kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1608] 🚨FAQs | 常见问题🚨 #1048

New issue
Open
opened 2026-10-29 14:47:55 +00:00 by bsakaguchi · 20 comments
bsakaguchi commented 2026-10-29 14:47:55 +00:00
Owner
Copy link

Originally created by @KMSorSMS on GitHub (Nov 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1608

Note

Please avoid creating issues regarding the following questions, as they might be closed without a response.
请避免创建与下述问题有关的 issues，这些 issues 可能不会被回复。

Tip

Documentation: https://kvcache-ai.github.io/ktransformers/
中文文档：https://github.com/kvcache-ai/ktransformers/tree/main/doc/zh

Most of problems / 大多数问题

How to install kt-kernel/ 怎么安装 kt-kernel

Please update repository and install again following the link below:
请参考文档进行安装，以及常见问题指导：
https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README.md

Inference with kt-kernel+Sglang/ 在 kt-kernel + Sglang 上推理

1. Install SGLang
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"
2. Prepare Weights

You need both GPU weights and CPU weights for heterogeneous inference:

GPU Weights: Use the original / quantized model weights.

CPU Weights: Quantize to AMX-optimized format using the conversion script:

python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \  # Depends on your GPU weights type: fp8, fp16, or bf16
  --output /path/to/cpu-weights \
  --quant-method int8  # or int4

Supported input formats: FP8, FP16, BF16 → INT4/INT8.

For more details, see:

Note: LLAMAFILE backend supports GGUF format directly, but this feature is still in preview.

3. Launch SGLang Server

Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference:

KT-Kernel Parameters to Add:

  • --kt-method: Backend method (AMXINT4, AMXINT8, or LLAMAFILE)
  • --kt-weight-path: Path to the converted CPU weights
  • --kt-cpuinfer: Number of CPU inference threads (set to physical cores)
  • --kt-threadpool-count: Number of thread pools (set to NUMA node count)
  • --kt-num-gpu-experts: Number of experts to keep on GPU
  • --kt-max-deferred-experts-per-token: Deferred experts for pipelined execution

Example:

python -m sglang.launch_server \
  [your normal SGLang parameters...] \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/cpu-weights \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

Support models/ 支持的模型列表

Model_name/ 模型名称 have_tested/ 是否测试过
GLM-4.5-Air ✅
GLM-4.5 ✅
Qwen3-30B-A3B ✅
Qwen3-235B-A22B-Thinking-2507 ✅
Qwen3-235B-A22B-Instruct-2507 ✅
Qwen3-Next-80B-A3B-Thinking ✅
DeepSeek-R1-0528 ✅

Note

In principle, we can support any model that SGLang supports. If a model cannot run, then SGLang running purely on GPU likely won't be able to either. So if you test additional models that work, feel free to share them.

Known Issues & Tips / 已知问题与提示

"KT fallback" log messages are normal / "KT fallback" 日志是正常的

If you see logs like KT fallback: layer X prepare weight = XXms, compute = XXms, this is expected behavior indicating that Layer-wise Prefill is working correctly. The "fallback" naming is misleading and will be updated in a future release.

如果看到类似 KT fallback: layer X prepare weight = XXms 的日志，这是预期行为，说明 Layer-wise Prefill 已正常启动。"fallback" 这个名称有误导性，后续版本会优化。

Low GPU utilization during prefill / Prefill 阶段 GPU 利用率低

If your GPU utilization is low during prefill, enable Layer-wise Prefill by adding:

--kt-gpu-prefill-token-threshold 2048

This allows the GPU to participate in prefill computation, significantly improving prefill throughput.

如果 prefill 阶段 GPU 利用率很低，请添加 --kt-gpu-prefill-token-threshold 2048 参数开启 Layer-wise Prefill。

GPTQ_INT4 + tensor-parallel-size > 1 / GPTQ_INT4 与多卡 TP 不兼容

GPTQ-Int4 quantized MoE models may fail with IndexError: start out of range when using tensor-parallel-size > 1 with GPU experts. This is an SGLang-side weight loader issue. Workarounds:

  • Use --tensor-parallel-size 1
  • Use FP8 weights instead of GPTQ-Int4

GPTQ-Int4 量化的 MoE 模型在 tensor-parallel-size > 1 且使用 GPU experts 时可能报错。这是 SGLang 侧的问题。建议降低 TP 或使用 FP8 权重。

Tip

If the problems still exist with the latest code, please create an issue.
若使用最新的代码仍然无法解决问题，请创建一个 issue。

Originally created by @KMSorSMS on GitHub (Nov 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1608 > [!NOTE] > Please **avoid** creating issues regarding the following questions, as they might be closed without a response. > 请**避免**创建与下述问题有关的 issues，这些 issues 可能不会被回复。 > [!TIP] > Documentation: https://kvcache-ai.github.io/ktransformers/ > 中文文档：https://github.com/kvcache-ai/ktransformers/tree/main/doc/zh ---- ### Most of problems / 大多数问题 #### How to install kt-kernel/ 怎么安装 kt-kernel Please update repository and install again following the link below: 请参考文档进行安装，以及常见问题指导： https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README.md #### Inference with kt-kernel+Sglang/ 在 kt-kernel + Sglang 上推理 ##### 1. Install SGLang ```bash git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[all]" ``` ##### 2. Prepare Weights You need both GPU weights and CPU weights for heterogeneous inference: **GPU Weights:** Use the original / quantized model weights. **CPU Weights:** Quantize to AMX-optimized format using the conversion script: ```bash python scripts/convert_cpu_weights.py \ --input-path /path/to/model \ --input-type bf16 \ # Depends on your GPU weights type: fp8, fp16, or bf16 --output /path/to/cpu-weights \ --quant-method int8 # or int4 ``` **Supported input formats:** FP8, FP16, BF16 → INT4/INT8. For more details, see: - [CPU Weights conversion](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README.md#cpu-weights-for-cold-experts-on-cpu) - [GPU Weights quantization](https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README.md#gpu-weights-for-hot-experts-on-gpu) **Note:** LLAMAFILE backend supports GGUF format directly, but this feature is still in preview. ##### 3. Launch SGLang Server Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference: **KT-Kernel Parameters to Add:** - `--kt-method`: Backend method (AMXINT4, AMXINT8, or LLAMAFILE) - `--kt-weight-path`: Path to the converted CPU weights - `--kt-cpuinfer`: Number of CPU inference threads (set to physical cores) - `--kt-threadpool-count`: Number of thread pools (set to NUMA node count) - `--kt-num-gpu-experts`: Number of experts to keep on GPU - `--kt-max-deferred-experts-per-token`: Deferred experts for pipelined execution Example: ```bash python -m sglang.launch_server \ [your normal SGLang parameters...] \ --kt-method AMXINT8 \ --kt-weight-path /path/to/cpu-weights \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ``` ---- ### Support models/ 支持的模型列表 | Model_name/ 模型名称 | have_tested/ 是否测试过 | |----------------------------------------|------------| | GLM-4.5-Air | :white_check_mark: | | GLM-4.5 | :white_check_mark: | | Qwen3-30B-A3B | :white_check_mark: | | Qwen3-235B-A22B-Thinking-2507 | :white_check_mark: | | Qwen3-235B-A22B-Instruct-2507 | :white_check_mark: | | Qwen3-Next-80B-A3B-Thinking | :white_check_mark: | | DeepSeek-R1-0528 | :white_check_mark: | >[!NOTE] In principle, we can support any model that SGLang supports. If a model cannot run, then SGLang running purely on GPU likely won't be able to either. So if you test additional models that work, feel free to share them. ---- ### Known Issues & Tips / 已知问题与提示 #### "KT fallback" log messages are normal / "KT fallback" 日志是正常的 If you see logs like `KT fallback: layer X prepare weight = XXms, compute = XXms`, this is **expected behavior** indicating that Layer-wise Prefill is working correctly. The "fallback" naming is misleading and will be updated in a future release. 如果看到类似 `KT fallback: layer X prepare weight = XXms` 的日志，这是**预期行为**，说明 Layer-wise Prefill 已正常启动。"fallback" 这个名称有误导性，后续版本会优化。 #### Low GPU utilization during prefill / Prefill 阶段 GPU 利用率低 If your GPU utilization is low during prefill, enable Layer-wise Prefill by adding: ```bash --kt-gpu-prefill-token-threshold 2048 ``` This allows the GPU to participate in prefill computation, significantly improving prefill throughput. 如果 prefill 阶段 GPU 利用率很低，请添加 `--kt-gpu-prefill-token-threshold 2048` 参数开启 Layer-wise Prefill。 #### GPTQ_INT4 + tensor-parallel-size > 1 / GPTQ_INT4 与多卡 TP 不兼容 GPTQ-Int4 quantized MoE models may fail with `IndexError: start out of range` when using `tensor-parallel-size > 1` with GPU experts. This is an SGLang-side weight loader issue. Workarounds: - Use `--tensor-parallel-size 1` - Use FP8 weights instead of GPTQ-Int4 GPTQ-Int4 量化的 MoE 模型在 `tensor-parallel-size > 1` 且使用 GPU experts 时可能报错。这是 SGLang 侧的问题。建议降低 TP 或使用 FP8 权重。 ---- > [!TIP] > If the problems still exist with the latest code, please create an issue. 若使用最新的代码仍然无法解决问题，请创建一个 issue。
bsakaguchi commented 2026-10-29 14:47:56 +00:00
Author
Owner
Copy link

@james0zan commented on GitHub (Nov 13, 2025):

We need a tracking matrix of model support.
For example, the currenct staus of GLM series model, what kind of weight are supported, what are not.

<!-- gh-comment-id:3527765158 --> @james0zan commented on GitHub (Nov 13, 2025): We need a tracking matrix of model support. For example, the currenct staus of GLM series model, what kind of weight are supported, what are not.
bsakaguchi commented 2026-10-29 14:47:56 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Nov 13, 2025):

You are welcome to report new models that are supported in the kt_kernel with your snapshot to prove

<!-- gh-comment-id:3527841912 --> @KMSorSMS commented on GitHub (Nov 13, 2025): You are welcome to report new models that are supported in the kt_kernel with your snapshot to prove
bsakaguchi commented 2026-10-29 14:47:56 +00:00
Author
Owner
Copy link

@kiankyars commented on GitHub (Dec 22, 2025):

I'm not too sure what the issue is here. Is this just an FAQ because I'm looking for a good first issue to contribute? I

<!-- gh-comment-id:3680629019 --> @kiankyars commented on GitHub (Dec 22, 2025): I'm not too sure what the issue is here. Is this just an FAQ because I'm looking for a good first issue to contribute? I
bsakaguchi commented 2026-10-29 14:47:56 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Dec 22, 2025):

@kiankyars Yes, this is just an FAQ. This FAQ contains answers to common questions that may help you. If you can’t find what you’re looking for here, please feel free to open a new issue.

<!-- gh-comment-id:3680730232 --> @KMSorSMS commented on GitHub (Dec 22, 2025): @kiankyars Yes, this is just an FAQ. This FAQ contains answers to common questions that may help you. If you can’t find what you’re looking for here, please feel free to open a new issue.
bsakaguchi commented 2026-10-29 14:47:56 +00:00
Author
Owner
Copy link

@wqshmzh commented on GitHub (Apr 12, 2026):

One has to prepare CPU+GPU models, which means he or she is supposed to download the model twice, right ? So that one model occupies the disk for two times. It is very space consuming. Previous versions of ktransformers do not require doing so. Why things changed after updates ?

<!-- gh-comment-id:4231467048 --> @wqshmzh commented on GitHub (Apr 12, 2026): One has to prepare CPU+GPU models, which means he or she is supposed to download the model twice, right ? So that one model occupies the disk for two times. It is very space consuming. Previous versions of ktransformers do not require doing so. Why things changed after updates ?
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Apr 12, 2026):

I think this is no longer necessary if you use the original-precision mode. Previously, two sets of weights were needed because the INT4/INT8 weights are not the same as the original weights. In low-precision mode, we use INT4/INT8 weights for the MoE experts, while the rest of the model stays unchanged, so both versions had to be kept. If you run in original-precision mode, such as rawint4, then this is no longer needed.

<!-- gh-comment-id:4231494167 --> @KMSorSMS commented on GitHub (Apr 12, 2026): I think this is no longer necessary if you use the original-precision mode. Previously, two sets of weights were needed because the INT4/INT8 weights are not the same as the original weights. In low-precision mode, we use INT4/INT8 weights for the MoE experts, while the rest of the model stays unchanged, so both versions had to be kept. If you run in original-precision mode, such as rawint4, then this is no longer needed.
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@wqshmzh commented on GitHub (Apr 21, 2026):

@KMSorSMS I dont have enough resources to run full-precision models. I always use unsloth's UD models, the precisions of moe and other parts are different in UD models. So do I need to download the original full precision model and unsloth's model twice now?

<!-- gh-comment-id:4285579210 --> @wqshmzh commented on GitHub (Apr 21, 2026): @KMSorSMS I dont have enough resources to run full-precision models. I always use unsloth's UD models, the precisions of moe and other parts are different in UD models. So do I need to download the original full precision model and unsloth's model twice now?
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Apr 21, 2026):

@wqshmzh I see — yes, in your case you'll need both. The original full-precision model is required (only its non-MoE parts are actually used, but there's no way to download just those, so the full model has to be pulled down), and Unsloth's UD model is needed for the MoE parts. So both downloads are necessary.

<!-- gh-comment-id:4285729182 --> @KMSorSMS commented on GitHub (Apr 21, 2026): @wqshmzh I see — yes, in your case you'll need both. The original full-precision model is required (only its non-MoE parts are actually used, but there's no way to download just those, so the full model has to be pulled down), and Unsloth's UD model is needed for the MoE parts. So both downloads are necessary.
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@mytbk commented on GitHub (May 28, 2026):

好像没什么关于现在的 SGLang+kt-kernel 架构的文档。我不清楚 kt-kernel 是专用于 CPU 还是也支持 GPU 的，我看 kt-kernel 的 setup.py 和 CMakeLists.txt 里面还提到了 CUDA/MUSA/ROCM，而且在 CUDA 环境的确会调用 nvcc. 另外 llama.cpp 的集成也看得不是很懂，是用来支持 GGUF 相关量化的吗？
我现在的主要部署方式是在自己的机器上先配好 venv 安装，再把 venv 挪到生产环境，所以想知道要构建 kt 有哪些是必须要构建的。

<!-- gh-comment-id:4561711443 --> @mytbk commented on GitHub (May 28, 2026): 好像没什么关于现在的 SGLang+kt-kernel 架构的文档。我不清楚 kt-kernel 是专用于 CPU 还是也支持 GPU 的，我看 kt-kernel 的 setup.py 和 CMakeLists.txt 里面还提到了 CUDA/MUSA/ROCM，而且在 CUDA 环境的确会调用 nvcc. 另外 llama.cpp 的集成也看得不是很懂，是用来支持 GGUF 相关量化的吗？ 我现在的主要部署方式是在自己的机器上先配好 venv 安装，再把 venv 挪到生产环境，所以想知道要构建 kt 有哪些是必须要构建的。
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (May 29, 2026):

@mytbk https://ktransformers.net/zh/docs

<!-- gh-comment-id:4569532856 --> @KMSorSMS commented on GitHub (May 29, 2026): @mytbk https://ktransformers.net/zh/docs
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@cxj01 commented on GitHub (Aug 10, 2026):

@KMSorSMS SFT时要求cpu要支持amx指令集？只支持avx指令集的cpu训练不了？

<!-- gh-comment-id:5237008803 --> @cxj01 commented on GitHub (Aug 10, 2026): @KMSorSMS SFT时要求cpu要支持amx指令集？只支持avx指令集的cpu训练不了？
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Aug 10, 2026):

应该不是的，目前能支持 avx 了。 @JimmyPeilinLi

<!-- gh-comment-id:5237068457 --> @KMSorSMS commented on GitHub (Aug 10, 2026): 应该不是的，目前能支持 avx 了。 @JimmyPeilinLi
bsakaguchi commented 2026-10-29 14:47:57 +00:00
Author
Owner
Copy link

@cxj01 commented on GitHub (Aug 10, 2026):

@KMSorSMS @JimmyPeilinLi 目前哪个版本之后开始支持，我安装了ktransformers==0.6.4 还不支持。

<!-- gh-comment-id:5237098888 --> @cxj01 commented on GitHub (Aug 10, 2026): @KMSorSMS @JimmyPeilinLi 目前哪个版本之后开始支持，我安装了ktransformers==0.6.4 还不支持。
bsakaguchi commented 2026-10-29 14:47:58 +00:00
Author
Owner
Copy link

@cxj01 commented on GitHub (Aug 10, 2026):

@KMSorSMS @JimmyPeilinLi
[rank0]: File "/aiproject/name/lixiufeng/miniconda3/envs/kt-sft/lib/python3.11/site-packages/kt_kernel/sft/amx.py", line 90, in init
[rank0]: raise RuntimeError(
[rank0]: RuntimeError: AMX SFT backend not available. kt_kernel_ext was not compiled with AMX SFT support.
[rank0]: Please recompile with AMX SFT enabled.

<!-- gh-comment-id:5237186802 --> @cxj01 commented on GitHub (Aug 10, 2026): @KMSorSMS @JimmyPeilinLi [rank0]: File "/aiproject/name/lixiufeng/miniconda3/envs/kt-sft/lib/python3.11/site-packages/kt_kernel/sft/amx.py", line 90, in __init__ [rank0]: raise RuntimeError( [rank0]: RuntimeError: AMX SFT backend not available. kt_kernel_ext was not compiled with AMX SFT support. [rank0]: Please recompile with AMX SFT enabled.
bsakaguchi commented 2026-10-29 14:47:58 +00:00
Author
Owner
Copy link

@JimmyPeilinLi commented on GitHub (Aug 10, 2026):

@cxj01 我认为可能是需要更新 llamafactory 的代码。https://github.com/hiyouga/LlamaFactory/pull/10738 是我们目前在 llamafactory 提交 AVX 的支持 PR，我们双方正在联合 review。后续合入会更新一版 cookbook，并发布新的 kt 的 pypi 包以确保支持

<!-- gh-comment-id:5237409434 --> @JimmyPeilinLi commented on GitHub (Aug 10, 2026): @cxj01 我认为可能是需要更新 llamafactory 的代码。https://github.com/hiyouga/LlamaFactory/pull/10738 是我们目前在 llamafactory 提交 AVX 的支持 PR，我们双方正在联合 review。后续合入会更新一版 cookbook，并发布新的 kt 的 pypi 包以确保支持
bsakaguchi commented 2026-10-29 14:47:58 +00:00
Author
Owner
Copy link

@JimmyPeilinLi commented on GitHub (Aug 10, 2026):

@cxj01 补充一下，avx的原精度微调应该是在v0.7 之后才支持的功能，v0.6.4 的 kt 应该不支持，可能需要源码编译一下目前。

<!-- gh-comment-id:5237522097 --> @JimmyPeilinLi commented on GitHub (Aug 10, 2026): @cxj01 补充一下，avx的原精度微调应该是在v0.7 之后才支持的功能，v0.6.4 的 kt 应该不支持，可能需要源码编译一下目前。
bsakaguchi commented 2026-10-29 14:47:58 +00:00
Author
Owner
Copy link

@cxj01 commented on GitHub (Aug 10, 2026):

@JimmyPeilinLi 直接拉最新的master（kt）代码，编译安装即可吧？不用切到特定的提交上吧？

<!-- gh-comment-id:5237927027 --> @cxj01 commented on GitHub (Aug 10, 2026): @JimmyPeilinLi 直接拉最新的master（kt）代码，编译安装即可吧？不用切到特定的提交上吧？
bsakaguchi commented 2026-10-29 14:47:58 +00:00
Author
Owner
Copy link

@yyj6666667 commented on GitHub (Aug 10, 2026):

@JimmyPeilinLi 直接拉最新的master（kt）代码，编译安装即可吧？不用切到特定的提交上吧？

对，后面直接拉取编译就行， 现在还需要等LlamaFactory 合入， 我们也在做相关的清理

<!-- gh-comment-id:5242659066 --> @yyj6666667 commented on GitHub (Aug 10, 2026): > [@JimmyPeilinLi](https://github.com/JimmyPeilinLi) 直接拉最新的master（kt）代码，编译安装即可吧？不用切到特定的提交上吧？ 对，后面直接拉取编译就行， 现在还需要等LlamaFactory 合入， 我们也在做相关的清理
bsakaguchi commented 2026-10-29 14:47:59 +00:00
Author
Owner
Copy link

@CodeIntegrity commented on GitHub (Aug 27, 2026):

docker镜像很久没更新了，看工作流好像是报错，是否能更新一下

<!-- gh-comment-id:5436735571 --> @CodeIntegrity commented on GitHub (Aug 27, 2026): docker镜像很久没更新了，看工作流好像是报错，是否能更新一下
bsakaguchi commented 2026-10-29 14:47:59 +00:00
Author
Owner
Copy link

@yyj6666667 commented on GitHub (Aug 27, 2026):

docker后面我们重新配置CI， 目前请先用pip install

<!-- gh-comment-id:5437514003 --> @yyj6666667 commented on GitHub (Aug 27, 2026): docker后面我们重新配置CI， 目前请先用pip install
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1048
No description provided.