[GH-ISSUE #1921] KTransformers Roadmap (2026 Q2) #1195

New issue

Open

opened 2026-10-29 14:49:47 +00:00 by bsakaguchi · 15 comments

bsakaguchi commented 2026-10-29 14:49:47 +00:00

Owner

Copy link

Originally created by @ErvinXie on GitHub (Apr 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1921

KTransformers Roadmap - 2026 Q2

Focus

• Fine-tuning: Make LoRA SFT run on the exact same hardware as inference — if you can run it, you can fine-tune it.
• Consumer Hardware: Deliver the best MoE inference performance on consumer-grade x86 + NVIDIA GPU.
• Windows: Native Windows support with full heterogeneous performance advantage.
• Performance: VNNI instruction set optimization and AI SSD exploration.

Fine-tuning Service @JimmyPeilinLi

Goal: Lower the barrier for LoRA SFT to match inference — same hardware, same setup, zero extra cost.

• Release fine-tuning service to main branch (basic version first, iterate later).
• Unify inference and SFT paths so both run on one set of hardware with shared model support, weight handling, and config.
• Support transformers v4 and v5, minimum 12G VRAM for 67B model. ✅

Consumer Hardware Support

Goal: Best-in-class MoE inference performance on consumer hardware.

• Windows native C++ inference support — investigate JIT compiler Windows compatibility.
• Evaluate S1 inference backend migration to Windows as alternative path.
• Complete VNNI instruction set optimization.
• Investigate AI SSD performance bottleneck (slow mmap reads from disk).
• Promote AVX2 to first-class supported tier.
• Improve support quality and stability for RTX 30/40/50 series.

Performance

• Optimize decode and prefill paths. @ouqingliang
• Optimize expert offloading and scheduling strategies. @ovowei
• Benchmark on representative AVX2 + RTX 30/40/50 configurations. @yyj6666667
• Focus on heterogeneous performance overhead in Windows path.
• Support nvfp4 and mxfp4 quantization formats (carried over from Q1).

Coverage

• Fast support for important frontier MoE models.
• Improve support quality and completeness for major model families.

Contribution / Maintenance

• Maintain CI/CD pipelines.
• Clearer documentation
• Establish recommended configurations for typical consumer hardware setups.
• Continue NUMA-aware optimization and CPU-GPU coordination improvements.

---

Any contribution is welcomed, please email yzwliam@126.com / ervinxie@qq.com if you want to join development WeChat group.

Originally created by @ErvinXie on GitHub (Apr 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1921 # KTransformers Roadmap - 2026 Q2 ## Focus - Fine-tuning: Make LoRA SFT run on the exact same hardware as inference — if you can run it, you can fine-tune it. - Consumer Hardware: Deliver the best MoE inference performance on consumer-grade x86 + NVIDIA GPU. - Windows: Native Windows support with full heterogeneous performance advantage. - Performance: VNNI instruction set optimization and AI SSD exploration. ## Fine-tuning Service @JimmyPeilinLi **Goal: Lower the barrier for LoRA SFT to match inference — same hardware, same setup, zero extra cost.** - Release fine-tuning service to main branch (basic version first, iterate later). - Unify inference and SFT paths so both run on one set of hardware with shared model support, weight handling, and config. - Support transformers v4 and v5, minimum 12G VRAM for 67B model. ✅ ## Consumer Hardware Support **Goal: Best-in-class MoE inference performance on consumer hardware.** - Windows native C++ inference support — investigate JIT compiler Windows compatibility. - Evaluate S1 inference backend migration to Windows as alternative path. - Complete VNNI instruction set optimization. - Investigate AI SSD performance bottleneck (slow mmap reads from disk). - Promote AVX2 to first-class supported tier. - Improve support quality and stability for RTX 30/40/50 series. ## Performance - Optimize decode and prefill paths. @ouqingliang - Optimize expert offloading and scheduling strategies. @ovowei - Benchmark on representative AVX2 + RTX 30/40/50 configurations. @yyj6666667 - Focus on heterogeneous performance overhead in Windows path. - Support nvfp4 and mxfp4 quantization formats (carried over from Q1). ## Coverage - Fast support for important frontier MoE models. - Improve support quality and completeness for major model families. ## Contribution / Maintenance - Maintain CI/CD pipelines. - Clearer documentation - Establish recommended configurations for typical consumer hardware setups. - Continue NUMA-aware optimization and CPU-GPU coordination improvements. --- Any contribution is welcomed, please email yzwliam@126.com / ervinxie@qq.com if you want to join development WeChat group.

bsakaguchi added the

enhancement

label 2026-10-29 14:49:47 +00:00

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@zklc commented on GitHub (Apr 9, 2026):

nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。

<!-- gh-comment-id:4214642711 --> @zklc commented on GitHub (Apr 9, 2026): nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@wanjunling168 commented on GitHub (Apr 13, 2026):

Windows native C++ inference support (JIT compiler) <- triton-windows probably enough?

<!-- gh-comment-id:4235276612 --> @wanjunling168 commented on GitHub (Apr 13, 2026): Windows native C++ inference support (JIT compiler) <- triton-windows probably enough?

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@rnett commented on GitHub (Apr 13, 2026):

nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。

Also interested (albeit mostly in the GPU side, since I have a blackwell GPU). It was in the Q1 roadmap, but wasn't checked off and the only related branch I see had its last commit 3 months ago.

<!-- gh-comment-id:4238074099 --> @rnett commented on GitHub (Apr 13, 2026): > nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。 Also interested (albeit mostly in the GPU side, since I have a blackwell GPU). It was in the Q1 roadmap, but wasn't checked off and the only related branch I see had its last commit 3 months ago.

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@aliez-ren commented on GitHub (Apr 18, 2026):

will be able to run rawint4 on avx2?

<!-- gh-comment-id:4274242484 --> @aliez-ren commented on GitHub (Apr 18, 2026): will be able to run rawint4 on avx2?

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@ErvinXie commented on GitHub (Apr 20, 2026):

@zklc @rnett Thanks for bringing this up. You're right that nvfp4 was on the Q1 roadmap but didn't get completed. We're planning to prioritize it in Q2 — along with mxfp4 support. Will update the roadmap accordingly. Stay tuned.

---

This reply was co-authored by an AI agent and a human.

<!-- gh-comment-id:4281558353 --> @ErvinXie commented on GitHub (Apr 20, 2026): @zklc @rnett Thanks for bringing this up. You're right that nvfp4 was on the Q1 roadmap but didn't get completed. We're planning to prioritize it in Q2 — along with mxfp4 support. Will update the roadmap accordingly. Stay tuned. --- *This reply was co-authored by an AI agent and a human.*

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@ErvinXie commented on GitHub (Apr 20, 2026):

@aliez-ren If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4.

That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 (#1926) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path.

---

This reply was co-authored by an AI agent and a human.

<!-- gh-comment-id:4281559609 --> @ErvinXie commented on GitHub (Apr 20, 2026): @aliez-ren If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4. That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 (#1926) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path. --- *This reply was co-authored by an AI agent and a human.*

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@yyj6666667 commented on GitHub (Apr 20, 2026):

@wanjunling168 Thanks for the pointer

Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA
C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch.

We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows
into the picture yet.

Thanks for pointing it out.

<!-- gh-comment-id:4281737852 --> @yyj6666667 commented on GitHub (Apr 20, 2026): @wanjunling168 Thanks for the pointer Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch. We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows into the picture yet. Thanks for pointing it out.

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@aliez-ren commented on GitHub (Apr 21, 2026):

@ErvinXie Thanks for your reply! My machine has 384G GDDR7 VRAM + 256G DDR4 RAM. I'm going to write a patch for running RAWINT4 on AVX2

<!-- gh-comment-id:4285108253 --> @aliez-ren commented on GitHub (Apr 21, 2026): @ErvinXie Thanks for your reply! My machine has 384G GDDR7 VRAM + 256G DDR4 RAM. I'm going to write a patch for running RAWINT4 on AVX2

bsakaguchi commented 2026-10-29 14:49:48 +00:00

Author

Owner

Copy link

@wanjunling168 commented on GitHub (Apr 21, 2026):

@wanjunling168 Thanks for the pointer

Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch.

We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows into the picture yet.

Thanks for pointing it out.

https://github.com/vllm-project/vllm/compare/main...SystemPanic:vllm-windows:vllm-for-windows
https://github.com/flashinfer-ai/flashinfer/compare/main...SystemPanic:flashinfer-windows:main

Seems we have some limited Windows ports for flashinfer and marlin kernels I found in this repo already (though they're meant for vllm), those marlin kernels in this repo can be good reference for you @yyj6666667 ?

<!-- gh-comment-id:4285277876 --> @wanjunling168 commented on GitHub (Apr 21, 2026): > [@wanjunling168](https://github.com/wanjunling168) Thanks for the pointer > > Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch. > > We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows into the picture yet. > > Thanks for pointing it out. `https://github.com/vllm-project/vllm/compare/main...SystemPanic:vllm-windows:vllm-for-windows` `https://github.com/flashinfer-ai/flashinfer/compare/main...SystemPanic:flashinfer-windows:main` Seems we have some limited Windows ports for flashinfer and marlin kernels I found in this repo already (though they're meant for vllm), those marlin kernels in this repo can be good reference for you @yyj6666667 ?

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@yyj6666667 commented on GitHub (Apr 21, 2026):

@wanjunling168 Definitely — those are good references, especially flashinfer-windows. We noticed it early on. Thanks for pointing it out.

<!-- gh-comment-id:4286264285 --> @yyj6666667 commented on GitHub (Apr 21, 2026): @wanjunling168 Definitely — those are good references, especially flashinfer-windows. We noticed it early on. Thanks for pointing it out.

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@aliez-ren commented on GitHub (Apr 23, 2026):

@aliez-ren If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4.

That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 (#1926) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path.

This reply was co-authored by an AI agent and a human.

created a pr to support rawint4 on avx2: https://github.com/kvcache-ai/ktransformers/pull/1942

<!-- gh-comment-id:4302741605 --> @aliez-ren commented on GitHub (Apr 23, 2026): > [@aliez-ren](https://github.com/aliez-ren) If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4. > > That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 ([#1926](https://github.com/kvcache-ai/ktransformers/pull/1926)) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path. > > _This reply was co-authored by an AI agent and a human._ created a pr to support rawint4 on avx2: https://github.com/kvcache-ai/ktransformers/pull/1942

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@willbed commented on GitHub (May 25, 2026):

我目前有个问题，当使用Hybird Prefill的时候，即kt-gpu-prefill-token-threshold 设为0， 此时cpu还是会加载全部权重至内存，DDR5的内存实在太贵了，目前只能用SSD swap解决这个问题。 这里是否能够只加载冷门专家至内存，我理解热门专家这个时候已经装载在显卡上，在内存里是完全没必要了。

<!-- gh-comment-id:4534374097 --> @willbed commented on GitHub (May 25, 2026): 我目前有个问题，当使用Hybird Prefill的时候，即kt-gpu-prefill-token-threshold 设为0， 此时cpu还是会加载全部权重至内存，DDR5的内存实在太贵了，目前只能用SSD swap解决这个问题。 这里是否能够只加载冷门专家至内存，我理解热门专家这个时候已经装载在显卡上，在内存里是完全没必要了。

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@WJwonjun commented on GitHub (Jun 6, 2026):

I ran a simple benchmark on an RTX 5000 GPU, but the performance was significantly poorer compared to the H100. I suspect this issue is due to the lack of sm120 support in your current SGLang fork. Are consumer-level GPUs (like the RTX 50/60 series) still unsupported, or did I configure something incorrectly?

<!-- gh-comment-id:4638129802 --> @WJwonjun commented on GitHub (Jun 6, 2026): I ran a simple benchmark on an RTX 5000 GPU, but the performance was significantly poorer compared to the H100. I suspect this issue is due to the lack of sm120 support in your current SGLang fork. Are consumer-level GPUs (like the RTX 50/60 series) still unsupported, or did I configure something incorrectly?

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@yyj6666667 commented on GitHub (Jun 6, 2026):

I ran a simple benchmark on an RTX 5000 GPU, but the performance was significantly poorer compared to the H100. I suspect this issue is due to the lack of sm120 support in your current SGLang fork. Are consumer-level GPUs (like the RTX 50/60 series) still unsupported, or did I configure something incorrectly?

For most models, the RTX50/60 is supported. Taking deepseek v4 as an example, we do have sm120 support in the sglang fork. The common reason why speed is slow is more likely ralated to the expert num on gpu, which is '' --kt-num-gpu-experts '' , the more expert you've got on gpu, the faster it would be.

<!-- gh-comment-id:4639118485 --> @yyj6666667 commented on GitHub (Jun 6, 2026): > I ran a simple benchmark on an RTX 5000 GPU, but the performance was significantly poorer compared to the H100. I suspect this issue is due to the lack of sm120 support in your current SGLang fork. Are consumer-level GPUs (like the RTX 50/60 series) still unsupported, or did I configure something incorrectly? For most models, the RTX50/60 is supported. Taking deepseek v4 as an example, we do have sm120 support in the sglang fork. The common reason why speed is slow is more likely ralated to the expert num on gpu, which is '' --kt-num-gpu-experts '' , the more expert you've got on gpu, the faster it would be.

bsakaguchi commented 2026-10-29 14:49:49 +00:00

Author

Owner

Copy link

@yyj6666667 commented on GitHub (Jun 6, 2026):

我目前有个问题，当使用Hybird Prefill的时候，即kt-gpu-prefill-token-threshold 设为0， 此时cpu还是会加载全部权重至内存，DDR5的内存实在太贵了，目前只能用SSD swap解决这个问题。 这里是否能够只加载冷门专家至内存，我理解热门专家这个时候已经装载在显卡上，在内存里是完全没必要了。

理论上可行， 实际可能需要考虑如下方面： 1. prefill 是计算密集型，多个token会散布到不同专家，gpu prefill过程中至少要保证模型单独一层的专家全量加载。 2. 专家从cpu ram加载到gpu通过一条三层流水线实现， 如果走ssd加载可能会显著拖慢流水线

<!-- gh-comment-id:4639204413 --> @yyj6666667 commented on GitHub (Jun 6, 2026): > 我目前有个问题，当使用Hybird Prefill的时候，即kt-gpu-prefill-token-threshold 设为0， 此时cpu还是会加载全部权重至内存，DDR5的内存实在太贵了，目前只能用SSD swap解决这个问题。 这里是否能够只加载冷门专家至内存，我理解热门专家这个时候已经装载在显卡上，在内存里是完全没必要了。 理论上可行， 实际可能需要考虑如下方面： 1. prefill 是计算密集型，多个token会散布到不同专家，gpu prefill过程中至少要保证模型单独一层的专家**全量加载**。 2. 专家从cpu ram加载到gpu通过一条三层流水线实现， 如果走ssd加载可能会显著拖慢流水线

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#1195

No description provided.