[GH-ISSUE #5] Ablation study? #3

New issue

Closed

opened 2026-10-29 14:32:32 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:32:32 +00:00

Owner

Originally created by @jukofyork on GitHub (Jul 27, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/5

It would be interesting to see the effect of each optimisation separately.

Some of the ideas might be very easy to use in other inference engines and others much harder to implement, so it would be very helpful to get an idea of the gains for each feature separately.

I've linked this to the llama.cpp discussions:

https://github.com/ggerganov/llama.cpp/discussions/8721

As the "Arithmetic Intensity Guided Offloading" is clearly something they could do quite easily and likely give a significant boost to MoE models.

Originally created by @jukofyork on GitHub (Jul 27, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/5 It would be interesting to see the effect of each optimisation separately. Some of the ideas might be very easy to use in other inference engines and others much harder to implement, so it would be very helpful to get an idea of the gains for each feature separately. I've linked this to the `llama.cpp` discussions: https://github.com/ggerganov/llama.cpp/discussions/8721 As the "Arithmetic Intensity Guided Offloading" is clearly something they could do quite easily and likely give a significant boost to MoE models.

bsakaguchi

2026-10-29 14:32:32 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:32:33 +00:00

Author

Owner

@james0zan commented on GitHub (Jul 27, 2024):

Yes, we will present more results later. Actually, our vision for KTransformers is to serve as an experimental platform used to develop prototypes faster. The more mature ones can then be adopted into more popular inference engines like llama.cpp and vLLM.

@james0zan commented on GitHub (Jul 27, 2024): Yes, we will present more results later. Actually, our vision for KTransformers is to serve as an experimental platform used to develop prototypes faster. The more mature ones can then be adopted into more popular inference engines like llama.cpp and vLLM.

bsakaguchi commented

2026-10-29 14:32:33 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

bsakaguchi referenced this issue

2026-10-29 14:37:20 +00:00

[GH-ISSUE #420] deepseek r1 using 4090 nvidia running, meet segmentation fault(core dumped) problem #288

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:38:09 +00:00

[GH-ISSUE #484] Docker run local chat raise error, please confirm whether AMD CPU can run ktransformers #342

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646