mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1816] Kimi 2.5 Bug and Performance #1135
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1135
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @trilog-inc on GitHub (Jan 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1816
Reminder
System Info
ktransformers - kimi_k2.5 branch
kvcache-ai/sglang - kimi_k2.5 branch
CPU: w7-3465x ( 28 physical cores, AMX enabled, 2 NUMA )
RAM: 768GB DDR5 5600
GPU: RTX 6000 pro
Reproduction
Congrats on the team for the day 0 support for Kimi 2.5! The multimodal feature is amazing.
I wanted to compare notes with other users running similar hardware ( RTX 6000 Pro, Xeon Sapphire Rapids ) and offer my feedback.
When I run the server with kt-cpuinfer matching the number of physical cores ( 28 ), I get about 10.3 token/s generation. When I go above the physical core counts, with for example 52, I get 11.8 token/s. Is this normal? I noticed in the log when I got above the number of physical cores, the logs print out "Core 14 inside NUMA node 0 not found ... to Core 25".
Are there any consequences to configuring the server this way?
Other note, the --kt-enable-dynamic-expert-update paramter increases the token generation to about 14token/s ( + 30-40%). I am looking forward to having this in the new kimi_2.5 branch!
Another note, the sglang server never outputs the opening tag. I imagine this is something to be fixed on the sglang side of the system. Will the kvcache-ai branch be able to be merged to the main one in the near term?
CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --host 0.0.0.0 --port 60000 --model /mnt/home_extend/models/kimi25/ --kt-weight-path /mnt/home_extend/models/kimi25/ --kt-cpuinfer 28 --kt-threadpool-count 2 --kt-num-gpu-experts 28 --kt-method RAWINT4 --attention-backend flashinfer --kt-gpu-prefill-token-threshold 400 --kt-max-deferred-experts-per-token 1 --trust-remote-code --mem-fraction-static 0.94 --chunked-prefill-size 16384 --max-running-requests 10 --max-total-tokens 115000 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion --served-model-name kimi --tool-call-parser kimi_k2 --reasoning-parser kimi_k2
Others
No response
@KebobZ commented on GitHub (Jan 28, 2026):
I have a similar setup (RTX 6000 Pro, 1TB DDR5, Xeon Sapphire Rapids).
Are you referring to the opening thinking tag? I noticed the same thing, but if I explicitly pass
{"chat_template_kwargs": {"thinking": true}}, it works fine. It's strange because if I don't, I see the closing</think>tag without the opening tag. Disabling thinking works as you would expect (no thinking and no opening or closing think tags).I am seeing another weird issue right now when I send a large prompt (like from Kimi Code or OpenCode), I am getting back a non-sensical response (the tokens are coherent in the sense that it seems related to the same topic, and I even see it trying to do tool calls sometimes, but it is completely unrelated to my prompt).
@trilog-inc commented on GitHub (Jan 28, 2026):
Yes exactly! Noted. I will try that as well.
I have noticed this as well. When I lower the kt-cpuinfer parameter to 20 or lower, I sometimes get garbled output ( ex "A B C D ..." or some unrelated output that eventually repeats itself.
@KebobZ What is your token generation speed like?
@KebobZ commented on GitHub (Jan 28, 2026):
I have 2x Xeon 8480+ (the QYFS ES) and the RTX 6000 Pro Max-Q, and I'm getting ~16.5t/s tg512. I'm in the process of testing Unsloth's
UD-Q4_K_XLquant to compare. This is how I'm running:That's how I was running Kimi K2 Thining. I haven't tried tweaking things yet.
@trilog-inc commented on GitHub (Jan 28, 2026):
Amazing. Please let me know what the UD quant yields. Otherwise, can you try increasing your cpuinfer count to ~78 to see how the performance changes? It boosted my tg by 20-30% for me.
@KebobZ commented on GitHub (Jan 28, 2026):
A modest gain for me. My tg512 went from 16.5 t/s -> 18 t/s going from cpuinfer 56 -> 78.
@trilog-inc commented on GitHub (Jan 28, 2026):
@KebobZ I've got a repository on my profile which proxies the request and modifies them if needed. In any case, it injects the thinking parameter to true when the model name contains "kimi", fixes the thinking tag bug for me when using Kilo Code etc
@KebobZ commented on GitHub (Jan 29, 2026):
Ah, we must think alike because I had to build something similar. I'll check it out! I did try out
ik_llama.cppand it worked well with OpenCode. Unfortunately it is quite a bit slower than using sglang + kt-kernel (~13t/s).@KebobZ commented on GitHub (Jan 30, 2026):
I did some more testing, and it seemed like it was related to
--kt-gpu-prefill-token-threshold, specifically when the prompt was above the threshold was when I could reliably reproduce the corruption issue. I tested running withCUDA_LAUNCH_BLOCKING=1and so far, that seems to have "fixed" the issue for me - at a performance penalty (though a lot faster than disabling the GPU-only fallback completely). It is much faster thanik_llama.cppwas though. I suspect there is a potential synchronization bug with the GPU-only fallback.@usrlocalben commented on GitHub (Feb 1, 2026):
I observed the same. If the layer-offload prefill mechanism is used, it executes impossibly fast (3000 tok/sec) and the output is gibberish.
@sousekd commented on GitHub (Mar 19, 2026):
It took me a while to make Kimi-K2.5 work on EPYC 9355 + 2x RTX PRO 6000.
For those who are still struggling, this is what worked for me:
Dockerfile
Run command
Performance (single user)
Performance (two concurrent users)
Both reasoning and function calling works well, but I have not tried on long context yet.
This is my first experience with SGLang + KT (and
aiperf).Tips to improve my "recipe to run" for better performance would be very welcome.
@usrlocalben commented on GitHub (Mar 30, 2026):
@sousekd thanks for posting. I'm not sure if it's due to cuda13 or a couple of new params I took from your invocation but I get a 5% bump in decode throughput over my cuda12 setup (epyc 9b14 + 1x rtx 6000 pro).
@ErvinXie commented on GitHub (Mar 31, 2026):
Thanks for the detailed report and for sharing your setup!
On kt-cpuinfer above physical core count: The "Core not found" warnings are expected when you set kt-cpuinfer higher than the number of physical cores in a NUMA node — the system simply skips the invalid core IDs. It's generally fine to go above the physical core count up to a reasonable limit; the thread scheduler will handle it. That said, setting it to match physical cores per NUMA node is the cleaner configuration.
On
--kt-enable-dynamic-expert-update: This feature is currently on the Kimi K2.5 branch. We plan to upstream it to the main branch — no specific timeline yet, but it's on our radar.On the missing
<think>opening tag: As @KebobZ noted, passing{"chat_template_kwargs": {"thinking": true}}in your request resolves this. This is a known behavior with the current chat template handling.On garbled output with large prompts (@KebobZ): This could be related to the prefill token threshold — when a very long prompt triggers the CPU-side prefill fallback, there can be edge cases. Could you share your full launch command and an example of the garbled output? That would help us narrow it down.
This reply was co-authored by an AI agent and a human.
@usrlocalben commented on GitHub (Mar 31, 2026):
@ErvinXie
Does this feature work? Does one just checkout the branch? branch kimi_k2.5 is 2 months old, 45 commits behind main at this time.
@ErvinXie commented on GitHub (Apr 9, 2026):
The
--kt-enable-dynamic-expert-updatefeature has already been merged into main. You can use it directly on the main branch, no need to checkout the kimi_k2.5 branch.This reply was co-authored by an AI agent and a human.
@kz0ltan commented on GitHub (Apr 26, 2026):
Before downloading and setting all this up, could somebody tell me if this will work with 512GB RAM and 2xRTX 6000 Pro (so the question is whether RAM is a hard requirement or RAM+VRAM works as well)? Thanks!
@sousekd commented on GitHub (Apr 26, 2026):
I don't think so. I see 600 GB RAM used after loading the model on one GPU with
--kt-num-gpu-experts 8. You can increase the number of offloaded experts with two PRO's 6000, but I don't think it will bypass the loading to RAM.That said, if you want to try, here is my current Dockerfile, updated just a couple of days ago:
If it doesn't work, use llama.cpp or ik_llama.cpp. The latter actually runs this model faster than KTransformers for a single user. I posted some comparison of inference engines here: https://huggingface.co/ubergarm/Kimi-K2.6-GGUF/discussions/3
@shrould8888 commented on GitHub (May 16, 2026):
you have to take advantage of the Intel AMX instructions which makes the prefill speed 10x faster.
but first you have to convert the INT4 weights to AMXINT4 weights.
I have found the solution to do the converstion and hope it could be useful to someone:
convert Kimi K2.X’s native INT4 weights to BF16 by using the script int4_cast_bf16_fixed.py from this project: https://github.com/csabakecskemeti/DQ_int4-to-bf16_dequant
using ktransformers/kt-kernel/scripts/convert_cpu_weights.py to convert bf16 weights to AMXINT4 weights.
I have successfully converted Kimi K2.6 INT4 weights to AMXINT4, it works flawlessly.