mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1866] Gibberish output when using GPU offload prefill with RAWINT4 Kimi K2.5 #1163
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1163
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @DocShotgun on GitHub (Feb 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1866
Originally assigned to: @ouqingliang on GitHub.
Reminder
System Info
2x 5th-gen Xeon Platinum 64c 128t with 768gb
1x RTX Pro 6000 Workstation Edition 96gb
Reproduction
I've followed the instructions in the guide for setting up Kimi-K2.5 native QAT weights using your fork of sglang (
github.com/kvcache-ai/sglang@a864420ee4) with ktransformersgithub.com/kvcache-ai/ktransformers@16a8b98f3e.Launch command:
I noticed that on generation, I was getting nonsense/garbage outputs, however if I sent a short enough prompt, the outputs were coherent. My guess was that the problem was related to the GPU offload prefill for large prompts. I re-launched the server, changing
--kt-gpu-prefill-token-thresholdto 999999 so that it would never trigger, and it was able to produce coherent output (although the prompt processing is very slow without full GPU offload and without AMX kernels).Others
No response
@shrould8888 commented on GitHub (Feb 26, 2026):
I have similar config(Xeon5 64 cores x1 + 768GB RAM + RTX PRO 6000 96GB) and use 10 feb 2026 github source code (ktransformation+sglang):
python -m sglang.launch_server
--host 0.0.0.0
--port 10005
--model /Kimi-K2.5
--kt-weight-path /Kimi-K2.5
--kt-cpuinfer 60
--kt-threadpool-count 1
--kt-num-gpu-experts 0
--kt-method RAWINT4 -chunked-prefill-size
--trust-remote-code
--mem-fraction-static 0.94
--served-model-name Kimi-K2.5
--enable-mixed-chunk
--tensor-parallel-size 1
--enable-p2p-check
--disable-shared-experts-fusion
--chunked-prefill-size 32768
--max-total-tokens 256000
--attention-backend flashinfer
--sleep-on-idle
--reasoning-parser kimi_k2
--tool-call-parser kimi_k2 --kt-gpu-prefill-token-threshold 500 --kt-enable-dynamic-expert-update
But I don't have any problem with running k2.5. The main difference is I don't use "PYTORCH_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0" and my --chunked-prefill-size is 32768 (in other model, changing this number may generate garbage output. (e.g. when using Deepseek V3.2 AMX weights, if I set it to 32768 something like that , the model will produce garbage output, have to set it to 4096) . you could use "kt" command to run the model, it should generate workable arguments/options.
@DocShotgun commented on GitHub (Feb 26, 2026):
I tried removing thos env vars and also adjusting the chunked prefill size to match yours, but it still produces gibberish with GPU offload prefill.
I noticed another difference is that I'm using 4 CPU threadpools (two sockets with 2 NUMA nodes per socket) while you're only using 1, while we both only have 1 GPU. So when I'm doing GPU offload prefill, the CPU weights that were previously split for TP=4 have to be re-combined to run on the single GPU. If done incorrectly, this could definitely cause gibberish. Perhaps this could be the source of error?
@trilog-inc commented on GitHub (Feb 26, 2026):
I am also interested in this issue. The original version of kt-kernel which supported K2.5 did not have this problem, however pulling and installing the latest does.
@DocShotgun commented on GitHub (Feb 26, 2026):
Do you happen to know what the last working commit was on your end?
@trilog-inc commented on GitHub (Feb 26, 2026):
On ktransformers:
Branch: origin/kimi_k2.5
Commit:
b941447a83On SGlang
Branch: origin/kimi_k2.5 ( manually updated xgrammar==0.1.29 )
Commit: 317ee674ee78b6c684ba342d054d65c7438d021b
Command:
CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=1
python -m sglang.launch_server --host 0.0.0.0 --port 60000
--model /mnt/home_extend/models/kimi25/
--kt-weight-path /mnt/home_extend/models/kimi25/
--kt-cpuinfer 52 --kt-threadpool-count 2 --kt-num-gpu-experts 26
--kt-method RAWINT4 --attention-backend flashinfer --kt-gpu-prefill-token-threshold 400
--kt-max-deferred-experts-per-token 1 --trust-remote-code
--mem-fraction-static 0.94 --chunked-prefill-size 16384 --max-running-requests 4
--max-total-tokens 120000 --enable-mixed-chunk --tensor-parallel-size 1 --enable-p2p-check --disable-shared-experts-fusion --served-model-name kimi
--tool-call-parser kimi_k2 --reasoning-parser kimi_k2
@DocShotgun commented on GitHub (Feb 26, 2026):
Might be worth doing a
git bisect, although it wouldn't exactly be quick to iterate lol.@trilog-inc commented on GitHub (Feb 26, 2026):
Won't be easy :s Although I love K2.5 and its multi modal capabilities, I have moved on to GLM5 as the function parsers are more mature. Even with those working branches, K2.5 would fail in inconsistent ways especially above 80K context. Although half the speed, GLM5 has been very solid
@ErvinXie commented on GitHub (Feb 28, 2026):
Thanks for the detailed report. We previously fixed a related bug in the GPU offload prefill path. Could you try using the latest
mainbranch for both KTransformers and SGLang and see if the issue is resolved?As a temporary workaround, setting
--kt-gpu-prefill-token-thresholdto a very large value (effectively disabling GPU prefill) should produce correct outputs, as you have already confirmed.Please let us know if the issue persists after updating.
This reply was generated by an AI agent and reviewed by a human.
@DocShotgun commented on GitHub (Feb 28, 2026):
Thanks for the update! I just tried the current latest commits of
github.com/kvcache-ai/ktransformers@9e69fccb02github.com/kvcache-ai/sglang@a8b821aee8and unfortunately I'm not able to get far enough to test this functionality due to a new error.During CUDA graph capture, JIT compilation of the fused marlin kernel fails due to an unexpected indent:
@ErvinXie commented on GitHub (Mar 1, 2026):
Thanks for the detailed follow-up and the traceback, @DocShotgun.
We've identified the root cause. The recent merge from upstream SGLang into our fork introduced two issues:
fused_marlin_moe.pyIndentationError — A merge conflict was not properly resolved, leaving an orphaned indented import on line 16. This is the error you encountered.RAWINT4 quantization path change — The upstream merge replaced the MoE quantization scheme (
compressed_tensors_wNa16→compressed_tensors_wNa16_moe), which now uses a JIT-compiled marlin MoE kernel (moe_wna16_marlin_gemm). This new kernel path has compatibility issues with CUDA graph capture on Blackwell (SM120) GPUs, causing thecudaErrorIllegalAddressafter the IndentationError is fixed.This is consistent with @trilog-inc's finding that the
kimi_k2.5branch (commit317ee674ee) still works — that branch predates the upstream merge and uses the original marlin MoE path.We are working on a fix. In the meantime, please use the
kimi_k2.5branch for both KTransformers and SGLang to run Kimi K2.5:We'll update this issue once the fix is merged to main.
This reply was co-authored by an AI agent and a human.
@ErvinXie commented on GitHub (Mar 1, 2026):
The fix has been pushed to kvcache-ai/sglang main branch:
8d06c33.Root cause
Three bugs introduced by the upstream SGLang merge (
a2f451315):fused_marlin_moe.pyIndentationError — merge conflict left imports outside theif _is_cuda:block, causingIndentationErroron import.fused_marlin_moe.pycrash on empty expert weights — when--kt-num-gpu-experts 0, GPU expert weight tensors have shape[0, ...]. The marlin MoE kernel crashes on these empty inputs. Fixed by returning zeros early whenE=0so CPU experts provide the full result.deepseek_v2.pydual-stream conflict with KT wrapper —forward_normal_dual_streamusesalt_streamfor shared expert parallelism, which conflicts with KT wrapper's internal_cpu_streamduring CUDA graph capture. Fixed by skipping dual-stream path when using KT wrapper.How to update
Verified on
--kt-num-gpu-experts 0 --kt-method RAWINT4 --attention-backend tritonWhat is 2+3?→5(correct reasoning + answer)Please update both SGLang and KTransformers to the latest main branch and let us know if the issue persists.
This reply was co-authored by an AI agent and a human.
@DocShotgun commented on GitHub (Mar 1, 2026):
Pulled everything to latest, got a bit further this time than last, but ran into a new error when the first inference request hits the server:
Using this launch command:
EDIT: Also, it seems like setting
--kt-num-gpu-expertsto anything above 0 results in a kernel launch error on CUDA graph capture:I don't think this is a VRAM-related issue as I have over 50gb of VRAM free at this point.
@triplenom commented on GitHub (Mar 1, 2026):
I'm seeing the same as @DocShotgun with a similar setup (Nvidia RTX Pro 6000 + Epyc 9654)
With --kt-num-gpu-experts 0 and no layerwise prefill generation is as expected.
If I provide a long prompt that hits layerwise prefill, see the same error:
File "/home/mike/sglang-kimi-mar1/ktransformers/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 520, in _prepare_weight_int4
num_bits = method.num_bits
^^^^^^^^^^^^^^^
AttributeError: 'CompressedTensorsFusedMoEMethod' object has no attribute 'num_bits'
@triplenom commented on GitHub (Mar 2, 2026):
If I force num_bits=4 and packed_factor=32 for the RAWINT case in kt_ep_wrapper.py then layerwise prefill seems to work fine, so that seems easy enough to resolve.
@DocShotgun commented on GitHub (Mar 2, 2026):
After a few rounds of trial and error, I figured out that the correct values are:
and I managed to get it to load and infer with the following launch command:
however, the model is still outputting nonsense whenever it goes down the GPU prefill codepath. For those of you who are getting it to generate coherent text with GPU prefill, is your
--kt-threadpool-count==--tensor-parallel-size?Also increasing
--kt-num-gpu-expertsto a non-zero value still results in the aboveRuntimeError: moe_sum_reduce CUDA kernel (small-token) launch failederror.@ErvinXie commented on GitHub (Mar 2, 2026):
Thanks for the continued debugging, @DocShotgun and @triplenom.
We've noted both issues:
num_bitsAttributeError — The upstream SGLang merge changed the MoE quantization scheme class (CompressedTensorsFusedMoEMethod), which no longer exposesnum_bits. Ourkt_ep_wrapper.pystill references the old attribute. We'll fix this.moe_sum_reducekernel launch failure & GPU prefill gibberish — These are likely caused by operator compatibility issues introduced by the upstream SGLang merge. The new marlin MoE kernel path behaves differently from the original one, and we need to verify compatibility across the full GPU offload prefill codepath.We need some time to investigate and fix these properly. In the meantime, please use the
kimi_k2.5branch as a workaround, or set--kt-gpu-prefill-token-thresholdto a very large value to disable GPU prefill.Will update this issue once the fixes are ready.
This reply was co-authored by an AI agent and a human.
@ErvinXie commented on GitHub (Mar 2, 2026):
It may take som time to fix this.
@DocShotgun commented on GitHub (Mar 2, 2026):
Alright, understood. Trying to test the
kimi_k2.5branches of both ktransformers and the sglang fork, but it's having some multimodal-related error on server launch:@ouqingliang commented on GitHub (Mar 2, 2026):
We’ve reproduced and fixed this on our side.
Root cause:
In the layerwise prefill fallback path for RAWINT4 (compressed-tensors), INT4 metadata was read from the outer fused method, while in this path the required fields are on the wrapped scheme/method chain. This could break prefill/offload behavior.
Fixes applied:
kt_ep_wrapper(resolvenum_bits/packed_factor/group_sizefrom wrapped method/scheme chain).kt_num_gpu_experts>0case (invalid expert-id path), which was causing incorrect behavior in this configuration.After the fix,
--kt-gpu-prefill-token-thresholdand--kt_num_gpu_expertscan stay enabled and long prompts are stable/coherent again.Both
sglangandktransformersare on themainbranch.Patch commit (our sglang side):
f1a12b9a9This reply was generated by an AI agent and reviewed by a human.
@DocShotgun commented on GitHub (Mar 2, 2026):
Hmm, pulled to latest main on both and reinstalled, now getting this:
EDIT: I think switching to the
kimi_k2.5branch and back perhaps left some stray files somewhere that git didn't catch. Did a clean clone of latest main and now the model is loading. Fingers crossed for it to work...@trilog-inc commented on GitHub (Mar 2, 2026):
The update has worked for me. No gibberish 👍
Only detail, update xgrammar to 0.1.31 as some of the function/thinking parsing is broken in previous versions. Also ensure the thinking argument is sent. Currently running a 75K context and everything is coherent and fast.
Thanks kt team!
@DocShotgun commented on GitHub (Mar 2, 2026):
Okay, everything runs without errors on the current main branches:
github.com/kvcache-ai/ktransformers@9e69fccb02github.com/kvcache-ai/sglang@f1a12b9a93Unfortunately the model is still producing nonsense for me when using GPU offload prefill:
With settings:
Using the recommended temp 0.6, top p 0.95 for non-thinking mode.
My hunch is still that it has something to do with NUMA TP != GPU TP since that seems to be the main factor unique to my machine here, in my case NUMA TP=4 with GPU TP=1.
@ouqingliang commented on GitHub (Mar 3, 2026):
Thanks for the very clear repro, this helped a lot.
We dug into this in detail, and the root cause is not the CPU TP/GPU TP mapping formula itself.
The main issue was a double-buffer reuse race in SGLang's RAWINT4 layerwise prefill fallback path.
Root cause
In
kt_ep_wrapper.py(_prepare_weight_int4), we use a 2-slot pinned host buffer pipeline:e+1into slot(e+1)%2efrom slote%2e-1For INT4, before reusing the next slot, we were missing the same synchronization guard that FP8/BF16 already had.
So under some topologies, CPU starts writing the next expert into a slot while GPU DMA is still reading that slot.
Why your machine is easier to trigger
Your case (
NUMA TP=4,GPU TP=1) makes this race window much larger:GPU TP=1=> per-slot H2D payload is large (copy takes longer)CPU TP=4=> writes are faster/more parallel (slot gets reused sooner)So this is very consistent with "runs without errors but output quality collapses under GPU offload prefill".
Fix
We added slot-reuse synchronization in the INT4 path before submitting the next write (same pattern as FP8/BF16).
b3356b6c4kvcache-ai/sglang(main)Please pull latest
mainfor bothsglangandktransformersand retest with the same command.This reply was generated by an AI agent and reviewed by a human.
@DocShotgun commented on GitHub (Mar 4, 2026):
By golly, that did it! This time I managed to generate coherent text with GPU layerwise prefill! Token gen speed dropped a tad compared to when it was generating nonsense, but the significance of that is unclear.
A few observations:
kt-kernel's token gen speed is noticeably faster thanllama.cppwith NUMA distribute and memory interleave (on an equivalent GGUF with the INT4-32g QAT mapped to q4_0) due to proper NUMA awareness.kt-kernel's GPU offload prefill is slower thanllama.cpp's and doesn't manage to saturate my PCIe 5.0 x16, peaking at around RX 10s-20s GB/s while llama.cpp is able to hit 50+ GB/s at times. I'm not sure if this is becausekt-kernelhas to do additional work to prepare the weights before transferring to GPU (around 350-500ms per layer) compared tollama.cpp's host buffer.The tokenization endpoint also keeps failing with
TypeError: Integer exceeds 64-bit rangeon JSON decode, although I don't think this error is related toktransformers.Anyhow, thanks for your hard work and attention!
This is the current launch command I've gotten, not sure if there's any more room for adjustment.
--record-kt-gpu-expert-distributiondidn't seem to write out anything while the server was running or after shutting it down.@DocShotgun commented on GitHub (Mar 6, 2026):
Closing this as the titular issue has been resolved.