mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #2213] Support for Deepseek V4.1 Flash #1304
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1304
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @RobertoFMBarreto on GitHub (Sep 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2213
Reminder
Description
When will you guys develop Deepseek V4.1 Flash support?
Pull Request
No response
@05yuki commented on GitHub (Oct 7, 2026):
In case it helps while this is open: DeepSeek V4.1-Flash runs on the KT hybrid today, and the whole recipe is public:
https://github.com/05yuki/dsv41-flash-dual-epyc
What it is: SGLang's
dsv4.1tree (1aa0e962) plus one tree patch, kt-kernel (95009ea) plus one tree patch, and an engram adapter that serves the two 101 GB engram tables straight from the NVMe instead of RAM. Measured on 2x RTX 5070 Ti 16 GB and 2x EPYC 7452 (512 GB), MXFP4 experts on the CPU, Marlin hot experts on the GPUs:The README lists the 18 things that had to change, in the order they were found (SWA pool sizing, the two-level indexer's memory at long prompts, arena pinning, the loader), each with its patch and its numbers. Two of them look upstreamable on their own (the kt-kernel loader prefetch and the CPU-expert skip in
load_weights); I asked about those in #2227.@andreasgru commented on GitHub (Oct 18, 2026):
Thanks for the recipe, we build on it. Recipe-state launcher (18 Sep) on 1 RTX PRO 6000 96 GB + 2× EPYC 7302, 1017 weights, 1 stream (median of 20): 23.6 tok/s. Our tuned path, same card: 64.2 to 64.7. Other expert map and memory cap: not kernel-for-kernel.
@disdrix commented on GitHub (Oct 21, 2026):
I just wanted to add that I tried the above method with DS4.1 1017 Q4_K_XL (Unsloth). Hardware is RTX 6000 Pro, Threadripper 7965WX, 512GB DDR5 6000. Engrams are loaded on m.2 drive.
Llama.cpp was around 24-32 tok/s decode with dspark and 310 tok/s prompt processing with 8k chunks.
I threw the latest version of ktransformers into opus 5.5 in claude code and let it cook. I've got ktransformers working fully in Windows 10. Prefill is running roughly 641 tok/s at 2,600 tokens, 1,135 tok/s at 10,000, and 1,438 tok/s at 20,500. Decode varies between 25 and 55. This depends on dspark and expert swapping. 40–55 tok/s in long sessions. 25–34 tok/s on fresh short prompts, while the GPU expert cache is still learning which experts get used.
I just starting doing rounds of optimization, so I should be able to get more out of it. So far, Opus 5.5 reports that accuracy is not impacted, so it should be good. It is kind of like we don't need devs anymore since AI just does it all now. I am not even sure what the point of contributing is anymore these days.
@05yuki commented on GitHub (Oct 21, 2026):
For 1017 specifically there is a newer recipe: https://github.com/05yuki/deepseek-v4.1-flash-1017-ktransformers
It is SGLang main (ed0b8f43) plus format-patch tiers, and kt-kernel
95009eaplus one tree patch.setup.shbuilds everything, andserve.shstarts it.On 2× RTX 5070 Ti 16 GB and 2× EPYC 7452 (512 GB DDR4), with a 1M-token context:
@disdrix, your note on expert swapping matches what we measured. A fixed hot set recorded on one kind of text caught only 3–6% of the routes of code, JSON or math.
The GPU experts in that repo now follow the text being written. Every 8 decode steps, all 21 a layer are swapped by decayed route counts: the 5 that always stay, and 16 more in the KV pool's free pages.
@robertofmb commented on GitHub (Oct 21, 2026):
This is @RobertoFMBarreto company account
<head></head>Thanks @05yuki for sharing your recipe and documenting your experiments. I originally opened this issue, and wanted to share where I have got to using your earlier recipe with some local changes.
My setup:
2× Intel Xeon Platinum 8362: 64 physical cores total, 1 TB RAM.
One NVIDIA RTX PRO 6000 Blackwell Max-Q with 96 GB VRAM dedicated to this model.
Linux and Docker, GPU TP=1, two CPU thread pools with 32 workers each.
deepseek-ai/DeepSeek-V4.1-Flash-1017, with 104 GPU-resident experts using Marlin and the remaining experts on CPU.Engram tables in RAM, prefix caching enabled, and prefill chunks of 2,048 tokens.
Context configured to 270,336 tokens, with one active request. We have not validated the entire configured context window.
Latest results, using medians of three runs:
Switching the CPU MXFP4 path from AVX2 to the AVX-512 implementation reduced uncached time to first output by approximately 23–24%, while generation throughput stayed almost unchanged. These Xeons do not support AMX; this uses the AVX-512 fallback in
AMXFP4_KGroup_MOE.Our targeted context-retrieval and tool-call round-trip tests passed, and I am also using it through Hermes Agent. These are functional checks, rather than a comprehensive accuracy evaluation.
Uncached prefill remains the main bottleneck. In the earlier zero-copy streaming code, we found that with two CPU partitions and GPU TP=1, the arena mapping selects only one CPU partition. We are experimenting with assembling both partitions correctly, but that change is still unvalidated.
I noticed you have now shared a newer recipe specifically for 1017—thanks for continuing to share this work. Have you tried two CPU partitions feeding a single GPU, particularly with the AVX-512 CPU backend?
If you have already tackled something similar in the newer recipe, I would be interested in comparing approaches. Happy to share our configuration and test results if useful.
Next, I plan to validate streamed prefill with both CPU partitions feeding the single GPU. After that, I want to compare expert placement based on representative Hermes workloads and investigate whether DSpark can improve generation speed in this setup. These are planned experiments; the results above do not include them.
@05yuki commented on GitHub (Oct 21, 2026):
@RobertoFMBarreto thanks for the detailed numbers. Yes: one GPU fed by two CPU partitions is exactly what we fixed today.
The two-partition fix
patches/sglang-tier1/14-stream-prefill.patch(_rank_segmentsinkt_stream_prefill.py).Two other things in that repo may fit your setup
gpu-moe-fp32). With Marlin's repacked layout, their self test will not match, and the slots stay static (not tried with Marlin here).KT_KV_HOT_EXTRA=auto).We only have AVX2 (Zen 2) CPUs, so the AVX-512 path is untested here. Your 23–24% on uncached time to first output is useful to know. Happy to compare configs.
Repo: https://github.com/05yuki/deepseek-v4.1-flash-1017-ktransformers
@disdrix commented on GitHub (Oct 21, 2026):
I was able to work on decode a bit by getting expert usage up. There is a lot of optimization to be done to get expert usage on GPU over 90%, which gives a huge boost to decode. I've got the average decode up to 42 tok/s with the minimum at 34 now. PCIe 5.0 instead of 4.0 if you have it might matter here more for this.
Best bet is probably have a frontier model like Opus 5.5 or similar just run overnight optimization with quality and accuracy checks at each step. I plan to do this tonight as I sleep.
@robertofmb commented on GitHub (Oct 21, 2026):
@05yuki Thanks! We ported your
b5ad85esegmented W16 fix into our existing SGLang build and adapted the AVX-512 backend to stream directly from its native CPU buffers, without another full weight copy.On our dual Xeon 8362 + RTX PRO 6000 96 GB, the best configuration measured so far is 96 GPU experts, streaming groups of four and 8,192-token chunks:
Streaming-versus-CPU checks on four layers gave cosine ≥0.99999 and relative L2 error of 0.30–0.40%. Targeted retrieval tests through ~32K and three tool-call round trips passed.
We have not migrated to the full new recipe or tested dynamic placement/DSpark yet. The configured 270K context remains unvalidated. Happy to share the AVX-512 changes if useful!
Next, I plan to test larger streaming expert groups, then investigate your dynamic expert placement and newer FP8 decode kernels. I’ll compare latency and generation throughput while repeating the numerical and functional checks.
@05yuki commented on GitHub (Oct 21, 2026):
@RobertoFMBarreto great result on the prefill (50.4 → 27.7 s), and thanks for the cosine and L2 checks.
On decode: 29 tok/s with 96 GPU experts on an RTX PRO 6000 looks bound by something other than the CPU experts. With a quarter of each layer's experts on the GPU, the CPU side of a token should be small.
Likely cause: the dense FP8 GEMMs. The RTX PRO 6000 is SM120, like our 5070 Ti. SM120 has no DeepGEMM build, so the dense FP8 GEMMs of decode take a slow generic path at batch 1. On our cards that was the longer side of a token.
What fixed it on our side:
fp8-decode(block-FP8 GEMV and split-K for decode-sized GEMMs) alone took a token from 62.7 to 36.6 ms (15.9 → 27.3 tok/s).engram-row-prefetchandwo-a-fp8on top, it reached 28.2–28.3 tok/s before any hot experts.How to check: an nsys trace of a few decode steps shows it. If the dense FP8 GEMMs dominate the token, rather than the wait for kt-kernel, that is the bound.
fp8-decodehas no prerequisites and applies alone on tier 0 (SGLANG_FEATURES=fp8-decode), so it can be tried without the rest.@disdrix commented on GitHub (Oct 21, 2026):
Okay, I let Opus 5.5 work on it for 8 hours as planned. The results were phenomenal. At the bottom is everything that was changed from ktransformers. Should help anyone that wants to optimize theirs too.
DeepSeek-V4.1-Flash on one workstation with KTransformers, native on Windows
Summary: about 80 tok/s decode on code, 55-61 tok/s on prose, and 950-1,140 tok/s prefill on a new 16-24K-token prompt. One user at a time, with 313 of each layer's 384 routed experts running on the CPU.
Hardware: AMD Threadripper PRO 7965WX (24 cores), 512 GB RAM, one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB). Windows, running natively (no WSL).
Model:
DeepSeek-V4.1-Flash-1017from Unsloth's GGUFs:UD-Q4_K_XL(MXFP4 routed experts, Q8_0 dense weights converted to FP8 at load),mmproj-BF16for vision, and the DSpark draft in Q8_0.Server: SGLang with DeepSeek-V4.1 support, plus kt-kernel for the CPU experts.
--vram-gib 80). The other 313 run on the CPU in kt-kernel (24 threads, AVX-512 BF16, MXFP4 read straight from the GGUF).Results
One request at a time, thinking off, streamed through the OpenAI-compatible API over the LAN. Runs are listed in the order they ran.
How these were measured
stream_interval=1), so this is DSpark's mean accept length including the bonus token (max 3).disaggregation/prefill.py(59 KB) plus "write a comprehensive pytest suite for this module". Run C usedmanagers/mm_utils.py, cut to 59 KB.What's different from upstream KTransformers 0.7.2
torch.distributed(no NCCL). Triton attention and PyTorch sampling replace the Linux-only FlashInfer, FA3, CUTLASS, DeepGEMM and TileLang. sgl-kernel's CUDA ops are reimplemented in PyTorch/Triton and checked against sgl-kernel's own tests. JIT kernels build with nvcc and MSVC, and kt-kernel builds with clang-cl.mmprojand DSpark draft load too. The Engram tables (about 190 GiB) are never loaded; each lookup reads its rows from the memory-mapped file.@disdrix commented on GitHub (Oct 22, 2026):
I had Opus 5.5 optimize for 8 hours again to squeeze more speed out of it. I am going to call these results final. More information on what I did is included so you can try to copy what I did with your specific setup. Since this is Windows 10, you could expect an increase of 10-15% on Linux.
Update: faster prompt processing, slightly faster decode (same hardware, same tests)
Summary: on a new 16-24K-token prompt, prefill now runs at 1,530-2,220 tok/s (was 950-1,140), and time to first token is 37-49% shorter. Decode is up to 6% faster (9 of the 10 runs faster, one even): about 82-84 tok/s on code and 62-64 tok/s on prose. Output quality is unchanged.
Setup changes since the first post (hardware, model and the rest of the server are the same):
-c 131072to match the first post.Results
Same harness and prompts as the first post: one request at a time, thinking off, streamed through the OpenAI-compatible API over the LAN, a fresh nonce line per request, runs in the order they ran, on a freshly started server.
What changed
@05yuki commented on GitHub (Oct 22, 2026):
A finding from checking our 1017 setup against the checkpoint's own reference implementation, in case it affects yours.
Routed experts scaled twice under KTransformers (SGLang main after sgl-project/sglang#44993, 2026-10-20).
routed_scaling_factorinto its weights whenever the quant method says so.KTEPWrapperMethodforwards that question to its GPU method, and the MXFP4 methods answer yes.DeepseekV2MoEstill multiplied the routed output by the factor for every KT layer. So on V4.1 every MoE layer added its routed experts at 2.25x instead of 1.5x.2026-10-22-2112.deepseek_v2.pystill hasor isinstance(self.experts.quant_method, KTEPWrapperMethod)withoutand not self.routed_scaling_factor_fused_in_topk.How we checked.
inference/model.py, streamed one layer at a time through a 16 GB card, scores those 11.5K tokens in about 5 minutes. We dumped the hidden states of piece 0 at matching points in both engines and compared them layer by layer.@disdrix commented on GitHub (Oct 22, 2026):
Thanks for the detailed write-up! We hit the same double scaling on 10-20 and fixed it on our side by having the KT wrapper keep the top-k weights unscaled (it answers "no" to fusing routed_scaling_factor), so DeepseekV2MoE applies the 1.5x once after the CPU+GPU sum. All the numbers in both posts are with that fix. Note that your quick check would flag our build anyway, since we kept the KTEPWrapperMethod line in deepseek_v2.py and fixed it in the wrapper instead. Would you share the 24 x 480-token text you scored? I'd like to run it on our build and compare with your reference NLL of 2.594.
@05yuki commented on GitHub (Oct 22, 2026):
A second finding, now in the 1017 repo (tag
2026-10-23-0135): on SM120 (RTX 50, RTX PRO 6000), SGLang runs V4.1 with the V4 KV layout.11-v41-kv-layout(sgl-project/sglang#45851) reads the V4.1 layouts on SM120 through FlashInfer's SM120 sparse MLA. It uses FlashInfer's full-BF16 route; the default route rounds Q and P to MXFP8.On 2× RTX 5070 Ti, decode went from 45.6 to 47.1 tok/s on Japanese prose and rose on every other text. On one card it went from 28.5 to 32.0.
@disdrix commented on GitHub (Oct 22, 2026):
Our Windows build also runs V4.1 with the V4 KV layout on the RTX PRO 6000 (584 bytes/token, FP8 blocks of 64, bf16 RoPE), with our own Triton attention kernels since FlashInfer isn't available there.
@mtcl commented on GitHub (Oct 25, 2026):
I'd like to try this on my 2x6kpro with Intel Xeon 3495x and 512gb ddr5. What would be the easiest way to follow this?
@05yuki commented on GitHub (Oct 25, 2026):
@mtcl The self-contained recipe is https://github.com/05yuki/deepseek-v4.1-flash-1017-ktransformers: download the checkpoint, then
MODEL=<checkpoint dir> ./setup.shbuilds the patched SGLang + kt-kernel andMODEL=<checkpoint dir> ./serve.shstarts it. serve.sh sets tensor parallelism from the GPUs it sees, so two cards give TP2; RTX PRO 6000 is SM120, the same architecture the results were measured on.One caveat: every number in the README comes from AMD Zen 2. On Sapphire Rapids the experts run on the same AVX2 kernels (the CPU has them too), but the recipe hasn't been run on Intel yet, so please open an issue there with the serve.sh log if anything misbehaves.
@disdrix commented on GitHub (Oct 25, 2026):
I ended up throwing what was available in to Opus 5.5 with claude code and had it do it all for me. I then had it run around 16 hours of optimizations trying different things. It would be much faster for you if you did it since I basically give you a recipe with all the things I did.