[GH-ISSUE #2118] Support DSpark speculative decoding for DeepSeek-V4-Flash-0731 in sglang-kt #1282

New issue

Open

opened 2026-10-29 14:50:50 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:50:50 +00:00

Owner

Originally created by @edwardyehuang on GitHub (Aug 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2118

Reminder

I have read the above rules and searched the existing issues.

Description

Summary

DeepSeek-V4-Flash-0731 (official release, 2026-07-31) ships a new DSpark speculative decoding module instead of the NextN-style MTP head used in DeepSeek-V4-Flash-Preview. sglang-kt==0.6.4 only supports EAGLE/NEXTN speculative algorithms and cannot load the new draft head, forcing users to disable speculative decoding entirely (~2x decode speed loss).

Request: port DSPARK support to sglang-kt (upstream SGLang supports it since v0.5.16).

Background

Per the official release notes, DeepSeek-V4-Flash-0731 "has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached."

The checkpoint layout changed:

Preview: single NextN MTP layer with top-level keys model.enorm/hnorm/e_proj/h_proj (+ weight_scale_inv)
0731: three full MoE draft layers mtp.0, mtp.1, mtp.2 (4,705 mtp.* keys in model.safetensors.index.json), each containing attention + 256 routed experts + 1 shared expert, plus mtp.2.confidence_head.proj and mtp.2.markov_head.markov_w1/w2

New config.json fields:

"dspark_block_size": 5,
"dspark_noise_token_id": 128799,
"dspark_target_layer_ids": [40, 41, 42],
"dspark_markov_rank": 256

### Pull Request

_No response_

Originally created by @edwardyehuang on GitHub (Aug 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2118 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description ## Summary DeepSeek-V4-Flash-0731 (official release, 2026-07-31) ships a new **DSpark** speculative decoding module instead of the NextN-style MTP head used in DeepSeek-V4-Flash-Preview. `sglang-kt==0.6.4` only supports `EAGLE`/`NEXTN` speculative algorithms and cannot load the new draft head, forcing users to disable speculative decoding entirely (~2x decode speed loss). Request: port DSPARK support to sglang-kt (upstream SGLang supports it since v0.5.16). ## Background Per the official release notes, DeepSeek-V4-Flash-0731 "has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached." The checkpoint layout changed: - **Preview**: single NextN MTP layer with top-level keys `model.enorm/hnorm/e_proj/h_proj` (+ `weight_scale_inv`) - **0731**: three full MoE draft layers `mtp.0`, `mtp.1`, `mtp.2` (4,705 `mtp.*` keys in `model.safetensors.index.json`), each containing attention + 256 routed experts + 1 shared expert, plus `mtp.2.confidence_head.proj` and `mtp.2.markov_head.markov_w1/w2` New `config.json` fields: ```json "dspark_block_size": 5, "dspark_noise_token_id": 128799, "dspark_target_layer_ids": [40, 41, 42], "dspark_markov_rank": 256 ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:50:50 +00:00

bsakaguchi commented

2026-10-29 14:50:50 +00:00

Author

Owner

@trilog-inc commented on GitHub (Aug 3, 2026):

Hi,

I managed to get Codex and 5.6 Sol to migrate the DSpark implementation from mainline. You can find the sglang update here https://github.com/trilog-inc/sglang/tree/codex/dspark-amx and the ktransformers update here https://github.com/trilog-inc/ktransformers/tree/codex/dspark-amx-native.

I also added the feature to select a different GPU for the draft model. In my case an RTX 4090 which frees up ~15G of VRAM on the RTX 6000.

It works on my system ( Xeon with AMX and a RTX 6000 pro ). Still doing tests but prefill is about 960t/s and decode from 25-35t/s.

EDIT: Averaging 45-60t/s with frequency expert placement via distribution recording.

@trilog-inc commented on GitHub (Aug 3, 2026): Hi, I managed to get Codex and 5.6 Sol to migrate the DSpark implementation from mainline. You can find the sglang update here https://github.com/trilog-inc/sglang/tree/codex/dspark-amx and the ktransformers update here https://github.com/trilog-inc/ktransformers/tree/codex/dspark-amx-native. I also added the feature to select a different GPU for the draft model. In my case an RTX 4090 which frees up ~15G of VRAM on the RTX 6000. It works on my system ( Xeon with AMX and a RTX 6000 pro ). Still doing tests but prefill is about 960t/s and decode from 25-35t/s. EDIT: Averaging 45-60t/s with frequency expert placement via distribution recording.