[GH-ISSUE #1815] Crash with Speculative decoding #1136

New issue

Open

opened 2026-10-29 14:49:01 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:49:01 +00:00

Owner

Originally created by @havetc on GitHub (Jan 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1815

Reminder

I have read the above rules and searched the existing issues.

System Info

ktransformer 0.5.0.post1, kv-cacheai/sglang:eaf832ef, 2*RTX PRO 6000,AMD EPYC 9645,Pop!_OS 22.04 LTS,

Reproduction

Hello!
I've been adding some issues with speculative decoding, which are not 100% deterministic but seems to happen very often, here is my setup:

#!/bin/bash
NCCL_P2P_LEVEL=PIX kt run --enable-shared-experts-fusion \
 --cpu-threads 96 \
 --chunked-prefill-size 60000 \
 --max-total-tokens 300000 \
 --gpu-experts 40 \
 --model-path /fast-data/ktransformer/MinimaxM2.1/ \
 -p 8000 \
 Minimax-M2.1 \
 --max-prefill-tokens 80000 \
 --max-running-requests 4 \
 --cuda-graph-max-bs 4 \
 --mem-fraction-static 0.75 \
 --speculative-algorithm STANDALONE \
 --speculative-draft-model-path Qwen/Qwen3-4B-Instruct-2507-FP8 \
 --enable-cache-report

It may or may not run for a short time, but then while processing a request it crashes with a very long and verbose stack trace. I think the main error seems to be Fatal Python error: Fatal Python error: Segmentation faultSegmentation fault, and the other errors are probably just cascading failures from there, I've put the full execution logs in there (there is a keyboardinterrupt at the end, otherwise it just loop over the error without ever stopping)

crash-home-workstation-ktransformers.txt

Before doing more investigation, I just wanted to know is speculative supposed to work properly with ktransformer ? If not what are the known issues / limitations ?

Others

No response

Originally created by @havetc on GitHub (Jan 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1815 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ktransformer 0.5.0.post1, kv-cacheai/sglang:eaf832ef, 2*RTX PRO 6000,AMD EPYC 9645,Pop!_OS 22.04 LTS, ### Reproduction Hello! I've been adding some issues with speculative decoding, which are not 100% deterministic but seems to happen very often, here is my setup: ```text #!/bin/bash NCCL_P2P_LEVEL=PIX kt run --enable-shared-experts-fusion \ --cpu-threads 96 \ --chunked-prefill-size 60000 \ --max-total-tokens 300000 \ --gpu-experts 40 \ --model-path /fast-data/ktransformer/MinimaxM2.1/ \ -p 8000 \ Minimax-M2.1 \ --max-prefill-tokens 80000 \ --max-running-requests 4 \ --cuda-graph-max-bs 4 \ --mem-fraction-static 0.75 \ --speculative-algorithm STANDALONE \ --speculative-draft-model-path Qwen/Qwen3-4B-Instruct-2507-FP8 \ --enable-cache-report ``` It may or may not run for a short time, but then while processing a request it crashes with a very long and verbose stack trace. I think the main error seems to be ```Fatal Python error: Fatal Python error: Segmentation faultSegmentation fault```, and the other errors are probably just cascading failures from there, I've put the full execution logs in there (there is a keyboardinterrupt at the end, otherwise it just loop over the error without ever stopping) [crash-home-workstation-ktransformers.txt](https://github.com/user-attachments/files/24881628/crash-home-workstation-ktransformers.txt) Before doing more investigation, I just wanted to know is speculative supposed to work properly with ktransformer ? If not what are the known issues / limitations ? ### Others _No response_

bsakaguchi commented

2026-10-29 14:49:02 +00:00

Author

Owner

@ouqingliang commented on GitHub (Jan 28, 2026):

We do not support speculative decoding currently.

@ouqingliang commented on GitHub (Jan 28, 2026): We do not support speculative decoding currently.