kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #2127] DeepSeek-V4-Flash-0731 MTP failed to start on avx2 + sm89 #1285

New issue
Open
opened 2026-10-29 14:50:51 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-29 14:50:51 +00:00
Owner
Copy link

Originally created by @cadslab on GitHub (Aug 3, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2127

Reminder

  • I have read the above rules and searched the existing issues.

System Info

ubuntu

Reproduction

MTP failed, avx2 + sm89, command to run:

export FLASHINFER_CUDA_ARCH_LIST=8.9a
export TORCH_CUDA_ARCH_LIST="8.9+PTX"
export SGLANG_DSV4_MODE=2604
export SGLANG_DSV4_2604_SUBMODE=2604B
export PYTORCH_ALLOC_CONF=expandable_segments:True,garbage_collection_threshold:0.3,max_split_size_mb:1024
export CUDA_HOME=/usr/local/cuda-12.8
export CUDA_LAUNCH_BLOCKING=0 
export FLASHINFER_DISABLE_TEMP_POOL=0
export FLASHINFER_DISABLE_VERSION_CHECK=1

python -m sglang.launch_server \
  --host 0.0.0.0 --port 30000 \
  --model /mnt/0_ssd/DeepSeek-V4-Flash-0731 \
  --kt-weight-path /mnt/0_ssd/DeepSeek-V4-Flash-0731 \
  --kt-method MXFP4 \
  --kt-num-gpu-experts 64 \
  --kt-cpuinfer 12 \
  --kt-threadpool-count 8 \
  --kt-gpu-prefill-token-threshold 8192 \
  --kt-expert-placement-strategy uniform \
  --kt-enable-dynamic-expert-update \
  --allow-auto-truncate \
  --attention-backend flashmla \
  --cuda-graph-bs 2 \
  --cuda-graph-max-bs 4 \
  --context-length 1048576 \
  --chunked-prefill-size 8192 \
  --enable-p2p-check \
  --enable-mixed-chunk \
  --enable-cudagraph-gc \
  --max-running-requests 2 \
  --max-total-tokens 1048576 \
  --max-prefill-tokens 8192 \
  --mem-fraction-static 0.85 \
  --reasoning-parser deepseek-v4 \
  --served-model-name local \
  --tensor-parallel-size 8 \
  --tool-call-parser deepseekv4 \
  --trust-remote-code \
  --watchdog-timeout 1200 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-moe-runner-backend auto

error:

Scheduler hit an exception: Traceback (most recent call last):
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3207, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 367, in __init__
    self.init_model_worker()
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 564, in init_model_worker
    self.maybe_init_draft_worker()
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 560, in maybe_init_draft_worker
    self.draft_worker = DraftWorkerClass(**draft_worker_kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 624, in __init__
    self._draft_worker = EagleDraftWorker(
                         ^^^^^^^^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 135, in __init__
    self.draft_worker = TpModelWorker(
                        ^^^^^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__
    self._init_model_runner()
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 418, in __init__
    self.initialize()
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 501, in initialize
    self.load_model()
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1060, in load_model
    self.model = self.loader.load_model(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model
    self.load_weights_and_postprocess(
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 686, in load_weights_and_postprocess
    model.load_weights(weights)
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/models/deepseek_v4_nextn.py", line 245, in load_weights
    super().load_weights(weights, is_nextn=True)
  File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/models/deepseek_v4.py", line 1991, in load_weights
    raise RuntimeError(
RuntimeError: Some weights are not initialized from checkpoints: {'model.h_proj.weight', 'model.e_proj.weight_scale_inv', 'model.enorm.weight', 'model.h_proj.weight_scale_inv', 'model.e_proj.weight', 'model.hnorm.weight'}

Others

No response

Originally created by @cadslab on GitHub (Aug 3, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2127 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ubuntu ### Reproduction MTP failed, avx2 + sm89, command to run: ``` export FLASHINFER_CUDA_ARCH_LIST=8.9a export TORCH_CUDA_ARCH_LIST="8.9+PTX" export SGLANG_DSV4_MODE=2604 export SGLANG_DSV4_2604_SUBMODE=2604B export PYTORCH_ALLOC_CONF=expandable_segments:True,garbage_collection_threshold:0.3,max_split_size_mb:1024 export CUDA_HOME=/usr/local/cuda-12.8 export CUDA_LAUNCH_BLOCKING=0 export FLASHINFER_DISABLE_TEMP_POOL=0 export FLASHINFER_DISABLE_VERSION_CHECK=1 python -m sglang.launch_server \ --host 0.0.0.0 --port 30000 \ --model /mnt/0_ssd/DeepSeek-V4-Flash-0731 \ --kt-weight-path /mnt/0_ssd/DeepSeek-V4-Flash-0731 \ --kt-method MXFP4 \ --kt-num-gpu-experts 64 \ --kt-cpuinfer 12 \ --kt-threadpool-count 8 \ --kt-gpu-prefill-token-threshold 8192 \ --kt-expert-placement-strategy uniform \ --kt-enable-dynamic-expert-update \ --allow-auto-truncate \ --attention-backend flashmla \ --cuda-graph-bs 2 \ --cuda-graph-max-bs 4 \ --context-length 1048576 \ --chunked-prefill-size 8192 \ --enable-p2p-check \ --enable-mixed-chunk \ --enable-cudagraph-gc \ --max-running-requests 2 \ --max-total-tokens 1048576 \ --max-prefill-tokens 8192 \ --mem-fraction-static 0.85 \ --reasoning-parser deepseek-v4 \ --served-model-name local \ --tensor-parallel-size 8 \ --tool-call-parser deepseekv4 \ --trust-remote-code \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-moe-runner-backend auto ``` error: ``` Scheduler hit an exception: Traceback (most recent call last): File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3207, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 367, in __init__ self.init_model_worker() File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 564, in init_model_worker self.maybe_init_draft_worker() File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 560, in maybe_init_draft_worker self.draft_worker = DraftWorkerClass(**draft_worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 624, in __init__ self._draft_worker = EagleDraftWorker( ^^^^^^^^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 135, in __init__ self.draft_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__ self._init_model_runner() File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 418, in __init__ self.initialize() File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 501, in initialize self.load_model() File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1060, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model self.load_weights_and_postprocess( File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 686, in load_weights_and_postprocess model.load_weights(weights) File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/models/deepseek_v4_nextn.py", line 245, in load_weights super().load_weights(weights, is_nextn=True) File "/home/k1/miniconda3/envs/kt/lib/python3.12/site-packages/sglang/srt/models/deepseek_v4.py", line 1991, in load_weights raise RuntimeError( RuntimeError: Some weights are not initialized from checkpoints: {'model.h_proj.weight', 'model.e_proj.weight_scale_inv', 'model.enorm.weight', 'model.h_proj.weight_scale_inv', 'model.e_proj.weight', 'model.hnorm.weight'} ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:50:52 +00:00
Author
Owner
Copy link

@yyj6666667 commented on GitHub (Aug 8, 2026):

Thanks for feedback. Does it work normally without turning on MTP ? It helps narrow the problem.

<!-- gh-comment-id:5225001296 --> @yyj6666667 commented on GitHub (Aug 8, 2026): Thanks for feedback. Does it work normally without turning on MTP ? It helps narrow the problem.
bsakaguchi commented 2026-10-29 14:50:52 +00:00
Author
Owner
Copy link

@cadslab commented on GitHub (Aug 8, 2026):

Thanks for feedback. Does it work normally without turning on MTP ? It helps narrow the problem.

worked fine without mtp

<!-- gh-comment-id:5226223711 --> @cadslab commented on GitHub (Aug 8, 2026): > Thanks for feedback. Does it work normally without turning on MTP ? It helps narrow the problem. worked fine without mtp
bsakaguchi commented 2026-10-29 14:50:52 +00:00
Author
Owner
Copy link

@yyj6666667 commented on GitHub (Aug 8, 2026):

I see.

<!-- gh-comment-id:5226296831 --> @yyj6666667 commented on GitHub (Aug 8, 2026): I see.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1285
No description provided.