mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1854] 启用DeepSeek-V3.2 GGUF sglang服务时，capture cuda graph失败 #1152
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1152
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @JYE-HC on GitHub (Feb 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1854
Reminder
System Info
Python 3.10
cuda 12.8
GPU NVIDIA RTX A6000
CPU Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz (32 核心 / 64 线程)
CPU 指令集 AVX512F, AVX512BW, AVX512VL, AVX2, AVX, FMA, SSE4.2, AVX512CD (+8 more)
NUMA 2
kt-kernel v0.5.1 (AVX512_VBMI)
SGLang Source (GitHub: kvcache-ai/sglang, branch: main)
SGLang kt-kernel 支持已验证
Reproduction
在modelscope下载的deepseek-ai/DeepSeek-V3.2原模型，以及unsloth/DeepSeek-V3.2-GGUF-Q4_K_M的权重。
执行命令：
CUDA_VISIBLE_DEVICES=7
python -m sglang.launch_server
--host 0.0.0.0
--port 30000
--model deepseek-ai/DeepSeek-V3.2
--kt-weight-path unsloth/DeepSeek-V3.2-GGUF/deepseek-v3.2-q4_k_m
--kt-cpuinfer 30
--kt-threadpool-count 2
--kt-num-gpu-experts 1
--kt-method LLAMAFILE
--attention-backend triton
--trust-remote-code
--mem-fraction-static 0.8
--chunked-prefill-size 4096
--max-running-requests 1
--max-total-tokens 40000
--served-model-name DeepSeek-V3.2
--enable-mixed-chunk
--tensor-parallel-size 1
--enable-p2p-check
--disable-shared-experts-fusion
--tool-call-parser deepseekv32
--reasoning-parser deepseek-v3
报错：
[2026-02-17 23:20:54] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32]
0%| | 0/8 [00:00<?, ?it/s]
Capturing batches (bs=32 avail_mem=8.34 GB): 0%| | 0/8 [00:00<?, ?it/s]
Capturing batches (bs=32 avail_mem=8.34 GB): 0%| | 0/8 [00:05<?, ?it/s]
[2026-02-17 23:21:00] Scheduler hit an exception: Traceback (most recent call last):
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 366, in init
self.capture()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 522, in capture
_capture_one_stream()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 509, in _capture_one_stream
) = self.capture_one_batch_size(bs, forward, stream_idx)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 728, in capture_one_batch_size
run_once()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 715, in run_once
logits_output_or_pp_proxy_tensors = forward(
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 2968, in forward
hidden_states = self.model(
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 2719, in forward
hidden_states, residual = layer(
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 2384, in forward
hidden_states = self.self_attn(
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 1368, in forward
return self.forward_core(s)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 1456, in forward_core
return self.forward_absorb_core(*inner_state)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/models/deepseek_v2.py", line 1806, in forward_absorb_core
attn_output = self.attn_mqa(
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/zhc/softwares/miniconda3/envs/kt/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/layers/radix_attention.py", line 126, in forward
return forward_batch.attn_backend.forward(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/layers/attention/base_attn_backend.py", line 93, in forward
return self.forward_decode(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 1020, in forward_decode
forward_batch.token_to_kv_pool.set_kv_buffer(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/mem_cache/memory_pool.py", line 977, in set_kv_buffer
_set_kv_buffer_impl(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/mem_cache/memory_pool.py", line 104, in _set_kv_buffer_impl
k.view(-1, row_dim),
RuntimeError: shape '[-1, 8192]' is invalid for input of size 18432
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/scheduler.py", line 3039, in run_scheduler_process
scheduler = Scheduler(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/scheduler.py", line 352, in init
self.init_model_worker()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/scheduler.py", line 541, in init_model_worker
self.init_tp_model_worker()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/scheduler.py", line 503, in init_tp_model_worker
self.tp_worker = TpModelWorker(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/tp_worker.py", line 242, in init
self._init_model_runner()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/managers/tp_worker.py", line 325, in _init_model_runner
self._model_runner = ModelRunner(
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/model_runner.py", line 403, in init
self.initialize(min_per_gpu_memory)
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/model_runner.py", line 600, in initialize
self.init_device_graphs()
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/model_runner.py", line 2107, in init_device_graphs
self.graph_runner = graph_runnersself.device
File "/home/zhc/workspace/llm/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 368, in init
raise Exception(
Exception: Capture cuda graph failed: shape '[-1, 8192]' is invalid for input of size 18432
Possible solutions:
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose
[2026-02-17 23:21:00] Received sigquit from a child process. It usually means the child failed.
Others
No response
@ErvinXie commented on GitHub (Feb 28, 2026):
DeepSeek-V3.2 是支持的，可参考官网 benchmark：https://ktransformers.net/zh/benchmarks#DeepSeek-V3.2-FP8-TP8
V3.2 使用了 NSA (Native Sparse Attention)，需要注意以下配置：
--attention-backend triton，flashinfer 不支持 NSA 的 decode 路径如果仍有问题，请提供完整的启动命令和报错日志。
This reply was co-authored by an AI agent and a human.