mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1885] Capture cuda graph failed: Assertion error when Running DeepSeek V3.2 with SGLang and KT-Kernel #1176
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1176
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @merries711 on GitHub (Mar 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1885
Reminder
System Info
CPU: INTEL(R) XEON(R) GOLD 6530 x2
GPU: NVIDIA RTX PRO 4500 Blackwell 32GB [ GB203GL ] x2
RAM: 1TB DDR5
KTransformers CLI v0.1.0
Python 3.11.14
平台 Linux 6.12.63+deb13-amd64
CUDA 13.1
Packages:
kt-kernel 0.5.2.post1
sglang-kt 0.5.2.post2 (sglang-kt)
installed from source .
Reproduction
I followed the tutorial and ran DeepSeek V3.2 with SGLang and KT-Kernel, but it prompted the following error; how should I adjust it? Additionally, can I run the DeepSeek V3.2 gguf of unsloth with the current kt-kernel?
Furthermore, when I was running MiniMax-M2.1 with Native Precision using SGLang and KT-Kernel, the speed was very slow — less than 1 TPS — and it crashed after just two queries.
Others
python -m sglang.launch_server
--host 0.0.0.0
--port 31666
--model /nas/DeepSeek-V3.2
--kt-weight-path /models/deepseek-v3.2-INT4
--kt-cpuinfer 30
--kt-threadpool-count 4
--kt-num-gpu-experts 4
--attention-backend triton
--trust-remote-code
--mem-fraction-static 0.7
--chunked-prefill-size 2048
--max-running-requests 16
--max-total-tokens 8192
--served-model-name DeepSeek-V3.2
--enable-mixed-chunk
--tensor-parallel-size 2
--enable-p2p-check
--disable-shared-experts-fusion
--kt-method AMXINT4
python -m sglang.launch_server
--host 0.0.0.0 --port 31666
--model /models/MiniMax-M2.1
--kt-weight-path /models/MiniMax-M2.1
--kt-cpuinfer 60 --kt-threadpool-count 2
--kt-num-gpu-experts 24 --kt-method FP8
--kt-gpu-prefill-token-threshold 4096
--attention-backend flashinfer --trust-remote-code
--mem-fraction-static 0.8 --chunked-prefill-size 8192
--max-running-requests 16 --max-total-tokens 20000
--enable-mixed-chunk --tensor-parallel-size 2
--enable-p2p-check --served-model-name MiniMax-M2.1
--disable-shared-experts-fusion --fp8-gemm-backend triton
@ErvinXie commented on GitHub (Mar 17, 2026):
Hi @merries711,
You have two separate issues here. Let me address them:
1. DeepSeek V3.2 — "Unknown recipe" in DeepGEMM
Your RTX PRO 4500 (GB203, Blackwell architecture) is not supported by DeepGEMM, which currently only works on Hopper (SM90) GPUs. Setting
SGLANG_ENABLE_JIT_DEEPGEMM=0is not enough — the FP8 GEMM backend still defaults to DeepGEMM.Fix: Add
--fp8-gemm-backend tritonto your DeepSeek V3.2 launch command, just like you already have for MiniMax-M2.1.2. MiniMax-M2.1 — Extremely slow (0.1 tok/s) and crash
With 2× 32GB GPUs (64GB total), running MiniMax-M2.1 is very tight. A few issues:
--kt-num-gpu-experts 24puts 12 experts per GPU (with TP2), which consumes significant VRAM. Try reducing to--kt-num-gpu-experts 8or even4to leave more room for KV cache.--max-total-tokens 20000may also be too high for your VRAM budget. Try8192first.You can also check GPU memory usage during startup with
nvidia-smito see how much VRAM is left after model loading.3. Regarding Unsloth GGUF
KTransformers / kt-kernel does not support Unsloth's custom GGUF quantization types (UD format). Please use standard GGUF quantizations (e.g., Q4_K_M) or the official FP8/INT4 weights.
This reply was co-authored by an AI agent and a human.