[GH-ISSUE #1317] [Bug]运行qwen3 moe后卡在kv_cache 不动了

bsakaguchi commented

2026-10-29 14:45:46 +00:00

Owner

Originally created by @tech112233445566 on GitHub (May 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1317

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

运行QWEN3 MOE 30B-A3B-GGUF Q4KM，卡死不动

Reproduction

运行命令：

Environment

Package Version

accelerate 1.7.0
annotated-types 0.7.0
anyio 4.9.0
blessed 1.21.0
blobfile 3.0.0
build 1.2.2.post1
certifi 2025.4.26
charset-normalizer 3.4.2
click 8.2.0
colorlog 6.9.0
distro 1.9.0
einops 0.8.1
fastapi 0.115.12
filelock 3.18.0
fire 0.7.0
flash_attn 2.7.4.post1
flashinfer-python 0.2.3
fsspec 2025.3.2
greenlet 3.2.2
h11 0.16.0
httpcore 1.0.9
httpx 0.28.1
huggingface-hub 0.31.2
idna 3.10
Jinja2 3.1.6
jiter 0.9.0
jsonpatch 1.33
jsonpointer 3.0.0
ktransformers 0.3+cu124torch26fancy
langchain 0.3.25
langchain-core 0.3.60
langchain-text-splitters 0.3.8
langsmith 0.3.42
lxml 5.4.0
MarkupSafe 3.0.2
mpmath 1.3.0
networkx 3.4.2
ninja 1.11.1.4
numpy 2.2.5
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
openai 1.79.0
orjson 3.10.18
packaging 24.2
pillow 11.0.0
pip 25.1
protobuf 6.31.0
psutil 7.0.0
pycryptodomex 3.22.0
pydantic 2.11.4
pydantic_core 2.33.2
pyproject_hooks 1.2.0
PyYAML 6.0.2
pyzmq 26.4.0
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
safetensors 0.5.3
sentencepiece 0.2.0
setuptools 78.1.1
sniffio 1.3.1
SQLAlchemy 2.0.41
starlette 0.46.2
sympy 1.13.1
tenacity 9.1.2
termcolor 3.1.0
tiktoken 0.9.0
tokenizers 0.21.1
torch 2.6.0+cu124
torchaudio 2.6.0+cu124
torchvision 0.21.0+cu124
tqdm 4.67.1
transformers 4.51.2
triton 3.2.0
typing_extensions 4.13.2
typing-inspection 0.4.0
urllib3 2.4.0
uvicorn 0.34.2
wcwidth 0.2.13
wheel 0.45.1
zmq 0.0.0
zstandard 0.23.0

Originally created by @tech112233445566 on GitHub (May 17, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1317 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug 运行QWEN3 MOE 30B-A3B-GGUF Q4KM，卡死不动 ![Image](https://github.com/user-attachments/assets/4550495b-2b1e-4276-8031-36f1960b0de3) ### Reproduction 运行命令： ![Image](https://github.com/user-attachments/assets/7fea1090-b828-4529-96e5-52d4ac5d484d) ### Environment Package Version ------------------------ --------------------- accelerate 1.7.0 annotated-types 0.7.0 anyio 4.9.0 blessed 1.21.0 blobfile 3.0.0 build 1.2.2.post1 certifi 2025.4.26 charset-normalizer 3.4.2 click 8.2.0 colorlog 6.9.0 distro 1.9.0 einops 0.8.1 fastapi 0.115.12 filelock 3.18.0 fire 0.7.0 flash_attn 2.7.4.post1 flashinfer-python 0.2.3 fsspec 2025.3.2 greenlet 3.2.2 h11 0.16.0 httpcore 1.0.9 httpx 0.28.1 huggingface-hub 0.31.2 idna 3.10 Jinja2 3.1.6 jiter 0.9.0 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.3+cu124torch26fancy langchain 0.3.25 langchain-core 0.3.60 langchain-text-splitters 0.3.8 langsmith 0.3.42 lxml 5.4.0 MarkupSafe 3.0.2 mpmath 1.3.0 networkx 3.4.2 ninja 1.11.1.4 numpy 2.2.5 nvidia-cublas-cu12 12.4.5.8 nvidia-cuda-cupti-cu12 12.4.127 nvidia-cuda-nvrtc-cu12 12.4.127 nvidia-cuda-runtime-cu12 12.4.127 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.2.1.3 nvidia-cufile-cu12 1.11.1.6 nvidia-curand-cu12 10.3.5.147 nvidia-cusolver-cu12 11.6.1.9 nvidia-cusparse-cu12 12.3.1.170 nvidia-cusparselt-cu12 0.6.2 nvidia-nccl-cu12 2.21.5 nvidia-nvjitlink-cu12 12.4.127 nvidia-nvtx-cu12 12.4.127 openai 1.79.0 orjson 3.10.18 packaging 24.2 pillow 11.0.0 pip 25.1 protobuf 6.31.0 psutil 7.0.0 pycryptodomex 3.22.0 pydantic 2.11.4 pydantic_core 2.33.2 pyproject_hooks 1.2.0 PyYAML 6.0.2 pyzmq 26.4.0 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.3 sentencepiece 0.2.0 setuptools 78.1.1 sniffio 1.3.1 SQLAlchemy 2.0.41 starlette 0.46.2 sympy 1.13.1 tenacity 9.1.2 termcolor 3.1.0 tiktoken 0.9.0 tokenizers 0.21.1 torch 2.6.0+cu124 torchaudio 2.6.0+cu124 torchvision 0.21.0+cu124 tqdm 4.67.1 transformers 4.51.2 triton 3.2.0 typing_extensions 4.13.2 typing-inspection 0.4.0 urllib3 2.4.0 uvicorn 0.34.2 wcwidth 0.2.13 wheel 0.45.1 zmq 0.0.0 zstandard 0.23.0

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@qiuqiu10 commented on GitHub (May 19, 2025):

我看它输出了sched_rpc started with pid，说明主进程的server已经起起来了（在BalanceServeInterface的init中），前面创建了新的进程来运行run_engine，而你的输出里显示kvcache loaded successfully说明Engine的init的前半部分都没有问题。所以你可以确定一下，要么是卡在cudagraph capture之前了，要么就是卡在cuda graph capture中了，还有一种可能是cudagraph capture本身就很慢，你多等一会儿就可以了

@qiuqiu10 commented on GitHub (May 19, 2025): 我看它输出了sched_rpc started with pid，说明主进程的server已经起起来了（在BalanceServeInterface的init中），前面创建了新的进程来运行run_engine，而你的输出里显示kvcache loaded successfully说明Engine的init的前半部分都没有问题。所以你可以确定一下，要么是卡在cudagraph capture之前了，要么就是卡在cuda graph capture中了，还有一种可能是cudagraph capture本身就很慢，你多等一会儿就可以了

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@tech112233445566 commented on GitHub (May 20, 2025):

上次基本上我等了10分钟都没有反应，运行的模型是qwen3 30b A3B

@tech112233445566 commented on GitHub (May 20, 2025): 上次基本上我等了10分钟都没有反应，运行的模型是qwen3 30b A3B

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@tech112233445566 commented on GitHub (May 20, 2025):

关掉cuda graph 可以吗，是不是就是最多会影响一些推理速度。我的显卡是RTX3060 12G, 内存64G，运行后内存还剩20多G

@tech112233445566 commented on GitHub (May 20, 2025): 关掉cuda graph 可以吗，是不是就是最多会影响一些推理速度。我的显卡是RTX3060 12G, 内存64G，运行后内存还剩20多G

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@ruralharry commented on GitHub (Jun 3, 2025):

我是卡在PID那里

❯ curl -X 'POST' 'http://localhost:8120/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"messages": [{"content": "你是谁", "role": "user"}], "model": "DeepSeek-V3", "stream": false}'
curl: (7) Failed to connect to localhost port 8120 after 0 ms: Connection refused

启动命令：

ktransformers
--port 8120
--cpu_infer 54
--model_path /mnt/sda5/DeepSeek-V3
--gguf_path /mnt/sda5/0324q4/DeepSeek-V3
--backend_type balance_serve
--max_new_tokens 4096
--cache_lens 131072
--chunk_size 2048

当我把命令简化成：
ktransformers
--port 8120
--cpu_infer 54
--model_path /mnt/sda5/DeepSeek-V3
--gguf_path /mnt/sda5/0324q4/DeepSeek-V3
--backend_type balance_serve
--max_new_tokens 1024

则出现了另外一个bug https://github.com/kvcache-ai/ktransformers/issues/1359

@ruralharry commented on GitHub (Jun 3, 2025): 我是卡在PID那里 ❯ curl -X 'POST' 'http://localhost:8120/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"messages": [{"content": "你是谁", "role": "user"}], "model": "DeepSeek-V3", "stream": false}' curl: (7) Failed to connect to localhost port 8120 after 0 ms: Connection refused 启动命令： ktransformers \ --port 8120 \ --cpu_infer 54 \ --model_path /mnt/sda5/DeepSeek-V3 \ --gguf_path /mnt/sda5/0324q4/DeepSeek-V3 \ --backend_type balance_serve \ --max_new_tokens 4096 \ --cache_lens 131072 \ --chunk_size 2048 当我把命令简化成： ktransformers \ --port 8120 \ --cpu_infer 54 \ --model_path /mnt/sda5/DeepSeek-V3 \ --gguf_path /mnt/sda5/0324q4/DeepSeek-V3 \ --backend_type balance_serve \ --max_new_tokens 1024 则出现了另外一个bug https://github.com/kvcache-ai/ktransformers/issues/1359

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@TheLegendOfKitty commented on GitHub (Jul 9, 2025):

I'm getting the same issue.

@TheLegendOfKitty commented on GitHub (Jul 9, 2025): I'm getting the same issue.

bsakaguchi commented

2026-10-29 14:45:47 +00:00

Author

Owner

@tech112233445566 commented on GitHub (Aug 1, 2025):

我跑localchat的时候发现了报错，但是不知道怎么解决

@tech112233445566 commented on GitHub (Aug 1, 2025): <img width="1481" height="996" alt="Image" src="https://github.com/user-attachments/assets/ef6c8411-8130-43c6-9d85-0f8035e474cd" /> 我跑localchat的时候发现了报错，但是不知道怎么解决

bsakaguchi referenced this issue

2026-10-29 14:52:58 +00:00

[PR #886] [MERGED] fix-singleton #1502

Rows
Columns

[GH-ISSUE #1317] [Bug]运行qwen3 moe后卡在kv_cache 不动了 #886

Checklist

Describe the bug

Reproduction

Environment