mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1254] [Bug]0.3版本运行Qwen3-235B-A22B 8bit版本全部装载完后卡住 #852
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#852
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @TTTiger2020 on GitHub (May 4, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1254
检查清单
问题描述
最新版本运行Qwen3-235B-A22B 8bit版本全部装载完后卡住：
这是最后一段的打印信息：
loading blk.89.attn_q_norm.weight to cuda
loading blk.89.attn_k_norm.weight to cuda
loading blk.89.attn_norm.weight to cuda
loading blk.89.ffn_norm.weight to cuda
loading blk.90.attn_q_norm.weight to cuda
loading blk.90.attn_k_norm.weight to cuda
loading blk.90.attn_norm.weight to cuda
loading blk.90.ffn_norm.weight to cuda
loading blk.91.attn_q_norm.weight to cuda
loading blk.91.attn_k_norm.weight to cuda
loading blk.91.attn_norm.weight to cuda
loading blk.91.ffn_norm.weight to cuda
loading blk.92.attn_q_norm.weight to cuda
loading blk.92.attn_k_norm.weight to cuda
loading blk.92.attn_norm.weight to cuda
loading blk.92.ffn_norm.weight to cuda
loading blk.93.attn_q_norm.weight to cuda
loading blk.93.attn_k_norm.weight to cuda
loading blk.93.attn_norm.weight to cuda
loading blk.93.ffn_norm.weight to cuda
loading output_norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 15339
卡在这里
然后可以看到内存使用量497.3/811.2GB（没有释放，看起来装载完了？）
显存使用量 4738MiB / 20480MiB
这是相关的程序占用，会一直卡在上面那个log的地方
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
tttiger 9134 1.1 0.0 13413400 84460 pts/0 Sl+ 20:36 0:16 python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path Qwen/Qwen3-235B-A22B --gguf_path /disk_2t/hf_model/q
tttiger 9309 0.0 0.0 23716 8320 pts/0 S+ 20:36 0:00 /home/tttiger/anaconda3/envs/ktransformers/bin/python -c from multiprocessing.resource_tracker import main;main(46)
tttiger 9380 268 79.7 746661532 631641860 pts/0 Sl+ 20:36 66:17 /home/tttiger/anaconda3/envs/ktransformers/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=47, pipe_
tttiger 15339 1.2 0.0 0 0 pts/0 Z+ 20:47 0:10 [python3]
复现步骤
运行指令：
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path Qwen/Qwen3-235B-A22B --gguf_path /disk_2t/hf_model/qwen3 --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --backend_type balance_serve
使用模型：
Qwen3-235B-A22B 8bit
环境信息
系统是ubuntu2204
CPU是双socket 四代Xeon 8470es + 48gb 4800 DDR5*16 + 3080 20g cuda 12.8
本地编译，编译时开启了
USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh
@withyou971 commented on GitHub (May 4, 2025):
相同的问题，window10 wsl2 ubutun2204
使用镜像尝试过 cuda 121，124，126，128，
sched_rpc started with PID: 163
Got inference context, sending it to subscribers.
Rebuilding kvcache
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 192, in init
inference_context = self.sched_client.rebuild_inferece_context(inference_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in rebuild_inferece_context
inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/balance_serve/sched_rpc.py", line 196, in
inference_context.k_cache = [fn(*args) for fn,args in data['k_cache']]
^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 181, in rebuild_cuda_tensor
storage = storage_cls._new_shared_cuda(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/storage.py", line 1452, in _new_shared_cuda
return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.使用编译环境，无论什么模型，balance_serve都会卡在
loading blk.47.attn_q_norm.weight to cuda
loading blk.47.attn_k_norm.weight to cuda
loading blk.47.attn_norm.weight to cuda
loading blk.47.ffn_norm.weight to cuda
loading output_norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 163
Got inference context, sending it to subscribers.
Rebuilding kvcache
之后，r1 ktransformers的backend可以使用
@TTTiger2020 commented on GitHub (May 5, 2025):
嗯嗯是的
@TTTiger2020 commented on GitHub (May 5, 2025):
好的感谢，这就去试试！
@zhinianqin commented on GitHub (May 5, 2025):
抱歉，我的回答有问题，我也遇到了卡住的问题，我查询成功案例以后发现只有Q4_K_M成功部署的，还在研究中
@TTTiger2020 commented on GitHub (May 5, 2025):
嗯嗯，我这边也试试不同的版本，看看哪个可以跑起来
@Flying-Cloud commented on GitHub (May 5, 2025):
根据我的经验，--model_path Qwen/Qwen3-235B-A22B这步有问题，需要将hf上的各种.json文件下载到本地，然后model_path指定一个本地的路径才行，不然会卡在sched_rpc started with PID
@BoruiXu commented on GitHub (May 5, 2025):
跑deepseek r1也遇到了同样问题，将huggingface上的各种配置文件下载到本地路径下解决了
@TTTiger2020 commented on GitHub (May 6, 2025):
感谢各位的建议，按照各位说的将配置下到本地，把模型路径换成本地路径可以正常运行了，指令如下：
（ktransformers) tttiger@tttiger-Super-Server:~/ktransformers$ python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path /disk_2t/hf_model/qwen3 --gguf_path /disk_2t/hf_model/qwen3 --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --backend_type balance_serve
@AlexBerko commented on GitHub (May 22, 2025):
hello, can you please tell me witch qwen3 model did you use?
because i install official model from hf: https://huggingface.co/Qwen/Qwen3-235B-A22B-GGUF/tree/main/Q8_0
and non-safetensors files from: https://huggingface.co/Qwen/Qwen3-235B-A22B/tree/main
run command:
TORCH_CUDA_ARCH_LIST="8.9" CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES="2" python ktransformers/server/main.py --host 0.0.0.0 --port 8150 --model_path /etc/models/gguf/Qwen3-235B-A22B/ --gguf_path /etc/models/gguf/Qwen3-235B-A22B-Q8-0/ --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --max_new_tokens 16384 --cache_lens 131072 --chunk_size 2048 --max_batch_size 4 --backend_type balance_serve --cpu_infer 24 --use_cuda_graph --model_name Qwen3-235B-A22B --temperature 0.2 --cache_q4 true --print_timings true --architectures Qwen3MoeForCausalLM
and still my program frozen on "Getting inference context from sched_client."