mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1843] AscendNPU run Qwen3-235b error #1151
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1151
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @zxt-triumph on GitHub (Feb 11, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1843
Reminder
System Info
System Info
我参考如下链接进行安装：
https://github.com/kvcache-ai/ktransformers/blob/main/doc/zh/Qwen3-MoE_tutorial_zh_for_Ascend_NPU.md
我的系统信息是：
NPU： 300I A2 (910B4-1 64GB) * 2
CPU: HUAWEI Kunpeng 920 5220
内存: 256GB
HDK: 25.3.rc1.2
CANN: CANN 8.3.RC1.alpha001，配套ToolKit，Kernel和NNAL均已安装好
OS: openEuler 22.03 LTS sp1
按照教程链接在docker镜像mindie2.2创建的容器中顺利安装好了，但是运行模型时遇到错误：
python ktransformers/server/main.py --port 10002 --model_path /opt/data/Qwen3-235B-A22B-q4km-w8a8 --gguf_path /opt/data/Qwen3-235B-A22B-q4km-w8a8 --model_name Qwen3MoeForCausalLM --cpu_infer 48 --optimize_config_path ./ktransformers/optimize/optimize_rules/npu/Qwen3-Chat-300IA2-npu-serve.yaml --max_new_tokens 1024 --cache_lens 8192 --max_batch_size 2 --use_cuda_graph --tp 1 --backend_type balance_serve
Reproduction
Injecting model.layers.93.mlp.experts.127 as default
Injecting model.layers.93.mlp.experts.127.gate_proj as default
Injecting model.layers.93.mlp.experts.127.up_proj as default
Injecting model.layers.93.mlp.experts.127.down_proj as default
Injecting model.layers.93.mlp.experts.127.act_fn as default
Injecting model.layers.93.input_layernorm as ktransformers.operators.ascend.ascend_layernorm . KQwen3MoeRMSNormW8A8
Injecting model.layers.93.post_attention_layernorm as ktransformers.operators.ascend.ascend_layernorm . KQwen3MoeRMSNormW8A8
Injecting model.norm as default
Injecting model.rotary_emb as default
Injecting cache as default
Injecting lm_head as ktransformers.operators.ascend.ascend_linear . KTransformersLinearW8A8A2
loading layer token_embd.weight to cpu
loading layer blk.0.ffn_gate_inp.weight to npu:0
/usr/lib64/python3.11/tempfile.py:904: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmph_z7j2yn'>
_warnings.warn(warn_message, ResourceWarning)
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 412, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 261, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/optimize/optimize.py", line 147, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 342, in load_weights
module.load()
File "/usr/local/lib64/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 342, in load_weights
module.load()
File "/usr/local/lib64/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 340, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 338, in load_weights
load_cur_state_dict(module, gguf_loader, prefix, device=device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 265, in load_cur_state_dict
load_cur_state_dict_npu(module, gguf_loader, prefix, device)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/util/utils.py", line 261, in load_cur_state_dict_npu
raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find blk.0.0.ffn_gate_exps in GGUF file!
/usr/lib64/python3.11/multiprocessing/process.py:330: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUB) at 0xfffe7872c6e0>
traceback.print_exc()
/usr/lib64/python3.11/multiprocessing/process.py:330: ResourceWarning: Unclosed context <zmq.Context() at 0xfffcfb3d3e90>
traceback.print_exc()