mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1268] [Bug]AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? #860
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#860
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sv777777 on GitHub (May 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1268
Originally assigned to: @Alisehen on GitHub.
Checklist
Describe the bug
0.3编译过程顺利，模型加载也正常，推理时出现报错：
Chat: 1
Traceback (most recent call last):
File "/home/i/ktransformers/ktransformers/util/utils.py", line 206, in prefill_and_generate
model._get_logits_warper(generation_config,device=inputs.device)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1944, in getattr
raise AttributeError(
AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/i/ktransformers/ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/util/utils.py", line 210, in prefill_and_generate
model._get_logits_warper(generation_config)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1944, in getattr
raise AttributeError(
AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?
Reproduction
python -m ktransformers.local_chat --model_path /home/Qwen3-235B-A22B/ --gguf_path /home/Qwen3-235B-IQ4_XS/ --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 22 --max_new_tokens 2048 --top_p 0.95 --temperature 0.7 --use_cuda_graph True --architectures Qwen3MoeForCausalLM
Environment
CUDA：12.6
SYSTEM:UBUNTU24
CPU:12900
GPU:4070
RAM:128
PYTHON:3.11
TORCH:2.8
@blime4 commented on GitHub (May 7, 2025):
maybe this pr can help you : https://github.com/kvcache-ai/ktransformers/pull/1247
@sv777777 commented on GitHub (May 8, 2025):
使用合并后了修复后的代码重新编译，推理时错误：
Chat: hi
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/i/ktransformers/ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/util/utils.py", line 281, in prefill_and_generate
logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/util/utils.py", line 244, in chunk_prefill
logits = model(
^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1755, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1766, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 1043, in forward
outputs: MoeModelOutputWithPast = self.model(
^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1755, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1766, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/ktransformers/ktransformers/operators/models.py", line 363, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1755, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1766, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py", line 375, in forward
hidden_states, self_attn_weights = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1755, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/i/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1766, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: KQwen3MoeAttention.forward() got an unexpected keyword argument 'attention_mask'
@zondie17 commented on GitHub (May 8, 2025):
maybe try the fix provided by @vickiegpt and @createthis in #1238
@sv777777 commented on GitHub (May 8, 2025):
I read your tip and found that qwen3 does not support local_chat mode yet. Thanks.