mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1111] [Bug] AttributeError: 'DeepseekV3Config' object has no attribute 'seq_aux' #775
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#775
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @iKpuc on GitHub (Apr 10, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1111
Checklist
Describe the bug
Hello!
The build was created without problems, however, an error occurs when launching Local_chat:
$ python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ~/DeepSeek-R1-UD-Q2_K_XL/ --cpu_infer 65 --max_new_tokens 8192 --temperature 0.6 --model_name unsloth/DeepSeek-R1-UD-Q2_K_XL
2025-04-09 21:00:11,184 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
using custom modeling_xxx.py.
Traceback (most recent call last):
File "/home/user/ktransformers/./ktransformers/local_chat.py", line 183, in
fire.Fire(local_chat)
File "/home/user/ktransformers/venv/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ktransformers/venv/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/user/ktransformers/venv/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ktransformers/./ktransformers/local_chat.py", line 91, in local_chat
model = custom_modelsconfig.architectures[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1605, in init
self.model = DeepseekV3Model(config)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1367, in init
[
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1368, in
DeepseekV3DecoderLayer(config, layer_idx)
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1154, in init
DeepseekV3MoE(config)
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 515, in init
self.gate = MoEGate(config)
^^^^^^^^^^^^^^^
File "/home/user/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 401, in init
self.seq_aux = config.seq_aux
^^^^^^^^^^^^^^
File "/home/user/ktransformers/venv/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in getattribute
return super().getattribute(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DeepseekV3Config' object has no attribute 'seq_aux'
Reproduction
python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ~/DeepSeek-R1-UD-Q2_K_XL/ --cpu_infer 65 --max_new_tokens 8192 --temperature 0.6 --model_name unsloth/DeepSeek-R1-UD-Q2_K_XL
Environment
Ubuntu 22.04
2*Xeon 8592+
512gb Ddr5
@iKpuc commented on GitHub (Apr 10, 2025):
The dependencies are installed as in flash-infer, I tried to run it through the API, the same thing...
@KMSorSMS commented on GitHub (Apr 11, 2025):
Could you please show your flash infer version,
pip list | grep flashandpip show flashinfer-python, and check if it is v0.2.3? As we use our own flash_infer in ./third_party/custom_flashinfer, you may need to install it instead of the official release. Runpip install third_party/custom_flashinfer/may help you.@iKpuc commented on GitHub (Apr 11, 2025):
Hi! I did what you wrote. It didn't help....
(ktransformers) user@user:
/ktransformers$ pip list | grep flash flashinfer-python 0.2.3/ktransformers$ pip show flashinfer-python Name: flashinfer-python(ktransformers) user@user:
Version: 0.2.3
Summary: FlashInfer: Kernel Library for LLM Serving
Home-page: https://github.com/flashinfer-ai/flashinfer
Author: FlashInfer team
Author-email:
License: Apache License 2.0
Location: /home/user/anaconda3/envs/ktransformers/lib/python3.11/site-packages
Requires: ninja, numpy, torch
Required-by:
@KMSorSMS commented on GitHub (Apr 12, 2025):
Seems like this issue is related to our latest fix. May I check your installed version? Try the
git logand also show it withpip list | grep ktrans. If older, maybe you can try reinstalling the latest version.@iKpuc commented on GitHub (Apr 12, 2025):
commit
7a19f3b781(HEAD)Merge:
85e2cc722df52eAuthor: wang jiahao 44601537+qiyuxinlin@users.noreply.github.com
Date: Thu Feb 27 21:01:21 2025 +0800
@iKpuc commented on GitHub (Apr 12, 2025):
When Building commit
6617549, the following problem occurs:(venv) (base) user@user:
/ktransformers$ export USE_NUMA=1/ktransformers$ KTRANSFORMERS_FORCE_BUILD=TRUE uv pip install . --no-build-isolation(venv) (base) user@user:
Using Python 3.11.11 environment at: venv
× Failed to build
ktransformers @ file:///home/user/ktransformers├─▶ The build backend returned an error
╰─▶ Call to
setuptools.build_meta.prepare_metadata_for_build_wheelfailed (exit status: 1)@iKpuc commented on GitHub (Apr 12, 2025):
And there is the same error with this commit.:
commit
c51818c39a(HEAD, tag: v0.2.3post2)Merge:
bda9cf13934b9dAuthor: Atream 80757050+Atream@users.noreply.github.com
Date: Sat Mar 15 23:09:30 2025 +0800
@KMSorSMS commented on GitHub (Apr 12, 2025):
@iKpuc commented on GitHub (Apr 12, 2025):
Thanks for the help! However, after creating the build, the following error occurred:
@KMSorSMS commented on GitHub (Apr 13, 2025):
This may be related to some weird problem when installing with the previous cache. You can reinstall it without any cache by rm all the build folders in the current workspace. Note that the build folder also occurs in csrc where you need to clean too. And then reinstall it.