[GH-ISSUE #1327] [Bug] failed to run the model on two A770 cards with ktransformers 0.3.1 #894

New issue

Closed

opened 2026-10-29 14:45:50 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:45:50 +00:00

Owner

Originally created by @luningxie on GitHub (May 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1327

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

I followed the tutorial which stress the new feature of using intel arc display card.
the compilation of ktransformers is ok. when i download the deepseek-R1-q4-k-m gguf and run the command:

"python ~/ktransformers/ktransformers/local_chat.py --model_path /home/deepseek/documents/deepseek --gguf_path /home/deepseek/documents/deepseek/DeepSeek-R1-Q4_K_M --optimize_config_path ktransformers/optimize/optimize_rules/xpu/DeepSeek-V3-Chat.yaml --cpu_infer 33 --device xpu --max_new_tokens 200"

there is a wrong message:

Traceback (most recent call last):
File "/home/deepseek/ktransformers/ktransformers/local_chat.py", line 190, in
fire.Fire(local_chat)
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/ktransformers/ktransformers/local_chat.py", line 77, in local_chat
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1123, in from_pretrained
config_class = get_class_from_dynamic_module(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 570, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module, force_reload=force_download)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 269, in get_class_in_module
return getattr(module, class_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

"AttributeError: module 'transformers_modules.deepseek.configuration_deepseek' has no attribute 'DeepseekV3Config'. Did you mean: 'DeepseekV2Config'?

Reproduction

using the model of DeepSeek-R1-Q4_K_M, with gguf form.

Environment

ubuntu 22.04.05
intel 8480, one cpu
384g ddr5

Originally created by @luningxie on GitHub (May 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1327 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug I followed the tutorial which stress the new feature of using intel arc display card. the compilation of ktransformers is ok. when i download the deepseek-R1-q4-k-m gguf and run the command: "python ~/ktransformers/ktransformers/local_chat.py --model_path /home/deepseek/documents/deepseek --gguf_path /home/deepseek/documents/deepseek/DeepSeek-R1-Q4_K_M --optimize_config_path ktransformers/optimize/optimize_rules/xpu/DeepSeek-V3-Chat.yaml --cpu_infer 33 --device xpu --max_new_tokens 200" there is a wrong message: Traceback (most recent call last): File "/home/deepseek/ktransformers/ktransformers/local_chat.py", line 190, in <module> fire.Fire(local_chat) File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/ktransformers/ktransformers/local_chat.py", line 77, in local_chat config = AutoConfig.from_pretrained(model_path, trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1123, in from_pretrained config_class = get_class_from_dynamic_module( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 570, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module, force_reload=force_download) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/deepseek/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 269, in get_class_in_module return getattr(module, class_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ "AttributeError: module 'transformers_modules.deepseek.configuration_deepseek' has no attribute 'DeepseekV3Config'. Did you mean: 'DeepseekV2Config'? ### Reproduction "python ~/ktransformers/ktransformers/local_chat.py --model_path /home/deepseek/documents/deepseek --gguf_path /home/deepseek/documents/deepseek/DeepSeek-R1-Q4_K_M --optimize_config_path ktransformers/optimize/optimize_rules/xpu/DeepSeek-V3-Chat.yaml --cpu_infer 33 --device xpu --max_new_tokens 200" using the model of DeepSeek-R1-Q4_K_M, with gguf form. ### Environment ubuntu 22.04.05 intel 8480, one cpu 384g ddr5

bsakaguchi closed this issue

2026-10-29 14:45:50 +00:00

bsakaguchi commented

2026-10-29 14:45:51 +00:00

Author

Owner

@aubreyli commented on GitHub (May 21, 2025):

It looks like your environment was messed up.

Can you please check if this folder

/home/deepseek/documents/deepseek/

has

configuration_deepseek.py

and this file has the following line?

class DeepseekV3Config(PretrainedConfig):

@aubreyli commented on GitHub (May 21, 2025): It looks like your environment was messed up. Can you please check if this folder `/home/deepseek/documents/deepseek/` has `configuration_deepseek.py` and this file has the following line? `class DeepseekV3Config(PretrainedConfig):`

bsakaguchi commented

2026-10-29 14:45:51 +00:00

Author

Owner

@luningxie commented on GitHub (May 21, 2025):

Problem solved. I copy the configuration_deepseek_v3.py to the model path, and rename it to configuration_deepseek.py.

It worked but very slowly.

prompt eval count: 6 token(s)
prompt eval duration: 6.846642971038818s
prompt eval rate: 0.8763418839539168 tokens/s
eval count: 77 token(s)
eval duration: 64.9632203578949s
eval rate: 1.1852860676516983 tokens/s

I have already used the Best Known Config (BKC) to obtain best performance
echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
echo 0 | sudo tee /sys/devices/system/cpu/cpu*/power/energy_perf_bias
sudo xpu-smi config -d 0 -t 0 --frequencyrange 2400,2400

    export SYCL_CACHE_PERSISTENT=1
    export ONEAPI_DEVICE_SELECTOR=level_zero:0
    export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1

@luningxie commented on GitHub (May 21, 2025): Problem solved. I copy the configuration_deepseek_v3.py to the model path, and rename it to configuration_deepseek.py. It worked but very slowly. prompt eval count: 6 token(s) prompt eval duration: 6.846642971038818s prompt eval rate: 0.8763418839539168 tokens/s eval count: 77 token(s) eval duration: 64.9632203578949s eval rate: 1.1852860676516983 tokens/s I have already used the Best Known Config (BKC) to obtain best performance echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor echo 0 | sudo tee /sys/devices/system/cpu/cpu*/power/energy_perf_bias sudo xpu-smi config -d 0 -t 0 --frequencyrange 2400,2400 export SYCL_CACHE_PERSISTENT=1 export ONEAPI_DEVICE_SELECTOR=level_zero:0 export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1

bsakaguchi commented

2026-10-29 14:45:51 +00:00

Author

Owner

@aubreyli commented on GitHub (May 21, 2025):

Can you please close this ticket and file a new one for another issue? Thanks!

@aubreyli commented on GitHub (May 21, 2025): Can you please close this ticket and file a new one for another issue? Thanks!