[GH-ISSUE #1462] [Bug] AssertionError: [Smallthinker config sanitizer] rope_layout must have the same length as num_hidden_layers #975

New issue

Open

opened 2026-10-29 14:47:05 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 14:47:05 +00:00

Owner

Originally created by @dhr613 on GitHub (Jul 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1462

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

python ktransformers/server/main.py
--model_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3
--gguf_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf
--cpu_infer 15
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--port 10002
--chunk_size 256
--max_new_tokens 1024
--max_batch_size 4
--port 10002
--cache_lens 3000
--backend_type balance_serve

是我模型下错了吗？但是我使用local_chat的时候是可行的

python3 -m ktransformers.local_chat
--model_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3"
--gguf_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf"

Reproduction

deepseek-v3

Environment

按照install.md中的步骤创建的环境

Originally created by @dhr613 on GitHub (Jul 29, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1462 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug python ktransformers/server/main.py \ --model_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3 \ --gguf_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf \ --cpu_infer 15 \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --port 10002 \ --chunk_size 256 \ --max_new_tokens 1024 \ --max_batch_size 4 \ --port 10002 \ --cache_lens 3000 \ --backend_type balance_serve 是我模型下错了吗？但是我使用local_chat的时候是可行的 python3 -m ktransformers.local_chat \ --model_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3" \ --gguf_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf" ### Reproduction deepseek-v3 ### Environment 按照install.md中的步骤创建的环境

bsakaguchi commented

2026-10-29 14:47:06 +00:00

Author

Owner

@luzamm commented on GitHub (Jul 30, 2025):

添加 --architectures DeepseekV3ForCausalLM

@luzamm commented on GitHub (Jul 30, 2025): 添加 --architectures DeepseekV3ForCausalLM

bsakaguchi commented

2026-10-29 14:47:06 +00:00

Author

Owner

@dhr613 commented on GitHub (Aug 1, 2025):

添加 --architectures DeepseekV3ForCausalLM

好的，已跑通，感谢

@dhr613 commented on GitHub (Aug 1, 2025): @ > 添加 --architectures DeepseekV3ForCausalLM 好的，已跑通，感谢

bsakaguchi commented

2026-10-29 14:47:06 +00:00

Author

Owner

@wxtt522 commented on GitHub (Sep 19, 2025):

同样的问题出现在/Qwen3-Next-80B-A3B模型上，使用--architectures Qwen3NextForCausalLM 没有用

@wxtt522 commented on GitHub (Sep 19, 2025): 同样的问题出现在/Qwen3-Next-80B-A3B模型上，使用--architectures Qwen3NextForCausalLM 没有用

bsakaguchi commented

2026-10-29 14:47:06 +00:00

Author

Owner

@ruralharry commented on GitHub (Sep 25, 2025):

我也遇到了，加了--architectures DeepseekV3ForCausalLM，也没用。使用的模型是deepseek-v3.1-terminus-gguf

@ruralharry commented on GitHub (Sep 25, 2025): 我也遇到了，加了--architectures DeepseekV3ForCausalLM，也没用。使用的模型是deepseek-v3.1-terminus-gguf