[GH-ISSUE #1445] [Bug] KT 0.3.2版本运行Kimi-K2-Instruct-UD-IQ2_XXS 能加载模型，进入对话框后报错。 #963

New issue

Open

opened 2026-10-29 14:46:59 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:46:59 +00:00

Owner

Originally created by @mikestut on GitHub (Jul 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1445

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

model_path: str = MODEL_PATH,
optimize_config_path: str = ktransformer_rules_dir+'DeepSeek-V3-Chat-serve.yaml',
gguf_path: str = GGUF_PATH, # | None = None,
max_new_tokens: int =1024,
cpu_infer: str = CPU_INFER ,#: int = Config().cpu_infer,
use_cuda_graph: bool = True , #False,
prompt_file : str | None = None,
mode: str = "normal",
force_think: bool = False,
chunk_size: int =12800

正常加载完成60层。
loading model.layers.60.self_attn.q_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.60.input_layernorm.weight to cuda
loading model.layers.60.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Chat: 你好
显存占用：8G

WARNING:transformers_modules.tokenization_kimi:Calling super().encode with {'add_special_tokens': False}
报错如下：

File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
1742 # If we don't have any hooks, we want to skip the rest of the logic in
1743 # this function, and just call forward.
1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1745 or _global_backward_pre_hooks or _global_backward_hooks
1746 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747 return forward_call(*args, **kwargs)
1749 result = None
1750 called_always_called_hooks = set()

TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'

复现步骤

python ktransformers/server/main.py
--port 10002
--model_path <path_to_safetensor_config>
--gguf_path <path_to_gguf_files>
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml
--max_new_tokens 1024
--cache_lens 32768
--chunk_size 256
--max_batch_size 4
--backend_type balance_serve \

反馈：

环境信息

ubuntu2204
X99
512 ddr4
2CPU 56线程
RTX3070 16G
docker 容器中跑（3.1.2.4都正常可以跑。

Originally created by @mikestut on GitHub (Jul 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1445 ### 检查清单 - [ ] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [ ] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [ ] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [ ] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 model_path: str = MODEL_PATH, optimize_config_path: str = ktransformer_rules_dir+'DeepSeek-V3-Chat-serve.yaml', gguf_path: str = GGUF_PATH, # | None = None, max_new_tokens: int =1024, cpu_infer: str = CPU_INFER ,#: int = Config().cpu_infer, use_cuda_graph: bool = True , #False, prompt_file : str | None = None, mode: str = "normal", force_think: bool = False, chunk_size: int =12800 正常加载完成60层。 loading model.layers.60.self_attn.q_a_layernorm.weight to cuda loading model.layers.60.self_attn.kv_a_layernorm.weight to cuda loading model.layers.60.self_attn.kv_b_proj.weight to cuda:0 loading model.layers.60.input_layernorm.weight to cuda loading model.layers.60.post_attention_layernorm.weight to cuda loading model.norm.weight to cuda Chat: 你好显存占用：8G WARNING:transformers_modules.tokenization_kimi:Calling super().encode with {'add_special_tokens': False} 报错如下： File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs) 1742 # If we don't have any hooks, we want to skip the rest of the logic in 1743 # this function, and just call forward. 1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks 1745 or _global_backward_pre_hooks or _global_backward_hooks 1746 or _global_forward_hooks or _global_forward_pre_hooks): -> 1747 return forward_call(*args, **kwargs) 1749 result = None 1750 called_always_called_hooks = set() **TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'** ### 复现步骤 python ktransformers/server/main.py \ --port 10002 \ --model_path <path_to_safetensor_config> \ --gguf_path <path_to_gguf_files> \ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \ --max_new_tokens 1024 \ --cache_lens 32768 \ --chunk_size 256 \ --max_batch_size 4 \ --backend_type balance_serve \ 反馈： <img width="2190" height="887" alt="Image" src="https://github.com/user-attachments/assets/bd7201db-6b7b-448e-87ff-3769dd338c83" /> ### 环境信息 ubuntu2204 X99 512 ddr4 2CPU 56线程 RTX3070 16G docker 容器中跑（3.1.2.4都正常可以跑。

bsakaguchi commented

2026-10-29 14:46:59 +00:00

Author

Owner

@apalesnail commented on GitHub (Jul 16, 2025):

Does it work with the ktransformers backend?

@apalesnail commented on GitHub (Jul 16, 2025): Does it work with the ktransformers backend?

bsakaguchi commented

2026-10-29 14:46:59 +00:00

Author

Owner

@mikestut commented on GitHub (Jul 17, 2025):

Does it work with the ktransformers backend?

Yes, are you have the same problem??

@mikestut commented on GitHub (Jul 17, 2025): > Does it work with the ktransformers backend? Yes, are you have the same problem??

bsakaguchi commented

2026-10-29 14:46:59 +00:00

Author

Owner

@apalesnail commented on GitHub (Jul 17, 2025):

I have seen this before with DeepSeek. Here was the workaround:

Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).

@apalesnail commented on GitHub (Jul 17, 2025): I have seen this before with DeepSeek. Here was the workaround: Hey all, a workaround is to copy all of the small files from hugging face such as modeling_deepseek.py must be in the directory for model_path (unlike the ktransformers backend, which automatically pulls from hugging face).