[GH-ISSUE #1333] [Bug] Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file! #897

New issue

Open

opened 2026-10-29 14:45:52 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:45:52 +00:00

Owner

Originally created by @XiongjieDai on GitHub (May 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1333

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

run deepseek-r1-q4_k_m.gguf model and got error

Reproduction

the deepseek-r1-q4_k_m.gguf model is converted by myself and it work fine with llama.cpp
files under the model path:

ls /data/test/
config.json  configuration_deepseek.py  deepseek-r1-q4_k_m.gguf  generation_config.json  modeling_deepseek.py  model.safetensors.index.json  tokenizer_config.json  tokenizer.json

the command:

python -m ktransformers.local_chat --model_path /data/test --gguf_path /data/test --cpu_infer 65 --max_new_tokens 1000

and got error:

flashinfer not found, use triton for linux                                                                                                                                                                                   
using custom modeling_xxx.py.                                                                                                                                                                                                
using default_optimize_rule for DeepseekV3ForCausalLM                                                                                                                                                                        
Injecting model as ktransformers.operators.models . KDeepseekV2Model                                                                                                                                                         
Injecting model.embed_tokens as default                                                                                                                                                                                      
Injecting model.layers as default                                                                                                                                                                                            
Injecting model.layers.0 as default                                                                                                                                                                                          
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention                                                                                                                               
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear                                                                                                                          
Injecting model.layers.0.self_attn.q_a_layernorm as default 

---

Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading model.embed_tokens.weight to cpu
loading model.layers.0.self_attn.q_a_layernorm.weight to cuda
loading model.layers.0.self_attn.kv_a_layernorm.weight to cuda
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ty/jack/ktransformers/ktransformers/local_chat.py", line 190, in <module>
    fire.Fire(local_chat)
  File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ty/jack/ktransformers/ktransformers/local_chat.py", line 117, in local_chat
    optimize_and_load_gguf(model, optimize_config_path, gguf_path, config, default_device=device)
  File "/home/ty/jack/ktransformers/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
    load_weights(module, weights_loader, device=default_device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights
    load_weights(child, gguf_loader, prefix+name+".", device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 154, in load_weights
    module.load()
  File "/home/ty/jack/ktransformers/ktransformers/operators/base_operator.py", line 63, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights
    load_weights(child, gguf_loader, prefix+name+".", device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights
    load_weights(child, gguf_loader, prefix+name+".", device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights
    load_weights(child, gguf_loader, prefix+name+".", device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 154, in load_weights
    module.load()
  File "/home/ty/jack/ktransformers/ktransformers/operators/base_operator.py", line 63, in load
    utils.load_weights(child, self.gguf_loader, self.key+".")
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights
    load_weights(child, gguf_loader, prefix+name+".", device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 150, in load_weights
    load_cur_state_dict(module, gguf_loader, prefix, device=device)
  File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 133, in load_cur_state_dict
    raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file!

Environment

ktransformers version: v0.3.1+cu124torch27fancy

Ubuntu 22.04.5 LTS
Python 3.11
torch = 2.7.0+cu126
CUDA = 12.6

GPU information:
8 NVIDIA H20 GPUs
Video memory per GPU: 97,871 MiB (about 95.6 GB)
All GPUs are currently idle
GPU temperature is between 31-35°C
Power consumption per GPU is between 108-117W (maximum power consumption 500W)

CPU information:
Model: Intel(R) Xeon(R) Platinum 8480
Number of CPU cores: 224 (2 physical CPUs, 56 cores each, 2 threads per core)
Base frequency: 800 MHz
Maximum frequency: 3.8 GHz
Cache:
L1d cache: 5.3 MiB
L1i cache: 3.5 MiB
L2 cache: 224 MiB
L3 cache: 210 MiB
NUMA supported, with 2 NUMA nodes

Memory information:
Total memory: 2.0 TB
Used: 15 GB
Available: 1.9 TB
Cache: 106 GB
Swap: 8 GB (unused)

Originally created by @XiongjieDai on GitHub (May 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1333 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug run ` deepseek-r1-q4_k_m.gguf ` model and got error ### Reproduction the ` deepseek-r1-q4_k_m.gguf ` model is converted by myself and it work fine with `llama.cpp` files under the model path: ``` ls /data/test/ config.json configuration_deepseek.py deepseek-r1-q4_k_m.gguf generation_config.json modeling_deepseek.py model.safetensors.index.json tokenizer_config.json tokenizer.json ``` the command: ``` python -m ktransformers.local_chat --model_path /data/test --gguf_path /data/test --cpu_infer 65 --max_new_tokens 1000 ``` and got error: ``` flashinfer not found, use triton for linux using custom modeling_xxx.py. using default_optimize_rule for DeepseekV3ForCausalLM Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.0.self_attn.q_a_layernorm as default --- Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as ktransformers.operators.linear . KTransformersLinear loading model.embed_tokens.weight to cpu loading model.layers.0.self_attn.q_a_layernorm.weight to cuda loading model.layers.0.self_attn.kv_a_layernorm.weight to cuda Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/ty/jack/ktransformers/ktransformers/local_chat.py", line 190, in <module> fire.Fire(local_chat) File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/ty/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ty/jack/ktransformers/ktransformers/local_chat.py", line 117, in local_chat optimize_and_load_gguf(model, optimize_config_path, gguf_path, config, default_device=device) File "/home/ty/jack/ktransformers/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf load_weights(module, weights_loader, device=default_device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 154, in load_weights module.load() File "/home/ty/jack/ktransformers/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 154, in load_weights module.load() File "/home/ty/jack/ktransformers/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 152, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 150, in load_weights load_cur_state_dict(module, gguf_loader, prefix, device=device) File "/home/ty/jack/ktransformers/ktransformers/util/utils.py", line 133, in load_cur_state_dict raise Exception(f"can't find {translated_key} in GGUF file!") Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file! ``` ### Environment **ktransformers version: v0.3.1+cu124torch27fancy** Ubuntu 22.04.5 LTS Python 3.11 torch = 2.7.0+cu126 CUDA = 12.6 GPU information: 8 NVIDIA H20 GPUs Video memory per GPU: 97,871 MiB (about 95.6 GB) All GPUs are currently idle GPU temperature is between 31-35°C Power consumption per GPU is between 108-117W (maximum power consumption 500W) CPU information: Model: Intel(R) Xeon(R) Platinum 8480 Number of CPU cores: 224 (2 physical CPUs, 56 cores each, 2 threads per core) Base frequency: 800 MHz Maximum frequency: 3.8 GHz Cache: L1d cache: 5.3 MiB L1i cache: 3.5 MiB L2 cache: 224 MiB L3 cache: 210 MiB NUMA supported, with 2 NUMA nodes Memory information: Total memory: 2.0 TB Used: 15 GB Available: 1.9 TB Cache: 106 GB Swap: 8 GB (unused)

bsakaguchi commented

2026-10-29 14:45:52 +00:00

Author

Owner

@XiongjieDai commented on GitHub (May 22, 2025):

similar issues are #473 and error 1 in #289

@XiongjieDai commented on GitHub (May 22, 2025): similar issues are #473 and error 1 in #289

bsakaguchi commented

2026-10-29 14:45:52 +00:00

Author

Owner

@luzamm commented on GitHub (May 23, 2025):

which gguf do you use? some new quants modifed the model architecture like unsloth/DeepSeek-V3-0324-GGUF-UD. The new model splitted attn_kv_b.weight to attn_k_b.weight and attn_v_b.weight. Check them on huggingface with

@luzamm commented on GitHub (May 23, 2025): which gguf do you use? some new quants modifed the model architecture like unsloth/DeepSeek-V3-0324-GGUF-UD. The new model splitted attn_kv_b.weight to attn_k_b.weight and attn_v_b.weight. Check them on huggingface with ![Image](https://github.com/user-attachments/assets/14917215-f54f-4bf7-86c1-4c00e3f5e93c)

bsakaguchi commented

2026-10-29 14:45:53 +00:00

Author

Owner

@luzamm commented on GitHub (May 23, 2025):

unsloth/DeepSeek-V3-0324-GGUF gguf that work well

unsloth/DeepSeek-V3-0324-GGUF-UD throw Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file!

@luzamm commented on GitHub (May 23, 2025): ![Image](https://github.com/user-attachments/assets/b26af163-0e51-4773-aaac-d2040af74bd5) unsloth/DeepSeek-V3-0324-GGUF gguf that work well ![Image](https://github.com/user-attachments/assets/98afba72-459a-4671-a956-e3883555e5f8) unsloth/DeepSeek-V3-0324-GGUF-UD throw Exception: can't find model.layers.0.self_attn.kv_b_proj.weight in GGUF file!

bsakaguchi referenced this issue

2026-10-29 14:53:01 +00:00

[PR #897] [MERGED] Add Unit Test for Local Chat #1507