kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #156] DeepSeek-R1-Q4_K_M throwing KeyError: 'blk.0.attn_q_a.weight' #86

New issue
Closed
opened 2026-10-29 14:33:52 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-29 14:33:52 +00:00
Owner
Copy link

Originally created by @hetong007 on GitHub (Feb 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/156

I compiled ktransformers following https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#v02-showcase. GGUF downloaded from https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q4_K_M.

Env: python 3.11, cuda 12.4, ubuntu 22.04

(ktransformer) ubuntu@localhost:~/ktransformers$ python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ~/.cache/huggingface/download/DeepSeek-R1-Q4_K_M/ --cpu_infer 65 --max_new_tokens 1000
tokenizer_config.json: 100%|███████████████████████████████████| 3.59k/3.59k [00:00<00:00, 32.3MB/s]
tokenizer.json: 100%|██████████████████████████████████████████| 7.85M/7.85M [00:00<00:00, 18.1MB/s]
config.json: 100%|█████████████████████████████████████████████| 1.73k/1.73k [00:00<00:00, 12.4MB/s]
configuration_deepseek.py: 100%|███████████████████████████████| 10.6k/10.6k [00:00<00:00, 87.2MB/s]
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1:
- configuration_deepseek.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Traceback (most recent call last):
  File "/home/ubuntu/ktransformers/./ktransformers/local_chat.py", line 179, in <module>
    fire.Fire(local_chat)
  File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ktransformers/./ktransformers/local_chat.py", line 110, in local_chat
    optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
  File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
    inject(module, optimize_config, model_config, gguf_loader)
  File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  [Previous line repeated 1 more time]
  File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 33, in inject
    inject_module=module_cls(key = inject_module_meta["key"], gguf_loader = gguf_loader, config = model_config, orig_module=child, **inject_module_meta["kwargs"])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ktransformers/./ktransformers/operators/linear.py", line 378, in __init__
    KLinearBase.__init__(self, key, gguf_loader, config, orig_module, generate_device, **kwargs)
  File "/home/ubuntu/ktransformers/./ktransformers/operators/linear.py", line 61, in __init__
    shape = self.gguf_loader.tensor_info[key + ".weight"]["shape"]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'blk.0.attn_q_a.weight'

Is there something on my side to fix?

Originally created by @hetong007 on GitHub (Feb 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/156 I compiled `ktransformers` following https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#v02-showcase. GGUF downloaded from https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q4_K_M. Env: python 3.11, cuda 12.4, ubuntu 22.04 ``` (ktransformer) ubuntu@localhost:~/ktransformers$ python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ~/.cache/huggingface/download/DeepSeek-R1-Q4_K_M/ --cpu_infer 65 --max_new_tokens 1000 tokenizer_config.json: 100%|███████████████████████████████████| 3.59k/3.59k [00:00<00:00, 32.3MB/s] tokenizer.json: 100%|██████████████████████████████████████████| 7.85M/7.85M [00:00<00:00, 18.1MB/s] config.json: 100%|█████████████████████████████████████████████| 1.73k/1.73k [00:00<00:00, 12.4MB/s] configuration_deepseek.py: 100%|███████████████████████████████| 10.6k/10.6k [00:00<00:00, 87.2MB/s] A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1: - configuration_deepseek.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. using custom modeling_xxx.py. using default_optimize_rule for DeepseekV3ForCausalLM Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Traceback (most recent call last): File "/home/ubuntu/ktransformers/./ktransformers/local_chat.py", line 179, in <module> fire.Fire(local_chat) File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/ktransformers/./ktransformers/local_chat.py", line 110, in local_chat optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config) File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf inject(module, optimize_config, model_config, gguf_loader) File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) [Previous line repeated 1 more time] File "/home/ubuntu/ktransformers/./ktransformers/optimize/optimize.py", line 33, in inject inject_module=module_cls(key = inject_module_meta["key"], gguf_loader = gguf_loader, config = model_config, orig_module=child, **inject_module_meta["kwargs"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/ktransformers/./ktransformers/operators/linear.py", line 378, in __init__ KLinearBase.__init__(self, key, gguf_loader, config, orig_module, generate_device, **kwargs) File "/home/ubuntu/ktransformers/./ktransformers/operators/linear.py", line 61, in __init__ shape = self.gguf_loader.tensor_info[key + ".weight"]["shape"] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ KeyError: 'blk.0.attn_q_a.weight' ``` Is there something on my side to fix?
bsakaguchi commented 2026-10-29 14:33:53 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Feb 12, 2025):

blk.0.attn_q_a.weight

Image
Seems, it has blk.0.attn_q_a.weight in the gguf. You can check if your cached gguf file is correct (by check the hash number). You can also redownload it.

<!-- gh-comment-id:2652848944 --> @KMSorSMS commented on GitHub (Feb 12, 2025): > blk.0.attn_q_a.weight ![Image](https://github.com/user-attachments/assets/c2708ec3-f923-4b8b-800e-7ee96e2993fc) Seems, it has blk.0.attn_q_a.weight in the gguf. You can check if your cached gguf file is correct (by check the hash number). You can also redownload it.
bsakaguchi commented 2026-10-29 14:33:54 +00:00
Author
Owner
Copy link

@hetong007 commented on GitHub (Feb 12, 2025):

Thanks for the quick reply! Indeed.

I fixed the gguf file, and the original error was gone. However it seems to by-default offload all weights into GPU and reported CUDA out of memory error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 504.00 MiB. GPU 0 has a total capacity of 44.31 GiB of which 341.00 MiB is free. Process 1086055 has 41.44 GiB memory in use. Including non-PyTorch memory, this process has 2.52 GiB memory in use. Of the allocated memory 1.61 GiB is allocated by PyTorch, and 444.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I briefly looked at the script, seems it loads some config from ktransformers/optimize/optimize_rules/*.yaml. Can I assume these config files needs to be modified locally (and carefully) to achieve best CPU/GPU balance? My hardware is multi-GPU with 8x48G VRAM. A pointer to a detail documentation is also appreciated, thanks!

<!-- gh-comment-id:2652882491 --> @hetong007 commented on GitHub (Feb 12, 2025): Thanks for the quick reply! Indeed. I fixed the gguf file, and the original error was gone. However it seems to by-default offload all weights into GPU and reported CUDA out of memory error: ``` torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 504.00 MiB. GPU 0 has a total capacity of 44.31 GiB of which 341.00 MiB is free. Process 1086055 has 41.44 GiB memory in use. Including non-PyTorch memory, this process has 2.52 GiB memory in use. Of the allocated memory 1.61 GiB is allocated by PyTorch, and 444.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) ``` I briefly looked at the script, seems it loads some config from `ktransformers/optimize/optimize_rules/*.yaml`. Can I assume these config files needs to be modified locally (and carefully) to achieve best CPU/GPU balance? My hardware is multi-GPU with 8x48G VRAM. A pointer to a detail documentation is also appreciated, thanks!
bsakaguchi commented 2026-10-29 14:33:54 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Feb 12, 2025):

Thanks for the quick reply! Indeed.

I fixed the gguf file, and the original error was gone. However it seems to by-default offload all weights into GPU and reported CUDA out of memory error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 504.00 MiB. GPU 0 has a total capacity of 44.31 GiB of which 341.00 MiB is free. Process 1086055 has 41.44 GiB memory in use. Including non-PyTorch memory, this process has 2.52 GiB memory in use. Of the allocated memory 1.61 GiB is allocated by PyTorch, and 444.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I briefly looked at the script, seems it loads some config from ktransformers/optimize/optimize_rules/*.yaml. Can I assume these config files needs to be modified locally (and carefully) to achieve best CPU/GPU balance? My hardware is multi-GPU with 8x48G VRAM. A pointer to a detail documentation is also appreciated, thanks!

This is abnormal. In fact we offload the weight of experts to CPU ( this process doesn't print out the log). So I guess if there is other process on your device use GPU? We only need 16G VRAM for Q4.

Furthermore, if you want to use multi GPU, you can set the arg: --optimize_rule_path <the path of ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml> . And also you can modify the yml file to fit your case (most of time modify it like change KLinearMarlin to KLinearTorch if your GPU is too old to support marlin or offload some experts layer to GPU if you don't have DRAM larger than 382G ).

<!-- gh-comment-id:2652921124 --> @KMSorSMS commented on GitHub (Feb 12, 2025): > Thanks for the quick reply! Indeed. > > I fixed the gguf file, and the original error was gone. However it seems to by-default offload all weights into GPU and reported CUDA out of memory error: > > ``` > torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 504.00 MiB. GPU 0 has a total capacity of 44.31 GiB of which 341.00 MiB is free. Process 1086055 has 41.44 GiB memory in use. Including non-PyTorch memory, this process has 2.52 GiB memory in use. Of the allocated memory 1.61 GiB is allocated by PyTorch, and 444.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) > ``` > > I briefly looked at the script, seems it loads some config from `ktransformers/optimize/optimize_rules/*.yaml`. Can I assume these config files needs to be modified locally (and carefully) to achieve best CPU/GPU balance? My hardware is multi-GPU with 8x48G VRAM. A pointer to a detail documentation is also appreciated, thanks! This is abnormal. In fact we offload the weight of experts to CPU ( this process doesn't print out the log). So I guess if there is other process on your device use GPU? We only need 16G VRAM for Q4.<br> Furthermore, if you want to use multi GPU, you can set the arg: `--optimize_rule_path <the path of ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml>` . And also you can modify the yml file to fit your case (most of time modify it like change KLinearMarlin to KLinearTorch if your GPU is too old to support marlin or offload some experts layer to GPU if you don't have DRAM larger than 382G ).
bsakaguchi commented 2026-10-29 14:33:54 +00:00
Author
Owner
Copy link

@hetong007 commented on GitHub (Feb 12, 2025):

BTW it might be that the layers are not named exactly as the matching rules in yamls. Would you please provide the source of your Q4 model files? I can double check if there's a discrepancy.

<!-- gh-comment-id:2652927215 --> @hetong007 commented on GitHub (Feb 12, 2025): BTW it might be that the layers are not named exactly as the matching rules in yamls. Would you please provide the source of your Q4 model files? I can double check if there's a discrepancy.
bsakaguchi commented 2026-10-29 14:33:54 +00:00
Author
Owner
Copy link

@KMSorSMS commented on GitHub (Feb 12, 2025):

BTW it might be that the layers are not named exactly as the matching rules in yamls. Would you please provide the source of your Q4 model files? I can double check if there's a discrepancy.

Sure:http://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q4_K_M

<!-- gh-comment-id:2652942343 --> @KMSorSMS commented on GitHub (Feb 12, 2025): > BTW it might be that the layers are not named exactly as the matching rules in yamls. Would you please provide the source of your Q4 model files? I can double check if there's a discrepancy. Sure:http://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q4_K_M
bsakaguchi commented 2026-10-29 14:33:54 +00:00
Author
Owner
Copy link

@hetong007 commented on GitHub (Feb 12, 2025):

I've got it resolved -- turned out to be other process taking GPU RAM. I compared 1 or 2 GPU's speed, they are similarly with impressive speed at 10 tokens/sec. This is about the same speed I achieve with dynamic quantization across all 8 GPUs.

Thanks for this great work!

<!-- gh-comment-id:2653126122 --> @hetong007 commented on GitHub (Feb 12, 2025): I've got it resolved -- turned out to be other process taking GPU RAM. I compared 1 or 2 GPU's speed, they are similarly with impressive speed at 10 tokens/sec. This is about the same speed I achieve with dynamic quantization across all 8 GPUs. Thanks for this great work!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#86
No description provided.