mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1347] [Feature] Support for unsloth/DeepSeek-R1-0528-GGUF #909
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#909
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @344303947 on GitHub (May 30, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1347
Checklist
Motivation
DeepSeek-R1-0528-UD-Q2_K_XL
Related resources
loading blk.0.attn_q_a_norm.weight to cuda
loading blk.0.attn_kv_a_norm.weight to cuda
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/sean/miniconda3/envs/kt/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 244, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 162, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 123, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/home/sean/miniconda3/envs/kt/lib/python3.10/site-packages/ktransformers/util/utils.py", line 118, in load_cur_state_dict
raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find blk.0.attn_kv_b.weight in GGUF file!
@apalesnail commented on GitHub (May 30, 2025):
Same issue observed in Q4_K_M deepseek r1 0528
@zhouye commented on GitHub (May 30, 2025):
The UD-Q2_K_XL version of DeepSeek-R1 0528 splits the KV tensor into standalone K and V tensors. The reason of doing this might be K and V needs to be quantized into different data types.
The support of this feature was in the upstream llama.cpp (https://github.com/ggml-org/llama.cpp/pull/12801) but not included in ktransformer master.
@qiyuxinlin commented on GitHub (Jun 1, 2025):
The issue has been resolved.