kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1374] [Bug] ktransformers 0.31跑deepseekr1报错assert self.gate_type == GGMLQuantizationType.BF16 #920

New issue
Open
opened 2026-10-29 14:46:08 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 14:46:08 +00:00
Owner
Copy link

Originally created by @south-ocean on GitHub (Jun 8, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1374

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

我使用ktransformers-0.3.1版本跑deepseekr1,CPU是Intel(R) Xeon(R) Platinum 8468V，运行指令是USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yaml，
报错日志如下：
loading model.layers.1.self_attn.q_a_layernorm.weight to cuda
loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.1.self_attn.kv_b_proj.weight to cuda
loading model.layers.1.input_layernorm.weight to cuda
loading model.layers.1.post_attention_layernorm.weight to cuda
loading model.layers.2.self_attn.q_a_layernorm.weight to cuda
loading model.layers.2.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.2.self_attn.kv_b_proj.weight to cuda
loading model.layers.2.input_layernorm.weight to cuda
loading model.layers.2.post_attention_layernorm.weight to cuda
loading model.layers.3.self_attn.q_a_layernorm.weight to cuda
loading model.layers.3.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.3.self_attn.kv_b_proj.weight to cuda
Traceback (most recent call last):
File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 191, in
fire.Fire(local_chat)
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 118, in local_chat
optimize_and_load_gguf(model, optimize_config_path, gguf_path, config, default_device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, weights_loader, device=default_device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 684, in load
self.generate_experts.load(w, warmup=warmup)
File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 226, in load
assert self.gate_type == GGMLQuantizationType.BF16
具体配置及报错见下图：
请问下intel的amx优化是只支持BF16的权重在线量化成int8（CPU端）及int4（GPU端）么？是否支持DeepSeek-R1-Q4_K_M？amx具体的优势场景有说明么？

Reproduction

USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yam
DeepSeek-R1-Q4_K_M

Environment

L20 GPU Intel(R) Xeon(R) Platinum 8468V CPU

Originally created by @south-ocean on GitHub (Jun 8, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1374 ### Checklist - [ ] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug 我使用ktransformers-0.3.1版本跑deepseekr1,CPU是Intel(R) Xeon(R) Platinum 8468V，运行指令是USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yaml， 报错日志如下： loading model.layers.1.self_attn.q_a_layernorm.weight to cuda loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda loading model.layers.1.self_attn.kv_b_proj.weight to cuda loading model.layers.1.input_layernorm.weight to cuda loading model.layers.1.post_attention_layernorm.weight to cuda loading model.layers.2.self_attn.q_a_layernorm.weight to cuda loading model.layers.2.self_attn.kv_a_layernorm.weight to cuda loading model.layers.2.self_attn.kv_b_proj.weight to cuda loading model.layers.2.input_layernorm.weight to cuda loading model.layers.2.post_attention_layernorm.weight to cuda loading model.layers.3.self_attn.q_a_layernorm.weight to cuda loading model.layers.3.self_attn.kv_a_layernorm.weight to cuda loading model.layers.3.self_attn.kv_b_proj.weight to cuda Traceback (most recent call last): File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 191, in <module> fire.Fire(local_chat) File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 118, in local_chat optimize_and_load_gguf(model, optimize_config_path, gguf_path, config, default_device=device) File "/home/kt-0.3/ktransformers/./ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf load_weights(module, weights_loader, device=default_device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights module.load() File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights module.load() File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights load_weights(child, gguf_loader, prefix+name+".", device=device) File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights module.load() File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 684, in load self.generate_experts.load(w, warmup=warmup) File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 226, in load assert self.gate_type == GGMLQuantizationType.BF16 具体配置及报错见下图： 请问下intel的amx优化是只支持BF16的权重在线量化成int8（CPU端）及int4（GPU端）么？是否支持DeepSeek-R1-Q4_K_M？amx具体的优势场景有说明么？ <!-- Failed to upload "image.png" --> <!-- Failed to upload "image.png" --> ### Reproduction USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yam DeepSeek-R1-Q4_K_M ### Environment L20 GPU Intel(R) Xeon(R) Platinum 8468V CPU
bsakaguchi commented 2026-10-29 14:46:09 +00:00
Author
Owner
Copy link

@MeJerry215 commented on GitHub (Jun 9, 2025):

r1-q4_k_M 的gate_type Q4_K = 12， 已经量化后的weight，所以如果要能用这个 看起来应该需要支持 将Q4_K 反量化为 BF16 load weight的时候。
不过我看他说使用deepseek-v3 int8 性能达到了 418 TPS, 这个性能怎么测试出来的。。

<!-- gh-comment-id:2954500643 --> @MeJerry215 commented on GitHub (Jun 9, 2025): r1-q4_k_M 的gate_type Q4_K = 12， 已经量化后的weight，所以如果要能用这个 看起来应该需要支持 将Q4_K 反量化为 BF16 load weight的时候。 不过我看他说使用deepseek-v3 int8 性能达到了 418 TPS, 这个性能怎么测试出来的。。
bsakaguchi commented 2026-10-29 14:46:09 +00:00
Author
Owner
Copy link

@qiuqiu10 commented on GitHub (Jun 10, 2025):

intel的amx优化目前只支持BF16的权重格式，DeepSeek-R1-Q4_K_M请用DeepSeek-V3-Chat-serve.yaml或DeepSeek-V3-Chat.yaml

<!-- gh-comment-id:2957526970 --> @qiuqiu10 commented on GitHub (Jun 10, 2025): intel的amx优化目前只支持BF16的权重格式，DeepSeek-R1-Q4_K_M请用DeepSeek-V3-Chat-serve.yaml或DeepSeek-V3-Chat.yaml
bsakaguchi commented 2026-10-29 14:46:09 +00:00
Author
Owner
Copy link

@Big2Wheel commented on GitHub (Oct 27, 2025):

大佬你好，请问下DeepSeek-R1-Q4_K_M模型权重现在可以支持AMXInt8了嘛

<!-- gh-comment-id:3451043152 --> @Big2Wheel commented on GitHub (Oct 27, 2025): 大佬你好，请问下DeepSeek-R1-Q4_K_M模型权重现在可以支持AMXInt8了嘛
bsakaguchi commented 2026-10-29 14:46:09 +00:00
Author
Owner
Copy link

@Big2Wheel commented on GitHub (Oct 27, 2025):

intel的amx优化目前只支持BF16的权重格式，DeepSeek-R1-Q4_K_M请用DeepSeek-V3-Chat-serve.yaml或DeepSeek-V3-Chat.yaml

大佬你好，请问下DeepSeek-R1-Q4_K_M模型权重现在可以支持AMXInt8了嘛

<!-- gh-comment-id:3451135091 --> @Big2Wheel commented on GitHub (Oct 27, 2025): > intel的amx优化目前只支持BF16的权重格式，DeepSeek-R1-Q4_K_M请用DeepSeek-V3-Chat-serve.yaml或DeepSeek-V3-Chat.yaml 大佬你好，请问下DeepSeek-R1-Q4_K_M模型权重现在可以支持AMXInt8了嘛
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#920
No description provided.