kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1402] 启动AMX功能报错 #940

New issue

Open

opened 2026-10-29 14:46:33 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented

2026-10-29 14:46:33 +00:00

Owner

Originally created by @kmnns on GitHub (Jun 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1402

在启动local_chat 命令 --optimize_config_path ./....../optimize_rules/DeepSeek-V3-Chat-amx.yaml时报错，在开始部署专家层时显示精度转换问题，请问该如何解决

Originally created by @kmnns on GitHub (Jun 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1402 在启动local_chat 命令 --optimize_config_path ./....../optimize_rules/DeepSeek-V3-Chat-amx.yaml时报错，在开始部署专家层时显示精度转换问题，请问该如何解决 ![Image](https://github.com/user-attachments/assets/3b59f8bd-8c16-4907-bf6e-1be4bf1e681c)

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 27, 2025):

You need to download BF16 GGUF, See AMX doc

@aubreyli commented on GitHub (Jun 27, 2025): You need to download BF16 GGUF, See [AMX doc](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/AMX.md)

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@kmnns commented on GitHub (Jun 27, 2025):

您需要下载 BF16 GGUF，请参阅 AMX 文档

我确认一下哈，我现在用的时q4_K_M的gguf ，我需要重新下载BF16版本的GGUF并替换它们是吗

@kmnns commented on GitHub (Jun 27, 2025): > 您需要下载 BF16 GGUF，请参阅 [AMX 文档](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/AMX.md) 我确认一下哈，我现在用的时q4_K_M的gguf ，我需要重新下载BF16版本的GGUF并替换它们是吗

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@kmnns commented on GitHub (Jun 27, 2025):

一共需要1.3T的那个版本吗

@kmnns commented on GitHub (Jun 27, 2025): 一共需要1.3T的那个版本吗

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 27, 2025):

您需要下载 BF16 GGUF，请参阅 AMX 文档

我确认一下哈，我现在用的时q4_K_M的gguf ，我需要重新下载BF16版本的GGUF并替换它们是吗

Yes

@aubreyli commented on GitHub (Jun 27, 2025): > > 您需要下载 BF16 GGUF，请参阅 [AMX 文档](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/AMX.md) > > 我确认一下哈，我现在用的时q4_K_M的gguf ，我需要重新下载BF16版本的GGUF并替换它们是吗 Yes

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 27, 2025):

一共需要1.3T的那个版本吗

If you have to use DeepSeek 671B, then yes. Otherwise, you might want to use a smaller model like the Qwen3-30B BF16 GGUF to try AMX.

@aubreyli commented on GitHub (Jun 27, 2025): > 一共需要1.3T的那个版本吗 If you have to use DeepSeek 671B, then yes. Otherwise, you might want to use a smaller model like the Qwen3-30B BF16 GGUF to try AMX.

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@trilog-inc commented on GitHub (Jun 27, 2025):

is there a timeline for the AMXInt4 backend?

@trilog-inc commented on GitHub (Jun 27, 2025): is there a timeline for the AMXInt4 backend?

bsakaguchi commented

2026-10-29 14:46:34 +00:00

Author

Owner

@aubreyli commented on GitHub (Jun 30, 2025):

is there a timeline for the AMXInt4 backend?

Currently, AMX hardware mainly supports BF16 and INT8 formats. If you have low-precision weights (such as 4-bit), they must first be dequantized into either BF16 or INT8 before AMX can be used for computation.

@aubreyli commented on GitHub (Jun 30, 2025): > is there a timeline for the AMXInt4 backend? Currently, AMX hardware mainly supports BF16 and INT8 formats. If you have low-precision weights (such as 4-bit), they must first be dequantized into either BF16 or INT8 before AMX can be used for computation.

bsakaguchi commented

2026-10-29 14:46:35 +00:00

Author

Owner

@trilog-inc commented on GitHub (Jun 30, 2025):

I understand. With only 512GB of ram, is there a future where R1/V3 can be used with AMX optimizations?

I only ask because in the AMX docs, the performance results table seems to hint at QWEN3-235B being loaded at 4-bit and consuming 160 GB of system ram. Rough calculations of R1 loaded this way would put it at ~460GB. Not everything is equal obviously..

@trilog-inc commented on GitHub (Jun 30, 2025): I understand. With only 512GB of ram, is there a future where R1/V3 can be used with AMX optimizations? I only ask because in the [AMX docs](https://private-user-images.githubusercontent.com/44601537/438466795-62567aad-353b-4c6f-ab87-2ea283ff2ba2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTEyOTI0MzEsIm5iZiI6MTc1MTI5MjEzMSwicGF0aCI6Ii80NDYwMTUzNy80Mzg0NjY3OTUtNjI1NjdhYWQtMzUzYi00YzZmLWFiODctMmVhMjgzZmYyYmEyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjMwVDE0MDIxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk2MDk1Nzg1MTdhNjQ2ZTZiNThjZDhmNGEwZTZiZThhMmViNjg0N2EwMjFmYzE3OGY1MjFhZjE1NTZlYTFlNTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0._LOxBJR3zoylAxVBACtK132usEK-UtEYPMZq4OKRuNQ), the performance results table seems to hint at QWEN3-235B being loaded at 4-bit and consuming 160 GB of system ram. Rough calculations of R1 loaded this way would put it at ~460GB. Not everything is equal obviously..

bsakaguchi commented

2026-10-29 14:46:35 +00:00

Author

Owner

@always-H commented on GitHub (Dec 3, 2025):

@aubreyli 请问是不是因为gguf的bf16版本和硬件中的bf16版本数据格式布局不同？否则直接使用huggingface上safetensor的bf16版本不行吗

@always-H commented on GitHub (Dec 3, 2025): @aubreyli 请问是不是因为gguf的bf16版本和硬件中的bf16版本数据格式布局不同？否则直接使用huggingface上safetensor的bf16版本不行吗

bsakaguchi commented

2026-10-29 14:46:35 +00:00

Author

Owner

@aubreyli commented on GitHub (Dec 4, 2025):

safetensor BF16 should work. The following webpage for your reference.
https://www.intel.com/content/www/us/en/developer/articles/code-sample/advanced-matrix-extensions-intrinsics-functions.html

@aubreyli commented on GitHub (Dec 4, 2025): safetensor BF16 should work. The following webpage for your reference. https://www.intel.com/content/www/us/en/developer/articles/code-sample/advanced-matrix-extensions-intrinsics-functions.html

bsakaguchi referenced this issue

2026-10-29 14:53:12 +00:00

[PR #940] [MERGED] Update gate.py #1526