kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #836] [Bug] v0.2.3 FP8 线性内核 4090报错f8e4m3nv is only supported on compute capability >= 90 #602

New issue

Open

opened 2026-10-29 14:41:47 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented

2026-10-29 14:41:47 +00:00

Owner

Originally created by @pitaozi on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/836

Originally assigned to: @Azure-Tang on GitHub.

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

https://github.com/kvcache-ai/ktransformers/blob/v0.2.3/doc/en/fp8_kernel.md
我尝试在v0.2.3上使用DeepSeek-V3/R1 的 FP8 线性内核，从下面的网址上下载预合并权重文件：

笔记
⚠️硬件要求

建议 FP8 内核至少配备 19GB 可用 VRAM。
需要支持 FP8 的 GPU（例如 4090）

KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid
并配置--optimize_config_path DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml，运行后出现如下图所示报错：
f8e4m3nv is only supported on compute capability >= 90

通过查询后得知：
f8e4m3nv 这类与 FP8 编码相关的特性（它是 NVIDIA Hopper 架构引入的一种 FP8 数据格式，用于高效深度学习计算），目前主要是在支持 SM90（如 H100 GPU 所采用的架构）及更新一代架构的 GPU 上才有，而像基于 SM89 架构（如 A100 GPU 所采用 Ampere 架构）的 GPU 并不直接支持。

Reproduction

模型：DeepSeek-R1-GGML-FP8-Hybrid
运行脚本：python -m ktransformers.local_chat --model_path /data/model/DeepSeek-R1-config --gguf_path /data/model/DeepSeek-R1-GGML-FP8-Hybrid --cpu_infer 80 --max_new_tokens 3000 --optimize_config_path DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml
分支：v0.2.3，env -u USE_NUMA make dev_install

Environment

CPU：Inter Xeon Gold 6342 CPU
GPU：4090，sm89
内存：512G
NUMA：2numa

Originally created by @pitaozi on GitHub (Mar 7, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/836 Originally assigned to: @Azure-Tang on GitHub. ### Checklist - [ ] 1. I have searched related issues but cannot get the expected help. - [ ] 2. The bug has not been fixed in the latest version. - [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug https://github.com/kvcache-ai/ktransformers/blob/v0.2.3/doc/en/fp8_kernel.md 我尝试在v0.2.3上使用DeepSeek-V3/R1 的 FP8 线性内核，从下面的网址上下载预合并权重文件：笔记 ⚠️硬件要求建议 FP8 内核至少配备 19GB 可用 VRAM。需要支持 FP8 的 GPU（例如 4090） [KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid](https://huggingface.co/KVCache-ai/DeepSeek-R1) 并配置--optimize_config_path DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml，运行后出现如下图所示报错： f8e4m3nv is only supported on compute capability >= 90 通过查询后得知： f8e4m3nv 这类与 FP8 编码相关的特性（它是 NVIDIA Hopper 架构引入的一种 FP8 数据格式，用于高效深度学习计算），目前主要是在支持 SM90（如 H100 GPU 所采用的架构）及更新一代架构的 GPU 上才有，而像基于 SM89 架构（如 A100 GPU 所采用 Ampere 架构）的 GPU 并不直接支持。 ![Image](https://github.com/user-attachments/assets/053f901e-4da1-4c90-88e8-f4076cc59c33) ### Reproduction 模型：DeepSeek-R1-GGML-FP8-Hybrid 运行脚本：python -m ktransformers.local_chat --model_path /data/model/DeepSeek-R1-config --gguf_path /data/model/DeepSeek-R1-GGML-FP8-Hybrid --cpu_infer 80 --max_new_tokens 3000 --optimize_config_path DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml 分支：v0.2.3，env -u USE_NUMA make dev_install ### Environment CPU：Inter Xeon Gold 6342 CPU GPU：4090，sm89 内存：512G NUMA：2numa

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Mar 8, 2025):

4090支持fp8运算。升级一下triton，triton版本不够也会报这个错误

@Azure-Tang commented on GitHub (Mar 8, 2025): 4090支持fp8运算。升级一下triton，triton版本不够也会报这个错误

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@pitaozi commented on GitHub (Mar 10, 2025):

4090支持fp8运算。升级一下triton，triton版本不够也会报这个错误

谢谢大佬，我升级版后解决了问题。
torch从2.3.0升级到2.6.0，triton自动从2.3.0升级到3.2.0版本了
torchvision从0.18.0升级到0.21.0
torchaudio从2.3.0升级到2.6.0

@pitaozi commented on GitHub (Mar 10, 2025): > 4090支持fp8运算。升级一下triton，triton版本不够也会报这个错误谢谢大佬，我升级版后解决了问题。 torch从2.3.0升级到2.6.0，triton自动从2.3.0升级到3.2.0版本了 torchvision从0.18.0升级到0.21.0 torchaudio从2.3.0升级到2.6.0

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@pitaozi commented on GitHub (Mar 10, 2025):

另外，问题解决后，我问了个默认的问题：
Please write a piece of quicksort code in C++.
然后回答感觉很乱，既有中文也有英文，如下图所示：

@pitaozi commented on GitHub (Mar 10, 2025): 另外，问题解决后，我问了个默认的问题： Please write a piece of quicksort code in C++. 然后回答感觉很乱，既有中文也有英文，如下图所示： ![Image](https://github.com/user-attachments/assets/3a634fd8-a2f7-411c-93c2-0a420bdd61b1)

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Mar 10, 2025):

/data/model/DeepSeek-R1-GGML-FP8-Hybrid跑的是哪一个hybrid权重？

@Azure-Tang commented on GitHub (Mar 10, 2025): /data/model/DeepSeek-R1-GGML-FP8-Hybrid跑的是哪一个hybrid权重？

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@pitaozi commented on GitHub (Mar 10, 2025):

/data/model/DeepSeek-R1-GGML-FP8-Hybrid跑的是哪一个hybrid权重？

https://huggingface.co/KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid/tree/main/DeepSeek-R1-q4km-fp8

@pitaozi commented on GitHub (Mar 10, 2025): > /data/model/DeepSeek-R1-GGML-FP8-Hybrid跑的是哪一个hybrid权重？ https://huggingface.co/KVCache-ai/DeepSeek-R1-GGML-FP8-Hybrid/tree/main/DeepSeek-R1-q4km-fp8

bsakaguchi commented

2026-10-29 14:41:48 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Mar 10, 2025):

我这边测试应该是正常的。你是哪个版本呢？在0.2.3post1之前有可能有flashinfer导致的降智问题。

Btw，用r1的话记得多传一个--force_think True

@Azure-Tang commented on GitHub (Mar 10, 2025): 我这边测试应该是正常的。你是哪个版本呢？在0.2.3post1之前有可能有flashinfer导致的降智问题。 Btw，用r1的话记得多传一个`--force_think True`

bsakaguchi commented

2026-10-29 14:41:49 +00:00

Author

Owner

@pitaozi commented on GitHub (Mar 10, 2025):

我这边测试应该是正常的。你是哪个版本呢？在0.2.3post1之前有可能有flashinfer导致的降智问题。

Btw，用r1的话记得多传一个--force_think True

切到下面这个分支，然后手动编译的：https://github.com/kvcache-ai/ktransformers/tree/v0.2.3
应该是没有用到flashinfer的，flashinfer not found, use triton for linux
我尝试了--force_think True，相比False，多了<think>.....</think>，</think>输出后也是有相同的问题
我拉取了v0.2.3post1版本的代码，然后重新编译（env -u USE_NUMA make dev_install），看起来问题还是存在的

@pitaozi commented on GitHub (Mar 10, 2025): > 我这边测试应该是正常的。你是哪个版本呢？在0.2.3post1之前有可能有flashinfer导致的降智问题。 > > Btw，用r1的话记得多传一个`--force_think True` 切到下面这个分支，然后手动编译的：https://github.com/kvcache-ai/ktransformers/tree/v0.2.3 应该是没有用到flashinfer的，flashinfer not found, use triton for linux 我尝试了--force_think True，相比False，多了\<think>.....\</think>，\</think>输出后也是有相同的问题我拉取了v0.2.3post1版本的代码，然后重新编译（env -u USE_NUMA make dev_install），看起来问题还是存在的

bsakaguchi commented

2026-10-29 14:41:49 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Mar 10, 2025):

跑其它比如q4km会有相同的问题吗？

@Azure-Tang commented on GitHub (Mar 10, 2025): 跑其它比如q4km会有相同的问题吗？

bsakaguchi commented

2026-10-29 14:41:49 +00:00

Author

Owner

@pitaozi commented on GitHub (Mar 10, 2025):