[PR #643] [MERGED] [feat] Support fp8 linear kernel; #1441

New issue

Closed

opened 2026-10-29 14:52:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:52:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/ktransformers/pull/643
Author: @Azure-Tang
Created: 2/24/2025
Status: ✅ Merged
Merged: 2/25/2025
Merged by: @Azure-Tang

Base: develop-0.2.2 ← Head: support-fp8

📝 Commits (5)

7b7c6a6 Add fp8 linear kernel;\n Add empty cache to fit in 16G VRAM; By 'wkGCaSS - 知乎 https://zhuanlan.zhihu.com/p/25491611225'
581a524 Add data loader to read special weights for fp8; Add special weight process script
ca7366d Merge remote-tracking branch 'upstream/develop-0.2.2' into support-fp8
4dc5518 update fp8 kernel tutorial
36fbeee Update doc

📊 Changes

21 files changed (+904 additions, -37 deletions)

View changed files

📝 README.md (+3 -2)
📝 README_ZH.md (+9 -8)
📝 doc/SUMMARY.md (+2 -1)
📝 doc/en/FAQ.md (+5 -2)
➕ doc/en/fp8_kernel.md (+74 -0)
📝 doc/en/injection_tutorial.md (+1 -0)
📝 doc/en/install.md (+1 -1)
➕ ktransformers/ktransformers_ext/triton/fp8gemm.py (+192 -0)
📝 ktransformers/local_chat.py (+5 -5)
📝 ktransformers/operators/experts.py (+10 -1)
📝 ktransformers/operators/gate.py (+10 -3)
📝 ktransformers/operators/linear.py (+68 -5)
➕ ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml (+63 -0)
📝 ktransformers/server/backend/interfaces/ktransformers.py (+3 -3)
📝 ktransformers/server/backend/interfaces/transformers.py (+1 -1)
➕ ktransformers/tests/triton_fp8gemm_test.py (+116 -0)
📝 ktransformers/util/custom_gguf.py (+24 -1)
➕ ktransformers/util/custom_loader.py (+86 -0)
📝 ktransformers/util/utils.py (+14 -3)
➕ merge_tensors/merge_safetensor_gguf.py (+214 -0)

...and 1 more files

📄 Description

Support fp8gemm linear kernel; (This requires to load special weights)
Add weight process script.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/ktransformers/pull/643 **Author:** [@Azure-Tang](https://github.com/Azure-Tang) **Created:** 2/24/2025 **Status:** ✅ Merged **Merged:** 2/25/2025 **Merged by:** [@Azure-Tang](https://github.com/Azure-Tang) **Base:** `develop-0.2.2` ← **Head:** `support-fp8` --- ### 📝 Commits (5) - [`7b7c6a6`](https://github.com/kvcache-ai/ktransformers/commit/7b7c6a657d8f4bb7a0e330dd91c03a3e81e802f4) Add fp8 linear kernel;\n Add empty cache to fit in 16G VRAM; By 'wkGCaSS - 知乎 https://zhuanlan.zhihu.com/p/25491611225' - [`581a524`](https://github.com/kvcache-ai/ktransformers/commit/581a524f65db422011d9a8db99439d658223db6f) Add data loader to read special weights for fp8; Add special weight process script - [`ca7366d`](https://github.com/kvcache-ai/ktransformers/commit/ca7366d2dbb81751e65f1fa8b7b5fba9165f9f42) Merge remote-tracking branch 'upstream/develop-0.2.2' into support-fp8 - [`4dc5518`](https://github.com/kvcache-ai/ktransformers/commit/4dc5518e4d2ae89a687709bcbe05d2f3f80e00ad) update fp8 kernel tutorial - [`36fbeee`](https://github.com/kvcache-ai/ktransformers/commit/36fbeee341e283a93b6befa2a4d9085b7a5dd2b1) Update doc ### 📊 Changes **21 files changed** (+904 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+3 -2) 📝 `README_ZH.md` (+9 -8) 📝 `doc/SUMMARY.md` (+2 -1) 📝 `doc/en/FAQ.md` (+5 -2) ➕ `doc/en/fp8_kernel.md` (+74 -0) 📝 `doc/en/injection_tutorial.md` (+1 -0) 📝 `doc/en/install.md` (+1 -1) ➕ `ktransformers/ktransformers_ext/triton/fp8gemm.py` (+192 -0) 📝 `ktransformers/local_chat.py` (+5 -5) 📝 `ktransformers/operators/experts.py` (+10 -1) 📝 `ktransformers/operators/gate.py` (+10 -3) 📝 `ktransformers/operators/linear.py` (+68 -5) ➕ `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-fp8-linear-ggml-experts.yaml` (+63 -0) 📝 `ktransformers/server/backend/interfaces/ktransformers.py` (+3 -3) 📝 `ktransformers/server/backend/interfaces/transformers.py` (+1 -1) ➕ `ktransformers/tests/triton_fp8gemm_test.py` (+116 -0) 📝 `ktransformers/util/custom_gguf.py` (+24 -1) ➕ `ktransformers/util/custom_loader.py` (+86 -0) 📝 `ktransformers/util/utils.py` (+14 -3) ➕ `merge_tensors/merge_safetensor_gguf.py` (+214 -0) _...and 1 more files_ </details> ### 📄 Description Support fp8gemm linear kernel; (This requires to load special weights) Add weight process script. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:52:27 +00:00