deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[PR #378] [CLOSED] feat: support sm90_fp8_fp4_g128 kernel #402

New issue
Closed
opened 2026-10-26 21:17:17 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:17:17 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/378
Author: @shiyang814-cpu
Created: 7/14/2026
Status: ❌ Closed

Base: main ← Head: sm90-fp8-fp4-g128-cleanup

📝 Commits (10+)

  • 6e12aeb Create test_sm90_fp8_fp4.py
  • b118d88 Create sm90_fp8_fp4_gemm_1d1d.cuh
  • 561cf21 Create sm90_fp8_fp4_gemm_1d2d.cuh
  • 4122a7d Update gemm.hpp
  • b3d92e0 Update init.py
  • 825cbdb Update init.py
  • 9625f4f Update test_sm90_fp8_fp4.py
  • 6a3b1ee Merge pull request #1 from zhangxiaolei123456/zhangxiaolei123456-patch-1
  • 043f007 Create sm90_fp8_fp4_gemm_1d2d.hpp
  • eebc4f4 Update sm90_fp8_fp4_gemm_1d2d.hpp

📊 Changes

16 files changed (+8670 additions, -10 deletions)

View changed files

📝 csrc/apis/gemm.hpp (+106 -0)
📝 csrc/apis/layout.hpp (+17 -2)
➕ csrc/jit_kernels/impls/sm90_fp8_fp4_gemm_1d2d.hpp (+1551 -0)
➕ csrc/jit_kernels/impls/sm90_fp8_fp4_gemm_1d2d_rs.hpp (+359 -0)
📝 csrc/jit_kernels/impls/smxx_layout.hpp (+19 -3)
📝 csrc/utils/layout.hpp (+14 -4)
📝 deep_gemm/__init__.py (+19 -0)
➕ deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d1d.cuh (+396 -0)
➕ deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d2d.cuh (+713 -0)
➕ deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d2d_rs.cuh (+3236 -0)
📝 deep_gemm/include/deep_gemm/mma/sm90.cuh (+70 -0)
📝 deep_gemm/include/deep_gemm/ptx/ld_st.cuh (+36 -0)
📝 deep_gemm/utils/math.py (+82 -1)
➕ tests/test_sm90_fp8_fp4.py (+921 -0)
➕ tests/test_sm90_fp8_fp4_g128.py (+607 -0)
➕ tests/test_sm90_int4_a8.py (+524 -0)

📄 Description

This PR adds support and validation for the SM90 FP8 × FP4 GEMM path with FP4 scales stored in FP32 and grouped by group_size=128 (g128). It differs from the previous feat: support sm90_fp8_fp4 kernel work which primarily focused on the existing FP4 scaling/fast-path configurations .

In addition, we simplified related masked-kernel code by removing redundant runtime switches for the masked operator, making the implementation easier to read and maintain.


python /sgl-workspace/sglang/test_sm90_fp8_fp4_g128.py --repo-root /sgl-workspace/DeepGEMM

Contiguous FP8xFP4(g128) benchmark

groups m/group n k W4 us W4 GB/s W4 diff FP8 us FP8 GB/s FP8 diff Speedup
8 1 4096 7168 327 431 0.0000 240 1074 0.0184 0.74x
8 1 7168 2048 178 444 0.0000 134 1030 0.0185 0.75x
8 1 4096 4096 195 430 0.0000 145 1045 0.0185 0.74x
8 4 4096 7168 326 431 0.0000 240 1076 0.0180 0.74x
8 4 7168 2048 179 444 0.0000 132 1042 0.0182 0.74x
8 4 4096 4096 197 427 0.0000 145 1039 0.0183 0.74x
8 8 4096 7168 326 432 0.0000 240 1075 0.0182 0.74x
8 8 7168 2048 178 444 0.0000 132 1044 0.0182 0.74x
8 8 4096 4096 196 428 0.0000 146 1036 0.0185 0.74x
8 16 4096 7168 327 430 0.0000 241 1073 0.0183 0.74x
8 16 7168 2048 179 443 0.0000 132 1043 0.0181 0.74x
8 16 4096 4096 196 429 0.0000 145 1044 0.0186 0.74x
8 32 4096 7168 326 432 0.0000 241 1073 0.0184 0.74x
8 32 7168 2048 178 444 0.0000 134 1033 0.0184 0.75x
8 32 4096 4096 196 429 0.0000 145 1042 0.0184 0.74x
8 64 4096 7168 327 431 0.0000 240 1076 0.0187 0.73x
8 64 7168 2048 179 442 0.0000 133 1037 0.0184 0.74x
8 64 4096 4096 197 427 0.0000 146 1039 0.0186 0.74x
16 1 4096 7168 637 442 0.0000 513 1006 0.0184 0.81x
16 1 7168 2048 334 474 0.0000 252 1095 0.0182 0.75x
16 1 4096 4096 380 442 0.0000 303 996 0.0183 0.80x
16 4 4096 7168 638 441 0.0000 514 1005 0.0182 0.81x
16 4 7168 2048 335 474 0.0000 252 1095 0.0185 0.75x
16 4 4096 4096 380 442 0.0000 303 996 0.0183 0.80x
16 8 4096 7168 638 441 0.0000 514 1005 0.0184 0.81x
16 8 7168 2048 334 475 0.0000 253 1090 0.0184 0.76x
16 8 4096 4096 380 442 0.0000 304 994 0.0182 0.80x
16 16 4096 7168 639 440 0.0000 514 1005 0.0183 0.80x
16 16 7168 2048 334 475 0.0000 252 1095 0.0184 0.76x
16 16 4096 4096 380 442 0.0000 303 996 0.0183 0.80x
16 32 4096 7168 639 441 0.0000 514 1005 0.0182 0.80x
16 32 7168 2048 335 473 0.0000 253 1091 0.0185 0.76x
16 32 4096 4096 381 442 0.0000 304 996 0.0181 0.80x
16 64 4096 7168 639 441 0.0000 515 1004 0.0182 0.81x
16 64 7168 2048 334 475 0.0000 252 1094 0.0182 0.76x
16 64 4096 4096 381 441 0.0000 304 995 0.0183 0.80x
32 1 4096 7168 1222 461 0.0000 937 1102 0.0182 0.77x
32 1 7168 2048 656 483 0.0000 491 1124 0.0181 0.75x
32 1 4096 4096 722 465 0.0000 548 1104 0.0184 0.76x
32 4 4096 7168 1221 461 0.0000 935 1104 0.0183 0.77x
32 4 7168 2048 656 483 0.0000 491 1124 0.0185 0.75x
32 4 4096 4096 722 465 0.0000 549 1102 0.0183 0.76x
32 8 4096 7168 1222 461 0.0000 937 1102 0.0183 0.77x
32 8 7168 2048 657 483 0.0000 491 1124 0.0183 0.75x
32 8 4096 4096 722 465 0.0000 549 1102 0.0183 0.76x
32 16 4096 7168 1222 461 0.0000 937 1102 0.0183 0.77x
32 16 7168 2048 657 483 0.0000 491 1124 0.0182 0.75x
32 16 4096 4096 722 465 0.0000 548 1104 0.0184 0.76x
32 32 4096 7168 1222 461 0.0000 937 1103 0.0184 0.77x
32 32 7168 2048 657 482 0.0000 491 1123 0.0184 0.75x
32 32 4096 4096 723 465 0.0000 548 1103 0.0184 0.76x
32 64 4096 7168 1224 460 0.0000 938 1101 0.0183 0.77x
32 64 7168 2048 658 482 0.0000 491 1123 0.0182 0.75x
32 64 4096 4096 726 463 0.0000 550 1100 0.0185 0.76x

Masked FP8xFP4(g128) benchmark

groups m/group n k W4 us W4 GB/s W4 diff FP8 us FP8 GB/s FP8 diff Speedup
8 1 4096 7168 99 1264 0.0000 142 1706 0.0196 1.44x
8 1 7168 2048 53 1180 0.0000 78 1552 0.0199 1.47x
8 1 4096 4096 62 1145 0.0000 88 1574 0.0189 1.41x
8 4 4096 7168 100 1256 0.0000 141 1717 0.0195 1.42x
8 4 7168 2048 53 1184 0.0000 79 1547 0.0189 1.48x
8 4 4096 4096 63 1137 0.0000 88 1575 0.0197 1.40x
8 8 4096 7168 99 1277 0.0000 141 1722 0.0190 1.43x
8 8 7168 2048 53 1204 0.0000 78 1562 0.0188 1.48x
8 8 4096 4096 63 1145 0.0000 88 1577 0.0195 1.40x
8 16 4096 7168 104 1220 0.0000 141 1730 0.0189 1.36x
8 16 7168 2048 55 1178 0.0000 78 1574 0.0193 1.43x
8 16 4096 4096 64 1130 0.0000 88 1594 0.0192 1.36x
8 32 4096 7168 114 1126 0.0000 142 1738 0.0184 1.24x
8 32 7168 2048 60 1114 0.0000 78 1603 0.0191 1.31x
8 32 4096 4096 71 1046 0.0000 88 1615 0.0185 1.23x
8 64 4096 7168 198 669 0.0000 142 1765 0.0189 0.71x
8 64 7168 2048 115 614 0.0000 78 1653 0.0190 0.68x
8 64 4096 4096 123 633 0.0000 88 1644 0.0189 0.72x
16 1 4096 7168 184 1357 0.0000 271 1791 0.0202 1.47x
16 1 7168 2048 94 1332 0.0000 144 1681 0.0194 1.54x
16 1 4096 4096 112 1271 0.0000 165 1683 0.0192 1.46x
16 4 4096 7168 184 1363 0.0000 269 1802 0.0193 1.47x
16 4 7168 2048 95 1326 0.0000 145 1681 0.0192 1.53x
16 4 4096 4096 113 1265 0.0000 166 1672 0.0191 1.46x
16 8 4096 7168 184 1368 0.0000 271 1797 0.0191 1.47x
16 8 7168 2048 93 1371 0.0000 144 1691 0.0191 1.56x
16 8 4096 4096 112 1283 0.0000 165 1686 0.0189 1.47x
16 16 4096 7168 196 1296 0.0000 270 1808 0.0187 1.38x
16 16 7168 2048 97 1325 0.0000 144 1707 0.0193 1.48x
16 16 4096 4096 115 1266 0.0000 165 1700 0.0194 1.43x
16 32 4096 7168 215 1197 0.0000 270 1827 0.0191 1.25x
16 32 7168 2048 107 1248 0.0000 144 1738 0.0198 1.35x
16 32 4096 4096 129 1158 0.0000 165 1713 0.0191 1.29x
16 64 4096 7168 379 700 0.0000 270 1853 0.0190 0.71x
16 64 7168 2048 213 664 0.0000 146 1779 0.0189 0.68x
16 64 4096 4096 231 672 0.0000 165 1755 0.0187 0.71x
32 1 4096 7168 318 1571 0.0000 506 1916 0.0202 1.59x
32 1 7168 2048 175 1432 0.0000 277 1751 0.0202 1.59x
32 1 4096 4096 190 1502 0.0000 302 1834 0.0198 1.59x
32 4 4096 7168 320 1567 0.0000 507 1915 0.0198 1.59x
32 4 7168 2048 176 1430 0.0000 278 1750 0.0192 1.58x
32 4 4096 4096 191 1498 0.0000 301 1842 0.0195 1.57x
32 8 4096 7168 319 1575 0.0000 508 1916 0.0191 1.59x
32 8 7168 2048 174 1461 0.0000 278 1760 0.0197 1.60x
32 8 4096 4096 190 1516 0.0000 302 1845 0.0196 1.59x
32 16 4096 7168 339 1496 0.0000 506 1932 0.0190 1.49x
32 16 7168 2048 182 1417 0.0000 277 1777 0.0195 1.52x
32 16 4096 4096 195 1493 0.0000 302 1856 0.0190 1.55x
32 32 4096 7168 374 1378 0.0000 508 1940 0.0192 1.36x
32 32 7168 2048 202 1319 0.0000 278 1806 0.0195 1.37x
32 32 4096 4096 220 1357 0.0000 302 1877 0.0193 1.37x
32 64 4096 7168 726 731 0.0000 507 1975 0.0189 0.70x
32 64 7168 2048 416 681 0.0000 278 1864 0.0194 0.67x
32 64 4096 4096 436 713 0.0000 301 1921 0.0192 0.69x

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/378 **Author:** [@shiyang814-cpu](https://github.com/shiyang814-cpu) **Created:** 7/14/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `sm90-fp8-fp4-g128-cleanup` --- ### 📝 Commits (10+) - [`6e12aeb`](https://github.com/deepseek-ai/DeepGEMM/commit/6e12aebefb6b9c120945e7f3b3464a807c541856) Create test_sm90_fp8_fp4.py - [`b118d88`](https://github.com/deepseek-ai/DeepGEMM/commit/b118d880cc3bff1e2dc41965c86a36e51d872e61) Create sm90_fp8_fp4_gemm_1d1d.cuh - [`561cf21`](https://github.com/deepseek-ai/DeepGEMM/commit/561cf215ea3464bfec014b49a0ffbdd47eff0b3b) Create sm90_fp8_fp4_gemm_1d2d.cuh - [`4122a7d`](https://github.com/deepseek-ai/DeepGEMM/commit/4122a7d7bd6687be8538871e74c9d562de6ce2d7) Update gemm.hpp - [`b3d92e0`](https://github.com/deepseek-ai/DeepGEMM/commit/b3d92e05b599d12d3e564d3220873779e93c4c5b) Update __init__.py - [`825cbdb`](https://github.com/deepseek-ai/DeepGEMM/commit/825cbdbe2a954764d35eecd3bc0b358577949537) Update __init__.py - [`9625f4f`](https://github.com/deepseek-ai/DeepGEMM/commit/9625f4f8e2dc5470bdc6fe150ff8086418d2af98) Update test_sm90_fp8_fp4.py - [`6a3b1ee`](https://github.com/deepseek-ai/DeepGEMM/commit/6a3b1eea52826591dc05817a3f05efa42d446ff2) Merge pull request #1 from zhangxiaolei123456/zhangxiaolei123456-patch-1 - [`043f007`](https://github.com/deepseek-ai/DeepGEMM/commit/043f007110d13b2154bfbbca053d7388e8733e4b) Create sm90_fp8_fp4_gemm_1d2d.hpp - [`eebc4f4`](https://github.com/deepseek-ai/DeepGEMM/commit/eebc4f473a8b87728fb8f452477b5647efeb261f) Update sm90_fp8_fp4_gemm_1d2d.hpp ### 📊 Changes **16 files changed** (+8670 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `csrc/apis/gemm.hpp` (+106 -0) 📝 `csrc/apis/layout.hpp` (+17 -2) ➕ `csrc/jit_kernels/impls/sm90_fp8_fp4_gemm_1d2d.hpp` (+1551 -0) ➕ `csrc/jit_kernels/impls/sm90_fp8_fp4_gemm_1d2d_rs.hpp` (+359 -0) 📝 `csrc/jit_kernels/impls/smxx_layout.hpp` (+19 -3) 📝 `csrc/utils/layout.hpp` (+14 -4) 📝 `deep_gemm/__init__.py` (+19 -0) ➕ `deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d1d.cuh` (+396 -0) ➕ `deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d2d.cuh` (+713 -0) ➕ `deep_gemm/include/deep_gemm/impls/sm90_fp8_fp4_gemm_1d2d_rs.cuh` (+3236 -0) 📝 `deep_gemm/include/deep_gemm/mma/sm90.cuh` (+70 -0) 📝 `deep_gemm/include/deep_gemm/ptx/ld_st.cuh` (+36 -0) 📝 `deep_gemm/utils/math.py` (+82 -1) ➕ `tests/test_sm90_fp8_fp4.py` (+921 -0) ➕ `tests/test_sm90_fp8_fp4_g128.py` (+607 -0) ➕ `tests/test_sm90_int4_a8.py` (+524 -0) </details> ### 📄 Description This PR adds support and validation for the SM90 FP8 × FP4 GEMM path with FP4 scales stored in FP32 and grouped by group_size=128 (g128). It differs from the previous feat: support sm90_fp8_fp4 kernel work which primarily focused on the existing FP4 scaling/fast-path configurations . In addition, we simplified related masked-kernel code by removing redundant runtime switches for the masked operator, making the implementation easier to read and maintain. ```bash python /sgl-workspace/sglang/test_sm90_fp8_fp4_g128.py --repo-root /sgl-workspace/DeepGEMM ``` ## Contiguous FP8xFP4(g128) benchmark groups | m/group | n | k | W4 us | W4 GB/s | W4 diff | FP8 us | FP8 GB/s | FP8 diff | Speedup -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 8 | 1 | 4096 | 7168 | 327 | 431 | 0.0000 | 240 | 1074 | 0.0184 | 0.74x 8 | 1 | 7168 | 2048 | 178 | 444 | 0.0000 | 134 | 1030 | 0.0185 | 0.75x 8 | 1 | 4096 | 4096 | 195 | 430 | 0.0000 | 145 | 1045 | 0.0185 | 0.74x 8 | 4 | 4096 | 7168 | 326 | 431 | 0.0000 | 240 | 1076 | 0.0180 | 0.74x 8 | 4 | 7168 | 2048 | 179 | 444 | 0.0000 | 132 | 1042 | 0.0182 | 0.74x 8 | 4 | 4096 | 4096 | 197 | 427 | 0.0000 | 145 | 1039 | 0.0183 | 0.74x 8 | 8 | 4096 | 7168 | 326 | 432 | 0.0000 | 240 | 1075 | 0.0182 | 0.74x 8 | 8 | 7168 | 2048 | 178 | 444 | 0.0000 | 132 | 1044 | 0.0182 | 0.74x 8 | 8 | 4096 | 4096 | 196 | 428 | 0.0000 | 146 | 1036 | 0.0185 | 0.74x 8 | 16 | 4096 | 7168 | 327 | 430 | 0.0000 | 241 | 1073 | 0.0183 | 0.74x 8 | 16 | 7168 | 2048 | 179 | 443 | 0.0000 | 132 | 1043 | 0.0181 | 0.74x 8 | 16 | 4096 | 4096 | 196 | 429 | 0.0000 | 145 | 1044 | 0.0186 | 0.74x 8 | 32 | 4096 | 7168 | 326 | 432 | 0.0000 | 241 | 1073 | 0.0184 | 0.74x 8 | 32 | 7168 | 2048 | 178 | 444 | 0.0000 | 134 | 1033 | 0.0184 | 0.75x 8 | 32 | 4096 | 4096 | 196 | 429 | 0.0000 | 145 | 1042 | 0.0184 | 0.74x 8 | 64 | 4096 | 7168 | 327 | 431 | 0.0000 | 240 | 1076 | 0.0187 | 0.73x 8 | 64 | 7168 | 2048 | 179 | 442 | 0.0000 | 133 | 1037 | 0.0184 | 0.74x 8 | 64 | 4096 | 4096 | 197 | 427 | 0.0000 | 146 | 1039 | 0.0186 | 0.74x 16 | 1 | 4096 | 7168 | 637 | 442 | 0.0000 | 513 | 1006 | 0.0184 | 0.81x 16 | 1 | 7168 | 2048 | 334 | 474 | 0.0000 | 252 | 1095 | 0.0182 | 0.75x 16 | 1 | 4096 | 4096 | 380 | 442 | 0.0000 | 303 | 996 | 0.0183 | 0.80x 16 | 4 | 4096 | 7168 | 638 | 441 | 0.0000 | 514 | 1005 | 0.0182 | 0.81x 16 | 4 | 7168 | 2048 | 335 | 474 | 0.0000 | 252 | 1095 | 0.0185 | 0.75x 16 | 4 | 4096 | 4096 | 380 | 442 | 0.0000 | 303 | 996 | 0.0183 | 0.80x 16 | 8 | 4096 | 7168 | 638 | 441 | 0.0000 | 514 | 1005 | 0.0184 | 0.81x 16 | 8 | 7168 | 2048 | 334 | 475 | 0.0000 | 253 | 1090 | 0.0184 | 0.76x 16 | 8 | 4096 | 4096 | 380 | 442 | 0.0000 | 304 | 994 | 0.0182 | 0.80x 16 | 16 | 4096 | 7168 | 639 | 440 | 0.0000 | 514 | 1005 | 0.0183 | 0.80x 16 | 16 | 7168 | 2048 | 334 | 475 | 0.0000 | 252 | 1095 | 0.0184 | 0.76x 16 | 16 | 4096 | 4096 | 380 | 442 | 0.0000 | 303 | 996 | 0.0183 | 0.80x 16 | 32 | 4096 | 7168 | 639 | 441 | 0.0000 | 514 | 1005 | 0.0182 | 0.80x 16 | 32 | 7168 | 2048 | 335 | 473 | 0.0000 | 253 | 1091 | 0.0185 | 0.76x 16 | 32 | 4096 | 4096 | 381 | 442 | 0.0000 | 304 | 996 | 0.0181 | 0.80x 16 | 64 | 4096 | 7168 | 639 | 441 | 0.0000 | 515 | 1004 | 0.0182 | 0.81x 16 | 64 | 7168 | 2048 | 334 | 475 | 0.0000 | 252 | 1094 | 0.0182 | 0.76x 16 | 64 | 4096 | 4096 | 381 | 441 | 0.0000 | 304 | 995 | 0.0183 | 0.80x 32 | 1 | 4096 | 7168 | 1222 | 461 | 0.0000 | 937 | 1102 | 0.0182 | 0.77x 32 | 1 | 7168 | 2048 | 656 | 483 | 0.0000 | 491 | 1124 | 0.0181 | 0.75x 32 | 1 | 4096 | 4096 | 722 | 465 | 0.0000 | 548 | 1104 | 0.0184 | 0.76x 32 | 4 | 4096 | 7168 | 1221 | 461 | 0.0000 | 935 | 1104 | 0.0183 | 0.77x 32 | 4 | 7168 | 2048 | 656 | 483 | 0.0000 | 491 | 1124 | 0.0185 | 0.75x 32 | 4 | 4096 | 4096 | 722 | 465 | 0.0000 | 549 | 1102 | 0.0183 | 0.76x 32 | 8 | 4096 | 7168 | 1222 | 461 | 0.0000 | 937 | 1102 | 0.0183 | 0.77x 32 | 8 | 7168 | 2048 | 657 | 483 | 0.0000 | 491 | 1124 | 0.0183 | 0.75x 32 | 8 | 4096 | 4096 | 722 | 465 | 0.0000 | 549 | 1102 | 0.0183 | 0.76x 32 | 16 | 4096 | 7168 | 1222 | 461 | 0.0000 | 937 | 1102 | 0.0183 | 0.77x 32 | 16 | 7168 | 2048 | 657 | 483 | 0.0000 | 491 | 1124 | 0.0182 | 0.75x 32 | 16 | 4096 | 4096 | 722 | 465 | 0.0000 | 548 | 1104 | 0.0184 | 0.76x 32 | 32 | 4096 | 7168 | 1222 | 461 | 0.0000 | 937 | 1103 | 0.0184 | 0.77x 32 | 32 | 7168 | 2048 | 657 | 482 | 0.0000 | 491 | 1123 | 0.0184 | 0.75x 32 | 32 | 4096 | 4096 | 723 | 465 | 0.0000 | 548 | 1103 | 0.0184 | 0.76x 32 | 64 | 4096 | 7168 | 1224 | 460 | 0.0000 | 938 | 1101 | 0.0183 | 0.77x 32 | 64 | 7168 | 2048 | 658 | 482 | 0.0000 | 491 | 1123 | 0.0182 | 0.75x 32 | 64 | 4096 | 4096 | 726 | 463 | 0.0000 | 550 | 1100 | 0.0185 | 0.76x ## Masked FP8xFP4(g128) benchmark groups | m/group | n | k | W4 us | W4 GB/s | W4 diff | FP8 us | FP8 GB/s | FP8 diff | Speedup -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 8 | 1 | 4096 | 7168 | 99 | 1264 | 0.0000 | 142 | 1706 | 0.0196 | 1.44x 8 | 1 | 7168 | 2048 | 53 | 1180 | 0.0000 | 78 | 1552 | 0.0199 | 1.47x 8 | 1 | 4096 | 4096 | 62 | 1145 | 0.0000 | 88 | 1574 | 0.0189 | 1.41x 8 | 4 | 4096 | 7168 | 100 | 1256 | 0.0000 | 141 | 1717 | 0.0195 | 1.42x 8 | 4 | 7168 | 2048 | 53 | 1184 | 0.0000 | 79 | 1547 | 0.0189 | 1.48x 8 | 4 | 4096 | 4096 | 63 | 1137 | 0.0000 | 88 | 1575 | 0.0197 | 1.40x 8 | 8 | 4096 | 7168 | 99 | 1277 | 0.0000 | 141 | 1722 | 0.0190 | 1.43x 8 | 8 | 7168 | 2048 | 53 | 1204 | 0.0000 | 78 | 1562 | 0.0188 | 1.48x 8 | 8 | 4096 | 4096 | 63 | 1145 | 0.0000 | 88 | 1577 | 0.0195 | 1.40x 8 | 16 | 4096 | 7168 | 104 | 1220 | 0.0000 | 141 | 1730 | 0.0189 | 1.36x 8 | 16 | 7168 | 2048 | 55 | 1178 | 0.0000 | 78 | 1574 | 0.0193 | 1.43x 8 | 16 | 4096 | 4096 | 64 | 1130 | 0.0000 | 88 | 1594 | 0.0192 | 1.36x 8 | 32 | 4096 | 7168 | 114 | 1126 | 0.0000 | 142 | 1738 | 0.0184 | 1.24x 8 | 32 | 7168 | 2048 | 60 | 1114 | 0.0000 | 78 | 1603 | 0.0191 | 1.31x 8 | 32 | 4096 | 4096 | 71 | 1046 | 0.0000 | 88 | 1615 | 0.0185 | 1.23x 8 | 64 | 4096 | 7168 | 198 | 669 | 0.0000 | 142 | 1765 | 0.0189 | 0.71x 8 | 64 | 7168 | 2048 | 115 | 614 | 0.0000 | 78 | 1653 | 0.0190 | 0.68x 8 | 64 | 4096 | 4096 | 123 | 633 | 0.0000 | 88 | 1644 | 0.0189 | 0.72x 16 | 1 | 4096 | 7168 | 184 | 1357 | 0.0000 | 271 | 1791 | 0.0202 | 1.47x 16 | 1 | 7168 | 2048 | 94 | 1332 | 0.0000 | 144 | 1681 | 0.0194 | 1.54x 16 | 1 | 4096 | 4096 | 112 | 1271 | 0.0000 | 165 | 1683 | 0.0192 | 1.46x 16 | 4 | 4096 | 7168 | 184 | 1363 | 0.0000 | 269 | 1802 | 0.0193 | 1.47x 16 | 4 | 7168 | 2048 | 95 | 1326 | 0.0000 | 145 | 1681 | 0.0192 | 1.53x 16 | 4 | 4096 | 4096 | 113 | 1265 | 0.0000 | 166 | 1672 | 0.0191 | 1.46x 16 | 8 | 4096 | 7168 | 184 | 1368 | 0.0000 | 271 | 1797 | 0.0191 | 1.47x 16 | 8 | 7168 | 2048 | 93 | 1371 | 0.0000 | 144 | 1691 | 0.0191 | 1.56x 16 | 8 | 4096 | 4096 | 112 | 1283 | 0.0000 | 165 | 1686 | 0.0189 | 1.47x 16 | 16 | 4096 | 7168 | 196 | 1296 | 0.0000 | 270 | 1808 | 0.0187 | 1.38x 16 | 16 | 7168 | 2048 | 97 | 1325 | 0.0000 | 144 | 1707 | 0.0193 | 1.48x 16 | 16 | 4096 | 4096 | 115 | 1266 | 0.0000 | 165 | 1700 | 0.0194 | 1.43x 16 | 32 | 4096 | 7168 | 215 | 1197 | 0.0000 | 270 | 1827 | 0.0191 | 1.25x 16 | 32 | 7168 | 2048 | 107 | 1248 | 0.0000 | 144 | 1738 | 0.0198 | 1.35x 16 | 32 | 4096 | 4096 | 129 | 1158 | 0.0000 | 165 | 1713 | 0.0191 | 1.29x 16 | 64 | 4096 | 7168 | 379 | 700 | 0.0000 | 270 | 1853 | 0.0190 | 0.71x 16 | 64 | 7168 | 2048 | 213 | 664 | 0.0000 | 146 | 1779 | 0.0189 | 0.68x 16 | 64 | 4096 | 4096 | 231 | 672 | 0.0000 | 165 | 1755 | 0.0187 | 0.71x 32 | 1 | 4096 | 7168 | 318 | 1571 | 0.0000 | 506 | 1916 | 0.0202 | 1.59x 32 | 1 | 7168 | 2048 | 175 | 1432 | 0.0000 | 277 | 1751 | 0.0202 | 1.59x 32 | 1 | 4096 | 4096 | 190 | 1502 | 0.0000 | 302 | 1834 | 0.0198 | 1.59x 32 | 4 | 4096 | 7168 | 320 | 1567 | 0.0000 | 507 | 1915 | 0.0198 | 1.59x 32 | 4 | 7168 | 2048 | 176 | 1430 | 0.0000 | 278 | 1750 | 0.0192 | 1.58x 32 | 4 | 4096 | 4096 | 191 | 1498 | 0.0000 | 301 | 1842 | 0.0195 | 1.57x 32 | 8 | 4096 | 7168 | 319 | 1575 | 0.0000 | 508 | 1916 | 0.0191 | 1.59x 32 | 8 | 7168 | 2048 | 174 | 1461 | 0.0000 | 278 | 1760 | 0.0197 | 1.60x 32 | 8 | 4096 | 4096 | 190 | 1516 | 0.0000 | 302 | 1845 | 0.0196 | 1.59x 32 | 16 | 4096 | 7168 | 339 | 1496 | 0.0000 | 506 | 1932 | 0.0190 | 1.49x 32 | 16 | 7168 | 2048 | 182 | 1417 | 0.0000 | 277 | 1777 | 0.0195 | 1.52x 32 | 16 | 4096 | 4096 | 195 | 1493 | 0.0000 | 302 | 1856 | 0.0190 | 1.55x 32 | 32 | 4096 | 7168 | 374 | 1378 | 0.0000 | 508 | 1940 | 0.0192 | 1.36x 32 | 32 | 7168 | 2048 | 202 | 1319 | 0.0000 | 278 | 1806 | 0.0195 | 1.37x 32 | 32 | 4096 | 4096 | 220 | 1357 | 0.0000 | 302 | 1877 | 0.0193 | 1.37x 32 | 64 | 4096 | 7168 | 726 | 731 | 0.0000 | 507 | 1975 | 0.0189 | 0.70x 32 | 64 | 7168 | 2048 | 416 | 681 | 0.0000 | 278 | 1864 | 0.0194 | 0.67x 32 | 64 | 4096 | 4096 | 436 | 713 | 0.0000 | 301 | 1921 | 0.0192 | 0.69x --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-26 21:17:17 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#402
No description provided.