deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[PR #168] [MERGED] Fix performance issue of m-grouped contiguous GEMMs. #273

New issue
Closed
opened 2026-10-26 21:15:38 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:15:38 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/168
Author: @coconut902
Created: 8/20/2025
Status: ✅ Merged
Merged: 8/22/2025
Merged by: @LyricZhao

Base: main ← Head: fix_grouped_gemm_performance_issue

📝 Commits (1)

  • 5b5d173 Fix grouped gemms performance issue.

📊 Changes

1 file changed (+7 additions, -12 deletions)

View changed files

📝 deep_gemm/include/deep_gemm/common/scheduler.cuh (+7 -12)

📄 Description

This PR fixed issue #160.

On my H800,
before fix:

Testing GEMM:
 > Perf (m=  128, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   50 us |   13 us |  302 TFLOPS | 1294 GB/s
 > Perf (m=  128, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch   57 us |   22 us |  443 TFLOPS | 2029 GB/s
 > Perf (m=  128, n=32768, k=  512, 1D2D, layout=NT, BF16, acc=0): launch   80 us |   13 us |  338 TFLOPS | 1983 GB/s
 > Perf (m=  128, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch   52 us |   55 us |  542 TFLOPS | 2190 GB/s
 > Perf (m=  128, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   51 us |   19 us |  399 TFLOPS | 1665 GB/s
 > Perf (m=  128, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch   63 us |   11 us |  351 TFLOPS | 1569 GB/s
 > Perf (m= 4096, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   56 us |   96 us | 1296 TFLOPS |  656 GB/s
 > Perf (m= 4096, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch   61 us |  245 us | 1263 TFLOPS | 1003 GB/s
 > Perf (m= 4096, n=32768, k=  512, 1D2D, layout=NT, BF16, acc=0): launch   69 us |  146 us |  938 TFLOPS | 1962 GB/s
 > Perf (m= 4096, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch   65 us |  648 us | 1484 TFLOPS |  379 GB/s
 > Perf (m= 4096, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   59 us |  169 us | 1424 TFLOPS |  552 GB/s
 > Perf (m= 4096, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch   55 us |   97 us | 1237 TFLOPS |  844 GB/s

Testing m-grouped contiguous GEMM:
 > Perf (num_groups=4, m=35456, n= 4096, k= 7168, 1D2D, layout=NT): 2061 us | 1010 TFLOPS |  325 GB/s
 > Perf (num_groups=4, m=36096, n= 7168, k= 2048, 1D2D, layout=NT): 1099 us |  964 TFLOPS |  594 GB/s
 > Perf (num_groups=8, m=32384, n= 4096, k= 7168, 1D2D, layout=NT): 1921 us |  990 TFLOPS |  385 GB/s
 > Perf (num_groups=8, m=31232, n= 7168, k= 2048, 1D2D, layout=NT):  920 us |  996 TFLOPS |  686 GB/s

Testing m-grouped masked GEMM:
 > Perf (num_groups=1, expected_m_per_group=1024, n=4096, k=7168, 1D2D):   80 us |  773 TFLOPS |  574 GB/s
 > Perf (num_groups=1, expected_m_per_group=1024, n=7168, k=2048, 1D2D):   45 us |  725 TFLOPS |  729 GB/s
 > Perf (num_groups=2, expected_m_per_group= 512, n=4096, k=7168, 1D2D):   92 us |  738 TFLOPS |  833 GB/s
 > Perf (num_groups=2, expected_m_per_group= 512, n=7168, k=2048, 1D2D):   44 us |  659 TFLOPS | 1040 GB/s
 > Perf (num_groups=4, expected_m_per_group= 256, n=4096, k=7168, 1D2D):   97 us |  669 TFLOPS | 1385 GB/s
 > Perf (num_groups=4, expected_m_per_group= 256, n=7168, k=2048, 1D2D):   48 us |  553 TFLOPS | 1531 GB/s

after fix:

Testing GEMM:
 > Perf (m=  128, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   51 us |   13 us |  303 TFLOPS | 1300 GB/s
 > Perf (m=  128, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch   48 us |   22 us |  443 TFLOPS | 2028 GB/s
 > Perf (m=  128, n=32768, k=  512, 1D2D, layout=NT, BF16, acc=0): launch   51 us |   13 us |  335 TFLOPS | 1971 GB/s
 > Perf (m=  128, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch   49 us |   56 us |  541 TFLOPS | 2185 GB/s
 > Perf (m=  128, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   51 us |   19 us |  404 TFLOPS | 1684 GB/s
 > Perf (m=  128, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch   57 us |   11 us |  353 TFLOPS | 1578 GB/s
 > Perf (m= 4096, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   51 us |   96 us | 1292 TFLOPS |  653 GB/s
 > Perf (m= 4096, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch   50 us |  245 us | 1261 TFLOPS | 1002 GB/s
 > Perf (m= 4096, n=32768, k=  512, 1D2D, layout=NT, BF16, acc=0): launch   59 us |  146 us |  941 TFLOPS | 1967 GB/s
 > Perf (m= 4096, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch   53 us |  649 us | 1482 TFLOPS |  378 GB/s
 > Perf (m= 4096, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch   49 us |  169 us | 1423 TFLOPS |  551 GB/s
 > Perf (m= 4096, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch   47 us |   97 us | 1237 TFLOPS |  844 GB/s

Testing m-grouped contiguous GEMM:
 > Perf (num_groups=4, m=35456, n= 4096, k= 7168, 1D2D, layout=NT): 1490 us | 1397 TFLOPS |  450 GB/s
 > Perf (num_groups=4, m=36096, n= 7168, k= 2048, 1D2D, layout=NT):  801 us | 1323 TFLOPS |  815 GB/s
 > Perf (num_groups=8, m=32384, n= 4096, k= 7168, 1D2D, layout=NT): 1370 us | 1388 TFLOPS |  540 GB/s
 > Perf (num_groups=8, m=31232, n= 7168, k= 2048, 1D2D, layout=NT):  699 us | 1312 TFLOPS |  903 GB/s

Testing m-grouped masked GEMM:
 > Perf (num_groups=1, expected_m_per_group=1024, n=4096, k=7168, 1D2D):   79 us |  775 TFLOPS |  576 GB/s
 > Perf (num_groups=1, expected_m_per_group=1024, n=7168, k=2048, 1D2D):   45 us |  726 TFLOPS |  731 GB/s
 > Perf (num_groups=2, expected_m_per_group= 512, n=4096, k=7168, 1D2D):   92 us |  740 TFLOPS |  835 GB/s
 > Perf (num_groups=2, expected_m_per_group= 512, n=7168, k=2048, 1D2D):   44 us |  659 TFLOPS | 1039 GB/s
 > Perf (num_groups=4, expected_m_per_group= 256, n=4096, k=7168, 1D2D):   95 us |  686 TFLOPS | 1421 GB/s
 > Perf (num_groups=4, expected_m_per_group= 256, n=7168, k=2048, 1D2D):   47 us |  562 TFLOPS | 1555 GB/s

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/168 **Author:** [@coconut902](https://github.com/coconut902) **Created:** 8/20/2025 **Status:** ✅ Merged **Merged:** 8/22/2025 **Merged by:** [@LyricZhao](https://github.com/LyricZhao) **Base:** `main` ← **Head:** `fix_grouped_gemm_performance_issue` --- ### 📝 Commits (1) - [`5b5d173`](https://github.com/deepseek-ai/DeepGEMM/commit/5b5d1731726fd2440e10f95d1ba347bdbef69159) Fix grouped gemms performance issue. ### 📊 Changes **1 file changed** (+7 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `deep_gemm/include/deep_gemm/common/scheduler.cuh` (+7 -12) </details> ### 📄 Description This PR fixed issue #160. On my H800, before fix: ``` Testing GEMM: > Perf (m= 128, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 50 us | 13 us | 302 TFLOPS | 1294 GB/s > Perf (m= 128, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch 57 us | 22 us | 443 TFLOPS | 2029 GB/s > Perf (m= 128, n=32768, k= 512, 1D2D, layout=NT, BF16, acc=0): launch 80 us | 13 us | 338 TFLOPS | 1983 GB/s > Perf (m= 128, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch 52 us | 55 us | 542 TFLOPS | 2190 GB/s > Perf (m= 128, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 51 us | 19 us | 399 TFLOPS | 1665 GB/s > Perf (m= 128, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch 63 us | 11 us | 351 TFLOPS | 1569 GB/s > Perf (m= 4096, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 56 us | 96 us | 1296 TFLOPS | 656 GB/s > Perf (m= 4096, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch 61 us | 245 us | 1263 TFLOPS | 1003 GB/s > Perf (m= 4096, n=32768, k= 512, 1D2D, layout=NT, BF16, acc=0): launch 69 us | 146 us | 938 TFLOPS | 1962 GB/s > Perf (m= 4096, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch 65 us | 648 us | 1484 TFLOPS | 379 GB/s > Perf (m= 4096, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 59 us | 169 us | 1424 TFLOPS | 552 GB/s > Perf (m= 4096, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch 55 us | 97 us | 1237 TFLOPS | 844 GB/s Testing m-grouped contiguous GEMM: > Perf (num_groups=4, m=35456, n= 4096, k= 7168, 1D2D, layout=NT): 2061 us | 1010 TFLOPS | 325 GB/s > Perf (num_groups=4, m=36096, n= 7168, k= 2048, 1D2D, layout=NT): 1099 us | 964 TFLOPS | 594 GB/s > Perf (num_groups=8, m=32384, n= 4096, k= 7168, 1D2D, layout=NT): 1921 us | 990 TFLOPS | 385 GB/s > Perf (num_groups=8, m=31232, n= 7168, k= 2048, 1D2D, layout=NT): 920 us | 996 TFLOPS | 686 GB/s Testing m-grouped masked GEMM: > Perf (num_groups=1, expected_m_per_group=1024, n=4096, k=7168, 1D2D): 80 us | 773 TFLOPS | 574 GB/s > Perf (num_groups=1, expected_m_per_group=1024, n=7168, k=2048, 1D2D): 45 us | 725 TFLOPS | 729 GB/s > Perf (num_groups=2, expected_m_per_group= 512, n=4096, k=7168, 1D2D): 92 us | 738 TFLOPS | 833 GB/s > Perf (num_groups=2, expected_m_per_group= 512, n=7168, k=2048, 1D2D): 44 us | 659 TFLOPS | 1040 GB/s > Perf (num_groups=4, expected_m_per_group= 256, n=4096, k=7168, 1D2D): 97 us | 669 TFLOPS | 1385 GB/s > Perf (num_groups=4, expected_m_per_group= 256, n=7168, k=2048, 1D2D): 48 us | 553 TFLOPS | 1531 GB/s ``` after fix: ``` Testing GEMM: > Perf (m= 128, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 51 us | 13 us | 303 TFLOPS | 1300 GB/s > Perf (m= 128, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch 48 us | 22 us | 443 TFLOPS | 2028 GB/s > Perf (m= 128, n=32768, k= 512, 1D2D, layout=NT, BF16, acc=0): launch 51 us | 13 us | 335 TFLOPS | 1971 GB/s > Perf (m= 128, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch 49 us | 56 us | 541 TFLOPS | 2185 GB/s > Perf (m= 128, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 51 us | 19 us | 404 TFLOPS | 1684 GB/s > Perf (m= 128, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch 57 us | 11 us | 353 TFLOPS | 1578 GB/s > Perf (m= 4096, n= 2112, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 51 us | 96 us | 1292 TFLOPS | 653 GB/s > Perf (m= 4096, n=24576, k= 1536, 1D2D, layout=NT, BF16, acc=0): launch 50 us | 245 us | 1261 TFLOPS | 1002 GB/s > Perf (m= 4096, n=32768, k= 512, 1D2D, layout=NT, BF16, acc=0): launch 59 us | 146 us | 941 TFLOPS | 1967 GB/s > Perf (m= 4096, n= 7168, k=16384, 1D2D, layout=NT, BF16, acc=0): launch 53 us | 649 us | 1482 TFLOPS | 378 GB/s > Perf (m= 4096, n= 4096, k= 7168, 1D2D, layout=NT, BF16, acc=0): launch 49 us | 169 us | 1423 TFLOPS | 551 GB/s > Perf (m= 4096, n= 7168, k= 2048, 1D2D, layout=NT, BF16, acc=0): launch 47 us | 97 us | 1237 TFLOPS | 844 GB/s Testing m-grouped contiguous GEMM: > Perf (num_groups=4, m=35456, n= 4096, k= 7168, 1D2D, layout=NT): 1490 us | 1397 TFLOPS | 450 GB/s > Perf (num_groups=4, m=36096, n= 7168, k= 2048, 1D2D, layout=NT): 801 us | 1323 TFLOPS | 815 GB/s > Perf (num_groups=8, m=32384, n= 4096, k= 7168, 1D2D, layout=NT): 1370 us | 1388 TFLOPS | 540 GB/s > Perf (num_groups=8, m=31232, n= 7168, k= 2048, 1D2D, layout=NT): 699 us | 1312 TFLOPS | 903 GB/s Testing m-grouped masked GEMM: > Perf (num_groups=1, expected_m_per_group=1024, n=4096, k=7168, 1D2D): 79 us | 775 TFLOPS | 576 GB/s > Perf (num_groups=1, expected_m_per_group=1024, n=7168, k=2048, 1D2D): 45 us | 726 TFLOPS | 731 GB/s > Perf (num_groups=2, expected_m_per_group= 512, n=4096, k=7168, 1D2D): 92 us | 740 TFLOPS | 835 GB/s > Perf (num_groups=2, expected_m_per_group= 512, n=7168, k=2048, 1D2D): 44 us | 659 TFLOPS | 1039 GB/s > Perf (num_groups=4, expected_m_per_group= 256, n=4096, k=7168, 1D2D): 95 us | 686 TFLOPS | 1421 GB/s > Perf (num_groups=4, expected_m_per_group= 256, n=7168, k=2048, 1D2D): 47 us | 562 TFLOPS | 1555 GB/s ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-26 21:15:38 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#273
No description provided.