[PR #116] [CLOSED] support group_gemm_offset, group_gemm_offset_swapAB #251

New issue

Closed

opened 2026-10-26 21:15:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:15:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/116
Author: @Wangzheee
Created: 6/19/2025
Status: ❌ Closed

Base: main ← Head: support_offset_and_offsetswapAB

📝 Commits (5)

d29b20c support group_gemm_offset, group_gemm_offset_swapAB
26a603f fix some bug
ccd63bb fix tma_d_offset_desc_swapAB, update unitest
7db1b0e update unitest
e29e996 update unitest

📊 Changes

10 files changed (+1593 additions, -53 deletions)

View changed files

📝 deep_gemm/__init__.py (+1 -0)
📝 deep_gemm/include/deep_gemm/fp8_gemm.cuh (+804 -1)
📝 deep_gemm/include/deep_gemm/mma_utils.cuh (+24 -0)
📝 deep_gemm/include/deep_gemm/scheduler.cuh (+262 -1)
📝 deep_gemm/jit_kernels/__init__.py (+2 -1)
📝 deep_gemm/jit_kernels/gemm.py (+90 -46)
📝 deep_gemm/jit_kernels/m_grouped_gemm.py (+150 -3)
📝 deep_gemm/jit_kernels/runtime.py (+154 -1)
📝 deep_gemm/jit_kernels/utils.py (+3 -0)
📝 tests/test_core.py (+103 -0)

📄 Description

support group gemm offset type: group_gemm_offset, and group_gemm_offset_swapAB

Performance:
- Remove random: group_ms = [int(expected_m_per_group * random.uniform(1, 1)) for _ in range(num_groups)]:
- m<64 about 30%~50% kernel speedup
  - Testing grouped masked GEMM:
    Perf (num_groups=2, expected_m_per_group= 16, n=4096, k=7168): 36 us | throughput: 53 TFLOPS, 1665 GB/s
    Perf (num_groups=4, expected_m_per_group= 16, n=4096, k=7168): 65 us | throughput: 58 TFLOPS, 1813 GB/s
    Perf (num_groups=2, expected_m_per_group= 32, n=4096, k=7168): 35 us | throughput: 106 TFLOPS, 1685 GB/s
    Perf (num_groups=9, expected_m_per_group= 32, n=4096, k=7168): 141 us | throughput: 120 TFLOPS, 1900 GB/s
    Perf (num_groups=2, expected_m_per_group= 32, n=4096, k=7168): 35 us | throughput: 106 TFLOPS, 1689 GB/s
    Perf (num_groups=4, expected_m_per_group= 32, n=4096, k=7168): 66 us | throughput: 115 TFLOPS, 1822 GB/s
    Perf (num_groups=32, expected_m_per_group= 64, n=4096, k=7168): 485 us | throughput: 248 TFLOPS, 2002 GB/s
  - Testing grouped offset GEMM:
    Perf (num_groups= 2, expected_m_per_group= 16, n=4096, k=7168): 27 us | throughput: 71 TFLOPS, 2226 GB/s
    Perf (num_groups= 4, expected_m_per_group= 16, n=4096, k=7168): 46 us | throughput: 82 TFLOPS, 2587 GB/s
    Perf (num_groups= 2, expected_m_per_group= 32, n=4096, k=7168): 28 us | throughput: 134 TFLOPS, 2136 GB/s
    Perf (num_groups= 9, expected_m_per_group= 32, n=4096, k=7168): 93 us | throughput: 183 TFLOPS, 2902 GB/s
    Perf (num_groups= 2, expected_m_per_group= 32, n=4096, k=7168): 28 us | throughput: 135 TFLOPS, 2143 GB/s
    Perf (num_groups= 4, expected_m_per_group= 32, n=4096, k=7168): 49 us | throughput: 152 TFLOPS, 2414 GB/s
    Perf (num_groups=32, expected_m_per_group= 64, n=4096, k=7168): 479 us | throughput: 251 TFLOPS, 2029 GB/s

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/116 **Author:** [@Wangzheee](https://github.com/Wangzheee) **Created:** 6/19/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `support_offset_and_offsetswapAB` --- ### 📝 Commits (5) - [`d29b20c`](https://github.com/deepseek-ai/DeepGEMM/commit/d29b20cd1664e769811f67d013b00ecf6f99519e) support group_gemm_offset, group_gemm_offset_swapAB - [`26a603f`](https://github.com/deepseek-ai/DeepGEMM/commit/26a603f51815cb6dd13bdba3ddd5d59c949843c2) fix some bug - [`ccd63bb`](https://github.com/deepseek-ai/DeepGEMM/commit/ccd63bb234b63658c79c4c04d1c87bbd5213727f) fix tma_d_offset_desc_swapAB, update unitest - [`7db1b0e`](https://github.com/deepseek-ai/DeepGEMM/commit/7db1b0ef63975773d353885e58c493044ad66f46) update unitest - [`e29e996`](https://github.com/deepseek-ai/DeepGEMM/commit/e29e996a42626810c5af15ad6be36c0dd2d48cdb) update unitest ### 📊 Changes **10 files changed** (+1593 additions, -53 deletions) <details> <summary>View changed files</summary> 📝 `deep_gemm/__init__.py` (+1 -0) 📝 `deep_gemm/include/deep_gemm/fp8_gemm.cuh` (+804 -1) 📝 `deep_gemm/include/deep_gemm/mma_utils.cuh` (+24 -0) 📝 `deep_gemm/include/deep_gemm/scheduler.cuh` (+262 -1) 📝 `deep_gemm/jit_kernels/__init__.py` (+2 -1) 📝 `deep_gemm/jit_kernels/gemm.py` (+90 -46) 📝 `deep_gemm/jit_kernels/m_grouped_gemm.py` (+150 -3) 📝 `deep_gemm/jit_kernels/runtime.py` (+154 -1) 📝 `deep_gemm/jit_kernels/utils.py` (+3 -0) 📝 `tests/test_core.py` (+103 -0) </details> ### 📄 Description support group gemm offset type: group_gemm_offset, and group_gemm_offset_swapAB - **Performance:** - Remove random: group_ms = [int(expected_m_per_group * random.uniform(1, 1)) for _ in range(num_groups)]: - **m<64 about 30%~50% kernel speedup** - Testing grouped masked GEMM: Perf (num_groups=2, expected_m_per_group= 16, n=4096, k=7168): 36 us | throughput: 53 TFLOPS, 1665 GB/s Perf (num_groups=4, expected_m_per_group= 16, n=4096, k=7168): 65 us | throughput: 58 TFLOPS, 1813 GB/s Perf (num_groups=2, expected_m_per_group= 32, n=4096, k=7168): 35 us | throughput: 106 TFLOPS, 1685 GB/s Perf (num_groups=9, expected_m_per_group= 32, n=4096, k=7168): 141 us | throughput: 120 TFLOPS, 1900 GB/s Perf (num_groups=2, expected_m_per_group= 32, n=4096, k=7168): 35 us | throughput: 106 TFLOPS, 1689 GB/s Perf (num_groups=4, expected_m_per_group= 32, n=4096, k=7168): 66 us | throughput: 115 TFLOPS, 1822 GB/s Perf (num_groups=32, expected_m_per_group= 64, n=4096, k=7168): 485 us | throughput: 248 TFLOPS, 2002 GB/s - Testing grouped offset GEMM: Perf (num_groups= 2, expected_m_per_group= 16, n=4096, k=7168): 27 us | throughput: 71 TFLOPS, 2226 GB/s Perf (num_groups= 4, expected_m_per_group= 16, n=4096, k=7168): 46 us | throughput: 82 TFLOPS, 2587 GB/s Perf (num_groups= 2, expected_m_per_group= 32, n=4096, k=7168): 28 us | throughput: 134 TFLOPS, 2136 GB/s Perf (num_groups= 9, expected_m_per_group= 32, n=4096, k=7168): 93 us | throughput: 183 TFLOPS, 2902 GB/s Perf (num_groups= 2, expected_m_per_group= 32, n=4096, k=7168): 28 us | throughput: 135 TFLOPS, 2143 GB/s Perf (num_groups= 4, expected_m_per_group= 32, n=4096, k=7168): 49 us | throughput: 152 TFLOPS, 2414 GB/s Perf (num_groups=32, expected_m_per_group= 64, n=4096, k=7168): 479 us | throughput: 251 TFLOPS, 2029 GB/s --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:15:21 +00:00