deepseek-ai/DeepGEMM
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00
Code Issues 162 Projects Packages Wiki Activity

[PR #183] [Feat] Single Batch Overlap (SBO): Overlaping of Down GEMM with Combine Send #282

New issue
Open
opened 2026-10-26 21:15:44 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 21:15:44 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/183
Author: @Sulfur6
Created: 9/2/2025
Status: 🔄 Open

Base: main ← Head: sbo.v2.public

📝 Commits (10+)

  • 9ac3853 feat: add signal gemm api for SBO (Single Batch Overlap).
  • 79b5a45 feat: add launch config args for coorperative groups.
  • 6cf6ca1 feat: add signal gemm impl&runtime in jit kernels.
  • 1e2c135 feat: add signal gemm kernel.
  • a5d23b0 feat: add signal gemm import in deep_gemm package.
  • 48c741a feat: add test for signal gemm.
  • 9a2d3e5 feat: add signal threshold as a config arg for signal gemm.
  • 03f61d2 add comparation test.
  • e86f1f9 avoid using bench kineto
  • e7ebff1 test: modify generators to fit bs32 down gemm.

📊 Changes

8 files changed (+106 additions, -31 deletions)

View changed files

📝 csrc/apis/gemm.hpp (+19 -5)
📝 csrc/jit_kernels/heuristics/common.hpp (+9 -3)
📝 csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp (+17 -9)
📝 deep_gemm/include/deep_gemm/common/utils.cuh (+10 -0)
📝 deep_gemm/include/deep_gemm/impls/sm90_fp8_gemm_1d2d.cuh (+14 -2)
📝 deep_gemm/testing/numeric.py (+14 -0)
📝 tests/generators.py (+10 -5)
📝 tests/test_fp8.py (+13 -7)

📄 Description

1. Motivation

The optimization effect of Two-Batch Overlap (TBO) is suboptimal for the Decode phase on low-compute-power cards (i.e., H20). This is due to two main factors: First, on the Hopper architecture, the WGMMA block_m is 64. Consequently, when TBO is enabled with a small Decode batch size, the MLP GEMM suffers from redundant computations. A positive throughput gain is only observed at larger batch sizes (e.g., 64, 128). Second, at these larger batch sizes, low-compute-power cards like the H20 fail to meet the SLA guarantees for TPOT/ITL.

Therefore, it is necessary to find a solution that can improve Decode throughput even with small batch sizes. Single Batch Overlap (SBO) presents itself as a viable solution.

We implement SBO for DeepSeek v3/R1 by modifying DeepEP and DeepGEMM, including the overlap of Shared Expert and Dispatch Recv, as well as the overlap of Down GEMM with Combine Send.

The overlap of Down GEMM with Combine Send is implemented by modifying SGlang, DeepEP and DeepGEMM, with the detailed implementation available in the PRs below:

We also conducted integration and evaluation in SGLang: https://github.com/sgl-project/sglang/pull/9660.

Since the latest version of SGLang depends on the branch https://github.com/sgl-project/DeepGEMM/tree/sgl, you should not use this branch when starting SGLang. Instead, you should use the branch developed based on the sgl branch https://github.com/Sulfur6/DeepGEMM/tree/sbo.v2.sgl.

2. Overlap Design

SBO implements two overlap for the MoE layers of DeepSeek-V3/R1. One is to overlap the Shared Expert computation with the Dispatch Recv communication, and the other is to overlap the Down GEMM computation with the Combine Send communication.
image
The interaction between Down GEMM and Combine Send is structured as a producer-consumer model synchronized by signals. For each local expert, a signal unit is allocated for every block_m tokens. The Down GEMM computes the results for these block_m tokens and atomically increments the signaling unit after completing a portion of the work. The Combine Send polls this signaling unit. Once the value reaches a threshold, it sends the corresponding block_m tokens.
image

3. Modifications

  • Add the m_grouped_fp8_gemm_nt_signal Python interface to support overlapping Down GEMM with Combine Send.
  • Add the SM90FP8SignalGemm1D2DRuntime and sm90_m_grouped_fp8_gemm_signal_1d2d to support Signal Down GEMM in SM90.
  • The sm90_fp8_signal_gemm_1d2d_impl kernel uses atomicAdd to write signal after the corresponding block_m tokens are computed.

4. Evaluation

We integrated the modified DeepEP and DeepGEMM into SGLang for performance evaluation.

4.1. Experiment Setup

  • 5 nodes, with 8 × H20 GPUs per node. Each prefill node uses TP8, and the other 2 decode nodes use DP_Attn 16 + EP 16.
  • Input length 4096, output length 1536.

4.2. Performance Evaluation

  • bs 32, origin
    image
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    4.8
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2359.16
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15672509
Request throughput (req/s):              4.34
Input token throughput (tok/s):          17778.82
Output token throughput (tok/s):         6667.06
Total token throughput (tok/s):          24445.88
Concurrency:                             490.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   112892.31
Median E2E Latency (ms):                 113847.19
---------------Time to First Token----------------
Mean TTFT (ms):                          640.62
Median TTFT (ms):                        545.06
P99 TTFT (ms):                           1543.37
---------------Inter-Token Latency----------------
Mean ITL (ms):                           73.11
Median ITL (ms):                         71.81
P95 ITL (ms):                            86.02
P99 ITL (ms):                            155.32
Max ITL (ms):                            1543.26
==================================================

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    5.0
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2357.80
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15673361
Request throughput (req/s):              4.34
Input token throughput (tok/s):          17789.05
Output token throughput (tok/s):         6670.89
Total token throughput (tok/s):          24459.95
Concurrency:                             490.83
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   113015.97
Median E2E Latency (ms):                 113951.58
---------------Time to First Token----------------
Mean TTFT (ms):                          724.98
Median TTFT (ms):                        624.73
P99 TTFT (ms):                           1693.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           73.13
Median ITL (ms):                         71.84
P95 ITL (ms):                            86.57
P99 ITL (ms):                            155.21
Max ITL (ms):                            1081.95
==================================================
  • bs 32, sbo
    image
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    4.8
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2211.76
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15673456
Request throughput (req/s):              4.63
Input token throughput (tok/s):          18963.67
Output token throughput (tok/s):         7111.38
Total token throughput (tok/s):          26075.05
Concurrency:                             481.58
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   104017.64
Median E2E Latency (ms):                 105363.65
---------------Time to First Token----------------
Mean TTFT (ms):                          606.28
Median TTFT (ms):                        508.61
P99 TTFT (ms):                           1475.44
---------------Inter-Token Latency----------------
Mean ITL (ms):                           67.35
Median ITL (ms):                         66.10
P95 ITL (ms):                            81.58
P99 ITL (ms):                            141.96
Max ITL (ms):                            1422.74
==================================================

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    5.0
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2194.12
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15672577
Request throughput (req/s):              4.67
Input token throughput (tok/s):          19116.14
Output token throughput (tok/s):         7168.55
Total token throughput (tok/s):          26284.70
Concurrency:                             487.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   104545.42
Median E2E Latency (ms):                 105483.50
---------------Time to First Token----------------
Mean TTFT (ms):                          619.03
Median TTFT (ms):                        511.23
P99 TTFT (ms):                           1504.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           67.68
Median ITL (ms):                         66.44
P95 ITL (ms):                            82.13
P99 ITL (ms):                            142.48
Max ITL (ms):                            1024.85
==================================================

4.3. Accuracy Tests

  • bs 32, origin
#python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000
100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:20<00:00, 12.41it/s]
Accuracy: 0.951
Invalid: 0.000
Latency: 80.802 s
Output throughput: 1183.468 token/s
  • bs 32, sbo
#python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000
100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:17<00:00, 12.87it/s]
Accuracy: 0.950
Invalid: 0.000
Latency: 78.056 s
Output throughput: 1217.443 token/s

4.4. Repro Script

Please refer to https://github.com/sgl-project/sglang/pull/9660.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/183 **Author:** [@Sulfur6](https://github.com/Sulfur6) **Created:** 9/2/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `sbo.v2.public` --- ### 📝 Commits (10+) - [`9ac3853`](https://github.com/deepseek-ai/DeepGEMM/commit/9ac3853d27d3e9a11167a35e2a2dd2e67e547e90) feat: add signal gemm api for SBO (Single Batch Overlap). - [`79b5a45`](https://github.com/deepseek-ai/DeepGEMM/commit/79b5a453806b98c9a8ba8f52c2a2e279035bed7f) feat: add launch config args for coorperative groups. - [`6cf6ca1`](https://github.com/deepseek-ai/DeepGEMM/commit/6cf6ca16bf01932385c44da713831866c1560a34) feat: add signal gemm impl&runtime in jit kernels. - [`1e2c135`](https://github.com/deepseek-ai/DeepGEMM/commit/1e2c135cb15834f8a2090fdcdccb92d2144c2bfc) feat: add signal gemm kernel. - [`a5d23b0`](https://github.com/deepseek-ai/DeepGEMM/commit/a5d23b0800795a0a54c9df6785ca3c161cd69ae8) feat: add signal gemm import in deep_gemm package. - [`48c741a`](https://github.com/deepseek-ai/DeepGEMM/commit/48c741af9d7f2a67a6264dff6d546bf8fd8e9f20) feat: add test for signal gemm. - [`9a2d3e5`](https://github.com/deepseek-ai/DeepGEMM/commit/9a2d3e55b63c2b329ce6bcb1bf8a1c125959aa6e) feat: add signal threshold as a config arg for signal gemm. - [`03f61d2`](https://github.com/deepseek-ai/DeepGEMM/commit/03f61d2b826381d97a1fdc3feaa7ce3ede5fd507) add comparation test. - [`e86f1f9`](https://github.com/deepseek-ai/DeepGEMM/commit/e86f1f91a320c0b8a8618b5001c1dddba43ce5b2) avoid using bench kineto - [`e7ebff1`](https://github.com/deepseek-ai/DeepGEMM/commit/e7ebff177d60bb693d5ea97a4af5edf79bd4022f) test: modify generators to fit bs32 down gemm. ### 📊 Changes **8 files changed** (+106 additions, -31 deletions) <details> <summary>View changed files</summary> 📝 `csrc/apis/gemm.hpp` (+19 -5) 📝 `csrc/jit_kernels/heuristics/common.hpp` (+9 -3) 📝 `csrc/jit_kernels/impls/sm90_fp8_gemm_1d2d.hpp` (+17 -9) 📝 `deep_gemm/include/deep_gemm/common/utils.cuh` (+10 -0) 📝 `deep_gemm/include/deep_gemm/impls/sm90_fp8_gemm_1d2d.cuh` (+14 -2) 📝 `deep_gemm/testing/numeric.py` (+14 -0) 📝 `tests/generators.py` (+10 -5) 📝 `tests/test_fp8.py` (+13 -7) </details> ### 📄 Description # 1. Motivation The optimization effect of Two-Batch Overlap (TBO) is suboptimal for the Decode phase on low-compute-power cards (i.e., H20). This is due to two main factors: First, on the Hopper architecture, the WGMMA block_m is 64. Consequently, when TBO is enabled with a small Decode batch size, the MLP GEMM suffers from redundant computations. A positive throughput gain is only observed at larger batch sizes (e.g., 64, 128). Second, at these larger batch sizes, low-compute-power cards like the H20 fail to meet the SLA guarantees for TPOT/ITL. Therefore, it is necessary to find a solution that can improve Decode throughput even with small batch sizes. Single Batch Overlap (SBO) presents itself as a viable solution. We implement SBO for DeepSeek v3/R1 by modifying DeepEP and DeepGEMM, including the overlap of Shared Expert and Dispatch Recv, as well as the overlap of Down GEMM with Combine Send. The overlap of Down GEMM with Combine Send is implemented by modifying SGlang, DeepEP and DeepGEMM, with the detailed implementation available in the PRs below: * DeepEP: https://github.com/deepseek-ai/DeepEP/pull/390. * DeepGEMM: this PR. We also conducted integration and evaluation in SGLang: https://github.com/sgl-project/sglang/pull/9660. Since the latest version of SGLang depends on the branch https://github.com/sgl-project/DeepGEMM/tree/sgl, you should not use this branch when starting SGLang. Instead, you should use the branch developed based on the sgl branch https://github.com/Sulfur6/DeepGEMM/tree/sbo.v2.sgl. # 2. Overlap Design SBO implements two overlap for the MoE layers of DeepSeek-V3/R1. One is to overlap the Shared Expert computation with the Dispatch Recv communication, and the other is to overlap the Down GEMM computation with the Combine Send communication. <img width="2484" height="772" alt="image" src="https://github.com/user-attachments/assets/b4d1f131-a1b5-4bf6-b623-fbf7b6fe6497" /> The interaction between Down GEMM and Combine Send is structured as a producer-consumer model synchronized by signals. For each local expert, a signal unit is allocated for every block_m tokens. The Down GEMM computes the results for these block_m tokens and atomically increments the signaling unit after completing a portion of the work. The Combine Send polls this signaling unit. Once the value reaches a threshold, it sends the corresponding block_m tokens. <img width="358" height="350" alt="image" src="https://github.com/user-attachments/assets/514134a5-0225-4698-855b-5a43c0a08bc7" /> # 3. Modifications * Add the `m_grouped_fp8_gemm_nt_signal` Python interface to support overlapping Down GEMM with Combine Send. * Add the `SM90FP8SignalGemm1D2DRuntime` and `sm90_m_grouped_fp8_gemm_signal_1d2d` to support Signal Down GEMM in SM90. * The `sm90_fp8_signal_gemm_1d2d_impl` kernel uses `atomicAdd` to write signal after the corresponding block_m tokens are computed. # 4. Evaluation We integrated the modified DeepEP and DeepGEMM into SGLang for performance evaluation. ## 4.1. Experiment Setup * 5 nodes, with 8 × H20 GPUs per node. Each prefill node uses TP8, and the other 2 decode nodes use DP_Attn 16 + EP 16. * Input length 4096, output length 1536. ## 4.2. Performance Evaluation * bs 32, origin <img width="3228" height="820" alt="image" src="https://github.com/user-attachments/assets/ccdc88bf-d3d4-4567-bded-4fe93bf456f8" /> ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 4.8 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2359.16 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15672509 Request throughput (req/s): 4.34 Input token throughput (tok/s): 17778.82 Output token throughput (tok/s): 6667.06 Total token throughput (tok/s): 24445.88 Concurrency: 490.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 112892.31 Median E2E Latency (ms): 113847.19 ---------------Time to First Token---------------- Mean TTFT (ms): 640.62 Median TTFT (ms): 545.06 P99 TTFT (ms): 1543.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.11 Median ITL (ms): 71.81 P95 ITL (ms): 86.02 P99 ITL (ms): 155.32 Max ITL (ms): 1543.26 ================================================== ``` ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 5.0 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2357.80 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15673361 Request throughput (req/s): 4.34 Input token throughput (tok/s): 17789.05 Output token throughput (tok/s): 6670.89 Total token throughput (tok/s): 24459.95 Concurrency: 490.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 113015.97 Median E2E Latency (ms): 113951.58 ---------------Time to First Token---------------- Mean TTFT (ms): 724.98 Median TTFT (ms): 624.73 P99 TTFT (ms): 1693.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.13 Median ITL (ms): 71.84 P95 ITL (ms): 86.57 P99 ITL (ms): 155.21 Max ITL (ms): 1081.95 ================================================== ``` * bs 32, sbo <img width="2990" height="822" alt="image" src="https://github.com/user-attachments/assets/e22b559c-db1c-46c1-8d4f-bf53b8fb12c4" /> ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 4.8 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2211.76 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15673456 Request throughput (req/s): 4.63 Input token throughput (tok/s): 18963.67 Output token throughput (tok/s): 7111.38 Total token throughput (tok/s): 26075.05 Concurrency: 481.58 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 104017.64 Median E2E Latency (ms): 105363.65 ---------------Time to First Token---------------- Mean TTFT (ms): 606.28 Median TTFT (ms): 508.61 P99 TTFT (ms): 1475.44 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.35 Median ITL (ms): 66.10 P95 ITL (ms): 81.58 P99 ITL (ms): 141.96 Max ITL (ms): 1422.74 ================================================== ``` ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 5.0 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2194.12 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15672577 Request throughput (req/s): 4.67 Input token throughput (tok/s): 19116.14 Output token throughput (tok/s): 7168.55 Total token throughput (tok/s): 26284.70 Concurrency: 487.92 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 104545.42 Median E2E Latency (ms): 105483.50 ---------------Time to First Token---------------- Mean TTFT (ms): 619.03 Median TTFT (ms): 511.23 P99 TTFT (ms): 1504.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.68 Median ITL (ms): 66.44 P95 ITL (ms): 82.13 P99 ITL (ms): 142.48 Max ITL (ms): 1024.85 ================================================== ``` ## 4.3. Accuracy Tests * bs 32, origin ``` #python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000 100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:20<00:00, 12.41it/s] Accuracy: 0.951 Invalid: 0.000 Latency: 80.802 s Output throughput: 1183.468 token/s ``` * bs 32, sbo ``` #python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000 100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:17<00:00, 12.87it/s] Accuracy: 0.950 Invalid: 0.000 Latency: 78.056 s Output throughput: 1217.443 token/s ``` ## 4.4. Repro Script Please refer to https://github.com/sgl-project/sglang/pull/9660. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
public-sync-20261014
0930-release
nv_dev
fix-mega-moe-fence
public-release-2609
public-release-2607
public-release-2606
mega-update
public-release-260416
ko3n1g/ci/build-wheels
nv_dev_f8e8fb5
nv_dev_950d31a
nv_dev_67fc648
nv_dev_c491439
nv_dev_4ff3f54
v2.1.1.post3
v2.1.1.post2
v2.1.1.post1
v2.1.1
v2.1.0
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepGEMM#282
No description provided.