[PR #390] [CLOSED] [Feat] Single Batch Overlap (SBO): Overlaping of Down GEMM with Combine Send #546

New issue

Closed

opened 2026-10-27 21:03:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/390
Author: @Zqy11
Created: 9/2/2025
Status: ❌ Closed

Base: main ← Head: feat/overlap

📝 Commits (10+)

9a58e8c feat: Overlap down gemm and combine send
433a6a4 merge main
42ed271 optimize put finishing flag for overlap mode
389e94b optimize put finishing flag for overlap mode
b596396 Merge branch 'main' of https://github.com/Zqy11/DeepEP into feat/overlap
988bad8 maintain: delete redundant code
11c5c66 maintain: add lambda send_finish_flag
99378d5 maintain: merge main
6601898 Merge branch 'main' of github.com:Zqy11/DeepEP into feat/overlap
b6f500c maintain: add kNumMaxExperts

📊 Changes

7 files changed (+218 additions, -67 deletions)

View changed files

📝 csrc/deep_ep.cpp (+16 -5)
📝 csrc/deep_ep.hpp (+2 -0)
📝 csrc/kernels/api.cuh (+4 -3)
📝 csrc/kernels/internode_ll.cu (+135 -31)
📝 deep_ep/buffer.py (+16 -2)
📝 tests/test_low_latency.py (+41 -26)
📝 tests/utils.py (+4 -0)

📄 Description

1. Motivation

The optimization effect of Two-Batch Overlap (TBO) is suboptimal for the Decode phase on low-compute-power cards (i.e., H20). This is due to two main factors: First, on the Hopper architecture, the WGMMA block_m is 64. Consequently, when TBO is enabled with a small Decode batch size, the MLP GEMM suffers from redundant computations. A positive throughput gain is only observed at larger batch sizes (e.g., 64, 128). Second, at these larger batch sizes, low-compute-power cards like the H20 fail to meet the SLA guarantees for TPOT/ITL.

Therefore, it is necessary to find a solution that can improve Decode throughput even with small batch sizes. Single Batch Overlap (SBO) presents itself as a viable solution.

We implement SBO for DeepSeek v3/R1 by modifying DeepEP and DeepGEMM, including the overlap of Shared Expert and Dispatch Recv, as well as the overlap of Down GEMM with Combine Send.

The overlap of Down GEMM with Combine Send is implemented by modifying SGlang, DeepEP and DeepGEMM, with the detailed implementation available in the PRs below:

DeepEP: this PR.
DeepGEMM: https://github.com/sgl-project/DeepGEMM/pull/14.

We also conduct integration and evaluation in SGLang: https://github.com/sgl-project/sglang/pull/9660.

2. Overlap Design

SBO implements two overlap for the MoE layers of DeepSeek-V3/R1. One is to overlap the Shared Expert computation with the Dispatch Recv communication, and the other is to overlap the Down GEMM computation with the Combine Send communication.

The interaction between Down GEMM and Combine Send is structured as a producer-consumer model synchronized by signals. For each local expert, a signal unit is allocated for every block_m tokens. The Down GEMM computes the results for these block_m tokens and atomically increments the signaling unit after completing a portion of the work. The Combine Send polls this signaling unit. Once the value reaches a threshold, it sends the corresponding block_m tokens.

3. Modifications

In low_latency_dispatch, change the type of packed_recv_src_info from kInt32 to kInt64 to track the src_rank for each token.
The internode_ll::combine kernel uses fewer SMs when the overlap parameter is enabled and integrates logic for overlap mode, such as polling signals, sending tokens and puting finish flag.

4. Evaluation

We integrated the modified DeepEP and DeepGEMM into SGLang for performance evaluation.

4.1. Experiment Setup

5 nodes, with 8 × H20 GPUs per node. Each prefill node uses TP8, and the other 2 decode nodes use DP_Attn 16 + EP 16.
Input length 4096, output length 1536.

4.2. Performance Evaluation

bs 32, origin

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    4.8
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2359.16
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15672509
Request throughput (req/s):              4.34
Input token throughput (tok/s):          17778.82
Output token throughput (tok/s):         6667.06
Total token throughput (tok/s):          24445.88
Concurrency:                             490.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   112892.31
Median E2E Latency (ms):                 113847.19
---------------Time to First Token----------------
Mean TTFT (ms):                          640.62
Median TTFT (ms):                        545.06
P99 TTFT (ms):                           1543.37
---------------Inter-Token Latency----------------
Mean ITL (ms):                           73.11
Median ITL (ms):                         71.81
P95 ITL (ms):                            86.02
P99 ITL (ms):                            155.32
Max ITL (ms):                            1543.26
==================================================

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    5.0
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2357.80
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15673361
Request throughput (req/s):              4.34
Input token throughput (tok/s):          17789.05
Output token throughput (tok/s):         6670.89
Total token throughput (tok/s):          24459.95
Concurrency:                             490.83
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   113015.97
Median E2E Latency (ms):                 113951.58
---------------Time to First Token----------------
Mean TTFT (ms):                          724.98
Median TTFT (ms):                        624.73
P99 TTFT (ms):                           1693.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           73.13
Median ITL (ms):                         71.84
P95 ITL (ms):                            86.57
P99 ITL (ms):                            155.21
Max ITL (ms):                            1081.95
==================================================

bs 32, sbo

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    4.8
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2211.76
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15673456
Request throughput (req/s):              4.63
Input token throughput (tok/s):          18963.67
Output token throughput (tok/s):         7111.38
Total token throughput (tok/s):          26075.05
Concurrency:                             481.58
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   104017.64
Median E2E Latency (ms):                 105363.65
---------------Time to First Token----------------
Mean TTFT (ms):                          606.28
Median TTFT (ms):                        508.61
P99 TTFT (ms):                           1475.44
---------------Inter-Token Latency----------------
Mean ITL (ms):                           67.35
Median ITL (ms):                         66.10
P95 ITL (ms):                            81.58
P99 ITL (ms):                            141.96
Max ITL (ms):                            1422.74
==================================================

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    5.0
Max request concurrency:                 512
Successful requests:                     10240
Benchmark duration (s):                  2194.12
Total input tokens:                      41943040
Total generated tokens:                  15728640
Total generated tokens (retokenized):    15672577
Request throughput (req/s):              4.67
Input token throughput (tok/s):          19116.14
Output token throughput (tok/s):         7168.55
Total token throughput (tok/s):          26284.70
Concurrency:                             487.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   104545.42
Median E2E Latency (ms):                 105483.50
---------------Time to First Token----------------
Mean TTFT (ms):                          619.03
Median TTFT (ms):                        511.23
P99 TTFT (ms):                           1504.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           67.68
Median ITL (ms):                         66.44
P95 ITL (ms):                            82.13
P99 ITL (ms):                            142.48
Max ITL (ms):                            1024.85
==================================================

4.3. Accuracy Tests

bs 32, origin

#python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000
100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:20<00:00, 12.41it/s]
Accuracy: 0.951
Invalid: 0.000
Latency: 80.802 s
Output throughput: 1183.468 token/s

bs 32, sbo

#python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000
100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:17<00:00, 12.87it/s]
Accuracy: 0.950
Invalid: 0.000
Latency: 78.056 s
Output throughput: 1217.443 token/s

4.4. Repro Script

Please refer to https://github.com/sgl-project/sglang/pull/9660.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/390 **Author:** [@Zqy11](https://github.com/Zqy11) **Created:** 9/2/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/overlap` --- ### 📝 Commits (10+) - [`9a58e8c`](https://github.com/deepseek-ai/DeepEP/commit/9a58e8c168cfa11a3253f0fbe2df8e05b195cbb5) feat: Overlap down gemm and combine send - [`433a6a4`](https://github.com/deepseek-ai/DeepEP/commit/433a6a4bdfc4d2a6a84d81e78a41bc4d5102b44c) merge main - [`42ed271`](https://github.com/deepseek-ai/DeepEP/commit/42ed2712538f047ba6d0664d6e1c48c20543a594) optimize put finishing flag for overlap mode - [`389e94b`](https://github.com/deepseek-ai/DeepEP/commit/389e94b55a8b271f4a71581ecee290e168f3626c) optimize put finishing flag for overlap mode - [`b596396`](https://github.com/deepseek-ai/DeepEP/commit/b596396a19e89714c94def7ae582609528523f03) Merge branch 'main' of https://github.com/Zqy11/DeepEP into feat/overlap - [`988bad8`](https://github.com/deepseek-ai/DeepEP/commit/988bad893e320cad2ddad0221b4e36b85ffd1d5e) maintain: delete redundant code - [`11c5c66`](https://github.com/deepseek-ai/DeepEP/commit/11c5c6646a80c043b7541f988792f0b78b95dbf4) maintain: add lambda send_finish_flag - [`99378d5`](https://github.com/deepseek-ai/DeepEP/commit/99378d586237530331b0b4e5b7f23871c8a67cfe) maintain: merge main - [`6601898`](https://github.com/deepseek-ai/DeepEP/commit/66018983803cd0f86994b7437db4c67e295ec199) Merge branch 'main' of github.com:Zqy11/DeepEP into feat/overlap - [`b6f500c`](https://github.com/deepseek-ai/DeepEP/commit/b6f500c1ad4db3d6478d8c8acbb009c6f4fb4ab7) maintain: add kNumMaxExperts ### 📊 Changes **7 files changed** (+218 additions, -67 deletions) <details> <summary>View changed files</summary> 📝 `csrc/deep_ep.cpp` (+16 -5) 📝 `csrc/deep_ep.hpp` (+2 -0) 📝 `csrc/kernels/api.cuh` (+4 -3) 📝 `csrc/kernels/internode_ll.cu` (+135 -31) 📝 `deep_ep/buffer.py` (+16 -2) 📝 `tests/test_low_latency.py` (+41 -26) 📝 `tests/utils.py` (+4 -0) </details> ### 📄 Description # 1. Motivation The optimization effect of Two-Batch Overlap (TBO) is suboptimal for the Decode phase on low-compute-power cards (i.e., H20). This is due to two main factors: First, on the Hopper architecture, the WGMMA block_m is 64. Consequently, when TBO is enabled with a small Decode batch size, the MLP GEMM suffers from redundant computations. A positive throughput gain is only observed at larger batch sizes (e.g., 64, 128). Second, at these larger batch sizes, low-compute-power cards like the H20 fail to meet the SLA guarantees for TPOT/ITL. Therefore, it is necessary to find a solution that can improve Decode throughput even with small batch sizes. Single Batch Overlap (SBO) presents itself as a viable solution. We implement SBO for DeepSeek v3/R1 by modifying DeepEP and DeepGEMM, including the overlap of Shared Expert and Dispatch Recv, as well as the overlap of Down GEMM with Combine Send. The overlap of Down GEMM with Combine Send is implemented by modifying SGlang, DeepEP and DeepGEMM, with the detailed implementation available in the PRs below: * DeepEP: this PR. * DeepGEMM: https://github.com/sgl-project/DeepGEMM/pull/14. We also conduct integration and evaluation in SGLang: https://github.com/sgl-project/sglang/pull/9660. # 2. Overlap Design SBO implements two overlap for the MoE layers of DeepSeek-V3/R1. One is to overlap the Shared Expert computation with the Dispatch Recv communication, and the other is to overlap the Down GEMM computation with the Combine Send communication. <img width="2484" height="772" alt="image" src="https://github.com/user-attachments/assets/b4d1f131-a1b5-4bf6-b623-fbf7b6fe6497" /> The interaction between Down GEMM and Combine Send is structured as a producer-consumer model synchronized by signals. For each local expert, a signal unit is allocated for every block_m tokens. The Down GEMM computes the results for these block_m tokens and atomically increments the signaling unit after completing a portion of the work. The Combine Send polls this signaling unit. Once the value reaches a threshold, it sends the corresponding block_m tokens. <img width="358" height="350" alt="image" src="https://github.com/user-attachments/assets/514134a5-0225-4698-855b-5a43c0a08bc7" /> # 3. Modifications * In `low_latency_dispatch`, change the type of packed_recv_src_info from kInt32 to kInt64 to track the src_rank for each token. * The `internode_ll::combine` kernel uses fewer SMs when the `overlap` parameter is enabled and integrates logic for overlap mode, such as polling signals, sending tokens and puting finish flag. # 4. Evaluation We integrated the modified DeepEP and DeepGEMM into SGLang for performance evaluation. ## 4.1. Experiment Setup * 5 nodes, with 8 × H20 GPUs per node. Each prefill node uses TP8, and the other 2 decode nodes use DP_Attn 16 + EP 16. * Input length 4096, output length 1536. ## 4.2. Performance Evaluation * bs 32, origin <img width="3228" height="820" alt="image" src="https://github.com/user-attachments/assets/ccdc88bf-d3d4-4567-bded-4fe93bf456f8" /> ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 4.8 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2359.16 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15672509 Request throughput (req/s): 4.34 Input token throughput (tok/s): 17778.82 Output token throughput (tok/s): 6667.06 Total token throughput (tok/s): 24445.88 Concurrency: 490.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 112892.31 Median E2E Latency (ms): 113847.19 ---------------Time to First Token---------------- Mean TTFT (ms): 640.62 Median TTFT (ms): 545.06 P99 TTFT (ms): 1543.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.11 Median ITL (ms): 71.81 P95 ITL (ms): 86.02 P99 ITL (ms): 155.32 Max ITL (ms): 1543.26 ================================================== ``` ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 5.0 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2357.80 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15673361 Request throughput (req/s): 4.34 Input token throughput (tok/s): 17789.05 Output token throughput (tok/s): 6670.89 Total token throughput (tok/s): 24459.95 Concurrency: 490.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 113015.97 Median E2E Latency (ms): 113951.58 ---------------Time to First Token---------------- Mean TTFT (ms): 724.98 Median TTFT (ms): 624.73 P99 TTFT (ms): 1693.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.13 Median ITL (ms): 71.84 P95 ITL (ms): 86.57 P99 ITL (ms): 155.21 Max ITL (ms): 1081.95 ================================================== ``` * bs 32, sbo <img width="2990" height="822" alt="image" src="https://github.com/user-attachments/assets/e22b559c-db1c-46c1-8d4f-bf53b8fb12c4" /> ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 4.8 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2211.76 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15673456 Request throughput (req/s): 4.63 Input token throughput (tok/s): 18963.67 Output token throughput (tok/s): 7111.38 Total token throughput (tok/s): 26075.05 Concurrency: 481.58 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 104017.64 Median E2E Latency (ms): 105363.65 ---------------Time to First Token---------------- Mean TTFT (ms): 606.28 Median TTFT (ms): 508.61 P99 TTFT (ms): 1475.44 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.35 Median ITL (ms): 66.10 P95 ITL (ms): 81.58 P99 ITL (ms): 141.96 Max ITL (ms): 1422.74 ================================================== ``` ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 5.0 Max request concurrency: 512 Successful requests: 10240 Benchmark duration (s): 2194.12 Total input tokens: 41943040 Total generated tokens: 15728640 Total generated tokens (retokenized): 15672577 Request throughput (req/s): 4.67 Input token throughput (tok/s): 19116.14 Output token throughput (tok/s): 7168.55 Total token throughput (tok/s): 26284.70 Concurrency: 487.92 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 104545.42 Median E2E Latency (ms): 105483.50 ---------------Time to First Token---------------- Mean TTFT (ms): 619.03 Median TTFT (ms): 511.23 P99 TTFT (ms): 1504.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.68 Median ITL (ms): 66.44 P95 ITL (ms): 82.13 P99 ITL (ms): 142.48 Max ITL (ms): 1024.85 ================================================== ``` ## 4.3. Accuracy Tests * bs 32, origin ``` #python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000 100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:20<00:00, 12.41it/s] Accuracy: 0.951 Invalid: 0.000 Latency: 80.802 s Output throughput: 1183.468 token/s ``` * bs 32, sbo ``` #python -m benchmark.gsm8k.bench_sglang --port 8000 --num-questions 1000 100%|█████████████████████████████████████████████████████████████| 1000/1000 [01:17<00:00, 12.87it/s] Accuracy: 0.950 Invalid: 0.000 Latency: 78.056 s Output throughput: 1217.443 token/s ``` ## 4.4. Repro Script Please refer to https://github.com/sgl-project/sglang/pull/9660. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:53 +00:00