[PR #492] [Feature] Per Expert Overlap (PEO) #594

New issue

Open

opened 2026-10-27 21:04:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/492
Author: @k-ling3
Created: 11/13/2025
Status: 🔄 Open

Base: main ← Head: PEO

📝 Commits (2)

b5fe7bb Support per-expert-overlap SBO.
ec07b5f Support PEO combine input x size divide by num_rounds.

📊 Changes

7 files changed (+374 additions, -142 deletions)

View changed files

📝 csrc/deep_ep.cpp (+93 -48)
📝 csrc/deep_ep.hpp (+13 -4)
📝 csrc/event.hpp (+3 -1)
📝 csrc/kernels/api.cuh (+9 -2)
📝 csrc/kernels/internode_ll.cu (+201 -71)
📝 deep_ep/buffer.py (+8 -3)
📝 tests/test_low_latency.py (+47 -13)

📄 Description

Background

The insight of this work is that we group the experts, allowing the communication of some experts to overlap with the computation of other experts. We call this approach Per Expert Overlap (PEO). Compared to existing methods, our approach has the following advantages:

1. Performance:

Compared to Non-overlap, PEO performs better at all batch sizes.
- For the DPSK model:
  - At batch size 4, PEO achieves an 11% improvement.
  - At batch size 128, PEO achieves a 31% improvement. The larger the batch size, the more significant the gain.
- For the QWEN model, PEO achieves up to a 51% improvement.
Compared to PR 390 (https://github.com/deepseek-ai/DeepEP/pull/390), PEO also performs better.

2. Usability

Compared to PR 390, PEO only modifies DeepEP and does not change DeepGEEM, making it easier to use.

In short, during the dispatch phase, we change the order of communication (by modifying DeepEP) to allow some experts to receive tokens first. During the GEMM phase, we change the order of computation (by modifying how the inference engine calls DeepGEMM) to allow some experts to compute first. In the combine phase, we let some experts send tokens first. Overall, this allows the communication of some experts to overlap with the computation of others.

Design

In the original DeepEP, each communication unit consists of num_experts or num_local_experts experts. That is, during the dispatch phase, each rank sends tokens to num_experts experts. During the combine phase, each rank sends tokens from num_local_experts experts to num_ranks ranks.

This solution modifies DeepEP by dividing the experts into num_rounds groups, and the communication is divided into num_rounds rounds.

During the dispatch phase, in each round, each rank sends tokens to num_experts // num_rounds experts.
During the combine phase, in each round, each rank sends tokens from num_local_experts // num_rounds local experts to num_ranks ranks.

The process is shown as follows:

Due to differences in model parameters, deployment scale, and batch size, this solution allows the following adjustable parameters to achieve the best overlap effect in different scenarios:

Parameters for Overlap:

Overlap method
- We tested different overlap methods and found they have different effects. Consider the following options:
  - overlap-1: After all dispatch sends are completed, then perform dispatch recv and GEMM.
  - overlap-2: Immediately after each dispatch send, perform recv + gemm.
  - overlap-3: Immediately after each dispatch send, perform recv + gemm, and allow DeepEP's send and recv to overlap.
  - overlap-4: No overlap between dispatch and GEMM.
num_rounds: Number of rounds for splitting dispatch/combine.
deepep_send_num_sms: Number of SMs used for dispatch/combine send.
deepep_recv_num_sms: Number of SMs used for dispatch/combine recv.
up_deepgemm_num_sms: Number of SMs used for UP GEMM.
down_deepgemm_num_sms: Number of SMs used for DOWN GEMM.

Performance

Configuration:

H20, EP16, QWEN, DPSK

Comparison Methods:

non-overlap
PR 390 (https://github.com/deepseek-ai/DeepEP/pull/390) (SBO)

Conclusion:

For both DPSK and QWEN, the overlap performance is the best at almost all batch sizes. For DPSK, PEO achieves a maximum of 31% improvement at batch size 128. For QWEN, PEO achieves a maximum of 50% improvement at batch size 16.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/492 **Author:** [@k-ling3](https://github.com/k-ling3) **Created:** 11/13/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `PEO` --- ### 📝 Commits (2) - [`b5fe7bb`](https://github.com/deepseek-ai/DeepEP/commit/b5fe7bbd9719a2d4af85591747bb54bb12e79da5) Support per-expert-overlap SBO. - [`ec07b5f`](https://github.com/deepseek-ai/DeepEP/commit/ec07b5fe97205f0a26c9d410160de37ee294c3c0) Support PEO combine input x size divide by num_rounds. ### 📊 Changes **7 files changed** (+374 additions, -142 deletions) <details> <summary>View changed files</summary> 📝 `csrc/deep_ep.cpp` (+93 -48) 📝 `csrc/deep_ep.hpp` (+13 -4) 📝 `csrc/event.hpp` (+3 -1) 📝 `csrc/kernels/api.cuh` (+9 -2) 📝 `csrc/kernels/internode_ll.cu` (+201 -71) 📝 `deep_ep/buffer.py` (+8 -3) 📝 `tests/test_low_latency.py` (+47 -13) </details> ### 📄 Description ## Background The insight of this work is that we group the experts, allowing the communication of some experts to overlap with the computation of other experts. We call this approach Per Expert Overlap (PEO). Compared to existing methods, our approach has the following advantages: ### 1. Performance: - **Compared to Non-overlap**, PEO performs better at all batch sizes. - For the DPSK model: - At batch size 4, PEO achieves an 11% improvement. - At batch size 128, PEO achieves a 31% improvement. The larger the batch size, the more significant the gain. - For the QWEN model, PEO achieves up to a 51% improvement. - **Compared to PR 390** (https://github.com/deepseek-ai/DeepEP/pull/390), PEO also performs better. ### 2. Usability - **Compared to PR 390**, PEO only modifies DeepEP and does not change DeepGEEM, making it easier to use. In short, during the dispatch phase, we change the order of communication (by modifying DeepEP) to allow some experts to receive tokens first. During the GEMM phase, we change the order of computation (by modifying how the inference engine calls DeepGEMM) to allow some experts to compute first. In the combine phase, we let some experts send tokens first. Overall, this allows the communication of some experts to overlap with the computation of others. --- ## Design In the original DeepEP, each communication unit consists of `num_experts` or `num_local_experts` experts. That is, during the dispatch phase, each rank sends tokens to `num_experts` experts. During the combine phase, each rank sends tokens from `num_local_experts` experts to `num_ranks` ranks. This solution modifies DeepEP by dividing the experts into `num_rounds` groups, and the communication is divided into `num_rounds` rounds. - **During the dispatch phase**, in each round, each rank sends tokens to `num_experts // num_rounds` experts. - **During the combine phase**, in each round, each rank sends tokens from `num_local_experts // num_rounds` local experts to `num_ranks` ranks. The process is shown as follows: <img src="https://github.com/user-attachments/assets/5a717059-4241-418d-a591-436b163d456f" width="300" height="200" > Due to differences in model parameters, deployment scale, and batch size, this solution allows the following adjustable parameters to achieve the best overlap effect in different scenarios: ### Parameters for Overlap: - **Overlap method** - We tested different overlap methods and found they have different effects. Consider the following options: - **overlap-1**: After all dispatch sends are completed, then perform dispatch recv and GEMM. <div style="text-align:center;"> <img src="https://github.com/user-attachments/assets/b24d51f3-47ef-4d7c-9a28-ce1b0e20bc53" width="300" height="200" alt="overlap-1 diagram"> </div> - **overlap-2**: Immediately after each dispatch send, perform recv + gemm. <div style="text-align:center;"> <img src="https://github.com/user-attachments/assets/c8ce2aee-cd27-44b9-a6e1-976ecd43ee83" width="300" height="200" alt="overlap-2 diagram"> </div> - **overlap-3**: Immediately after each dispatch send, perform recv + gemm, and allow DeepEP's send and recv to overlap. <div style="text-align:center;"> <img src="https://github.com/user-attachments/assets/30b76a17-91f0-46b7-856d-c7170baabbb1" width="300" height="200" alt="overlap-2 diagram"> </div> - **overlap-4**: No overlap between dispatch and GEMM. <div style="text-align:center;"> <img src="https://github.com/user-attachments/assets/49a8d27a-2265-418d-883a-4bfdc8f619fa" width="300" height="200" alt="overlap-2 diagram"> </div> - **num_rounds**: Number of rounds for splitting dispatch/combine. - **deepep_send_num_sms**: Number of SMs used for dispatch/combine send. - **deepep_recv_num_sms**: Number of SMs used for dispatch/combine recv. - **up_deepgemm_num_sms**: Number of SMs used for UP GEMM. - **down_deepgemm_num_sms**: Number of SMs used for DOWN GEMM. --- ## Performance ### Configuration: - H20, EP16, QWEN, DPSK ### Comparison Methods: - non-overlap - PR 390 (https://github.com/deepseek-ai/DeepEP/pull/390) (SBO) ### Conclusion: For both DPSK and QWEN, the overlap performance is the best at almost all batch sizes. For DPSK, PEO achieves a maximum of 31% improvement at batch size 128. For QWEN, PEO achieves a maximum of 50% improvement at batch size 16. <img src="https://github.com/user-attachments/assets/e51ee801-5bd4-49dd-8969-b25b83e70534" > <img src="https://github.com/user-attachments/assets/3ac09ea1-f5d8-4e41-8661-a1faf13821e9" > --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:04:21 +00:00

bsakaguchi referenced this issue

2026-10-27 21:04:52 +00:00

[PR #594] [CLOSED] ll dispatch: bulk put per expert to reduce RDMA overhead #649