[PR #500] [MERGED] opt ll dispatch layered algo #598

New issue

Closed

opened 2026-10-27 21:04:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:25 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/500
Author: @alpha-baby
Created: 11/21/2025
Status: ✅ Merged
Merged: 12/4/2025
Merged by: @wangfakang

Base: antgroup-opt ← Head: fujh/pr/ll_dispatch_opt

📝 Commits (2)

86f8836 opt ll dispatch
a3c90be fix comments and format code

📊 Changes

7 files changed (+392 additions, -108 deletions)

View changed files

📝 csrc/config.hpp (+24 -4)
📝 csrc/deep_ep.cpp (+11 -7)
📝 csrc/deep_ep.hpp (+3 -1)
📝 csrc/kernels/api.cuh (+4 -2)
📝 csrc/kernels/internode_ll.cu (+303 -61)
📝 deep_ep/buffer.py (+20 -8)
📝 tests/test_low_latency.py (+27 -25)

📄 Description

introduce

algo opt for dispatch in low-latency mode:

In the dispatch kernel of DeepEP's low-latency mode, the original algorithm directly sends data to the destination rank via the RDMA cross-orbit network. A drawback of this algorithm is that it results in excessive duplicate data being transmitted over the RDMA network. Now, drawing inspiration from the approach used in normal mode, we can improve the dispatch kernel in low-latency mode by first sending data to the same-orbit rank on the cross-node, and then forwarding it to the actual destination rank via the NVLink interconnect.

Note: This feature conflicts with the existing Elasticity Support to DeepEP for Fault-Tolerant EP Inference functionality, and the two features cannot be enabled simultaneously.

before:

after:

performance

benchmark:

use

This feature is enabled by default and requires no additional activation from the user. To disable it, please set the following environment variable: DEEPEP_DISABLE_LL_DISPATCH_OPT=1.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/500 **Author:** [@alpha-baby](https://github.com/alpha-baby) **Created:** 11/21/2025 **Status:** ✅ Merged **Merged:** 12/4/2025 **Merged by:** [@wangfakang](https://github.com/wangfakang) **Base:** `antgroup-opt` ← **Head:** `fujh/pr/ll_dispatch_opt` --- ### 📝 Commits (2) - [`86f8836`](https://github.com/deepseek-ai/DeepEP/commit/86f88369995a1b65d0499f6209d39cc64dc3e233) opt ll dispatch - [`a3c90be`](https://github.com/deepseek-ai/DeepEP/commit/a3c90be63e283cc8e4b159c1f8bc0f8543b2b364) fix comments and format code ### 📊 Changes **7 files changed** (+392 additions, -108 deletions) <details> <summary>View changed files</summary> 📝 `csrc/config.hpp` (+24 -4) 📝 `csrc/deep_ep.cpp` (+11 -7) 📝 `csrc/deep_ep.hpp` (+3 -1) 📝 `csrc/kernels/api.cuh` (+4 -2) 📝 `csrc/kernels/internode_ll.cu` (+303 -61) 📝 `deep_ep/buffer.py` (+20 -8) 📝 `tests/test_low_latency.py` (+27 -25) </details> ### 📄 Description # introduce algo opt for dispatch in low-latency mode: In the dispatch kernel of DeepEP's low-latency mode, the original algorithm directly sends data to the destination rank via the RDMA cross-orbit network. A drawback of this algorithm is that it results in excessive duplicate data being transmitted over the RDMA network. Now, drawing inspiration from the approach used in normal mode, we can improve the dispatch kernel in low-latency mode by first sending data to the same-orbit rank on the cross-node, and then forwarding it to the actual destination rank via the NVLink interconnect. Note: This feature conflicts with the existing [Elasticity Support to DeepEP for Fault-Tolerant EP Inference](https://github.com/deepseek-ai/DeepEP/pull/370) functionality, and the two features cannot be enabled simultaneously. before: <img width="476" height="544" alt="image" src="https://github.com/user-attachments/assets/5cb02acd-cb43-474e-a209-8ac9a5a4eb95" /> after: <img width="520" height="588" alt="image" src="https://github.com/user-attachments/assets/ce4d5361-c5b9-45d4-ae17-184da84a6619" /> # performance benchmark: <img width="744" height="1096" alt="image" src="https://github.com/user-attachments/assets/bdb0a20e-2d43-45df-9477-95497725a894" /> # use This feature is enabled by default and requires no additional activation from the user. To disable it, please set the following environment variable: `DEEPEP_DISABLE_LL_DISPATCH_OPT=1`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:04:25 +00:00