[PR #375] [MERGED] [Feature] Support normal mode for non-nvlink environment #543

New issue

Closed

opened 2026-10-27 21:03:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:03:50 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/375
Author: @zhou9402
Created: 8/25/2025
Status: ✅ Merged
Merged: 9/25/2025
Merged by: @jershi425

Base: hybrid-ep ← Head: pcie

📝 Commits (10+)

b5eab60 add PCIE
5277b0d for sender architecture
bd91666 Revert "for sender architecture"
9e2acbe sender pesudocode
5e0b78f merge
9cd706b remove is_token_in_rank,added topk_ranks_idx
04bafa0 add deepep_cpp pcie dispatch
57bb2e6 internode_dispatcCh_sender:use topk_idx to get dst_rank for simplicity
b805c0e Add notify_dispatch_pcie for non-nvlink GPU
3ff8113 fix

📊 Changes

15 files changed (+2046 additions, -63 deletions)

View changed files

📝 README.md (+3 -0)
📝 csrc/config.hpp (+25 -0)
📝 csrc/deep_ep.cpp (+385 -15)
📝 csrc/deep_ep.hpp (+20 -1)
📝 csrc/kernels/CMakeLists.txt (+2 -1)
📝 csrc/kernels/api.cuh (+45 -1)
📝 csrc/kernels/internode.cu (+1 -3)
📝 csrc/kernels/launch.cuh (+1 -0)
➕ csrc/kernels/pcie.cu (+1146 -0)
📝 csrc/kernels/runtime.cu (+2 -2)
📝 deep_ep/buffer.py (+97 -7)
📝 deep_ep/utils.py (+34 -32)
➕ docs/README_PCIe.md (+63 -0)
📝 setup.py (+4 -1)
➕ tests/test_normal_without_nvl.py (+218 -0)

📄 Description

Description

DeepEP was originally designed for high-end data center GPUs with NVLink connectivity. However, many GPU configurations lack NVLink support, such as RTX professional series cards and consumer GPUs. This PR extends DeepEP to support non-NVLink environments.

Design

1D Communication Architecture: All communication (both intra-node and inter-node) is routed exclusively through the RDMA NIC, eliminating the complexity of PCIe/RDMA traffic splitting. This unified approach is motivated by several technical considerations:
1. Limited PCIe Atomic Support: PCIe atomic operations are not well supported , which leads to reliability issues.
2. PCIe Contention: Heavy bus contention significantly degrades performance when multiple GPUs compete for limited PCIe bandwidth.
3. CPU Interconnect Bandwidth Constraints: The limited bandwidth of CPU interconnects (UPI/QPI) creates bottlenecks in multi-socket systems, particularly under high communication loads
Bounce Buffer Strategy: To maintain predictable memory usage, we employ an intermediate buffer mechanism, just similar to current normal mode implementation. These buffers are pre-registered to the NIC to optimize RDMA transfer performance while preventing excessive memory consumption.

NOTE: The current PCIE mode supports communication with up to 32 ranks.

Dispatch dataflow

Performance

Enviroment

GPU: H20 with disabled NVLink
NIC: 8 CX7, one GPU one NIC
Config: 24 sms，4096 tokens per batch, 7168 hidden, top-8 experts, FP8 dispatching and BF16 combining

Result

Current Progress

Now we have added a flag allow_nvlink_for_normal_mode to the Python interface that allows users to select the transport path. However, we still maintain a separate PCIe kernel implementation on the C++ side. We are working to integrate the PCIe kernel into the existing kernel architecture. Due to differences in buffer allocation and synchronization mechanisms, this will require excessive additional conditional handling in deep_ep.cpp or the internode.cu interface.
We would appreciate your suggestions and feedback regarding this design!

Future work

Add TMA support
Reduce SM usage by merging sender and receiver
Optimization for CX8
PCIE path support (under discussion)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/375 **Author:** [@zhou9402](https://github.com/zhou9402) **Created:** 8/25/2025 **Status:** ✅ Merged **Merged:** 9/25/2025 **Merged by:** [@jershi425](https://github.com/jershi425) **Base:** `hybrid-ep` ← **Head:** `pcie` --- ### 📝 Commits (10+) - [`b5eab60`](https://github.com/deepseek-ai/DeepEP/commit/b5eab60719248affbfe00f361e2367b3a88904d6) add PCIE - [`5277b0d`](https://github.com/deepseek-ai/DeepEP/commit/5277b0d5ada7bffb992fe781a82b44b2165bdf86) for sender architecture - [`bd91666`](https://github.com/deepseek-ai/DeepEP/commit/bd91666581330c2bc58819bf7f3ebfcf6e9b4aea) Revert "for sender architecture" - [`9e2acbe`](https://github.com/deepseek-ai/DeepEP/commit/9e2acbe339a130d048762eb85a1d09cd5976d324) sender pesudocode - [`5e0b78f`](https://github.com/deepseek-ai/DeepEP/commit/5e0b78fa26967138b084a6a5a482494a5eb7aaea) merge - [`9cd706b`](https://github.com/deepseek-ai/DeepEP/commit/9cd706b0bb9a4651eb6b7ee9402c615483a860be) remove is_token_in_rank,added topk_ranks_idx - [`04bafa0`](https://github.com/deepseek-ai/DeepEP/commit/04bafa0f90b345cc751034ad8b218043ab1087ac) add deepep_cpp pcie dispatch - [`57bb2e6`](https://github.com/deepseek-ai/DeepEP/commit/57bb2e685f7335c135a888ab4a6190063b0686aa) internode_dispatcCh_sender:use topk_idx to get dst_rank for simplicity - [`b805c0e`](https://github.com/deepseek-ai/DeepEP/commit/b805c0e15e0d697c06d5db05fabcac2f0edb6e30) Add notify_dispatch_pcie for non-nvlink GPU - [`3ff8113`](https://github.com/deepseek-ai/DeepEP/commit/3ff8113764f57f4a25824fda1ba40294a910b663) fix ### 📊 Changes **15 files changed** (+2046 additions, -63 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+3 -0) 📝 `csrc/config.hpp` (+25 -0) 📝 `csrc/deep_ep.cpp` (+385 -15) 📝 `csrc/deep_ep.hpp` (+20 -1) 📝 `csrc/kernels/CMakeLists.txt` (+2 -1) 📝 `csrc/kernels/api.cuh` (+45 -1) 📝 `csrc/kernels/internode.cu` (+1 -3) 📝 `csrc/kernels/launch.cuh` (+1 -0) ➕ `csrc/kernels/pcie.cu` (+1146 -0) 📝 `csrc/kernels/runtime.cu` (+2 -2) 📝 `deep_ep/buffer.py` (+97 -7) 📝 `deep_ep/utils.py` (+34 -32) ➕ `docs/README_PCIe.md` (+63 -0) 📝 `setup.py` (+4 -1) ➕ `tests/test_normal_without_nvl.py` (+218 -0) </details> ### 📄 Description ### **Description** DeepEP was originally designed for high-end data center GPUs with NVLink connectivity. However, many GPU configurations lack NVLink support, such as RTX professional series cards and consumer GPUs. This PR extends DeepEP to support non-NVLink environments. ### **Design** - **1D Communication Architecture**: All communication (both intra-node and inter-node) is routed exclusively through the RDMA NIC, eliminating the complexity of PCIe/RDMA traffic splitting. This unified approach is motivated by several technical considerations: 1. Limited PCIe Atomic Support: PCIe atomic operations are not well supported , which leads to reliability issues. 2. PCIe Contention: Heavy bus contention significantly degrades performance when multiple GPUs compete for limited PCIe bandwidth. 3. CPU Interconnect Bandwidth Constraints: The limited bandwidth of CPU interconnects (UPI/QPI) creates bottlenecks in multi-socket systems, particularly under high communication loads - **Bounce Buffer Strategy**: To maintain predictable memory usage, we employ an intermediate buffer mechanism, just similar to current normal mode implementation. These buffers are pre-registered to the NIC to optimize RDMA transfer performance while preventing excessive memory consumption. **NOTE: The current PCIE mode supports communication with up to 32 ranks.** > Dispatch dataflow <img width="846" height="417" alt="image" src="https://github.com/user-attachments/assets/d8711acf-16e2-46fe-bc74-80bb510321f2" /> ### **Performance** **Enviroment** 1. GPU: H20 with disabled NVLink 2. NIC: 8 CX7, one GPU one NIC 3. Config: 24 sms，4096 tokens per batch, 7168 hidden, top-8 experts, FP8 dispatching and BF16 combining **Result** <img width="1153" height="212" alt="image" src="https://github.com/user-attachments/assets/2a87d0a2-4ebb-4d90-8bac-15bbbd40b665" /> ### **Current Progress** Now we have added a flag `allow_nvlink_for_normal_mode` to the Python interface that allows users to select the transport path. However, we still maintain a separate PCIe kernel implementation on the C++ side. We are working to integrate the PCIe kernel into the existing kernel architecture. Due to differences in buffer allocation and synchronization mechanisms, this will require excessive additional conditional handling in `deep_ep.cpp` or the `internode.cu` interface. We would appreciate your suggestions and feedback regarding this design! ### **Future work** - [ ] Add TMA support - [ ] Reduce SM usage by merging sender and receiver - [ ] Optimization for CX8 - [ ] PCIE path support (under discussion) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-27 21:03:50 +00:00