[PR #650] Add support to multi-plane NICs and multi-NIC per PE to DeepEP v1 #687

New issue

Open

opened 2026-10-27 21:05:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:17 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/650
Author: @aahouzi
Created: 5/28/2026
Status: 🔄 Open

Base: main ← Head: multi-nic-fix

📝 Commits (4)

20aa88b Add support for multi-plane NICs / multi-NIC per PE
532d7e2 Merge branch 'deepseek-ai:main' into multi-nic-fix
01e4c91 Merge branch 'deepseek-ai:main' into multi-nic-fix
d2a1fdc fix(legacy): use the v2 IBGDA device-state RC indexing in ibgda_get_rc

📊 Changes

3 files changed (+27 additions, -14 deletions)

View changed files

📝 csrc/kernels/legacy/ibgda_device.cuh (+1 -3)
📝 csrc/kernels/legacy/internode.cu (+14 -6)
📝 csrc/kernels/legacy/internode_ll.cu (+12 -5)

📄 Description

Description

On multi-plane NICs (e.g. CX-8 dual-plane) or multi-NIC per PE, enabling NVSHMEM's NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1 was not enough to actually distribute traffic across planes in DeepEP, and reach full line rate of CX8, even though shmem_put_bw perftest was able to reach line rate of CX8 NICs.

After some investigation, it seems that the DeepEP v1 kernels only addressed the first half of QP pool, so all RDMA traffic stayed on a single plane even when NVSHMEM had correctly allocated QPs on all available planes. So we end up with this scenario on a DGX B300 NVL8 + CX8 dual-plane cluster where only plane p0 of each of the 8 NICs is being utilized by DeepEP:

Solution

This PR fixes the QP selection logic in both legacy DeepEP v1 kernels (Internode + LL), so traffic is distributed equally across all available planes/NICs per PE. Also, the PR should work well for multi-NIC per PE scenarios.

With multi-port enabled, NVSHMEM allocates num_rc_per_pe × num_devs QPs per peer, laid out as [NIC 0 QPs | NIC 1 QPs | ... | NIC N-1 QPs]. The new QP selection logic:

Internode: Different channels bind to different NICs for their whole lifetime, distributing the traffic across all NICs assigned per PE, and also preserving RC ordering.
LL: Binding the QP choice to expert_local_idx, and since different warp groups have different expert indices, traffic ends up spread across all NICs assigned per PE naturally. This requires num_local_experts >= num_devs since with fewer local experts than NICs, some NICs stay unused.

The fix also is fully backward compatible, so if a user doesn't provide NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1, the qp_id reverts back to the original value it had for 1 NIC per PE.

Also, users are not required to provide any NIC to PE mapping via NVSHMEM_HCA_PE_MAPPING, as the default PCI-path of NVSHMEM can correctly detect when 2 NICs are closer to the same PE and assign it correctly whenever the user provides NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1.

Results

Setup: DGX B300 NVL8 with CX-8 dual-plane
Internode test for 2N and 8N:

LL test for 4N and 8N across num_local_experts={4,8}

NIC counters during the run for all planes are fully utilized and balanced:

Requirements

NVSHMEM ≥ 3.7, which is now released
Multi-port feature enabled via: NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1
This PR builds on top of #288 to actually distribute traffic across the num_rc_per_pe × num_devs QP pool.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/650 **Author:** [@aahouzi](https://github.com/aahouzi) **Created:** 5/28/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `multi-nic-fix` --- ### 📝 Commits (4) - [`20aa88b`](https://github.com/deepseek-ai/DeepEP/commit/20aa88b9b8b5719bc58aa67b23c7851482177283) Add support for multi-plane NICs / multi-NIC per PE - [`532d7e2`](https://github.com/deepseek-ai/DeepEP/commit/532d7e21d3f476efb9082300254d515174c57cb5) Merge branch 'deepseek-ai:main' into multi-nic-fix - [`01e4c91`](https://github.com/deepseek-ai/DeepEP/commit/01e4c91a8c4bff30ef3e61303ff4263e33eccf18) Merge branch 'deepseek-ai:main' into multi-nic-fix - [`d2a1fdc`](https://github.com/deepseek-ai/DeepEP/commit/d2a1fdcbc9304afd4017a9e81cc32dc9722daa27) fix(legacy): use the v2 IBGDA device-state RC indexing in ibgda_get_rc ### 📊 Changes **3 files changed** (+27 additions, -14 deletions) <details> <summary>View changed files</summary> 📝 `csrc/kernels/legacy/ibgda_device.cuh` (+1 -3) 📝 `csrc/kernels/legacy/internode.cu` (+14 -6) 📝 `csrc/kernels/legacy/internode_ll.cu` (+12 -5) </details> ### 📄 Description ## Description On multi-plane NICs (e.g. CX-8 dual-plane) or multi-NIC per PE, enabling NVSHMEM's `NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1` was not enough to actually distribute traffic across planes in DeepEP, and reach full line rate of CX8, even though `shmem_put_bw` perftest was able to reach line rate of CX8 NICs. After some investigation, it seems that the DeepEP v1 kernels only addressed the first half of QP pool, so all RDMA traffic stayed on a single plane even when NVSHMEM had correctly allocated QPs on all available planes. So we end up with this scenario on a DGX B300 NVL8 + CX8 dual-plane cluster where only plane p0 of each of the 8 NICs is being utilized by DeepEP: <img width="351" height="267" alt="image" src="https://github.com/user-attachments/assets/4ed85044-6115-4cdd-a032-4d5befa8cc0e" /> ## Solution This PR fixes the QP selection logic in both legacy DeepEP v1 kernels (Internode + LL), so traffic is distributed equally across all available planes/NICs per PE. Also, the PR should work well for multi-NIC per PE scenarios. With multi-port enabled, NVSHMEM allocates `num_rc_per_pe × num_devs` QPs per peer, laid out as `[NIC 0 QPs | NIC 1 QPs | ... | NIC N-1 QPs]`. The new QP selection logic: - **Internode:** Different channels bind to different NICs for their whole lifetime, distributing the traffic across all NICs assigned per PE, and also preserving RC ordering. - **LL:** Binding the QP choice to `expert_local_idx`, and since different warp groups have different expert indices, traffic ends up spread across all NICs assigned per PE naturally. This requires `num_local_experts >= num_devs` since with fewer local experts than NICs, some NICs stay unused. The fix also is fully backward compatible, so if a user doesn't provide `NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1`, the `qp_id` reverts back to the original value it had for 1 NIC per PE. Also, users are not required to provide any NIC to PE mapping via `NVSHMEM_HCA_PE_MAPPING`, as the default PCI-path of NVSHMEM can correctly detect when 2 NICs are closer to the same PE and assign it correctly whenever the user provides `NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1`. ## Results - Setup: DGX B300 NVL8 with CX-8 dual-plane - Internode test for 2N and 8N: <img width="1232" height="540" alt="image" src="https://github.com/user-attachments/assets/ccf1d423-34c3-42bb-86e2-b4b5627d5dd3" /> - LL test for 4N and 8N across `num_local_experts={4,8}` <img width="1228" height="602" alt="image" src="https://github.com/user-attachments/assets/f4301dbe-2b04-4895-bc0b-b3524ec2deaf" /> - NIC counters during the run for all planes are fully utilized and balanced: <img width="600" height="400" alt="image" src="https://github.com/user-attachments/assets/b614b2f6-dd0b-4bba-b27d-020eb91ba5d2" /> ## Requirements - **NVSHMEM ≥ 3.7**, which is now released - Multi-port feature enabled via: `NVSHMEM_IBGDA_ENABLE_MULTI_PORT=1` - This PR builds on top of [#288](https://github.com/deepseek-ai/DeepEP/pull/288) to actually distribute traffic across the `num_rc_per_pe × num_devs` QP pool. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:17 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:27 +00:00

[PR #687] [CLOSED] NCCL devComm: allocate according to devCommRuntimeVersionSize #706

bsakaguchi referenced this issue

2026-10-27 21:05:27 +00:00

[PR #688] [MERGED] NCCL devComm: allocate according to devCommRuntimeVersionSize #707