deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[PR #797] Add EP_RDMA_GBS env to declare RDMA link bandwidth when ibstat under-reports it (fixes under-provisioned num_sms on CX8 800G) #791

New issue

Open

opened 2026-10-27 21:06:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:06:25 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/797
Author: @sssssakuraaaaa
Created: 10/26/2026
Status: 🔄 Open

Base: main ← Head: fix/env-rdma-gbs

📝 Commits (1)

b7ac457 Add EP_RDMA_GBS env to declare RDMA link bandwidth when ibstat under-reports it

📊 Changes

1 file changed (+10 additions, -0 deletions)

View changed files

📝 deep_ep/utils/envs.py (+10 -0)

📄 Description

Problem

get_theoretical_num_sms (deep_ep/buffers/ep.py) auto-estimates the dispatch/combine SM count
from the RDMA bandwidth returned by get_rdma_gbs() (deep_ep/utils/envs.py), which parses
ibstat's Rate field. In a bonded (802.3ad LAG) RDMA setup this under-detects bandwidth and
under-provisions the SM count for the inter-node (scale-out) path:

With a bond, ibstat (and the kernel's /sys/class/net/<dev>/speed) report a single member
link's rate, not the bonded aggregate. On a ConnectX-8 800G NIC that member shows 400
(4X NDR), so get_rdma_gbs returns 50 GB/s instead of 100.
The default also queries mlx5_0, which on some hosts is a 200G front-end/management NIC (not
the GPU rail), yielding 25 GB/s.

Either way the auto num_sms comes out far too low (e.g. ~6–8 instead of ~28), which directly
starves the scale-out path. Users currently work around it by hard-coding --num-sms, a magic
number that doesn't adapt to token/expert/topk changes.

Fix

Add an opt-in EP_RDMA_GBS env that declares the true per-GPU RDMA link bandwidth (GB/s). When
set, get_rdma_gbs returns it directly; when unset, behavior is unchanged. This feeds the correct
input to the existing SM model so num_sms is computed correctly and adapts to the workload,
instead of overriding the output with --num-sms.

    # Allow declaring the true link bandwidth. In a bonded (802.3ad LAG) RDMA
    # setup, ibstat reports a single member link's rate (e.g. a ConnectX-8 800G
    # NIC shows 400, 4X NDR) rather than the bonded aggregate, so detection
    # under-reports. When unset, detection is unchanged.
    env_gbs = os.environ.get('EP_RDMA_GBS')
    if env_gbs:
        try:
            return float(env_gbs)
        except Exception:
            pass

(os is already imported at the top of the file.)

Verification

B300, 2 nodes × 8 GPU (EP16), hybrid mode, NCCL 2.32.3 / CUDA 13, tests/ep/test_ep.py
experts=256 hidden=7168 topk=8 token=4096, EP_OVERRIDE_RDMA_SL=162, --num-sms 0 (auto):

config	auto #SM	dispatch SO	time
default (ibstat → mlx5_0 200G → 25 GB/s)	8	89 GB/s	689 µs
right NIC via ibstat (bond member 400G NDR → 50 GB/s)	16	126 GB/s	485 µs
`EP_RDMA_GBS=100` (true CX8 800G)	28	132 GB/s	463 µs

Declaring the true bandwidth lets the model pick 28 SM and matches a manual --num-sms 32
(132 GB/s / 464 µs), without the magic number. The middle row shows that even selecting the correct
rail NIC still under-reports (bond member 400G → only 16 SM), which is why a declaration — not
better auto-detection — is the robust fix in a bonded 800G setup.

Usage

export EP_RDMA_GBS=100   # per-GPU link GB/s
# then leave num_sms on auto

Scope / safety

Additive and opt-in; no change when the env is unset. Fixing the mlx5_0 default / auto rail-NIC
selection is intentionally out of scope — on a bond it still can't reach the true 800G because
ibstat/sysfs only expose a member link, so it wouldn't remove the need to declare.

Independent of #791 (symmetric-window ordering, EP_RELAXED_WIN_ORDERING) and #794 (tail-signal
layout); the three don't depend on each other.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/797 **Author:** [@sssssakuraaaaa](https://github.com/sssssakuraaaaa) **Created:** 10/26/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/env-rdma-gbs` --- ### 📝 Commits (1) - [`b7ac457`](https://github.com/deepseek-ai/DeepEP/commit/b7ac457b6bbed00c4db08a9057694693474f3c08) Add EP_RDMA_GBS env to declare RDMA link bandwidth when ibstat under-reports it ### 📊 Changes **1 file changed** (+10 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/utils/envs.py` (+10 -0) </details> ### 📄 Description ## Problem `get_theoretical_num_sms` (`deep_ep/buffers/ep.py`) auto-estimates the dispatch/combine SM count from the RDMA bandwidth returned by `get_rdma_gbs()` (`deep_ep/utils/envs.py`), which parses `ibstat`'s `Rate` field. In a **bonded (802.3ad LAG)** RDMA setup this under-detects bandwidth and under-provisions the SM count for the inter-node (scale-out) path: - With a bond, `ibstat` (and the kernel's `/sys/class/net/<dev>/speed`) report a **single member link's** rate, not the bonded aggregate. On a **ConnectX-8 800G** NIC that member shows `400` (4X NDR), so `get_rdma_gbs` returns **50 GB/s** instead of 100. - The default also queries `mlx5_0`, which on some hosts is a 200G front-end/management NIC (not the GPU rail), yielding **25 GB/s**. Either way the auto `num_sms` comes out far too low (e.g. ~6–8 instead of ~28), which directly starves the scale-out path. Users currently work around it by hard-coding `--num-sms`, a magic number that doesn't adapt to token/expert/topk changes. ## Fix Add an opt-in **`EP_RDMA_GBS`** env that declares the true per-GPU RDMA link bandwidth (GB/s). When set, `get_rdma_gbs` returns it directly; when unset, behavior is unchanged. This feeds the correct input to the existing SM model so `num_sms` is computed correctly and adapts to the workload, instead of overriding the output with `--num-sms`. ```python # Allow declaring the true link bandwidth. In a bonded (802.3ad LAG) RDMA # setup, ibstat reports a single member link's rate (e.g. a ConnectX-8 800G # NIC shows 400, 4X NDR) rather than the bonded aggregate, so detection # under-reports. When unset, detection is unchanged. env_gbs = os.environ.get('EP_RDMA_GBS') if env_gbs: try: return float(env_gbs) except Exception: pass ``` (`os` is already imported at the top of the file.) ## Verification B300, 2 nodes × 8 GPU (EP16), hybrid mode, NCCL 2.32.3 / CUDA 13, `tests/ep/test_ep.py` experts=256 hidden=7168 topk=8 token=4096, `EP_OVERRIDE_RDMA_SL=162`, `--num-sms 0` (auto): | config | auto #SM | dispatch SO | time | | --- | --- | --- | --- | | default (ibstat → mlx5_0 200G → 25 GB/s) | 8 | 89 GB/s | 689 µs | | right NIC via ibstat (bond member 400G NDR → 50 GB/s) | 16 | 126 GB/s | 485 µs | | `EP_RDMA_GBS=100` (true CX8 800G) | 28 | 132 GB/s | 463 µs | Declaring the true bandwidth lets the model pick 28 SM and matches a manual `--num-sms 32` (132 GB/s / 464 µs), without the magic number. The middle row shows that even selecting the correct rail NIC still under-reports (bond member 400G → only 16 SM), which is why a **declaration** — not better auto-detection — is the robust fix in a bonded 800G setup. ## Usage ```bash export EP_RDMA_GBS=100 # per-GPU link GB/s # then leave num_sms on auto ``` ## Scope / safety Additive and opt-in; no change when the env is unset. Fixing the `mlx5_0` default / auto rail-NIC selection is intentionally out of scope — on a bond it still can't reach the true 800G because `ibstat`/sysfs only expose a member link, so it wouldn't remove the need to declare. Independent of #791 (symmetric-window ordering, `EP_RELAXED_WIN_ORDERING`) and #794 (tail-signal layout); the three don't depend on each other. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:06:25 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#791

No description provided.

Rows
Columns