[PR #705] fix: correctly detect RDMA bandwidth for bonded NICs #720

New issue

Open

opened 2026-10-27 21:05:34 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:05:34 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/705
Author: @cynton503
Created: 7/27/2026
Status: 🔄 Open

Base: main ← Head: fix/rdma-bond-speed-detection

📝 Commits (1)

13fb806 fix: correctly detect RDMA bandwidth for bonded NICs

📊 Changes

1 file changed (+39 additions, -0 deletions)

View changed files

📝 deep_ep/utils/envs.py (+39 -0)

📄 Description

Summary

Fix incorrect RDMA bandwidth detection for bonded NIC devices (e.g., mlx5_bond_*), which causes suboptimal SM allocation and reduced dispatch/combine throughput.

Problem

In RDMA LAG (bond) mode, ibstat reports only a single logical port with per-port rate:

CA 'mlx5_bond_7'
        CA type: MT4131
        Number of ports: 1
        Port 1:
                Rate: 400

The existing get_rdma_gbs() parses this single port rate and computes 400 / 8 = 50 GB/s. However, dual-port CX7/CX8 NICs in bond mode aggregate two 400 Gb/s physical ports, so the actual available bandwidth is 800 / 8 = 100 GB/s.

This under-reported bandwidth causes get_theoretical_num_sms() to recommend fewer SMs than optimal, directly reducing dispatch/combine kernel performance.

Changes

Add get_net_device_speed_gbs(): maps IB device to its network interface via sysfs (/sys/class/infiniband/<dev>/ports/<port>/gid_attrs/ndevs/0), then reads the kernel-reported aggregated link speed from /sys/class/net/<iface>/speed (in Mbps).
Update get_rdma_gbs() with a three-level detection priority:
1. EP_OVERRIDE_RDMA_GBS environment variable — manual override, consistent with EP_OVERRIDE_RDMA_SL convention
2. sysfs aggregated speed — for bond devices (NIC name containing bond)
3. ibstat Rate parsing — original fallback for non-bond devices or when sysfs is unavailable

Validation

Verified on a multi-node cluster with CX8 dual-port NICs in RDMA LAG mode (EP_NIC_NAME=mlx5_bond_0):

Config	QP / SM (before)	QP / SM (after)	Dispatch GB/s (before → after)	Combine GB/s (before → after)
2-node	65 / 12	65 / 22	49 → 58	53 → 79
4-node	65 / 6	65 / 10	53 → 77	53 → 81

Compatibility

Non-bond devices: behavior unchanged — the sysfs branch is skipped when NIC name does not contain bond
sysfs unavailable: silently falls back to ibstat parsing (returns 0 on any failure)
Manual override: EP_OVERRIDE_RDMA_GBS=<value_in_GBs> takes highest priority for any device type

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/705 **Author:** [@cynton503](https://github.com/cynton503) **Created:** 7/27/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/rdma-bond-speed-detection` --- ### 📝 Commits (1) - [`13fb806`](https://github.com/deepseek-ai/DeepEP/commit/13fb806bcb8139a3db0ffb0990569bab591b65ca) fix: correctly detect RDMA bandwidth for bonded NICs ### 📊 Changes **1 file changed** (+39 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `deep_ep/utils/envs.py` (+39 -0) </details> ### 📄 Description ## Summary Fix incorrect RDMA bandwidth detection for bonded NIC devices (e.g., `mlx5_bond_*`), which causes suboptimal SM allocation and reduced dispatch/combine throughput. ## Problem In RDMA LAG (bond) mode, `ibstat` reports only a single logical port with per-port rate: ``` CA 'mlx5_bond_7' CA type: MT4131 Number of ports: 1 Port 1: Rate: 400 ``` The existing `get_rdma_gbs()` parses this single port rate and computes `400 / 8 = 50 GB/s`. However, dual-port CX7/CX8 NICs in bond mode aggregate two 400 Gb/s physical ports, so the actual available bandwidth is `800 / 8 = 100 GB/s`. This under-reported bandwidth causes `get_theoretical_num_sms()` to recommend fewer SMs than optimal, directly reducing dispatch/combine kernel performance. ## Changes - Add `get_net_device_speed_gbs()`: maps IB device to its network interface via sysfs (`/sys/class/infiniband/<dev>/ports/<port>/gid_attrs/ndevs/0`), then reads the kernel-reported aggregated link speed from `/sys/class/net/<iface>/speed` (in Mbps). - Update `get_rdma_gbs()` with a three-level detection priority: 1. `EP_OVERRIDE_RDMA_GBS` environment variable — manual override, consistent with `EP_OVERRIDE_RDMA_SL` convention 2. sysfs aggregated speed — for bond devices (NIC name containing `bond`) 3. `ibstat` Rate parsing — original fallback for non-bond devices or when sysfs is unavailable ## Validation Verified on a multi-node cluster with CX8 dual-port NICs in RDMA LAG mode (`EP_NIC_NAME=mlx5_bond_0`): | Config | QP / SM (before) | QP / SM (after) | Dispatch GB/s (before → after)| Combine GB/s (before → after)| |--------|-------------------|-----------------|---------------|--------------| | 2-node | 65 / 12 | 65 / 22 | 49 → 58 | 53 → 79 | | 4-node | 65 / 6 | 65 / 10 | 53 → 77 | 53 → 81 | ## Compatibility - **Non-bond devices**: behavior unchanged — the sysfs branch is skipped when NIC name does not contain `bond` - **sysfs unavailable**: silently falls back to `ibstat` parsing (returns 0 on any failure) - **Manual override**: `EP_OVERRIDE_RDMA_GBS=<value_in_GBs>` takes highest priority for any device type --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:05:34 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:43 +00:00

[PR #720] feat(normal): add optional diagnostic probes for DeepEP normal mode #733