[GH-ISSUE #759] [Bug] low-latency (internode_ll) unusable on non-Mellanox NICs: NVSHMEM IBGDA transport is mlx5-only (reproduced on Alibaba eRDMA) #416

New issue

Open

opened 2026-10-26 10:00:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:20 +00:00

Owner

Originally created by @luoyuctl on GitHub (Sep 21, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/759

Summary

DeepEP's low-latency / internode path cannot run on RDMA NICs that are not Mellanox mlx5. On Alibaba Cloud eRDMA (ERI) the failure surfaces as a single dlopen warning that reads like a benign capability downgrade, but it is fatal for internode_ll. Reporting so it can at least be documented up front.

Environment

Item	Value
GPU	2 nodes × 8× NVIDIA H20 96GB (SM90)
RDMA	Alibaba Cloud eRDMA (`aliyun/erdma` device plugin) → `erdma_0`, `erdma_1`
Driver / CUDA	595.58.03 / CUDA 13.0
NVSHMEM	3.4.5 (cu13)
deep_ep	2.0.0

What works

ibv_devinfo on erdma_0: PORT_ACTIVE, link_layer: Ethernet
ibv_open_device(erdma_0) → OK
Standard NCCL over RoCE works: we run 16-rank cross-node EP with NCCL collectives, including NET/IB/GDRDMA, without issue

What fails — every IBGDA-dependent path

NVSHMEM: Unable to dlopen libmlx5dv. Disabling directNIC features.

mlx5dv_open_device(erdma_0)  -> FAIL (not an mlx5 device)
/sys/class/infiniband/erdma_0/device/driver -> erdma      # not mlx5_core

We also checked whether any alternate transport could supply IBGDA symbols instead — all three export 0:

ibrc        -> 0 IBGDA symbols
libfabric   -> 0 IBGDA symbols
ucx         -> 0 IBGDA symbols

Reading NVSHMEM 3.4.5's IBGDA transport source (transport_mlx5_common.cpp), it is mlx5-specific throughout: mlx5dv_init_obj, mlx5dv_create_qp, mlx5dv_devx_alloc_uar, mlx5dv_devx_obj_modify. There is no generic verbs fallback that we could find.

Impact

deepep_low_latency (and test_low_latency.py / test_internode.py) cannot be enabled on this fleet. We are forced onto an NCCL all-gather/reduce-scatter all2all backend, which is the throughput bottleneck on our deployment.

Why this seems worth a documentation change

The log line Disabling directNIC features reads as a degraded-but-working state. It is not — it is a hard capability wall for the low-latency path. We spent substantial time assuming it was noise.
eRDMA / ERI is the default RDMA path on Alibaba Cloud GPU instances, so this is a large fleet rather than an exotic one.
docs/legacy.md describes IB as fully validated and RoCE as theoretically supported, without noting the mlx5 dependency — which is precisely the distinction that bites here: RoCEv2 on mlx5 works, Ethernet RDMA on a non-mlx5 driver does not.

Asks

Document the mlx5 / IBGDA requirement explicitly, ideally alongside the existing NVSHMEM requirement.
Make the failure explicit and loud rather than a dlopen warning.
Confirm intent: is a non-mlx5 IBGDA path planned, or is the intended route for such fleets a different backend (NIXL / libfabric)? Cross-ref #659, which asks the adjacent question for AWS EFA.

Happy to provide full NVSHMEM init logs, ibv_devinfo output, and the symbol-count checks.

Originally created by @luoyuctl on GitHub (Sep 21, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/759 ### Summary DeepEP's low-latency / internode path cannot run on RDMA NICs that are not Mellanox mlx5. On Alibaba Cloud eRDMA (ERI) the failure surfaces as a single `dlopen` warning that reads like a benign capability downgrade, but it is fatal for `internode_ll`. Reporting so it can at least be documented up front. ### Environment | Item | Value | |---|---| | GPU | 2 nodes × 8× NVIDIA H20 96GB (SM90) | | RDMA | Alibaba Cloud eRDMA (`aliyun/erdma` device plugin) → `erdma_0`, `erdma_1` | | Driver / CUDA | 595.58.03 / CUDA 13.0 | | NVSHMEM | 3.4.5 (cu13) | | deep_ep | 2.0.0 | ### What works - `ibv_devinfo` on `erdma_0`: `PORT_ACTIVE`, `link_layer: Ethernet` - `ibv_open_device(erdma_0)` → OK - Standard NCCL over RoCE works: we run 16-rank cross-node EP with NCCL collectives, including `NET/IB/GDRDMA`, without issue ### What fails — every IBGDA-dependent path ``` NVSHMEM: Unable to dlopen libmlx5dv. Disabling directNIC features. ``` ``` mlx5dv_open_device(erdma_0) -> FAIL (not an mlx5 device) /sys/class/infiniband/erdma_0/device/driver -> erdma # not mlx5_core ``` We also checked whether any alternate transport could supply IBGDA symbols instead — all three export **0**: ``` ibrc -> 0 IBGDA symbols libfabric -> 0 IBGDA symbols ucx -> 0 IBGDA symbols ``` Reading NVSHMEM 3.4.5's IBGDA transport source (`transport_mlx5_common.cpp`), it is mlx5-specific throughout: `mlx5dv_init_obj`, `mlx5dv_create_qp`, `mlx5dv_devx_alloc_uar`, `mlx5dv_devx_obj_modify`. There is no generic verbs fallback that we could find. ### Impact `deepep_low_latency` (and `test_low_latency.py` / `test_internode.py`) cannot be enabled on this fleet. We are forced onto an NCCL all-gather/reduce-scatter all2all backend, which is the throughput bottleneck on our deployment. ### Why this seems worth a documentation change 1. The log line `Disabling directNIC features` reads as a degraded-but-working state. It is not — it is a hard capability wall for the low-latency path. We spent substantial time assuming it was noise. 2. eRDMA / ERI is the default RDMA path on Alibaba Cloud GPU instances, so this is a large fleet rather than an exotic one. 3. `docs/legacy.md` describes IB as fully validated and RoCE as theoretically supported, without noting the mlx5 dependency — which is precisely the distinction that bites here: RoCEv2 *on mlx5* works, Ethernet RDMA on a non-mlx5 driver does not. ### Asks 1. Document the mlx5 / IBGDA requirement explicitly, ideally alongside the existing NVSHMEM requirement. 2. Make the failure explicit and loud rather than a `dlopen` warning. 3. Confirm intent: is a non-mlx5 IBGDA path planned, or is the intended route for such fleets a different backend (NIXL / libfabric)? Cross-ref #659, which asks the adjacent question for AWS EFA. Happy to provide full NVSHMEM init logs, `ibv_devinfo` output, and the symbol-count checks.

bsakaguchi referenced this issue

2026-10-27 21:04:02 +00:00

[PR #416] Support per tensor transfer #560