[GH-ISSUE #628] ElasticBuffer asserts NCCL_GIN_TYPE_NONE on host with MLNX_OFED 24.10 (libmlx5 < MLX5_1.25) #376

New issue

Open

opened 2026-10-26 09:59:43 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:59:43 +00:00

Owner

Originally created by @xiaoyaojun666 on GitHub (May 9, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/628

DeepEP version: 2.0.0 (commit <你那 commit>)
NCCL: 2.30.4+cuda13.2
GPU: B300 SXM6, driver 590.48.01
NIC: ConnectX-7, fw_ver 40.46.5500
Host OFED: MLNX_OFED_LINUX-24.10-3.2.5.0 (libmlx5 max symbol MLX5_1.17)

NCCL_DEBUG=INFO shows:
dlvsym failed on mlx5dv_reg_dmabuf_mr ... version MLX5_1.25
dlvsym failed on mlx5dv_get_data_direct_sysfs_path ... version MLX5_1.25
Assigned GIN plugin GIN_IB_GDAKI to comm
GPU Direct RDMA Enabled for HCA 0..7
但 ncclCommQueryProperties() 返回 props.ginType = NCCL_GIN_TYPE_NONE,
ElasticBuffer ctor 在 csrc/kernels/backend/nccl.cu:87 断言失败。

Tried (all fail same):

allow_hybrid_mode={True, False}
NCCL_GIN_ENABLE=1
intra-node 8-rank NVLink-only test (no inter-node fabric involved)

Question:

Is upgrading host MLNX_OFED to 25.x the only path? Or is there an env / NCCL flag to skip the MLX5_1.25 dmabuf
path?
Could README mention minimum OFED version explicitly?

Originally created by @xiaoyaojun666 on GitHub (May 9, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/628 DeepEP version: 2.0.0 (commit <你那 commit>) NCCL: 2.30.4+cuda13.2 GPU: B300 SXM6, driver 590.48.01 NIC: ConnectX-7, fw_ver 40.46.5500 Host OFED: MLNX_OFED_LINUX-24.10-3.2.5.0 (libmlx5 max symbol MLX5_1.17) NCCL_DEBUG=INFO shows: dlvsym failed on mlx5dv_reg_dmabuf_mr ... version MLX5_1.25 dlvsym failed on mlx5dv_get_data_direct_sysfs_path ... version MLX5_1.25 Assigned GIN plugin GIN_IB_GDAKI to comm GPU Direct RDMA Enabled for HCA 0..7 但 ncclCommQueryProperties() 返回 props.ginType = NCCL_GIN_TYPE_NONE, ElasticBuffer ctor 在 csrc/kernels/backend/nccl.cu:87 断言失败。 Tried (all fail same): - allow_hybrid_mode={True, False} - NCCL_GIN_ENABLE=1 - intra-node 8-rank NVLink-only test (no inter-node fabric involved) Question: 1. Is upgrading host MLNX_OFED to 25.x the only path? Or is there an env / NCCL flag to skip the MLX5_1.25 dmabuf path? 2. Could README mention minimum OFED version explicitly?

bsakaguchi commented

2026-10-26 09:59:46 +00:00

Author

Owner

@jiangzhuti commented on GitHub (May 25, 2026):

试试 export NCCL_IB_MERGE_NICS=0

@jiangzhuti commented on GitHub (May 25, 2026): 试试 export NCCL_IB_MERGE_NICS=0

bsakaguchi commented

2026-10-26 09:59:46 +00:00

Author

Owner

@Functionhx commented on GitHub (Jul 9, 2026):

I ran into this too. Instead of asserting, I made the GIN check conditional: if GIN isn't available, it prints a debug message and falls back to non-GIN path. Also added ncclAllReduce consensus so heterogeneous ranks don't mismatch. Want a PR?

@Functionhx commented on GitHub (Jul 9, 2026): I ran into this too. Instead of asserting, I made the GIN check conditional: if GIN isn't available, it prints a debug message and falls back to non-GIN path. Also added ncclAllReduce consensus so heterogeneous ranks don't mismatch. Want a PR?

bsakaguchi referenced this issue

2026-10-27 21:03:49 +00:00

[PR #376] [MERGED] Tiny fix comments about package format #541