deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #386] Error after changing NIC name from mlx5_x #263

New issue
Closed
opened 2026-10-26 09:57:36 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:57:36 +00:00
Owner
Copy link

Originally created by @aahouzi on GitHub (Sep 1, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/386

Type of issue

  • Changing the NIC names from the default mlx5_x to roce_railx names, results in this error log being printed out for every rank:
[config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8
[config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8
[layout] Kernel performance: 0.039 ms

[layout] Kernel performance: 0.039 ms

[testing] Running with BF16, without top-k (async=False, previous=False) ...NVSHMEM v3.3.9
WARN: device roce_rail1 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail2 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail3 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail4 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail5 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail6 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail7 is not enumerated as an mlx5 device. Skipping...

WARN: device roce_rail8 is not enumerated as an mlx5 device. Skipping...

Global rank: 3, num_recv_tokens: -1, num_rdma_recv_tokens: -1
moe_recv_expert_counter[0]: -1
WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.
  • I tried setting these env variables, but this still didn't solve the issue:
export NVSHMEM_ENABLE_NIC_PE_MAPPING=1
export NVSHMEM_HCA_LIST=roce_rail1:1,roce_rail2:1,roce_rail3:1,roce_rail4:1,roce_rail5:1,roce_rail6:1,roce_rail7:1,roce_rail8:1
export NCCL_IB_HCA=roce_rail1:1,roce_rail2:1,roce_rail3:1,roce_rail4:1,roce_rail5:1,roce_rail6:1,roce_rail7:1,roce_rail8:1
  • Can you guys provide some guidance on how to solve this issue ? Thanks :)
Originally created by @aahouzi on GitHub (Sep 1, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/386 ## Type of issue - Changing the NIC names from the default mlx5_x to roce_railx names, results in this error log being printed out for every rank: ``` [config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8 [config] num_tokens=4096, hidden=7168, num_topk_groups=2, num_topk=8 [layout] Kernel performance: 0.039 ms [layout] Kernel performance: 0.039 ms [testing] Running with BF16, without top-k (async=False, previous=False) ...NVSHMEM v3.3.9 WARN: device roce_rail1 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail2 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail3 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail4 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail5 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail6 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail7 is not enumerated as an mlx5 device. Skipping... WARN: device roce_rail8 is not enumerated as an mlx5 device. Skipping... Global rank: 3, num_recv_tokens: -1, num_rdma_recv_tokens: -1 moe_recv_expert_counter[0]: -1 WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources. ``` - I tried setting these env variables, but this still didn't solve the issue: ``` export NVSHMEM_ENABLE_NIC_PE_MAPPING=1 export NVSHMEM_HCA_LIST=roce_rail1:1,roce_rail2:1,roce_rail3:1,roce_rail4:1,roce_rail5:1,roce_rail6:1,roce_rail7:1,roce_rail8:1 export NCCL_IB_HCA=roce_rail1:1,roce_rail2:1,roce_rail3:1,roce_rail4:1,roce_rail5:1,roce_rail6:1,roce_rail7:1,roce_rail8:1 ``` - Can you guys provide some guidance on how to solve this issue ? Thanks :)
bsakaguchi commented 2026-10-26 09:57:38 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Sep 1, 2025):

I think this is an NVSHMEM NIC selection issue. You can report it to the NVSHMEM team, or you can make a simple change to the NVSHMEM code yourself to avoid skipping these NICs.

<!-- gh-comment-id:3241559672 --> @sphish commented on GitHub (Sep 1, 2025): I think this is an NVSHMEM NIC selection issue. You can report it to the NVSHMEM team, or you can make a simple change to the NVSHMEM code yourself to avoid skipping these NICs.
bsakaguchi commented 2026-10-26 09:57:38 +00:00
Author
Owner
Copy link

@aahouzi commented on GitHub (Sep 5, 2025):

Thanks @sphish. Indeed, it's a NIC selection issue and will be fixed in the next releases.

<!-- gh-comment-id:3257489457 --> @aahouzi commented on GitHub (Sep 5, 2025): Thanks @sphish. Indeed, it's a NIC selection issue and will be fixed in the next releases.
bsakaguchi commented 2026-10-26 09:57:38 +00:00
Author
Owner
Copy link

@qijiaxing commented on GitHub (Oct 23, 2025):

Met the same issue. Need to install NVSHMEM 3.4.5 with environment variable NVSHMEM_HCA_PREFIX to enable IB transports on systems which name their HCA devices in a non-standard way (for example, ipb* instead of mlx5*).
see here.

<!-- gh-comment-id:3436300441 --> @qijiaxing commented on GitHub (Oct 23, 2025): Met the same issue. Need to install NVSHMEM 3.4.5 with environment variable `NVSHMEM_HCA_PREFIX `to enable IB transports on systems which name their HCA devices in a non-standard way (for example, `ipb*` instead of `mlx5*`). see [here](https://github.com/NVIDIA/nvshmem/releases/tag/v3.4.5-0).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#263
No description provided.