= num_local_experts` assert); worked with NVSHMEM <= 3.5 #418

New issue

Open

opened 2026-10-26 10:00:22 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 10:00:22 +00:00

Owner

Originally created by @laithsakka on GitHub (Oct 8, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/774

causing VLLM failure on torch 2.15 see https://github.com/pytorch/test-infra/issues/8890#issuecomment-6067431855

Low-latency mode aborts on nodes without RDMA NIC since NVSHMEM 3.6

Problem
On a single node with no usable RDMA NIC, low-latency mode worked with NVSHMEM <= 3.5 (traffic goes over NVLink). With NVSHMEM >= 3.6 it aborts on the first low_latency_dispatch:

init failed for transport: IBGDA
Assertion failed: csrc/kernels/internode_ll.cu:285, condition: ibgda_get_state()->num_rc_per_pe >= num_local_experts

PyTorch 2.15 now ships nvidia-nvshmem-cu13==3.7.2 (pytorch/pytorch#194279), so users on these nodes hit it by default (e.g. vLLM CI, pytorch/test-infra#8890).

Evidence (2x H100, same DeepEP binary, only the NVSHMEM library changed)

NVSHMEM	With RDMA NIC	Without RDMA NIC
3.4.5, 3.5.21	pass	pass
3.6.5, 3.7.0, 3.7.2, 3.8.0	pass (3.7.2)	abort

Cause
The assert never checked whether IBGDA actually started:

When IBGDA fails to start, NVSHMEM <= 3.5 leaves num_rc_per_pe at its "unset" value 0xFFFFFFFF, so the check passes by accident.
NVSHMEM >= 3.6 no longer leaves that value, so the check fails even though the NVLink path would work.

Suggested fix
If IBGDA didn't initialize and all peers are reachable over NVLink, skip the check and use NVLink. Otherwise, raise a clear host-side error instead of a device assert.

Repro
Run a 2-GPU low-latency dispatch with NVSHMEM_HCA_LIST=mlx5_99 (hides the RDMA NICs) against NVSHMEM 3.5.21 vs 3.6.5. Script available on request.

Originally created by @laithsakka on GitHub (Oct 8, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/774 causing VLLM failure on torch 2.15 see https://github.com/pytorch/test-infra/issues/8890#issuecomment-6067431855 ### Low-latency mode aborts on nodes without RDMA NIC since NVSHMEM 3.6 **Problem** On a single node with no usable RDMA NIC, low-latency mode worked with NVSHMEM <= 3.5 (traffic goes over NVLink). With NVSHMEM >= 3.6 it aborts on the first `low_latency_dispatch`: init failed for transport: IBGDA Assertion failed: csrc/kernels/internode_ll.cu:285, condition: ibgda_get_state()->num_rc_per_pe >= num_local_experts PyTorch 2.15 now ships `nvidia-nvshmem-cu13==3.7.2` (pytorch/pytorch#194279), so users on these nodes hit it by default (e.g. vLLM CI, pytorch/test-infra#8890). **Evidence** (2x H100, same DeepEP binary, only the NVSHMEM library changed) | NVSHMEM | With RDMA NIC | Without RDMA NIC | |---|---|---| | 3.4.5, 3.5.21 | pass | pass | | 3.6.5, 3.7.0, 3.7.2, 3.8.0 | pass (3.7.2) | **abort** | **Cause** The assert never checked whether IBGDA actually started: - When IBGDA fails to start, NVSHMEM <= 3.5 leaves `num_rc_per_pe` at its "unset" value `0xFFFFFFFF`, so the check passes by accident. - NVSHMEM >= 3.6 no longer leaves that value, so the check fails even though the NVLink path would work. **Suggested fix** If IBGDA didn't initialize and all peers are reachable over NVLink, skip the check and use NVLink. Otherwise, raise a clear host-side error instead of a device assert. **Repro** Run a 2-GPU low-latency dispatch with `NVSHMEM_HCA_LIST=mlx5_99` (hides the RDMA NICs) against NVSHMEM 3.5.21 vs 3.6.5. Script available on request.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#418

No description provided.

Rows
Columns

[GH-ISSUE #774] Low-latency mode aborts on nodes without a usable RDMA NIC with NVSHMEM >= 3.6 (num_rc_per_pe >= num_local_experts assert); worked with NVSHMEM <= 3.5 #418

Low-latency mode aborts on nodes without RDMA NIC since NVSHMEM 3.6

[GH-ISSUE #774] Low-latency mode aborts on nodes without a usable RDMA NIC with NVSHMEM >= 3.6 (`num_rc_per_pe >= num_local_experts` assert); worked with NVSHMEM <= 3.5 #418