[GH-ISSUE #582] AssertionError: No NVLink connection between GPU 0 and GPU 0 #358

New issue

Open

opened 2026-10-26 09:59:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:59:07 +00:00

Owner

Originally created by @pramodk on GitHub (Feb 17, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/582

While running some MoE training examples with Megatron-LM, we have seen below errors:

[rank1]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True
[rank2]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True

The topology / nvidia-smi output on the compute node:

 nodes:
	GPU0	GPU1	GPU2	GPU3	
GPU0	 X 	NV4	NV4	NV4	
GPU1	NV4	 X 	NV4	NV4	
GPU2	NV4	NV4	 X 	NV4	
GPU3	NV4	NV4	NV4	 X

Looking at the implementation here, I wonder if we end up in a situation where device ID 0 is duplicated across ranks and the NVLink check end up comparing the same GPU against itself (i.e. this check is not sufficient). By the way, here is the older implementation.

cc: @LyricZhao

Originally created by @pramodk on GitHub (Feb 17, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/582 While running some MoE training examples with Megatron-LM, we have seen below errors: ``` [rank1]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True [rank2]: AssertionError: No NVLink connection between GPU 0 and GPU 0, but allow_nvlink_for_normal_mode=True ``` The topology / `nvidia-smi` output on the compute node: ``` nodes: GPU0 GPU1 GPU2 GPU3 GPU0 X NV4 NV4 NV4 GPU1 NV4 X NV4 NV4 GPU2 NV4 NV4 X NV4 GPU3 NV4 NV4 NV4 X ``` Looking at the [implementation here](https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L87), I wonder if we end up in a situation where device ID 0 is duplicated across ranks and the NVLink check end up comparing the same GPU against itself (i.e. [this check](https://github.com/deepseek-ai/DeepEP/blame/567632dd59810d77b3cc05553df953cc0f779799/deep_ep/utils.py#L94) is not sufficient). By the way, here is [the older implementation](https://github.com/deepseek-ai/DeepEP/commit/9ec061204e6763c12a9dd9f4cc5ca3b6c868b552). cc: @LyricZhao

bsakaguchi commented

2026-10-26 09:59:08 +00:00

Author

Owner

@pramodk commented on GitHub (Feb 25, 2026):

Hello @LyricZhao, do you have any insight / suggestion about this? Thanks!

@pramodk commented on GitHub (Feb 25, 2026): Hello @LyricZhao, do you have any insight / suggestion about this? Thanks!

bsakaguchi referenced this issue

2026-10-27 21:04:02 +00:00

[PR #422] [MERGED] Make dtype of topk_idx configurable #559