[GH-ISSUE #294] [BUG]DeepEP internode timeout #210

New issue

Open

opened 2026-10-26 09:56:40 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:56:40 +00:00

Owner

Originally created by @jimmy-evo on GitHub (Jul 10, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/294

4node H100

As shown in image, it stuck here then got "check timeout failed" Error.
I notice in bottom-left panel, there is a nvshmem log : in nvshmem_finalize

num_token 4096 is ok, but it timeout when num_token=8192/16384

set -ex
export NVSHMEM_DEBUG=INFO
export NVSHMEM_IB_ENABLE_IBGDA=true
export NCCL_DEBUG=TRACE
export MASTER_ADDR=10.137.1.127
export MASTER_PORT=5003
export WORLD_SIZE=4
export RANK=0
python test_internode.py

code commit: 9ec061204e
I can't use later commit because June12's commit 21efbe9b48 will stuck

ibv_devinfo

hca_id: mlx5_ib0
transport:                      InfiniBand (0)
fw_ver:                         28.43.1014
node_guid:                      58a2:e103:00e4:2166
sys_image_guid:                 58a2:e103:00e4:2166
vendor_id:                      0x02c9
vendor_part_id:                 4129
hw_ver:                         0x0
board_id:                       MT_0000000800
phys_port_cnt:                  1
port:   1
state:                  PORT_ACTIVE (4)
max_mtu:                4096 (5)
active_mtu:             4096 (5)
sm_lid:                 1
port_lid:               727
port_lmc:               0x00
link_layer:             InfiniBand

Originally created by @jimmy-evo on GitHub (Jul 10, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/294 4node H100 <img width="2328" height="1306" alt="Image" src="https://github.com/user-attachments/assets/fbafa4cc-9003-4ff6-97f1-8b0329384019" /> As shown in image, it stuck here then got "check timeout failed" Error. *I notice in bottom-left panel, there is a nvshmem log : in nvshmem_finalize* **num_token 4096 is ok**, but it timeout when num_token=8192/16384 ``` set -ex export NVSHMEM_DEBUG=INFO export NVSHMEM_IB_ENABLE_IBGDA=true export NCCL_DEBUG=TRACE export MASTER_ADDR=10.137.1.127 export MASTER_PORT=5003 export WORLD_SIZE=4 export RANK=0 python test_internode.py ``` code commit: 9ec061204e6763c12a9dd9f4cc5ca3b6c868b552 I can't use later commit because June12's commit 21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942 will stuck ibv_devinfo ``` hca_id: mlx5_ib0 transport: InfiniBand (0) fw_ver: 28.43.1014 node_guid: 58a2:e103:00e4:2166 sys_image_guid: 58a2:e103:00e4:2166 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000800 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 727 port_lmc: 0x00 link_layer: InfiniBand ```

bsakaguchi commented

2026-10-26 09:56:42 +00:00

Author

Owner

@sphish commented on GitHub (Jul 15, 2025):

https://github.com/deepseek-ai/DeepEP/pull/292 Can you set explicitly_destroy and rerun the test? I suspect there might be an assert error.

@sphish commented on GitHub (Jul 15, 2025): https://github.com/deepseek-ai/DeepEP/pull/292 Can you set `explicitly_destroy` and rerun the test? I suspect there might be an assert error.

bsakaguchi referenced this issue

2026-10-26 10:01:06 +00:00

[PR #210] [MERGED] Super tiny remove extra semicolon #467