[GH-ISSUE #553] DeepEP timeout when using NVSHMEM v3.5.19 #340

New issue

Closed

opened 2026-10-26 09:58:50 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 09:58:50 +00:00

Owner

Originally created by @aahouzi on GitHub (Jan 2, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/553

Type of issue

I upgraded to NVSHMEM 3.5.19 to resolve a specific RoCE issue I was facing on my RoCE cluster, for which the fix was commited 2 weeks ago. However, when running the internode kernels with this release, I am consistently encountering a runtime timeout. This issue is reproducible on two separate H100 RoCE clusters:

Traceback (most recent call last):
  File "/home/dpsk_a2a/DeepEP/tests/test_internode.py", line 339, in test_loop
    ref_hash += test_main(args, i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group,
  File "/home/dpsk_a2a/DeepEP/tests/test_internode.py", line 138, in test_main
    recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(
  File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.2.1+309785a-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 377, in dispatch
    return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank,
  File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.2.1+309785a-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 490, in internode_dispatch
    recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch(
RuntimeError: DeepEP error: timeout (dispatch CPU)

I tried running shmem_put_bw perftest on my cluster with IBGDA enabled, and it runs correctly with expected performance:

#shmem_put_bw_uni
size(B)     scope     BW (GB/sec)     
1024        None      0.049474        
2048        None      0.091586        
4096        None      0.186671        
8192        None      0.376194        
16384       None      0.720315        
32768       None      1.548231        
65536       None      3.125763        
131072      None      6.272588        
262144      None      11.936471       
524288      None      24.526947       
1048576     None      41.753315       
2097152     None      44.419140       
4194304     None      45.893559       
8388608     None      46.719658       
16777216    None      47.129551

I saw in the NVSHMEM v3.5.19 release notes that there were significant architectural changes, and I suspect this may have impacted DeepEP somehow. Any idea what's the gap here ? Thanks

Versions

NVSHMEM 3.5.19
gdrcopy v2.5.1
DeepEP 309785a
RoCE H100 cluster

Originally created by @aahouzi on GitHub (Jan 2, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/553 ## Type of issue - I upgraded to NVSHMEM 3.5.19 to resolve a specific [RoCE issue](https://github.com/NVIDIA/nvshmem/issues/21) I was facing on my RoCE cluster, for which the fix was commited 2 weeks ago. However, when running the internode kernels with this release, I am consistently encountering a runtime timeout. This issue is reproducible on two separate H100 RoCE clusters: ``` Traceback (most recent call last): File "/home/dpsk_a2a/DeepEP/tests/test_internode.py", line 339, in test_loop ref_hash += test_main(args, i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group, File "/home/dpsk_a2a/DeepEP/tests/test_internode.py", line 138, in test_main recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch( File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.2.1+309785a-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 377, in dispatch return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.2.1+309785a-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 490, in internode_dispatch recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch( RuntimeError: DeepEP error: timeout (dispatch CPU) ``` - I tried running shmem_put_bw perftest on my cluster with IBGDA enabled, and it runs correctly with expected performance: ``` #shmem_put_bw_uni size(B) scope BW (GB/sec) 1024 None 0.049474 2048 None 0.091586 4096 None 0.186671 8192 None 0.376194 16384 None 0.720315 32768 None 1.548231 65536 None 3.125763 131072 None 6.272588 262144 None 11.936471 524288 None 24.526947 1048576 None 41.753315 2097152 None 44.419140 4194304 None 45.893559 8388608 None 46.719658 16777216 None 47.129551 ``` - I saw in the [NVSHMEM v3.5.19 release notes](https://github.com/NVIDIA/nvshmem/releases/tag/v3.5.19-1) that there were significant architectural changes, and I suspect this may have impacted DeepEP somehow. Any idea what's the gap here ? Thanks ## Versions - NVSHMEM 3.5.19 - gdrcopy v2.5.1 - DeepEP 309785a - RoCE H100 cluster

bsakaguchi closed this issue

2026-10-26 09:58:51 +00:00

bsakaguchi commented

2026-10-26 09:58:51 +00:00

Author

Owner

@skeyyk commented on GitHub (Jan 13, 2026):

Is there a recommended way to upgrade to NVSHMEM 3.5.19?

@skeyyk commented on GitHub (Jan 13, 2026): Is there a recommended way to upgrade to NVSHMEM 3.5.19?

bsakaguchi commented

2026-10-26 09:58:51 +00:00

Author

Owner

@seth-howell commented on GitHub (Jan 16, 2026):

There is a real compatibility issue between 3.5.19 and the current main of DeepEP which will lead to hangs. Please see PR #564 to see if it addresses your specific case.

@seth-howell commented on GitHub (Jan 16, 2026): There is a real compatibility issue between 3.5.19 and the current main of DeepEP which will lead to hangs. Please see PR #564 to see if it addresses your specific case.

bsakaguchi commented

2026-10-26 09:58:51 +00:00

Author

Owner

@aahouzi commented on GitHub (Jan 23, 2026):

Thanks @seth-howell for the fix, it's running on my cluster :)

@aahouzi commented on GitHub (Jan 23, 2026): Thanks @seth-howell for the fix, it's running on my cluster :)

bsakaguchi referenced this issue

2026-10-27 21:03:39 +00:00

[PR #340] feat(buffer): implement dynamic buffer resizing for simplicity #525