[GH-ISSUE #255] When testing on H20, an error occurs when using more than two nodes, but any two nodes can test normally without issues #190

New issue

Open

opened 2026-10-26 09:56:15 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-26 09:56:15 +00:00

Owner

Copy link

Originally created by @ZhenshengWu on GitHub (Jun 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/255

error logs

[testing] Running with BF16, without top-k (async=False, previous=False) ...ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:941084 [6] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:941087 [3] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938500:941090 [1] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:941087 [3] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938503:941086 [4] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:941084 [6] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938506:941083 [7] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938501:941088 [2] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:941086 [4] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938504:941091 [5] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938506:941083 [7] NCCL INFO misc/socket.cc:881 -> 3
[rank0]:[W625 01:10:05.149569580 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938499:941095 [0] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938500:941090 [1] NCCL INFO misc/socket.cc:881 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO comm 0x97e3d00 rank 3 nranks 24 cudaDev 3 busId c6000 - Abort COMPLETE
ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO comm 0x859f5c0 rank 6 nranks 24 cudaDev 6 busId 1a3000 - Abort COMPLETE
ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO comm 0x8fbdd10 rank 1 nranks 24 cudaDev 1 busId 7e000 - Abort COMPLETE
ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO comm 0x9198390 rank 7 nranks 24 cudaDev 7 busId 1c7000 - Abort COMPLETE
ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO comm 0x8f00270 rank 4 nranks 24 cudaDev 4 busId 109000 - Abort COMPLETE
ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO comm 0x9583580 rank 2 nranks 24 cudaDev 2 busId a2000 - Abort COMPLETE
ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO comm 0x7fe9250 rank 5 nranks 24 cudaDev 5 busId 17f000 - Abort COMPLETE
ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:64 -> 3
ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:80 -> 3
ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:829 -> 3
ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO comm 0x7c92cc0 rank 0 nranks 24 cudaDev 0 busId 8000 - Abort COMPLETE
ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO comm 0x8316580 rank 6 nranks 24 cudaDev 6 busId 1a3000 - Abort COMPLETE
ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO comm 0x8f0f540 rank 7 nranks 24 cudaDev 7 busId 1c7000 - Abort COMPLETE
ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO comm 0x955af30 rank 3 nranks 24 cudaDev 3 busId c6000 - Abort COMPLETE
ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO comm 0x8d34ec0 rank 1 nranks 24 cudaDev 1 busId 7e000 - Abort COMPLETE
ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO comm 0x8c77230 rank 4 nranks 24 cudaDev 4 busId 109000 - Abort COMPLETE
W0625 01:10:06.967000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938499 via signal SIGTERM
W0625 01:10:06.968000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938501 via signal SIGTERM
W0625 01:10:06.968000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938504 via signal SIGTERM
Traceback (most recent call last):
  File "/data1/workdata/DeepEP/tests/test_internode.py", line 252, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 6 terminated with the following error:
Traceback (most recent call last):
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/data1/workdata/DeepEP/tests/test_internode.py", line 236, in test_loop
    test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
  File "/data1/workdata/DeepEP/tests/test_internode.py", line 110, in test_main
    recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args)
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/deep_ep-1.1.0-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 311, in dispatch
    return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert,
  File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/deep_ep-1.1.0-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 421, in internode_dispatch
    recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch(
RuntimeError: Failed: Assertion error /data1/workdata/DeepEP/csrc/kernels/internode.cu:328 'false and "Unsupported RDMA ranks" 

IB : ROCE 4 * （200G * 2 ）

env

 385  export NCCL_IB_HCA=mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3
  386  export NCCL_IB_GID_INDEX=3
  387  export NCCL_IB_QPS_PER_CONNECTION=8
  388  export NCCL_IB_TIMEOUT=23
  389  export NCCL_IB_RETRY_CNT=7
  390  export UCX_TLS=tcp
  391  export UCX_NET_DEVICES=eth0
  392  export NCCL_SET_THREAD_NAME=1
  393  export MASTER_ADDR=10.0.0.114
  394  export WORLD_SIZE=3
  395  export RANK=2
  397  export NCCL_SOCKET_IFNAME=eth0

The NCCL test across the entire cluster works as expected.
In addition, the low_latency test with three nodes also runs normally.

How can i solve this issue？
code of DeepEP：the last version
nvshmem_src: 3.2.5-1

Originally created by @ZhenshengWu on GitHub (Jun 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/255 error logs ``` [testing] Running with BF16, without top-k (async=False, previous=False) ...ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:941084 [6] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:941087 [3] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938500:941090 [1] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:941087 [3] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938503:941086 [4] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:941084 [6] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938506:941083 [7] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938501:941088 [2] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:941086 [4] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938504:941091 [5] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938506:941083 [7] NCCL INFO misc/socket.cc:881 -> 3 [rank0]:[W625 01:10:05.149569580 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938499:941095 [0] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938500:941090 [1] NCCL INFO misc/socket.cc:881 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938502:942218 [3] NCCL INFO comm 0x97e3d00 rank 3 nranks 24 cudaDev 3 busId c6000 - Abort COMPLETE ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938505:942217 [6] NCCL INFO comm 0x859f5c0 rank 6 nranks 24 cudaDev 6 busId 1a3000 - Abort COMPLETE ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938500:942233 [1] NCCL INFO comm 0x8fbdd10 rank 1 nranks 24 cudaDev 1 busId 7e000 - Abort COMPLETE ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938506:942237 [7] NCCL INFO comm 0x9198390 rank 7 nranks 24 cudaDev 7 busId 1c7000 - Abort COMPLETE ali-f8xhrryi-0001:938503:942235 [4] NCCL INFO comm 0x8f00270 rank 4 nranks 24 cudaDev 4 busId 109000 - Abort COMPLETE ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938501:942245 [2] NCCL INFO comm 0x9583580 rank 2 nranks 24 cudaDev 2 busId a2000 - Abort COMPLETE ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938501:942465 [2] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938504:942252 [5] NCCL INFO comm 0x7fe9250 rank 5 nranks 24 cudaDev 5 busId 17f000 - Abort COMPLETE ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:64 -> 3 ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:80 -> 3 ali-f8xhrryi-0001:938504:942530 [5] NCCL INFO misc/socket.cc:829 -> 3 ali-f8xhrryi-0001:938499:942265 [0] NCCL INFO comm 0x7c92cc0 rank 0 nranks 24 cudaDev 0 busId 8000 - Abort COMPLETE ali-f8xhrryi-0001:938505:942315 [6] NCCL INFO comm 0x8316580 rank 6 nranks 24 cudaDev 6 busId 1a3000 - Abort COMPLETE ali-f8xhrryi-0001:938506:942449 [7] NCCL INFO comm 0x8f0f540 rank 7 nranks 24 cudaDev 7 busId 1c7000 - Abort COMPLETE ali-f8xhrryi-0001:938502:942313 [3] NCCL INFO comm 0x955af30 rank 3 nranks 24 cudaDev 3 busId c6000 - Abort COMPLETE ali-f8xhrryi-0001:938500:942446 [1] NCCL INFO comm 0x8d34ec0 rank 1 nranks 24 cudaDev 1 busId 7e000 - Abort COMPLETE ali-f8xhrryi-0001:938503:942451 [4] NCCL INFO comm 0x8c77230 rank 4 nranks 24 cudaDev 4 busId 109000 - Abort COMPLETE W0625 01:10:06.967000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938499 via signal SIGTERM W0625 01:10:06.968000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938501 via signal SIGTERM W0625 01:10:06.968000 938221 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 938504 via signal SIGTERM Traceback (most recent call last): File "/data1/workdata/DeepEP/tests/test_internode.py", line 252, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 6 terminated with the following error: Traceback (most recent call last): File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/data1/workdata/DeepEP/tests/test_internode.py", line 236, in test_loop test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group) File "/data1/workdata/DeepEP/tests/test_internode.py", line 110, in test_main recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args) File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/deep_ep-1.1.0-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 311, in dispatch return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert, File "/data1/anaconda/envs/test-deepep/lib/python3.10/site-packages/deep_ep-1.1.0-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 421, in internode_dispatch recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch( RuntimeError: Failed: Assertion error /data1/workdata/DeepEP/csrc/kernels/internode.cu:328 'false and "Unsupported RDMA ranks" ``` IB : ROCE 4 * （200G * 2 ） env ``` 385 export NCCL_IB_HCA=mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 386 export NCCL_IB_GID_INDEX=3 387 export NCCL_IB_QPS_PER_CONNECTION=8 388 export NCCL_IB_TIMEOUT=23 389 export NCCL_IB_RETRY_CNT=7 390 export UCX_TLS=tcp 391 export UCX_NET_DEVICES=eth0 392 export NCCL_SET_THREAD_NAME=1 393 export MASTER_ADDR=10.0.0.114 394 export WORLD_SIZE=3 395 export RANK=2 397 export NCCL_SOCKET_IFNAME=eth0 ``` The NCCL test across the entire cluster works as expected. In addition, the low_latency test with three nodes also runs normally. How can i solve this issue？ code of DeepEP：the last version nvshmem_src: 3.2.5-1

bsakaguchi commented 2026-10-26 09:56:17 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Jun 27, 2025):

#define SWITCH_RDMA_RANKS(case_macro) \
    switch (num_ranks / NUM_MAX_NVL_PEERS) { \
        case 2: case_macro(2); \
        case 4: case_macro(4); \
        case 8: case_macro(8); \
        case 16: case_macro(16); \
        default: EP_HOST_ASSERT(false and "Unsupported RDMA ranks"); \
    } while (false)

The main repo only supports 2/4/8/16 nodes, if you have a specific number needed. Please add a case and submit a PR, thanks!

<!-- gh-comment-id:3011793516 --> @LyricZhao commented on GitHub (Jun 27, 2025): ``` #define SWITCH_RDMA_RANKS(case_macro) \ switch (num_ranks / NUM_MAX_NVL_PEERS) { \ case 2: case_macro(2); \ case 4: case_macro(4); \ case 8: case_macro(8); \ case 16: case_macro(16); \ default: EP_HOST_ASSERT(false and "Unsupported RDMA ranks"); \ } while (false) ``` The main repo only supports 2/4/8/16 nodes, if you have a specific number needed. Please add a case and submit a PR, thanks!

bsakaguchi commented 2026-10-26 09:56:17 +00:00

Author

Owner

Copy link

@grglym commented on GitHub (Jul 29, 2025):

@ZhenshengWu 请问下，8GPU使用8个200G 和 8GPU使用4个400G(由2个200G bond)，跑deepep性能有差异吗？

<!-- gh-comment-id:3130541518 --> @grglym commented on GitHub (Jul 29, 2025): @ZhenshengWu 请问下，8GPU使用8个200G 和 8GPU使用4个400G(由2个200G bond)，跑deepep性能有差异吗？

bsakaguchi referenced this issue 2026-10-26 10:00:55 +00:00

[PR #190] [MERGED] Reduce NVSHMEM gpu memory usage and disable MNNVL. #454

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

hybrid-ep

chore/remove-clang-format-yapf

v2.5

nvDev

try-fix-cu128

mori-ep

antgroup-opt

compat-old-torch

add-linter

eager-rdma

v1.2.1

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#190

No description provided.