mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #417] test_internode bug on h20 with 2nodes #281
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#281
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @junchenglai-mthreads on GitHub (Sep 18, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/417
root@mccx:/sgl-workspace/DeepEP# MASTER_ADDR=xxxxx MASTER_PORT=29500 WORLD_SIZE=2 RANK=0 python tests/test_internode.py
W0918 10:57:42.299000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1683 via signal SIGTERM
W0918 10:57:42.300000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1684 via signal SIGTERM
W0918 10:57:42.301000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1685 via signal SIGTERM
W0918 10:57:42.301000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1686 via signal SIGTERM
W0918 10:57:42.302000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1687 via signal SIGTERM
W0918 10:57:42.303000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1688 via signal SIGTERM
W0918 10:57:42.303000 1617 torch/multiprocessing/spawn.py:169] Terminating process 1689 via signal SIGTERM
Traceback (most recent call last):
File "/sgl-workspace/DeepEP/tests/test_internode.py", line 247, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/sgl-workspace/DeepEP/tests/test_internode.py", line 221, in test_loop
rank, num_ranks, group = init_dist(local_rank, num_local_ranks)
File "/sgl-workspace/DeepEP/tests/utils.py", line 17, in init_dist
dist.init_process_group(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
func_return = func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1714, in init_process_group
store, rank, world_size = next(rendezvous_iterator)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 226, in _tcp_rendezvous_handler
store = _create_c10d_store(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 194, in _create_c10d_store
return TCPStore(
RuntimeError: The server socket has failed to listen on any local network address. port: 29500, useIpv6: 0, code: -98, name: EADDRINUSE, message: address already in use
root@mccx:/sgl-workspace/DeepEP# MASTER_ADDR=10.10.142.201 MASTER_PORT=29503 WORLD_SIZE=2 RANK=0 python tests/test_internode.py
mccx:2259:2259 [0] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2259:2259 [0] NCCL INFO cudaDriverVersion 12080
NCCL version 2.21.5+cuda12.4
mccx:2259:2259 [0] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7fd79ba01400
mccx:2259:2259 [0] NCCL INFO Comm config Blocking set to 1
mccx:2266:2266 [7] NCCL INFO cudaDriverVersion 12080
mccx:2266:2266 [7] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2263:2263 [4] NCCL INFO cudaDriverVersion 12080
mccx:2263:2263 [4] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2262:2262 [3] NCCL INFO cudaDriverVersion 12080
mccx:2262:2262 [3] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2264:2264 [5] NCCL INFO cudaDriverVersion 12080
mccx:2264:2264 [5] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2261:2261 [2] NCCL INFO cudaDriverVersion 12080
mccx:2261:2261 [2] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2260:2260 [1] NCCL INFO cudaDriverVersion 12080
mccx:2265:2265 [6] NCCL INFO cudaDriverVersion 12080
mccx:2260:2260 [1] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2265:2265 [6] NCCL INFO Bootstrap : Using ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2263:2263 [4] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f3497a01400
mccx:2263:2263 [4] NCCL INFO Comm config Blocking set to 1
mccx:2266:2266 [7] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f1c3fa01400
mccx:2266:2266 [7] NCCL INFO Comm config Blocking set to 1
mccx:2264:2264 [5] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f2d53a01400
mccx:2264:2264 [5] NCCL INFO Comm config Blocking set to 1
mccx:2261:2261 [2] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f2ec7a01400
mccx:2261:2261 [2] NCCL INFO Comm config Blocking set to 1
mccx:2265:2265 [6] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f3d9ba01400
mccx:2265:2265 [6] NCCL INFO Comm config Blocking set to 1
mccx:2262:2262 [3] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f4f73a01400
mccx:2262:2262 [3] NCCL INFO Comm config Blocking set to 1
mccx:2260:2260 [1] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f896ba01400
mccx:2260:2260 [1] NCCL INFO Comm config Blocking set to 1
mccx:2259:2822 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2259:2822 [0] NCCL INFO P2P plugin IBext_v8
mccx:2262:2834 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2262:2834 [3] NCCL INFO P2P plugin IBext_v8
mccx:2266:2826 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2266:2826 [7] NCCL INFO P2P plugin IBext_v8
mccx:2263:2824 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2263:2824 [4] NCCL INFO P2P plugin IBext_v8
mccx:2264:2828 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2264:2828 [5] NCCL INFO P2P plugin IBext_v8
mccx:2261:2830 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2261:2830 [2] NCCL INFO P2P plugin IBext_v8
mccx:2265:2832 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2265:2832 [6] NCCL INFO P2P plugin IBext_v8
mccx:2260:2836 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:2260:2836 [1] NCCL INFO P2P plugin IBext_v8
mccx:2259:2822 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2259:2822 [0] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2259:2822 [0] NCCL INFO Using non-device net plugin version 0
mccx:2259:2822 [0] NCCL INFO Using network IBext_v8
mccx:2263:2824 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2263:2824 [4] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2263:2824 [4] NCCL INFO Using non-device net plugin version 0
mccx:2263:2824 [4] NCCL INFO Using network IBext_v8
mccx:2265:2832 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2265:2832 [6] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2265:2832 [6] NCCL INFO Using non-device net plugin version 0
mccx:2265:2832 [6] NCCL INFO Using network IBext_v8
mccx:2262:2834 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2262:2834 [3] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2262:2834 [3] NCCL INFO Using non-device net plugin version 0
mccx:2262:2834 [3] NCCL INFO Using network IBext_v8
mccx:2264:2828 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2264:2828 [5] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2264:2828 [5] NCCL INFO Using non-device net plugin version 0
mccx:2264:2828 [5] NCCL INFO Using network IBext_v8
mccx:2266:2826 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2266:2826 [7] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2266:2826 [7] NCCL INFO Using non-device net plugin version 0
mccx:2266:2826 [7] NCCL INFO Using network IBext_v8
mccx:2260:2836 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2260:2836 [1] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2260:2836 [1] NCCL INFO Using non-device net plugin version 0
mccx:2260:2836 [1] NCCL INFO Using network IBext_v8
mccx:2261:2830 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB ibs1130:fe80::ba3f:d203:56:876a%ibs1130<0>
mccx:2261:2830 [2] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:2261:2830 [2] NCCL INFO Using non-device net plugin version 0
mccx:2261:2830 [2] NCCL INFO Using network IBext_v8
mccx:2259:2822 [0] NCCL INFO DMA-BUF is available on GPU device 0
mccx:2263:2824 [4] NCCL INFO DMA-BUF is available on GPU device 4
mccx:2265:2832 [6] NCCL INFO DMA-BUF is available on GPU device 6
mccx:2262:2834 [3] NCCL INFO DMA-BUF is available on GPU device 3
mccx:2264:2828 [5] NCCL INFO DMA-BUF is available on GPU device 5
mccx:2266:2826 [7] NCCL INFO DMA-BUF is available on GPU device 7
mccx:2260:2836 [1] NCCL INFO DMA-BUF is available on GPU device 1
mccx:2261:2830 [2] NCCL INFO DMA-BUF is available on GPU device 2
mccx:2266:2826 [7] misc/socket.cc:484 NCCL WARN socketStartConnect: Connect to fe80::ba3f:d203:56:871a%4<40625> failed : Network is unreachable
mccx:2266:2826 [7] NCCL INFO misc/socket.cc:567 -> 2
mccx:2266:2826 [7] NCCL INFO misc/socket.cc:621 -> 2
mccx:2266:2826 [7] NCCL INFO bootstrap.cc:297 -> 2
mccx:2266:2826 [7] NCCL INFO init.cc:1534 -> 2
mccx:2266:2826 [7] NCCL INFO group.cc:64 -> 2 [Async thread]
mccx:2259:2821 [0] misc/socket.cc:484 NCCL WARN socketStartConnect: Connect to fe80::ba3f:d203:56:871a%4<33631> failed : Network is unreachable
mccx:2259:2821 [0] NCCL INFO misc/socket.cc:567 -> 2
mccx:2259:2821 [0] NCCL INFO misc/socket.cc:621 -> 2
mccx:2259:2821 [0] NCCL INFO bootstrap.cc:163 -> 2
mccx:2266:2266 [7] NCCL INFO group.cc:418 -> 2
mccx:2266:2266 [7] NCCL INFO init.cc:1929 -> 2
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /sgl-workspace/DeepEP/csrc/deep_ep.cpp:89 'an illegal memory access was encountered'
W0918 10:58:41.800000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2259 via signal SIGTERM
W0918 10:58:41.801000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2260 via signal SIGTERM
W0918 10:58:41.802000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2261 via signal SIGTERM
W0918 10:58:41.803000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2262 via signal SIGTERM
W0918 10:58:41.806000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2263 via signal SIGTERM
W0918 10:58:41.807000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2264 via signal SIGTERM
W0918 10:58:41.808000 2194 torch/multiprocessing/spawn.py:169] Terminating process 2265 via signal SIGTERM
Traceback (most recent call last):
File "/sgl-workspace/DeepEP/tests/test_internode.py", line 247, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 7 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/sgl-workspace/DeepEP/tests/test_internode.py", line 229, in test_loop
buffer = deep_ep.Buffer(group, int(1e9), int(1e9), low_latency_mode=test_ll_compatibility,
File "/usr/local/lib/python3.10/dist-packages/deep_ep/buffer.py", line 64, in init
dist.all_gather_object(device_ids, local_device_id, group)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object
all_gather(object_size_list, local_size, group=group)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather
work = group.allgather([tensor_list], [tensor])
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
socketStartConnect: Connect to fe80::ba3f:d203:56:871a%4<40625> failed : Network is unreachable
my nvshmem version is 3.2.5
@alpha-baby commented on GitHub (Sep 18, 2025):
socketStartConnect: Connect to fe80::ba3f:d203:56:871a%4<40625> failed : Network is unreachable
Ipv6 network is blocked, so you should use ipv4, and you should ensure that your tcp network is connected.
@junchenglai-mthreads commented on GitHub (Sep 18, 2025):
mccx:8974:8974 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8974:8974 [0] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8974:8974 [0] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8974:8974 [0] NCCL INFO cudaDriverVersion 12080
NCCL version 2.21.5+cuda12.4
mccx:8981:8981 [7] NCCL INFO cudaDriverVersion 12080
mccx:8981:8981 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8981:8981 [7] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8981:8981 [7] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8977:8977 [3] NCCL INFO cudaDriverVersion 12080
mccx:8977:8977 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8977:8977 [3] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8977:8977 [3] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8980:8980 [6] NCCL INFO cudaDriverVersion 12080
mccx:8980:8980 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8980:8980 [6] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8980:8980 [6] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8977:8977 [3] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7ef8e3400000
mccx:8977:8977 [3] NCCL INFO Comm config Blocking set to 0
mccx:8981:8981 [7] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f0d39400000
mccx:8981:8981 [7] NCCL INFO Comm config Blocking set to 0
mccx:8980:8980 [6] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f6057400000
mccx:8980:8980 [6] NCCL INFO Comm config Blocking set to 0
mccx:8975:8975 [1] NCCL INFO cudaDriverVersion 12080
mccx:8975:8975 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8975:8975 [1] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8975:8975 [1] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8974:8974 [0] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f0e17400000
mccx:8974:8974 [0] NCCL INFO Comm config Blocking set to 0
mccx:8978:8978 [4] NCCL INFO cudaDriverVersion 12080
mccx:8978:8978 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8978:8978 [4] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8978:8978 [4] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8978:8978 [4] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7efe77400000
mccx:8978:8978 [4] NCCL INFO Comm config Blocking set to 0
mccx:8975:8975 [1] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f4239400000
mccx:8975:8975 [1] NCCL INFO Comm config Blocking set to 0
mccx:8979:8979 [5] NCCL INFO cudaDriverVersion 12080
mccx:8979:8979 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8979:8979 [5] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8976:8976 [2] NCCL INFO cudaDriverVersion 12080
mccx:8979:8979 [5] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8976:8976 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8976:8976 [2] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8976:8976 [2] NCCL INFO Bootstrap : Using bond0:10.10.142.201<0>
mccx:8976:8976 [2] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f8eb9400000
mccx:8976:8976 [2] NCCL INFO Comm config Blocking set to 0
mccx:8979:8979 [5] NCCL INFO init.cc:1785 Cuda Host Alloc Size 4 pointer 0x7f84df400000
mccx:8979:8979 [5] NCCL INFO Comm config Blocking set to 0
mccx:8977:9523 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8977:9523 [3] NCCL INFO P2P plugin IBext_v8
mccx:8977:9523 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8977:9523 [3] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8981:9526 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8981:9526 [7] NCCL INFO P2P plugin IBext_v8
mccx:8981:9526 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8981:9526 [7] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8980:9529 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8980:9529 [6] NCCL INFO P2P plugin IBext_v8
mccx:8980:9529 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8980:9529 [6] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8974:9531 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8974:9531 [0] NCCL INFO P2P plugin IBext_v8
mccx:8974:9531 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8974:9531 [0] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8977:9523 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8977:9523 [3] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8977:9523 [3] NCCL INFO Using non-device net plugin version 0
mccx:8977:9523 [3] NCCL INFO Using network IBext_v8
mccx:8978:9535 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8978:9535 [4] NCCL INFO P2P plugin IBext_v8
mccx:8978:9535 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8978:9535 [4] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8980:9529 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8980:9529 [6] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8980:9529 [6] NCCL INFO Using non-device net plugin version 0
mccx:8980:9529 [6] NCCL INFO Using network IBext_v8
mccx:8975:9537 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8975:9537 [1] NCCL INFO P2P plugin IBext_v8
mccx:8975:9537 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8975:9537 [1] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8981:9526 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8981:9526 [7] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8981:9526 [7] NCCL INFO Using non-device net plugin version 0
mccx:8981:9526 [7] NCCL INFO Using network IBext_v8
mccx:8978:9535 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8978:9535 [4] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8978:9535 [4] NCCL INFO Using non-device net plugin version 0
mccx:8978:9535 [4] NCCL INFO Using network IBext_v8
mccx:8974:9531 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8974:9531 [0] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8974:9531 [0] NCCL INFO Using non-device net plugin version 0
mccx:8974:9531 [0] NCCL INFO Using network IBext_v8
mccx:8975:9537 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8975:9537 [1] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8975:9537 [1] NCCL INFO Using non-device net plugin version 0
mccx:8975:9537 [1] NCCL INFO Using network IBext_v8
mccx:8977:9523 [3] NCCL INFO DMA-BUF is available on GPU device 3
mccx:8976:9540 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8976:9540 [2] NCCL INFO P2P plugin IBext_v8
mccx:8976:9540 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8976:9540 [2] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8979:9543 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
mccx:8979:9543 [5] NCCL INFO P2P plugin IBext_v8
mccx:8979:9543 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to bond0
mccx:8979:9543 [5] NCCL INFO NCCL_SOCKET_IFNAME set to bond0
mccx:8976:9540 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8976:9540 [2] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8976:9540 [2] NCCL INFO Using non-device net plugin version 0
mccx:8976:9540 [2] NCCL INFO Using network IBext_v8
mccx:8979:9543 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB bond0:10.10.142.201<0>
mccx:8979:9543 [5] NCCL INFO NET/IB : GPU Direct RDMA (DMABUF) enabled for HCA 0 'mlx5_0
mccx:8979:9543 [5] NCCL INFO Using non-device net plugin version 0
mccx:8979:9543 [5] NCCL INFO Using network IBext_v8
mccx:8980:9529 [6] NCCL INFO DMA-BUF is available on GPU device 6
mccx:8981:9526 [7] NCCL INFO DMA-BUF is available on GPU device 7
mccx:8978:9535 [4] NCCL INFO DMA-BUF is available on GPU device 4
mccx:8974:9531 [0] NCCL INFO DMA-BUF is available on GPU device 0
mccx:8975:9537 [1] NCCL INFO DMA-BUF is available on GPU device 1
mccx:8976:9540 [2] NCCL INFO DMA-BUF is available on GPU device 2
mccx:8979:9543 [5] NCCL INFO DMA-BUF is available on GPU device 5
now i set NCCL_SOCKET_IFNAME=boud0，
and sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 to forbidden ipv6,
now it hang here, what's the problem
@alpha-baby commented on GitHub (Sep 19, 2025):
maybe you could use:
cuda-gdb --pid {pid}to look backtrace@junchenglai-mthreads commented on GitHub (Sep 19, 2025):
root@mccx:/sgl-workspace/ljc/DeepEP# MASTER_ADDR=xxxxxx MASTER_PORT=29500 WORLD_SIZE=2 RANK=1 python tests/test_internode.py
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
IPC failure: Sending data over socket failed: No such file or directory
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/mem/mem_heap.cpp:exchange_heap_memory_handle:781: Fatal IPC Failure
W0919 06:48:16.657000 2789 torch/multiprocessing/spawn.py:169] Terminating process 2857 via signal SIGTERM
W0919 06:48:16.658000 2789 torch/multiprocessing/spawn.py:169] Terminating process 2859 via signal SIGTERM
Traceback (most recent call last):
File "/sgl-workspace/ljc/DeepEP/tests/test_internode.py", line 328, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 204, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 6 terminated with exit code 255
I met a new bug，can u help me
@junchenglai-mthreads commented on GitHub (Sep 19, 2025):
I solved， the host name between node0 and node1 should be different