deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #274] [Bug] test_internode.py timeout on 4 * 8 H800 #204

New issue
Open
opened 2026-10-26 09:56:33 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-26 09:56:33 +00:00
Owner
Copy link

Originally created by @newfarmer on GitHub (Jul 2, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/274

Environment：

Image

script:

export NCCL_DEBUG=INFO
export TORCH_CPP_LOG_LEVEL=INFO
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_IFNAME=bond0
export GLOO_SOCKET_IFNAME=bond0
export NCCL_IB_HCA='^=mlx5_bond_0'
export NCCL_IB_TC=128
export NCCL_IB_TIMEOUT=22
export NCCL_IB_RETRY_CNT=15

export DISABLE_SM90_FEATURES=1
export TORCH_CUDA_ARCH_LIST="9.0"
export DISABLE_AGGRESSIVE_PTX_INSTRS=0
export NCCL_IB_QPS_PER_CONNECTION=128

# node 0：
 MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=0 python test_internode.py
# node 1：
 MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=1 python test_internode.py
# node 2：
 MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=2 python test_internode.py
# node 3：
 MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=3 python test_internode.py

logs on node 0:

DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 05/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 03/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 07/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 09/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 07/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 11/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 11/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 15/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 13/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 15/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 05/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 01/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 09/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 05/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 07/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 13/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 09/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 15/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 11/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 13/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA
DeepEP-test:3081461:3082596 [4] NCCL INFO Connected NVLS tree
DeepEP-test:3081461:3082596 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081461:3082596 [4] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081459:3082607 [2] NCCL INFO Connected NVLS tree
DeepEP-test:3081459:3082607 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081459:3082607 [2] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081457:3082608 [0] NCCL INFO Connected NVLS tree
DeepEP-test:3081457:3082608 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081457:3082608 [0] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081464:3082593 [7] NCCL INFO Connected NVLS tree
DeepEP-test:3081464:3082593 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081464:3082593 [7] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081460:3082601 [3] NCCL INFO Connected NVLS tree
DeepEP-test:3081460:3082601 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081460:3082601 [3] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081463:3082611 [6] NCCL INFO Connected NVLS tree
DeepEP-test:3081463:3082611 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081463:3082611 [6] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081462:3082590 [5] NCCL INFO Connected NVLS tree
DeepEP-test:3081462:3082590 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081462:3082590 [5] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081458:3082602 [1] NCCL INFO Connected NVLS tree
DeepEP-test:3081458:3082602 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512
DeepEP-test:3081458:3082602 [1] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
DeepEP-test:3081458:3082602 [1] NCCL INFO ncclCommSplit comm 0x88506c0 rank 1 nranks 32 cudaDev 1 nvmlDev 1 busId 38000 parent 0x863f950 color 698429859 key 1 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081462:3082590 [5] NCCL INFO ncclCommSplit comm 0x71d3420 rank 5 nranks 32 cudaDev 5 nvmlDev 5 busId bb000 parent 0x6fc26a0 color 698429859 key 5 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081460:3082601 [3] NCCL INFO ncclCommSplit comm 0x73d21c0 rank 3 nranks 32 cudaDev 3 nvmlDev 3 busId 59000 parent 0x71c1410 color 698429859 key 3 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081464:3082593 [7] NCCL INFO ncclCommSplit comm 0x8af9980 rank 7 nranks 32 cudaDev 7 nvmlDev 7 busId da000 parent 0x88e8470 color 698429859 key 7 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081461:3082596 [4] NCCL INFO ncclCommSplit comm 0x7ad94b0 rank 4 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 parent 0x78c8730 color 698429859 key 4 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081457:3082608 [0] NCCL INFO ncclCommSplit comm 0x72ab6d0 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 18000 parent 0x709b0f0 color 698429859 key 0 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081463:3082611 [6] NCCL INFO ncclCommSplit comm 0x898dc00 rank 6 nranks 32 cudaDev 6 nvmlDev 6 busId ca000 parent 0x877ce70 color 698429859 key 6 commId 0x725286ef14dca1f4 - Init COMPLETE
DeepEP-test:3081459:3082607 [2] NCCL INFO ncclCommSplit comm 0x8327690 rank 2 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 parent 0x8116900 color 698429859 key 2 commId 0x725286ef14dca1f4 - Init COMPLETE
[rank5]:[I702 21:26:24.214150102 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 5] ProcessGroupNCCL created ncclComm_ 0x71d3420 on CUDA device:
[rank5]:[I702 21:26:24.214206655 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 5] NCCL_DEBUG: INFO
[rank7]:[I702 21:26:24.214185277 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 7] ProcessGroupNCCL created ncclComm_ 0x8af9980 on CUDA device:
[rank1]:[I702 21:26:24.214184812 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 1] ProcessGroupNCCL created ncclComm_ 0x88506c0 on CUDA device:
[rank4]:[I702 21:26:24.214206895 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 4] ProcessGroupNCCL created ncclComm_ 0x7ad94b0 on CUDA device:
[rank3]:[I702 21:26:24.214220130 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 3] ProcessGroupNCCL created ncclComm_ 0x73d21c0 on CUDA device:
[rank7]:[I702 21:26:24.214239687 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 7] NCCL_DEBUG: INFO
[rank1]:[I702 21:26:24.214253689 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 1] NCCL_DEBUG: INFO
[rank4]:[I702 21:26:24.214265286 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 4] NCCL_DEBUG: INFO
[rank3]:[I702 21:26:24.214274454 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 3] NCCL_DEBUG: INFO
[rank6]:[I702 21:26:24.214295010 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 6] ProcessGroupNCCL created ncclComm_ 0x898dc00 on CUDA device:
[rank2]:[I702 21:26:24.214303675 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 2] ProcessGroupNCCL created ncclComm_ 0x8327690 on CUDA device:
[rank6]:[I702 21:26:24.214355397 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 6] NCCL_DEBUG: INFO
[rank2]:[I702 21:26:24.214364662 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 2] NCCL_DEBUG: INFO
[rank0]:[I702 21:26:24.214582605 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 0] ProcessGroupNCCL created ncclComm_ 0x72ab6d0 on CUDA device:
[rank0]:[I702 21:26:24.214675581 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 0] NCCL_DEBUG: INFO
xterm-256colorWARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
[config] num_tokens=4096, hidden=7168, num_topk_groups=4, num_topk=8
[layout] Kernel performance: 0.057 ms

[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=False, previous=False) ... passed
[testing] Running with BF16, with top-k (async=False, previous=False) ... passed
[testing] Running with FP8, without top-k (async=False, previous=False) ... passed
[testing] Running with FP8, with top-k (async=False, previous=False) ... passed
[testing] Running with BF16, without top-k (async=True, previous=False) ... passed
[testing] Running with BF16, with top-k (async=True, previous=False) ... passed
[testing] Running with BF16, without top-k (async=True, previous=False) ... passed
[testing] Running with BF16, with top-k (async=True, previous=False) ... passed
[testing] Running with FP8, without top-k (async=True, previous=False) ... passed
[testing] Running with FP8, with top-k (async=True, previous=False) ... passed
[testing] Running with BF16, without top-k (async=False, previous=True) ... passed
[testing] Running with BF16, with top-k (async=False, previous=True) ... passed
[testing] Running with BF16, without top-k (async=False, previous=True) ... passed
[testing] Running with BF16, with top-k (async=False, previous=True) ... passed
[testing] Running with FP8, without top-k (async=False, previous=True) ... passed
[testing] Running with FP8, with top-k (async=False, previous=True) ... passed
[testing] Running with BF16, without top-k (async=True, previous=True) ... passed
[testing] Running with BF16, with top-k (async=True, previous=True) ...
DeepEP timeout check failed: rank = 7, thread = 0)
DeepEP timeout check failed: rank = 0, thread = 0)
DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 0, tail: 309, remaining: 0
DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 1, tail: 297, remaining: 0
DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 2, tail: 240, remaining: 70
DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 3, tail: 224, remaining: 75
DeepEP timeout check failed: rank = 3, thread = 0)
DeepEP timeout check failed: rank = 2, thread = 0)
DeepEP timeout check failed: rank = 6, thread = 0)
DeepEP timeout check failed: rank = 5, thread = 0)
DeepEP timeout check failed: rank = 1, thread = 0)
W0702 21:28:25.026000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081457 via signal SIGTERM
W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081458 via signal SIGTERM
W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081459 via signal SIGTERM
W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081460 via signal SIGTERM
W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081462 via signal SIGTERM
W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081463 via signal SIGTERM
W0702 21:28:25.028000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081464 via signal SIGTERM
Traceback (most recent call last):
  File "/root/hfy/DeepEP-main/tests/test_internode.py", line 254, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
  File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes
    while not context.join():
  File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 203, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/root/hfy/DeepEP-main/tests/test_internode.py", line 238, in test_loop
    test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
  File "/root/hfy/DeepEP-main/tests/test_internode.py", line 116, in test_main
    assert gbl_num_tokens_per_rank[rank].item() == recv_x.size(0), f'{gbl_num_tokens_per_rank[rank].item()} != {recv_x.size(0)}'
  File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/utils/_device.py", line 106, in __torch_function__
    return func(*args, **kwargs)
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Originally created by @newfarmer on GitHub (Jul 2, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/274 Environment： ![Image](https://github.com/user-attachments/assets/d9741e28-59bc-4b41-963f-9c282a7bf1f1) script: ```shell export NCCL_DEBUG=INFO export TORCH_CPP_LOG_LEVEL=INFO export NCCL_IB_GID_INDEX=3 export NCCL_SOCKET_IFNAME=bond0 export GLOO_SOCKET_IFNAME=bond0 export NCCL_IB_HCA='^=mlx5_bond_0' export NCCL_IB_TC=128 export NCCL_IB_TIMEOUT=22 export NCCL_IB_RETRY_CNT=15 export DISABLE_SM90_FEATURES=1 export TORCH_CUDA_ARCH_LIST="9.0" export DISABLE_AGGRESSIVE_PTX_INSTRS=0 export NCCL_IB_QPS_PER_CONNECTION=128 # node 0： MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=0 python test_internode.py # node 1： MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=1 python test_internode.py # node 2： MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=2 python test_internode.py # node 3： MASTER_ADDR=my_master_node MASTER_PORT=my_master_port WORLD_SIZE=4 RANK=3 python test_internode.py ``` logs on node 0: ``` DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 01/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 05/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 03/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 07/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 05/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 09/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 07/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 11/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 11/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081462:3082590 [5] NCCL INFO Channel 15/0 : 13[5] -> 5[5] [receive] via NET/IB/1/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 13/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081458:3082602 [1] NCCL INFO Channel 15/0 : 9[1] -> 1[1] [receive] via NET/IB/6/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 03/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 05/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 01/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 07/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 09/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 05/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 07/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 13/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 09/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081460:3082601 [3] NCCL INFO Channel 15/0 : 11[3] -> 3[3] [receive] via NET/IB/2/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 11/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081464:3082593 [7] NCCL INFO Channel 13/0 : 15[7] -> 7[7] [receive] via NET/IB/7/GDRDMA DeepEP-test:3081461:3082596 [4] NCCL INFO Connected NVLS tree DeepEP-test:3081461:3082596 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081461:3082596 [4] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081459:3082607 [2] NCCL INFO Connected NVLS tree DeepEP-test:3081459:3082607 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081459:3082607 [2] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081457:3082608 [0] NCCL INFO Connected NVLS tree DeepEP-test:3081457:3082608 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081457:3082608 [0] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081464:3082593 [7] NCCL INFO Connected NVLS tree DeepEP-test:3081464:3082593 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081464:3082593 [7] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081460:3082601 [3] NCCL INFO Connected NVLS tree DeepEP-test:3081460:3082601 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081460:3082601 [3] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081463:3082611 [6] NCCL INFO Connected NVLS tree DeepEP-test:3081463:3082611 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081463:3082611 [6] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081462:3082590 [5] NCCL INFO Connected NVLS tree DeepEP-test:3081462:3082590 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081462:3082590 [5] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081458:3082602 [1] NCCL INFO Connected NVLS tree DeepEP-test:3081458:3082602 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 DeepEP-test:3081458:3082602 [1] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer DeepEP-test:3081458:3082602 [1] NCCL INFO ncclCommSplit comm 0x88506c0 rank 1 nranks 32 cudaDev 1 nvmlDev 1 busId 38000 parent 0x863f950 color 698429859 key 1 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081462:3082590 [5] NCCL INFO ncclCommSplit comm 0x71d3420 rank 5 nranks 32 cudaDev 5 nvmlDev 5 busId bb000 parent 0x6fc26a0 color 698429859 key 5 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081460:3082601 [3] NCCL INFO ncclCommSplit comm 0x73d21c0 rank 3 nranks 32 cudaDev 3 nvmlDev 3 busId 59000 parent 0x71c1410 color 698429859 key 3 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081464:3082593 [7] NCCL INFO ncclCommSplit comm 0x8af9980 rank 7 nranks 32 cudaDev 7 nvmlDev 7 busId da000 parent 0x88e8470 color 698429859 key 7 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081461:3082596 [4] NCCL INFO ncclCommSplit comm 0x7ad94b0 rank 4 nranks 32 cudaDev 4 nvmlDev 4 busId 9b000 parent 0x78c8730 color 698429859 key 4 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081457:3082608 [0] NCCL INFO ncclCommSplit comm 0x72ab6d0 rank 0 nranks 32 cudaDev 0 nvmlDev 0 busId 18000 parent 0x709b0f0 color 698429859 key 0 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081463:3082611 [6] NCCL INFO ncclCommSplit comm 0x898dc00 rank 6 nranks 32 cudaDev 6 nvmlDev 6 busId ca000 parent 0x877ce70 color 698429859 key 6 commId 0x725286ef14dca1f4 - Init COMPLETE DeepEP-test:3081459:3082607 [2] NCCL INFO ncclCommSplit comm 0x8327690 rank 2 nranks 32 cudaDev 2 nvmlDev 2 busId 49000 parent 0x8116900 color 698429859 key 2 commId 0x725286ef14dca1f4 - Init COMPLETE [rank5]:[I702 21:26:24.214150102 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 5] ProcessGroupNCCL created ncclComm_ 0x71d3420 on CUDA device: [rank5]:[I702 21:26:24.214206655 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 5] NCCL_DEBUG: INFO [rank7]:[I702 21:26:24.214185277 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 7] ProcessGroupNCCL created ncclComm_ 0x8af9980 on CUDA device: [rank1]:[I702 21:26:24.214184812 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 1] ProcessGroupNCCL created ncclComm_ 0x88506c0 on CUDA device: [rank4]:[I702 21:26:24.214206895 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 4] ProcessGroupNCCL created ncclComm_ 0x7ad94b0 on CUDA device: [rank3]:[I702 21:26:24.214220130 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 3] ProcessGroupNCCL created ncclComm_ 0x73d21c0 on CUDA device: [rank7]:[I702 21:26:24.214239687 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 7] NCCL_DEBUG: INFO [rank1]:[I702 21:26:24.214253689 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 1] NCCL_DEBUG: INFO [rank4]:[I702 21:26:24.214265286 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 4] NCCL_DEBUG: INFO [rank3]:[I702 21:26:24.214274454 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 3] NCCL_DEBUG: INFO [rank6]:[I702 21:26:24.214295010 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 6] ProcessGroupNCCL created ncclComm_ 0x898dc00 on CUDA device: [rank2]:[I702 21:26:24.214303675 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 2] ProcessGroupNCCL created ncclComm_ 0x8327690 on CUDA device: [rank6]:[I702 21:26:24.214355397 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 6] NCCL_DEBUG: INFO [rank2]:[I702 21:26:24.214364662 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 2] NCCL_DEBUG: INFO [rank0]:[I702 21:26:24.214582605 ProcessGroupNCCL.cpp:2301] [PG ID 1 PG GUID 1 Rank 0] ProcessGroupNCCL created ncclComm_ 0x72ab6d0 on CUDA device: [rank0]:[I702 21:26:24.214675581 ProcessGroupNCCL.cpp:2306] [PG ID 1 PG GUID 1 Rank 0] NCCL_DEBUG: INFO xterm-256colorWARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument [config] num_tokens=4096, hidden=7168, num_topk_groups=4, num_topk=8 [layout] Kernel performance: 0.057 ms [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=False, previous=False) ... passed [testing] Running with BF16, with top-k (async=False, previous=False) ... passed [testing] Running with FP8, without top-k (async=False, previous=False) ... passed [testing] Running with FP8, with top-k (async=False, previous=False) ... passed [testing] Running with BF16, without top-k (async=True, previous=False) ... passed [testing] Running with BF16, with top-k (async=True, previous=False) ... passed [testing] Running with BF16, without top-k (async=True, previous=False) ... passed [testing] Running with BF16, with top-k (async=True, previous=False) ... passed [testing] Running with FP8, without top-k (async=True, previous=False) ... passed [testing] Running with FP8, with top-k (async=True, previous=False) ... passed [testing] Running with BF16, without top-k (async=False, previous=True) ... passed [testing] Running with BF16, with top-k (async=False, previous=True) ... passed [testing] Running with BF16, without top-k (async=False, previous=True) ... passed [testing] Running with BF16, with top-k (async=False, previous=True) ... passed [testing] Running with FP8, without top-k (async=False, previous=True) ... passed [testing] Running with FP8, with top-k (async=False, previous=True) ... passed [testing] Running with BF16, without top-k (async=True, previous=True) ... passed [testing] Running with BF16, with top-k (async=True, previous=True) ... DeepEP timeout check failed: rank = 7, thread = 0) DeepEP timeout check failed: rank = 0, thread = 0) DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 0, tail: 309, remaining: 0 DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 1, tail: 297, remaining: 0 DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 2, tail: 240, remaining: 70 DeepEP RDMA sender coordinator timeout, channel: 4, IB: 0, nvl 4, dst IB: 3, tail: 224, remaining: 75 DeepEP timeout check failed: rank = 3, thread = 0) DeepEP timeout check failed: rank = 2, thread = 0) DeepEP timeout check failed: rank = 6, thread = 0) DeepEP timeout check failed: rank = 5, thread = 0) DeepEP timeout check failed: rank = 1, thread = 0) W0702 21:28:25.026000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081457 via signal SIGTERM W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081458 via signal SIGTERM W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081459 via signal SIGTERM W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081460 via signal SIGTERM W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081462 via signal SIGTERM W0702 21:28:25.027000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081463 via signal SIGTERM W0702 21:28:25.028000 3081433 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 3081464 via signal SIGTERM Traceback (most recent call last): File "/root/hfy/DeepEP-main/tests/test_internode.py", line 254, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes while not context.join(): File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 203, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/root/hfy/DeepEP-main/tests/test_internode.py", line 238, in test_loop test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group) File "/root/hfy/DeepEP-main/tests/test_internode.py", line 116, in test_main assert gbl_num_tokens_per_rank[rank].item() == recv_x.size(0), f'{gbl_num_tokens_per_rank[rank].item()} != {recv_x.size(0)}' File "/root/miniconda3/envs/test_deepep_py310/lib/python3.10/site-packages/torch/utils/_device.py", line 106, in __torch_function__ return func(*args, **kwargs) RuntimeError: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ```
bsakaguchi commented 2026-10-26 09:56:33 +00:00
Author
Owner
Copy link

@dfshshw commented on GitHub (Jul 2, 2025):

meet the same problem on 4*8 h800，after fp8 dispatch test, "DeepEP dispatch RDMA sender timeout" happened in bf16 dispatch test. more info is

Image

Image

<!-- gh-comment-id:3028416667 --> @dfshshw commented on GitHub (Jul 2, 2025): meet the same problem on 4*8 h800，after fp8 dispatch test, "DeepEP dispatch RDMA sender timeout" happened in bf16 dispatch test. more info is ![Image](https://github.com/user-attachments/assets/36e1d299-f0d2-4038-8003-2e2388cf8a75) ![Image](https://github.com/user-attachments/assets/ce6d3401-ef34-421b-82de-2832327d0cb6)
bsakaguchi commented 2026-10-26 09:56:34 +00:00
Author
Owner
Copy link

@hongfangyu commented on GitHub (Jul 3, 2025):

same error with ROCE. 2 nodes 16 GPUs under the same TOR is OK, but 4 nodes 32 gpus fail.

<!-- gh-comment-id:3030196070 --> @hongfangyu commented on GitHub (Jul 3, 2025): same error with ROCE. 2 nodes 16 GPUs under the same TOR is OK, but 4 nodes 32 gpus fail.
bsakaguchi commented 2026-10-26 09:56:34 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 3, 2025):

It might be the same issue as https://github.com/deepseek-ai/DeepEP/issues/270. I will try to debug it.

<!-- gh-comment-id:3031721199 --> @sphish commented on GitHub (Jul 3, 2025): It might be the same issue as https://github.com/deepseek-ai/DeepEP/issues/270. I will try to debug it.
bsakaguchi commented 2026-10-26 09:56:34 +00:00
Author
Owner
Copy link

@Roysky commented on GitHub (Jul 8, 2025):

meet the same problem on 4*8 h800

<!-- gh-comment-id:3047111501 --> @Roysky commented on GitHub (Jul 8, 2025): meet the same problem on 4*8 h800
bsakaguchi commented 2026-10-26 09:56:34 +00:00
Author
Owner
Copy link

@hongfangyu commented on GitHub (Jul 8, 2025):

this repo is ok，about 45GB/s
https://github.com/Infrawaves/DeepEP_ibrc_dual-ports_multiQP

<!-- gh-comment-id:3048719791 --> @hongfangyu commented on GitHub (Jul 8, 2025): this repo is ok，about 45GB/s https://github.com/Infrawaves/DeepEP_ibrc_dual-ports_multiQP
bsakaguchi commented 2026-10-26 09:56:34 +00:00
Author
Owner
Copy link

@ericschreiber commented on GitHub (Jul 8, 2025):

Hi I have the same issue.

<!-- gh-comment-id:3049646218 --> @ericschreiber commented on GitHub (Jul 8, 2025): Hi I have the same issue.
bsakaguchi commented 2026-10-26 09:56:35 +00:00
Author
Owner
Copy link

@pengchengneo commented on GitHub (Jul 9, 2025):

same error on 4 * 8 * H800, any progress with that @newfarmer
thanks !

<!-- gh-comment-id:3051697884 --> @pengchengneo commented on GitHub (Jul 9, 2025): same error on 4 * 8 * H800, any progress with that @newfarmer thanks !
bsakaguchi commented 2026-10-26 09:56:35 +00:00
Author
Owner
Copy link

@GuangguanWang commented on GitHub (Jul 17, 2025):

Try this PR #310.

<!-- gh-comment-id:3083784615 --> @GuangguanWang commented on GitHub (Jul 17, 2025): Try this PR #310.
bsakaguchi commented 2026-10-26 09:56:36 +00:00
Author
Owner
Copy link

@spacegoing commented on GitHub (Oct 22, 2025):


WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
DeepEP dispatch NVL receiver timeout, channel: 6, RDMA: 0, nvl: 0, src NVL: 5, head: 315, tail: 315
DeepEP dispatch NVL receiver timeout, channel: 10, RDMA: 0, nvl: 0, src NVL: 5, head: 300, tail: 300
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure'
ment
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
WARN: IB: read failed in ib_roce_get_version_num: Invalid argument
DeepEP dispatch NVL receiver timeout, channel: 8, RDMA: 0, nvl: 3, src NVL: 5, head: 324, tail: 324
DeepEP dispatch NVL receiver timeout, channel: 6, RDMA: 0, nvl: 3, src NVL: 5, head: 330, tail: 330
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure'
W1022 04:10:47.910000 808 torch/multiprocessing/spawn.py:169] Terminating process 873 via signal SIGTERM
W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 874 via signal SIGTERM
W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 875 via signal SIGTERM
W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 877 via signal SIGTERM
W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 878 via signal SIGTERM
W1022 04:10:47.912000 808 torch/multiprocessing/spawn.py:169] Terminating process 879 via signal SIGTERM
W1022 04:10:47.912000 808 torch/multiprocessing/spawn.py:169] Terminating process 880 via signal SIGTERM
Traceback (most recent call last):
  File "/workspace/DeepEP/tests/test_internode.py", line 244, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 3 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/workspace/DeepEP/tests/test_internode.py", line 232, in test_loop
    test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
  File "/workspace/DeepEP/tests/test_internode.py", line 109, in test_main
    recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args)
                                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.0.0+a84a248-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 283, in dispatch
    return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.0.0+a84a248-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 391, in internode_dispatch
    recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch(
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed: CUDA error /workspace/DeepEP/csrc/kernels/internode.cu:1068 'unspecified launch failure'

--- Inter-Node Test Finished on Node 0 ---

same issue here on 20 H200 nodes

<!-- gh-comment-id:3430426329 --> @spacegoing commented on GitHub (Oct 22, 2025): ``` WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument DeepEP dispatch NVL receiver timeout, channel: 6, RDMA: 0, nvl: 0, src NVL: 5, head: 315, tail: 315 DeepEP dispatch NVL receiver timeout, channel: 10, RDMA: 0, nvl: 0, src NVL: 5, head: 300, tail: 300 terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure' ment WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument WARN: IB: read failed in ib_roce_get_version_num: Invalid argument DeepEP dispatch NVL receiver timeout, channel: 8, RDMA: 0, nvl: 3, src NVL: 5, head: 324, tail: 324 DeepEP dispatch NVL receiver timeout, channel: 6, RDMA: 0, nvl: 3, src NVL: 5, head: 330, tail: 330 terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure' terminate called after throwing an instance of 'EPException' what(): Failed: CUDA error /workspace/DeepEP/csrc/deep_ep.cpp:83 'unspecified launch failure' W1022 04:10:47.910000 808 torch/multiprocessing/spawn.py:169] Terminating process 873 via signal SIGTERM W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 874 via signal SIGTERM W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 875 via signal SIGTERM W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 877 via signal SIGTERM W1022 04:10:47.911000 808 torch/multiprocessing/spawn.py:169] Terminating process 878 via signal SIGTERM W1022 04:10:47.912000 808 torch/multiprocessing/spawn.py:169] Terminating process 879 via signal SIGTERM W1022 04:10:47.912000 808 torch/multiprocessing/spawn.py:169] Terminating process 880 via signal SIGTERM Traceback (most recent call last): File "/workspace/DeepEP/tests/test_internode.py", line 244, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes) File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 3 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/workspace/DeepEP/tests/test_internode.py", line 232, in test_loop test_main(i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group) File "/workspace/DeepEP/tests/test_internode.py", line 109, in test_main recv_x, recv_topk_idx, recv_topk_weights, recv_num_tokens_per_expert_list, handle, event = buffer.dispatch(**dispatch_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.0.0+a84a248-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 283, in dispatch return self.internode_dispatch(x, handle, num_tokens_per_rank, num_tokens_per_rdma_rank, is_token_in_rank, num_tokens_per_expert, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/deep_ep-1.0.0+a84a248-py3.12-linux-x86_64.egg/deep_ep/buffer.py", line 391, in internode_dispatch recv_src_meta, send_rdma_head, send_nvl_head, event = self.runtime.internode_dispatch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Failed: CUDA error /workspace/DeepEP/csrc/kernels/internode.cu:1068 'unspecified launch failure' --- Inter-Node Test Finished on Node 0 --- ``` same issue here on 20 H200 nodes
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#204
No description provided.