mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #122] 一直卡着不动 #106
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#106
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @yushuai1 on GitHub (Apr 15, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/122
一直被卡死在这里
2*8 gpu h200
启动命令
NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=0 python tests/test_internode.py
NCCL_DEBUG=INFO MASTER_ADDR=172.31.1.12 WORLD_SIZE=2 RANK=1 python tests/test_internode.py
日志到这里就卡死了
hd03-gpu2-0012:39115:39820 [4] NCCL INFO Channel 15/0 : 4[4] -> 12[4] [send] via NET/IB/5/GDRDMA
hd03-gpu2-0012:39113:39814 [2] NCCL INFO Channel 12/0 : 2[2] -> 10[2] [send] via NET/IB/3/GDRDMA
hd03-gpu2-0012:39113:39814 [2] NCCL INFO Channel 13/0 : 2[2] -> 10[2] [send] via NET/IB/3/GDRDMA
hd03-gpu2-0012:39113:39814 [2] NCCL INFO Channel 14/0 : 2[2] -> 10[2] [send] via NET/IB/3/GDRDMA
hd03-gpu2-0012:39113:39814 [2] NCCL INFO Channel 15/0 : 2[2] -> 10[2] [send] via NET/IB/3/GDRDMA
hd03-gpu2-0012:39111:39809 [0] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39111:39809 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39111:39809 [0] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39115:39820 [4] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39115:39820 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39115:39820 [4] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39116:39815 [5] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39116:39815 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39116:39815 [5] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39112:39822 [1] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39112:39822 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39112:39822 [1] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39117:39821 [6] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39117:39821 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39117:39821 [6] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39113:39814 [2] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39113:39814 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39113:39814 [2] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39118:39816 [7] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39118:39816 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39118:39816 [7] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39114:39817 [3] NCCL INFO Connected NVLS tree
hd03-gpu2-0012:39114:39817 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
hd03-gpu2-0012:39114:39817 [3] NCCL INFO 16 coll channels, 16 collnet channels, 16 nvls channels, 16 p2p channels, 2 p2p channels per peer
hd03-gpu2-0012:39118:39816 [7] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39118:39816 [7] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39118:39816 [7] NCCL INFO ncclCommInitRank comm 0x1129f670 rank 7 nranks 16 cudaDev 7 nvmlDev 7 busId db000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39115:39820 [4] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39116:39815 [5] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39117:39821 [6] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39115:39820 [4] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39116:39815 [5] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39117:39821 [6] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39115:39820 [4] NCCL INFO ncclCommInitRank comm 0x9381970 rank 4 nranks 16 cudaDev 4 nvmlDev 4 busId 9b000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39117:39821 [6] NCCL INFO ncclCommInitRank comm 0x2927f8d0 rank 6 nranks 16 cudaDev 6 nvmlDev 6 busId cb000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39116:39815 [5] NCCL INFO ncclCommInitRank comm 0x1e7b45a0 rank 5 nranks 16 cudaDev 5 nvmlDev 5 busId bb000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39113:39814 [2] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39113:39814 [2] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39113:39814 [2] NCCL INFO ncclCommInitRank comm 0x2ffb4a30 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 4c000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39111:39809 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39114:39817 [3] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39111:39809 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39114:39817 [3] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39111:39809 [0] NCCL INFO ncclCommInitRank comm 0xb607c60 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39114:39817 [3] NCCL INFO ncclCommInitRank comm 0x21f65970 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId 5d000 commId 0x5cbac1e9e398f569 - Init COMPLETE
hd03-gpu2-0012:39112:39822 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
hd03-gpu2-0012:39112:39822 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
hd03-gpu2-0012:39112:39822 [1] NCCL INFO ncclCommInitRank comm 0x2604f640 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 3b000 commId 0x5cbac1e9e398f569 - Init COMPLETE
@sphish commented on GitHub (Apr 16, 2025):
可以看一下具体卡在哪行 python 代码上了
@yushuai1 commented on GitHub (Apr 16, 2025):
buffer = deep_ep.Buffer(group, int(1e9), int(1e9), low_latency_mode=test_ll_compatibility,
num_qps_per_rank=(ll_num_experts // num_ranks if test_ll_compatibility else 1))
只要num_rdma_bytes >0 就会卡死
@sphish commented on GitHub (Apr 16, 2025):
那可能是 nvshmem init 卡了，可以把 nvshmem 的 log 打出来看看，
export NVSHMEM_DEBUG=INFO@HermesCui commented on GitHub (Jun 23, 2025):
@yushuai1 Could you show how to figure out this issue, and we have same issue
@alpha-baby commented on GitHub (Jun 23, 2025):
you can use GDB to print the backtrace