mirror of
https://github.com/deepseek-ai/DeepEP.git
synced 2026-11-05 00:47:32 +00:00
[GH-ISSUE #411] Internode hang #278
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepEP#278
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @junchenglai-mthreads on GitHub (Sep 16, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/411
MASTER_ADDR=xxxxxxxx WORLD_SIZE=2 RANK=0 python tests/test_i
nternode.py：
[rank7]:[E916 08:53:53.794121333 ProcessGroupNCCL.cpp:633] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600023 milliseconds before timing out.
[rank7]:[E916 08:53:53.794631106 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 7] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank7]:[E916 08:53:53.794661113 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank7]:[E916 08:53:53.794766404 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 7] First PG on this rank to signal dumping.
[rank3]:[E916 08:53:53.813635437 ProcessGroupNCCL.cpp:633] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600042 milliseconds before timing out.
[rank3]:[E916 08:53:53.814173265 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 3] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank3]:[E916 08:53:53.814207094 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank3]:[E916 08:53:53.814325498 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 3] First PG on this rank to signal dumping.
[rank5]:[E916 08:53:53.825187251 ProcessGroupNCCL.cpp:633] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600044 milliseconds before timing out.
[rank5]:[E916 08:53:53.825654273 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 5] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank5]:[E916 08:53:53.825686902 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank5]:[E916 08:53:53.825782986 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 5] First PG on this rank to signal dumping.
[rank0]:[E916 08:53:53.842429275 ProcessGroupNCCL.cpp:633] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600053 milliseconds before timing out.
[rank0]:[E916 08:53:53.842904155 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 0] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank0]:[E916 08:53:53.842936883 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank0]:[E916 08:53:53.843021352 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 0] First PG on this rank to signal dumping.
[rank3]:[E916 08:53:53.843724490 ProcessGroupNCCL.cpp:1764] [PG ID 0 PG GUID 0 Rank 3] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank4]:[E916 08:53:53.843937012 ProcessGroupNCCL.cpp:1703] [PG ID 0 PG GUID 0 Rank 4] Observed flight recorder dump signal from another rank via TCPStore.
[rank4]:[E916 08:53:53.844169601 ProcessGroupNCCL.cpp:1764] [PG ID 0 PG GUID 0 Rank 4] Received a dump signal due to a collective timeout from rank 0 and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank3]:[E916 08:53:53.844278749 ProcessGroupNCCL.cpp:1554] [PG ID 0 PG GUID 0 Rank 3] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank4]:[E916 08:53:53.844607703 ProcessGroupNCCL.cpp:1554] [PG ID 0 PG GUID 0 Rank 4] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank4]:[E916 08:53:53.863061275 ProcessGroupNCCL.cpp:633] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600074 milliseconds before timing out.
[rank4]:[E916 08:53:53.863502669 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 4] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank4]:[E916 08:53:53.863528115 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank4]:[E916 08:53:53.863594849 ProcessGroupNCCL.cpp:685] [Rank 4] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank4]:[E916 08:53:53.863609204 ProcessGroupNCCL.cpp:699] [Rank 4] To avoid data inconsistency, we are taking the entire process down.
[rank4]:[E916 08:53:53.866564418 ProcessGroupNCCL.cpp:1917] [PG ID 1 PG GUID 1 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600074 milliseconds before timing out.
Exception raised from checkTimeout at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:636 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f3e5c932368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f3e5d6950cd in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc08 (0x7f3e5d696dd8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f3e5d697cbd in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xecdb4 (0x7f3e42cc7db4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: + 0x9caa4 (0x7f3ec4f0daa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x129c3c (0x7f3ec4f9ac3c in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 1 PG GUID 1 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600074 milliseconds before timing out.
Exception raised from checkTimeout at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:636 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f3e5c932368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f3e5d6950cd in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc08 (0x7f3e5d696dd8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f3e5d697cbd in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xecdb4 (0x7f3e42cc7db4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: + 0x9caa4 (0x7f3ec4f0daa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x129c3c (0x7f3ec4f9ac3c in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1923 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f3e5c932368 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: + 0xcc036e (0x7f3e5d66736e in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x91669b (0x7f3e5d2bd69b in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: + 0xecdb4 (0x7f3e42cc7db4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #4: + 0x9caa4 (0x7f3ec4f0daa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: + 0x129c3c (0x7f3ec4f9ac3c in /lib/x86_64-linux-gnu/libc.so.6)
[rank2]:[E916 08:53:53.870272601 ProcessGroupNCCL.cpp:633] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600077 milliseconds before timing out.
[rank2]:[E916 08:53:53.870771387 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 2] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank2]:[E916 08:53:53.870804389 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E916 08:53:53.870906102 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 2] First PG on this rank to signal dumping.
[rank6]:[E916 08:53:53.875104669 ProcessGroupNCCL.cpp:633] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=600000) ran for 600086 milliseconds before timing out.
[rank6]:[E916 08:53:53.875581780 ProcessGroupNCCL.cpp:2289] [PG ID 1 PG GUID 1 Rank 6] failure detected by watchdog at work sequence id: 1 PG status: last enqueued work: 1, last completed work: -1
[rank6]:[E916 08:53:53.875610402 ProcessGroupNCCL.cpp:671] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank6]:[E916 08:53:53.875713810 ProcessGroupNCCL.cpp:2124] [PG ID 1 PG GUID 1 Rank 6] First PG on this rank to signal dumping.
[rank1]:[E916 08:53:53.993593628 ProcessGroupNCCL.cpp:1703] [PG ID 0 PG GUID 0 Rank 1] Observed flight recorder dump signal from another rank via TCPStore.
[rank1]:[E916 08:53:53.993783629 ProcessGroupNCCL.cpp:1764] [PG ID 0 PG GUID 0 Rank 1] Received a dump signal due to a collective timeout from rank 11 and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank1]:[E916 08:53:53.994200980 ProcessGroupNCCL.cpp:1554] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
W0916 08:53:53.532000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20510 via signal SIGTERM
W0916 08:53:53.537000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20511 via signal SIGTERM
W0916 08:53:53.538000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20512 via signal SIGTERM
W0916 08:53:53.539000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20513 via signal SIGTERM
[rank5]:[E916 08:53:53.258212774 ProcessGroupNCCL.cpp:1764] [PG ID 0 PG GUID 0 Rank 5] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: -1, last completed NCCL work: -1.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
[rank5]:[E916 08:53:53.258915146 ProcessGroupNCCL.cpp:1554] [PG ID 0 PG GUID 0 Rank 5] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
W0916 08:53:53.541000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20515 via signal SIGTERM
W0916 08:53:53.542000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20516 via signal SIGTERM
W0916 08:53:53.543000 20444 torch/multiprocessing/spawn.py:169] Terminating process 20517 via signal SIGTERM
Traceback (most recent call last):
File "/workspace/deepep-test/juncheng.lai/DeepEP/tests/test_internode.py", line 247, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 196, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 4 terminated with signal SIGABRT
multi nodes hang, what's wrong,I can run test_intranode.py, but i cant run test_internode