deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #328] Timeout caused by 'bench_kineto' #231

New issue
Closed
opened 2026-10-26 09:57:03 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:57:03 +00:00
Owner
Copy link

Originally created by @ZhiyiHu1999 on GitHub (Jul 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/328

Hello! I found sometimes using bench_kineto for kernel execution time estimation may cause timeout. Also, the timeout becomes extremely frequent if barrier_comm_profiling=True is set. I guess in some cases, maybe one kernel (in so many tests) on one GPU is not launched or executed if bench_kineto is used, causing the timeout of other peers.

Thanks a lot!

[tuning] SMs 24, NVL chunk 28, RDMA chunk 12, transmit: 1397.00 us, notify: 54.16 us, BW: 43.18 GB/s (RDMA), 141.40 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 28, RDMA chunk 16, transmit: 1404.00 us, notify: 53.66 us, BW: 42.96 GB/s (RDMA), 140.69 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 28, RDMA chunk 20, transmit: 1419.00 us, notify: 49.28 us, BW: 42.51 GB/s (RDMA), 139.20 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 28, RDMA chunk 24, transmit: 1423.00 us, notify: 56.33 us, BW: 42.39 GB/s (RDMA), 138.81 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 28, RDMA chunk 28, transmit: 1429.00 us, notify: 51.15 us, BW: 42.21 GB/s (RDMA), 138.23 GB/s (NVL) 
DeepEP timeout check failed: rank = 7, thread = 0, value = 0)
DeepEP timeout check failed: rank = 7, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 7, thread = 2, value = 0)
DeepEP timeout check failed: rank = 7, thread = 3, value = 0)
DeepEP timeout check failed: rank = 7, thread = 4, value = 0)
DeepEP timeout check failed: rank = 7, thread = 5, value = 0)
DeepEP timeout check failed: rank = 7, thread = 6, value = 0)
DeepEP timeout check failed: rank = 7, thread = 7, value = 0)
DeepEP timeout check failed: rank = 3, thread = 0, value = 0)
DeepEP timeout check failed: rank = 3, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 3, thread = 2, value = 0)
DeepEP timeout check failed: rank = 3, thread = 3, value = 0)
DeepEP timeout check failed: rank = 3, thread = 4, value = 0)
DeepEP timeout check failed: rank = 3, thread = 5, value = 0)
DeepEP timeout check failed: rank = 3, thread = 6, value = 0)
DeepEP timeout check failed: rank = 3, thread = 7, value = 0)
DeepEP timeout check failed: rank = 2, thread = 0, value = 0)
DeepEP timeout check failed: rank = 2, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 2, thread = 2, value = 0)
DeepEP timeout check failed: rank = 2, thread = 3, value = 0)
DeepEP timeout check failed: rank = 2, thread = 4, value = 0)
DeepEP timeout check failed: rank = 2, thread = 5, value = 0)
DeepEP timeout check failed: rank = 2, thread = 6, value = 0)
DeepEP timeout check failed: rank = 2, thread = 7, value = 0)
DeepEP timeout check failed: rank = 5, thread = 0, value = 0)
DeepEP timeout check failed: rank = 5, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 5, thread = 2, value = 0)
DeepEP timeout check failed: rank = 5, thread = 3, value = 0)
DeepEP timeout check failed: rank = 5, thread = 4, value = 0)
DeepEP timeout check failed: rank = 5, thread = 5, value = 0)
DeepEP timeout check failed: rank = 5, thread = 6, value = 0)
DeepEP timeout check failed: rank = 5, thread = 7, value = 0)
DeepEP timeout check failed: rank = 6, thread = 0, value = 0)
DeepEP timeout check failed: rank = 6, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 6, thread = 2, value = 0)
DeepEP timeout check failed: rank = 6, thread = 3, value = 0)
DeepEP timeout check failed: rank = 6, thread = 4, value = 0)
DeepEP timeout check failed: rank = 6, thread = 5, value = 0)
DeepEP timeout check failed: rank = 6, thread = 6, value = 0)
DeepEP timeout check failed: rank = 6, thread = 7, value = 0)
DeepEP timeout check failed: rank = 0, thread = 0, value = 0)
DeepEP timeout check failed: rank = 0, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 0, thread = 2, value = 0)
DeepEP timeout check failed: rank = 0, thread = 3, value = 0)
DeepEP timeout check failed: rank = 0, thread = 4, value = 0)
DeepEP timeout check failed: rank = 0, thread = 5, value = 0)
DeepEP timeout check failed: rank = 0, thread = 6, value = 0)
DeepEP timeout check failed: rank = 0, thread = 7, value = 0)
DeepEP timeout check failed: rank = 4, thread = 0, value = 0)
DeepEP timeout check failed: rank = 4, thread = 1, value = 1024)
DeepEP timeout check failed: rank = 4, thread = 2, value = 0)
DeepEP timeout check failed: rank = 4, thread = 3, value = 0)
DeepEP timeout check failed: rank = 4, thread = 4, value = 0)
DeepEP timeout check failed: rank = 4, thread = 5, value = 0)
DeepEP timeout check failed: rank = 4, thread = 6, value = 0)
DeepEP timeout check failed: rank = 4, thread = 7, value = 0)
W0725 16:05:28.707000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97566 via signal SIGTERM
W0725 16:05:28.707000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97567 via signal SIGTERM
W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97568 via signal SIGTERM
W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97569 via signal SIGTERM
W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97570 via signal SIGTERM
W0725 16:05:28.709000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97572 via signal SIGTERM
W0725 16:05:28.709000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97573 via signal SIGTERM
Traceback (most recent call last):
  File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 282, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 5 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 243, in test_loop
    test_main(args, i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group)
  File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 188, in test_main
    t, notify_t = bench_kineto(lambda: buffer.dispatch(**tune_args), ('dispatch', 'notify'), barrier_comm_profiling=True, suppress_kineto_output=False)
  File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/utils.py", line 174, in bench_kineto
    prof.step()
  File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 832, in step
    self._transit_action(prev_action, self.current_action)
  File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 866, in _transit_action
    action()
  File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 243, in stop_trace
    self.profiler.__exit__(None, None, None)
  File "/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py", line 381, in __exit__
    device_module.synchronize()
  File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1040, in synchronize
    return torch._C._cuda_synchronize()
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Originally created by @ZhiyiHu1999 on GitHub (Jul 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/328 Hello! I found sometimes using `bench_kineto` for kernel execution time estimation may cause timeout. Also, the timeout becomes extremely frequent if `barrier_comm_profiling=True` is set. I guess in some cases, maybe one kernel (in so many tests) on one GPU is not launched or executed if `bench_kineto` is used, causing the timeout of other peers. Thanks a lot! ```log [tuning] SMs 24, NVL chunk 28, RDMA chunk 12, transmit: 1397.00 us, notify: 54.16 us, BW: 43.18 GB/s (RDMA), 141.40 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 16, transmit: 1404.00 us, notify: 53.66 us, BW: 42.96 GB/s (RDMA), 140.69 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 20, transmit: 1419.00 us, notify: 49.28 us, BW: 42.51 GB/s (RDMA), 139.20 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 24, transmit: 1423.00 us, notify: 56.33 us, BW: 42.39 GB/s (RDMA), 138.81 GB/s (NVL) [tuning] SMs 24, NVL chunk 28, RDMA chunk 28, transmit: 1429.00 us, notify: 51.15 us, BW: 42.21 GB/s (RDMA), 138.23 GB/s (NVL) DeepEP timeout check failed: rank = 7, thread = 0, value = 0) DeepEP timeout check failed: rank = 7, thread = 1, value = 1024) DeepEP timeout check failed: rank = 7, thread = 2, value = 0) DeepEP timeout check failed: rank = 7, thread = 3, value = 0) DeepEP timeout check failed: rank = 7, thread = 4, value = 0) DeepEP timeout check failed: rank = 7, thread = 5, value = 0) DeepEP timeout check failed: rank = 7, thread = 6, value = 0) DeepEP timeout check failed: rank = 7, thread = 7, value = 0) DeepEP timeout check failed: rank = 3, thread = 0, value = 0) DeepEP timeout check failed: rank = 3, thread = 1, value = 1024) DeepEP timeout check failed: rank = 3, thread = 2, value = 0) DeepEP timeout check failed: rank = 3, thread = 3, value = 0) DeepEP timeout check failed: rank = 3, thread = 4, value = 0) DeepEP timeout check failed: rank = 3, thread = 5, value = 0) DeepEP timeout check failed: rank = 3, thread = 6, value = 0) DeepEP timeout check failed: rank = 3, thread = 7, value = 0) DeepEP timeout check failed: rank = 2, thread = 0, value = 0) DeepEP timeout check failed: rank = 2, thread = 1, value = 1024) DeepEP timeout check failed: rank = 2, thread = 2, value = 0) DeepEP timeout check failed: rank = 2, thread = 3, value = 0) DeepEP timeout check failed: rank = 2, thread = 4, value = 0) DeepEP timeout check failed: rank = 2, thread = 5, value = 0) DeepEP timeout check failed: rank = 2, thread = 6, value = 0) DeepEP timeout check failed: rank = 2, thread = 7, value = 0) DeepEP timeout check failed: rank = 5, thread = 0, value = 0) DeepEP timeout check failed: rank = 5, thread = 1, value = 1024) DeepEP timeout check failed: rank = 5, thread = 2, value = 0) DeepEP timeout check failed: rank = 5, thread = 3, value = 0) DeepEP timeout check failed: rank = 5, thread = 4, value = 0) DeepEP timeout check failed: rank = 5, thread = 5, value = 0) DeepEP timeout check failed: rank = 5, thread = 6, value = 0) DeepEP timeout check failed: rank = 5, thread = 7, value = 0) DeepEP timeout check failed: rank = 6, thread = 0, value = 0) DeepEP timeout check failed: rank = 6, thread = 1, value = 1024) DeepEP timeout check failed: rank = 6, thread = 2, value = 0) DeepEP timeout check failed: rank = 6, thread = 3, value = 0) DeepEP timeout check failed: rank = 6, thread = 4, value = 0) DeepEP timeout check failed: rank = 6, thread = 5, value = 0) DeepEP timeout check failed: rank = 6, thread = 6, value = 0) DeepEP timeout check failed: rank = 6, thread = 7, value = 0) DeepEP timeout check failed: rank = 0, thread = 0, value = 0) DeepEP timeout check failed: rank = 0, thread = 1, value = 1024) DeepEP timeout check failed: rank = 0, thread = 2, value = 0) DeepEP timeout check failed: rank = 0, thread = 3, value = 0) DeepEP timeout check failed: rank = 0, thread = 4, value = 0) DeepEP timeout check failed: rank = 0, thread = 5, value = 0) DeepEP timeout check failed: rank = 0, thread = 6, value = 0) DeepEP timeout check failed: rank = 0, thread = 7, value = 0) DeepEP timeout check failed: rank = 4, thread = 0, value = 0) DeepEP timeout check failed: rank = 4, thread = 1, value = 1024) DeepEP timeout check failed: rank = 4, thread = 2, value = 0) DeepEP timeout check failed: rank = 4, thread = 3, value = 0) DeepEP timeout check failed: rank = 4, thread = 4, value = 0) DeepEP timeout check failed: rank = 4, thread = 5, value = 0) DeepEP timeout check failed: rank = 4, thread = 6, value = 0) DeepEP timeout check failed: rank = 4, thread = 7, value = 0) W0725 16:05:28.707000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97566 via signal SIGTERM W0725 16:05:28.707000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97567 via signal SIGTERM W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97568 via signal SIGTERM W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97569 via signal SIGTERM W0725 16:05:28.708000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97570 via signal SIGTERM W0725 16:05:28.709000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97572 via signal SIGTERM W0725 16:05:28.709000 97501 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 97573 via signal SIGTERM Traceback (most recent call last): File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 282, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes) File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 5 terminated with the following error: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 243, in test_loop test_main(args, i, local_rank, num_local_ranks, num_ranks, num_nodes, rank, buffer, group) File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/test_internode.py", line 188, in test_main t, notify_t = bench_kineto(lambda: buffer.dispatch(**tune_args), ('dispatch', 'notify'), barrier_comm_profiling=True, suppress_kineto_output=False) File "/home/nas/zhiyihu/test_origin_ant_deepep/DeepEP/tests/utils.py", line 174, in bench_kineto prof.step() File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 832, in step self._transit_action(prev_action, self.current_action) File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 866, in _transit_action action() File "/opt/conda/lib/python3.10/site-packages/torch/profiler/profiler.py", line 243, in stop_trace self.profiler.__exit__(None, None, None) File "/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py", line 381, in __exit__ device_module.synchronize() File "/opt/conda/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1040, in synchronize return torch._C._cuda_synchronize() RuntimeError: CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
bsakaguchi commented 2026-10-26 09:57:04 +00:00
Author
Owner
Copy link

@GeofferyGeng commented on GitHub (Aug 1, 2025):

Encounter same issue, temporary resolev by change the lines related to bench_kineto to bench.

<!-- gh-comment-id:3142027818 --> @GeofferyGeng commented on GitHub (Aug 1, 2025): Encounter same issue, temporary resolev by change the lines related to bench_kineto to bench.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#231
No description provided.