deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200

New issue
Closed
opened 2026-10-26 09:56:27 +00:00 by bsakaguchi · 10 comments
bsakaguchi commented 2026-10-26 09:56:27 +00:00
Owner
Copy link

Originally created by @yhyang201 on GitHub (Jul 1, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/270

A timeout error occurred when running test_internode.py on four devices. (The two-device test ran successfully.)
The error message is as follows:

[tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 1.22 GB/s (RDMA), 2.45 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 1.41 GB/s (RDMA), 2.84 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 1.63 GB/s (RDMA), 3.29 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 2.00 GB/s (RDMA), 4.03 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 2.19 GB/s (RDMA), 4.41 GB/s (NVL) 
[tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 2.41 GB/s (RDMA), 4.85 GB/s (NVL) 
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_allgather:480: rank 2 nranks 4 size 1536
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_allgather:503: rank 2 nranks 4 size 1536 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER
/workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE
DeepEP dispatch NVL receiver timeout, channel: 8, RDMA: 2, nvl: 2, src NVL: 4, head: 313, tail: 313
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I have already reviewed the following link: https://github.com/deepseek-ai/DeepEP/issues/137, but it seems the methods mentioned there are ineffective.

Please feel free to ask if you need any additional environment information.

Originally created by @yhyang201 on GitHub (Jul 1, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/270 A timeout error occurred when running test_internode.py on four devices. (The two-device test ran successfully.) The error message is as follows: ``` [tuning] SMs 24, NVL chunk 4, RDMA chunk 4: 1.22 GB/s (RDMA), 2.45 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 8: 1.41 GB/s (RDMA), 2.84 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 12: 1.63 GB/s (RDMA), 3.29 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 16: 2.00 GB/s (RDMA), 4.03 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 20: 2.19 GB/s (RDMA), 4.41 GB/s (NVL) [tuning] SMs 24, NVL chunk 4, RDMA chunk 24: 2.41 GB/s (RDMA), 4.85 GB/s (NVL) /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_allgather:480: rank 2 nranks 4 size 1536 /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_allgather:503: rank 2 nranks 4 size 1536 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 4 tag 0 - ENTER /workspace/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 4 tag 2 - DONE DeepEP dispatch NVL receiver timeout, channel: 8, RDMA: 2, nvl: 2, src NVL: 4, head: 313, tail: 313 terminate called after throwing an instance of 'c10::Error' what(): CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` I have already reviewed the following link: https://github.com/deepseek-ai/DeepEP/issues/137, but it seems the methods mentioned there are ineffective. Please feel free to ask if you need any additional environment information.
bsakaguchi commented 2026-10-26 09:56:28 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Jul 2, 2025):

are you try this PR ? #181

<!-- gh-comment-id:3026954421 --> @alpha-baby commented on GitHub (Jul 2, 2025): are you try this PR ? #181
bsakaguchi commented 2026-10-26 09:56:28 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 2, 2025):

We were not able to reproduce this issue on the main branch code. Could you please provide the output of ibv_devinfo and nvidia-smi topo -m, as well as the test log after setting NVSHMEM_DEBUG=INFO?

<!-- gh-comment-id:3026961454 --> @sphish commented on GitHub (Jul 2, 2025): We were not able to reproduce this issue on the main branch code. Could you please provide the output of `ibv_devinfo` and `nvidia-smi topo -m`, as well as the test log after setting `NVSHMEM_DEBUG=INFO`?
bsakaguchi commented 2026-10-26 09:56:29 +00:00
Author
Owner
Copy link

@yhyang201 commented on GitHub (Jul 2, 2025):

are you try this PR ? #181

Thank you very much for your attention!

I believe I tried this PR because the commit I used is from the main branch:

commit a15faa9ff07a2942c7f27bd3aa4615909a53cb9f (HEAD)  
Author: Chenggang Zhao <chenggangz@deepseek.com>  
Date:   Tue Jun 24 09:21:35 2025 +0800
<!-- gh-comment-id:3027216279 --> @yhyang201 commented on GitHub (Jul 2, 2025): > are you try this PR ? [#181](https://github.com/deepseek-ai/DeepEP/pull/181) Thank you very much for your attention! I believe I tried this PR because the commit I used is from the main branch: ```` commit a15faa9ff07a2942c7f27bd3aa4615909a53cb9f (HEAD) Author: Chenggang Zhao <chenggangz@deepseek.com> Date: Tue Jun 24 09:21:35 2025 +0800 ````
bsakaguchi commented 2026-10-26 09:56:29 +00:00
Author
Owner
Copy link

@yhyang201 commented on GitHub (Jul 2, 2025):

We were not able to reproduce this issue on the main branch code. Could you please provide the output of ibv_devinfo and nvidia-smi topo -m, as well as the test log after setting NVSHMEM_DEBUG=INFO?

Thank you very much for your attention!

I have provided the following information at this link: https://gist.github.com/yhyang201/c6e7a0afe4eca3abe7291e3274c4360f

  1. ibv_devinfo
  2. nvidia-smi topo -m
  3. Environment Variables
  4. Test log after setting NVSHMEM_DEBUG=INFO
  5. The commit I used
<!-- gh-comment-id:3027221454 --> @yhyang201 commented on GitHub (Jul 2, 2025): > We were not able to reproduce this issue on the main branch code. Could you please provide the output of `ibv_devinfo` and `nvidia-smi topo -m`, as well as the test log after setting `NVSHMEM_DEBUG=INFO`? Thank you very much for your attention! I have provided the following information at this link: https://gist.github.com/yhyang201/c6e7a0afe4eca3abe7291e3274c4360f 1. ibv\_devinfo 2. nvidia-smi topo -m 3. Environment Variables 4. Test log after setting NVSHMEM\_DEBUG=INFO 5. The commit I used
bsakaguchi commented 2026-10-26 09:56:29 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Jul 3, 2025):

It looks like your testing method is correct, and NVSHMEM is also running fine. I suspect this issue might be related to the bonded NIC. Could you add me on WeChat: Sphizzz? Maybe we can debug this together.

<!-- gh-comment-id:3031716804 --> @sphish commented on GitHub (Jul 3, 2025): It looks like your testing method is correct, and NVSHMEM is also running fine. I suspect this issue might be related to the bonded NIC. Could you add me on WeChat: Sphizzz? Maybe we can debug this together.
bsakaguchi commented 2026-10-26 09:56:29 +00:00
Author
Owner
Copy link

@liweiqing1997 commented on GitHub (Jul 8, 2025):

My be same error?

[rank28]:[W708 04:17:38.756339127 CUDAGuardImpl.h:119] Warning: CUDA warning: unspecified launch failure (function destroyEvent)
[rank28]:[W708 04:17:38.756353798 CUDAGuardImpl.h:120] Warning: CUDA warning: CUDA-capable device(s) is/are busy or unavailable (function destroyEvent)
terminate called after throwing an instance of 'c10::Error'
, psutil._psutil_linux, psutil._psutil_posix, zmq.backend.cython._zmq, PIL._imaging, setproctitle._setproctitle, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft what(): CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f59ef3785e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe0 (0x7f59ef30d4a2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7f59ef7a5422 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x1e9a9 (0x7f59ef76d9a9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x20060 (0x7f59ef76f060 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x2028c (0x7f59ef76f28c in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #6: + 0x457242 (0x7f59e7057242 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f59ef352f39 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: + 0x7186d8 (0x7f59e73186d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #9: + 0x718af1 (0x7f59e7318af1 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #10: + 0x181390 (0x561755181390 in sglang::scheduler_DP28_TP28)
frame #11: + 0x1946c8 (0x5617551946c8 in sglang::scheduler_DP28_TP28)
frame #12: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #13: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #14: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #15: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #16: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #17: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #18: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #19: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #20: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #21: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #22: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #23: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #24: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #25: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #26: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #27: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #28: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #29: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #30: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #31: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #32: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #33: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #34: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #35: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #36: + 0x181390 (0x561755181390 in sglang::scheduler_DP28_TP28)
frame #37: + 0x1946c8 (0x5617551946c8 in sglang::scheduler_DP28_TP28)
frame #38: + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28)
frame #39: _PyEval_EvalFrameDefault + 0x2bfe (0x561755177e6e in sglang::scheduler_DP28_TP28)
frame #40: + 0x198b81 (0x561755198b81 in sglang::scheduler_DP28_TP28)
frame #41: _PyEval_EvalFrameDefault + 0x2a7b (0x561755177ceb in sglang::scheduler_DP28_TP28)
frame #42: _PyFunction_Vectorcall + 0x7c (0x56175518b6ac in sglang::scheduler_DP28_TP28)
frame #43: _PyEval_EvalFrameDefault + 0x807 (0x561755175a77 in sglang::scheduler_DP28_TP28)
frame #44: _PyFunction_Vectorcall + 0x7c (0x56175518b6ac in sglang::scheduler_DP28_TP28)
frame #45: _PyEval_EvalFrameDefault + 0x807 (0x561755175a77 in sglang::scheduler_DP28_TP28)
frame #46: + 0x198b81 (0x561755198b81 in sglang::scheduler_DP28_TP28)
frame #47: + 0x2ad9fa (0x5617552ad9fa in sglang::scheduler_DP28_TP28)
frame #48: + 0x2a3598 (0x5617552a3598 in sglang::scheduler_DP28_TP28)
frame #49: + 0x94ac3 (0x7f5b72094ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: + 0x126850 (0x7f5b72126850 in /lib/x86_64-linux-gnu/libc.so.6)

Fatal Python error: Aborted

Thread 0x00007f34abfff640 (most recent call first):
File "/usr/lib/python3.10/threading.py", line 324 in wait
File "/usr/lib/python3.10/threading.py", line 607 in wait
File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f34b5ffe640 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1987 in watchdog_thread
File "/usr/lib/python3.10/threading.py", line 953 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007f34b9fff640 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 131 in forward_thread_func
File "/usr/lib/python3.10/threading.py", line 953 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f4981fff640 (most recent call first):
File "/usr/lib/python3.10/threading.py", line 324 in wait
File "/usr/lib/python3.10/threading.py", line 607 in wait
File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f56f78ff640 (most recent call first):
File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 55 in _recv_msg
File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 191 in _read_thread
File "/usr/lib/python3.10/threading.py", line 953 in run
File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b71fff280 (most recent call first):
File "/usr/lib/python3.10/threading.py", line 320 in wait
File "/usr/lib/python3.10/threading.py", line 607 in wait
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 203 in resolve_last_batch_result
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1757 in process_batch_result
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 776 in event_loop_overlap
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2645 in run_scheduler_process
File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run
File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main
File "", line 1 in
, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, charset_normalizer.md, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, multidict._multidict, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, uvloop.loop, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, msgspec._core, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, psutil._psutil_linux, scipy.linalg._decomp_update, psutil._psutil_posix, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, zmq.backend.cython._zmq, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, , PIL._imagingscipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, , setproctitle._setproctitlescipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, , scipy.optimize._group_columnsyaml._yaml, scipy._lib.messagestream, scipy.optimize._trlib._trlib, , scipy.optimize._lbfgsbregex._regex, _moduleTNC, scipy.optimize._moduleTNC, markupsafe._speedups, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, PIL._imagingft, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, cuda_utils, numpy.random._common, numpy.random.bit_generator, __triton_launcher, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64 (total: 108)
, numpy.random._sfc64, numpy.random._generator, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, cuda_utils, __triton_launcher (total: 108)

<!-- gh-comment-id:3047312159 --> @liweiqing1997 commented on GitHub (Jul 8, 2025): My be same error? [rank28]:[W708 04:17:38.756339127 CUDAGuardImpl.h:119] Warning: CUDA warning: unspecified launch failure (function destroyEvent) [rank28]:[W708 04:17:38.756353798 CUDAGuardImpl.h:120] Warning: CUDA warning: CUDA-capable device(s) is/are busy or unavailable (function destroyEvent) terminate called after throwing an instance of 'c10::Error' , psutil._psutil_linux, psutil._psutil_posix, zmq.backend.cython._zmq, PIL._imaging, setproctitle._setproctitle, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft what(): CUDA error: unspecified launch failure CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f59ef3785e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7f59ef30d4a2 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7f59ef7a5422 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so) frame #3: <unknown function> + 0x1e9a9 (0x7f59ef76d9a9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so) frame #4: <unknown function> + 0x20060 (0x7f59ef76f060 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so) frame #5: <unknown function> + 0x2028c (0x7f59ef76f28c in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so) frame #6: <unknown function> + 0x457242 (0x7f59e7057242 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so) frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f59ef352f39 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #8: <unknown function> + 0x7186d8 (0x7f59e73186d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so) frame #9: <unknown function> + 0x718af1 (0x7f59e7318af1 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so) frame #10: <unknown function> + 0x181390 (0x561755181390 in sglang::scheduler_DP28_TP28) frame #11: <unknown function> + 0x1946c8 (0x5617551946c8 in sglang::scheduler_DP28_TP28) frame #12: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #13: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #14: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #15: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #16: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #17: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #18: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #19: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #20: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #21: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #22: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #23: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #24: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #25: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #26: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #27: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #28: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #29: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #30: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #31: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #32: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #33: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #34: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #35: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #36: <unknown function> + 0x181390 (0x561755181390 in sglang::scheduler_DP28_TP28) frame #37: <unknown function> + 0x1946c8 (0x5617551946c8 in sglang::scheduler_DP28_TP28) frame #38: <unknown function> + 0x1946dc (0x5617551946dc in sglang::scheduler_DP28_TP28) frame #39: _PyEval_EvalFrameDefault + 0x2bfe (0x561755177e6e in sglang::scheduler_DP28_TP28) frame #40: <unknown function> + 0x198b81 (0x561755198b81 in sglang::scheduler_DP28_TP28) frame #41: _PyEval_EvalFrameDefault + 0x2a7b (0x561755177ceb in sglang::scheduler_DP28_TP28) frame #42: _PyFunction_Vectorcall + 0x7c (0x56175518b6ac in sglang::scheduler_DP28_TP28) frame #43: _PyEval_EvalFrameDefault + 0x807 (0x561755175a77 in sglang::scheduler_DP28_TP28) frame #44: _PyFunction_Vectorcall + 0x7c (0x56175518b6ac in sglang::scheduler_DP28_TP28) frame #45: _PyEval_EvalFrameDefault + 0x807 (0x561755175a77 in sglang::scheduler_DP28_TP28) frame #46: <unknown function> + 0x198b81 (0x561755198b81 in sglang::scheduler_DP28_TP28) frame #47: <unknown function> + 0x2ad9fa (0x5617552ad9fa in sglang::scheduler_DP28_TP28) frame #48: <unknown function> + 0x2a3598 (0x5617552a3598 in sglang::scheduler_DP28_TP28) frame #49: <unknown function> + 0x94ac3 (0x7f5b72094ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #50: <unknown function> + 0x126850 (0x7f5b72126850 in /lib/x86_64-linux-gnu/libc.so.6) Fatal Python error: Aborted Thread 0x00007f34abfff640 (most recent call first): File "/usr/lib/python3.10/threading.py", line 324 in wait File "/usr/lib/python3.10/threading.py", line 607 in wait File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f34b5ffe640 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1987 in watchdog_thread File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Current thread 0x00007f34b9fff640 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 131 in forward_thread_func File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f4981fff640 (most recent call first): File "/usr/lib/python3.10/threading.py", line 324 in wait File "/usr/lib/python3.10/threading.py", line 607 in wait File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f56f78ff640 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 55 in _recv_msg File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 191 in _read_thread File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f5b71fff280 (most recent call first): File "/usr/lib/python3.10/threading.py", line 320 in wait File "/usr/lib/python3.10/threading.py", line 607 in wait File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 203 in resolve_last_batch_result File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 1757 in process_batch_result File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 776 in event_loop_overlap File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context File "/usr/local/lib/python3.10/dist-packages/sglang/srt/managers/scheduler.py", line 2645 in run_scheduler_process File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main File "<string>", line 1 in <module> , numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, charset_normalizer.md, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, multidict._multidict, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, uvloop.loop, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, msgspec._core, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, psutil._psutil_linux, scipy.linalg._decomp_update, psutil._psutil_posix, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, zmq.backend.cython._zmq, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, , PIL._imagingscipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, , setproctitle._setproctitlescipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, , scipy.optimize._group_columnsyaml._yaml, scipy._lib.messagestream, scipy.optimize._trlib._trlib, , scipy.optimize._lbfgsbregex._regex, _moduleTNC, scipy.optimize._moduleTNC, markupsafe._speedups, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, PIL._imagingft, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, cuda_utils, numpy.random._common, numpy.random.bit_generator, __triton_launcher, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64 (total: 108) , numpy.random._sfc64, numpy.random._generator, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, cuda_utils, __triton_launcher (total: 108)
bsakaguchi commented 2026-10-26 09:56:30 +00:00
Author
Owner
Copy link

@pengchengneo commented on GitHub (Jul 9, 2025):

@sphish @yhyang201 hi, I also encountered with this error , can u share the method with this issue?

<!-- gh-comment-id:3051678961 --> @pengchengneo commented on GitHub (Jul 9, 2025): @sphish @yhyang201 hi, I also encountered with this error , can u share the method with this issue?
bsakaguchi commented 2026-10-26 09:56:30 +00:00
Author
Owner
Copy link

@bighammerExt commented on GitHub (Aug 21, 2025):

I encountered the same error and solved by git checkout 77f97f79bd && NVSHMEM_DIR=/sgl-workspace/nvshmem/install python setup.py install, but i don't know why

<!-- gh-comment-id:3209426610 --> @bighammerExt commented on GitHub (Aug 21, 2025): I encountered the same error and solved by git checkout 77f97f79bda051d991aa9681c04138007faa0366 && NVSHMEM_DIR=/sgl-workspace/nvshmem/install python setup.py install, but i don't know why
bsakaguchi commented 2026-10-26 09:56:31 +00:00
Author
Owner
Copy link

@fungaren commented on GitHub (Sep 9, 2025):

I got the same error, after upgrading to the latest DeepEP commit (main branch) this problem has gone.

<!-- gh-comment-id:3270282439 --> @fungaren commented on GitHub (Sep 9, 2025): I got the same error, after upgrading to the latest DeepEP commit (main branch) this problem has gone.
bsakaguchi commented 2026-10-26 09:56:31 +00:00
Author
Owner
Copy link

@yhyang201 commented on GitHub (Sep 9, 2025):

close this issue for now

<!-- gh-comment-id:3270287731 --> @yhyang201 commented on GitHub (Sep 9, 2025): close this issue for now
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#200
No description provided.