[GH-ISSUE #15] Test test_low_latency.py failed #14

New issue

Closed

opened 2026-10-26 09:52:26 +00:00 by bsakaguchi · 15 comments

bsakaguchi commented 2026-10-26 09:52:26 +00:00

Owner

Copy link

Originally created by @Nekofish-L on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/15

I encountered an issue while running the test python test_low_latency.py. The test is failing with the following error:

Allocating buffer size: 2116.2912 MB ...
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
malloc(): unsorted double linked list corrupted
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
corrupted double-linked list
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
malloc(): unsorted double linked list corrupted
[/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch 
malloc(): unsorted double linked list corrupted
malloc(): unsorted double linked list corrupted
malloc(): unsorted double linked list corrupted
malloc(): unsorted double linked list corrupted
W0225 20:24:08.312000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54153 via signal SIGTERM
W0225 20:24:08.313000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54154 via signal SIGTERM
W0225 20:24:08.313000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54157 via signal SIGTERM
W0225 20:24:08.314000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54158 via signal SIGTERM

However, another related test tests/test_intranode.py runs successfully and produces the expected output.

Environment:

• TencentOS Server release 3.1 (CentOS)
• H20 * 8
• CUDA 12.3

Originally created by @Nekofish-L on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/15 I encountered an issue while running the test `python test_low_latency.py`. The test is failing with the following error: ``` Allocating buffer size: 2116.2912 MB ... [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch malloc(): unsorted double linked list corrupted [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch corrupted double-linked list [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch malloc(): unsorted double linked list corrupted [/root/nvshmem_src/src/host/stream/coll/barrier/barrier.cu:36] cuda failed with too many blocks in cooperative launch malloc(): unsorted double linked list corrupted malloc(): unsorted double linked list corrupted malloc(): unsorted double linked list corrupted malloc(): unsorted double linked list corrupted W0225 20:24:08.312000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54153 via signal SIGTERM W0225 20:24:08.313000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54154 via signal SIGTERM W0225 20:24:08.313000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54157 via signal SIGTERM W0225 20:24:08.314000 54083 /data/root/miniconda3/lib/python3.12/site-packages/torch/multiprocessing/spawn.py:169] Terminating process 54158 via signal SIGTERM ``` However, another related test `tests/test_intranode.py` runs successfully and produces the expected output. **Environment:** - TencentOS Server release 3.1 (CentOS) - H20 * 8 - CUDA 12.3

bsakaguchi closed this issue 2026-10-26 09:52:26 +00:00

bsakaguchi commented 2026-10-26 09:52:27 +00:00

Author

Owner

Copy link

@haswelliris commented on GitHub (Feb 25, 2025):

Please provide additional information, including:

• Network Card: You can obtain this information using the command ibstatus.
• GPU Topology: Use the command nvidia-smi topo -m to retrieve this information.

Furthermore, please share the results of the following reports, if available:

• nvshmem perftest Report:

Alternatively, you can simply provide the output of the command:

/path/to/nvshmem/dir/bin/nvshmem-info -a

<!-- gh-comment-id:2681983329 --> @haswelliris commented on GitHub (Feb 25, 2025): Please provide additional information, including: - **Network Card**: You can obtain this information using the command `ibstatus`. - **GPU Topology**: Use the command `nvidia-smi topo -m` to retrieve this information. Furthermore, please share the results of the following reports, if available: - **`nvshmem perftest` Report**: Alternatively, you can simply provide the output of the command: ```bash /path/to/nvshmem/dir/bin/nvshmem-info -a ```

bsakaguchi commented 2026-10-26 09:52:27 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Feb 25, 2025):

There is a line num_tokens, hidden, num_topk, num_experts = 128, 7168, 8, 288 in the test, indicating the SM used is num_experts / 3 = 96 (refer to https://github.com/deepseek-ai/DeepEP/blob/main/csrc/kernels/internode_ll.cu#L305C26-L305C63), which exceeds the H20 SM count.

You can try to reduce the number of experts.

<!-- gh-comment-id:2682343603 --> @LyricZhao commented on GitHub (Feb 25, 2025): There is a line `num_tokens, hidden, num_topk, num_experts = 128, 7168, 8, 288` in the test, indicating the SM used is `num_experts / 3 = 96` (refer to `https://github.com/deepseek-ai/DeepEP/blob/main/csrc/kernels/internode_ll.cu#L305C26-L305C63`), which exceeds the H20 SM count. You can try to reduce the number of experts.

bsakaguchi commented 2026-10-26 09:52:27 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Feb 25, 2025):

We may add H20 compatible changes later.

<!-- gh-comment-id:2682345549 --> @LyricZhao commented on GitHub (Feb 25, 2025): We may add H20 compatible changes later.

bsakaguchi commented 2026-10-26 09:52:27 +00:00

Author

Owner

Copy link

@MengYu10151 commented on GitHub (Mar 10, 2025):

We may add H20 compatible changes later.

Hi Lyric, in addition to reducing the number of experts, is it possible to enable 288 experts by changing other parameters? and I also wonder if the H20 support change will be launched soon? Thanks!

<!-- gh-comment-id:2709683580 --> @MengYu10151 commented on GitHub (Mar 10, 2025): > We may add H20 compatible changes later. Hi Lyric, in addition to reducing the number of experts, is it possible to enable 288 experts by changing other parameters? and I also wonder if the H20 support change will be launched soon? Thanks!

bsakaguchi commented 2026-10-26 09:52:28 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Mar 10, 2025):

is it possible to enable 288 experts by changing other parameters

It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine)

constexpr int kNumWarpsPerGroup = 10;
constexpr int kNumWarpGroups = 3;

into

constexpr int kNumWarpsPerGroup = 8;
constexpr int kNumWarpGroups = 4;

and remove EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);.

I don't have H20 for testing, if the performance is not good, please report and file an issue :)

<!-- gh-comment-id:2709715782 --> @LyricZhao commented on GitHub (Mar 10, 2025): > is it possible to enable 288 experts by changing other parameters It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine) ```cpp constexpr int kNumWarpsPerGroup = 10; constexpr int kNumWarpGroups = 3; ``` into ```cpp constexpr int kNumWarpsPerGroup = 8; constexpr int kNumWarpGroups = 4; ``` and remove `EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);`. I don't have H20 for testing, if the performance is not good, please report and file an issue :)

bsakaguchi commented 2026-10-26 09:52:28 +00:00

Author

Owner

Copy link

@alpha-baby commented on GitHub (Mar 24, 2025):

is it possible to enable 288 experts by changing other parameters

It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine)

constexpr int kNumWarpsPerGroup = 10;
constexpr int kNumWarpGroups = 3;
into

constexpr int kNumWarpsPerGroup = 8;
constexpr int kNumWarpGroups = 4;
and remove EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);.

I don't have H20 for testing, if the performance is not good, please report and file an issue :)

In my H20 machine, I used a single server to run the test : tests/test_low_latency.py.
result:

[rank 4] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.56 us, min_t=885.41 us, max_t=916.00 us
[rank 2] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.39 us, min_t=885.25 us, max_t=918.11 us
[rank 5] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.30 us, min_t=881.60 us, max_t=912.86 us
[rank 1] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.89 us, min_t=885.54 us, max_t=923.97 us
[rank 3] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.87 us, min_t=886.46 us, max_t=920.22 us
[rank 0] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.66 us, min_t=884.74 us, max_t=922.21 us
[rank 7] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.98 us, min_t=882.98 us, max_t=916.29 us
[rank 6] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.56 us, min_t=879.04 us, max_t=912.61 us
[rank 1] Dispatch bandwidth: 24.30 GB/s, avg_t=309.08 us | Combine bandwidth: 24.95 GB/s, avg_t=582.56 us
[rank 0] Dispatch bandwidth: 24.50 GB/s, avg_t=306.61 us | Combine bandwidth: 24.86 GB/s, avg_t=584.66 us
[rank 7] Dispatch bandwidth: 24.51 GB/s, avg_t=306.45 us | Combine bandwidth: 24.85 GB/s, avg_t=585.01 us
[rank 3] Dispatch bandwidth: 24.67 GB/s, avg_t=304.43 us | Combine bandwidth: 24.72 GB/s, avg_t=588.12 us
[rank 6] Dispatch bandwidth: 24.61 GB/s, avg_t=305.22 us | Combine bandwidth: 24.73 GB/s, avg_t=587.85 us
[rank 4] Dispatch bandwidth: 25.01 GB/s, avg_t=300.36 us | Combine bandwidth: 24.64 GB/s, avg_t=590.02 us
[rank 2] Dispatch bandwidth: 25.10 GB/s, avg_t=299.25 us | Combine bandwidth: 24.52 GB/s, avg_t=592.90 us
[rank 5] Dispatch bandwidth: 24.82 GB/s, avg_t=302.67 us | Combine bandwidth: 24.71 GB/s, avg_t=588.28 us
[rank 5] Dispatch send/recv time: 41.94 us | Combine send/recv time: 55.67 us
[rank 4] Dispatch send/recv time: 389.45 us | Combine send/recv time: 535.86 us
[rank 1] Dispatch send/recv time: 1369.67 us | Combine send/recv time: 1706.07 us
[rank 3] Dispatch send/recv time: 148.48 us | Combine send/recv time: 199.69 us
[rank 6] Dispatch send/recv time: 419.30 us | Combine send/recv time: 614.06 us
[rank 7] Dispatch send/recv time: 29.74 us | Combine send/recv time: 30.63 us
[rank 2] Dispatch send/recv time: 29.31 us | Combine send/recv time: 30.06 us
[rank 0] Dispatch send/recv time: 28.72 us | Combine send/recv time: 30.39 us

but the program for testing will SIGSEGV.

log:

/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 0 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 5 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 4 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 3 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 6 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 1 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 7 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 8 tag 0 - ENTER
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 7 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 3 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 1 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 5 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 0 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 6 nranks 8 tag 3 - DONE
/root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 4 nranks 8 tag 3 - DONE
W0324 21:01:46.298366 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9089 via signal SIGTERM
W0324 21:01:46.298805 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9090 via signal SIGTERM
W0324 21:01:46.298865 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9091 via signal SIGTERM
W0324 21:01:46.299215 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9093 via signal SIGTERM
W0324 21:01:46.299256 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9094 via signal SIGTERM
W0324 21:01:46.299292 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9095 via signal SIGTERM
Traceback (most recent call last):
  File "tests/test_low_latency.py", line 172, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 170, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 7 terminated with signal SIGSEGV

gdb debug coredump file:

#0  0x00007fb10c4e6337 in ibv_dealloc_pd () from /lib64/libibverbs.so
#1  0x00007fb10c0de560 in nvshmemt_ibrc_finalize(nvshmem_transport*) () from /opt/nvshmem/lib/nvshmem_transport_ibrc.so.3
#2  0x00007fb10dc05342 in nvshmemi_transport_finalize(nvshmemi_state_dec*) () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so
#3  0x00007fb10daaf809 in nvshmemid_hostlib_finalize () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so
#4  0x00007fb10dba3078 in nvshmemi_finalize () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so
#5  0x00007fb10da594da in deep_ep::Buffer::~Buffer (this=0x727fcc0, __in_chrg=<optimized out>) at /home/root/DeepEP/csrc/deep_ep.cpp:106
#6  0x00007fb10da6aeba in std::default_delete<deep_ep::Buffer>::operator() (this=<optimized out>, __ptr=0x727fcc0) at /usr/include/c++/10/bits/unique_ptr.h:79
#7  std::unique_ptr<deep_ep::Buffer, std::default_delete<deep_ep::Buffer> >::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/10/bits/unique_ptr.h:361
#8  pybind11::class_<deep_ep::Buffer>::dealloc (v_h=...) at /opt/conda/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h:1939
#9  0x00007fb2670ad753 in pybind11::detail::clear_instance(_object*) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#10 0x00007fb2670ae6d1 in pybind11_object_dealloc () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#11 0x00000000004d3dae in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207
#12 _Py_DECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478
#13 _Py_XDECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:541
#14 dict_dealloc (mp=0x7fb111649d00) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1990
#15 0x00000000004f9b46 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207
#16 _Py_DECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478
#17 subtype_dealloc (self=0x7fb11162a460) at /usr/local/src/conda/python-3.8.20/Modules/condvar.h:1277
#18 0x00000000004e0c20 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207
#19 _Py_DECREF (op=<optimized out>, lineno=430, filename=0x5f56e0 "/croot/python-split_1727968608067/work/Objects/frameobject.c") at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478
#20 frame_dealloc (f=0x62fe7f0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/genobject.c:430
#21 0x00000000004e84b8 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207
#22 _Py_DECREF (op=0x62fe7f0, op@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=292, lineno@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<synthetic pointer>,
    filename@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478
#23 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=2, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>)
    at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:292
#24 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411
#25 0x00000000004f7b0a in PyVectorcall_Call (kwargs=0x0, tuple=<optimized out>, callable=0x7fb111632d30) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1802
#26 PyObject_Call (callable=0x7fb111632d30, args=<optimized out>, kwargs=0x0) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:228
#27 0x00000000004da48a in do_call_core (kwdict=0x0, callargs=0x7fb269006b00, func=0x7fb111632d30, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:5010
#28 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3559
#29 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738
#30 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=4, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>)
    at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284
#31 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb111bf2cd8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411
#32 0x00000000004f7b0a in PyVectorcall_Call (kwargs=0x7fb11162b9c0, tuple=<optimized out>, callable=0x7fb112d70c10) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1802
#33 PyObject_Call (callable=0x7fb112d70c10, args=<optimized out>, kwargs=0x7fb11162b9c0) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:228
#34 0x00000000004da48a in do_call_core (kwdict=0x7fb11162b9c0, callargs=0x7fb111bf2cc0, func=0x7fb112d70c10, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:5010
#35 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3559
#36 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738
#37 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>)
    at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284
#38 _PyFunction_Vectorcall (func=<optimized out>, stack=0x595b220, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411
#39 0x00000000004d8ac2 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x595b220, callable=0x7fb2691d09d0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127
#40 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963
#41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3486
#42 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741
#43 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0xf8c9c8, kwcount=<optimized out>, kwstep=1, defs=0x7fb2691b3178, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x7fb2693c4730, qualname=0x7fb2691b27b0)
    at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298
#44 0x00000000004e851c in _PyFunction_Vectorcall (func=<optimized out>, stack=0xf8c9b8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:436
#45 0x00000000004d8ac2 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0xf8c9b8, callable=0x7fb2691b83a0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127
#46 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963
#47 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3486
#48 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738
#49 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=2, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>)
    at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284
#50 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb269291608, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411
#51 0x00000000004d87a9 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fb269291608, callable=0x7fb268fe3940) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127
#52 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963
#53 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3500
#54 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741
#55 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7fb26938ad50, kwcount=<optimized out>, kwstep=1, defs=0x7fb2691fb118, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x7fb269285970, qualname=0x7fb269285970)
    at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298
#56 0x00000000004e851c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb26938ad50, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:436
#57 0x00000000004d9566 in _PyObject_Vectorcall (kwnames=0x7fb26923aa80, nargsf=<optimized out>, args=<optimized out>, callable=0x7fb268fe38b0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127
#58 call_function (kwnames=0x7fb26923aa80, oparg=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963
#59 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3515
#60 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741
#61 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298
#62 0x0000000000586079 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4327
#63 0x000000000058603b in PyEval_EvalCode (co=co@entry=0x7fb269344870, globals=globals@entry=0x7fb269359fc0, locals=locals@entry=0x7fb269359fc0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:718
#64 0x00000000005a5da1 in run_eval_code_obj (co=0x7fb269344870, globals=0x7fb269359fc0, locals=0x7fb269359fc0) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1166
#65 0x00000000005a4daf in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7fb269359fc0, locals=0x7fb269359fc0, flags=<optimized out>, arena=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1188
#66 0x00000000005a27bb in PyRun_StringFlags (str=<optimized out>, start=257, globals=0x7fb269359fc0, locals=0x7fb269359fc0, flags=0x7fffb85d7808) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1061
#67 0x0000000000450b4b in PyRun_SimpleStringFlags (command=0x7fb2692e29d0 "from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=4, pipe_handle=14)\n", flags=flags@entry=0x7fffb85d7808) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:486
#68 0x00000000005a1bb8 in pymain_run_command (cf=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, command=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/work/build-static/python.c:264
#69 pymain_run_python (exitcode=0x7fffb85d7800) at /croot/python-split_1727968608067/work/build-static/python.c:611
#70 Py_RunMain () at /croot/python-split_1727968608067/work/build-static/python.c:699
#71 0x000000000057a0d9 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /croot/python-split_1727968608067/work/build-static/python.c:1131
#72 0x00007fb26979e192 in __libc_start_main () from /lib64/libc.so.6
#73 0x0000000000579f8d in _start () at /usr/local/src/conda/python-3.8.20/Objects/clinic/import.c.h:521

Is this because verbs has a bug?

<!-- gh-comment-id:2748119001 --> @alpha-baby commented on GitHub (Mar 24, 2025): > > is it possible to enable 288 experts by changing other parameters > > It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine) > > constexpr int kNumWarpsPerGroup = 10; > constexpr int kNumWarpGroups = 3; > into > > constexpr int kNumWarpsPerGroup = 8; > constexpr int kNumWarpGroups = 4; > and remove `EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);`. > > I don't have H20 for testing, if the performance is not good, please report and file an issue :) In my H20 machine, I used a single server to run the test : tests/test_low_latency.py. result: ``` [rank 4] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.56 us, min_t=885.41 us, max_t=916.00 us [rank 2] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.39 us, min_t=885.25 us, max_t=918.11 us [rank 5] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.30 us, min_t=881.60 us, max_t=912.86 us [rank 1] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.89 us, min_t=885.54 us, max_t=923.97 us [rank 3] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.87 us, min_t=886.46 us, max_t=920.22 us [rank 0] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.66 us, min_t=884.74 us, max_t=922.21 us [rank 7] Dispatch + combine bandwidth: 24.53 GB/s, avg_t=898.98 us, min_t=882.98 us, max_t=916.29 us [rank 6] Dispatch + combine bandwidth: 24.54 GB/s, avg_t=898.56 us, min_t=879.04 us, max_t=912.61 us [rank 1] Dispatch bandwidth: 24.30 GB/s, avg_t=309.08 us | Combine bandwidth: 24.95 GB/s, avg_t=582.56 us [rank 0] Dispatch bandwidth: 24.50 GB/s, avg_t=306.61 us | Combine bandwidth: 24.86 GB/s, avg_t=584.66 us [rank 7] Dispatch bandwidth: 24.51 GB/s, avg_t=306.45 us | Combine bandwidth: 24.85 GB/s, avg_t=585.01 us [rank 3] Dispatch bandwidth: 24.67 GB/s, avg_t=304.43 us | Combine bandwidth: 24.72 GB/s, avg_t=588.12 us [rank 6] Dispatch bandwidth: 24.61 GB/s, avg_t=305.22 us | Combine bandwidth: 24.73 GB/s, avg_t=587.85 us [rank 4] Dispatch bandwidth: 25.01 GB/s, avg_t=300.36 us | Combine bandwidth: 24.64 GB/s, avg_t=590.02 us [rank 2] Dispatch bandwidth: 25.10 GB/s, avg_t=299.25 us | Combine bandwidth: 24.52 GB/s, avg_t=592.90 us [rank 5] Dispatch bandwidth: 24.82 GB/s, avg_t=302.67 us | Combine bandwidth: 24.71 GB/s, avg_t=588.28 us [rank 5] Dispatch send/recv time: 41.94 us | Combine send/recv time: 55.67 us [rank 4] Dispatch send/recv time: 389.45 us | Combine send/recv time: 535.86 us [rank 1] Dispatch send/recv time: 1369.67 us | Combine send/recv time: 1706.07 us [rank 3] Dispatch send/recv time: 148.48 us | Combine send/recv time: 199.69 us [rank 6] Dispatch send/recv time: 419.30 us | Combine send/recv time: 614.06 us [rank 7] Dispatch send/recv time: 29.74 us | Combine send/recv time: 30.63 us [rank 2] Dispatch send/recv time: 29.31 us | Combine send/recv time: 30.06 us [rank 0] Dispatch send/recv time: 28.72 us | Combine send/recv time: 30.39 us ``` but the program for testing will SIGSEGV. log: ``` /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 0 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 5 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 4 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 3 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 6 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 1 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 7 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 8 tag 0 - ENTER /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 7 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 3 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 1 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 5 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 0 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 6 nranks 8 tag 3 - DONE /root/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 4 nranks 8 tag 3 - DONE W0324 21:01:46.298366 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9089 via signal SIGTERM W0324 21:01:46.298805 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9090 via signal SIGTERM W0324 21:01:46.298865 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9091 via signal SIGTERM W0324 21:01:46.299215 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9093 via signal SIGTERM W0324 21:01:46.299256 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9094 via signal SIGTERM W0324 21:01:46.299292 140365217399232 torch/multiprocessing/spawn.py:146] Terminating process 9095 via signal SIGTERM Traceback (most recent call last): File "tests/test_low_latency.py", line 172, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes while not context.join(): File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 170, in join raise ProcessExitedException( torch.multiprocessing.spawn.ProcessExitedException: process 7 terminated with signal SIGSEGV ``` gdb debug coredump file: ``` #0 0x00007fb10c4e6337 in ibv_dealloc_pd () from /lib64/libibverbs.so #1 0x00007fb10c0de560 in nvshmemt_ibrc_finalize(nvshmem_transport*) () from /opt/nvshmem/lib/nvshmem_transport_ibrc.so.3 #2 0x00007fb10dc05342 in nvshmemi_transport_finalize(nvshmemi_state_dec*) () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so #3 0x00007fb10daaf809 in nvshmemid_hostlib_finalize () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so #4 0x00007fb10dba3078 in nvshmemi_finalize () from /opt/conda/lib/python3.8/site-packages/deep_ep-1.0.0+c4b8ffc-py3.8-linux-x86_64.egg/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so #5 0x00007fb10da594da in deep_ep::Buffer::~Buffer (this=0x727fcc0, __in_chrg=<optimized out>) at /home/root/DeepEP/csrc/deep_ep.cpp:106 #6 0x00007fb10da6aeba in std::default_delete<deep_ep::Buffer>::operator() (this=<optimized out>, __ptr=0x727fcc0) at /usr/include/c++/10/bits/unique_ptr.h:79 #7 std::unique_ptr<deep_ep::Buffer, std::default_delete<deep_ep::Buffer> >::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/10/bits/unique_ptr.h:361 #8 pybind11::class_<deep_ep::Buffer>::dealloc (v_h=...) at /opt/conda/lib/python3.8/site-packages/torch/include/pybind11/pybind11.h:1939 #9 0x00007fb2670ad753 in pybind11::detail::clear_instance(_object*) () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so #10 0x00007fb2670ae6d1 in pybind11_object_dealloc () from /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so #11 0x00000000004d3dae in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207 #12 _Py_DECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478 #13 _Py_XDECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:541 #14 dict_dealloc (mp=0x7fb111649d00) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1990 #15 0x00000000004f9b46 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207 #16 _Py_DECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478 #17 subtype_dealloc (self=0x7fb11162a460) at /usr/local/src/conda/python-3.8.20/Modules/condvar.h:1277 #18 0x00000000004e0c20 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207 #19 _Py_DECREF (op=<optimized out>, lineno=430, filename=0x5f56e0 "/croot/python-split_1727968608067/work/Objects/frameobject.c") at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478 #20 frame_dealloc (f=0x62fe7f0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/genobject.c:430 #21 0x00000000004e84b8 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/abstract.c:2207 #22 _Py_DECREF (op=0x62fe7f0, op@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, lineno=292, lineno@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, filename=<synthetic pointer>, filename@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Python/stdio2.h:478 #23 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=2, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:292 #24 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411 #25 0x00000000004f7b0a in PyVectorcall_Call (kwargs=0x0, tuple=<optimized out>, callable=0x7fb111632d30) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1802 #26 PyObject_Call (callable=0x7fb111632d30, args=<optimized out>, kwargs=0x0) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:228 #27 0x00000000004da48a in do_call_core (kwdict=0x0, callargs=0x7fb269006b00, func=0x7fb111632d30, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:5010 #28 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3559 #29 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738 #30 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=4, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284 #31 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb111bf2cd8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411 #32 0x00000000004f7b0a in PyVectorcall_Call (kwargs=0x7fb11162b9c0, tuple=<optimized out>, callable=0x7fb112d70c10) at /usr/local/src/conda/python-3.8.20/Objects/objimpl.h:1802 #33 PyObject_Call (callable=0x7fb112d70c10, args=<optimized out>, kwargs=0x7fb11162b9c0) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:228 #34 0x00000000004da48a in do_call_core (kwdict=0x7fb11162b9c0, callargs=0x7fb111bf2cc0, func=0x7fb112d70c10, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:5010 #35 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3559 #36 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738 #37 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284 #38 _PyFunction_Vectorcall (func=<optimized out>, stack=0x595b220, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411 #39 0x00000000004d8ac2 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x595b220, callable=0x7fb2691d09d0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127 #40 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963 #41 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3486 #42 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741 #43 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0xf8c9c8, kwcount=<optimized out>, kwstep=1, defs=0x7fb2691b3178, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x7fb2693c4730, qualname=0x7fb2691b27b0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298 #44 0x00000000004e851c in _PyFunction_Vectorcall (func=<optimized out>, stack=0xf8c9b8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:436 #45 0x00000000004d8ac2 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0xf8c9b8, callable=0x7fb2691b83a0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127 #46 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963 #47 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3486 #48 0x00000000004e8486 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:738 #49 function_code_fastcall (globals=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, nargs=2, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, co=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:284 #50 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb269291608, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:411 #51 0x00000000004d87a9 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fb269291608, callable=0x7fb268fe3940) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127 #52 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0xe98320) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963 #53 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3500 #54 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741 #55 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7fb26938ad50, kwcount=<optimized out>, kwstep=1, defs=0x7fb2691fb118, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x7fb269285970, qualname=0x7fb269285970) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298 #56 0x00000000004e851c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fb26938ad50, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/opcode_targets.h:436 #57 0x00000000004d9566 in _PyObject_Vectorcall (kwnames=0x7fb26923aa80, nargsf=<optimized out>, args=<optimized out>, callable=0x7fb268fe38b0) at /croot/python-split_1727968608067/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/namespaceobject.c:127 #58 call_function (kwnames=0x7fb26923aa80, oparg=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4963 #59 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:3515 #60 0x00000000004d73d1 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:741 #61 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=<optimized out>, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4298 #62 0x0000000000586079 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:4327 #63 0x000000000058603b in PyEval_EvalCode (co=co@entry=0x7fb269344870, globals=globals@entry=0x7fb269359fc0, locals=locals@entry=0x7fb269359fc0) at /usr/local/src/conda/python-3.8.20/Programs/gcmodule.c:718 #64 0x00000000005a5da1 in run_eval_code_obj (co=0x7fb269344870, globals=0x7fb269359fc0, locals=0x7fb269359fc0) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1166 #65 0x00000000005a4daf in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7fb269359fc0, locals=0x7fb269359fc0, flags=<optimized out>, arena=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1188 #66 0x00000000005a27bb in PyRun_StringFlags (str=<optimized out>, start=257, globals=0x7fb269359fc0, locals=0x7fb269359fc0, flags=0x7fffb85d7808) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:1061 #67 0x0000000000450b4b in PyRun_SimpleStringFlags (command=0x7fb2692e29d0 "from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=4, pipe_handle=14)\n", flags=flags@entry=0x7fffb85d7808) at /usr/local/src/conda/python-3.8.20/Objects/clinic/marshal.c.h:486 #68 0x00000000005a1bb8 in pymain_run_command (cf=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, command=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /croot/python-split_1727968608067/work/build-static/python.c:264 #69 pymain_run_python (exitcode=0x7fffb85d7800) at /croot/python-split_1727968608067/work/build-static/python.c:611 #70 Py_RunMain () at /croot/python-split_1727968608067/work/build-static/python.c:699 #71 0x000000000057a0d9 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /croot/python-split_1727968608067/work/build-static/python.c:1131 #72 0x00007fb26979e192 in __libc_start_main () from /lib64/libc.so.6 #73 0x0000000000579f8d in _start () at /usr/local/src/conda/python-3.8.20/Objects/clinic/import.c.h:521 ``` Is this because verbs has a bug?

bsakaguchi commented 2026-10-26 09:52:28 +00:00

Author

Owner

Copy link

@sphish commented on GitHub (Apr 3, 2025):

@alpha-baby I believe verbs is a well-tested library and likely doesn't contain bugs in this case. The issue might be caused by something else. For example, is nvshmem_finalize called twice in your test?

<!-- gh-comment-id:2774102263 --> @sphish commented on GitHub (Apr 3, 2025): @alpha-baby I believe verbs is a well-tested library and likely doesn't contain bugs in this case. The issue might be caused by something else. For example, is `nvshmem_finalize` called twice in your test?

bsakaguchi commented 2026-10-26 09:52:28 +00:00

Author

Owner

Copy link

@alpha-baby commented on GitHub (Apr 11, 2025):

is it possible to enable 288 experts by changing other parameters

It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine)

constexpr int kNumWarpsPerGroup = 10;
constexpr int kNumWarpGroups = 3;
into

constexpr int kNumWarpsPerGroup = 8;
constexpr int kNumWarpGroups = 4;
and remove EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);.

I don't have H20 for testing, if the performance is not good, please report and file an issue :)

Can we make this variable directly configurable? Then, users are recommended how to configure different types of GPUs, and users can also customize them.

<!-- gh-comment-id:2795631099 --> @alpha-baby commented on GitHub (Apr 11, 2025): > > is it possible to enable 288 experts by changing other parameters > > It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine) > > constexpr int kNumWarpsPerGroup = 10; > constexpr int kNumWarpGroups = 3; > into > > constexpr int kNumWarpsPerGroup = 8; > constexpr int kNumWarpGroups = 4; > and remove `EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);`. > > I don't have H20 for testing, if the performance is not good, please report and file an issue :) Can we make this variable directly configurable? Then, users are recommended how to configure different types of GPUs, and users can also customize them.

bsakaguchi commented 2026-10-26 09:52:29 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Apr 11, 2025):

Can we make this variable directly configurable? Then, users are recommended how to configure different types of GPUs, and users can also customize them.

OK, will do it later.

<!-- gh-comment-id:2795774405 --> @LyricZhao commented on GitHub (Apr 11, 2025): > Can we make this variable directly configurable? Then, users are recommended how to configure different types of GPUs, and users can also customize them. OK, will do it later.

bsakaguchi commented 2026-10-26 09:52:30 +00:00

Author

Owner

Copy link

@whybeyoung commented on GitHub (Apr 16, 2025):

and remove EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);

is there any update on this ? i can't find this line. @LyricZhao

<!-- gh-comment-id:2808923256 --> @whybeyoung commented on GitHub (Apr 16, 2025): > and remove `EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);` is there any update on this ? i can't find this line. @LyricZhao

bsakaguchi commented 2026-10-26 09:52:30 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Apr 16, 2025):

Yep, I removed it. No more checks, performace may degrade a little bit, but not a big deal.

<!-- gh-comment-id:2809006645 --> @LyricZhao commented on GitHub (Apr 16, 2025): Yep, I removed it. No more checks, performace may degrade a little bit, but not a big deal.

bsakaguchi commented 2026-10-26 09:52:30 +00:00

Author

Owner

Copy link

@whybeyoung commented on GitHub (Apr 17, 2025):

Yep, I removed it. No more checks, performace may degrade a little bit, but not a big deal.

tested. it's ok after do that in h20 node.

<!-- gh-comment-id:2811721435 --> @whybeyoung commented on GitHub (Apr 17, 2025): > Yep, I removed it. No more checks, performace may degrade a little bit, but not a big deal. tested. it's ok after do that in h20 node.

bsakaguchi commented 2026-10-26 09:52:30 +00:00

Author

Owner

Copy link

@liusy58 commented on GitHub (Apr 27, 2025):

is it possible to enable 288 experts by changing other parameters

It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine)

constexpr int kNumWarpsPerGroup = 10;
constexpr int kNumWarpGroups = 3;
into

constexpr int kNumWarpsPerGroup = 8;
constexpr int kNumWarpGroups = 4;
and remove EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);.

I don't have H20 for testing, if the performance is not good, please report and file an issue :)

@LyricZhao Not sure why we also need to change kNumWarpsPerGroup. Could you please give me an explanation?

<!-- gh-comment-id:2833189493 --> @liusy58 commented on GitHub (Apr 27, 2025): > > is it possible to enable 288 experts by changing other parameters > > It is possible, and for an easier solution (I will update the mainstream later), you can simply change (2 places, both dispatch and combine) > > constexpr int kNumWarpsPerGroup = 10; > constexpr int kNumWarpGroups = 3; > into > > constexpr int kNumWarpsPerGroup = 8; > constexpr int kNumWarpGroups = 4; > and remove `EP_HOST_ASSERT(cell_div(static_cast<int>(hidden * 2 / sizeof(int4)), 32 * (num_warps - 1)) <= 2);`. > > I don't have H20 for testing, if the performance is not good, please report and file an issue :) @LyricZhao Not sure why we also need to change `kNumWarpsPerGroup`. Could you please give me an explanation?

bsakaguchi commented 2026-10-26 09:52:31 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Apr 27, 2025):

It is related to performance, see https://github.com/deepseek-ai/DeepEP/issues/63#issuecomment-2712529850 for more discussion.

<!-- gh-comment-id:2833344131 --> @LyricZhao commented on GitHub (Apr 27, 2025): It is related to performance, see https://github.com/deepseek-ai/DeepEP/issues/63#issuecomment-2712529850 for more discussion.

bsakaguchi commented 2026-10-26 09:52:31 +00:00

Author

Owner

Copy link

@LyricZhao commented on GitHub (Jun 16, 2025):

Fixed in #213, close it now.

<!-- gh-comment-id:2975029212 --> @LyricZhao commented on GitHub (Jun 16, 2025): Fixed in #213, close it now.

bsakaguchi referenced this issue 2026-10-26 09:53:39 +00:00

[GH-ISSUE #74] test_internode.py hangs at internode.cu:dispatch(...) #68

bsakaguchi referenced this issue 2026-10-26 09:56:29 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200

bsakaguchi referenced this issue 2026-10-26 09:56:40 +00:00

[GH-ISSUE #290] Running the test case on a 2-node H20 3E machine failed with internode.cu:1029 'unspecified launch failure' #208

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

hybrid-ep

chore/remove-clang-format-yapf

v2.5

nvDev

try-fix-cu128

mori-ep

antgroup-opt

compat-old-torch

add-linter

eager-rdma

v1.2.1

Labels

Clear labels   

pull-request

Mirrored from GitHub Pull Request

No labels

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#14

No description provided.