[GH-ISSUE #1999] [Bug]: test_mooncake_ep.py failed #556

New issue

Closed

opened 2026-10-29 21:16:23 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented 2026-10-29 21:16:23 +00:00

Owner

Copy link

Originally created by @zbnlala on GitHub (Apr 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1999

Originally assigned to: @UNIDY2002 on GitHub.

Bug Report

I installed Mooncake v0.3.10.post1 by:

mkdir build && cd build
cmake .. -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1"  -DBUILD_SHARED_LIBS=ON -DWITH_STORE=ON -DUSE_HTTP=ON -DUSE_REDIS=ON -DWITH_P2P_STORE=ON
make -j$(nproc)
make install # optional, make it ready to be used by vLLM/SGLang

cd ../
export BUILD_WITH_EP=1
export EP_TORCH_VERSIONS=2.9.1
export TORCH_CUDA_ARCH_LIST="8.9"
./scripts/build_wheel.sh 3.12 dist-3.12

pip install mooncake-wheel/dist-3.12/*.whl --force-reinstall

However when i run test_mooncake_ep.py ,it fails sometimes as shown below:

I0428 06:58:14.118758 29488 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0428 06:58:15.334108 29575 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0428 06:58:15.334112 29574 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0428 06:58:16.448208 29575 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001
I0428 06:58:16.448295 29575 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15019
I0428 06:58:16.448388 29575 transfer_engine_impl.cpp:227] Auto-discovering topology...
I0428 06:58:16.448396 29575 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json
I0428 06:58:16.448558 29575 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs.
I0428 06:58:16.448568 29575 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I0428 06:58:16.448626 29575 rdma_context.cpp:77] Using SIEVE endpoint store
I0428 06:58:16.453070 29574 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001
I0428 06:58:16.453145 29574 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:16688
I0428 06:58:16.453255 29574 transfer_engine_impl.cpp:227] Auto-discovering topology...
I0428 06:58:16.453263 29574 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json
I0428 06:58:16.453418 29574 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs.
I0428 06:58:16.453428 29574 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I0428 06:58:16.453487 29574 rdma_context.cpp:77] Using SIEVE endpoint store
I0428 06:58:16.460618 29574 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device)
I0428 06:58:16.462071 29574 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06
I0428 06:58:16.462180 29574 transfer_engine_impl.cpp:308] installTransport, type=rdma
I0428 06:58:16.474431 29575 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device)
I0428 06:58:16.475899 29575 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06
I0428 06:58:16.475983 29575 transfer_engine_impl.cpp:308] installTransport, type=rdma
Allocating buffer size: 2118.652416 MB ...
I0428 06:58:16.763548 29574 mooncake_ep_buffer.cpp:468] [EP] GPU 0 uses NIC 5 out of 12 NIC(s)
I0428 06:58:16.809283 29575 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s)
I0428 06:58:17.538609 29575 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s)
[rank 1] Dispatch + combine bandwidth: 38.25 GB/s, avg_t=576.37 us, min_t=570.37 us, max_t=581.63 us
[rank 0] Dispatch + combine bandwidth: 38.24 GB/s, avg_t=576.51 us, min_t=570.37 us, max_t=582.66 us
[rank 1] Dispatch bandwidth: 36.55 GB/s, avg_t=205.53 us | Combine bandwidth: 38.16 GB/s, avg_t=380.98 us
[rank 0] Dispatch bandwidth: 35.73 GB/s, avg_t=210.26 us | Combine bandwidth: 38.41 GB/s, avg_t=378.49 us
[rank 0] Dispatch send/recv time: 505.72 us | Combine send/recv time: 804.44 us
[rank 1] Dispatch send/recv time: 207.16 us | Combine send/recv time: 377.90 us
root@lenovo1296:/sgl-workspace/Mooncake/mooncake-wheel/tests# python test_mooncake_ep.py 
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0428 06:58:29.396224 29896 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0428 06:58:30.634552 29983 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0428 06:58:30.851174 29982 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0428 06:58:32.345678 29983 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001
I0428 06:58:32.345789 29983 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15807
I0428 06:58:32.345919 29983 transfer_engine_impl.cpp:227] Auto-discovering topology...
I0428 06:58:32.345929 29983 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json
I0428 06:58:32.346184 29983 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs.
I0428 06:58:32.346197 29983 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I0428 06:58:32.346283 29983 rdma_context.cpp:77] Using SIEVE endpoint store
I0428 06:58:32.349664 29982 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001
I0428 06:58:32.349784 29982 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15719
I0428 06:58:32.349921 29982 transfer_engine_impl.cpp:227] Auto-discovering topology...
I0428 06:58:32.349931 29982 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json
I0428 06:58:32.350170 29982 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs.
I0428 06:58:32.350184 29982 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering.
I0428 06:58:32.350265 29982 rdma_context.cpp:77] Using SIEVE endpoint store
I0428 06:58:32.359449 29982 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device)
I0428 06:58:32.361066 29982 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06
I0428 06:58:32.361155 29982 transfer_engine_impl.cpp:308] installTransport, type=rdma
I0428 06:58:32.378625 29983 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device)
I0428 06:58:32.380272 29983 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06
I0428 06:58:32.380378 29983 transfer_engine_impl.cpp:308] installTransport, type=rdma
Allocating buffer size: 2118.652416 MB ...
I0428 06:58:32.666940 29982 mooncake_ep_buffer.cpp:468] [EP] GPU 0 uses NIC 5 out of 12 NIC(s)
I0428 06:58:32.713523 29983 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s)
I0428 06:58:33.469841 29983 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s)
W0428 06:58:35.587000 29896 torch/multiprocessing/spawn.py:174] Terminating process 29982 via signal SIGTERM
Traceback (most recent call last):
  File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 198, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 364, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 320, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 220, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 95, in _wrap
    fn(i, *args)
  File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 177, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, cpu_group, buffer, seed=1)
  File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 64, in test_main
    assert torch.equal(recv_x_amin, recv_x[:, :-128].amax(dim=-1)), f'recv_x_amin={recv_x_amin}, recv_x_amax={recv_x[:, :-128].amax(dim=-1)}'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: recv_x_amin=tensor([-127., -127., -127., -128., -128., -128.], device='cuda:1'), recv_x_amax=tensor([-127., -127., -127., -128., -128.,    0.], device='cuda:1')

Is there something wrong with my settings? All other tests are running normally.

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

Originally created by @zbnlala on GitHub (Apr 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1999 Originally assigned to: @UNIDY2002 on GitHub. ### Bug Report I installed Mooncake v0.3.10.post1 by: ```bash dependencies.sh mkdir build && cd build cmake .. -DUSE_CUDA=ON -DWITH_EP=ON -DEP_TORCH_VERSIONS="2.9.1" -DBUILD_SHARED_LIBS=ON -DWITH_STORE=ON -DUSE_HTTP=ON -DUSE_REDIS=ON -DWITH_P2P_STORE=ON make -j$(nproc) make install # optional, make it ready to be used by vLLM/SGLang cd ../ export BUILD_WITH_EP=1 export EP_TORCH_VERSIONS=2.9.1 export TORCH_CUDA_ARCH_LIST="8.9" ./scripts/build_wheel.sh 3.12 dist-3.12 pip install mooncake-wheel/dist-3.12/*.whl --force-reinstall ``` However when i run test_mooncake_ep.py ,it fails sometimes as shown below: ```WARNING: Logging before InitGoogleLogging() is written to STDERR I0428 06:58:14.118758 29488 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) WARNING: Logging before InitGoogleLogging() is written to STDERR WARNING: Logging before InitGoogleLogging() is written to STDERR I0428 06:58:15.334108 29575 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0428 06:58:15.334112 29574 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0428 06:58:16.448208 29575 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001 I0428 06:58:16.448295 29575 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15019 I0428 06:58:16.448388 29575 transfer_engine_impl.cpp:227] Auto-discovering topology... I0428 06:58:16.448396 29575 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json I0428 06:58:16.448558 29575 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs. I0428 06:58:16.448568 29575 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering. I0428 06:58:16.448626 29575 rdma_context.cpp:77] Using SIEVE endpoint store I0428 06:58:16.453070 29574 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001 I0428 06:58:16.453145 29574 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:16688 I0428 06:58:16.453255 29574 transfer_engine_impl.cpp:227] Auto-discovering topology... I0428 06:58:16.453263 29574 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json I0428 06:58:16.453418 29574 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs. I0428 06:58:16.453428 29574 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering. I0428 06:58:16.453487 29574 rdma_context.cpp:77] Using SIEVE endpoint store I0428 06:58:16.460618 29574 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device) I0428 06:58:16.462071 29574 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06 I0428 06:58:16.462180 29574 transfer_engine_impl.cpp:308] installTransport, type=rdma I0428 06:58:16.474431 29575 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device) I0428 06:58:16.475899 29575 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06 I0428 06:58:16.475983 29575 transfer_engine_impl.cpp:308] installTransport, type=rdma Allocating buffer size: 2118.652416 MB ... I0428 06:58:16.763548 29574 mooncake_ep_buffer.cpp:468] [EP] GPU 0 uses NIC 5 out of 12 NIC(s) I0428 06:58:16.809283 29575 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s) I0428 06:58:17.538609 29575 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s) [rank 1] Dispatch + combine bandwidth: 38.25 GB/s, avg_t=576.37 us, min_t=570.37 us, max_t=581.63 us [rank 0] Dispatch + combine bandwidth: 38.24 GB/s, avg_t=576.51 us, min_t=570.37 us, max_t=582.66 us [rank 1] Dispatch bandwidth: 36.55 GB/s, avg_t=205.53 us | Combine bandwidth: 38.16 GB/s, avg_t=380.98 us [rank 0] Dispatch bandwidth: 35.73 GB/s, avg_t=210.26 us | Combine bandwidth: 38.41 GB/s, avg_t=378.49 us [rank 0] Dispatch send/recv time: 505.72 us | Combine send/recv time: 804.44 us [rank 1] Dispatch send/recv time: 207.16 us | Combine send/recv time: 377.90 us root@lenovo1296:/sgl-workspace/Mooncake/mooncake-wheel/tests# python test_mooncake_ep.py WARNING: Logging before InitGoogleLogging() is written to STDERR I0428 06:58:29.396224 29896 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) WARNING: Logging before InitGoogleLogging() is written to STDERR I0428 06:58:30.634552 29983 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) WARNING: Logging before InitGoogleLogging() is written to STDERR I0428 06:58:30.851174 29982 transfer_engine_impl.cpp:597] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0428 06:58:32.345678 29983 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001 I0428 06:58:32.345789 29983 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15807 I0428 06:58:32.345919 29983 transfer_engine_impl.cpp:227] Auto-discovering topology... I0428 06:58:32.345929 29983 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json I0428 06:58:32.346184 29983 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs. I0428 06:58:32.346197 29983 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering. I0428 06:58:32.346283 29983 rdma_context.cpp:77] Using SIEVE endpoint store I0428 06:58:32.349664 29982 transfer_engine_impl.cpp:105] Transfer Engine parseHostNameWithPort. server_name: 127.0.0.1 port: 12001 I0428 06:58:32.349784 29982 transfer_engine_impl.cpp:172] Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:15719 I0428 06:58:32.349921 29982 transfer_engine_impl.cpp:227] Auto-discovering topology... I0428 06:58:32.349931 29982 transfer_engine_impl.cpp:230] Using custom topology from: /sgl-workspace/topology.json I0428 06:58:32.350170 29982 transfer_engine_impl.cpp:242] Topology discovery complete. Found 1 HCAs. I0428 06:58:32.350184 29982 rdma_transport.cpp:63] [RDMA] Relaxed ordering disabled via MC_IB_PCI_RELAXED_ORDERING=0. Falling back to strict ordering. I0428 06:58:32.350265 29982 rdma_context.cpp:77] Using SIEVE endpoint store I0428 06:58:32.359449 29982 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device) I0428 06:58:32.361066 29982 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06 I0428 06:58:32.361155 29982 transfer_engine_impl.cpp:308] installTransport, type=rdma I0428 06:58:32.378625 29983 rdma_context.cpp:605] Find best gid index: 3 on mlx5_5/ (network state: with network device) I0428 06:58:32.380272 29983 rdma_context.cpp:140] RDMA device: mlx5_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:c0:a8:3d:06 I0428 06:58:32.380378 29983 transfer_engine_impl.cpp:308] installTransport, type=rdma Allocating buffer size: 2118.652416 MB ... I0428 06:58:32.666940 29982 mooncake_ep_buffer.cpp:468] [EP] GPU 0 uses NIC 5 out of 12 NIC(s) I0428 06:58:32.713523 29983 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s) I0428 06:58:33.469841 29983 mooncake_ep_buffer.cpp:468] [EP] GPU 1 uses NIC 5 out of 12 NIC(s) W0428 06:58:35.587000 29896 torch/multiprocessing/spawn.py:174] Terminating process 29982 via signal SIGTERM Traceback (most recent call last): File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 198, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes) File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 364, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 320, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 220, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 1 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 95, in _wrap fn(i, *args) File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 177, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, cpu_group, buffer, seed=1) File "/sgl-workspace/Mooncake/mooncake-wheel/tests/test_mooncake_ep.py", line 64, in test_main assert torch.equal(recv_x_amin, recv_x[:, :-128].amax(dim=-1)), f'recv_x_amin={recv_x_amin}, recv_x_amax={recv_x[:, :-128].amax(dim=-1)}' ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: recv_x_amin=tensor([-127., -127., -127., -128., -128., -128.], device='cuda:1'), recv_x_amax=tensor([-127., -127., -127., -128., -128., 0.], device='cuda:1') ``` Is there something wrong with my settings? All other tests are running normally. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi 2026-10-29 21:16:23 +00:00

• closed this issue
• added the
  bug
  label

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@UNIDY2002 commented on GitHub (Apr 28, 2026):

Hi @zbnlala! Can you share your GPU model and the NIC model? Thanks!

<!-- gh-comment-id:4333704758 --> @UNIDY2002 commented on GitHub (Apr 28, 2026): Hi @zbnlala! Can you share your GPU model and the NIC model? Thanks!

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@zbnlala commented on GitHub (Apr 28, 2026):

GPUs are L20

$ lspci | grep -i -E 'ethernet|network|infiniband'
1b:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
1b:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
32:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
32:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
43:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
43:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
97:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
97:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
b3:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
b3:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
c3:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
c3:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

$sudo lshw -class network
  *-network:0               
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:1b:00.0
       logical name: ens5f0np0
       version: 00
       serial: 58:a2:e1:da:e6:26
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.50.10 latency=0 link=yes multicast=yes
       resources: iomemory:20c00-20bff irq:16 memory:20c046000000-20c047ffffff memory:9f100000-9f1fffff memory:20c048800000-20c048ffffff
  *-network:1
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:1b:00.1
       logical name: ens5f1np1
       version: 00
       serial: 58:a2:e1:da:e6:27
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.51.10 latency=0 link=yes multicast=yes
       resources: iomemory:20c00-20bff irq:17 memory:20c044000000-20c045ffffff memory:9f000000-9f0fffff memory:20c048000000-20c0487fffff
  *-network:0
       description: Ethernet interface
       product: MT2894 Family [ConnectX-6 Lx]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:32:00.0
       logical name: ens16f0np0
       version: 00
       serial: c4:70:bd:dc:9a:c6
       size: 10Gbit/s
       capacity: 25Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=26.39.1002 (LNV0000000037) ip=10.121.129.10 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
       resources: iomemory:20df0-20def irq:16 memory:20dffc000000-20dffdffffff memory:a7600000-a76fffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: MT2894 Family [ConnectX-6 Lx]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:32:00.1
       logical name: ens16f1np1
       version: 00
       serial: c4:70:bd:dc:9a:c7
       capacity: 25Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=26.39.1002 (LNV0000000037) latency=0 link=no multicast=yes port=fibre
       resources: iomemory:20df0-20def irq:17 memory:20dffa000000-20dffbffffff memory:a7500000-a75fffff
  *-network:0
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:43:00.0
       logical name: ens6f0np0
       version: 00
       serial: 58:a2:e1:e2:1f:5c
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.60.10 latency=0 link=yes multicast=yes
       resources: iomemory:20ef0-20eef irq:16 memory:20effc000000-20effdffffff memory:b0600000-b06fffff memory:20effe800000-20effeffffff
  *-network:1
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:43:00.1
       logical name: ens6f1np1
       version: 00
       serial: 58:a2:e1:e2:1f:5d
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.61.10 latency=0 link=yes multicast=yes
       resources: iomemory:20ef0-20eef irq:17 memory:20effa000000-20effbffffff memory:b0500000-b05fffff memory:20effe000000-20effe7fffff
  *-network:0
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:97:00.0
       logical name: ens8f0np0
       version: 00
       serial: 58:a2:e1:da:e7:76
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.80.10 latency=0 link=yes multicast=yes
       resources: iomemory:21e00-21dff irq:16 memory:21e046000000-21e047ffffff memory:d1100000-d11fffff memory:21e048800000-21e048ffffff
  *-network:1
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:97:00.1
       logical name: ens8f1np1
       version: 00
       serial: 58:a2:e1:da:e7:77
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.81.10 latency=0 link=yes multicast=yes
       resources: iomemory:21e00-21dff irq:17 memory:21e044000000-21e045ffffff memory:d1000000-d10fffff memory:21e048000000-21e0487fffff
  *-network:0
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:b3:00.0
       logical name: ens7f0np0
       version: 00
       serial: 58:a2:e1:e2:6c:74
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.70.10 latency=0 link=yes multicast=yes
       resources: iomemory:21ff0-21fef irq:16 memory:21fffc000000-21fffdffffff memory:d9200000-d92fffff memory:21fffe800000-21fffeffffff
  *-network:1
       description: Ethernet interface
       product: MT2910 Family [ConnectX-7]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:b3:00.1
       logical name: ens7f1np1
       version: 00
       serial: 58:a2:e1:e2:6c:75
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.71.10 latency=0 link=yes multicast=yes
       resources: iomemory:21ff0-21fef irq:17 memory:21fffa000000-21fffbffffff memory:d9100000-d91fffff memory:21fffe000000-21fffe7fffff
  *-network:0 DISABLED
       description: Ethernet interface
       product: MT2892 Family [ConnectX-6 Dx]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:c3:00.0
       logical name: ens9f0np0
       version: 00
       serial: b8:3f:d2:2d:65:02
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=22.31.1014 (LNV0000000029) latency=0 link=no multicast=yes
       resources: iomemory:220f0-220ef irq:16 memory:220ffc000000-220ffdffffff memory:e0600000-e06fffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: MT2892 Family [ConnectX-6 Dx]
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:c3:00.1
       logical name: ens9f1np1
       version: 00
       serial: b8:3f:d2:2d:65:03
       capacity: 40Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=22.31.1014 (LNV0000000029) latency=0 link=no multicast=yes
       resources: iomemory:220f0-220ef irq:17 memory:220ffa000000-220ffbffffff memory:e0500000-e05fffff

$ ibstat
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300dae626
        System image GUID: 0x58a2e10300dae626
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffedae626
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300dae627
        System image GUID: 0x58a2e10300dae626
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffedae627
                Link layer: Ethernet
CA 'mlx5_10'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.31.1014
        Hardware version: 0
        Node GUID: 0xb83fd203002d6502
        System image GUID: 0xb83fd203002d6502
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xba3fd2fffe2d6502
                Link layer: Ethernet
CA 'mlx5_11'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.31.1014
        Hardware version: 0
        Node GUID: 0xb83fd203002d6503
        System image GUID: 0xb83fd203002d6502
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xba3fd2fffe2d6503
                Link layer: Ethernet
CA 'mlx5_2'
        CA type: MT4127
        Number of ports: 1
        Firmware version: 26.39.1002
        Hardware version: 0
        Node GUID: 0xc470bd0300dc9ac6
        System image GUID: 0xc470bd0300dc9ac6
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xc670bdfffedc9ac6
                Link layer: Ethernet
CA 'mlx5_3'
        CA type: MT4127
        Number of ports: 1
        Firmware version: 26.39.1002
        Hardware version: 0
        Node GUID: 0xc470bd0300dc9ac7
        System image GUID: 0xc470bd0300dc9ac6
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xc670bdfffedc9ac7
                Link layer: Ethernet
CA 'mlx5_4'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300e21f5c
        System image GUID: 0x58a2e10300e21f5c
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffee21f5c
                Link layer: Ethernet
CA 'mlx5_5'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300e21f5d
        System image GUID: 0x58a2e10300e21f5c
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffee21f5d
                Link layer: Ethernet
CA 'mlx5_6'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300dae776
        System image GUID: 0x58a2e10300dae776
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffedae776
                Link layer: Ethernet
CA 'mlx5_7'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300dae777
        System image GUID: 0x58a2e10300dae776
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffedae777
                Link layer: Ethernet
CA 'mlx5_8'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300e26c74
        System image GUID: 0x58a2e10300e26c74
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffee26c74
                Link layer: Ethernet
CA 'mlx5_9'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x58a2e10300e26c75
        System image GUID: 0x58a2e10300e26c74
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffee26c75
                Link layer: Ethernet

<!-- gh-comment-id:4333911457 --> @zbnlala commented on GitHub (Apr 28, 2026): GPUs are L20 ``` $ lspci | grep -i -E 'ethernet|network|infiniband' 1b:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] 1b:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] 32:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 32:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 43:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] 43:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] 97:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] 97:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] b3:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] b3:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7] c3:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] c3:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] ``` ``` $sudo lshw -class network *-network:0 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:1b:00.0 logical name: ens5f0np0 version: 00 serial: 58:a2:e1:da:e6:26 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.50.10 latency=0 link=yes multicast=yes resources: iomemory:20c00-20bff irq:16 memory:20c046000000-20c047ffffff memory:9f100000-9f1fffff memory:20c048800000-20c048ffffff *-network:1 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:1b:00.1 logical name: ens5f1np1 version: 00 serial: 58:a2:e1:da:e6:27 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.51.10 latency=0 link=yes multicast=yes resources: iomemory:20c00-20bff irq:17 memory:20c044000000-20c045ffffff memory:9f000000-9f0fffff memory:20c048000000-20c0487fffff *-network:0 description: Ethernet interface product: MT2894 Family [ConnectX-6 Lx] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:32:00.0 logical name: ens16f0np0 version: 00 serial: c4:70:bd:dc:9a:c6 size: 10Gbit/s capacity: 25Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=26.39.1002 (LNV0000000037) ip=10.121.129.10 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s resources: iomemory:20df0-20def irq:16 memory:20dffc000000-20dffdffffff memory:a7600000-a76fffff *-network:1 DISABLED description: Ethernet interface product: MT2894 Family [ConnectX-6 Lx] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:32:00.1 logical name: ens16f1np1 version: 00 serial: c4:70:bd:dc:9a:c7 capacity: 25Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=26.39.1002 (LNV0000000037) latency=0 link=no multicast=yes port=fibre resources: iomemory:20df0-20def irq:17 memory:20dffa000000-20dffbffffff memory:a7500000-a75fffff *-network:0 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:43:00.0 logical name: ens6f0np0 version: 00 serial: 58:a2:e1:e2:1f:5c capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.60.10 latency=0 link=yes multicast=yes resources: iomemory:20ef0-20eef irq:16 memory:20effc000000-20effdffffff memory:b0600000-b06fffff memory:20effe800000-20effeffffff *-network:1 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:43:00.1 logical name: ens6f1np1 version: 00 serial: 58:a2:e1:e2:1f:5d capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.61.10 latency=0 link=yes multicast=yes resources: iomemory:20ef0-20eef irq:17 memory:20effa000000-20effbffffff memory:b0500000-b05fffff memory:20effe000000-20effe7fffff *-network:0 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:97:00.0 logical name: ens8f0np0 version: 00 serial: 58:a2:e1:da:e7:76 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.80.10 latency=0 link=yes multicast=yes resources: iomemory:21e00-21dff irq:16 memory:21e046000000-21e047ffffff memory:d1100000-d11fffff memory:21e048800000-21e048ffffff *-network:1 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:97:00.1 logical name: ens8f1np1 version: 00 serial: 58:a2:e1:da:e7:77 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.81.10 latency=0 link=yes multicast=yes resources: iomemory:21e00-21dff irq:17 memory:21e044000000-21e045ffffff memory:d1000000-d10fffff memory:21e048000000-21e0487fffff *-network:0 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:b3:00.0 logical name: ens7f0np0 version: 00 serial: 58:a2:e1:e2:6c:74 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.70.10 latency=0 link=yes multicast=yes resources: iomemory:21ff0-21fef irq:16 memory:21fffc000000-21fffdffffff memory:d9200000-d92fffff memory:21fffe800000-21fffeffffff *-network:1 description: Ethernet interface product: MT2910 Family [ConnectX-7] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:b3:00.1 logical name: ens7f1np1 version: 00 serial: 58:a2:e1:e2:6c:75 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 duplex=full firmware=28.39.1002 (LNV0000000058) ip=192.168.71.10 latency=0 link=yes multicast=yes resources: iomemory:21ff0-21fef irq:17 memory:21fffa000000-21fffbffffff memory:d9100000-d91fffff memory:21fffe000000-21fffe7fffff *-network:0 DISABLED description: Ethernet interface product: MT2892 Family [ConnectX-6 Dx] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:c3:00.0 logical name: ens9f0np0 version: 00 serial: b8:3f:d2:2d:65:02 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=22.31.1014 (LNV0000000029) latency=0 link=no multicast=yes resources: iomemory:220f0-220ef irq:16 memory:220ffc000000-220ffdffffff memory:e0600000-e06fffff *-network:1 DISABLED description: Ethernet interface product: MT2892 Family [ConnectX-6 Dx] vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:c3:00.1 logical name: ens9f1np1 version: 00 serial: b8:3f:d2:2d:65:03 capacity: 40Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=24.10-4.1.4 firmware=22.31.1014 (LNV0000000029) latency=0 link=no multicast=yes resources: iomemory:220f0-220ef irq:17 memory:220ffa000000-220ffbffffff memory:e0500000-e05fffff ``` ``` $ ibstat CA 'mlx5_0' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300dae626 System image GUID: 0x58a2e10300dae626 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffedae626 Link layer: Ethernet CA 'mlx5_1' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300dae627 System image GUID: 0x58a2e10300dae626 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffedae627 Link layer: Ethernet CA 'mlx5_10' CA type: MT4125 Number of ports: 1 Firmware version: 22.31.1014 Hardware version: 0 Node GUID: 0xb83fd203002d6502 System image GUID: 0xb83fd203002d6502 Port 1: State: Down Physical state: Disabled Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba3fd2fffe2d6502 Link layer: Ethernet CA 'mlx5_11' CA type: MT4125 Number of ports: 1 Firmware version: 22.31.1014 Hardware version: 0 Node GUID: 0xb83fd203002d6503 System image GUID: 0xb83fd203002d6502 Port 1: State: Down Physical state: Disabled Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba3fd2fffe2d6503 Link layer: Ethernet CA 'mlx5_2' CA type: MT4127 Number of ports: 1 Firmware version: 26.39.1002 Hardware version: 0 Node GUID: 0xc470bd0300dc9ac6 System image GUID: 0xc470bd0300dc9ac6 Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xc670bdfffedc9ac6 Link layer: Ethernet CA 'mlx5_3' CA type: MT4127 Number of ports: 1 Firmware version: 26.39.1002 Hardware version: 0 Node GUID: 0xc470bd0300dc9ac7 System image GUID: 0xc470bd0300dc9ac6 Port 1: State: Down Physical state: Disabled Rate: 10 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xc670bdfffedc9ac7 Link layer: Ethernet CA 'mlx5_4' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300e21f5c System image GUID: 0x58a2e10300e21f5c Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffee21f5c Link layer: Ethernet CA 'mlx5_5' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300e21f5d System image GUID: 0x58a2e10300e21f5c Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffee21f5d Link layer: Ethernet CA 'mlx5_6' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300dae776 System image GUID: 0x58a2e10300dae776 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffedae776 Link layer: Ethernet CA 'mlx5_7' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300dae777 System image GUID: 0x58a2e10300dae776 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffedae777 Link layer: Ethernet CA 'mlx5_8' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300e26c74 System image GUID: 0x58a2e10300e26c74 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffee26c74 Link layer: Ethernet CA 'mlx5_9' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x58a2e10300e26c75 System image GUID: 0x58a2e10300e26c74 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffee26c75 Link layer: Ethernet ```

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@zbnlala commented on GitHub (May 8, 2026):

@UNIDY2002 hi，any progress？

<!-- gh-comment-id:4402681513 --> @zbnlala commented on GitHub (May 8, 2026): @UNIDY2002 hi，any progress？

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@UNIDY2002 commented on GitHub (May 8, 2026):

The problem may be related to PCIe P2P. Would you please try whether commenting out these lines will help?
github.com/kvcache-ai/Mooncake@5ba016ac1c/mooncake-wheel/mooncake/mooncake_ep_buffer.py (L180-L203)

<!-- gh-comment-id:4402746298 --> @UNIDY2002 commented on GitHub (May 8, 2026): The problem may be related to PCIe P2P. Would you please try whether commenting out these lines will help? https://github.com/kvcache-ai/Mooncake/blob/5ba016ac1cf585ea045a218abb036fa818961777/mooncake-wheel/mooncake/mooncake_ep_buffer.py#L180-L203

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@zbnlala commented on GitHub (May 8, 2026):

@UNIDY2002 Thank you! After commenting it out, there won't be any more bugs. However, I’m still wondering whether this indicates an issue with my environment or configuration. And will commenting it out have any impact on the functionality of mooncake-ep?

<!-- gh-comment-id:4403018964 --> @zbnlala commented on GitHub (May 8, 2026): @UNIDY2002 Thank you! After commenting it out, there won't be any more bugs. However, I’m still wondering whether this indicates an issue with my environment or configuration. And will commenting it out have any impact on the functionality of mooncake-ep?

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@UNIDY2002 commented on GitHub (May 8, 2026):

That part was for NVLink, so since your environment does not involve NVLink (correct me if I'm wrong), it will not have a negative impact. However, I will add an environment variable flag so that you will not have to manually comment out the unrelated code.

<!-- gh-comment-id:4403100623 --> @UNIDY2002 commented on GitHub (May 8, 2026): That part was for NVLink, so since your environment does not involve NVLink (correct me if I'm wrong), it will not have a negative impact. However, I will add an environment variable flag so that you will not have to manually comment out the unrelated code.

bsakaguchi commented 2026-10-29 21:16:24 +00:00

Author

Owner

Copy link

@zbnlala commented on GitHub (May 8, 2026):

@UNIDY2002 hi, I have another question related to SGLang + Mooncake-ep. When starting SGLang, I used the following command:

python3 -m sglang.launch_server --model-path  downmodels/Qwen3-30B-A3B-FP8 --tp 4  --ep 4  --moe-a2a-backend mooncake --deepep-mode low_latency --attention-backend triton --enable-dp-attention --dp 4 --enable-dp-lm-head --elastic-ep-backend mooncake --moe-dense-tp-size 1 --enable-two-batch-overlap --disable-custom-all-reduce --enable-eplb --ep-num-redundant-experts 72 --chunked-prefill-size 512 --cuda-graph-max-bs 128 --max-running-requests 128 --mem-fraction-static 0.5 --enable-elastic-expert-backup

There is error:

ScheduleBatch(forward_mode=EXTEND, #req=1)
ScheduleBatch(forward_mode=EXTEND, #req=1)
ScheduleBatch(forward_mode=IDLE, #req=0)
ScheduleBatch(forward_mode=IDLE, #req=0)
[2026-05-08 08:24:53 DP3 TP3 EP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 3617, in run_scheduler_process
    scheduler.run_event_loop()
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 1300, in run_event_loop
    dispatch_event_loop(self)
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 3498, in dispatch_event_loop
    scheduler.event_loop_overlap()
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 1362, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 2685, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2739, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2855, in _forward_raw
    ret = self.forward_idle(forward_batch, pp_proxy_tensors=pp_proxy_tensors)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2697, in forward_idle
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 991, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen2_moe.py", line 681, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 839, in forward
    hidden_states = self.mlp(
                    ^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 298, in forward
    return self.forward_normal(
           ^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 326, in forward_normal
    final_hidden_states = self.experts(hidden_states, topk_output)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/ep_moe/layer.py", line 208, in forward
    return self.forward_impl(hidden_states, topk_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/ep_moe/layer.py", line 223, in forward_impl
    dispatch_output = self.dispatcher.dispatch(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/batch_overlap/two_batch_overlap.py", line 1050, in dispatch
    return self._execute("dispatch", **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/batch_overlap/two_batch_overlap.py", line 1047, in _execute
    return getattr(self._inners[tbo_subbatch_index or 0], name)(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 326, in dispatch
    self.dispatch_a(hidden_states, topk_output)
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 336, in dispatch_a
    inner_state = self._get_impl().dispatch_a(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 160, in dispatch_a
    hidden_states, masked_m, event, hook = self._dispatch_core(
                                           ^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 214, in _dispatch_core
    buffer.dispatch(
  File "/usr/local/lib/python3.12/dist-packages/mooncake/mooncake_ep_buffer.py", line 192, in dispatch
    self.runtime.dispatch(x, topk_idx, active_ranks,
torch.AcceleratorError: CUDA error: invalid configuration argument
Search for `cudaErrorInvalidConfiguration' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I printed the ScheduleBatch, and it seems that the error occurs whenever a batch in the idle state is being processed.
I also tried setting self._use_fallback = True in the buffer so that I could trace into the self._fallback_dispatch() function. The error then appeared at the following line with the same failure:

x_padded = torch.cat(
    [x, torch.zeros((pad_size, hidden), dtype=x.dtype, device=x.device)],
    dim=0
)

Interestingly, if I change device=x.device to device="cpu", this line can execute successfully, but the same error appears again at the next GPU-related operation. This makes me suspect that the issue may be related to GPU blocking. Do you have any suggestions on how to debug or resolve this?

<!-- gh-comment-id:4404993411 --> @zbnlala commented on GitHub (May 8, 2026): @UNIDY2002 hi, I have another question related to SGLang + Mooncake-ep. When starting SGLang, I used the following command: ``` python3 -m sglang.launch_server --model-path downmodels/Qwen3-30B-A3B-FP8 --tp 4 --ep 4 --moe-a2a-backend mooncake --deepep-mode low_latency --attention-backend triton --enable-dp-attention --dp 4 --enable-dp-lm-head --elastic-ep-backend mooncake --moe-dense-tp-size 1 --enable-two-batch-overlap --disable-custom-all-reduce --enable-eplb --ep-num-redundant-experts 72 --chunked-prefill-size 512 --cuda-graph-max-bs 128 --max-running-requests 128 --mem-fraction-static 0.5 --enable-elastic-expert-backup ``` There is error: ``` ScheduleBatch(forward_mode=EXTEND, #req=1) ScheduleBatch(forward_mode=EXTEND, #req=1) ScheduleBatch(forward_mode=IDLE, #req=0) ScheduleBatch(forward_mode=IDLE, #req=0) [2026-05-08 08:24:53 DP3 TP3 EP3] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 3617, in run_scheduler_process scheduler.run_event_loop() File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 1300, in run_event_loop dispatch_event_loop(self) File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 3498, in dispatch_event_loop scheduler.event_loop_overlap() File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 1362, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/scheduler.py", line 2685, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation out = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2739, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2855, in _forward_raw ret = self.forward_idle(forward_batch, pp_proxy_tensors=pp_proxy_tensors) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/model_executor/model_runner.py", line 2697, in forward_idle return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 991, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen2_moe.py", line 681, in forward hidden_states, residual = layer( ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 839, in forward hidden_states = self.mlp( ^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 298, in forward return self.forward_normal( ^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/models/qwen3_moe.py", line 326, in forward_normal final_hidden_states = self.experts(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/ep_moe/layer.py", line 208, in forward return self.forward_impl(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/ep_moe/layer.py", line 223, in forward_impl dispatch_output = self.dispatcher.dispatch( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/batch_overlap/two_batch_overlap.py", line 1050, in dispatch return self._execute("dispatch", **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/batch_overlap/two_batch_overlap.py", line 1047, in _execute return getattr(self._inners[tbo_subbatch_index or 0], name)(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 326, in dispatch self.dispatch_a(hidden_states, topk_output) File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 336, in dispatch_a inner_state = self._get_impl().dispatch_a( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 160, in dispatch_a hidden_states, masked_m, event, hook = self._dispatch_core( ^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang-5.10/python/sglang/srt/layers/moe/token_dispatcher/mooncake.py", line 214, in _dispatch_core buffer.dispatch( File "/usr/local/lib/python3.12/dist-packages/mooncake/mooncake_ep_buffer.py", line 192, in dispatch self.runtime.dispatch(x, topk_idx, active_ranks, torch.AcceleratorError: CUDA error: invalid configuration argument Search for `cudaErrorInvalidConfiguration' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` I printed the ```ScheduleBatch```, and it seems that the error occurs whenever a batch in the ```idle``` state is being processed. I also tried setting ```self._use_fallback = True``` in the buffer so that I could trace into the ```self._fallback_dispatch()``` function. The error then appeared at the following line with the same failure: ``` x_padded = torch.cat( [x, torch.zeros((pad_size, hidden), dtype=x.dtype, device=x.device)], dim=0 ) ``` Interestingly, if I change device=x.device to device="cpu", this line can execute successfully, but the same error appears again at the next GPU-related operation. This makes me suspect that the issue may be related to GPU blocking. Do you have any suggestions on how to debug or resolve this?

bsakaguchi commented 2026-10-29 21:16:25 +00:00

Author

Owner

Copy link

@UNIDY2002 commented on GitHub (May 9, 2026):

I haven't yet allocated enough resource to reproduce this case, but I have a clue: there are several conditionals missing for models other than DeepSeek V3. Would you try adding some conditions in the qwen3 model file just like in the following commit (although it was for qwen2)?
github.com/HanHan009527/sglang@bf6488fb52

<!-- gh-comment-id:4411397030 --> @UNIDY2002 commented on GitHub (May 9, 2026): I haven't yet allocated enough resource to reproduce this case, but I have a clue: there are several conditionals missing for models other than DeepSeek V3. Would you try adding some conditions in the qwen3 model file just like in the following commit (although it was for qwen2)? https://github.com/HanHan009527/sglang/commit/bf6488fb529ac14ed08ce0895b0dcba03f331f38

bsakaguchi commented 2026-10-29 21:16:25 +00:00

Author

Owner

Copy link

@zbnlala commented on GitHub (May 11, 2026):

@UNIDY2002 It works！I have replaced all get_moe_a2a_backend().is_deepep() in qwen2_moe.py and qwen3_moe.py with (get_moe_a2a_backend().is_deepep() or get_moe_a2a_backend().is_mooncake()), The service can proceed normally.

Thanks for answering!

<!-- gh-comment-id:4418193906 --> @zbnlala commented on GitHub (May 11, 2026): @UNIDY2002 It works！I have replaced all ```get_moe_a2a_backend().is_deepep()``` in ```qwen2_moe.py``` and ```qwen3_moe.py``` with ```(get_moe_a2a_backend().is_deepep() or get_moe_a2a_backend().is_mooncake())```, The service can proceed normally. Thanks for answering!

bsakaguchi referenced this issue 2026-10-29 21:29:10 +00:00

[PR #556] [MERGED] support batch put/get api in python module #1648

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#556

No description provided.