kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #507] how to deal the mooncake error: worker_pool.cpp:291] Worker: Process failed for slice #173

New issue
Closed
opened 2026-10-29 21:10:30 +00:00 by bsakaguchi · 6 comments
bsakaguchi commented 2026-10-29 21:10:30 +00:00
Owner
Copy link

Originally created by @XiaobinZhao on GitHub (Jun 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/507

i run 1p1d sglang with mooncake,but failed. the error log is: ... Worker: Process failed for slice ...; i searched the issue in the github mooncake; and try the mooncake Example: Transfer Engine Bench , the error is same，

root@ubuntu-104:/Mooncake-main/build/mooncake-transfer-engine/example# ./transfer_engine_bench  --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2  --local_server_name=10.10.251.104  --segment_id=10.10.251.105
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0617 08:42:42.470683   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_6/
I0617 08:42:42.471800   413 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:12:55:cc
I0617 08:42:42.498806   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_7/
I0617 08:42:42.499879   413 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 10, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:12:55:a4
I0617 08:42:42.526409   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_8/
I0617 08:42:42.527472   413 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 12, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:18:db:50
I0617 08:42:42.554749   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_9/
I0617 08:42:42.555806   413 rdma_context.cpp:125] RDMA device: mlx5_9, LID: 14, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:18:cb:50
I0617 08:42:42.582837   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_0/
I0617 08:42:42.583910   413 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:19:5c:ea
I0617 08:42:42.611243   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_1/
I0617 08:42:42.612282   413 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 2, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:b2:6f:64
I0617 08:42:42.638762   413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_2/
I0617 08:42:42.639797   413 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 4, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:ad:ee:b8
E0617 08:42:47.012193   417 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f60b0027000, length: 4096, dest_addr: 139879971450880, local_nic: mlx5_6, peer_nic: 10.10.251.105@mlx5_7, dest_rkey: 2096896, retry_cnt: 0): transport retry counter exceeded
....

how to debug? @ShangmingCai @alogfans @xiaguan

my steps

  1. run docker instance
docker run -itd --privileged --ipc=host --gpus all --cap-add=IPC_LOCK --device=/dev/infiniband --network=host --ulimit memlock=-1 --ulimit stack=67108864 -v /deepseek-r1:/root/.cache/huggingface --name moon_ws alogfans/mooncake:latest bash
  1. run the etcd on the node 105etcd --listen-client-urls http://0.0.0.0:12379 --advertise-client-urls http://localhost:12379
  2. run the target on the 105
cd /Mooncake-main/build/mooncake-transfer-engine/example
./transfer_engine_bench --mode=target   --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2  --local_server_name=10.10.251.105
  1. run the initiator on the 104
cd /Mooncake-main/build/mooncake-transfer-engine/example
./transfer_engine_bench  --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2  --local_server_name=10.10.251.104  --segment_id=10.10.251.105
  1. error occer on the initiator

my host env

exec python3 check_env.py in the container of initiator

root@ubuntu-104:/usr/local/lib/python3.10/dist-packages/sglang# python3 check_env.py
/bin/sh: 1: /usr/local/cuda/bin/nvcc: not found
Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Not Available
CUDA Driver Version: 570.124.06
PyTorch: 2.3.0+cu121
sglang: 0.4.7.post1
sgl_kernel: Module Not Found
flashinfer_python: Module Not Found
triton: 2.3.0
transformers: Module Not Found
torchao: Module Not Found
numpy: 1.26.4
aiohttp: 3.12.13
fastapi: 0.115.12
hf_transfer: Module Not Found
huggingface_hub: Module Not Found
interegular: Module Not Found
modelscope: Module Not Found
orjson: Module Not Found
outlines: Module Not Found
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: Module Not Found
pyzmq: 27.0.0
uvicorn: Module Not Found
uvloop: Module Not Found
vllm: Module Not Found
xgrammar: Module Not Found
openai: Module Not Found
tiktoken: Module Not Found
anthropic: Module Not Found
litellm: Module Not Found
decord: Module Not Found
NVIDIA Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     56-111,168-223  1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      PIX     NODE    SYS     SYS     SYS     SYS
NIC4    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX      X      NODE    SYS     SYS     SYS     SYS
NIC5    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC6    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC7    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC8    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC9    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9


ulimit soft: 640000

the ib info

root@ubuntu-104:~# ibstat
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c20300195cea
        System image GUID: 0xa088c20300195cea
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 16
                LMC: 0
                SM lid: 16
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c20300195cea
                Link layer: InfiniBand
CA 'mlx5_1'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0x946dae0300b26f64
        System image GUID: 0x946dae0300b26f64
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 2
                LMC: 0
                SM lid: 2
                Capability mask: 0xa751e84a
                Port GUID: 0x946dae0300b26f64
                Link layer: InfiniBand
CA 'mlx5_2'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c20300adeeb8
        System image GUID: 0xa088c20300adeeb8
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 4
                LMC: 0
                SM lid: 4
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c20300adeeb8
                Link layer: InfiniBand
CA 'mlx5_3'
        CA type: MT41692
        Number of ports: 1
        Firmware version: 32.41.1000
        Hardware version: 1
        Node GUID: 0x58a2e10300669170
        System image GUID: 0x58a2e10300669170
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffe669170
                Link layer: Ethernet
CA 'mlx5_4'
        CA type: MT41692
        Number of ports: 1
        Firmware version: 32.41.1000
        Hardware version: 1
        Node GUID: 0x58a2e10300669171
        System image GUID: 0x58a2e10300669170
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x5aa2e1fffe669171
                Link layer: Ethernet
CA 'mlx5_5'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c20300125594
        System image GUID: 0xa088c20300125594
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 6
                LMC: 0
                SM lid: 6
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c20300125594
                Link layer: InfiniBand
CA 'mlx5_6'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c203001255cc
        System image GUID: 0xa088c203001255cc
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 8
                LMC: 0
                SM lid: 8
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c203001255cc
                Link layer: InfiniBand
CA 'mlx5_7'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c203001255a4
        System image GUID: 0xa088c203001255a4
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 10
                LMC: 0
                SM lid: 10
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c203001255a4
                Link layer: InfiniBand
CA 'mlx5_8'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c2030018db50
        System image GUID: 0xa088c2030018db50
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 12
                LMC: 0
                SM lid: 12
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c2030018db50
                Link layer: InfiniBand
CA 'mlx5_9'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.1002
        Hardware version: 0
        Node GUID: 0xa088c2030018cb50
        System image GUID: 0xa088c2030018cb50
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 14
                LMC: 0
                SM lid: 14
                Capability mask: 0xa751e84a
                Port GUID: 0xa088c2030018cb50
                Link layer: InfiniBand
Originally created by @XiaobinZhao on GitHub (Jun 17, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/507 i run 1p1d sglang with mooncake,but failed. the error log is: `... Worker: Process failed for slice ...`; i searched the issue in the github mooncake; and try the mooncake [Example: Transfer Engine Bench](https://kvcache-ai.github.io/Mooncake/design/transfer-engine.html#example-transfer-engine-bench) , the error is same， ``` root@ubuntu-104:/Mooncake-main/build/mooncake-transfer-engine/example# ./transfer_engine_bench --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2 --local_server_name=10.10.251.104 --segment_id=10.10.251.105 WARNING: Logging before InitGoogleLogging() is written to STDERR I0617 08:42:42.470683 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_6/ I0617 08:42:42.471800 413 rdma_context.cpp:125] RDMA device: mlx5_6, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:12:55:cc I0617 08:42:42.498806 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_7/ I0617 08:42:42.499879 413 rdma_context.cpp:125] RDMA device: mlx5_7, LID: 10, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:12:55:a4 I0617 08:42:42.526409 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_8/ I0617 08:42:42.527472 413 rdma_context.cpp:125] RDMA device: mlx5_8, LID: 12, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:18:db:50 I0617 08:42:42.554749 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_9/ I0617 08:42:42.555806 413 rdma_context.cpp:125] RDMA device: mlx5_9, LID: 14, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:18:cb:50 I0617 08:42:42.582837 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_0/ I0617 08:42:42.583910 413 rdma_context.cpp:125] RDMA device: mlx5_0, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:19:5c:ea I0617 08:42:42.611243 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_1/ I0617 08:42:42.612282 413 rdma_context.cpp:125] RDMA device: mlx5_1, LID: 2, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:94:6d:ae:03:00:b2:6f:64 I0617 08:42:42.638762 413 rdma_context.cpp:426] Find best gid index: 0 on mlx5_2/ I0617 08:42:42.639797 413 rdma_context.cpp:125] RDMA device: mlx5_2, LID: 4, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:a0:88:c2:03:00:ad:ee:b8 E0617 08:42:47.012193 417 worker_pool.cpp:291] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f60b0027000, length: 4096, dest_addr: 139879971450880, local_nic: mlx5_6, peer_nic: 10.10.251.105@mlx5_7, dest_rkey: 2096896, retry_cnt: 0): transport retry counter exceeded .... ``` how to debug? @ShangmingCai @alogfans @xiaguan #### my steps 1. run docker instance ``` docker run -itd --privileged --ipc=host --gpus all --cap-add=IPC_LOCK --device=/dev/infiniband --network=host --ulimit memlock=-1 --ulimit stack=67108864 -v /deepseek-r1:/root/.cache/huggingface --name moon_ws alogfans/mooncake:latest bash ``` 2. run the etcd on the node 105`etcd --listen-client-urls http://0.0.0.0:12379 --advertise-client-urls http://localhost:12379` 3. run the target on the 105 ``` cd /Mooncake-main/build/mooncake-transfer-engine/example ./transfer_engine_bench --mode=target --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2 --local_server_name=10.10.251.105 ``` 4. run the initiator on the 104 ``` cd /Mooncake-main/build/mooncake-transfer-engine/example ./transfer_engine_bench --metadata_server=etcd://10.10.251.105:12379 --device_name=mlx5_6,mlx5_7,mlx5_8,mlx5_9,mlx5_0,mlx5_1,mlx5_2 --local_server_name=10.10.251.104 --segment_id=10.10.251.105 ``` 5. error occer on the initiator #### my host env exec `python3 check_env.py` in the container of initiator ``` root@ubuntu-104:/usr/local/lib/python3.10/dist-packages/sglang# python3 check_env.py /bin/sh: 1: /usr/local/cuda/bin/nvcc: not found Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H200 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Not Available CUDA Driver Version: 570.124.06 PyTorch: 2.3.0+cu121 sglang: 0.4.7.post1 sgl_kernel: Module Not Found flashinfer_python: Module Not Found triton: 2.3.0 transformers: Module Not Found torchao: Module Not Found numpy: 1.26.4 aiohttp: 3.12.13 fastapi: 0.115.12 hf_transfer: Module Not Found huggingface_hub: Module Not Found interegular: Module Not Found modelscope: Module Not Found orjson: Module Not Found outlines: Module Not Found packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.7 python-multipart: Module Not Found pyzmq: 27.0.0 uvicorn: Module Not Found uvloop: Module Not Found vllm: Module Not Found xgrammar: Module Not Found openai: Module Not Found tiktoken: Module Not Found anthropic: Module Not Found litellm: Module Not Found decord: Module Not Found NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE NODE NODE SYS SYS SYS SYS 0-55,112-167 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE NODE NODE SYS SYS SYS SYS 0-55,112-167 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE NODE SYS SYS SYS SYS 0-55,112-167 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE PIX SYS SYS SYS SYS 0-55,112-167 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE 56-111,168-223 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE 56-111,168-223 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE PIX NODE 56-111,168-223 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX 56-111,168-223 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NODE NODE SYS SYS SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X PIX NODE SYS SYS SYS SYS NIC4 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE PIX X NODE SYS SYS SYS SYS NIC5 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE NODE X SYS SYS SYS SYS NIC6 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS X NODE NODE NODE NIC7 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS NODE X NODE NODE NIC8 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS NODE NODE X NODE NIC9 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 ulimit soft: 640000 ``` the ib info ``` root@ubuntu-104:~# ibstat CA 'mlx5_0' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c20300195cea System image GUID: 0xa088c20300195cea Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 16 LMC: 0 SM lid: 16 Capability mask: 0xa751e84a Port GUID: 0xa088c20300195cea Link layer: InfiniBand CA 'mlx5_1' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0x946dae0300b26f64 System image GUID: 0x946dae0300b26f64 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 2 LMC: 0 SM lid: 2 Capability mask: 0xa751e84a Port GUID: 0x946dae0300b26f64 Link layer: InfiniBand CA 'mlx5_2' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c20300adeeb8 System image GUID: 0xa088c20300adeeb8 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 4 LMC: 0 SM lid: 4 Capability mask: 0xa751e84a Port GUID: 0xa088c20300adeeb8 Link layer: InfiniBand CA 'mlx5_3' CA type: MT41692 Number of ports: 1 Firmware version: 32.41.1000 Hardware version: 1 Node GUID: 0x58a2e10300669170 System image GUID: 0x58a2e10300669170 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffe669170 Link layer: Ethernet CA 'mlx5_4' CA type: MT41692 Number of ports: 1 Firmware version: 32.41.1000 Hardware version: 1 Node GUID: 0x58a2e10300669171 System image GUID: 0x58a2e10300669170 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0x5aa2e1fffe669171 Link layer: Ethernet CA 'mlx5_5' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c20300125594 System image GUID: 0xa088c20300125594 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 6 LMC: 0 SM lid: 6 Capability mask: 0xa751e84a Port GUID: 0xa088c20300125594 Link layer: InfiniBand CA 'mlx5_6' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c203001255cc System image GUID: 0xa088c203001255cc Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 8 LMC: 0 SM lid: 8 Capability mask: 0xa751e84a Port GUID: 0xa088c203001255cc Link layer: InfiniBand CA 'mlx5_7' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c203001255a4 System image GUID: 0xa088c203001255a4 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 10 LMC: 0 SM lid: 10 Capability mask: 0xa751e84a Port GUID: 0xa088c203001255a4 Link layer: InfiniBand CA 'mlx5_8' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c2030018db50 System image GUID: 0xa088c2030018db50 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 12 LMC: 0 SM lid: 12 Capability mask: 0xa751e84a Port GUID: 0xa088c2030018db50 Link layer: InfiniBand CA 'mlx5_9' CA type: MT4129 Number of ports: 1 Firmware version: 28.39.1002 Hardware version: 0 Node GUID: 0xa088c2030018cb50 System image GUID: 0xa088c2030018cb50 Port 1: State: Active Physical state: LinkUp Rate: 400 Base lid: 14 LMC: 0 SM lid: 14 Capability mask: 0xa751e84a Port GUID: 0xa088c2030018cb50 Link layer: InfiniBand ```
bsakaguchi 2026-10-29 21:10:30 +00:00
bsakaguchi commented 2026-10-29 21:10:30 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Jun 17, 2025):

@alogfans Please check this. It seems IB device fault tolerance is broken even with only 2 NICs.

<!-- gh-comment-id:2979948321 --> @ShangmingCai commented on GitHub (Jun 17, 2025): @alogfans Please check this. It seems IB device fault tolerance is broken even with only 2 NICs.
bsakaguchi commented 2026-10-29 21:10:30 +00:00
Author
Owner
Copy link

@XiaobinZhao commented on GitHub (Jun 18, 2025):

@alogfans
i test 1p1d sglang with mooncake again. I test the mooncake 2 versions:

  • mooncake-transfer-engine 0.3.1
  • mooncake-transfer-engine 0.3.3.post2
    they all failed with --disaggregation-ib-device mlx5_0,mlx5_1; but success with only set one NIC.

the command success

# run docker
docker run -itd --privileged --ipc=host --gpus all --cap-add=IPC_LOCK --device=/dev/infiniband --network=host --ulimit memlock=-1 --ulimit stack=67108864 -v /deepseek-r1:/root/.cache/huggingface --name sglang_ws sglang-deepep:dev-deepep-0.4.7 bash

# prefill
SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512  NCCL_SOCKET_IFNAME="eno1"  python3 -m sglang.launch_server --model-path /root/.cache/huggingface/DeepSeek-R1  --disaggregation-mode prefill --host 0.0.0.0 --port 9000 --trust-remote-code --dist-init-addr 10.10.251.104:7000 --nnodes 1 --node-rank 0 --tp-size 8 --dp-size 8 --enable-dp-attention --enable-deepep-moe --deepep-mode normal --mem-fraction-static 0.8 --disaggregation-ib-device mlx5_0

# decode
SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512  NCCL_SOCKET_IFNAME="eno1"  python3 -m sglang.launch_server --model-path /root/.cache/huggingface/DeepSeek-R1  --disaggregation-mode decode --host 0.0.0.0 --port 9000 --trust-remote-code --dist-init-addr 10.10.251.105:7000 --nnodes 1 --node-rank 0 --tp-size 8 --dp-size 8 --enable-dp-attention --enable-deepep-moe --deepep-mode low_latency --mem-fraction-static 0.8 --disaggregation-ib-device mlx5_0

# ib
python3 -m sglang.srt.disaggregation.mini_lb   --prefill http://10.10.251.104:9000   --decode http://10.10.251.105:9000   --host 0.0.0.0   --port 40000

# test
curl -X 'POST' 'http://127.0.0.1:40000/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
    "model": "/root/.cache/huggingface/DeepSeek-R1",
    "messages": [{"role":"user", "content":"Which number is larger, 9.11 or 9.8?"}],
    "max_tokens": 64
}'

change --disaggregation-ib-device to mlx5_1; it works; but set --disaggregation-ib-device to mlx5_0,mlx5_1 it failed; the failed log still is：

[2025-06-18 01:14:08 DP3 TP3] Prefill batch. #new-seq: 1, #new-token: 19, #cached-token: 1, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0, time: 60.99
E0618 01:14:14.351258  4492 worker_pool.cpp:289] Worker: Process failed for slice (opcode: 1, source_addr: 0x7fb552000480, length: 1152, dest_addr: 0x7f902e003a80, local_nic: mlx5_0, peer_nic: 10.10.251.105:16245@mlx5_1, dest_rkey: 2112968, retry_cnt: 0): transport retry counter exceeded
E0618 01:14:14.491779  4507 worker_pool.cpp:289] Worker: Process failed for slice (opcode: 1, source_addr: 0x7fb564000480, length: 1152, dest_addr: 0x7f9040003a80, local_nic: mlx5_1, peer_nic: 10.10.251.105:16245@mlx5_0, dest_rkey: 2110860, retry_cnt: 0): transport retry counter exceeded
<!-- gh-comment-id:2982289070 --> @XiaobinZhao commented on GitHub (Jun 18, 2025): @alogfans i test 1p1d sglang with mooncake again. I test the mooncake 2 versions: - mooncake-transfer-engine 0.3.1 - mooncake-transfer-engine 0.3.3.post2 they all failed with `--disaggregation-ib-device mlx5_0,mlx5_1`; but success with only set one NIC. ### the command success ``` # run docker docker run -itd --privileged --ipc=host --gpus all --cap-add=IPC_LOCK --device=/dev/infiniband --network=host --ulimit memlock=-1 --ulimit stack=67108864 -v /deepseek-r1:/root/.cache/huggingface --name sglang_ws sglang-deepep:dev-deepep-0.4.7 bash # prefill SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512 NCCL_SOCKET_IFNAME="eno1" python3 -m sglang.launch_server --model-path /root/.cache/huggingface/DeepSeek-R1 --disaggregation-mode prefill --host 0.0.0.0 --port 9000 --trust-remote-code --dist-init-addr 10.10.251.104:7000 --nnodes 1 --node-rank 0 --tp-size 8 --dp-size 8 --enable-dp-attention --enable-deepep-moe --deepep-mode normal --mem-fraction-static 0.8 --disaggregation-ib-device mlx5_0 # decode SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=512 NCCL_SOCKET_IFNAME="eno1" python3 -m sglang.launch_server --model-path /root/.cache/huggingface/DeepSeek-R1 --disaggregation-mode decode --host 0.0.0.0 --port 9000 --trust-remote-code --dist-init-addr 10.10.251.105:7000 --nnodes 1 --node-rank 0 --tp-size 8 --dp-size 8 --enable-dp-attention --enable-deepep-moe --deepep-mode low_latency --mem-fraction-static 0.8 --disaggregation-ib-device mlx5_0 # ib python3 -m sglang.srt.disaggregation.mini_lb --prefill http://10.10.251.104:9000 --decode http://10.10.251.105:9000 --host 0.0.0.0 --port 40000 # test curl -X 'POST' 'http://127.0.0.1:40000/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ "model": "/root/.cache/huggingface/DeepSeek-R1", "messages": [{"role":"user", "content":"Which number is larger, 9.11 or 9.8?"}], "max_tokens": 64 }' ``` change `--disaggregation-ib-device` to `mlx5_1`; it works; but set `--disaggregation-ib-device` to `mlx5_0,mlx5_1` it failed; the failed log still is： ``` [2025-06-18 01:14:08 DP3 TP3] Prefill batch. #new-seq: 1, #new-token: 19, #cached-token: 1, token usage: 0.00, #running-req: 0, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0, time: 60.99 E0618 01:14:14.351258 4492 worker_pool.cpp:289] Worker: Process failed for slice (opcode: 1, source_addr: 0x7fb552000480, length: 1152, dest_addr: 0x7f902e003a80, local_nic: mlx5_0, peer_nic: 10.10.251.105:16245@mlx5_1, dest_rkey: 2112968, retry_cnt: 0): transport retry counter exceeded E0618 01:14:14.491779 4507 worker_pool.cpp:289] Worker: Process failed for slice (opcode: 1, source_addr: 0x7fb564000480, length: 1152, dest_addr: 0x7f9040003a80, local_nic: mlx5_1, peer_nic: 10.10.251.105:16245@mlx5_0, dest_rkey: 2110860, retry_cnt: 0): transport retry counter exceeded ```
bsakaguchi commented 2026-10-29 21:10:31 +00:00
Author
Owner
Copy link

@mugglew commented on GitHub (Jun 18, 2025):

@XiaobinZhao 您好，请问你的镜像可以上传到dockerhub吗？我的环境安装sglang和mooncake遇到了困难，可以直接用您的镜像吗？不胜感谢

<!-- gh-comment-id:2983211077 --> @mugglew commented on GitHub (Jun 18, 2025): @XiaobinZhao 您好，请问你的镜像可以上传到dockerhub吗？我的环境安装sglang和mooncake遇到了困难，可以直接用您的镜像吗？不胜感谢
bsakaguchi commented 2026-10-29 21:10:31 +00:00
Author
Owner
Copy link

@XiaobinZhao commented on GitHub (Jun 18, 2025):

@XiaobinZhao 您好，请问你的镜像可以上传到dockerhub吗？我的环境安装sglang和mooncake遇到了困难，可以直接用您的镜像吗？不胜感谢

pushing... you will get it after 30mins

https://hub.docker.com/r/zxb000/sglang-deepep

<!-- gh-comment-id:2983255386 --> @XiaobinZhao commented on GitHub (Jun 18, 2025): > [@XiaobinZhao](https://github.com/XiaobinZhao) 您好，请问你的镜像可以上传到dockerhub吗？我的环境安装sglang和mooncake遇到了困难，可以直接用您的镜像吗？不胜感谢 pushing... you will get it after 30mins https://hub.docker.com/r/zxb000/sglang-deepep
bsakaguchi commented 2026-10-29 21:10:31 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 4, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4880606909 --> @github-actions[bot] commented on GitHub (Jul 4, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:10:31 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 13, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4954429072 --> @github-actions[bot] commented on GitHub (Jul 13, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#173
No description provided.