s #147

New issue

Closed

opened 2026-10-29 21:10:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:10:08 +00:00

Owner

Originally created by @Bruce-x-1997 on GitHub (May 30, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/429

my cmd just like:
pip3 install mooncake-transfer-engine && python3 -m
sglang.launch_server --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --model-path
deepseek-ai/DeepSeek-V3-0324
--tp 16 --trust-remote-code --disaggregation-mode decode --port
30000 --host 0.0.0.0 --disaggregation-ib-device
mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_11
--dist-init-addr
$(LWS_LEADER_ADDRESS):5757
--cuda-graph-bs 128 --disable-radix-cache --watchdog-timeout 1000000
--mem-fraction-static 0.82 --context-length 4096
--dp 16 --enable-dp-attention --enable-dp-lm-head

and the connectivity between prefill & decode nodes is ok.(just node-leafswitch-spineswitch architecture).
and in node, we can see nvidia-smi topo -m:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS NODE NODE PIX NODE NODE NODE 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS NODE NODE NODE NODE PIX NODE 48-95,144-191 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS
NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NODE SYS SYS SYS SYS SYS SYS
NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NODE SYS SYS SYS SYS SYS SYS
NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X NODE SYS SYS SYS SYS SYS SYS
NIC4 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE X SYS SYS SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS X NODE NODE NODE NODE NODE
NIC6 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS NODE X NODE NODE NODE NODE
NIC7 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS NODE NODE X NODE NODE NODE
NIC8 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS NODE NODE NODE X NODE PXB
NIC9 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS NODE NODE NODE NODE X NODE
NIC10 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS NODE NODE NODE PXB NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_11
NIC10: mlx5_bond_0

so we see that the nics I used(mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_11) are all directly connected to GPU(PIX)

then when I run pd disaggregation, I see bandwith of decode pod:
from perfquery, I see bandwidth of mlx5_0, mlx5_4, mlx5_5, mlx5_11 could reach 2-4 GB/s, but mlx5_1, mlx5_2, mlx5_6, mlx5_7 just reach 10-20 KB/s
Could anyone explain this Phenomenon？thanks

Originally created by @Bruce-x-1997 on GitHub (May 30, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/429 my cmd just like: pip3 install mooncake-transfer-engine && python3 -m sglang.launch_server --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --model-path deepseek-ai/DeepSeek-V3-0324 --tp 16 --trust-remote-code --disaggregation-mode decode --port 30000 --host 0.0.0.0 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_11 --dist-init-addr $(LWS_LEADER_ADDRESS):5757 --cuda-graph-bs 128 --disable-radix-cache --watchdog-timeout 1000000 --mem-fraction-static 0.82 --context-length 4096 --dp 16 --enable-dp-attention --enable-dp-lm-head and the connectivity between prefill & decode nodes is ok.(just node-leafswitch-spineswitch architecture). and in node, we can see nvidia-smi topo -m: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS PIX NODE NODE NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS NODE PIX NODE NODE NODE NODE 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS NODE NODE PIX NODE NODE NODE 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS NODE NODE NODE NODE PIX NODE 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NODE SYS SYS SYS SYS SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE X NODE SYS SYS SYS SYS SYS SYS NIC4 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE X SYS SYS SYS SYS SYS SYS NIC5 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS X NODE NODE NODE NODE NODE NIC6 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS NODE X NODE NODE NODE NODE NIC7 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS NODE NODE X NODE NODE NODE NIC8 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS NODE NODE NODE X NODE PXB NIC9 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS NODE NODE NODE NODE X NODE NIC10 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS NODE NODE NODE PXB NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_11 NIC10: mlx5_bond_0 so we see that the nics I used(mlx5_0,mlx5_1,mlx5_2,mlx5_4,mlx5_5,mlx5_6,mlx5_7,mlx5_11) are all directly connected to GPU(PIX) then when I run pd disaggregation, I see bandwith of decode pod: from perfquery, I see bandwidth of mlx5_0, mlx5_4, mlx5_5, mlx5_11 could reach 2-4 GB/s, but mlx5_1, mlx5_2, mlx5_6, mlx5_7 just reach 10-20 KB/s Could anyone explain this Phenomenon？thanks

bsakaguchi

2026-10-29 21:10:08 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:09 +00:00

Author

Owner

@alogfans commented on GitHub (May 30, 2025):

During the Mooncake Transfer Engine's initialization steps, does it print all NIC cards that Mooncake detected? The output format likes below:

RDMA device: mlx5_X, LID: X, GID: (GID_Index X) XX:XX:XX;

Is there any message of "Failed to establish peer endpoints in local RNIC ..." or "Worker: Process failed for slice"?

@alogfans commented on GitHub (May 30, 2025): 1. During the Mooncake Transfer Engine's initialization steps, does it print all NIC cards that Mooncake detected? The output format likes below: ``` RDMA device: mlx5_X, LID: X, GID: (GID_Index X) XX:XX:XX; ``` 2. Is there any message of "Failed to establish peer endpoints in local RNIC ..." or "Worker: Process failed for slice"?

bsakaguchi commented

2026-10-29 21:10:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 9, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 9, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 17, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 17, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:43 +00:00

[PR #147] [CLOSED] [Feature] Add shared memory transport and refactor metadata format #1356

bsakaguchi referenced this issue

2026-10-29 21:26:49 +00:00

[PR #170] [CLOSED] [TransferEngine] Add shared memory transport and refactor metadata format #1369