mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #397] vllm+mooncake store，pd分离多rdma卡场景bench卡住 #136
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#136
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @qyzhaoxun on GitHub (May 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/397
诉求
vllm+mooncake-store怎么在有多张rdma卡的机器上跑pd分离
问题描述
两台机器，每台机器8张gpu和rdma卡。按照教程部署了pd分离服务，bench时会一直卡住
prefill日志报错
INFO 05-23 06:50:39 [engine.py:310] Added request cmpl-ca60681e919c4882a450a6d5c5c23538-0.
I0523 06:50:40.052094 223 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.32.5.183:12728 address 0x7f3cf6020000--0x7f3cf6020094
I0523 06:50:40.052116 223 transfer_metadata_dump.cpp:63] -----------------------------------------------------------
I0523 06:50:40.052117 223 transfer_metadata_dump.cpp:64] TransferMetadata::dumpMetadataContent
I0523 06:50:40.052119 223 transfer_metadata_dump.cpp:65] -----------------------------------------------------------
I0523 06:50:40.052120 223 transfer_metadata_dump.cpp:66] === Cached Segment Descriptors ===
I0523 06:50:40.052120 223 transfer_metadata_dump.cpp:72] segment id: 1, ref object 0x2e464850
I0523 06:50:40.052124 223 transfer_metadata_dump.cpp:20] segment name: 10.32.5.183:12728
I0523 06:50:40.052125 223 transfer_metadata_dump.cpp:21] protocol: rdma
I0523 06:50:40.052126 223 transfer_metadata_dump.cpp:22] topology: {
"cpu:0" :
[
[
"mlx5_bond_0",
"mlx5_bond_1",
"mlx5_bond_2",
"mlx5_bond_3",
"mlx5_bond_4",
"mlx5_bond_5",
"mlx5_bond_6",
"mlx5_bond_7"
],
[]
]
}
I0523 06:50:40.052135 223 transfer_metadata_dump.cpp:23] devices:
I0523 06:50:40.052136 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_0, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8a
I0523 06:50:40.052137 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_1, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:b2
I0523 06:50:40.052138 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_2, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:a6
I0523 06:50:40.052139 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_3, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8e
I0523 06:50:40.052140 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_4, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:da
I0523 06:50:40.052141 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_5, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:d6
I0523 06:50:40.052141 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_6, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:de
I0523 06:50:40.052142 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_7, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:ba
I0523 06:50:40.052143 223 transfer_metadata_dump.cpp:28] buffers:
I0523 06:50:40.052145 223 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f9637000000--0x7f9677000000
I0523 06:50:40.052145 223 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f956e000000--0x7f9636000000
I0523 06:50:40.052151 223 transfer_metadata_dump.cpp:34] nvmeof buffers: 0 items
I0523 06:50:40.052153 223 transfer_metadata_dump.cpp:35] timestamp: 2025-05-23 06:36:16.122743
I0523 06:50:40.052155 223 transfer_metadata_dump.cpp:72] segment id: 0, ref object 0x14549f30
"cpu:0" :
[
[
"mlx5_bond_0",
"mlx5_bond_1",
"mlx5_bond_2",
"mlx5_bond_3",
"mlx5_bond_4",
"mlx5_bond_5",
"mlx5_bond_6",
"mlx5_bond_7"
],
[]
]
}
I0523 06:50:40.052165 223 transfer_metadata_dump.cpp:23] devices:
I0523 06:50:40.052166 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_0, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:82
I0523 06:50:40.052168 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_1, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:a2
I0523 06:50:40.052170 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_2, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:9e
I0523 06:50:40.052173 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_3, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:96
I0523 06:50:40.052175 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_4, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:c6
I0523 06:50:40.052178 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_5, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:be
I0523 06:50:40.052181 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_6, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:ce
I0523 06:50:40.052181 223 transfer_metadata_dump.cpp:25] device name mlx5_bond_7, lid 0, 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:b6
I0523 06:50:40.052182 223 transfer_metadata_dump.cpp:28] buffers:
I0523 06:50:40.052184 223 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f4637000000--0x7f4677000000
I0523 06:50:40.052186 223 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f456e000000--0x7f4636000000
I0523 06:50:40.052188 223 transfer_metadata_dump.cpp:34] nvmeof buffers: 0 items
I0523 06:50:40.052191 223 transfer_metadata_dump.cpp:35] timestamp:
I0523 06:50:40.052192 223 transfer_metadata_dump.cpp:77] === Local RPC Route ===
I0523 06:50:40.052193 223 transfer_metadata_dump.cpp:78] location: 10.32.5.203:15337
I0523 06:50:40.052196 223 transfer_metadata_dump.cpp:80] === Remote RPC Routes ===
E0523 06:50:40.054558 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.054589 223 client.cpp:450] Transfer incomplete, retrying... (attempt 1/3)
E0523 06:50:40.064651 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.064657 223 client.cpp:450] Transfer incomplete, retrying... (attempt 2/3)
E0523 06:50:40.074718 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.074728 223 client.cpp:450] Transfer incomplete, retrying... (attempt 3/3)
E0523 06:50:40.084792 223 client.cpp:460] transfer_incomplete max_attempts=3
E0523 06:50:40.085733 223 store_py.cpp:367] Put operation failed with error: TRANSFER_FAIL
E0523 06:50:40.087774 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.087782 223 client.cpp:450] Transfer incomplete, retrying... (attempt 1/3)
E0523 06:50:40.097843 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.097848 223 client.cpp:450] Transfer incomplete, retrying... (attempt 2/3)
E0523 06:50:40.107908 223 client.cpp:444] Transfer failed for task0
W0523 06:50:40.107914 223 client.cpp:450] Transfer incomplete, retrying... (attempt 3/3)
E0523 06:50:40.117973 223 client.cpp:460] transfer_incomplete max_attempts=3
E0523 06:50:40.118625 223 store_py.cpp:367] Put operation failed with error: TRANSFER_FAIL
decode日志报错
WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank0]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=217) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank1]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=218) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank2]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=219) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank3]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=221) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank5]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=223) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank7]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=220) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank4]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=222) WARNING 05-23 06:39:54 [mooncake_store_connector.py:181] [rank6]: Failed to receive all KVs and hidden states, redo model forwarding.
mooncake配置文件
prefill mooncake配置
mooncake.json: |
{
"local_hostname": "10.32.5.203",
"prefill_url": "10.32.5.203:13003",
"decode_url": "10.32.5.183:13003",
"master_server_address": "10.32.5.215:50051",
"metadata_server": "etcd-headless.vllm-mooncake.svc.cluster.local:2379",
"metadata_backend": "etcd",
"protocol": "rdma",
"device_name": "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7"
}
decode mooncake配置
mooncake.json: |
{
"local_hostname": "10.32.5.183",
"prefill_url": "10.32.5.203:13003",
"decode_url": "10.32.5.183:13003",
"master_server_address": "10.32.5.215:50051",
"metadata_server": "etcd-headless.vllm-mooncake.svc.cluster.local:2379",
"metadata_backend": "etcd",
"protocol": "rdma",
"device_name": "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3,mlx5_bond_4,mlx5_bond_5,mlx5_bond_6,mlx5_bond_7"
}
补充说明
E0524 12:53:38.269760 773 rdma_endpoint.cpp:149] Invalid argument: received packet mismatch, local.local_nic_path: 10.32.5.183@mlx5_bond_2, local.peer_nic_path: 10.32.5.203@mlx5_bond_2, peer.local_nic_path: , peer.peer_nic_path:
E0524 12:53:38.269788 773 worker_pool.cpp:227] Worker: Cannot make connection for endpoint: 10.32.5.203@mlx5_bond_2, mark it inactive
I0524 12:53:38.276801 779 transfer_engine_bench.cpp:215] FAILED
W0524 12:53:38.276849 779 transport.h:148] detected slice leak: allocated 128 freed 0
I0524 12:53:38.276801 776 transfer_engine_bench.cpp:215] FAILED
W0524 12:53:38.276886 776 transport.h:148] detected slice leak: allocated 128 freed 0
@jiafuzha commented on GitHub (May 26, 2025):
You can try to increase listening backlog size. Please check #388.
@qyzhaoxun commented on GitHub (May 26, 2025):
@jiafuzha 非常感谢，那我应该怎么验证这个mr可以符合预期呢？我看vllm应该还没有适配0.3.2，vllm我是直接用的官方的镜像，没有自己编译vllm，是直接通过pip3 install --upgrade mooncake-transfer-engine==0.3.2就搞定了吗？另外这里有方式通过transfer_engine_bench工具验证吗？
@xiaguan commented on GitHub (May 26, 2025):
你可以从388 pr的ci里下载whl包，比如，https://github.com/kvcache-ai/Mooncake/suites/39108435316/artifacts/3194443374
@jiafuzha commented on GitHub (May 26, 2025):
#388 里面有错误描述。如果你没有发现这些错误，应该就好了。当然也可能你的错误本来就是别的问题造成的。
@mugglew commented on GitHub (Jul 17, 2025):
@qyzhaoxun hi i met same problem, have you solve that?
@mgw2168-1 commented on GitHub (Sep 5, 2025):
@mugglew 你好，我也遇到了这个问题，有啥解决办法吗？我mooncake v0.3.5
@thqq479 commented on GitHub (Mar 3, 2026):
The same problem occurs in sglang+mooncake
@thqq479 commented on GitHub (Mar 3, 2026):
I0303 12:07:47.329501 159911 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.92.241.56:15917 address 0x7f7454c94000--0x7f7454ca4000
@alogfans I encountered this error when deploying DeepSeek PD in separation mode, especially when the QPS value was high
@github-actions[bot] commented on GitHub (Jul 11, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Jul 19, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.