kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #371] Failed to retrieve segment descriptor with tensor-parallelism #131

New issue
Closed
opened 2026-10-29 21:09:53 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-29 21:09:53 +00:00
Owner
Copy link

Originally created by @qingqiuhe on GitHub (May 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/371

image Dockerfile

FROM vllm/vllm-openai:v0.8.5.post1
RUN git clone https://github.com/kvcache-ai/Mooncake.git /mooncake && apt-get update && cd /mooncake && bash dependencies.sh && mkdir build && cd build && cmake .. && make -j && make install && cp -r /mooncake/build /mooncake-build && rm -rf /mooncake

2 nodes x 2 gpus

mooncake.json for node1:

{
    "local_hostname": "172.17.120.17:13230",
    "metadata_server": "http://172.17.120.12:8080/metadata",
    "protocol": "rdma",
    "device_name": "mlx5_2",
    "master_server_address": "172.17.120.12:50001",
    "global_segment_size": 33554432000
}

mooncake.json for node2:

{
    "local_hostname": "172.17.120.16:13230",
    "metadata_server": "http://172.17.120.12:8080/metadata",
    "protocol": "rdma",
    "device_name": "mlx5_2",
    "master_server_address": "172.17.120.12:50001",
    "global_segment_size": 33554432000
}

vllm is runned in container created by "docker run --privileged --net host ...."
both command is :

MC_TE_METRIC=1 CUDA_VISIBLE_DEVICES=6,7 VLLM_USE_V1=0 MOONCAKE_CONFIG_PATH=/models/mooncake.json python3 -m vllm.entrypoints.openai.api_server --port 18083 --model /models/Qwen3-32B  --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}' --disable-log-requests  --max_model_len 2048 --tensor-parallel-size 2

error logs:

INFO:     172.17.120.18:55274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     172.17.120.18:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     172.17.120.18:55282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     172.17.120.18:55286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     172.17.120.18:55290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
WARNING 05-15 01:26:55 [mooncake_store_connector.py:181] [rank0]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=4894) WARNING 05-15 01:26:55 [mooncake_store_connector.py:181] [rank1]: Failed to receive all KVs and hidden states, redo model forwarding.
I0515 01:26:55.973487  4822 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 172.17.120.17:13230 address 0x7fbcef000000--0x7fbcef010000
I0515 01:26:55.973522  4822 transfer_metadata_dump.cpp:63] -----------------------------------------------------------
I0515 01:26:55.973529  4822 transfer_metadata_dump.cpp:64] TransferMetadata::dumpMetadataContent
I0515 01:26:55.973536  4822 transfer_metadata_dump.cpp:65] -----------------------------------------------------------
I0515 01:26:55.973541  4822 transfer_metadata_dump.cpp:66] === Cached Segment Descriptors ===
I0515 01:26:55.973544  4822 transfer_metadata_dump.cpp:72] segment id: 0, ref object 0x12fab1d0
I0515 01:26:55.973587  4822 transfer_metadata_dump.cpp:20]   segment name: 172.17.120.17:13230
I0515 01:26:55.973608  4822 transfer_metadata_dump.cpp:21]   protocol: rdma
I0515 01:26:55.973616  4822 transfer_metadata_dump.cpp:22]   topology: {
        "cpu:0" :
        [
                [
                        "mlx5_2"
                ],
                []
        ]
}
I0515 01:26:55.973865  4822 transfer_metadata_dump.cpp:23]   devices:
I0515 01:26:55.973913  4822 transfer_metadata_dump.cpp:25]     device name mlx5_2, lid 21, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:42:32
I0515 01:26:55.973932  4822 transfer_metadata_dump.cpp:28]   buffers:
I0515 01:26:55.973938  4822 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7f39b3000000--0x7f39f3000000
I0515 01:26:55.973956  4822 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7f31e2000000--0x7f39b2000000
I0515 01:26:55.973970  4822 transfer_metadata_dump.cpp:34]   nvmeof buffers: 0 items
I0515 01:26:55.974001  4822 transfer_metadata_dump.cpp:35]   timestamp:
I0515 01:26:55.974009  4822 transfer_metadata_dump.cpp:77] === Local RPC Route ===
I0515 01:26:55.974016  4822 transfer_metadata_dump.cpp:78] location: 172.17.120.17:15899
I0515 01:26:55.974022  4822 transfer_metadata_dump.cpp:80] === Remote RPC Routes ===
E0515 01:26:55.974072  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:55.974085  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:55.984163  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:55.984179  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:55.994268  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:55.994282  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.004354  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.004510  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
E0515 01:26:56.010552  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.010571  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.020640  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.020655  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.030745  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.030761  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.040831  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.040928  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
I0515 01:26:56.077957  4894 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 172.17.120.16:13230 address 0x7f3ce3000000--0x7f3ce3010000
I0515 01:26:56.077991  4894 transfer_metadata_dump.cpp:63] -----------------------------------------------------------
I0515 01:26:56.078022  4894 transfer_metadata_dump.cpp:64] TransferMetadata::dumpMetadataContent
I0515 01:26:56.078027  4894 transfer_metadata_dump.cpp:65] -----------------------------------------------------------
I0515 01:26:56.078032  4894 transfer_metadata_dump.cpp:66] === Cached Segment Descriptors ===
I0515 01:26:56.078058  4894 transfer_metadata_dump.cpp:72] segment id: 1, ref object 0x2d553120
I0515 01:26:56.078076  4894 transfer_metadata_dump.cpp:20]   segment name: 172.17.120.16:13230
I0515 01:26:56.078125  4894 transfer_metadata_dump.cpp:21]   protocol: rdma
I0515 01:26:56.078145  4894 transfer_metadata_dump.cpp:22]   topology: {
        "cpu:0" :
        [
                [
                        "mlx5_2"
                ],
                []
        ]
}
I0515 01:26:56.078310  4894 transfer_metadata_dump.cpp:23]   devices:
I0515 01:26:56.078326  4894 transfer_metadata_dump.cpp:25]     device name mlx5_2, lid 22, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:41:52
I0515 01:26:56.078352  4894 transfer_metadata_dump.cpp:28]   buffers:
I0515 01:26:56.078366  4894 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7f28ef000000--0x7f292f000000
I0515 01:26:56.078372  4894 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7f211e000000--0x7f28ee000000
I0515 01:26:56.078393  4894 transfer_metadata_dump.cpp:34]   nvmeof buffers: 0 items
I0515 01:26:56.078414  4894 transfer_metadata_dump.cpp:35]   timestamp: 2025-05-15 01:26:35.584654
I0515 01:26:56.078418  4894 transfer_metadata_dump.cpp:72] segment id: 0, ref object 0x109c9ce0
I0515 01:26:56.078430  4894 transfer_metadata_dump.cpp:20]   segment name: 172.17.120.17:13230
I0515 01:26:56.078451  4894 transfer_metadata_dump.cpp:21]   protocol: rdma
I0515 01:26:56.078464  4894 transfer_metadata_dump.cpp:22]   topology: {
        "cpu:0" :
        [
                [
                        "mlx5_2"
                ],
                []
        ]
}
I0515 01:26:56.078552  4894 transfer_metadata_dump.cpp:23]   devices:
I0515 01:26:56.078562  4894 transfer_metadata_dump.cpp:25]     device name mlx5_2, lid 21, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:42:32
I0515 01:26:56.078564  4894 transfer_metadata_dump.cpp:28]   buffers:
I0515 01:26:56.078572  4894 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7fc4bf000000--0x7fc4ff000000
I0515 01:26:56.078574  4894 transfer_metadata_dump.cpp:30]     buffer type cpu:0, address 0x7fbcee000000--0x7fc4be000000
I0515 01:26:56.078576  4894 transfer_metadata_dump.cpp:34]   nvmeof buffers: 0 items
I0515 01:26:56.078583  4894 transfer_metadata_dump.cpp:35]   timestamp:
I0515 01:26:56.078584  4894 transfer_metadata_dump.cpp:77] === Local RPC Route ===
I0515 01:26:56.078585  4894 transfer_metadata_dump.cpp:78] location: 172.17.120.17:15567
I0515 01:26:56.078626  4894 transfer_metadata_dump.cpp:80] === Remote RPC Routes ===
E0515 01:26:56.088551  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.088569  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.089118  4894 client.cpp:448] Transfer failed for task0
W0515 01:26:56.089134  4894 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.098659  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.098675  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.099221  4894 client.cpp:448] Transfer failed for task0
W0515 01:26:56.099237  4894 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.108767  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.108781  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.109308  4894 client.cpp:448] Transfer failed for task0
W0515 01:26:56.109323  4894 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.118866  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.118963  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
E0515 01:26:56.119407  4894 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.119513  4894 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
E0515 01:26:56.140933  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.140951  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.151016  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.151032  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.161123  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.161139  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.171217  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.171298  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
E0515 01:26:56.220357  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.220376  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.230461  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.230477  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.240569  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.240584  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.250670  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.250767  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
E0515 01:26:56.257088  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.257105  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3)
E0515 01:26:56.267187  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.267203  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3)
E0515 01:26:56.277294  4822 client.cpp:448] Transfer failed for task0
W0515 01:26:56.277309  4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3)
E0515 01:26:56.287396  4822 client.cpp:464] transfer_incomplete max_attempts=3
E0515 01:26:56.287503  4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL
I0515 01:26:57.103857  4969 transfer_engine.cpp:387] [Metrics] Transfer Engine Throughput: 6.55 MB/s (over last 5s)
E0515 01:26:59.246706  4894 transfer_metadata_plugin.cpp:181] Error from http client, GET http://172.17.120.12:8080/metadata?key=mooncake%2Fram%2F172.17.120.16%3A13230 error: Timeout was reached
W0515 01:26:59.246766  4894 transfer_metadata.cpp:280] Failed to retrieve segment descriptor, name 172.17.120.16:13230
E0515 01:26:59.246784  4894 worker_pool.cpp:113] Cannot reload target segment #1
E0515 01:26:59.340948  4822 transfer_metadata_plugin.cpp:181] Error from http client, GET http://172.17.120.12:8080/metadata?key=mooncake%2Fram%2F172.17.120.16%3A13230 error: Timeout was reached
W0515 01:26:59.341012  4822 transfer_metadata.cpp:280] Failed to retrieve segment descriptor, name 172.17.120.16:13230
E0515 01:26:59.341090  4822 worker_pool.cpp:113] Cannot reload target segment #1
INFO:     172.17.120.18:55298 - "GET /metrics HTTP/1.1" 200 OK
INFO:     172.17.120.18:55308 - "GET /metrics HTTP/1.1" 200 OK
ERROR 05-15 01:27:27 [client.py:305] RuntimeError('Engine process (pid 4822) died.')
ERROR 05-15 01:27:27 [client.py:305] NoneType: None

mooncake_master also warnings:

W0515 01:00:32.142334 353781 master_service.cpp:27] segment_name=172.17.120.16:13230, error=segment_already_exists
W0515 01:01:34.353374 353783 master_service.cpp:27] segment_name=172.17.120.17:13230, error=segment_already_exists

It looks like two processes (tp=2) of the same vllm instance have used a conflicting segment_name.

when running vllm without tensor-parallelism, it's ok.

Originally created by @qingqiuhe on GitHub (May 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/371 image Dockerfile ``` FROM vllm/vllm-openai:v0.8.5.post1 RUN git clone https://github.com/kvcache-ai/Mooncake.git /mooncake && apt-get update && cd /mooncake && bash dependencies.sh && mkdir build && cd build && cmake .. && make -j && make install && cp -r /mooncake/build /mooncake-build && rm -rf /mooncake ``` 2 nodes x 2 gpus mooncake.json for node1: ``` { "local_hostname": "172.17.120.17:13230", "metadata_server": "http://172.17.120.12:8080/metadata", "protocol": "rdma", "device_name": "mlx5_2", "master_server_address": "172.17.120.12:50001", "global_segment_size": 33554432000 } ``` mooncake.json for node2: ``` { "local_hostname": "172.17.120.16:13230", "metadata_server": "http://172.17.120.12:8080/metadata", "protocol": "rdma", "device_name": "mlx5_2", "master_server_address": "172.17.120.12:50001", "global_segment_size": 33554432000 } ``` vllm is runned in container created by "docker run --privileged --net host ...." both command is : ``` MC_TE_METRIC=1 CUDA_VISIBLE_DEVICES=6,7 VLLM_USE_V1=0 MOONCAKE_CONFIG_PATH=/models/mooncake.json python3 -m vllm.entrypoints.openai.api_server --port 18083 --model /models/Qwen3-32B --gpu-memory-utilization 0.8 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}' --disable-log-requests --max_model_len 2048 --tensor-parallel-size 2 ``` error logs: ``` INFO: 172.17.120.18:55274 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO: 172.17.120.18:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO: 172.17.120.18:55282 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO: 172.17.120.18:55286 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO: 172.17.120.18:55290 - "POST /v1/chat/completions HTTP/1.1" 200 OK WARNING 05-15 01:26:55 [mooncake_store_connector.py:181] [rank0]: Failed to receive all KVs and hidden states, redo model forwarding. (VllmWorkerProcess pid=4894) WARNING 05-15 01:26:55 [mooncake_store_connector.py:181] [rank1]: Failed to receive all KVs and hidden states, redo model forwarding. I0515 01:26:55.973487 4822 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 172.17.120.17:13230 address 0x7fbcef000000--0x7fbcef010000 I0515 01:26:55.973522 4822 transfer_metadata_dump.cpp:63] ----------------------------------------------------------- I0515 01:26:55.973529 4822 transfer_metadata_dump.cpp:64] TransferMetadata::dumpMetadataContent I0515 01:26:55.973536 4822 transfer_metadata_dump.cpp:65] ----------------------------------------------------------- I0515 01:26:55.973541 4822 transfer_metadata_dump.cpp:66] === Cached Segment Descriptors === I0515 01:26:55.973544 4822 transfer_metadata_dump.cpp:72] segment id: 0, ref object 0x12fab1d0 I0515 01:26:55.973587 4822 transfer_metadata_dump.cpp:20] segment name: 172.17.120.17:13230 I0515 01:26:55.973608 4822 transfer_metadata_dump.cpp:21] protocol: rdma I0515 01:26:55.973616 4822 transfer_metadata_dump.cpp:22] topology: { "cpu:0" : [ [ "mlx5_2" ], [] ] } I0515 01:26:55.973865 4822 transfer_metadata_dump.cpp:23] devices: I0515 01:26:55.973913 4822 transfer_metadata_dump.cpp:25] device name mlx5_2, lid 21, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:42:32 I0515 01:26:55.973932 4822 transfer_metadata_dump.cpp:28] buffers: I0515 01:26:55.973938 4822 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f39b3000000--0x7f39f3000000 I0515 01:26:55.973956 4822 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f31e2000000--0x7f39b2000000 I0515 01:26:55.973970 4822 transfer_metadata_dump.cpp:34] nvmeof buffers: 0 items I0515 01:26:55.974001 4822 transfer_metadata_dump.cpp:35] timestamp: I0515 01:26:55.974009 4822 transfer_metadata_dump.cpp:77] === Local RPC Route === I0515 01:26:55.974016 4822 transfer_metadata_dump.cpp:78] location: 172.17.120.17:15899 I0515 01:26:55.974022 4822 transfer_metadata_dump.cpp:80] === Remote RPC Routes === E0515 01:26:55.974072 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:55.974085 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:55.984163 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:55.984179 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:55.994268 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:55.994282 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.004354 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.004510 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL E0515 01:26:56.010552 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.010571 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.020640 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.020655 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.030745 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.030761 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.040831 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.040928 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL I0515 01:26:56.077957 4894 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 172.17.120.16:13230 address 0x7f3ce3000000--0x7f3ce3010000 I0515 01:26:56.077991 4894 transfer_metadata_dump.cpp:63] ----------------------------------------------------------- I0515 01:26:56.078022 4894 transfer_metadata_dump.cpp:64] TransferMetadata::dumpMetadataContent I0515 01:26:56.078027 4894 transfer_metadata_dump.cpp:65] ----------------------------------------------------------- I0515 01:26:56.078032 4894 transfer_metadata_dump.cpp:66] === Cached Segment Descriptors === I0515 01:26:56.078058 4894 transfer_metadata_dump.cpp:72] segment id: 1, ref object 0x2d553120 I0515 01:26:56.078076 4894 transfer_metadata_dump.cpp:20] segment name: 172.17.120.16:13230 I0515 01:26:56.078125 4894 transfer_metadata_dump.cpp:21] protocol: rdma I0515 01:26:56.078145 4894 transfer_metadata_dump.cpp:22] topology: { "cpu:0" : [ [ "mlx5_2" ], [] ] } I0515 01:26:56.078310 4894 transfer_metadata_dump.cpp:23] devices: I0515 01:26:56.078326 4894 transfer_metadata_dump.cpp:25] device name mlx5_2, lid 22, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:41:52 I0515 01:26:56.078352 4894 transfer_metadata_dump.cpp:28] buffers: I0515 01:26:56.078366 4894 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f28ef000000--0x7f292f000000 I0515 01:26:56.078372 4894 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7f211e000000--0x7f28ee000000 I0515 01:26:56.078393 4894 transfer_metadata_dump.cpp:34] nvmeof buffers: 0 items I0515 01:26:56.078414 4894 transfer_metadata_dump.cpp:35] timestamp: 2025-05-15 01:26:35.584654 I0515 01:26:56.078418 4894 transfer_metadata_dump.cpp:72] segment id: 0, ref object 0x109c9ce0 I0515 01:26:56.078430 4894 transfer_metadata_dump.cpp:20] segment name: 172.17.120.17:13230 I0515 01:26:56.078451 4894 transfer_metadata_dump.cpp:21] protocol: rdma I0515 01:26:56.078464 4894 transfer_metadata_dump.cpp:22] topology: { "cpu:0" : [ [ "mlx5_2" ], [] ] } I0515 01:26:56.078552 4894 transfer_metadata_dump.cpp:23] devices: I0515 01:26:56.078562 4894 transfer_metadata_dump.cpp:25] device name mlx5_2, lid 21, fe:80:00:00:00:00:00:00:94:6d:ae:03:00:88:42:32 I0515 01:26:56.078564 4894 transfer_metadata_dump.cpp:28] buffers: I0515 01:26:56.078572 4894 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7fc4bf000000--0x7fc4ff000000 I0515 01:26:56.078574 4894 transfer_metadata_dump.cpp:30] buffer type cpu:0, address 0x7fbcee000000--0x7fc4be000000 I0515 01:26:56.078576 4894 transfer_metadata_dump.cpp:34] nvmeof buffers: 0 items I0515 01:26:56.078583 4894 transfer_metadata_dump.cpp:35] timestamp: I0515 01:26:56.078584 4894 transfer_metadata_dump.cpp:77] === Local RPC Route === I0515 01:26:56.078585 4894 transfer_metadata_dump.cpp:78] location: 172.17.120.17:15567 I0515 01:26:56.078626 4894 transfer_metadata_dump.cpp:80] === Remote RPC Routes === E0515 01:26:56.088551 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.088569 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.089118 4894 client.cpp:448] Transfer failed for task0 W0515 01:26:56.089134 4894 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.098659 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.098675 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.099221 4894 client.cpp:448] Transfer failed for task0 W0515 01:26:56.099237 4894 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.108767 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.108781 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.109308 4894 client.cpp:448] Transfer failed for task0 W0515 01:26:56.109323 4894 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.118866 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.118963 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL E0515 01:26:56.119407 4894 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.119513 4894 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL E0515 01:26:56.140933 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.140951 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.151016 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.151032 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.161123 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.161139 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.171217 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.171298 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL E0515 01:26:56.220357 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.220376 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.230461 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.230477 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.240569 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.240584 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.250670 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.250767 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL E0515 01:26:56.257088 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.257105 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 1/3) E0515 01:26:56.267187 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.267203 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 2/3) E0515 01:26:56.277294 4822 client.cpp:448] Transfer failed for task0 W0515 01:26:56.277309 4822 client.cpp:454] Transfer incomplete, retrying... (attempt 3/3) E0515 01:26:56.287396 4822 client.cpp:464] transfer_incomplete max_attempts=3 E0515 01:26:56.287503 4822 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL I0515 01:26:57.103857 4969 transfer_engine.cpp:387] [Metrics] Transfer Engine Throughput: 6.55 MB/s (over last 5s) E0515 01:26:59.246706 4894 transfer_metadata_plugin.cpp:181] Error from http client, GET http://172.17.120.12:8080/metadata?key=mooncake%2Fram%2F172.17.120.16%3A13230 error: Timeout was reached W0515 01:26:59.246766 4894 transfer_metadata.cpp:280] Failed to retrieve segment descriptor, name 172.17.120.16:13230 E0515 01:26:59.246784 4894 worker_pool.cpp:113] Cannot reload target segment #1 E0515 01:26:59.340948 4822 transfer_metadata_plugin.cpp:181] Error from http client, GET http://172.17.120.12:8080/metadata?key=mooncake%2Fram%2F172.17.120.16%3A13230 error: Timeout was reached W0515 01:26:59.341012 4822 transfer_metadata.cpp:280] Failed to retrieve segment descriptor, name 172.17.120.16:13230 E0515 01:26:59.341090 4822 worker_pool.cpp:113] Cannot reload target segment #1 INFO: 172.17.120.18:55298 - "GET /metrics HTTP/1.1" 200 OK INFO: 172.17.120.18:55308 - "GET /metrics HTTP/1.1" 200 OK ERROR 05-15 01:27:27 [client.py:305] RuntimeError('Engine process (pid 4822) died.') ERROR 05-15 01:27:27 [client.py:305] NoneType: None ``` mooncake_master also warnings: ``` W0515 01:00:32.142334 353781 master_service.cpp:27] segment_name=172.17.120.16:13230, error=segment_already_exists W0515 01:01:34.353374 353783 master_service.cpp:27] segment_name=172.17.120.17:13230, error=segment_already_exists ``` It looks like two processes (tp=2) of the same vllm instance have used a conflicting segment_name. when running vllm without tensor-parallelism, it's ok.
bsakaguchi 2026-10-29 21:09:53 +00:00
bsakaguchi commented 2026-10-29 21:09:54 +00:00
Author
Owner
Copy link

@xiaguan commented on GitHub (May 15, 2025):

You can try removing the port from the "local_hostname" field in json, so it would just be "172.17.120.16" and "172.17.120.16:13230".

Thanks for pointing this out! We'll make sure to update the documentation.

<!-- gh-comment-id:2883118093 --> @xiaguan commented on GitHub (May 15, 2025): You can try removing the port from the "local_hostname" field in json, so it would just be "172.17.120.16" and "172.17.120.16:13230". Thanks for pointing this out! We'll make sure to update the documentation.
bsakaguchi commented 2026-10-29 21:09:54 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 12, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4949900594 --> @github-actions[bot] commented on GitHub (Jul 12, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:09:54 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 20, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:5018723456 --> @github-actions[bot] commented on GitHub (Jul 20, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#131
No description provided.