kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #456] Failed to start put operation: RPC_FAIL #162

New issue

Closed

opened 2026-10-29 21:10:18 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented

2026-10-29 21:10:18 +00:00

Owner

Originally created by @mugglew on GitHub (Jun 6, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/456

I use mooncake to test PD Disaggrated
It starts successful but can handle llm chat request
but it reports error in prefill log

E0606 09:39:00.285498 382 master_client.cpp:136] Failed to start put operation:
2025-06-06 09:39:00.285457 ERROR [382] [coro_rpc_client.hpp:923] client has been closed, please re-connect
E0606 09:39:00.285537 382 client.cpp:323] Failed to start put operation: RPC_FAIL
E0606 09:39:00.285573 382 store_py.cpp:372] Put operation failed with error: RPC_FAIL
E0606 09:39:00.286211 382 master_client.cpp:136] Failed to start put operation:
2025-06-06 09:39:00.286185 ERROR [382] [coro_rpc_client.hpp:923] client has been closed, please re-connect
E0606 09:39:00.286233 382 client.cpp:323] Failed to start put operation: RPC_FAIL
E0606 09:39:00.286243 382 store_py.cpp:372] Put operation failed with error: RPC_FAIL

and in decode log

WARNING 06-06 09:39:00 [mooncake_store_connector.py:182] [rank0]: Failed to receive all KVs and hidden states, redo model forwarding.
(VllmWorkerProcess pid=524) WARNING 06-06 09:39:00 [mooncake_store_connector.py:182] [rank1]: Failed to receive all KVs and hidden states, redo model forwarding.

my devices are 4*H200
and here is my start shell

CONF="./mooncake.json"
cat > "$CONF" << 'EOF'
{
    "local_hostname": "0.0.0.0",
    "metadata_server": "etcd://0.0.0.0:2379",
    "protocol": "tcp",
    "device_name": "",
    "global_segment_size": 33554432000,
    "local_buffer_size": 10737418240,
    "master_server_address": "0.0.0.0:50001"
}
EOF

unset http_proxy
unset https_proxy
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 &

mooncake_master --port 50001 --enable_gc=true &

# 3. Run multiple vllm instances
# kv_producer role
CUDA_VISIBLE_DEVICES=0,1 \
MOONCAKE_CONFIG_PATH=$CONF \
VLLM_USE_V1=0 \
VLLM_ATTENTION_BACKEND=FLASH_ATTN \
VLLM_FLASH_ATTN_VERSION=3 \
vllm serve /QWEN-25-72B-INSTRUCT/Qwen2.5-72B-Instruct \
  --port 8100 \
  --served-model-name llm \
  --max-model-len 32000 \
  --max-num-batched-tokens 32000 \
  --max-num-seqs 2048 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8 \
  --kv-transfer-config \
  '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' > /prefiller.log 2>&1 &

# kv_consumer role
CUDA_VISIBLE_DEVICES=2,3 \
MOONCAKE_CONFIG_PATH=$CONF \
VLLM_USE_V1=0 \
VLLM_ATTENTION_BACKEND=FLASH_ATTN \
VLLM_FLASH_ATTN_VERSION=3 \
vllm serve /QWEN-25-72B-INSTRUCT/Qwen2.5-72B-Instruct \
    --served-model-name llm \
    --port 8200 \
    --max-model-len 32000 \
    --max-num-seqs 2048 \
    --gpu-memory-utilization 0.9 \
    --tensor-parallel-size 2 \
    --no-enable-chunked-prefill \
    --max-num-batched-tokens 32000 \
    --max-seq-len-to-capture 32000 \
    --kv-cache-dtype fp8 \
    --kv-transfer-config \
    '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}' > /decoder.log 2>&1 &

python3 /workspace/vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py \
    --host 0.0.0.0 \
    --prefiller-host 0.0.0.0 \
    --decoder-host 0.0.0.0

tail -f /dev/null

Actually i used http meta server before, it's okay and just a little bit slow
After using etcd this issue occurs

CAN SOMEONE HELP ME PLZ
THANK YOU SO MUCH

Originally created by @mugglew on GitHub (Jun 6, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/456 I use mooncake to test PD Disaggrated It starts successful but can handle llm chat request but it reports error in prefill log > E0606 09:39:00.285498 382 master_client.cpp:136] Failed to start put operation: 2025-06-06 09:39:00.285457 ERROR [382] [coro_rpc_client.hpp:923] client has been closed, please re-connect E0606 09:39:00.285537 382 client.cpp:323] Failed to start put operation: RPC_FAIL E0606 09:39:00.285573 382 store_py.cpp:372] Put operation failed with error: RPC_FAIL E0606 09:39:00.286211 382 master_client.cpp:136] Failed to start put operation: 2025-06-06 09:39:00.286185 ERROR [382] [coro_rpc_client.hpp:923] client has been closed, please re-connect E0606 09:39:00.286233 382 client.cpp:323] Failed to start put operation: RPC_FAIL E0606 09:39:00.286243 382 store_py.cpp:372] Put operation failed with error: RPC_FAIL and in decode log > WARNING 06-06 09:39:00 [mooncake_store_connector.py:182] [rank0]: Failed to receive all KVs and hidden states, redo model forwarding. (VllmWorkerProcess pid=524) WARNING 06-06 09:39:00 [mooncake_store_connector.py:182] [rank1]: Failed to receive all KVs and hidden states, redo model forwarding. my devices are 4*H200 and here is my start shell ```sh CONF="./mooncake.json" cat > "$CONF" << 'EOF' { "local_hostname": "0.0.0.0", "metadata_server": "etcd://0.0.0.0:2379", "protocol": "tcp", "device_name": "", "global_segment_size": 33554432000, "local_buffer_size": 10737418240, "master_server_address": "0.0.0.0:50001" } EOF unset http_proxy unset https_proxy etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 & mooncake_master --port 50001 --enable_gc=true & # 3. Run multiple vllm instances # kv_producer role CUDA_VISIBLE_DEVICES=0,1 \ MOONCAKE_CONFIG_PATH=$CONF \ VLLM_USE_V1=0 \ VLLM_ATTENTION_BACKEND=FLASH_ATTN \ VLLM_FLASH_ATTN_VERSION=3 \ vllm serve /QWEN-25-72B-INSTRUCT/Qwen2.5-72B-Instruct \ --port 8100 \ --served-model-name llm \ --max-model-len 32000 \ --max-num-batched-tokens 32000 \ --max-num-seqs 2048 \ --gpu-memory-utilization 0.9 \ --tensor-parallel-size 2 \ --kv-cache-dtype fp8 \ --kv-transfer-config \ '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' > /prefiller.log 2>&1 & # kv_consumer role CUDA_VISIBLE_DEVICES=2,3 \ MOONCAKE_CONFIG_PATH=$CONF \ VLLM_USE_V1=0 \ VLLM_ATTENTION_BACKEND=FLASH_ATTN \ VLLM_FLASH_ATTN_VERSION=3 \ vllm serve /QWEN-25-72B-INSTRUCT/Qwen2.5-72B-Instruct \ --served-model-name llm \ --port 8200 \ --max-model-len 32000 \ --max-num-seqs 2048 \ --gpu-memory-utilization 0.9 \ --tensor-parallel-size 2 \ --no-enable-chunked-prefill \ --max-num-batched-tokens 32000 \ --max-seq-len-to-capture 32000 \ --kv-cache-dtype fp8 \ --kv-transfer-config \ '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}' > /decoder.log 2>&1 & python3 /workspace/vllm/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py \ --host 0.0.0.0 \ --prefiller-host 0.0.0.0 \ --decoder-host 0.0.0.0 tail -f /dev/null ``` Actually i used http meta server before, it's okay and just a little bit slow After using etcd this issue occurs CAN SOMEONE HELP ME PLZ THANK YOU SO MUCH

bsakaguchi

2026-10-29 21:10:18 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Jun 6, 2025):

It means your decode node does the prefill run by itself. @xiaguan can you take a look at this?

@ShangmingCai commented on GitHub (Jun 6, 2025): It means your decode node does the prefill run by itself. @xiaguan can you take a look at this?

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 6, 2025):

I suspect the issue arises because the whl package on PyPI lacks the etcd plugin, which prevents the client from initializing successfully. As a temporary measure, you can use the HTTP metadata server instead, as it's not on the performance-critical path. We should aim to release a package with the etcd plugin @ShangmingCai ?

@xiaguan commented on GitHub (Jun 6, 2025): I suspect the issue arises because the whl package on PyPI lacks the etcd plugin, which prevents the client from initializing successfully. As a temporary measure, you can use the HTTP metadata server instead, as it's not on the performance-critical path. We should aim to release a package with the etcd plugin @ShangmingCai ?

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@mugglew commented on GitHub (Jun 6, 2025):

@xiaguan Thank u for ur reply
Yeah I do tried http meta server with same config above before，it worked，but much slow than non PD with 4tp（same nums of gpu）. is that normal？ i thought it is because i did not use etcd so it is slow

@mugglew commented on GitHub (Jun 6, 2025): @xiaguan Thank u for ur reply Yeah I do tried http meta server with same config above before，it worked，but much slow than non PD with 4tp（same nums of gpu）. is that normal？ i thought it is because i did not use etcd so it is slow

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 6, 2025):

The metadata server is not on the critical performance path. I recall that the performance in v0 was due to the overhead caused by serialization and deserialization?

@xiaguan commented on GitHub (Jun 6, 2025): The metadata server is not on the critical performance path. I recall that the performance in v0 was due to the overhead caused by serialization and deserialization?

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@mugglew commented on GitHub (Jun 7, 2025):

The metadata server is not on the critical performance path. I recall that the performance in v0 was due to the overhead caused by serialization and deserialization?

@xiaguan It seems so.
Is that v1 with mooncake lmcache would perform much better？

@mugglew commented on GitHub (Jun 7, 2025): > The metadata server is not on the critical performance path. I recall that the performance in v0 was due to the overhead caused by serialization and deserialization? @xiaguan It seems so. Is that v1 with mooncake lmcache would perform much better？

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@xiaguan commented on GitHub (Jun 9, 2025):

Yeah, I’d guess so. And there should be more performance improvements coming for lmcache in the future too.

@xiaguan commented on GitHub (Jun 9, 2025): Yeah, I’d guess so. And there should be more performance improvements coming for lmcache in the future too.

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 8, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 8, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:49 +00:00

[PR #170] [CLOSED] [TransferEngine] Add shared memory transport and refactor metadata format #1369