mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #476] Unable to find metadata storage plugin etcd with conn string: etcd://10.0.0.103:2379 #167
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#167
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @67lc on GitHub (Jun 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/476
mooncake.json
{ "prefill_url": "10.0.0.103:13003", "decode_url": "10.0.0.103:13103", "local_hostname": "10.0.0.103", "metadata_server": "etcd://10.0.0.103:2379", "protocol": "tcp", "device_name": "", "master_server_address": "10.0.0.103:50001" }command
`CUDA_VISIBLE_DEVICES=0 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server
--model /share/models/meta-llama/Llama-3.2-1B-Instruct
--port 8100
--served-model-name llama
--tensor_parallel_size 1
--dtype half
--enforce-eager
--max-model-len 10000
--gpu-memory-utilization 0.8
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}' &
python3 proxy_server.py`
I install Mooncake with pip,when I use the command.The error occur:
INFO 06-11 12:06:31 [mooncake_pipe.py:74] Mooncake Configuration loaded successfully. WARNING: Logging before InitGoogleLogging() is written to STDERR I0611 12:06:31.998637 2235985 transfer_engine.cpp:350] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0611 12:06:31.998659 2235985 transfer_engine.cpp:44] Transfer Engine starting. Server: 10.0.0.103:13103, Metadata: etcd://10.0.0.103:2379, ip_or_host_name: , rpc_port: 0 I0611 12:06:31.999114 2235985 transfer_engine.cpp:100] Transfer Engine RPC using new RPC mapping, listening on 10.0.0.103:16227 F0611 12:06:31.999123 2235985 transfer_metadata_plugin.cpp:442] Unable to find metadata storage plugin etcd with conn string: etcd://10.0.0.103:2379My etcd is running.
@alogfans commented on GitHub (Jun 12, 2025):
We recently removed the default support of etcd. To use etcd service in transfer engine, in
mooncake-common/common.cmake, change the switch ofUSE_ETCDfromOFFtoON.@67lc commented on GitHub (Jun 12, 2025):
Thank you for your answer, I think the guide of integration of vllm should be updated.
@67lc commented on GitHub (Jun 12, 2025):
When I use RDMA,the error ouccr:
ERROR 06-12 13:12:27 [mooncake_store.py:97] An error occurred while loading the configuration: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [mooncake_store.py:97] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [mooncake_store.py:97] ERROR 06-12 13:12:27 [mooncake_store.py:97] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] Traceback (most recent call last): ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 447, in run_mp_engine ERROR 06-12 13:12:27 [engine.py:459] engine = MQLLMEngine.from_vllm_config( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 134, in from_vllm_config ERROR 06-12 13:12:27 [engine.py:459] return cls( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 88, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.engine = LLMEngine(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/llm_engine.py", line 265, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.model_executor = executor_class(vllm_config=vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/executor_base.py", line 53, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self._init_executor() ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 47, in _init_executor ERROR 06-12 13:12:27 [engine.py:459] self.collective_rpc("init_device") ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc ERROR 06-12 13:12:27 [engine.py:459] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/utils.py", line 2680, in run_method ERROR 06-12 13:12:27 [engine.py:459] return func(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker_base.py", line 606, in init_device ERROR 06-12 13:12:27 [engine.py:459] self.worker.init_device() # type: ignore ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 193, in init_device ERROR 06-12 13:12:27 [engine.py:459] init_worker_distributed_environment(self.vllm_config, self.rank, ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 537, in init_worker_distributed_environment ERROR 06-12 13:12:27 [engine.py:459] ensure_kv_transfer_initialized(vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_transfer_state.py", line 67, in ensure_kv_transfer_initialized ERROR 06-12 13:12:27 [engine.py:459] _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v0( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 50, in create_connector_v0 ERROR 06-12 13:12:27 [engine.py:459] return connector_cls(rank, local_rank, config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 55, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.kv_store = MooncakeStore(config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_lookup_buffer/mooncake_store.py", line 86, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.store.setup(self.config.local_hostname, ERROR 06-12 13:12:27 [engine.py:459] TypeError: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', Nonemy mooncake.json is:
{ "local_hostname": "192.168.1.10" "prefill_url": "192.168.1.10:13003", "decode_url": "192.168.1.10:13103", "metadata_server": "192.168.1.10:2379", "metadata_backend": "etcd", "protocol": "rdma", "device_name": "erdma_0" "master_server_address": "192.168.1.10:50001" }Is this my configuration problem?
@iblisylvie commented on GitHub (Jun 13, 2025):
I also met the same problem.
So is it now to change the metadata_server from "etd://xx.xx.xx.xx:2379" to "http://xx.xx.xx.xx:2379"?
I changed it, but this time it returns
E0613 17:05:31.807329 2384597 transfer_metadata_plugin.cpp:235] Unexpected code in http response, PUT http://10.11.60.9:2379?key=mooncake%2Frpc_meta%2F10.11.60.9%3A13609 response code: 404 response body: 404 page not found
E0613 17:05:31.807374 2384597 transfer_metadata.cpp:453] Failed to set location of 10.11.60.9:13609
F0613 17:05:31.807412 2384597 client.cpp:120] Check failed: rc == 0 (-200 vs. 0) Failed to initialize transfer engine
@alogfans
@stmatengss commented on GitHub (Jun 13, 2025):
Hi, before that, you should start http server. https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py
@air6677 commented on GitHub (Jun 13, 2025):
@alogfans HI,what is the reason for disabling etcd support? If in new version to use etcd, how to enable it quickly? For future production environments, should we prioritize using an HTTP server for the metadata server?
@alogfans commented on GitHub (Jul 1, 2025):
@air6677 The dependenies of etcd support is complex and it's not used by most users. To enable it, builds it manually, with --USE_ETCD=ON. We prefer to use HTTP or P2P-based support.
@github-actions[bot] commented on GitHub (Jul 4, 2026):
This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
@github-actions[bot] commented on GitHub (Jul 12, 2026):
Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.