[GH-ISSUE #476] Unable to find metadata storage plugin etcd with conn string: etcd://10.0.0.103:2379 #167

Closed
opened 2026-10-29 21:10:24 +00:00 by bsakaguchi · 9 comments
Owner

Originally created by @67lc on GitHub (Jun 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/476

mooncake.json
{ "prefill_url": "10.0.0.103:13003", "decode_url": "10.0.0.103:13103", "local_hostname": "10.0.0.103", "metadata_server": "etcd://10.0.0.103:2379", "protocol": "tcp", "device_name": "", "master_server_address": "10.0.0.103:50001" }

command
`CUDA_VISIBLE_DEVICES=0 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server
--model /share/models/meta-llama/Llama-3.2-1B-Instruct
--port 8100
--served-model-name llama
--tensor_parallel_size 1
--dtype half
--enforce-eager
--max-model-len 10000
--gpu-memory-utilization 0.8
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}' &

CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \
--model /share/models/meta-llama/Llama-3.2-1B-Instruct \
--port 8200 \
--served-model-name llama \
--tensor_parallel_size 1 \
--enforce-eager \
--dtype half \
--max-model-len 10000 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}' &

python3 proxy_server.py`

I install Mooncake with pip,when I use the command.The error occur:
INFO 06-11 12:06:31 [mooncake_pipe.py:74] Mooncake Configuration loaded successfully. WARNING: Logging before InitGoogleLogging() is written to STDERR I0611 12:06:31.998637 2235985 transfer_engine.cpp:350] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0611 12:06:31.998659 2235985 transfer_engine.cpp:44] Transfer Engine starting. Server: 10.0.0.103:13103, Metadata: etcd://10.0.0.103:2379, ip_or_host_name: , rpc_port: 0 I0611 12:06:31.999114 2235985 transfer_engine.cpp:100] Transfer Engine RPC using new RPC mapping, listening on 10.0.0.103:16227 F0611 12:06:31.999123 2235985 transfer_metadata_plugin.cpp:442] Unable to find metadata storage plugin etcd with conn string: etcd://10.0.0.103:2379
My etcd is running.

Originally created by @67lc on GitHub (Jun 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/476 **mooncake.json** `{ "prefill_url": "10.0.0.103:13003", "decode_url": "10.0.0.103:13103", "local_hostname": "10.0.0.103", "metadata_server": "etcd://10.0.0.103:2379", "protocol": "tcp", "device_name": "", "master_server_address": "10.0.0.103:50001" }` **command** `CUDA_VISIBLE_DEVICES=0 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \ --model /share/models/meta-llama/Llama-3.2-1B-Instruct \ --port 8100 \ --served-model-name llama \ --tensor_parallel_size 1 \ --dtype half \ --enforce-eager \ --max-model-len 10000 \ --gpu-memory-utilization 0.8 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}' & CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server \ --model /share/models/meta-llama/Llama-3.2-1B-Instruct \ --port 8200 \ --served-model-name llama \ --tensor_parallel_size 1 \ --enforce-eager \ --dtype half \ --max-model-len 10000 \ --gpu-memory-utilization 0.8 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}' & python3 proxy_server.py` I install Mooncake with pip,when I use the command.The error occur: `INFO 06-11 12:06:31 [mooncake_pipe.py:74] Mooncake Configuration loaded successfully. WARNING: Logging before InitGoogleLogging() is written to STDERR I0611 12:06:31.998637 2235985 transfer_engine.cpp:350] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0611 12:06:31.998659 2235985 transfer_engine.cpp:44] Transfer Engine starting. Server: 10.0.0.103:13103, Metadata: etcd://10.0.0.103:2379, ip_or_host_name: , rpc_port: 0 I0611 12:06:31.999114 2235985 transfer_engine.cpp:100] Transfer Engine RPC using new RPC mapping, listening on 10.0.0.103:16227 F0611 12:06:31.999123 2235985 transfer_metadata_plugin.cpp:442] Unable to find metadata storage plugin etcd with conn string: etcd://10.0.0.103:2379 ` My etcd is running.
bsakaguchi 2026-10-29 21:10:24 +00:00
Author
Owner

@alogfans commented on GitHub (Jun 12, 2025):

We recently removed the default support of etcd. To use etcd service in transfer engine, in mooncake-common/common.cmake, change the switch of USE_ETCD from OFF to ON.

<!-- gh-comment-id:2964809564 --> @alogfans commented on GitHub (Jun 12, 2025): We recently removed the default support of etcd. To use etcd service in transfer engine, in `mooncake-common/common.cmake`, change the switch of `USE_ETCD` from `OFF` to `ON`.
Author
Owner

@67lc commented on GitHub (Jun 12, 2025):

Thank you for your answer, I think the guide of integration of vllm should be updated.

<!-- gh-comment-id:2966162995 --> @67lc commented on GitHub (Jun 12, 2025): Thank you for your answer, I think the guide of integration of vllm should be updated.
Author
Owner

@67lc commented on GitHub (Jun 12, 2025):

When I use RDMA,the error ouccr:
ERROR 06-12 13:12:27 [mooncake_store.py:97] An error occurred while loading the configuration: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [mooncake_store.py:97] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [mooncake_store.py:97] ERROR 06-12 13:12:27 [mooncake_store.py:97] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] Traceback (most recent call last): ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 447, in run_mp_engine ERROR 06-12 13:12:27 [engine.py:459] engine = MQLLMEngine.from_vllm_config( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 134, in from_vllm_config ERROR 06-12 13:12:27 [engine.py:459] return cls( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 88, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.engine = LLMEngine(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/llm_engine.py", line 265, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.model_executor = executor_class(vllm_config=vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/executor_base.py", line 53, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self._init_executor() ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 47, in _init_executor ERROR 06-12 13:12:27 [engine.py:459] self.collective_rpc("init_device") ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc ERROR 06-12 13:12:27 [engine.py:459] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/utils.py", line 2680, in run_method ERROR 06-12 13:12:27 [engine.py:459] return func(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker_base.py", line 606, in init_device ERROR 06-12 13:12:27 [engine.py:459] self.worker.init_device() # type: ignore ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 193, in init_device ERROR 06-12 13:12:27 [engine.py:459] init_worker_distributed_environment(self.vllm_config, self.rank, ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 537, in init_worker_distributed_environment ERROR 06-12 13:12:27 [engine.py:459] ensure_kv_transfer_initialized(vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_transfer_state.py", line 67, in ensure_kv_transfer_initialized ERROR 06-12 13:12:27 [engine.py:459] _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v0( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 50, in create_connector_v0 ERROR 06-12 13:12:27 [engine.py:459] return connector_cls(rank, local_rank, config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 55, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.kv_store = MooncakeStore(config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_lookup_buffer/mooncake_store.py", line 86, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.store.setup(self.config.local_hostname, ERROR 06-12 13:12:27 [engine.py:459] TypeError: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None
my mooncake.json is:
{ "local_hostname": "192.168.1.10" "prefill_url": "192.168.1.10:13003", "decode_url": "192.168.1.10:13103", "metadata_server": "192.168.1.10:2379", "metadata_backend": "etcd", "protocol": "rdma", "device_name": "erdma_0" "master_server_address": "192.168.1.10:50001" }
Is this my configuration problem?

<!-- gh-comment-id:2966708135 --> @67lc commented on GitHub (Jun 12, 2025): When I use RDMA,the error ouccr: `ERROR 06-12 13:12:27 [mooncake_store.py:97] An error occurred while loading the configuration: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [mooncake_store.py:97] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [mooncake_store.py:97] ERROR 06-12 13:12:27 [mooncake_store.py:97] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ERROR 06-12 13:12:27 [engine.py:459] Traceback (most recent call last): ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 447, in run_mp_engine ERROR 06-12 13:12:27 [engine.py:459] engine = MQLLMEngine.from_vllm_config( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 134, in from_vllm_config ERROR 06-12 13:12:27 [engine.py:459] return cls( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/multiprocessing/engine.py", line 88, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.engine = LLMEngine(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/engine/llm_engine.py", line 265, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.model_executor = executor_class(vllm_config=vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/executor_base.py", line 53, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self._init_executor() ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 47, in _init_executor ERROR 06-12 13:12:27 [engine.py:459] self.collective_rpc("init_device") ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc ERROR 06-12 13:12:27 [engine.py:459] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/utils.py", line 2680, in run_method ERROR 06-12 13:12:27 [engine.py:459] return func(*args, **kwargs) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker_base.py", line 606, in init_device ERROR 06-12 13:12:27 [engine.py:459] self.worker.init_device() # type: ignore ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 193, in init_device ERROR 06-12 13:12:27 [engine.py:459] init_worker_distributed_environment(self.vllm_config, self.rank, ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/worker/worker.py", line 537, in init_worker_distributed_environment ERROR 06-12 13:12:27 [engine.py:459] ensure_kv_transfer_initialized(vllm_config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_transfer_state.py", line 67, in ensure_kv_transfer_initialized ERROR 06-12 13:12:27 [engine.py:459] _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v0( ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 50, in create_connector_v0 ERROR 06-12 13:12:27 [engine.py:459] return connector_cls(rank, local_rank, config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 55, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.kv_store = MooncakeStore(config) ERROR 06-12 13:12:27 [engine.py:459] File "/vllm/vllm/distributed/kv_transfer/kv_lookup_buffer/mooncake_store.py", line 86, in __init__ ERROR 06-12 13:12:27 [engine.py:459] self.store.setup(self.config.local_hostname, ERROR 06-12 13:12:27 [engine.py:459] TypeError: setup(): incompatible function arguments. The following argument types are supported: ERROR 06-12 13:12:27 [engine.py:459] 1. (self: mooncake.store.MooncakeDistributedStore, arg0: str, arg1: str, arg2: int, arg3: int, arg4: str, arg5: str, arg6: str) -> int ERROR 06-12 13:12:27 [engine.py:459] ERROR 06-12 13:12:27 [engine.py:459] Invoked with: <mooncake.store.MooncakeDistributedStore object at 0x7af2a2487430>, None, '192.168.1.10:2379', 3355443200, 1073741824, 'rdma', 'erdma_0', None ` my mooncake.json is: ` { "local_hostname": "192.168.1.10" "prefill_url": "192.168.1.10:13003", "decode_url": "192.168.1.10:13103", "metadata_server": "192.168.1.10:2379", "metadata_backend": "etcd", "protocol": "rdma", "device_name": "erdma_0" "master_server_address": "192.168.1.10:50001" } ` Is this my configuration problem?
Author
Owner

@iblisylvie commented on GitHub (Jun 13, 2025):

I also met the same problem.
So is it now to change the metadata_server from "etd://xx.xx.xx.xx:2379" to "http://xx.xx.xx.xx:2379"?
I changed it, but this time it returns

E0613 17:05:31.807329 2384597 transfer_metadata_plugin.cpp:235] Unexpected code in http response, PUT http://10.11.60.9:2379?key=mooncake%2Frpc_meta%2F10.11.60.9%3A13609 response code: 404 response body: 404 page not found
E0613 17:05:31.807374 2384597 transfer_metadata.cpp:453] Failed to set location of 10.11.60.9:13609
F0613 17:05:31.807412 2384597 client.cpp:120] Check failed: rc == 0 (-200 vs. 0) Failed to initialize transfer engine

@alogfans

<!-- gh-comment-id:2969661188 --> @iblisylvie commented on GitHub (Jun 13, 2025): I also met the same problem. So is it now to change the metadata_server from "etd://xx.xx.xx.xx:2379" to "http://xx.xx.xx.xx:2379"? I changed it, but this time it returns E0613 17:05:31.807329 2384597 transfer_metadata_plugin.cpp:235] Unexpected code in http response, PUT http://10.11.60.9:2379?key=mooncake%2Frpc_meta%2F10.11.60.9%3A13609 response code: 404 response body: 404 page not found E0613 17:05:31.807374 2384597 transfer_metadata.cpp:453] Failed to set location of 10.11.60.9:13609 F0613 17:05:31.807412 2384597 client.cpp:120] Check failed: rc == 0 (-200 vs. 0) Failed to initialize transfer engine @alogfans
Author
Owner

@stmatengss commented on GitHub (Jun 13, 2025):

Hi, before that, you should start http server. https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py

<!-- gh-comment-id:2969839660 --> @stmatengss commented on GitHub (Jun 13, 2025): Hi, before that, you should start http server. https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py
Author
Owner

@air6677 commented on GitHub (Jun 13, 2025):

We recently removed the default support of etcd. To use etcd service in transfer engine, in mooncake-common/common.cmake, change the switch of USE_ETCD from OFF to ON.

@alogfans HI,what is the reason for disabling etcd support? If in new version to use etcd, how to enable it quickly? For future production environments, should we prioritize using an HTTP server for the metadata server?

<!-- gh-comment-id:2970104799 --> @air6677 commented on GitHub (Jun 13, 2025): > We recently removed the default support of etcd. To use etcd service in transfer engine, in `mooncake-common/common.cmake`, change the switch of `USE_ETCD` from `OFF` to `ON`. @alogfans HI,what is the reason for disabling etcd support? If in new version to use etcd, how to enable it quickly? For future production environments, should we prioritize using an HTTP server for the metadata server?
Author
Owner

@alogfans commented on GitHub (Jul 1, 2025):

@air6677 The dependenies of etcd support is complex and it's not used by most users. To enable it, builds it manually, with --USE_ETCD=ON. We prefer to use HTTP or P2P-based support.

<!-- gh-comment-id:3022069909 --> @alogfans commented on GitHub (Jul 1, 2025): @air6677 The dependenies of etcd support is complex and it's not used by most users. To enable it, builds it manually, with --USE_ETCD=ON. We prefer to use HTTP or P2P-based support.
Author
Owner

@github-actions[bot] commented on GitHub (Jul 4, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4880607128 --> @github-actions[bot] commented on GitHub (Jul 4, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
Author
Owner

@github-actions[bot] commented on GitHub (Jul 12, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4949899934 --> @github-actions[bot] commented on GitHub (Jul 12, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#167
No description provided.