[GH-ISSUE #375] Unable to find metadata storage plugin etcd with conn string: etcd://X.X.X.X:2379 #130

Closed
opened 2026-10-29 21:09:53 +00:00 by bsakaguchi · 5 comments
Owner

Originally created by @CSEEduanyu on GitHub (May 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/375

ON 10.92.129.174:
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.92.129.174:2379
2025-05-18 13:10:15.455204 I | etcdmain: etcd Version: 3.3.25
2025-05-18 13:10:15.455221 I | etcdmain: Git SHA: Not provided (use ./build instead of go build)
2025-05-18 13:10:15.455227 I | etcdmain: Go Version: go1.18.1
2025-05-18 13:10:15.455229 I | etcdmain: Go OS/Arch: linux/amd64
2025-05-18 13:10:15.455232 I | etcdmain: setting maximum number of CPUs to 96, total number of available CPUs is 96
2025-05-18 13:10:15.455237 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2025-05-18 13:10:15.462762 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2025-05-18 13:10:15.463049 I | embed: listening for peers on http://localhost:2380
2025-05-18 13:10:15.463090 I | embed: listening for client requests on 0.0.0.0:2379
2025-05-18 13:10:15.523978 I | etcdserver: name = default
2025-05-18 13:10:15.523987 I | etcdserver: data dir = default.etcd
2025-05-18 13:10:15.523990 I | etcdserver: member dir = default.etcd/member
2025-05-18 13:10:15.523993 I | etcdserver: heartbeat = 100ms
2025-05-18 13:10:15.523996 I | etcdserver: election = 1000ms
2025-05-18 13:10:15.524000 I | etcdserver: snapshot count = 100000
2025-05-18 13:10:15.524106 I | etcdserver: advertise client URLs = http://10.92.129.174:2379
2025-05-18 13:10:15.535853 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 12
2025-05-18 13:10:15.535887 I | raft: 8e9e05c52164694d became follower at term 6
2025-05-18 13:10:15.535902 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 6, commit: 12, applied: 0, lastindex: 12, lastterm: 6]
2025-05-18 13:10:15.545054 W | auth: simple token is not cryptographically signed
2025-05-18 13:10:15.549244 I | etcdserver: starting server... [version: 3.3.25, cluster version: to_be_decided]
2025-05-18 13:10:15.550397 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2025-05-18 13:10:15.550514 N | etcdserver/membership: set the initial cluster version to 3.3
2025-05-18 13:10:15.550543 I | etcdserver/api: enabled capabilities for version 3.3
2025-05-18 13:10:17.043801 I | raft: 8e9e05c52164694d is starting a new election at term 6
2025-05-18 13:10:17.043871 I | raft: 8e9e05c52164694d became candidate at term 7
2025-05-18 13:10:17.043891 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 7
2025-05-18 13:10:17.043902 I | raft: 8e9e05c52164694d became leader at term 7
2025-05-18 13:10:17.043908 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 7
2025-05-18 13:10:17.044122 I | embed: ready to serve client requests
2025-05-18 13:10:17.044146 I | etcdserver: published {Name:default ClientURLs:[http://10.92.129.174:2379]} to cluster cdf818194e3a8c32
2025-05-18 13:10:17.044615 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!

ON 10.92.133.70:
./transfer_engine_bench --mode=target
--metadata_server=10.92.129.174:2379
--local_server_name=10.92.133.70:12345
--auto_discovery
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0518 13:39:45.080192 119241 transfer_engine.cpp:345] Metrics reporting thread started (interval: 5s)
I0518 13:39:45.080736 119240 transfer_engine.cpp:91] Transfer Engine uses address 10.92.133.70 and port 15543 for serving local TCP service
F0518 13:39:45.080753 119240 transfer_metadata_plugin.cpp:442] Unable to find metadata storage plugin etcd with conn string: 10.92.129.174:2379
*** Check failure stack trace: ***
Aborted (core dumped)

Originally created by @CSEEduanyu on GitHub (May 18, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/375 ON 10.92.129.174: etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.92.129.174:2379 2025-05-18 13:10:15.455204 I | etcdmain: etcd Version: 3.3.25 2025-05-18 13:10:15.455221 I | etcdmain: Git SHA: Not provided (use ./build instead of go build) 2025-05-18 13:10:15.455227 I | etcdmain: Go Version: go1.18.1 2025-05-18 13:10:15.455229 I | etcdmain: Go OS/Arch: linux/amd64 2025-05-18 13:10:15.455232 I | etcdmain: setting maximum number of CPUs to 96, total number of available CPUs is 96 2025-05-18 13:10:15.455237 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd 2025-05-18 13:10:15.462762 N | etcdmain: the server is already initialized as member before, starting as etcd member... 2025-05-18 13:10:15.463049 I | embed: listening for peers on http://localhost:2380 2025-05-18 13:10:15.463090 I | embed: listening for client requests on 0.0.0.0:2379 2025-05-18 13:10:15.523978 I | etcdserver: name = default 2025-05-18 13:10:15.523987 I | etcdserver: data dir = default.etcd 2025-05-18 13:10:15.523990 I | etcdserver: member dir = default.etcd/member 2025-05-18 13:10:15.523993 I | etcdserver: heartbeat = 100ms 2025-05-18 13:10:15.523996 I | etcdserver: election = 1000ms 2025-05-18 13:10:15.524000 I | etcdserver: snapshot count = 100000 2025-05-18 13:10:15.524106 I | etcdserver: advertise client URLs = http://10.92.129.174:2379 2025-05-18 13:10:15.535853 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 12 2025-05-18 13:10:15.535887 I | raft: 8e9e05c52164694d became follower at term 6 2025-05-18 13:10:15.535902 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 6, commit: 12, applied: 0, lastindex: 12, lastterm: 6] 2025-05-18 13:10:15.545054 W | auth: simple token is not cryptographically signed 2025-05-18 13:10:15.549244 I | etcdserver: starting server... [version: 3.3.25, cluster version: to_be_decided] 2025-05-18 13:10:15.550397 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 2025-05-18 13:10:15.550514 N | etcdserver/membership: set the initial cluster version to 3.3 2025-05-18 13:10:15.550543 I | etcdserver/api: enabled capabilities for version 3.3 2025-05-18 13:10:17.043801 I | raft: 8e9e05c52164694d is starting a new election at term 6 2025-05-18 13:10:17.043871 I | raft: 8e9e05c52164694d became candidate at term 7 2025-05-18 13:10:17.043891 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 7 2025-05-18 13:10:17.043902 I | raft: 8e9e05c52164694d became leader at term 7 2025-05-18 13:10:17.043908 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 7 2025-05-18 13:10:17.044122 I | embed: ready to serve client requests 2025-05-18 13:10:17.044146 I | etcdserver: published {Name:default ClientURLs:[http://10.92.129.174:2379]} to cluster cdf818194e3a8c32 2025-05-18 13:10:17.044615 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged! ON 10.92.133.70: ./transfer_engine_bench --mode=target \ --metadata_server=10.92.129.174:2379 \ --local_server_name=10.92.133.70:12345 \ --auto_discovery WARNING: Logging before InitGoogleLogging() is written to STDERR I0518 13:39:45.080192 119241 transfer_engine.cpp:345] Metrics reporting thread started (interval: 5s) I0518 13:39:45.080736 119240 transfer_engine.cpp:91] Transfer Engine uses address 10.92.133.70 and port 15543 for serving local TCP service F0518 13:39:45.080753 119240 transfer_metadata_plugin.cpp:442] Unable to find metadata storage plugin etcd with conn string: 10.92.129.174:2379 *** Check failure stack trace: *** Aborted (core dumped)
Author
Owner

@CSEEduanyu commented on GitHub (May 18, 2025):

ON 10.92.133.70:
curl -L http://10.92.129.174:2379/version
{"etcdserver":"3.3.25","etcdcluster":"3.3.0"}

ETCDCTL_API=3 etcdctl --endpoints=http://10.92.129.174:2379 endpoint status
http://10.92.129.174:2379, 8e9e05c52164694d, 3.3.25, 20 kB, true, 7, 14

<!-- gh-comment-id:2888998002 --> @CSEEduanyu commented on GitHub (May 18, 2025): ON 10.92.133.70: curl -L http://10.92.129.174:2379/version {"etcdserver":"3.3.25","etcdcluster":"3.3.0"} ETCDCTL_API=3 etcdctl --endpoints=http://10.92.129.174:2379 endpoint status http://10.92.129.174:2379, 8e9e05c52164694d, 3.3.25, 20 kB, true, 7, 14
Author
Owner

@CSEEduanyu commented on GitHub (May 18, 2025):

https://github.com/kvcache-ai/Mooncake/blob/main/doc/zh/run-examples.md
I followed these instructions for testing, but I couldn't connect to the metadata server @ShangmingCai

<!-- gh-comment-id:2888999440 --> @CSEEduanyu commented on GitHub (May 18, 2025): https://github.com/kvcache-ai/Mooncake/blob/main/doc/zh/run-examples.md I followed these instructions for testing, but I couldn't connect to the metadata server @ShangmingCai
Author
Owner

@CSEEduanyu commented on GitHub (May 18, 2025):

issue cloesd,The reason is that the documentation does not indicate the need for camek -DUSE_ETCD=ON

<!-- gh-comment-id:2889010089 --> @CSEEduanyu commented on GitHub (May 18, 2025): issue cloesd,The reason is that the documentation does not indicate the need for camek -DUSE_ETCD=ON
Author
Owner

@MountainGG commented on GitHub (May 20, 2025):

I directly used pip install, but also came up with this error.

<!-- gh-comment-id:2894007033 --> @MountainGG commented on GitHub (May 20, 2025): I directly used pip install, but also came up with this error.
Author
Owner

@CSEEduanyu commented on GitHub (May 20, 2025):

I directly used pip install, but also came up with this error.

maybe you need to compile from source and use -DUSE_ETCD=ON

<!-- gh-comment-id:2894050999 --> @CSEEduanyu commented on GitHub (May 20, 2025): > I directly used pip install, but also came up with this error. maybe you need to compile from source and use -DUSE_ETCD=ON
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#130
No description provided.