[GH-ISSUE #16] Error in running transfer engine sample #14

New issue

Closed

opened 2026-10-29 21:07:53 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:07:53 +00:00

Owner

Originally created by @Little-rookie-ee on GitHub (Dec 4, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/16

After building the repo，I tried to run the transfer engine example. I ran etcd sample in target node and initiator node without error. But ,when I ran in target node，an error happened like below.

./transfer_engine_bench --mode=target
--metadata_server=10.0.0.1:2379
--local_server_name=10.0.0.2:12345
--device_name=mlx5_0
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1204 10:45:22.126729 108700 transfer_metadata.cpp:74] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing
E1204 10:45:22.126777 108700 transfer_metadata.cpp:426] Failed to insert rpc meta of 10.0.0.2:12345
I1204 10:45:22.131613 108700 rdma_context.cpp:131] RDMA device: mlx5_0, LID: 28, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:9e:d2:3e
E1204 10:45:22.131757 108700 transfer_metadata.cpp:74] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing
E1204 10:45:22.131764 108700 transfer_metadata.cpp:178] Failed to put segment description: 10.0.0.2:12345
E1204 10:45:22.131772 108700 rdma_transport.cpp:92] Transfer engine cannot be initialized: cannot publish segments. Check the connectivity between this server and etcd servers
E1204 10:45:22.131799 108700 transfer_metadata.cpp:85] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing
E1204 10:45:22.131804 108700 transfer_metadata.cpp:187] Failed to remove segment description: 10.0.0.2:12345

It seems that it was not initialized correctly? Also, when I try to use TCP for communication way, a similar error happened. Is it because my environment is not properly configured？

device: 2 A100 * 80G

Originally created by @Little-rookie-ee on GitHub (Dec 4, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/16 After building the repo，I tried to run the transfer engine example. I ran etcd sample in target node and initiator node without error. But ,when I ran in target node，an error happened like below. ./transfer_engine_bench --mode=target \ --metadata_server=10.0.0.1:2379 \ --local_server_name=10.0.0.2:12345 \ --device_name=mlx5_0 WARNING: Logging before InitGoogleLogging() is written to STDERR E1204 10:45:22.126729 108700 transfer_metadata.cpp:74] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing E1204 10:45:22.126777 108700 transfer_metadata.cpp:426] Failed to insert rpc meta of 10.0.0.2:12345 I1204 10:45:22.131613 108700 rdma_context.cpp:131] RDMA device: mlx5_0, LID: 28, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:9e:d2:3e E1204 10:45:22.131757 108700 transfer_metadata.cpp:74] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing E1204 10:45:22.131764 108700 transfer_metadata.cpp:178] Failed to put segment description: 10.0.0.2:12345 E1204 10:45:22.131772 108700 rdma_transport.cpp:92] Transfer engine cannot be initialized: cannot publish segments. Check the connectivity between this server and etcd servers E1204 10:45:22.131799 108700 transfer_metadata.cpp:85] Error from etcd client, etcd uri: 10.0.0.1:2379 error: 14 message: connections to all backends failing E1204 10:45:22.131804 108700 transfer_metadata.cpp:187] Failed to remove segment description: 10.0.0.2:12345 It seems that it was not initialized correctly? Also, when I try to use TCP for communication way, a similar error happened. Is it because my environment is not properly configured？ device: 2 A100 * 80G

bsakaguchi closed this issue

2026-10-29 21:07:53 +00:00

bsakaguchi commented

2026-10-29 21:07:54 +00:00

Author

Owner

@Little-rookie-ee commented on GitHub (Dec 4, 2024):

I solved this problem. I changed the metadata_server and local_server_name to the node`s IP. And I started etcd server in a terminal and started target node command in another one. Then it can work.

@Little-rookie-ee commented on GitHub (Dec 4, 2024): I solved this problem. I changed the metadata_server and local_server_name to the node`s IP. And I started etcd server in a terminal and started target node command in another one. Then it can work.

bsakaguchi commented

2026-10-29 21:07:54 +00:00

Author

Owner

@LLJING-ER commented on GitHub (May 23, 2025):

I solved this problem. I changed the metadata_server and local_server_name to the node`s IP. And I started etcd server in a terminal and started target node command in another one. Then it can work.

can you share your etcd version, command and log of start metadata service for me? thanks for your help

@LLJING-ER commented on GitHub (May 23, 2025): > I solved this problem. I changed the metadata_server and local_server_name to the node`s IP. And I started etcd server in a terminal and started target node command in another one. Then it can work. can you share your **etcd version, command and log of start metadata service** for me? thanks for your help

bsakaguchi referenced this issue

2026-10-29 21:17:35 +00:00

[GH-ISSUE #2440] [Bug]: Execute mooncake+hixl, final destruction phase core #646