[GH-ISSUE #55] Start the initiator node crashed #35

New issue

Closed

opened 2026-10-29 21:08:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:08:14 +00:00

Owner

Originally created by @nctu6 on GitHub (Dec 30, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/55

Dears,
I'm not sure why there are issues:

Invalid argument when running the target
Assert failed: port

Below are the commands and logs.
Regards.

Start the metadata service.

etcd --listen-client-urls http://0.0.0.0:8910  --advertise-client-urls http://0.0.0.0:8911

{"level":"info","ts":"2024-12-30T06:17:46.653336Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c5
2164694d became leader at term 2"}                                                                                     
{"level":"info","ts":"2024-12-30T06:17:46.653347Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.nod
e: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"}                                                        
{"level":"info","ts":"2024-12-30T06:17:46.656601Z","caller":"etcdserver/server.go:2651","msg":"setting up initial clust
er version using v2 API","cluster-version":"3.5"}                                                                      
{"level":"info","ts":"2024-12-30T06:17:46.658377Z","caller":"etcdserver/server.go:2140","msg":"published local member t
o cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http:
//0.0.0.0:8911]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-time
out":"7s"}                                                                                                             
{"level":"info","ts":"2024-12-30T06:17:46.658480Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"
}                                                                                                                      
{"level":"info","ts":"2024-12-30T06:17:46.658822Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}       
{"level":"info","ts":"2024-12-30T06:17:46.659054Z","caller":"etcdmain/main.go:50","msg":"successfully notified init dae
mon"}                                                                                                                  
{"level":"info","ts":"2024-12-30T06:17:46.660167Z","caller":"membership/cluster.go:584","msg":"set initial cluster vers
ion","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}                     
{"level":"info","ts":"2024-12-30T06:17:46.660235Z","caller":"api/capability.go:75","msg":"enabled capabilities for vers
ion","cluster-version":"3.5"}                                                                                          
{"level":"info","ts":"2024-12-30T06:17:46.660261Z","caller":"etcdserver/server.go:2675","msg":"cluster version is updat
ed","cluster-version":"3.5"}                                                                                           
{"level":"info","ts":"2024-12-30T06:17:46.660601Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","s
ervice":"","status":"SERVING"}                                                                                         
{"level":"info","ts":"2024-12-30T06:17:46.661780Z","caller":"embed/serve.go:187","msg":"serving client traffic insecure
ly; this is strongly discouraged!","traffic":"grpc+http","address":"[::]:8910"}

Start the target node.

./transfer_engine_bench --mode=target \
                        --metadata_server=127.0.0.1:8911 \
                        --local_server_name=127.0.0.1:8912 \
                        --protocol=tcp

WARNING: Logging before InitGoogleLogging() is written to STDERR                                                       
E1230 16:39:13.654633 76510 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa
ge: connections to all backends failing                                                                                
E1230 16:39:13.654750 76510 transfer_metadata.cpp:675] Failed to insert rpc meta of 127.0.0.1:8912                     
E1230 16:39:13.655081 76510 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa
ge: connections to all backends failing                                                                                
E1230 16:39:13.655104 76510 transfer_metadata.cpp:427] Failed to put segment description: 127.0.0.1:8912               
E1230 16:39:13.655109 76510 tcp_transport.cpp:218] *** Transfer engine cannot be initialized: cannot publish segments  
E1230 16:39:13.655115 76510 tcp_transport.cpp:220] *** Check the connectivity between this server and metadata server (
etcd server)                                                                                                           
E1230 16:39:13.655438 76510 transfer_metadata.cpp:313] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa
ge: connections to all backends failing                                                                                
E1230 16:39:13.655460 76510 transfer_metadata.cpp:436] Failed to remove segment description: 127.0.0.1:8912            
mbind: Invalid argument

Start the initiator node.

./transfer_engine_bench --metadata_server=127.0.0.1:8911 \
                        --segment_id=127.0.0.1:8912 \
                        --local_server_name=127.0.0.1:8913 \
                        --protocol=tcp

WARNING: Logging before InitGoogleLogging() is written to STDERR                                                      
E1230 16:40:16.850381 76515 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess
age: connections to all backends failing                                                                              
E1230 16:40:16.850457 76515 transfer_metadata.cpp:675] Failed to insert rpc meta of 127.0.0.1:8913                    
E1230 16:40:16.850755 76515 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess
age: connections to all backends failing                                                                              
E1230 16:40:16.850777 76515 transfer_metadata.cpp:427] Failed to put segment description: 127.0.0.1:8913              
E1230 16:40:16.850781 76515 tcp_transport.cpp:218] *** Transfer engine cannot be initialized: cannot publish segments 
E1230 16:40:16.850787 76515 tcp_transport.cpp:220] *** Check the connectivity between this server and metadata server 
(etcd server)                                                                                                         
E1230 16:40:16.851032 76515 transfer_metadata.cpp:313] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess
age: connections to all backends failing                                                                              
E1230 16:40:16.851044 76515 transfer_metadata.cpp:436] Failed to remove segment description: 127.0.0.1:8913           
F1230 16:40:16.851049 76515 transfer_engine_bench.cpp:248] Assert failed: xport                                       
*** Check failure stack trace: ***                                                                                    
Aborted (core dumped)

Originally created by @nctu6 on GitHub (Dec 30, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/55 Dears, I'm not sure why there are issues: * Invalid argument when running the target * Assert failed: port Below are the commands and logs. Regards. 1. Start the metadata service. ``` etcd --listen-client-urls http://0.0.0.0:8910 --advertise-client-urls http://0.0.0.0:8911 ``` ``` {"level":"info","ts":"2024-12-30T06:17:46.653336Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c5 2164694d became leader at term 2"} {"level":"info","ts":"2024-12-30T06:17:46.653347Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.nod e: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2"} {"level":"info","ts":"2024-12-30T06:17:46.656601Z","caller":"etcdserver/server.go:2651","msg":"setting up initial clust er version using v2 API","cluster-version":"3.5"} {"level":"info","ts":"2024-12-30T06:17:46.658377Z","caller":"etcdserver/server.go:2140","msg":"published local member t o cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http: //0.0.0.0:8911]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-time out":"7s"} {"level":"info","ts":"2024-12-30T06:17:46.658480Z","caller":"embed/serve.go:103","msg":"ready to serve client requests" } {"level":"info","ts":"2024-12-30T06:17:46.658822Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"} {"level":"info","ts":"2024-12-30T06:17:46.659054Z","caller":"etcdmain/main.go:50","msg":"successfully notified init dae mon"} {"level":"info","ts":"2024-12-30T06:17:46.660167Z","caller":"membership/cluster.go:584","msg":"set initial cluster vers ion","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"} {"level":"info","ts":"2024-12-30T06:17:46.660235Z","caller":"api/capability.go:75","msg":"enabled capabilities for vers ion","cluster-version":"3.5"} {"level":"info","ts":"2024-12-30T06:17:46.660261Z","caller":"etcdserver/server.go:2675","msg":"cluster version is updat ed","cluster-version":"3.5"} {"level":"info","ts":"2024-12-30T06:17:46.660601Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","s ervice":"","status":"SERVING"} {"level":"info","ts":"2024-12-30T06:17:46.661780Z","caller":"embed/serve.go:187","msg":"serving client traffic insecure ly; this is strongly discouraged!","traffic":"grpc+http","address":"[::]:8910"} ``` 2. Start the target node. ``` ./transfer_engine_bench --mode=target \ --metadata_server=127.0.0.1:8911 \ --local_server_name=127.0.0.1:8912 \ --protocol=tcp ``` ``` WARNING: Logging before InitGoogleLogging() is written to STDERR E1230 16:39:13.654633 76510 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa ge: connections to all backends failing E1230 16:39:13.654750 76510 transfer_metadata.cpp:675] Failed to insert rpc meta of 127.0.0.1:8912 E1230 16:39:13.655081 76510 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa ge: connections to all backends failing E1230 16:39:13.655104 76510 transfer_metadata.cpp:427] Failed to put segment description: 127.0.0.1:8912 E1230 16:39:13.655109 76510 tcp_transport.cpp:218] *** Transfer engine cannot be initialized: cannot publish segments E1230 16:39:13.655115 76510 tcp_transport.cpp:220] *** Check the connectivity between this server and metadata server ( etcd server) E1230 16:39:13.655438 76510 transfer_metadata.cpp:313] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 messa ge: connections to all backends failing E1230 16:39:13.655460 76510 transfer_metadata.cpp:436] Failed to remove segment description: 127.0.0.1:8912 mbind: Invalid argument ``` 3. Start the initiator node. ``` ./transfer_engine_bench --metadata_server=127.0.0.1:8911 \ --segment_id=127.0.0.1:8912 \ --local_server_name=127.0.0.1:8913 \ --protocol=tcp ``` ``` WARNING: Logging before InitGoogleLogging() is written to STDERR E1230 16:40:16.850381 76515 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess age: connections to all backends failing E1230 16:40:16.850457 76515 transfer_metadata.cpp:675] Failed to insert rpc meta of 127.0.0.1:8913 E1230 16:40:16.850755 76515 transfer_metadata.cpp:302] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess age: connections to all backends failing E1230 16:40:16.850777 76515 transfer_metadata.cpp:427] Failed to put segment description: 127.0.0.1:8913 E1230 16:40:16.850781 76515 tcp_transport.cpp:218] *** Transfer engine cannot be initialized: cannot publish segments E1230 16:40:16.850787 76515 tcp_transport.cpp:220] *** Check the connectivity between this server and metadata server (etcd server) E1230 16:40:16.851032 76515 transfer_metadata.cpp:313] Error from etcd client, etcd uri: 127.0.0.1:8911 error: 14 mess age: connections to all backends failing E1230 16:40:16.851044 76515 transfer_metadata.cpp:436] Failed to remove segment description: 127.0.0.1:8913 F1230 16:40:16.851049 76515 transfer_engine_bench.cpp:248] Assert failed: xport *** Check failure stack trace: *** Aborted (core dumped) ```

bsakaguchi closed this issue

2026-10-29 21:08:14 +00:00