kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2798] [Bug]:mooncake_master startup failed with "bad address: 0.0.0.0 error: Host not found (authoritative)" #728

New issue

Closed

opened 2026-10-29 21:18:46 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:18:46 +00:00

Owner

Originally created by @enkco on GitHub (Jul 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2798

Bug Report

I installed Mooncake v0.3.9 following the steps in the documentation:

Then I just run

mooncake_master

with an error occurred:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260708 17:18:15.555217 23659 master.cpp:515] Master service started on port 50051, max_threads=4, enable_metric_reporting=1, metrics_port=9003, default_kv_lease_ttl=5000, default_kv_soft_pin_ttl=1800000, allow_evict_soft_pinned_objects=1, eviction_ratio=0.05, eviction_high_watermark_ratio=0.95, enable_ha=0, enable_offload=0, etcd_endpoints=, client_ttl=10, rpc_thread_num=4, rpc_port=50051, rpc_address=0.0.0.0, rpc_conn_timeout_seconds=0, rpc_enable_tcp_no_delay=1, rpc protocol=tcp, cluster_id=mooncake_cluster, root_fs_dir=, global_file_segment_size=9223372036854775807, memory_allocator=offset, enable_http_metadata_server=0, http_metadata_server_port=8080, http_metadata_server_host=0.0.0.0, put_start_discard_timeout_sec=30, put_start_release_timeout_sec=600, max_total_finished_tasks=10000, max_total_pending_tasks=10000, max_total_processing_tasks=10000, pending_task_timeout_sec=300, processing_task_timeout_sec=300, enable_cxl=0, cxl_path=/dev/dax0.0, cxl_size=8589934592
I20260708 17:18:15.561800 23666 master_service.cpp:218] Task cleanup thread started
I20260708 17:18:15.561969 23659 rpc_service.cpp:251] HTTP metrics server started on port 9003
2026-07-08 17:18:15.561937 ERROR    [23659] [coro_http_server.hpp:725] bad address: 0.0.0.0 error: Host not found (authoritative)
2026-07-08 17:18:15.562157 ERROR    [23659] [coro_rpc_server.hpp:394] resolve address 0.0.0.0 error: Host not found (authoritative)
I20260708 17:18:15.562461 23667 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
I20260708 17:18:25.562783 23666 master_service.cpp:236] Task cleanup thread stopped

and the mooncake_master service disappeared.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @enkco on GitHub (Jul 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2798 ### Bug Report I installed Mooncake v0.3.9 following the steps in the documentation: <img width="886" height="796" alt="Image" src="https://github.com/user-attachments/assets/31b9270d-bfe2-41af-9846-5d375c5e89a0" /> Then I just run ``` mooncake_master ``` with an error occurred: ``` WARNING: Logging before InitGoogleLogging() is written to STDERR I20260708 17:18:15.555217 23659 master.cpp:515] Master service started on port 50051, max_threads=4, enable_metric_reporting=1, metrics_port=9003, default_kv_lease_ttl=5000, default_kv_soft_pin_ttl=1800000, allow_evict_soft_pinned_objects=1, eviction_ratio=0.05, eviction_high_watermark_ratio=0.95, enable_ha=0, enable_offload=0, etcd_endpoints=, client_ttl=10, rpc_thread_num=4, rpc_port=50051, rpc_address=0.0.0.0, rpc_conn_timeout_seconds=0, rpc_enable_tcp_no_delay=1, rpc protocol=tcp, cluster_id=mooncake_cluster, root_fs_dir=, global_file_segment_size=9223372036854775807, memory_allocator=offset, enable_http_metadata_server=0, http_metadata_server_port=8080, http_metadata_server_host=0.0.0.0, put_start_discard_timeout_sec=30, put_start_release_timeout_sec=600, max_total_finished_tasks=10000, max_total_pending_tasks=10000, max_total_processing_tasks=10000, pending_task_timeout_sec=300, processing_task_timeout_sec=300, enable_cxl=0, cxl_path=/dev/dax0.0, cxl_size=8589934592 I20260708 17:18:15.561800 23666 master_service.cpp:218] Task cleanup thread started I20260708 17:18:15.561969 23659 rpc_service.cpp:251] HTTP metrics server started on port 9003 2026-07-08 17:18:15.561937 ERROR [23659] [coro_http_server.hpp:725] bad address: 0.0.0.0 error: Host not found (authoritative) 2026-07-08 17:18:15.562157 ERROR [23659] [coro_rpc_server.hpp:394] resolve address 0.0.0.0 error: Host not found (authoritative) I20260708 17:18:15.562461 23667 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B I20260708 17:18:25.562783 23666 master_service.cpp:236] Task cleanup thread stopped ``` and the mooncake_master service disappeared. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:46 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:47 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 8, 2026):

Thanks for opening this issue, @enkco!

Field	Value
Issue	#2798
GitHub user ID	`48650046`
Reporter	@enkco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 8, 2026): Thanks for opening this issue, @enkco! | Field | Value | |-------|-------| | **Issue** | #2798 | | **GitHub user ID** | `48650046` | | **Reporter** | @enkco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:47 +00:00

Author

Owner

@Lin-z-w commented on GitHub (Jul 9, 2026):

Thanks for the report. I looked into the startup failure. The direct cause appears to be in the yalantinglibs version currently used by Mooncake's RPC/HTTP servers.

0.0.0.0 is a valid server bind address; it means "listen on all IPv4 interfaces". However, the yalantinglibs version pinned by Mooncake currently resolves the listen address through Asio/system resolver before binding. In some environments, resolving 0.0.0.0 can return Host not found, which matches the errors in your log:

bad address: 0.0.0.0 error: Host not found (authoritative)
resolve address 0.0.0.0 error: Host not found (authoritative)

So this is not caused by Mooncake master service logic itself, but Mooncake exposes it because mooncake_master uses 0.0.0.0 as the default bind address. The proper fix is for yalantinglibs to handle numeric addresses such as 0.0.0.0, 127.0.0.1, and :: by directly constructing a TCP endpoint, and only use resolver lookup for hostnames like localhost.

I checked upstream yalantinglibs. Their current main branch already contains this fix via coro_io/detail/listen_endpoint.hpp, but the latest yalantinglibs release/tag (0.6.1) does not contain it yet. Mooncake is still pinned to an older yalantinglibs commit, so the released Mooncake binaries can still hit this issue in affected environments.

We plan to pick up the fixed yalantinglibs code in a future dependency upgrade. Until then, possible workarounds are:

If you only need local testing, bind the RPC service to loopback, for example --rpc_address=127.0.0.1.
If the metrics/admin HTTP server also fails in the same environment, disable metric reporting if it is not needed.
For multi-node or HA deployments, do not use 0.0.0.0 as the advertised/client address. Configure a real reachable IP address or use --rpc_interface=<network-interface> so Mooncake can derive the correct IP.

For clients, 0.0.0.0 should not be used as the connection address. Use 127.0.0.1:50051 for single-machine testing, or <master-reachable-ip>:50051 for multi-node deployments.

@Lin-z-w commented on GitHub (Jul 9, 2026): Thanks for the report. I looked into the startup failure. The direct cause appears to be in the `yalantinglibs` version currently used by Mooncake's RPC/HTTP servers. `0.0.0.0` is a valid server bind address; it means "listen on all IPv4 interfaces". However, the yalantinglibs version pinned by Mooncake currently resolves the listen address through Asio/system resolver before binding. In some environments, resolving `0.0.0.0` can return `Host not found`, which matches the errors in your log: ```text bad address: 0.0.0.0 error: Host not found (authoritative) resolve address 0.0.0.0 error: Host not found (authoritative) ``` So this is not caused by Mooncake master service logic itself, but Mooncake exposes it because `mooncake_master` uses `0.0.0.0` as the default bind address. The proper fix is for yalantinglibs to handle numeric addresses such as `0.0.0.0`, `127.0.0.1`, and `::` by directly constructing a TCP endpoint, and only use resolver lookup for hostnames like `localhost`. I checked upstream yalantinglibs. Their current `main` branch already contains this fix via `coro_io/detail/listen_endpoint.hpp`, but the latest yalantinglibs release/tag (`0.6.1`) does not contain it yet. Mooncake is still pinned to an older yalantinglibs commit, so the released Mooncake binaries can still hit this issue in affected environments. We plan to pick up the fixed yalantinglibs code in a future dependency upgrade. Until then, possible workarounds are: - If you only need local testing, bind the RPC service to loopback, for example `--rpc_address=127.0.0.1`. - If the metrics/admin HTTP server also fails in the same environment, disable metric reporting if it is not needed. - For multi-node or HA deployments, do not use `0.0.0.0` as the advertised/client address. Configure a real reachable IP address or use `--rpc_interface=<network-interface>` so Mooncake can derive the correct IP. For clients, `0.0.0.0` should not be used as the connection address. Use `127.0.0.1:50051` for single-machine testing, or `<master-reachable-ip>:50051` for multi-node deployments.

bsakaguchi commented

2026-10-29 21:18:47 +00:00

Author

Owner

@enkco commented on GitHub (Jul 9, 2026):

Thanks for your reply! So is it an error limited to environments where 0.0.0.0 cannot be resolved successfully using Asio?
In addition, when I instead run mooncake_master --rpc_address $MY_HOST_IPV6, the mooncake_master service also reports a host not found error:

I20260709 21:40:35.397544 12449 rpc_service.cpp:251] HTTP metrics server started on port 9003
2026-07-09 21:40:35.397501 ERROR    [12449] [coro_http_server.hpp:725] bad address: 0.0.0.0 error: Host not found (authoritative)
I20260709 21:40:35.397734 12458 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0),  | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
2026-07-09 21:40:35.397797 ERROR    [12449] [coro_rpc_server.hpp:394] resolve address 2605:340:cd51:4900:14f:ea12:cb9e error: Host not found (authoritative)
I20260709 21:40:45.398020 12457 master_service.cpp:236] Task cleanup thread stopped

whereas the actual IP address is 2605:340:cd51:4900:14f:ea12:cb9e:820.

@enkco commented on GitHub (Jul 9, 2026): Thanks for your reply! So is it an error limited to environments where `0.0.0.0` cannot be resolved successfully using Asio? In addition, when I instead run `mooncake_master --rpc_address $MY_HOST_IPV6`, the mooncake_master service also reports a `host not found` error: ``` I20260709 21:40:35.397544 12449 rpc_service.cpp:251] HTTP metrics server started on port 9003 2026-07-09 21:40:35.397501 ERROR [12449] [coro_http_server.hpp:725] bad address: 0.0.0.0 error: Host not found (authoritative) I20260709 21:40:35.397734 12458 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B 2026-07-09 21:40:35.397797 ERROR [12449] [coro_rpc_server.hpp:394] resolve address 2605:340:cd51:4900:14f:ea12:cb9e error: Host not found (authoritative) I20260709 21:40:45.398020 12457 master_service.cpp:236] Task cleanup thread stopped ``` whereas the actual IP address is `2605:340:cd51:4900:14f:ea12:cb9e:820`.

bsakaguchi referenced this issue

2026-10-29 21:30:15 +00:00

[PR #728] [CLOSED] add ascend direct transport #1777