[GH-ISSUE #2185] [Usage]: Could the client reconnect to new master when it restarts? #599

New issue

Closed

opened 2026-10-29 21:17:01 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:17:01 +00:00

Owner

Originally created by @ckeyever on GitHub (May 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2185

Describe your usage question

The IP of master is set in mooncake.json(master_server_address). In my scenario, the IP of master changes when it restarts, and I don't want to restart the inference services. How could the client connector the new master with the new IP?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @ckeyever on GitHub (May 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2185 ### Describe your usage question The IP of master is set in mooncake.json(master_server_address). In my scenario, the IP of master changes when it restarts, and I don't want to restart the inference services. How could the client connector the new master with the new IP? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:17:01 +00:00

bsakaguchi commented

2026-10-29 21:17:01 +00:00

Author

Owner

@leonzzhu commented on GitHub (May 24, 2026):

@ckeyever HI

For your scenario, here are two approaches you can consider:

1. Service discovery via etcd (single master)

You can deploy an etcd cluster and have the master register its current address to a well-known etcd key (e.g., /mooncake/master/address) on startup. Clients watch this key and automatically reconnect when the master IP changes. This way, even with a single master, clients don't need to be restarted after master IP changes.

2. HA mode with leader + standby (recommended)

Deploy two (or more) master instances with etcd-based leader election. One acts as the active leader, the other stays in standby mode. When the leader goes down, the standby automatically promotes itself to leader and registers its address in etcd. Clients discover the new leader via etcd without any manual intervention or restart.

I've been working on the HA feature recently and verified that leader failover + snapshot restore works well in production. If you're interested in approach #2, feel free to check out the HA-related configuration in mooncake.json (ha_enabled, etcd_endpoints, etc.).

Hope this helps!

@leonzzhu commented on GitHub (May 24, 2026): @ckeyever HI For your scenario, here are two approaches you can consider: **1. Service discovery via etcd (single master)** You can deploy an etcd cluster and have the master register its current address to a well-known etcd key (e.g., `/mooncake/master/address`) on startup. Clients watch this key and automatically reconnect when the master IP changes. This way, even with a single master, clients don't need to be restarted after master IP changes. **2. HA mode with leader + standby (recommended)** Deploy two (or more) master instances with etcd-based leader election. One acts as the active leader, the other stays in standby mode. When the leader goes down, the standby automatically promotes itself to leader and registers its address in etcd. Clients discover the new leader via etcd without any manual intervention or restart. I've been working on the HA feature recently and verified that leader failover + snapshot restore works well in production. If you're interested in approach #2, feel free to check out the HA-related configuration in `mooncake.json` (`ha_enabled`, `etcd_endpoints`, etc.). Hope this helps!

bsakaguchi referenced this issue

2026-10-29 21:29:26 +00:00

[PR #599] [CLOSED] [TransferEngine] Detect unconnected remote endpoints and avoid using them #1679