mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #905] [Bug]: failed use enable-ha with etcd #287
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#287
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jieyang-su on GitHub (Sep 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/905
Bug Report
I'm encountering the same issue: After enabling High Availability (HA) mode in mooncake_master, the client service automatically deregisters from mooncake_master after running for a period of time, and the corresponding records in mooncake_master get released automatically.
Besides, another problem occurs when I change the master_server_address in MooncakeDistributedStore from the original 192.168.2.38:xxxx to etcd://192.168.2.38:xxxx, accompanied by the following error messages:
Before submitting...
@jieyang-su commented on GitHub (Sep 30, 2025):
I used the code of mooncake version 0.3.5 and compiled the version myself,
cmake .. -DUSE_ETCD=ON -DUSE_ETCD_LEGACY=ON -DSTORE_USE_ETCD=ON
@jieyang-su commented on GitHub (Sep 30, 2025):
I see when i use --enable-ha=true，Clients always equal 0
when i use --enable-ha=false，the metric have not the Clients。
@stmatengss commented on GitHub (Sep 30, 2025):
Could you update to 0.3.6.post1? It is recommended to use pip install as the first choice.
@ykwd commented on GitHub (Sep 30, 2025):
In older versions of Mooncake, if HA was enabled on the master but not on the client (i.e., the
master_server_addressin the client was not an etcd-prefixed address), the client could connect to the master but would not send heartbeat requests. As a result, after 30 seconds of timeout, the master would mark the client as disconnected.Solution 1: Enable HA on both the master and the client, or disable HA on both sides.
Solution 2: Update to the latest version of Mooncake.
@ykwd commented on GitHub (Sep 30, 2025):
This error seems to be caused by the client not connecting to etcd. You may want to check whether master_server_address is set correctly.
If you only want to use the HA feature of the Mooncake store, only need to set STORE_USE_ETCD to ON; the other two options can be set to OFF.
@jieyang-su commented on GitHub (Sep 30, 2025):
Thank you for your reply @ykwd . I confuse that how to set mooncake_master_server for using enable-ha. The startup command for mooncanke_master now is as follows:
nohup mooncake_master --enable-ha=true --etcd-endpoints="0.0.0.0:2379;0.0.0.0:2381;0.0.0.0:2383" --client-ttl=30 > /mooncake_master.log 2>&1
For convenience, I wrote a simple test myself: (This problem was encountered in sglang+mooncake)
`from mooncake.store import MooncakeDistributedStore
import time
import threading
store = MooncakeDistributedStore()
store.setup(
"192.168.2.31", # Your node's ip address
"192.168.2.38:2379", # P2P handshake (no HTTP metadata)
51210241024, # 512MB segment size
12810241024, # 128MB local buffer
"rdma", # Use TCP (RDMA for high performance)
"mlx5_2", # Leave empty; Mooncake auto-picks RDMA devices when needed
"192.168.2.38:50051" # master_server_address
)
attempt_count = 0
while True:
attempt_count += 1
keys = ['fd8886064cd208934888ddc8781fd6bff9ad72bbacab47d6b11419a0de792b30_0_k', 'bdcbac5573bd436b906cd5dba328c02090f4fe0f59055a0d8ad6e7144f0c6011_0_k']
values = [b"value1", b"value2"]
results = store.put_batch(keys, values)
status = "success" if results == 0 else f"failed with code: {results}"
time.sleep(1800)`
After one ttl, the registered memory is automatically logged out。I have checked mooncake-store-preview.md and I have also changed mooncake_master_service to etcd://IP:Port ;IP:Port; ...; IP:Port，but there will be another problem, as mentioned above
@jieyang-su commented on GitHub (Sep 30, 2025):
Of course, but there are some parameter changes in the new version, so it seems that the md file has not been updated at startup. Therefore, I chose to use Mooncake's 0.3.5 version to adapt to sglangv0.5.2.
By the way, pip seems to be unable to enable enable-ha, which I can use without enabling it, but it seems that enabling it requires compiling myself.
@ykwd commented on GitHub (Sep 30, 2025):
Yes. This result is expected. In older versions, if HA was enabled on the master, then HA also had to be enabled on the client (i.e., you must provide the address in the form of etcd://IP:Port;IP:Port;...;IP:Port). Otherwise, the client would not periodically send heartbeats to the master, and the master would consider the client timed out and disconnected.
From the previous logs, the error seems to be caused by the client timing out when fetching data from etcd. You should check whether the server where the client is running can actually connect to etcd. Also, I noticed that the etcd endpoints you provided contain three different IPs—please confirm that you have indeed started three etcd instances and configured them with the corresponding IPs. For easier debugging, you might try starting with just a single etcd instance first.
@jieyang-su commented on GitHub (Sep 30, 2025):
Thank you for your suggestion @ykwd . I have reviewed my parameters and found that I forgot to set the rpc-address when starting mooncake_master_server. I found that the default is 0.0.0.0, so it cannot be used across hosts.
Now, it is working normally in preliminary testing.
nohup mooncake_master --enable-ha=true --etcd-endpoints="0.0.0.0:2379;0.0.0.0:2381;0.0.0.0:2383" --rpc-address=xxx.xxx.xxx.xxx --client-ttl=30 > /mooncake_master.log 2>&1
@ykwd commented on GitHub (Sep 30, 2025):
Glad to hear the issue is resolved! Happy to help anytime.