[PR #451] [MERGED] [Store] High Availability 1: Master Failover #1574

New issue

Closed

opened 2026-10-29 21:28:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:28:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/451
Author: @ykwd
Created: 6/5/2025
Status: ✅ Merged
Merged: 6/11/2025
Merged by: @xiaguan

Base: main ← Head: feature/ha

📝 Commits (10+)

2896379 A temp version. Better to continue development after merging the latest main branch
483762f Resolve merge conflicts
b1abc1f Temp version to merge the latest main branch
8c87ecb merge main
0bbcef9 Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug.
fb55405 Refactor the etcd_helper
6e10b11 refactor ha_helper
778f2c4 Add some unit tests. Refactor the code
aa60561 Update cmakelists: build etcd_wrapper in default
3df656c Merge main

📊 Changes

21 files changed (+1453 additions, -101 deletions)

View changed files

📝 .github/workflows/ci.yml (+10 -0)
📝 mooncake-common/CMakeLists.txt (+2 -2)
📝 mooncake-common/etcd/etcd_wrapper.go (+301 -10)
📝 mooncake-store/CMakeLists.txt (+4 -1)
📝 mooncake-store/include/client.h (+18 -4)
➕ mooncake-store/include/etcd_helper.h (+102 -0)
➕ mooncake-store/include/ha_helper.h (+111 -0)
📝 mooncake-store/include/master_client.h (+8 -0)
📝 mooncake-store/include/master_metric_manager.h (+3 -0)
📝 mooncake-store/include/rpc_service.h (+75 -14)
📝 mooncake-store/include/types.h (+12 -0)
📝 mooncake-store/src/CMakeLists.txt (+6 -2)
📝 mooncake-store/src/client.cpp (+148 -15)
➕ mooncake-store/src/etcd_helper.cpp (+151 -0)
➕ mooncake-store/src/ha_helper.cpp (+181 -0)
📝 mooncake-store/src/master.cpp (+61 -42)
📝 mooncake-store/src/master_client.cpp (+27 -1)
📝 mooncake-store/src/master_metric_manager.cpp (+14 -2)
📝 mooncake-store/src/types.cpp (+5 -0)
📝 mooncake-store/tests/CMakeLists.txt (+5 -8)

...and 1 more files

📄 Description

Currently, the high availability goal consists following contents:

The client can detect when the master goes down and automatically remount segments after the master restarts. This way, if the master crashes, only the master needs to be restarted without having to restart the entire cluster.
The master service contains multiple instances, one of which is the leader and serves the client requests. When the current leader fails, other instances can campaign to become the new leader and continue providing services. However, since the previous metadata is lost, it's equivalent to the cluster's key-value store being cleared, requiring the cache to be repopulated.
Tolerant client failure. The master can detect when a client fails and mark its associated data as invalid. When the client comes back online, it can re-register with the master.
KV metadata failover. Persist or replicate the kvs' metadata so that the newly elected leader can sync to the latest state, and the previously existing key-value data is still preserved.
Add fault-tolerant logic to the existing code to make the system more robust.
Add more tests to validate the robustness of the system under various failures.

This PR is the first step of a series of attempts to make mooncake-store achieve high availability. This PR achieves the 1st and 2nd goals listed above.

More precisely, this PR adds an optional HA mode that depends on etcd service. In HA mode, users can deploy multiple master instances, and at any time, only one instance is the leader to serve client requests. If N instances are deployed, the cluster can tolerate at most N-1 instances fail as long as the etcd service is alive. Currently, we also support non-HA mode, which is the default mode, and the system behaviour is equivalent to before.

As the high availability depends on etcd service, the installation of go and the compilation of etcd-wrapper become necessary if mooncake-store is to be compiled. Both mooncake-store and transferengine have an independent etcd client instance, so they won't interfere with each other.

Additionally, to do the corresponding unit tests in CI, we also need to deploy etcd service in the CI environment.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/451 **Author:** [@ykwd](https://github.com/ykwd) **Created:** 6/5/2025 **Status:** ✅ Merged **Merged:** 6/11/2025 **Merged by:** [@xiaguan](https://github.com/xiaguan) **Base:** `main` ← **Head:** `feature/ha` --- ### 📝 Commits (10+) - [`2896379`](https://github.com/kvcache-ai/Mooncake/commit/2896379077d83f0c428c9afc0103846f8be11117) A temp version. Better to continue development after merging the latest main branch - [`483762f`](https://github.com/kvcache-ai/Mooncake/commit/483762f8d459043edb0bb717af8bd56becc85323) Resolve merge conflicts - [`b1abc1f`](https://github.com/kvcache-ai/Mooncake/commit/b1abc1f106fd4d7ffe687ec3d018b8226eb5dbf0) Temp version to merge the latest main branch - [`8c87ecb`](https://github.com/kvcache-ai/Mooncake/commit/8c87ecb95e56708c2811b15768ed6dd9c77555c0) merge main - [`0bbcef9`](https://github.com/kvcache-ai/Mooncake/commit/0bbcef96abd6aa240c82fd06af601448605e9235) Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug. - [`fb55405`](https://github.com/kvcache-ai/Mooncake/commit/fb55405982630793696d13983098e782bf49d455) Refactor the etcd_helper - [`6e10b11`](https://github.com/kvcache-ai/Mooncake/commit/6e10b1168c20a2ff2ffa6eccbb3ca04ea5024e29) refactor ha_helper - [`778f2c4`](https://github.com/kvcache-ai/Mooncake/commit/778f2c41c880f5602ccf9bb8b895b20a7074cdc0) Add some unit tests. Refactor the code - [`aa60561`](https://github.com/kvcache-ai/Mooncake/commit/aa60561a17acbe1cab82bfa4a77f429511092a75) Update cmakelists: build etcd_wrapper in default - [`3df656c`](https://github.com/kvcache-ai/Mooncake/commit/3df656c21cf3ced020f8f9727cccb06c614f7994) Merge main ### 📊 Changes **21 files changed** (+1453 additions, -101 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+10 -0) 📝 `mooncake-common/CMakeLists.txt` (+2 -2) 📝 `mooncake-common/etcd/etcd_wrapper.go` (+301 -10) 📝 `mooncake-store/CMakeLists.txt` (+4 -1) 📝 `mooncake-store/include/client.h` (+18 -4) ➕ `mooncake-store/include/etcd_helper.h` (+102 -0) ➕ `mooncake-store/include/ha_helper.h` (+111 -0) 📝 `mooncake-store/include/master_client.h` (+8 -0) 📝 `mooncake-store/include/master_metric_manager.h` (+3 -0) 📝 `mooncake-store/include/rpc_service.h` (+75 -14) 📝 `mooncake-store/include/types.h` (+12 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+6 -2) 📝 `mooncake-store/src/client.cpp` (+148 -15) ➕ `mooncake-store/src/etcd_helper.cpp` (+151 -0) ➕ `mooncake-store/src/ha_helper.cpp` (+181 -0) 📝 `mooncake-store/src/master.cpp` (+61 -42) 📝 `mooncake-store/src/master_client.cpp` (+27 -1) 📝 `mooncake-store/src/master_metric_manager.cpp` (+14 -2) 📝 `mooncake-store/src/types.cpp` (+5 -0) 📝 `mooncake-store/tests/CMakeLists.txt` (+5 -8) _...and 1 more files_ </details> ### 📄 Description Currently, the high availability goal consists following contents: 1. The client can detect when the master goes down and automatically remount segments after the master restarts. This way, if the master crashes, only the master needs to be restarted without having to restart the entire cluster. 2. The master service contains multiple instances, one of which is the leader and serves the client requests. When the current leader fails, other instances can campaign to become the new leader and continue providing services. However, since the previous metadata is lost, it's equivalent to the cluster's key-value store being cleared, requiring the cache to be repopulated. 3. Tolerant client failure. The master can detect when a client fails and mark its associated data as invalid. When the client comes back online, it can re-register with the master. 4. KV metadata failover. Persist or replicate the kvs' metadata so that the newly elected leader can sync to the latest state, and the previously existing key-value data is still preserved. 5. Add fault-tolerant logic to the existing code to make the system more robust. 6. Add more tests to validate the robustness of the system under various failures. This PR is the first step of a series of attempts to make mooncake-store achieve high availability. This PR achieves the 1st and 2nd goals listed above. More precisely, this PR adds an optional HA mode that depends on etcd service. In HA mode, users can deploy multiple master instances, and at any time, only one instance is the leader to serve client requests. If N instances are deployed, the cluster can tolerate at most N-1 instances fail as long as the etcd service is alive. Currently, we also support non-HA mode, which is the default mode, and the system behaviour is equivalent to before. As the high availability depends on etcd service, the installation of go and the compilation of etcd-wrapper become necessary if mooncake-store is to be compiled. Both mooncake-store and transferengine have an independent etcd client instance, so they won't interfere with each other. Additionally, to do the corresponding unit tests in CI, we also need to deploy etcd service in the CI environment. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:28:35 +00:00