[PR #501] [MERGED] [Store] High Availability V2: Client Failover #1609

New issue

Closed

opened 2026-10-29 21:28:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:28:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/501
Author: @ykwd
Created: 6/16/2025
Status: ✅ Merged
Merged: 6/20/2025
Merged by: @ykwd

Base: main ← Head: feature/ha2

📝 Commits (10+)

2896379 A temp version. Better to continue development after merging the latest main branch
483762f Resolve merge conflicts
b1abc1f Temp version to merge the latest main branch
8c87ecb merge main
0bbcef9 Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug.
fb55405 Refactor the etcd_helper
6e10b11 refactor ha_helper
778f2c4 Add some unit tests. Refactor the code
aa60561 Update cmakelists: build etcd_wrapper in default
3df656c Merge main

📊 Changes

30 files changed (+2261 additions, -559 deletions)

View changed files

📝 mooncake-integration/store/store_py.cpp (+1 -2)
📝 mooncake-store/include/allocation_strategy.h (+34 -80)
📝 mooncake-store/include/client.h (+10 -13)
📝 mooncake-store/include/ha_helper.h (+2 -0)
📝 mooncake-store/include/master_client.h (+21 -8)
📝 mooncake-store/include/master_metric_manager.h (+25 -0)
📝 mooncake-store/include/master_service.h (+80 -55)
📝 mooncake-store/include/rpc_service.h (+58 -23)
➕ mooncake-store/include/segment.h (+182 -0)
📝 mooncake-store/include/types.h (+61 -5)
📝 mooncake-store/src/CMakeLists.txt (+1 -0)
📝 mooncake-store/src/client.cpp (+103 -74)
📝 mooncake-store/src/ha_helper.cpp (+5 -3)
📝 mooncake-store/src/master.cpp (+9 -4)
📝 mooncake-store/src/master_client.cpp (+42 -16)
📝 mooncake-store/src/master_metric_manager.cpp (+71 -3)
📝 mooncake-store/src/master_service.cpp (+282 -135)
➕ mooncake-store/src/segment.cpp (+229 -0)
📝 mooncake-store/src/types.cpp (+17 -1)
📝 mooncake-store/tests/CMakeLists.txt (+14 -1)

...and 10 more files

📄 Description

Following https://github.com/kvcache-ai/Mooncake/pull/451, this PR serves as the 2nd step towards store's high availability.

This PR mainly focuses on the failover on the client side. More specifically, this PR adds the following features:

When the master has not heard from a certain client through Pings for a pre-defined time period, either because the client terminated or there are some network partitions, the master will remount all segments on this client.
When the client regains connections to the master after a network partition, it can automatically remount all the local segments.

There are many corner cases, such as:

When a client crashes and then restarts, it may attempt to mount a segment with the same name as before, e.g., using its hostname as the segment name. Though the two segments have the same name, the old one is no longer available. The master must identify this situation, unmount the old segment, and mount the new one.
When a client gets partitioned from the master, the master unmounts its segments, but has not finished deleting all the replicas that are allocated on the unmounted segments. At this time, the client connects to the master and remounts its segment. Then the segment shall not be mounted and allocated memory to new key-values, as the old key-values that reside on this segment may still exist.
There is a leader view change. The client needs to remount all the segments to the new leader.
etc.

To deal with all the corner cases:

A UUID is associated with each client instance and each segment.
The segment unmounting is divided into two phases: prepare and commit. In the prepare phase, the segment's buffer allocator is deleted and is marked as unmounting. After all the associated replicas are deleted, in the commit phase, the segment is fully deleted. A segment cannot be mounted if the segment with the same UUID is unmounting.
On the master side, the client status and segment status shall be updated together in an atomic way to avoid state inconsistency.

Currently, there is no distributed testing or chaos testing for the HA features. Thus, the HA features shall be considered as highly unstable. So this PR still does not change the default behavior of the system. HA features will be toggled on only when users explicitly specify --enable-ha=true. We will design and run more testing, including integration testing, distributed testing, and chaos testing, in the subsequent PRs. Only after passing all these tests would it be safe to introduce the HA features in the docs.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/501 **Author:** [@ykwd](https://github.com/ykwd) **Created:** 6/16/2025 **Status:** ✅ Merged **Merged:** 6/20/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `feature/ha2` --- ### 📝 Commits (10+) - [`2896379`](https://github.com/kvcache-ai/Mooncake/commit/2896379077d83f0c428c9afc0103846f8be11117) A temp version. Better to continue development after merging the latest main branch - [`483762f`](https://github.com/kvcache-ai/Mooncake/commit/483762f8d459043edb0bb717af8bd56becc85323) Resolve merge conflicts - [`b1abc1f`](https://github.com/kvcache-ai/Mooncake/commit/b1abc1f106fd4d7ffe687ec3d018b8226eb5dbf0) Temp version to merge the latest main branch - [`8c87ecb`](https://github.com/kvcache-ai/Mooncake/commit/8c87ecb95e56708c2811b15768ed6dd9c77555c0) merge main - [`0bbcef9`](https://github.com/kvcache-ai/Mooncake/commit/0bbcef96abd6aa240c82fd06af601448605e9235) Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug. - [`fb55405`](https://github.com/kvcache-ai/Mooncake/commit/fb55405982630793696d13983098e782bf49d455) Refactor the etcd_helper - [`6e10b11`](https://github.com/kvcache-ai/Mooncake/commit/6e10b1168c20a2ff2ffa6eccbb3ca04ea5024e29) refactor ha_helper - [`778f2c4`](https://github.com/kvcache-ai/Mooncake/commit/778f2c41c880f5602ccf9bb8b895b20a7074cdc0) Add some unit tests. Refactor the code - [`aa60561`](https://github.com/kvcache-ai/Mooncake/commit/aa60561a17acbe1cab82bfa4a77f429511092a75) Update cmakelists: build etcd_wrapper in default - [`3df656c`](https://github.com/kvcache-ai/Mooncake/commit/3df656c21cf3ced020f8f9727cccb06c614f7994) Merge main ### 📊 Changes **30 files changed** (+2261 additions, -559 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-integration/store/store_py.cpp` (+1 -2) 📝 `mooncake-store/include/allocation_strategy.h` (+34 -80) 📝 `mooncake-store/include/client.h` (+10 -13) 📝 `mooncake-store/include/ha_helper.h` (+2 -0) 📝 `mooncake-store/include/master_client.h` (+21 -8) 📝 `mooncake-store/include/master_metric_manager.h` (+25 -0) 📝 `mooncake-store/include/master_service.h` (+80 -55) 📝 `mooncake-store/include/rpc_service.h` (+58 -23) ➕ `mooncake-store/include/segment.h` (+182 -0) 📝 `mooncake-store/include/types.h` (+61 -5) 📝 `mooncake-store/src/CMakeLists.txt` (+1 -0) 📝 `mooncake-store/src/client.cpp` (+103 -74) 📝 `mooncake-store/src/ha_helper.cpp` (+5 -3) 📝 `mooncake-store/src/master.cpp` (+9 -4) 📝 `mooncake-store/src/master_client.cpp` (+42 -16) 📝 `mooncake-store/src/master_metric_manager.cpp` (+71 -3) 📝 `mooncake-store/src/master_service.cpp` (+282 -135) ➕ `mooncake-store/src/segment.cpp` (+229 -0) 📝 `mooncake-store/src/types.cpp` (+17 -1) 📝 `mooncake-store/tests/CMakeLists.txt` (+14 -1) _...and 10 more files_ </details> ### 📄 Description Following https://github.com/kvcache-ai/Mooncake/pull/451, this PR serves as the 2nd step towards store's high availability. This PR mainly focuses on the failover on the client side. More specifically, this PR adds the following features: - When the master has not heard from a certain client through Pings for a pre-defined time period, either because the client terminated or there are some network partitions, the master will remount all segments on this client. - When the client regains connections to the master after a network partition, it can automatically remount all the local segments. There are many corner cases, such as: - When a client crashes and then restarts, it may attempt to mount a segment with the same name as before, e.g., using its hostname as the segment name. Though the two segments have the same name, the old one is no longer available. The master must identify this situation, unmount the old segment, and mount the new one. - When a client gets partitioned from the master, the master unmounts its segments, but has not finished deleting all the replicas that are allocated on the unmounted segments. At this time, the client connects to the master and remounts its segment. Then the segment shall not be mounted and allocated memory to new key-values, as the old key-values that reside on this segment may still exist. - There is a leader view change. The client needs to remount all the segments to the new leader. - etc. To deal with all the corner cases: - A UUID is associated with each client instance and each segment. - The segment unmounting is divided into two phases: prepare and commit. In the prepare phase, the segment's buffer allocator is deleted and is marked as unmounting. After all the associated replicas are deleted, in the commit phase, the segment is fully deleted. A segment cannot be mounted if the segment with the same UUID is unmounting. - On the master side, the client status and segment status shall be updated together in an atomic way to avoid state inconsistency. Currently, there is no distributed testing or chaos testing for the HA features. Thus, the HA features shall be considered as highly unstable. So this PR still does not change the default behavior of the system. HA features will be toggled on only when users explicitly specify --enable-ha=true. We will design and run more testing, including integration testing, distributed testing, and chaos testing, in the subsequent PRs. Only after passing all these tests would it be safe to introduce the HA features in the docs. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:28:52 +00:00