[PR #690] [MERGED] [Store]feat: Migrate Persistence Metadata from Client to Master Service #1747

New issue

Closed

opened 2026-10-29 21:30:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:30:00 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/690
Author: @SgtPepperr
Created: 7/29/2025
Status: ✅ Merged
Merged: 8/14/2025
Merged by: @stmatengss

Base: main ← Head: ssd_master

📝 Commits (10+)

30c50d6 initial commit
c18d2be fix client::query return fault
09176c3 fix isexist return fault
c5fef16 fix test bug
dba12af fix clearinvalidhandles problem
3f4d614 add file description for 3fs
d0f3929 change ssd function start from client to master
5bea6e9 fix conflict
1a91f2d fix naming error
40d2a5c edit doc description

📊 Changes

34 files changed (+875 additions, -492 deletions)

View changed files

📝 doc/en/mooncake-store-preview.md (+12 -8)
📝 doc/zh/mooncake-store-preview.md (+11 -6)
📝 docs/source/design/mooncake-store-preview.md (+12 -8)
📝 mooncake-store/include/client.h (+3 -5)
📝 mooncake-store/include/config.h (+1 -0)
📝 mooncake-store/include/ha_helper.h (+2 -0)
📝 mooncake-store/include/master_client.h (+5 -2)
📝 mooncake-store/include/master_service.h (+58 -6)
📝 mooncake-store/include/rpc_service.h (+5 -2)
📝 mooncake-store/include/storage_backend.h (+14 -44)
📝 mooncake-store/include/transfer_task.h (+2 -2)
📝 mooncake-store/include/types.h (+124 -32)
📝 mooncake-store/src/client.cpp (+104 -102)
📝 mooncake-store/src/client_buffer.cpp (+2 -2)
📝 mooncake-store/src/ha_helper.cpp (+3 -2)
📝 mooncake-store/src/hf3fs/README.md (+3 -2)
📝 mooncake-store/src/hf3fs/hf3fs_file.cpp (+2 -24)
📝 mooncake-store/src/master.cpp (+12 -1)
📝 mooncake-store/src/master_client.cpp (+8 -4)
📝 mooncake-store/src/master_service.cpp (+125 -43)

...and 14 more files

📄 Description

Background

Based on the iteration roadmap discussed in previous PRs #610 and issues #578 , this release migrates the metadata management of the SSD KV-cache in the hierarchical caching feature from the client to the master service, preserving the control-data separation design philosophy of Mooncake.
The main benefits are:

Improved BatchGet read performance on 3FS.
Stronger data-consistency guarantees.

Design & Implementation

The key idea is to extend the notion of replica so it can be either memory or disk.

PutStart now returns both memory replica and a disk replica.
The client writes data to each independently.
Disk-replica file writes are asynchronous (handled by a dedicated file-writer thread pool) and therefore do not block the synchronous path.
The PutEnd RPC is extended:
– The synchronous path issues PutEnd for the memory replica.
– The asynchronous path issues PutEnd for the disk replica.
Get chooses the appropriate read path depending on replica type.
Evict is modified: instead of deleting the entire metadata entry, only the memory replica portion is removed.
The persistence switch is still controlled by the client:
– If a storage path is specified → enable disk replica (Put/Get).
– If not specified → skip all disk-related operations.

The persistence feature's switch has been moved from the client to the master side, where enabling persistence is now controlled by specifying (--root_fs_dir=/path/to/dir)during master startup, requiring all client hosts to mount their DFS directories under this specified path to avoid potential abnormal behavior in Mooncake Store.

State Transitions

Moving metadata from the client to the master increases the complexity of state management:

Old model	New model
`memory` only	`mem`, `mem+disk`, `disk`

Further, each replica has its own status (e.g., PROCESSING, COMPLETED), so the combined state space explodes.
Explicit tests and eventually state-machine diagrams will be added to prevent bugs.

Basic transition rules

stateDiagram
    [*] --> empty

    empty --> mem      : Put
    empty --> disk     : Put
    empty --> mem+disk  : Put

    mem     --> empty : Remove
    mem     --> empty : Evict

    disk    --> empty : Remove

    mem+disk --> disk  : Evict
    mem+disk --> empty     : Remove

Corner Cases

Mixed persistence settings
3 clients, 2 with valid mount directory, 1 with invalid mount directory.
– potential abnormal behavior may happen in Mooncake Store.
Memory write fails, disk write succeeds (mem+disk path)
– Memory failure triggers PutRevoke, removing the memory replica.
– Disk success triggers PutEnd, leaving a COMPLETED disk replica usable by later reads.
New Put on an existing pure-disk replica
– Currently rejected with “object exists”.
– Future plan: allow re-adding a memory replica, transitioning disk → mem+disk.
Get while disk replica is still processing
– If the memory replica exist and are COMPLETED, the master returns all completed replicas, ignoring the still-processing disk replica.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/690 **Author:** [@SgtPepperr](https://github.com/SgtPepperr) **Created:** 7/29/2025 **Status:** ✅ Merged **Merged:** 8/14/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `ssd_master` --- ### 📝 Commits (10+) - [`30c50d6`](https://github.com/kvcache-ai/Mooncake/commit/30c50d640e8f892f2356e95f2aae497bd8206785) initial commit - [`c18d2be`](https://github.com/kvcache-ai/Mooncake/commit/c18d2bed98ed0b7ce910c219a7fe5b589976aed8) fix client::query return fault - [`09176c3`](https://github.com/kvcache-ai/Mooncake/commit/09176c3885c104e228fa8cf599a59efdc5005e9c) fix isexist return fault - [`c5fef16`](https://github.com/kvcache-ai/Mooncake/commit/c5fef1695da7cc822a0c7f0b170f529cc8c3a339) fix test bug - [`dba12af`](https://github.com/kvcache-ai/Mooncake/commit/dba12afe8f2287dd317ef2bcbcfca9178142379d) fix clearinvalidhandles problem - [`3f4d614`](https://github.com/kvcache-ai/Mooncake/commit/3f4d61403fd3c3fccf49440a556bbed3cd3e91a8) add file description for 3fs - [`d0f3929`](https://github.com/kvcache-ai/Mooncake/commit/d0f39293067b986c35ee3974bd6fa41aa48933f4) change ssd function start from client to master - [`5bea6e9`](https://github.com/kvcache-ai/Mooncake/commit/5bea6e9318eb34fb7b4858a653d3ab626fd8f241) fix conflict - [`1a91f2d`](https://github.com/kvcache-ai/Mooncake/commit/1a91f2dde5e7fa4b012394d9522672b06d1bd864) fix naming error - [`40d2a5c`](https://github.com/kvcache-ai/Mooncake/commit/40d2a5c3270b279fea8708873dfbbcfcf9c856c8) edit doc description ### 📊 Changes **34 files changed** (+875 additions, -492 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/mooncake-store-preview.md` (+12 -8) 📝 `doc/zh/mooncake-store-preview.md` (+11 -6) 📝 `docs/source/design/mooncake-store-preview.md` (+12 -8) 📝 `mooncake-store/include/client.h` (+3 -5) 📝 `mooncake-store/include/config.h` (+1 -0) 📝 `mooncake-store/include/ha_helper.h` (+2 -0) 📝 `mooncake-store/include/master_client.h` (+5 -2) 📝 `mooncake-store/include/master_service.h` (+58 -6) 📝 `mooncake-store/include/rpc_service.h` (+5 -2) 📝 `mooncake-store/include/storage_backend.h` (+14 -44) 📝 `mooncake-store/include/transfer_task.h` (+2 -2) 📝 `mooncake-store/include/types.h` (+124 -32) 📝 `mooncake-store/src/client.cpp` (+104 -102) 📝 `mooncake-store/src/client_buffer.cpp` (+2 -2) 📝 `mooncake-store/src/ha_helper.cpp` (+3 -2) 📝 `mooncake-store/src/hf3fs/README.md` (+3 -2) 📝 `mooncake-store/src/hf3fs/hf3fs_file.cpp` (+2 -24) 📝 `mooncake-store/src/master.cpp` (+12 -1) 📝 `mooncake-store/src/master_client.cpp` (+8 -4) 📝 `mooncake-store/src/master_service.cpp` (+125 -43) _...and 14 more files_ </details> ### 📄 Description ## Background Based on the iteration roadmap discussed in previous PRs #610 and issues #578 , this release migrates the metadata management of the SSD KV-cache in the hierarchical caching feature from the client to the master service, preserving the control-data separation design philosophy of Mooncake. The main benefits are: * Improved `BatchGet` read performance on 3FS. * Stronger data-consistency guarantees. ## Design & Implementation The key idea is to extend the notion of **replica** so it can be either **memory** or **disk**. * **`PutStart`** now returns **both** memory replica and a disk replica. The client writes data to each independently. Disk-replica file writes are **asynchronous** (handled by a dedicated file-writer thread pool) and therefore do **not** block the synchronous path. * The **`PutEnd`** RPC is extended: – The synchronous path issues `PutEnd` for the **memory** replica. – The asynchronous path issues `PutEnd` for the **disk** replica. * **`Get`** chooses the appropriate read path depending on replica type. * **`Evict`** is modified: instead of deleting the entire metadata entry, only the **memory** replica portion is removed. The persistence **switch** is still controlled by the client: – If a storage path is specified → enable disk replica (`Put`/`Get`). – If not specified → skip all disk-related operations. The persistence feature's switch has been moved from the client to the master side, where enabling persistence is now controlled by specifying (--root_fs_dir=/path/to/dir)during master startup, requiring all client hosts to mount their DFS directories under this specified path to avoid potential abnormal behavior in Mooncake Store. ## State Transitions Moving metadata from the client to the master increases the complexity of state management: | Old model | New model | |-----------|-----------| | `memory` only | `mem`, `mem+disk`, `disk` | Further, each replica has its own status (e.g., `PROCESSING`, `COMPLETED`), so the combined state space explodes. Explicit tests and eventually state-machine diagrams will be added to prevent bugs. ### Basic transition rules ```mermaid stateDiagram [*] --> empty empty --> mem : Put empty --> disk : Put empty --> mem+disk : Put mem --> empty : Remove mem --> empty : Evict disk --> empty : Remove mem+disk --> disk : Evict mem+disk --> empty : Remove ``` ## Corner Cases 1. **Mixed persistence settings** 3 clients, 2 with valid mount directory, 1 with invalid mount directory. – potential abnormal behavior may happen in Mooncake Store. 2. **Memory write fails, disk write succeeds (mem+disk path)** – Memory failure triggers `PutRevoke`, removing the memory replica. – Disk success triggers `PutEnd`, leaving a `COMPLETED` disk replica usable by later reads. 3. **New `Put` on an existing pure-disk replica** – Currently rejected with “object exists”. – Future plan: allow re-adding a memory replica, transitioning `disk` → `mem+disk`. 4. **Get while disk replica is still processing** – If the memory replica exist and are `COMPLETED`, the master returns all completed replicas, ignoring the still-processing disk replica. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:30:00 +00:00