[PR #993] [MERGED] [Store]: Cleanup processing objects if transferring timedout (#975) #1938

New issue

Closed

opened 2026-10-29 21:31:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:31:32 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/993
Author: @nickyc975
Created: 10/31/2025
Status: ✅ Merged
Merged: 11/6/2025
Merged by: @ykwd

Base: main ← Head: jinlong/fix-put-start-upstream

📝 Commits (4)

4bbc371 [Store]: Do not remove processing objects in RemoveAll
f460e17 [Store]: Keep client_id in master_client to reduce parameters
cd617bb [Store]: Ensure that PutStart and PutEnd/PutRevoke on a key is called by the same client
232d1e5 [Store]: Cleanup expired PutStart operations

📊 Changes

21 files changed (+1093 additions, -232 deletions)

View changed files

📝 doc/en/mooncake-store.md (+12 -0)
📝 doc/zh/mooncake-store.md (+12 -0)
📝 docs/source/design/mooncake-store.md (+12 -0)
📝 mooncake-store/include/client.h (+3 -3)
📝 mooncake-store/include/master_client.h (+9 -10)
📝 mooncake-store/include/master_config.h (+35 -0)
📝 mooncake-store/include/master_metric_manager.h (+16 -0)
📝 mooncake-store/include/master_service.h (+119 -31)
📝 mooncake-store/include/replica.h (+56 -1)
📝 mooncake-store/include/rpc_service.h (+9 -6)
📝 mooncake-store/include/types.h (+4 -0)
📝 mooncake-store/src/client.cpp (+8 -9)
📝 mooncake-store/src/master.cpp (+33 -1)
📝 mooncake-store/src/master_client.cpp (+20 -21)
📝 mooncake-store/src/master_metric_manager.cpp (+61 -1)
📝 mooncake-store/src/master_service.cpp (+159 -16)
📝 mooncake-store/src/rpc_service.cpp (+28 -21)
📝 mooncake-store/src/types.cpp (+1 -0)
📝 mooncake-store/tests/master_metrics_test.cpp (+22 -12)
📝 mooncake-store/tests/master_service_ssd_test.cpp (+144 -18)

...and 1 more files

📄 Description

As discussed in #975, this PR implemented the following mechenism:

Add client_id parameter to put_start/put_end/put_revoke, and save the client_id in object metadata. This is needed because now we may have more than one client that is processing the same key. We must identify the valid client and prevent unexpected put_end/put_revoke.
For a successful put_start, save the client_id and the current timestamp in object metadata. When another put_start on the same object arrived, if the object has been in processing state for more than 30s (maybe configurable), we discard existing replicas of the object and renew its client_id and timestamp.
Discarded replicas will be put into a staging list, and be released after 10-20 minutes (maybe configurable).
A new thread put_start_monitor_thread_ is added to monitor the ongoing Put operations. If an object is found stucking in processing state for more than 10-20 minutes (can be the same as timeout of discarded replicas), we remove it and release the space immediately.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/993 **Author:** [@nickyc975](https://github.com/nickyc975) **Created:** 10/31/2025 **Status:** ✅ Merged **Merged:** 11/6/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `jinlong/fix-put-start-upstream` --- ### 📝 Commits (4) - [`4bbc371`](https://github.com/kvcache-ai/Mooncake/commit/4bbc3715ee7cb0d535f86381e09590aea34c2a36) [Store]: Do not remove processing objects in RemoveAll - [`f460e17`](https://github.com/kvcache-ai/Mooncake/commit/f460e1740af79200b04dcb5f1e3c63c689fc44cf) [Store]: Keep client_id in master_client to reduce parameters - [`cd617bb`](https://github.com/kvcache-ai/Mooncake/commit/cd617bb796249d850d8f93666dea07570d7f4b12) [Store]: Ensure that PutStart and PutEnd/PutRevoke on a key is called by the same client - [`232d1e5`](https://github.com/kvcache-ai/Mooncake/commit/232d1e513432049828b48d2a6b2d2a0af39876c8) [Store]: Cleanup expired PutStart operations ### 📊 Changes **21 files changed** (+1093 additions, -232 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/mooncake-store.md` (+12 -0) 📝 `doc/zh/mooncake-store.md` (+12 -0) 📝 `docs/source/design/mooncake-store.md` (+12 -0) 📝 `mooncake-store/include/client.h` (+3 -3) 📝 `mooncake-store/include/master_client.h` (+9 -10) 📝 `mooncake-store/include/master_config.h` (+35 -0) 📝 `mooncake-store/include/master_metric_manager.h` (+16 -0) 📝 `mooncake-store/include/master_service.h` (+119 -31) 📝 `mooncake-store/include/replica.h` (+56 -1) 📝 `mooncake-store/include/rpc_service.h` (+9 -6) 📝 `mooncake-store/include/types.h` (+4 -0) 📝 `mooncake-store/src/client.cpp` (+8 -9) 📝 `mooncake-store/src/master.cpp` (+33 -1) 📝 `mooncake-store/src/master_client.cpp` (+20 -21) 📝 `mooncake-store/src/master_metric_manager.cpp` (+61 -1) 📝 `mooncake-store/src/master_service.cpp` (+159 -16) 📝 `mooncake-store/src/rpc_service.cpp` (+28 -21) 📝 `mooncake-store/src/types.cpp` (+1 -0) 📝 `mooncake-store/tests/master_metrics_test.cpp` (+22 -12) 📝 `mooncake-store/tests/master_service_ssd_test.cpp` (+144 -18) _...and 1 more files_ </details> ### 📄 Description As discussed in #975, this PR implemented the following mechenism: 1. Add client_id parameter to put_start/put_end/put_revoke, and save the client_id in object metadata. This is needed because now we may have more than one client that is processing the same key. We must identify the valid client and prevent unexpected put_end/put_revoke. 2. For a successful put_start, save the client_id and the current timestamp in object metadata. When another put_start on the same object arrived, if the object has been in processing state for more than 30s (maybe configurable), we discard existing replicas of the object and renew its client_id and timestamp. 3. Discarded replicas will be put into a staging list, and be released after 10-20 minutes (maybe configurable). 4. A new thread `put_start_monitor_thread_` is added to monitor the ongoing Put operations. If an object is found stucking in processing state for more than 10-20 minutes (can be the same as timeout of discarded replicas), we remove it and release the space immediately. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:31:32 +00:00