kvcache-ai/Mooncake

Fork

You've already forked Mooncake

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

Code Issues 368 Projects Packages Wiki Activity

[PR #1191] [MERGED] [Store] feat: Add BatchReplicaClear API for manual cache cleanup #2063

New issue

Closed

opened 2026-10-29 21:32:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:32:29 +00:00

Owner

Copy link

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1191
Author: @Bo-Vincent
Created: 12/10/2025
Status: ✅ Merged
Merged: 12/12/2025
Merged by: @stmatengss

Base: main ← Head: vin/replicaclear

📝 Commits (2)

89d1efe [Store] feat: Add BatchReplicaClear API for manual cache cleanup
d4c5216 [Store] feat: Fix BatchReplicaClear monitor issues

📊 Changes

15 files changed (+915 additions, -25 deletions)

View changed files

📝 doc/en/mooncake-store.md (+39 -6)
📝 doc/zh/mooncake-store.md (+46 -13)
📝 docs/source/design/mooncake-store.md (+39 -6)
📝 mooncake-store/include/client_service.h (+14 -0)
📝 mooncake-store/include/master_client.h (+14 -0)
📝 mooncake-store/include/master_metric_manager.h (+13 -0)
📝 mooncake-store/include/master_service.h (+16 -0)
📝 mooncake-store/include/rpc_service.h (+4 -0)
📝 mooncake-store/src/client_service.cpp (+8 -0)
📝 mooncake-store/src/master_client.cpp (+20 -0)
📝 mooncake-store/src/master_metric_manager.cpp (+71 -0)
📝 mooncake-store/src/master_service.cpp (+125 -0)
📝 mooncake-store/src/rpc_service.cpp (+41 -0)
📝 mooncake-store/tests/client_integration_test.cpp (+122 -0)
📝 mooncake-store/tests/master_service_test.cpp (+343 -0)

📄 Description

Implement the BatchReplicaClear API to allow clients to manually clear cached object replicas. This feature enables explicit storage management by deleting replicas for specified keys. The implementation is end-to-end, from the client to the master service.

Description

This pull request introduces a new RPC endpoint, BatchReplicaClear, designed to provide a deterministic way for applications or cleanup jobs to manually clear cached object replicas. This is particularly useful for reclaiming storage space from stale or unused entries that may persist beyond their intended lifecycle.

The implementation includes the following key aspects:

End-to-End API: A new BatchReplicaClear method is exposed through the Client, MasterClient, and MasterService, with a corresponding RPC handler.
Flexible Clearing Logic:
- Supports clearing all replicas for a key (by providing an empty segment_name).
- Supports clearing replicas only from a specific storage segment.
Robust Safety and Security Checks:
- Client Ownership Verification: Strictly ensures that the requesting client_id owns the object, preventing unauthorized data deletion.
- Active Lease Protection: Refuses to clear replicas of objects with an active lease, ensuring data consistency and preventing race conditions with active readers.
- In-Progress Write Protection: Safely skips clearing objects that have incomplete replicas, avoiding interference with ongoing Put operations.
Comprehensive Observability:
- New metrics have been added to track requests, failures, partial successes, and item counts for this new endpoint.
- Important operational events (e.g., skipping a key due to an active lease) are logged at the WARNING level for better visibility.

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

This feature has been thoroughly tested through a combination of unit and integration tests:

Unit Tests (master_service_test.cpp): A comprehensive suite of tests has been added to validate the core logic in MasterService. These tests cover numerous scenarios, including:
- Successfully clearing replicas from all segments and a specific segment.
- Correctly skipping objects with an active lease.
- Enforcing client ID ownership.
- Gracefully handling non-existent keys, empty key lists, and mixed-validity batches.
Integration Tests (client_integration_test.cpp): End-to-end tests have been implemented to verify the full functionality through the Client interface. These tests confirm that the RPC call works as expected and correctly handles lease expiration by using a reliable polling mechanism.

All new and existing tests pass successfully.

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1191 **Author:** [@Bo-Vincent](https://github.com/Bo-Vincent) **Created:** 12/10/2025 **Status:** ✅ Merged **Merged:** 12/12/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `vin/replicaclear` --- ### 📝 Commits (2) - [`89d1efe`](https://github.com/kvcache-ai/Mooncake/commit/89d1efe16aa60ab5ccfa50db54160b79447fb3cf) [Store] feat: Add BatchReplicaClear API for manual cache cleanup - [`d4c5216`](https://github.com/kvcache-ai/Mooncake/commit/d4c521635a6c8f0007268675c61528c5384e1ca2) [Store] feat: Fix BatchReplicaClear monitor issues ### 📊 Changes **15 files changed** (+915 additions, -25 deletions) <details> <summary>View changed files</summary> 📝 `doc/en/mooncake-store.md` (+39 -6) 📝 `doc/zh/mooncake-store.md` (+46 -13) 📝 `docs/source/design/mooncake-store.md` (+39 -6) 📝 `mooncake-store/include/client_service.h` (+14 -0) 📝 `mooncake-store/include/master_client.h` (+14 -0) 📝 `mooncake-store/include/master_metric_manager.h` (+13 -0) 📝 `mooncake-store/include/master_service.h` (+16 -0) 📝 `mooncake-store/include/rpc_service.h` (+4 -0) 📝 `mooncake-store/src/client_service.cpp` (+8 -0) 📝 `mooncake-store/src/master_client.cpp` (+20 -0) 📝 `mooncake-store/src/master_metric_manager.cpp` (+71 -0) 📝 `mooncake-store/src/master_service.cpp` (+125 -0) 📝 `mooncake-store/src/rpc_service.cpp` (+41 -0) 📝 `mooncake-store/tests/client_integration_test.cpp` (+122 -0) 📝 `mooncake-store/tests/master_service_test.cpp` (+343 -0) </details> ### 📄 Description Implement the BatchReplicaClear API to allow clients to manually clear cached object replicas. This feature enables explicit storage management by deleting replicas for specified keys. The implementation is end-to-end, from the client to the master service. ## Description This pull request introduces a new RPC endpoint, `BatchReplicaClear`, designed to provide a deterministic way for applications or cleanup jobs to manually clear cached object replicas. This is particularly useful for reclaiming storage space from stale or unused entries that may persist beyond their intended lifecycle. The implementation includes the following key aspects: - **End-to-End API**: A new `BatchReplicaClear` method is exposed through the `Client`, `MasterClient`, and `MasterService`, with a corresponding RPC handler. - **Flexible Clearing Logic**: - Supports clearing all replicas for a key (by providing an empty `segment_name`). - Supports clearing replicas only from a specific storage segment. - **Robust Safety and Security Checks**: - **Client Ownership Verification**: Strictly ensures that the requesting `client_id` owns the object, preventing unauthorized data deletion. - **Active Lease Protection**: Refuses to clear replicas of objects with an active lease, ensuring data consistency and preventing race conditions with active readers. - **In-Progress Write Protection**: Safely skips clearing objects that have incomplete replicas, avoiding interference with ongoing `Put` operations. - **Comprehensive Observability**: - New metrics have been added to track requests, failures, partial successes, and item counts for this new endpoint. - Important operational events (e.g., skipping a key due to an active lease) are logged at the `WARNING` level for better visibility. ## Type of Change * Types - [ ] Bug fix - [x] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested? This feature has been thoroughly tested through a combination of unit and integration tests: - **Unit Tests (`master_service_test.cpp`)**: A comprehensive suite of tests has been added to validate the core logic in `MasterService`. These tests cover numerous scenarios, including: - Successfully clearing replicas from all segments and a specific segment. - Correctly skipping objects with an active lease. - Enforcing client ID ownership. - Gracefully handling non-existent keys, empty key lists, and mixed-validity batches. - **Integration Tests (`client_integration_test.cpp`)**: End-to-end tests have been implemented to verify the full functionality through the `Client` interface. These tests confirm that the RPC call works as expected and correctly handles lease expiration by using a reliable polling mechanism. All new and existing tests pass successfully. ## Checklist - [x] I have performed a self-review of my own code. - [x] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:32:29 +00:00

closed this issue
added the
pull-request
label

No Branch/Tag specified

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels

Mirrored from GitHub Pull Request

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#2063

Reference in a new issue

Repository

kvcache-ai/Mooncake

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns