kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3030] [RFC]: Replica-ID Validation for PutEnd / PutRevoke and GetReplicaList #780

New issue

Open

opened 2026-10-29 21:19:26 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:26 +00:00

Owner

Originally created by @ykwd on GitHub (Jul 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3030

Changes proposed

Status: Proposed
Module: Mooncake Store (mooncake-store)

Motivation

1. Same-client stale PutEnd / PutRevoke after PutStart preemption

PutEnd / PutRevoke currently only check that client_id matches the object's writer. This rejects a different client after preemption, but fails when the same client starts a second write that preempts an earlier in-progress write:

Client C: PutStart#1(key) → replicas R1 (PROCESSING)
          … discard timeout / UpsertStart preempts …
Client C: PutStart#2(key) → replicas R2 (PROCESSING); client_id still C
Client C: PutEnd#1 / PutRevoke#1(key) → accepted (client_id matches) → incorrectly completes or revokes R2

PutStart can reclaim after put_start_discard_timeout; UpsertStart preempts immediately. A stale PutEnd may mark R2 COMPLETE before data is written; a stale PutRevoke may delete R2 while Put#2 is still in progress.

2. False Get failures after lease expiry

On Get, the client first calls GetReplicaList (or equivalent) to obtain a replica list and a lease TTL, then transfers data. If the lease expires before the transfer finishes, the client returns LEASE_EXPIRED even when the underlying replicas are still valid, to avoid reading data that may have been evicted or replaced. For slow or large transfers this causes avoidable failures: the object may still exist unchanged, but without the replica IDs from GetReplicaList, the client cannot ask the master whether those same replicas are still live.

Proposal

Use replica IDs as the durable identity of allocated replicas. PutStart / UpsertStart / GetReplicaList all return them to the client; PutEnd / PutRevoke and lease-expired Get revalidation consume them.

Globally unique ReplicaID

ReplicaID (uint64_t) already exists on each replica. Strengthen allocation as follows:

On master fresh init, set next_id_ to a random uint64.
On each new replica, assign id = next_id_.fetch_add(1) (atomic).

Return ReplicaID from PutStart / UpsertStart / GetReplicaList

Every replica descriptor returned by PutStart, UpsertStart, and GetReplicaList (including batch variants) must include a non-zero ReplicaID. Clients retain these IDs for the matching write completion or for Get revalidation after lease expiry.

PutEnd / PutRevoke validate replica IDs

Extend end/revoke RPCs to carry the replica ID from the matching PutStart / UpsertStart:

PutEnd(client_id, key, tenant_id, replica_type, replica_id)
PutRevoke(client_id, key, tenant_id, replica_type, replica_id)

Lease-expired Get revalidation via GetReplicaList IDs

The client keeps the replica IDs from the initial GetReplicaList. After transfer, if the lease has expired, it re-queries the master with those ID. If the same IDs still exist and are COMPLETE, return success; otherwise fail as today (LEASE_EXPIRED / INVALID_REPLICA).

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @ykwd on GitHub (Jul 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3030 ### Changes proposed - **Status**: Proposed - **Module**: Mooncake Store (`mooncake-store`) ## Motivation ### 1. Same-client stale PutEnd / PutRevoke after PutStart preemption `PutEnd` / `PutRevoke` currently only check that `client_id` matches the object's writer. This rejects a different client after preemption, but fails when the **same client** starts a second write that preempts an earlier in-progress write: ```text Client C: PutStart#1(key) → replicas R1 (PROCESSING) … discard timeout / UpsertStart preempts … Client C: PutStart#2(key) → replicas R2 (PROCESSING); client_id still C Client C: PutEnd#1 / PutRevoke#1(key) → accepted (client_id matches) → incorrectly completes or revokes R2 ``` `PutStart` can reclaim after `put_start_discard_timeout`; `UpsertStart` preempts immediately. A stale `PutEnd` may mark R2 `COMPLETE` before data is written; a stale `PutRevoke` may delete R2 while Put#2 is still in progress. ### 2. False Get failures after lease expiry On Get, the client first calls `GetReplicaList` (or equivalent) to obtain a replica list and a lease TTL, then transfers data. If the lease expires before the transfer finishes, the client returns `LEASE_EXPIRED` even when the underlying replicas are still valid, to avoid reading data that may have been evicted or replaced. For slow or large transfers this causes avoidable failures: the object may still exist unchanged, but without the replica IDs from `GetReplicaList`, the client cannot ask the master whether those same replicas are still live. ## Proposal Use **replica IDs** as the durable identity of allocated replicas. `PutStart` / `UpsertStart` / `GetReplicaList` all return them to the client; `PutEnd` / `PutRevoke` and lease-expired Get revalidation consume them. ### Globally unique ReplicaID `ReplicaID` (`uint64_t`) already exists on each replica. Strengthen allocation as follows: - On master fresh init, set `next_id_` to a random `uint64`. - On each new replica, assign `id = next_id_.fetch_add(1)` (atomic). ### Return ReplicaID from PutStart / UpsertStart / GetReplicaList Every replica descriptor returned by `PutStart`, `UpsertStart`, and `GetReplicaList` (including batch variants) must include a non-zero `ReplicaID`. Clients retain these IDs for the matching write completion or for Get revalidation after lease expiry. ### PutEnd / PutRevoke validate replica IDs Extend end/revoke RPCs to carry the replica ID from the matching `PutStart` / `UpsertStart`: ```text PutEnd(client_id, key, tenant_id, replica_type, replica_id) PutRevoke(client_id, key, tenant_id, replica_type, replica_id) ``` ### Lease-expired Get revalidation via GetReplicaList IDs The client keeps the replica IDs from the initial `GetReplicaList`. After transfer, if the lease has expired, it re-queries the master with those ID. If the same IDs still exist and are `COMPLETE`, return success; otherwise fail as today (`LEASE_EXPIRED` / `INVALID_REPLICA`). ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:19:26 +00:00

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 21, 2026):

Thanks for opening this issue, @ykwd!

Field	Value
Issue	#3030
GitHub user ID	`11404354`
Reporter	@ykwd

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 21, 2026): Thanks for opening this issue, @ykwd! | Field | Value | |-------|-------| | **Issue** | #3030 | | **GitHub user ID** | `11404354` | | **Reporter** | @ykwd | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 20, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 20, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:27 +00:00

Author

Owner

@sssqqeer commented on GitHub (Oct 24, 2026):

@ykwd I'd like to take the lease-expired Get revalidation part of this RFC. I have a working version and will open a PR soon.

@sssqqeer commented on GitHub (Oct 24, 2026): @ykwd I'd like to take the lease-expired Get revalidation part of this RFC. I have a working version and will open a PR soon.

bsakaguchi referenced this issue

2026-10-29 21:30:39 +00:00

[PR #793] [CLOSED] [TransferEngine] Introduce generic NVMeoF transport implementation (#780) #1827