kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2953] [RFC]: [Store] Recoverable client liveness states and asynchronous offboarding #763

New issue

Closed

opened 2026-10-29 21:19:15 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:15 +00:00

Owner

Originally created by @Doube-Suzerain on GitHub (Jul 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2953

Changes proposed

Motivation

Mooncake Store currently treats one client heartbeat timeout as a permanent failure. Temporary network jitter, a client pause, or a delayed Master monitor loop can therefore irreversibly unmount recoverable resources and delete their metadata.

We propose adding a recovery window: after the first TTL expires, the Master retains the client's resources but stops routing new requests and work to them; a valid liveness signal makes them serving again. Terminal cleanup starts only if the client remains unavailable for a second TTL.

Resource availability checks are currently distributed across allocation, replica lookup, Copy/Move, task dispatch, drain, offload, and promotion. This proposal also centralizes that decision so the new liveness semantics are applied consistently.

Changes proposed

Introduce an Active → Suspected → Offline state machine for each client incarnation, with separate active and suspicion TTLs.
Add a typed ServingEligibility seam for all client-owned resource routing, while retaining Suspected resources for possible recovery.
Separate client liveness from remount readiness and segment lifecycle.
Express terminal cleanup as a retryable ClientOffboardingJob; complete it synchronously first, then move the same job to an asynchronous executor.
Preserve the existing Ping RPC schema and legacy TTL configuration.
Rebuild process-local liveness records when restoring Master snapshots; keep NoF on its independent health model.

Proposed design

Client liveness

Each client incarnation has a ClientLivenessRecord:

Active --active TTL--> Suspected --suspicion TTL--> Offline
   ^                       |
   +---- liveness signal --+

Active resources may serve new requests.
Suspected resources are retained but cannot receive new allocations, descriptors, or work. A valid liveness signal recovers the same incarnation.
Offline is terminal for that incarnation and starts offboarding.

The suspicion TTL starts when the Master actually commits Active -> Suspected, so a delayed monitor does not shorten the recovery window. Ping and successful segment mounts are liveness signals; ordinary Put/Get traffic and task/offload/promotion heartbeats are not.

Ping remains wire-compatible: a ready Active client receives OK; an unknown or Offline client receives NEED_REMOUNT; Ping may recover a Suspected client before applying the existing remount-readiness check.

Client liveness, remount readiness (ok_client_), and segment lifecycle remain independent. In particular, ok_client_ will no longer determine whether resources may serve or must be retained.

`ServingEligibility`

Allocation, metadata reads, task routing, offload, promotion, and drain will use a small typed interface with the following common semantics:

resource can serve
  = resource is intrinsically valid
  && segment lifecycle permits serving
  && owning ClientLivenessRecord is Active

ShouldRetainResources
  = ClientLivenessRecord is Active or Suspected

This supplements existing segment checks rather than replacing them. NoF resources continue to use their own probe and lifecycle rules.

While Suspected, the Master blocks new dependencies on the client, including allocations, descriptors, Copy/Move routing, task delivery, offload/promotion work, and drain selection. End, Revoke, failure, and completion callbacks for already-started work remain accepted so refcounts, quota, and task state can converge.

Memory allocator registrations and buffers retain the specific ClientLivenessRecord that created them, preventing a later incarnation from reactivating old memory resources.

Offboarding

Terminal retirement uses one job interface:

PrepareClientOffboarding(record) -> ClientOffboardingJob
CompleteClientOffboarding(job)

Prepare commits Offline and immediately isolates the incarnation. Complete performs the expensive work: metadata cleanup, memory segment unmount, LocalDisk/offload/promotion cleanup, quota convergence, and external metadata cleanup. The old record is removed only after all steps succeed; until then it acts as a Mount/ReMount barrier.

The first implementation calls Complete synchronously from the client monitor. A follow-up submits the same job to an asynchronous executor with batching, retry, shutdown draining, and queue observability. This keeps state and cleanup semantics unchanged when execution becomes asynchronous.

HA restore and compatibility

After snapshot restore, each restored client starts as Active with a fresh active TTL, preserving the current recovery semantics. The Master rebuilds the liveness associations needed by restored resources, while readiness still follows the existing ReMount protocol.

Snapshot creation must not capture partially completed offboarding. The synchronous phase preserves the current exclusion behavior; asynchronous offboarding will add explicit snapshot/offboarding coordination.

New configuration:

client_active_ttl_sec: 10
client_suspicion_ttl_sec: 20

client_live_ttl_sec and --client_ttl remain deprecated aliases for the active TTL. If an active/legacy TTL is explicitly configured but suspicion TTL is omitted, suspicion inherits the resolved active TTL. No RPC schema change is required.

LocalDisk and task metadata currently identify ownership by client ID. Initially, complete cleanup plus the Offline remount barrier prevents old resources from overlapping a new incarnation. An explicit incarnation token would provide stricter isolation for artificially reused client IDs, but requires a wire/schema change and is outside the initial scope.

Development plan

Add ClientLivenessRecord, dual-TTL configuration, Ping/Mount integration, and the three-state monitor.
Introduce ServingEligibility across allocation, descriptors, Copy/Move, task/drain, LocalDisk/offload, and promotion.
Add synchronous ClientOffboardingJob, retry semantics, HA rebinding, metrics, and regression coverage.
Move job completion to the asynchronous executor without changing state semantics.

Implementation may proceed while this RFC is discussed, in reviewable commits that can be adjusted before merge.

Open questions

Should Admin descriptor queries hide Suspected resources while diagnostic status queries still expose them?
Is it acceptable for new offload/promotion work rejected during Suspected to require a later rescan rather than automatic replay initially?
Should snapshots wait for pending asynchronous offboarding jobs, or should offboarding state eventually be serialized?
Is preserving the current RPC schema preferable to adding an explicit incarnation token in the first version?

Feedback on the state semantics, ServingEligibility seam, and staged synchronous-to-asynchronous offboarding plan is especially welcome.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Doube-Suzerain on GitHub (Jul 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2953 ### Changes proposed ## Motivation Mooncake Store currently treats one client heartbeat timeout as a permanent failure. Temporary network jitter, a client pause, or a delayed Master monitor loop can therefore irreversibly unmount recoverable resources and delete their metadata. We propose adding a recovery window: after the first TTL expires, the Master retains the client's resources but stops routing new requests and work to them; a valid liveness signal makes them serving again. Terminal cleanup starts only if the client remains unavailable for a second TTL. Resource availability checks are currently distributed across allocation, replica lookup, Copy/Move, task dispatch, drain, offload, and promotion. This proposal also centralizes that decision so the new liveness semantics are applied consistently. ## Changes proposed - Introduce an **Active → Suspected → Offline** state machine for each client incarnation, with separate active and suspicion TTLs. - Add a typed `ServingEligibility` seam for all client-owned resource routing, while retaining Suspected resources for possible recovery. - Separate client liveness from remount readiness and segment lifecycle. - Express terminal cleanup as a retryable `ClientOffboardingJob`; complete it synchronously first, then move the same job to an asynchronous executor. - Preserve the existing Ping RPC schema and legacy TTL configuration. - Rebuild process-local liveness records when restoring Master snapshots; keep NoF on its independent health model. ## Proposed design ### Client liveness Each client incarnation has a `ClientLivenessRecord`: ```text Active --active TTL--> Suspected --suspicion TTL--> Offline ^ | +---- liveness signal --+ ``` - **Active** resources may serve new requests. - **Suspected** resources are retained but cannot receive new allocations, descriptors, or work. A valid liveness signal recovers the same incarnation. - **Offline** is terminal for that incarnation and starts offboarding. The suspicion TTL starts when the Master actually commits `Active -> Suspected`, so a delayed monitor does not shorten the recovery window. Ping and successful segment mounts are liveness signals; ordinary Put/Get traffic and task/offload/promotion heartbeats are not. Ping remains wire-compatible: a ready Active client receives `OK`; an unknown or Offline client receives `NEED_REMOUNT`; Ping may recover a Suspected client before applying the existing remount-readiness check. Client liveness, remount readiness (`ok_client_`), and segment lifecycle remain independent. In particular, `ok_client_` will no longer determine whether resources may serve or must be retained. ### `ServingEligibility` Allocation, metadata reads, task routing, offload, promotion, and drain will use a small typed interface with the following common semantics: ```text resource can serve = resource is intrinsically valid && segment lifecycle permits serving && owning ClientLivenessRecord is Active ShouldRetainResources = ClientLivenessRecord is Active or Suspected ``` This supplements existing segment checks rather than replacing them. NoF resources continue to use their own probe and lifecycle rules. While Suspected, the Master blocks new dependencies on the client, including allocations, descriptors, Copy/Move routing, task delivery, offload/promotion work, and drain selection. End, Revoke, failure, and completion callbacks for already-started work remain accepted so refcounts, quota, and task state can converge. Memory allocator registrations and buffers retain the specific `ClientLivenessRecord` that created them, preventing a later incarnation from reactivating old memory resources. ### Offboarding Terminal retirement uses one job interface: ```text PrepareClientOffboarding(record) -> ClientOffboardingJob CompleteClientOffboarding(job) ``` Prepare commits Offline and immediately isolates the incarnation. Complete performs the expensive work: metadata cleanup, memory segment unmount, LocalDisk/offload/promotion cleanup, quota convergence, and external metadata cleanup. The old record is removed only after all steps succeed; until then it acts as a Mount/ReMount barrier. The first implementation calls Complete synchronously from the client monitor. A follow-up submits the same job to an asynchronous executor with batching, retry, shutdown draining, and queue observability. This keeps state and cleanup semantics unchanged when execution becomes asynchronous. ### HA restore and compatibility After snapshot restore, each restored client starts as Active with a fresh active TTL, preserving the current recovery semantics. The Master rebuilds the liveness associations needed by restored resources, while readiness still follows the existing ReMount protocol. Snapshot creation must not capture partially completed offboarding. The synchronous phase preserves the current exclusion behavior; asynchronous offboarding will add explicit snapshot/offboarding coordination. New configuration: ```yaml client_active_ttl_sec: 10 client_suspicion_ttl_sec: 20 ``` `client_live_ttl_sec` and `--client_ttl` remain deprecated aliases for the active TTL. If an active/legacy TTL is explicitly configured but suspicion TTL is omitted, suspicion inherits the resolved active TTL. No RPC schema change is required. LocalDisk and task metadata currently identify ownership by client ID. Initially, complete cleanup plus the Offline remount barrier prevents old resources from overlapping a new incarnation. An explicit incarnation token would provide stricter isolation for artificially reused client IDs, but requires a wire/schema change and is outside the initial scope. ## Development plan 1. Add `ClientLivenessRecord`, dual-TTL configuration, Ping/Mount integration, and the three-state monitor. 2. Introduce `ServingEligibility` across allocation, descriptors, Copy/Move, task/drain, LocalDisk/offload, and promotion. 3. Add synchronous `ClientOffboardingJob`, retry semantics, HA rebinding, metrics, and regression coverage. 4. Move job completion to the asynchronous executor without changing state semantics. Implementation may proceed while this RFC is discussed, in reviewable commits that can be adjusted before merge. ## Open questions 1. Should Admin descriptor queries hide Suspected resources while diagnostic status queries still expose them? 2. Is it acceptable for new offload/promotion work rejected during Suspected to require a later rescan rather than automatic replay initially? 3. Should snapshots wait for pending asynchronous offboarding jobs, or should offboarding state eventually be serialized? 4. Is preserving the current RPC schema preferable to adding an explicit incarnation token in the first version? Feedback on the state semantics, `ServingEligibility` seam, and staged synchronous-to-asynchronous offboarding plan is especially welcome. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:19:15 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Thanks for opening this issue, @Doube-Suzerain!

Field	Value
Issue	#2953
GitHub user ID	`43774695`
Reporter	@Doube-Suzerain

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Thanks for opening this issue, @Doube-Suzerain! | Field | Value | |-------|-------| | **Issue** | #2953 | | **GitHub user ID** | `43774695` | | **Reporter** | @Doube-Suzerain | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 15, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 15, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 22, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 22, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:29 +00:00

[PR #763] [MERGED] refactor(store): remove garbage collection implementation #1804