kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3808] [RoadMap]: Mooncake Store HA and OpLog Roadmap #971

New issue

Open

opened 2026-10-29 21:21:29 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:21:29 +00:00

Owner

Originally created by @Icedcoco on GitHub (Sep 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3808

Last reviewed: 2026-08-31
Code baseline: main at 73471dc28
Scope: durable batch OpLog, HA leadership, standby recovery, snapshot migration, replica lifecycle, entity ownership, observability, codecs, and conditional Segment persistence work.

This is the roadmap for the work tracked under oplog-ha-prs. It consolidates the PR-level specifications and the earlier RFC direction into one delivery view. It is broader than the snapshot implementation: snapshots are the recovery and retention track, while writer correctness, fencing, lifecycle, identity, ownership, metrics, and codec work are independent but connected workstreams.

End state

Mooncake Store should converge on one production HA state model:

a fenced, ordered batch OpLog is the durable mutation history;
standby masters apply that history and periodically publish immutable materialized snapshots;
a new or promoted node restores a verified snapshot and replays its suffix before serving;
replica lifecycle events, resource ownership, Segment state, and access recency have stable identities and deterministic replay semantics;
OpLog and snapshot growth are bounded by a reader-visible compaction floor and tested etcd operations;
the legacy primary snapshot writer and other transitional writers are removed after rollout evidence, not merely after compilation.

The program does not introduce multiple authorities for the same state. During migration, old readers may remain for explicit import or historical compatibility, but production selection is explicit and never based on bucket/catalog listing.

Program invariants

Durability precedes visibility. A mutation or lifecycle transition is externally visible only after its required durable record is committed. No lock is held across an asynchronous durable wait.
Fencing is fail-closed. A stale producer view cannot append a batch. Fencing, corruption, and non-retryable storage failures stop admission and eventually remove serving, rather than silently falling back to an unfenced writer.
Recovery is deterministic. Restore validates identity, cursor, checksum, ordering, and resource descriptors. Unknown history, malformed state, or infrastructure failures never become an empty-cluster success.
One identity follows an entity. ObjectIdentity, ReplicaID, and the future created_at generation field are reused by asynchronous callbacks, lifecycle records, restore, promotion, and eviction. No global registry or operation-ID framework is required for the first version.
Ownership is explicit. Objects own Replicas, Replicas own allocation handles, and Segments own allocators. Release is durable and identity-checked before memory is freed.
Runtime and wire clocks are separate. Runtime leases use steady_clock; raw steady-clock values are never persisted. Any persisted time field requires a versioned wire contract and defined clock mapping.
Compatibility is staged. New and legacy writers do not double-write. Readers may support mixed historical records where specified, and cleanup happens only after a measured rollout window.

Baseline on latest main

The following capabilities are already present in upstream/main:

Capability	State	Reference
Snapshot descriptor/cursor protocol	Complete	#3178
Complete-batch standby capture	Complete	#3326
Chunked immutable standby artifact writer	Complete, with follow-up fixes	#3447
Fenced etcd maintenance lease and `latest`/`fallback` publication	Complete	#3640
Pointer bootstrap and OpLog suffix replay	Complete	#3642
Writer retry backoff and terminal state	Complete	#3166, #3204
Durable-prefix validation and producer-view fencing	Complete	#3165, #3201, #3384
Supervisor view-flow coverage and explicit restore errors	Complete	#3352, #3354
Public production wiring for the new snapshot mode	Not enabled	Owned by N08
Legacy primary snapshot manager and global snapshot barrier	Still active	Remove after migration gates

N06 is the next active snapshot item. PR #3794 is a draft implementation based on the current N04/N05 components; it is deliberately not a public configuration switch or a production-readiness claim.

Workstreams

1. Writer and storage correctness

This track makes the batch history trustworthy before more recovery features depend on it.

Item	Delivery	Key result	State
PR-00	Baseline	Pin the shared upstream base, focused commands, E2E artifacts, and rollback assumptions.	Maintained
W01	Retry backoff	Replace fixed retry spinning with bounded exponential backoff and interruptible `Stop()`.	Complete, #3166
W02	Durable awaiter	Add `AwaitDurable(sequence)` using the existing async primitives; wake covered waiters on prefix advancement or terminal failure.	Planned after W03
W03	Terminal writer state	Expose one terminal reason/state and callback for timeout, fencing, and non-retryable write errors.	Complete, #3204
S01	Prefix validation	Prove that a non-zero durable prefix points to an existing, internally consistent terminal batch; fail closed on contradictions.	Complete, #3165
S02	Producer-view compatibility	Preserve the historical view field during the compatibility phase.	Complete, #3201; superseded as authority by S03
S03	Producer-view fencing	Claim a monotonically higher standalone producer view and compare it in every batch transaction.	Complete, #3384

W02 is needed by lifecycle durable barriers, not by the first snapshot publication path. S03 is the current producer-view layout: the durable prefix carries the batch/sequence cursor, while the standalone view key fences writers.

2. Leadership, serving, and restore safety

These items turn storage errors and restore errors into visible service-state decisions.

Item	Delivery	Key result	State
L01	View-flow test	Verify that the supervisor's acquired view reaches the production writer; keep view-zero only for explicit unmanaged fixtures.	Complete, #3352
F01	Fenced writer switch	Require fenced transactions for production `enable_oplog + HA + etcd`; reject missing view instead of downgrading.	Planned; N08 gate
F02	Writer fail-stop	On terminal writer failure, close mutation admission, set admin not-ready, remove the RPC delegate, stop the service, and release leadership.	Planned after F01; N08 gate
F03	Unfenced-writer removal	Do not create a separate PR. F01/F02 make the legal production path fenced directly.	Conditionally closed
R01	Explicit restore errors	Return structured restore errors for endpoint, descriptor, allocation, and rebuild failures.	Complete, #3354
R02	Restore serving gate	Keep a restoring or failed node non-serving until the candidate is fully installed and validated.	Complete, #3497
X01	Real-etcd fault matrix	Exercise stale leaders, fencing, writer fail-stop, snapshot recovery, promotion, pruning rebootstrap, client rediscovery, and acknowledged-data audits.	Planned after F02 and N11

The leadership gate is behavioral: a node that cannot prove durable history, fencing ownership, or complete restore must not accept traffic, even if its process is alive.

3. Standby snapshot, recovery, and bounded retention

Snapshots are one workstream within the program. They materialize the state applied by a standby after a complete batch; they are not copies of individual OpLog records.

Item	Delivery	Key result	State
N01	Descriptor and cursor	Versioned descriptor, complete batch/sequence cursor, checksum fields, and strict validation.	Complete, #3178
N02	Bounded capture	Capture a complete applied batch without the primary snapshot mutex and expose chunk traversal.	Complete, #3326
N03	Artifact writer	Encode objects and Segments in bounded chunks, upload immutable artifacts, and verify size/checksum.	Complete, #3447 plus mainline fixes
N04	Maintenance lease and pointers	Serialize maintenance with etcd and atomically publish independently addressable `latest` and `fallback`.	Complete, #3640
N05	Bootstrap and suffix	Restore `latest`, then `fallback`, then a proven complete OpLog; replay only the suffix after the snapshot cursor.	Complete, #3642
N06	Coordinator	Schedule eligible attempts, gate on catch-up, compose lease/capture/writer/publisher, and isolate worker failures.	In progress, draft #3794
N07	Bounded promotion	Move the standby store and drain it by chunks without a second full object vector; preserve ReplicaID and serving gates.	Planned
N08	Production switch and E2E	Add default-off `enable_batch_oplog_snapshot`, validate configuration, and prove cold restore, suffix replay, promotion, and legacy regression with real etcd.	Planned; first production gate
N09	Floor-aware rebootstrap	A reader below `compaction_floor` rebuilds temporary state from a valid snapshot and atomically swaps it in.	Planned; deploy before pruning
N10	Safe batch delete	Add an idempotent range-bounded delete that can touch only batch records through an explicit inclusive bound.	Planned
N11	Snapshot object GC	Delete only unreferenced candidate prefixes after publication while protecting both pointer targets.	Planned
N12	Floor and pruning coordinator	Revalidate both snapshots, CAS the monotonic floor, then call N10. Never delete history before publishing the floor.	Planned
N13	Etcd capacity operations	Document and soak MVCC compaction, defragmentation, quota, `NOSPACE`, and long-running bounded-history behavior.	Planned

The N06 coordinator follows a 600-second default interval, skips an occupied maintenance lease, records the catch-up target observed when capture releases standby state, cancels only the in-memory capture on promotion/stop, and allows post-capture publication to finish independently. It remains inert unless explicitly constructed.

The N08 mode contract is:

enable_oplog=false
  -> legacy snapshot mode during migration

enable_oplog=true, enable_batch_oplog_snapshot=false
  -> batch OpLog without standby snapshot scheduling

enable_oplog=true, enable_batch_oplog_snapshot=true
  -> standby-generated snapshot mode
  -> requires HA with etcd and a complete shared object store
  -> requires enable_snapshot=false and enable_snapshot_restore=false

The new flag is default-off. Invalid combinations fail before workers, leadership, or RPC serving start. N08 must validate at least two snapshots, cold bootstrap, suffix replay, fallback recovery, corruption handling, crash points, lease competition, promotion, object-store outage, and target-hardware freeze/RSS/upload measurements.

Retention is deliberately ordered: N09 rollout, N11 protected-object cleanup, N12 floor publication, N10 deletion, then N13 backend capacity operations. GC or object-store failures may leak capacity temporarily, but must not stop a healthy OpLog apply loop or change authoritative pointers.

4. Replica lifecycle and durable mutation semantics

The lifecycle track changes the durable model from whole-object mutation payloads to small, identity-scoped Replica events. It is independent of the N-series snapshot schedule, but its reader and identity guarantees must be ready before lifecycle state becomes authoritative.

Item	Delivery	Key result	State
RL01	Lifecycle records and reader	Add `REPLICA_CREATE`, `COMPLETE`, `FAILED`, and `REMOVE`; read new and legacy history with identity-aware idempotence.	Planned after E06
RL02	Put/Upsert CREATE barrier	Make CREATE durable before returning a descriptor; use W02 without holding metadata or Segment locks across await.	Planned after RL01/W02
RL03	Copy/Move CREATE barrier	Reuse RL02 for targets; Move retains the source until target CREATE is durable.	Planned after RL02
RL04	Put terminal events	Persist COMPLETE or FAILED with the CREATE identity; late callbacks become no-ops.	Planned after RL03
RL05	Copy/Move terminal order	Enforce target CREATE -> transfer -> target terminal -> source REMOVE for Move.	Planned after RL04
RL06	Remove/evict/release	Route remove, eviction, lease expiry, and delayed release through one durable identity-scoped REMOVE and finalizer.	Planned after RL05
RL07	Lifecycle authoritative switch	Select either the old whole-object writer or lifecycle writer; support mixed historical reads without double-writing.	Planned after RL01-RL06 and reader rollout
RL08	Remove legacy lifecycle writer	Delete the old writer and transition-only fields after one complete rollout window; retain historical import readers as agreed.	Rollout cleanup

The hard invariant is that a successful Put/Copy/Move response has durable CREATE evidence, and a Move cannot create a source/target double-loss window. This track also supplies the event model needed for future GET access records.

5. Entity ownership and identity

This track makes restore, promotion, asynchronous callbacks, and resource release refer to the same logical entities.

Item	Delivery	Key result	State
E01	Allocation accounting fields	Distinguish requested bytes from allocator-reserved bytes while keeping old descriptors readable.	Planned after baseline
E02	Active allocation count	Make allocator-owned active allocation count the single counter across allocate, rebuild, move, and release.	Planned after E01/E03
E03	Atomic Segment admission	Use one atomic admission state for lock-free checks while keeping transitions under the owner lock.	Planned
I00	Preserve ReplicaID	Restore descriptor IDs exactly and set `next_id = max_live + 1` only after complete restore.	N07/N08 hard dependency
I01	Immutable Replica `created_at`	Add a stable generation timestamp for recreate/callback disambiguation, with compatibility fallback for old data.	Planned after I00
I02	ReplicaIdentity helper	Reuse `MasterService::ObjectIdentity` plus ReplicaID and created_at for short, lock-scoped lookup.	Planned after I01
I03	Put callback identity	Guard Put/Upsert callbacks against remove-and-recreate races.	Planned after I02
I04	Other callback identity	Apply the same helper to Copy, Move, SSD offload, promotion, and rebind callbacks.	Planned after I03
E04	Heap ownership compatibility	Add a minimal `unique_ptr<Replica>` side path while the value container remains default.	Planned after E02/I04
E05	Pointer container switch	Make pointer ownership the production default without changing serialized logical fields; retain one rollback window.	Planned after E04
E06	Unified REMOVED finalizer	Recheck full identity under shard lock, release the buffer first, then remove Replica and empty Objects safely.	Planned after E05/I04
E07	Remove value adapter	Delete the value-container compatibility layer after pointer and lifecycle rollout; retain historical readers.	Rollout cleanup

I00 is a snapshot/promotion gate. I01-I04 and E01-E07 improve lifecycle correctness and callback safety but should not be stuffed into N06 or the first snapshot switch.

6. Observability and codec evolution

Operational signals and serialization performance are separate tracks with explicit gates.

Item	Delivery	Key result	State
M01	Writer/admin metrics	Expose accepting state, retries, durable cursor, waiters, callback queue, last error, stuck range, and terminal reason from one state view.	Planned after W03
M02	Snapshot/capacity metrics	Expose snapshot ages, bytes/chunks, capture pause, upload/bootstrap/replay latency, lease/GC skips, floor, and uncompacted batches without high-cardinality labels.	Planned after N12 and M01
P01	Codec benchmark	Compare JSON and candidate binary formats using reproducible payload/CPU/throughput measurements without changing production code.	Planned; can run in parallel
P02	Binary reader	Add explicit format/version discrimination and read JSON plus binary with the same validation/checksum rules.	Conditional on P01 evidence
P03	Binary writer switch	Select one writer format at construction time; support mixed-history readers and rollback to JSON.	Conditional after P02 rollout
P04	Remove JSON writer	Delete the transitional JSON writer after a full binary rollout; keep JSON historical readers/import fixtures.	Conditional cleanup

P02-P04 are not assumed improvements. If P01 does not show a meaningful benefit on representative batches, the binary line stays closed and the existing JSON writer remains the simpler production choice.

7. Segment persistence and `SegmentView` decision

The Segment line is conditional, not part of the mandatory snapshot migration.

Item	Delivery	Key result	State
V01	Persist before publish	Write an idempotent Segment lifecycle record before publishing the new local state.	Planned after X01
V02	Reader/apply gate	Validate lifecycle transitions, mixed history, endpoint and allocator consistency before serving.	Planned after V01
V03	Decision	Use Segment write rate, OpLog size, restore ambiguity, allocator failures, snapshot size, and incidents to decide whether a separate durable view is justified.	Waiting for data
V04	Shadow `SegmentView`	If V03 chooses separation, write and compare a versioned shadow view while the existing OpLog/snapshot remains authoritative.	Conditional, not approved
V05	Authority switch	Make `SegmentView` the sole Segment recovery authority only after a zero-difference rollout and real etcd evidence.	Conditional
V06	Remove legacy Segment state	Delete duplicate state and fields after the switch window; retain historical import support.	Conditional cleanup

RFC #2979 provides the ownership and SegmentView architecture, but the current decision is evidence-driven: continue with Segment OpLog plus snapshot unless measurements or an incident demonstrate that it cannot provide an authoritative recoverable lifecycle. Do not add a second registry or keyspace just for conceptual cleanliness.

Milestones and dependency gates

Milestone	Required work	Exit evidence
M0: Durable history	W01/W03, S01-S03, L01, R01	Prefix and view fencing tests pass; invalid history fails closed.
M1: Safe service state	F01/F02, R02, M01	Stale writer and terminal backend failures cannot remain serving.
M2: New snapshot path	N01-N07, I00	Standby capture, publication, restore, suffix replay, and bounded promotion pass focused tests.
M3: Production switch	N08, X01, target hardware measurements	Two verified snapshots, cold bootstrap, promotion, fault matrix, and acknowledged-data audit pass in real etcd.
M4: Bounded history	N09-N13, M02	Readers recover across the floor; GC, deletion, MVCC compaction, quota, and `NOSPACE` behavior are repeatable.
M5: Lifecycle authority	RL01-RL07, I01-I04, E01-E06	Durable lifecycle records, stable callback identity, and one release order survive restart and promotion.
M6: Cleanup	Snapshot cleanup, RL08, E07, optional P04/V06	Old writers/configuration are unused for a measured window and can be removed without losing historical import.
M7: Access recency	GET OpLog plus snapshot schema update	A promoted node makes the same eviction decision as the former primary for acknowledged access history.

PRs that touch shared production files remain serial even when their designs can be developed in parallel. In particular, master_service.*, hot_standby_service.*, OpLog storage/writer files, and the E2E harness are dependency bottlenecks. Every parallel branch must rebase onto the current snapshot/HA mainline before merge, followed by combined focused tests.

Legacy snapshot retirement

The desired outcome is to remove the old primary snapshot implementation and use only the new HA batch OpLog plus standby-generated snapshot in production. Retirement is a rollout phase, not an N06 feature.

After M4 and a complete migration window:

stop producing primary snapshots and reject the legacy snapshot-only configuration;
make pointer-selected standby snapshots and OpLog suffix replay the only production HA authority;
keep a read-only legacy importer/reader only while old artifacts still require conversion;
remove the primary fork manager, primary scheduler, catalog fallback from production recovery, and the global snapshot barrier after auditing its remaining lock users;
remove legacy readers, flags, fixtures, and operator instructions when the import window closes.

Once floor publication and OpLog deletion begin, rollback requires a binary that understands the new pointer, snapshot, and floor protocol. Older binaries are not a supported rollback target.

Future GET access OpLog

After M3/M4 stabilize the new HA snapshot path, add a separate ordered OpLog event for successful object access. This is a follow-up protocol change and should not expand N06 or delay the first snapshot switch.

The design must define:

Event identity: use stable tenant_id plus object key and the applicable Replica generation identity; failed reads do not advance recency.
Ordering: place the event in an ordered batch/sequence. Persist a defined wall-clock epoch timestamp for policy and diagnostics, with OpLog sequence as the deterministic tie-breaker. Never persist a raw process-local steady-clock value.
Materialized state: replay updates last_access and the new snapshot format carries the value required for post-recovery eviction without replaying every historical GET.
Eviction policy: choose the oldest (last_access_time, last_access_sequence) among eligible objects/Replicas while excluding hard pins, active leases, and non-terminal lifecycle states.
Load control: begin with every successful GET for correctness, measure OpLog/etcd amplification, and only then consider coalescing or sampling with an explicit recency-accuracy bound.
Compatibility: version the new field, define snapshots without access history, and keep legacy snapshot data outside this reader.

Tests must cover new-node replay, equal/skewed timestamps, duplicate events, remove-and-recreate, hard-pin/lease exclusions, snapshot restore, floor pruning, and deterministic eviction after promotion.

RFC alignment

RFC #3167 defines standby-generated snapshots, dual pointers, floor-before-delete retention, and the direction to remove the primary snapshot writer.
RFC #2979 defines explicit Object/Replica/AllocatedBuffer/Segment ownership, stable identity, primitive lifecycle events, and the conditional SegmentView path.
RFC #3321 defines recovery leases and clock domains: runtime lease decisions use steady_clock, while persisted clock information requires an explicit versioned mapping. The v1 standby snapshot therefore does not persist lease deadlines or TTLs.

Completion criteria

The full roadmap is complete when HA mutations are fenced and fail-stop, restore and promotion are fail-closed and bounded, lifecycle and identity events are durable and replayable, metrics expose every recovery/capacity decision, optional codec/Segment migrations have evidence, legacy snapshot/lifecycle writers are retired, and GET access history yields deterministic recency-aware eviction after promotion.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Icedcoco on GitHub (Sep 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3808 **Last reviewed:** 2026-08-31 **Code baseline:** [`main` at `73471dc28`](https://github.com/kvcache-ai/Mooncake/commit/73471dc28) **Scope:** durable batch OpLog, HA leadership, standby recovery, snapshot migration, replica lifecycle, entity ownership, observability, codecs, and conditional Segment persistence work. This is the roadmap for the work tracked under [`oplog-ha-prs`](../specs/oplog-ha-prs/README.md). It consolidates the PR-level specifications and the earlier RFC direction into one delivery view. It is broader than the snapshot implementation: snapshots are the recovery and retention track, while writer correctness, fencing, lifecycle, identity, ownership, metrics, and codec work are independent but connected workstreams. ## End state Mooncake Store should converge on one production HA state model: - a fenced, ordered batch OpLog is the durable mutation history; - standby masters apply that history and periodically publish immutable materialized snapshots; - a new or promoted node restores a verified snapshot and replays its suffix before serving; - replica lifecycle events, resource ownership, Segment state, and access recency have stable identities and deterministic replay semantics; - OpLog and snapshot growth are bounded by a reader-visible compaction floor and tested etcd operations; - the legacy primary snapshot writer and other transitional writers are removed after rollout evidence, not merely after compilation. The program does not introduce multiple authorities for the same state. During migration, old readers may remain for explicit import or historical compatibility, but production selection is explicit and never based on bucket/catalog listing. ## Program invariants 1. **Durability precedes visibility.** A mutation or lifecycle transition is externally visible only after its required durable record is committed. No lock is held across an asynchronous durable wait. 2. **Fencing is fail-closed.** A stale producer view cannot append a batch. Fencing, corruption, and non-retryable storage failures stop admission and eventually remove serving, rather than silently falling back to an unfenced writer. 3. **Recovery is deterministic.** Restore validates identity, cursor, checksum, ordering, and resource descriptors. Unknown history, malformed state, or infrastructure failures never become an empty-cluster success. 4. **One identity follows an entity.** `ObjectIdentity`, `ReplicaID`, and the future `created_at` generation field are reused by asynchronous callbacks, lifecycle records, restore, promotion, and eviction. No global registry or operation-ID framework is required for the first version. 5. **Ownership is explicit.** Objects own Replicas, Replicas own allocation handles, and Segments own allocators. Release is durable and identity-checked before memory is freed. 6. **Runtime and wire clocks are separate.** Runtime leases use `steady_clock`; raw steady-clock values are never persisted. Any persisted time field requires a versioned wire contract and defined clock mapping. 7. **Compatibility is staged.** New and legacy writers do not double-write. Readers may support mixed historical records where specified, and cleanup happens only after a measured rollout window. ## Baseline on latest main The following capabilities are already present in `upstream/main`: | Capability | State | Reference | | --- | --- | --- | | Snapshot descriptor/cursor protocol | Complete | [#3178](https://github.com/kvcache-ai/Mooncake/pull/3178) | | Complete-batch standby capture | Complete | [#3326](https://github.com/kvcache-ai/Mooncake/pull/3326) | | Chunked immutable standby artifact writer | Complete, with follow-up fixes | [#3447](https://github.com/kvcache-ai/Mooncake/pull/3447) | | Fenced etcd maintenance lease and `latest`/`fallback` publication | Complete | [#3640](https://github.com/kvcache-ai/Mooncake/pull/3640) | | Pointer bootstrap and OpLog suffix replay | Complete | [#3642](https://github.com/kvcache-ai/Mooncake/pull/3642) | | Writer retry backoff and terminal state | Complete | [#3166](https://github.com/kvcache-ai/Mooncake/pull/3166), [#3204](https://github.com/kvcache-ai/Mooncake/pull/3204) | | Durable-prefix validation and producer-view fencing | Complete | [#3165](https://github.com/kvcache-ai/Mooncake/pull/3165), [#3201](https://github.com/kvcache-ai/Mooncake/pull/3201), [#3384](https://github.com/kvcache-ai/Mooncake/pull/3384) | | Supervisor view-flow coverage and explicit restore errors | Complete | [#3352](https://github.com/kvcache-ai/Mooncake/pull/3352), [#3354](https://github.com/kvcache-ai/Mooncake/pull/3354) | | Public production wiring for the new snapshot mode | Not enabled | Owned by N08 | | Legacy primary snapshot manager and global snapshot barrier | Still active | Remove after migration gates | N06 is the next active snapshot item. [PR #3794](https://github.com/kvcache-ai/Mooncake/pull/3794) is a draft implementation based on the current N04/N05 components; it is deliberately not a public configuration switch or a production-readiness claim. ## Workstreams ### 1. Writer and storage correctness This track makes the batch history trustworthy before more recovery features depend on it. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | PR-00 | Baseline | Pin the shared upstream base, focused commands, E2E artifacts, and rollback assumptions. | Maintained | | W01 | Retry backoff | Replace fixed retry spinning with bounded exponential backoff and interruptible `Stop()`. | Complete, #3166 | | W02 | Durable awaiter | Add `AwaitDurable(sequence)` using the existing async primitives; wake covered waiters on prefix advancement or terminal failure. | Planned after W03 | | W03 | Terminal writer state | Expose one terminal reason/state and callback for timeout, fencing, and non-retryable write errors. | Complete, #3204 | | S01 | Prefix validation | Prove that a non-zero durable prefix points to an existing, internally consistent terminal batch; fail closed on contradictions. | Complete, #3165 | | S02 | Producer-view compatibility | Preserve the historical view field during the compatibility phase. | Complete, #3201; superseded as authority by S03 | | S03 | Producer-view fencing | Claim a monotonically higher standalone producer view and compare it in every batch transaction. | Complete, #3384 | W02 is needed by lifecycle durable barriers, not by the first snapshot publication path. S03 is the current producer-view layout: the durable prefix carries the batch/sequence cursor, while the standalone view key fences writers. ### 2. Leadership, serving, and restore safety These items turn storage errors and restore errors into visible service-state decisions. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | L01 | View-flow test | Verify that the supervisor's acquired view reaches the production writer; keep view-zero only for explicit unmanaged fixtures. | Complete, #3352 | | F01 | Fenced writer switch | Require fenced transactions for production `enable_oplog + HA + etcd`; reject missing view instead of downgrading. | Planned; N08 gate | | F02 | Writer fail-stop | On terminal writer failure, close mutation admission, set admin not-ready, remove the RPC delegate, stop the service, and release leadership. | Planned after F01; N08 gate | | F03 | Unfenced-writer removal | Do not create a separate PR. F01/F02 make the legal production path fenced directly. | Conditionally closed | | R01 | Explicit restore errors | Return structured restore errors for endpoint, descriptor, allocation, and rebuild failures. | Complete, #3354 | | R02 | Restore serving gate | Keep a restoring or failed node non-serving until the candidate is fully installed and validated. | Complete, #3497 | | X01 | Real-etcd fault matrix | Exercise stale leaders, fencing, writer fail-stop, snapshot recovery, promotion, pruning rebootstrap, client rediscovery, and acknowledged-data audits. | Planned after F02 and N11 | The leadership gate is behavioral: a node that cannot prove durable history, fencing ownership, or complete restore must not accept traffic, even if its process is alive. ### 3. Standby snapshot, recovery, and bounded retention Snapshots are one workstream within the program. They materialize the state applied by a standby after a complete batch; they are not copies of individual OpLog records. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | N01 | Descriptor and cursor | Versioned descriptor, complete batch/sequence cursor, checksum fields, and strict validation. | Complete, #3178 | | N02 | Bounded capture | Capture a complete applied batch without the primary snapshot mutex and expose chunk traversal. | Complete, #3326 | | N03 | Artifact writer | Encode objects and Segments in bounded chunks, upload immutable artifacts, and verify size/checksum. | Complete, #3447 plus mainline fixes | | N04 | Maintenance lease and pointers | Serialize maintenance with etcd and atomically publish independently addressable `latest` and `fallback`. | Complete, #3640 | | N05 | Bootstrap and suffix | Restore `latest`, then `fallback`, then a proven complete OpLog; replay only the suffix after the snapshot cursor. | Complete, #3642 | | N06 | Coordinator | Schedule eligible attempts, gate on catch-up, compose lease/capture/writer/publisher, and isolate worker failures. | In progress, draft #3794 | | N07 | Bounded promotion | Move the standby store and drain it by chunks without a second full object vector; preserve ReplicaID and serving gates. | Planned | | N08 | Production switch and E2E | Add default-off `enable_batch_oplog_snapshot`, validate configuration, and prove cold restore, suffix replay, promotion, and legacy regression with real etcd. | Planned; first production gate | | N09 | Floor-aware rebootstrap | A reader below `compaction_floor` rebuilds temporary state from a valid snapshot and atomically swaps it in. | Planned; deploy before pruning | | N10 | Safe batch delete | Add an idempotent range-bounded delete that can touch only batch records through an explicit inclusive bound. | Planned | | N11 | Snapshot object GC | Delete only unreferenced candidate prefixes after publication while protecting both pointer targets. | Planned | | N12 | Floor and pruning coordinator | Revalidate both snapshots, CAS the monotonic floor, then call N10. Never delete history before publishing the floor. | Planned | | N13 | Etcd capacity operations | Document and soak MVCC compaction, defragmentation, quota, `NOSPACE`, and long-running bounded-history behavior. | Planned | The N06 coordinator follows a 600-second default interval, skips an occupied maintenance lease, records the catch-up target observed when capture releases standby state, cancels only the in-memory capture on promotion/stop, and allows post-capture publication to finish independently. It remains inert unless explicitly constructed. The N08 mode contract is: ```text enable_oplog=false -> legacy snapshot mode during migration enable_oplog=true, enable_batch_oplog_snapshot=false -> batch OpLog without standby snapshot scheduling enable_oplog=true, enable_batch_oplog_snapshot=true -> standby-generated snapshot mode -> requires HA with etcd and a complete shared object store -> requires enable_snapshot=false and enable_snapshot_restore=false ``` The new flag is default-off. Invalid combinations fail before workers, leadership, or RPC serving start. N08 must validate at least two snapshots, cold bootstrap, suffix replay, fallback recovery, corruption handling, crash points, lease competition, promotion, object-store outage, and target-hardware freeze/RSS/upload measurements. Retention is deliberately ordered: N09 rollout, N11 protected-object cleanup, N12 floor publication, N10 deletion, then N13 backend capacity operations. GC or object-store failures may leak capacity temporarily, but must not stop a healthy OpLog apply loop or change authoritative pointers. ### 4. Replica lifecycle and durable mutation semantics The lifecycle track changes the durable model from whole-object mutation payloads to small, identity-scoped Replica events. It is independent of the N-series snapshot schedule, but its reader and identity guarantees must be ready before lifecycle state becomes authoritative. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | RL01 | Lifecycle records and reader | Add `REPLICA_CREATE`, `COMPLETE`, `FAILED`, and `REMOVE`; read new and legacy history with identity-aware idempotence. | Planned after E06 | | RL02 | Put/Upsert CREATE barrier | Make CREATE durable before returning a descriptor; use W02 without holding metadata or Segment locks across await. | Planned after RL01/W02 | | RL03 | Copy/Move CREATE barrier | Reuse RL02 for targets; Move retains the source until target CREATE is durable. | Planned after RL02 | | RL04 | Put terminal events | Persist COMPLETE or FAILED with the CREATE identity; late callbacks become no-ops. | Planned after RL03 | | RL05 | Copy/Move terminal order | Enforce target CREATE -> transfer -> target terminal -> source REMOVE for Move. | Planned after RL04 | | RL06 | Remove/evict/release | Route remove, eviction, lease expiry, and delayed release through one durable identity-scoped REMOVE and finalizer. | Planned after RL05 | | RL07 | Lifecycle authoritative switch | Select either the old whole-object writer or lifecycle writer; support mixed historical reads without double-writing. | Planned after RL01-RL06 and reader rollout | | RL08 | Remove legacy lifecycle writer | Delete the old writer and transition-only fields after one complete rollout window; retain historical import readers as agreed. | Rollout cleanup | The hard invariant is that a successful Put/Copy/Move response has durable CREATE evidence, and a Move cannot create a source/target double-loss window. This track also supplies the event model needed for future GET access records. ### 5. Entity ownership and identity This track makes restore, promotion, asynchronous callbacks, and resource release refer to the same logical entities. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | E01 | Allocation accounting fields | Distinguish requested bytes from allocator-reserved bytes while keeping old descriptors readable. | Planned after baseline | | E02 | Active allocation count | Make allocator-owned active allocation count the single counter across allocate, rebuild, move, and release. | Planned after E01/E03 | | E03 | Atomic Segment admission | Use one atomic admission state for lock-free checks while keeping transitions under the owner lock. | Planned | | I00 | Preserve ReplicaID | Restore descriptor IDs exactly and set `next_id = max_live + 1` only after complete restore. | N07/N08 hard dependency | | I01 | Immutable Replica `created_at` | Add a stable generation timestamp for recreate/callback disambiguation, with compatibility fallback for old data. | Planned after I00 | | I02 | ReplicaIdentity helper | Reuse `MasterService::ObjectIdentity` plus ReplicaID and created_at for short, lock-scoped lookup. | Planned after I01 | | I03 | Put callback identity | Guard Put/Upsert callbacks against remove-and-recreate races. | Planned after I02 | | I04 | Other callback identity | Apply the same helper to Copy, Move, SSD offload, promotion, and rebind callbacks. | Planned after I03 | | E04 | Heap ownership compatibility | Add a minimal `unique_ptr<Replica>` side path while the value container remains default. | Planned after E02/I04 | | E05 | Pointer container switch | Make pointer ownership the production default without changing serialized logical fields; retain one rollback window. | Planned after E04 | | E06 | Unified REMOVED finalizer | Recheck full identity under shard lock, release the buffer first, then remove Replica and empty Objects safely. | Planned after E05/I04 | | E07 | Remove value adapter | Delete the value-container compatibility layer after pointer and lifecycle rollout; retain historical readers. | Rollout cleanup | I00 is a snapshot/promotion gate. I01-I04 and E01-E07 improve lifecycle correctness and callback safety but should not be stuffed into N06 or the first snapshot switch. ### 6. Observability and codec evolution Operational signals and serialization performance are separate tracks with explicit gates. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | M01 | Writer/admin metrics | Expose accepting state, retries, durable cursor, waiters, callback queue, last error, stuck range, and terminal reason from one state view. | Planned after W03 | | M02 | Snapshot/capacity metrics | Expose snapshot ages, bytes/chunks, capture pause, upload/bootstrap/replay latency, lease/GC skips, floor, and uncompacted batches without high-cardinality labels. | Planned after N12 and M01 | | P01 | Codec benchmark | Compare JSON and candidate binary formats using reproducible payload/CPU/throughput measurements without changing production code. | Planned; can run in parallel | | P02 | Binary reader | Add explicit format/version discrimination and read JSON plus binary with the same validation/checksum rules. | Conditional on P01 evidence | | P03 | Binary writer switch | Select one writer format at construction time; support mixed-history readers and rollback to JSON. | Conditional after P02 rollout | | P04 | Remove JSON writer | Delete the transitional JSON writer after a full binary rollout; keep JSON historical readers/import fixtures. | Conditional cleanup | P02-P04 are not assumed improvements. If P01 does not show a meaningful benefit on representative batches, the binary line stays closed and the existing JSON writer remains the simpler production choice. ### 7. Segment persistence and `SegmentView` decision The Segment line is conditional, not part of the mandatory snapshot migration. | Item | Delivery | Key result | State | | --- | --- | --- | --- | | V01 | Persist before publish | Write an idempotent Segment lifecycle record before publishing the new local state. | Planned after X01 | | V02 | Reader/apply gate | Validate lifecycle transitions, mixed history, endpoint and allocator consistency before serving. | Planned after V01 | | V03 | Decision | Use Segment write rate, OpLog size, restore ambiguity, allocator failures, snapshot size, and incidents to decide whether a separate durable view is justified. | Waiting for data | | V04 | Shadow `SegmentView` | If V03 chooses separation, write and compare a versioned shadow view while the existing OpLog/snapshot remains authoritative. | Conditional, not approved | | V05 | Authority switch | Make `SegmentView` the sole Segment recovery authority only after a zero-difference rollout and real etcd evidence. | Conditional | | V06 | Remove legacy Segment state | Delete duplicate state and fields after the switch window; retain historical import support. | Conditional cleanup | RFC #2979 provides the ownership and SegmentView architecture, but the current decision is evidence-driven: continue with Segment OpLog plus snapshot unless measurements or an incident demonstrate that it cannot provide an authoritative recoverable lifecycle. Do not add a second registry or keyspace just for conceptual cleanliness. ## Milestones and dependency gates | Milestone | Required work | Exit evidence | | --- | --- | --- | | M0: Durable history | W01/W03, S01-S03, L01, R01 | Prefix and view fencing tests pass; invalid history fails closed. | | M1: Safe service state | F01/F02, R02, M01 | Stale writer and terminal backend failures cannot remain serving. | | M2: New snapshot path | N01-N07, I00 | Standby capture, publication, restore, suffix replay, and bounded promotion pass focused tests. | | M3: Production switch | N08, X01, target hardware measurements | Two verified snapshots, cold bootstrap, promotion, fault matrix, and acknowledged-data audit pass in real etcd. | | M4: Bounded history | N09-N13, M02 | Readers recover across the floor; GC, deletion, MVCC compaction, quota, and `NOSPACE` behavior are repeatable. | | M5: Lifecycle authority | RL01-RL07, I01-I04, E01-E06 | Durable lifecycle records, stable callback identity, and one release order survive restart and promotion. | | M6: Cleanup | Snapshot cleanup, RL08, E07, optional P04/V06 | Old writers/configuration are unused for a measured window and can be removed without losing historical import. | | M7: Access recency | GET OpLog plus snapshot schema update | A promoted node makes the same eviction decision as the former primary for acknowledged access history. | PRs that touch shared production files remain serial even when their designs can be developed in parallel. In particular, `master_service.*`, `hot_standby_service.*`, OpLog storage/writer files, and the E2E harness are dependency bottlenecks. Every parallel branch must rebase onto the current snapshot/HA mainline before merge, followed by combined focused tests. ## Legacy snapshot retirement The desired outcome is to remove the old primary snapshot implementation and use only the new HA batch OpLog plus standby-generated snapshot in production. Retirement is a rollout phase, not an N06 feature. After M4 and a complete migration window: 1. stop producing primary snapshots and reject the legacy snapshot-only configuration; 2. make pointer-selected standby snapshots and OpLog suffix replay the only production HA authority; 3. keep a read-only legacy importer/reader only while old artifacts still require conversion; 4. remove the primary fork manager, primary scheduler, catalog fallback from production recovery, and the global snapshot barrier after auditing its remaining lock users; 5. remove legacy readers, flags, fixtures, and operator instructions when the import window closes. Once floor publication and OpLog deletion begin, rollback requires a binary that understands the new pointer, snapshot, and floor protocol. Older binaries are not a supported rollback target. ## Future GET access OpLog After M3/M4 stabilize the new HA snapshot path, add a separate ordered OpLog event for successful object access. This is a follow-up protocol change and should not expand N06 or delay the first snapshot switch. The design must define: - **Event identity:** use stable `tenant_id` plus object key and the applicable Replica generation identity; failed reads do not advance recency. - **Ordering:** place the event in an ordered batch/sequence. Persist a defined wall-clock epoch timestamp for policy and diagnostics, with OpLog sequence as the deterministic tie-breaker. Never persist a raw process-local steady-clock value. - **Materialized state:** replay updates `last_access` and the new snapshot format carries the value required for post-recovery eviction without replaying every historical GET. - **Eviction policy:** choose the oldest `(last_access_time, last_access_sequence)` among eligible objects/Replicas while excluding hard pins, active leases, and non-terminal lifecycle states. - **Load control:** begin with every successful GET for correctness, measure OpLog/etcd amplification, and only then consider coalescing or sampling with an explicit recency-accuracy bound. - **Compatibility:** version the new field, define snapshots without access history, and keep legacy snapshot data outside this reader. Tests must cover new-node replay, equal/skewed timestamps, duplicate events, remove-and-recreate, hard-pin/lease exclusions, snapshot restore, floor pruning, and deterministic eviction after promotion. ## RFC alignment - [RFC #3167](https://github.com/kvcache-ai/Mooncake/issues/3167) defines standby-generated snapshots, dual pointers, floor-before-delete retention, and the direction to remove the primary snapshot writer. - [RFC #2979](https://github.com/kvcache-ai/Mooncake/issues/2979) defines explicit Object/Replica/AllocatedBuffer/Segment ownership, stable identity, primitive lifecycle events, and the conditional SegmentView path. - [RFC #3321](https://github.com/kvcache-ai/Mooncake/issues/3321) defines recovery leases and clock domains: runtime lease decisions use `steady_clock`, while persisted clock information requires an explicit versioned mapping. The v1 standby snapshot therefore does not persist lease deadlines or TTLs. ## Completion criteria The full roadmap is complete when HA mutations are fenced and fail-stop, restore and promotion are fail-closed and bounded, lifecycle and identity events are durable and replayable, metrics expose every recovery/capacity decision, optional codec/Segment migrations have evidence, legacy snapshot/lifecycle writers are retired, and GET access history yields deterministic recency-aware eviction after promotion. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

Roadmap

label

2026-10-29 21:21:29 +00:00

bsakaguchi commented

2026-10-29 21:21:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 1, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#3808
GitHub user ID	`102317026`
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 1, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #3808 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:21:30 +00:00

Author

Owner

@WEIXILIN0307 commented on GitHub (Sep 27, 2026):

Hi @Icedcoco, I'd like to take E02 — Active allocation count, provided it is still available.

I checked the roadmap discussion and related PRs and did not find an E02-specific claim or implementation. I also noticed that OffsetAllocator already maintains m_allocated_num, so I would reuse that rather than introduce another independently maintained counter in the buffer or Segment layer.

My proposed scope is:

Expose a consistent allocator-level active-allocation-count API, reusing the existing OffsetAllocator count and providing equivalent accounting for CacheLib.
Cover successful allocation/release and both restore paths. Rebinding handles to already-restored allocations must not increment the count again, and temporary gap allocations used during import must not remain in the final count.
Verify ownership moves, including assignment over a non-empty destination, without treating handle movement as a new allocation.
Add focused regression tests for allocation failures, restore/import, handle rebinding, moves, and concurrent allocation/release, plus a small before/after overhead check.

I would keep this limited to allocator accounting and its tests, preserving RPC/persisted formats and existing byte-accounting semantics. I would not introduce a new registry or change the ownership model. A zero count alone would not become a new unmount/reclamation gate; admission and lifecycle synchronization remain prerequisites.

I see E02 is listed after E01/E03, and #4222 is already implementing E01, including relevant move semantics. I'll coordinate with @0z5a rather than duplicate that work.

Could you confirm that E02 is available and the intended landing order relative to E03?

@WEIXILIN0307 commented on GitHub (Sep 27, 2026): Hi @Icedcoco, I'd like to take E02 — Active allocation count, provided it is still available. I checked the roadmap discussion and related PRs and did not find an E02-specific claim or implementation. I also noticed that OffsetAllocator already maintains `m_allocated_num`, so I would reuse that rather than introduce another independently maintained counter in the buffer or Segment layer. My proposed scope is: - Expose a consistent allocator-level active-allocation-count API, reusing the existing OffsetAllocator count and providing equivalent accounting for CacheLib. - Cover successful allocation/release and both restore paths. Rebinding handles to already-restored allocations must not increment the count again, and temporary gap allocations used during import must not remain in the final count. - Verify ownership moves, including assignment over a non-empty destination, without treating handle movement as a new allocation. - Add focused regression tests for allocation failures, restore/import, handle rebinding, moves, and concurrent allocation/release, plus a small before/after overhead check. I would keep this limited to allocator accounting and its tests, preserving RPC/persisted formats and existing byte-accounting semantics. I would not introduce a new registry or change the ownership model. A zero count alone would not become a new unmount/reclamation gate; admission and lifecycle synchronization remain prerequisites. I see E02 is listed after E01/E03, and #4222 is already implementing E01, including relevant move semantics. I'll coordinate with @0z5a rather than duplicate that work. Could you confirm that E02 is available and the intended landing order relative to E03?