mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #4666] [RFC]: [Store] Shutdown handoff: index-preserving planned restart for a standalone master #1186
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#1186
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Juhyun-Kim-Memphis on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4666
Changes proposed
Summary
A standalone (non-HA) Mooncake master loses its whole object index on every restart. In our deployment, almost every master restart is planned: an operator rolls a new image or changes a flag, and the process gets SIGTERM first.
This RFC proposes an opt-in shutdown handoff for that case. On SIGTERM, the old master stops serving, stops its background mutators, writes its final state to a local directory, and exits. The next master claims that state once, rebuilds the index and allocator state, and only then opens RPC.
The proposal rests on three properties:
--handoff_diris set.It cannot be enabled together with HA and does not change the HA path. We run it in a fork and would like to upstream it in small PRs (Section 7), after hearing the maintainers' view on the direction and the failure policy (Section 5).
Related: #3774, #3808, #3167, #3771, #4406.
1. Problem
After #4406, a standalone master has no restart path that keeps the index. The remaining path is HA: batch OpLog in etcd plus standbys (#3808, #3167), which requires
enable_hawith etcd. The removed legacy path did not keep the index in practice either: its default restore dropped objects whose lease had expired (#3761), and on245e7106it kept 0 of 25M keys.In #3774 (an HA deployment), @Icedcoco explained that the batch OpLog design rebuilds the index from durable etcd history, and that recovery must fail closed. We agree with both points. This RFC is about deployments that do not run HA.
What this costs us. One production cell has 12 stores and up to about 25M keys. In the 30 days before early October 2026, its master was replaced 6 times. All 6 were operator rolls, with no crash. Each roll dropped the whole index, and after one of them the cell took about 8 minutes to refill 17.3M keys. Because a master roll empties the cache, we also postponed rolling a master image that carried upstream fixes.
The restart that hurts is the one the operator starts on purpose, which is also the easiest one to make safe.
2. Proposal
Configuration.
--handoff_dir=<path>names a directory on the master node's local disk. It is empty by default, which disables the feature. Config validation rejects it together with--enable_habefore any worker or server starts. The first version supports only the OFFSET allocator; with any other allocator, the master logs the reason and starts empty.Shutdown (old master, on SIGTERM).
READYbyrename, and exit.Startup (new master).
flockon the directory. Refuse to restore if aPOISONmarker shows that two masters shared it.READYtoCLAIMEDbefore allocating anything, so each snapshot is restored at most once. Ignore snapshots older than--handoff_max_age_sec.Remount by segment identity. The memory tier needs no client or wire change. Stores call
ReMountSegmentafterNEED_REMOUNT, and theSegmentthey send already carries a per-mount random id and the base address.client_id(a random UUID per process), segment id, base, size, and endpoint all match, the segment opens. The check takes about 0.01 ms, whatever the key count.Today's
ValidateStandbyRemountSegmentcompares only name, endpoint, and capacity. That is why A3 adds segment id and base to the persisted segment description.Not carried, so each becomes a miss: in-flight puts (their ranges are quarantined), offload and promotion tasks, and Transfer Engine metadata. Stores re-publish their Transfer Engine metadata within 0.6–1.6 s.
3. Why this cannot serve wrong data
Principle: a missing entry is a cache miss; a wrong entry is silent corruption. Every failure path here ends in a missing entry.
#3771 was closed because a restored snapshot can point at memory that was reused after the snapshot, which was taken by a master that kept serving. The handoff rules this out by construction. Its snapshot is the final state of the only allocator. Nothing allocates while no master runs. The new master rebuilds the same allocator state before allocating anything, and restored entries open only after their store proves it has not changed.
put_start_release_timeout)READY, claimed before allocating;flock,POISON, maximum agedefault_kv_lease_ttl)We wrote the argument as a step-by-step proof that ties each step to the code enforcing it. AI-assisted review passes checked it against our fork's code. We fixed the gaps they found, and the last pass found none. The proof also lists what code cannot enforce. The main items are that a writer stops using an allocation within
put_start_release_timeout, and that both masters use the same timeouts Appendix §D.4. Evidence
Where these numbers come from. The handoff was measured on our fork implementation: a branch on upstream
245e7106, and its port to our production fork (vessl-ai/Mooncake#26). We have ported it to current upstreammain(1204367d) as the commit series in Section 7; each commit builds and passes its unit tests, but we have not re-run the rig on that port.Recreate, 4 CPU-only stores over TCP with L4 on. 12M keys plus remove/re-put churn, checked by a byte-exact oracle. One run.245e7106branch, same rig design. One run.245e7106branch, 14-CPU Xeon pod, no load. Median of 5.The handoff makes the master gap longer (13.8 s instead of 6.6 s in the same rig). In exchange, clients come back to a full cache, where a cold restart left our production cell refilling for about 8 minutes Appendix §A.
Why not HA for planned restarts? HA covers more, including crashes, but carries structural costs under any implementation: a 3-member etcd cluster to operate (compaction and quota are open as N13), a standby holding a full metadata copy, and an OpLog record on every PutEnd. Our measurement on current
mainalso showed its durability coupled to the steady-state path through eviction's use of the OpLog writer Appendix §B. A deployment that needs only planned restarts pays for that all the time; the handoff costs nothing outside the restart.5. Failure policy
In #3774, the maintainers said that silently serving an empty or partially reconstructed index "is not an acceptable fallback." They also said that dropping a descriptor needs explicit identity, ownership, and durability semantics. Our answer:
Baseline. A standalone master already serves an empty index after every restart. A failed handoff returns to exactly that state, so it adds no new fallback.
Durability. The deployment guide calls empty serving a bug in HA because it disagrees with the durable OpLog and poisons later recovery. A standalone master has no durable history to disagree with. The handoff file is not durable state: it has one producer (the quiesced old master), at most one consumer (the next master, which claims it before allocating), and no reader after the claim.
Never a wrong index. Restore is all or nothing for each snapshot. Dropping is all or nothing for each segment, and it means the same as that store restarting today. Upstream already skips replicas on unloaded segments in standby restore (#4553).
Identity and ownership. An entry belongs to a segment, and a segment belongs to a store process (identity as in Section 2). Store memory is written only into ranges that the master allocated. So the same process with the same mount still holds the recorded bytes, apart from the quarantined ranges. Anything else is treated as possibly reused.
Not silent. Every outcome has a result and a reason in the metrics (
master_handoff_last_snapshot_result,master_handoff_restore_result,master_handoff_segments_{confirmed,dropped,pending}) and in the logs.Operator choice (proposal).
--handoff_failure_policy:serve_empty(default): start empty, as today, and report why.refuse: if a handoff was expected but cannot be used, stay non-serving like the R02 gate until the operator clears the directory. "Expected but unusable" means aREADYthat is invalid, too old, or over budget, or a failed write recorded at shutdown.We are happy to make
refusethe default, or to require an explicit choice, if maintainers prefer.6. Relation to the HA roadmap (#3808)
245e7106, about 195 s of a 279 s snapshot restore was spent inRestoreFromStandbyState's per-chunk loop, which promotion also runs. A shard-parallel restore read the same file in 11.2 s. An identity-only remount takes about 0.01 ms, where the key scan took 26.5 s per store (#4291 is open) Appendix §C.7. Delivery plan
Each PR builds and passes tests on its own. We will split further wherever reviewers ask.
StandbySegmentInfo; identity-only remountQuiesce()that stops and joins background mutators, separate from the destructorREADY/CLAIMEDlifecycle, restore wiring, failure policy, metrics. Default off.LOCAL_DISK) state in the handoffTrack B (prerequisite for A6 only): a separate bug report and fix for wrong reads in the existing L4 offload path. We reproduced these without any master restart.
Separately from this RFC, we will file the observations in Appendix §F as their own issues with reproductions.
8. Non-goals
Crashes and OOM kills (HA's job), zero-downtime restarts (a gap of about 15 s remains), any change to the HA paths, the memory of a store that restarts during the gap, and allocators other than OFFSET.
9. Questions for maintainers
serve_emptyacceptable as the standalone default, or do you preferrefuse, or an explicit choice?StandbySegmentInfo, which HA shares, or into a handoff-only manifest? Does this touch the roadmap's V-series?BatchOpLogSnapshotWriterand the N03 manifest, so that one reader serves both paths?default_kv_lease_ttl. Does this meet the rule for persisted time fields (#3808 invariant 6, #3321)? Or should they start without a lease, protected by the remount gate?AI assistance disclosure
Claude Code helped with the design, the fork implementation and tests, the rig runs, the proof review passes, and this draft. The submitter checked every number against the source data and is responsible for defending the design and, in each PR, every changed line.
Before submitting a new issue...
@Juhyun-Kim-Memphis commented on GitHub (Oct 15, 2026):
Appendix to the shutdown handoff RFC
This appendix holds the measurement details, the full list of safety mechanisms, and a summary of the correctness argument. The RFC body is self-contained; read this only for the details.
All handoff numbers were measured on our fork implementation: a branch on upstream
245e7106, and its port to our production fork. The port to current upstreammainwill be verified with unit tests; the rig was not re-run for it.Commits named here:
b140aa92: upstreammainon 2026-10-14. Used for the HA cost measurement (§B) and the restore path comparison (§C).245e7106: upstreammainthat our first implementation was built on. Used for the scale benchmarks (§A.3).145c50db,778aeb07: our implementation on top of245e7106.Contents:
§A Handoff measurements
A.1 Production baseline (cold restart, before the handoff)
L4 context, which belongs to track B: after the 2026-09-10 restart, the stores re-registered their disk keys all at once, from 0 to 24M keys in 40 s. The engines saw tens of thousands of RPC failures during that time.
A.2 End-to-end rolls on Kubernetes
Rig.
Recreateand a hostPath handoff directory. 4 CPU, 32 GiB. Flags--client_ttl=60 -enable_offload=true.Production fork port (vessl-ai/Mooncake#26 lineage).
--handoff_dir)In the handoff roll, the old master exported 5,191,681 objects. The new master claimed the snapshot once and confirmed all 4 segments, with 0 dropped. No store saw
SEGMENT_NOT_FOUNDor re-mounted, and no store expired.Build
145c50dbon top of245e7106.Note: the control's 92.31% is a liveness cost of a time-based offload hold that this build used after a cold start. Objects put during that hold had no disk copy, so eviction removed them. The later L4 task-id change (track B) removed the hold.
Earlier finding from the same rig (build
828b43d7). Before the L4 fixes, wrong reads appeared even without any roll: 8 in 4.05M gets (about 2 per million) under remove/re-put churn. After a cold restart there were 33,209, because the stores re-registered their disk keys. We traced every observed wrong read to a cause in the existing L4 offload path:This is the subject of track B. With the track B fixes, every phase above has 0 wrong reads.
A.3 Scale (25M keys, 12 segments)
Hardware and method. CPU-only pod with 14 CPU and 80 GiB, Intel Xeon 8462Y+, cgroup v2, no concurrent load. The base was upstream
245e7106. The synthetic keys are about 1,030 B of metadata per key; production is about 1,250 B.MountSegment9.5 ms; background deletion 16.6 sReal
mooncake_masterbinary, 25M keys. Production flags, an index published by a test tool on 12 fake segments, 3 runs.A crash during the restore led to a cold start on the next start (
aborted_previous), 1.06 s later.With L4 state (track B / A6, for reference). 25M memory keys plus 24M disk keys.
The admission model is
(memory objects × 1,144 B + extra disk replicas × 184 B + disk-only objects × 904 B) × 1.15. It was 15–17% above every measured peak in the cases above.§B HA steady-state cost (M3)
This section backs one sentence of the RFC: on current
main, HA's durability is coupled to the steady-state path. It is not an argument that HA is slow. Most of the cost below comes from an implementation detail of eviction, which can be fixed. We will report it separately (§F).B.1 Setup
mainb140aa92, Release,STORE_USE_ETCD=ON. Both modes ran the same binary; only the flags differed.docs/source/deployment/mooncake-store-deployment-guide.md:enable_ha, 3 etcd endpoints,enable_oplog,oplog_poll_interval_ms=1000,oplog_batch_max_entries=1024,enable_oplog_snapshot(600 s, local object store).--client_ttl=60 -enable_offload=true, eviction defaults (high watermark 0.90, ratio 0.05).master_bench.cppthat calls only master RPCs. 12 fake segments × 675 GiB (8.7 TB, the same as production). Fill: 16.4M objects of 470,000 B at 40k items/s (88.6% full). Measured phase: 360 s of put 9,300 items/s (64 per request) and get 64,000 items/s (128 per request), both at a constant rate. This is our 7-day production median.B.2 Code reading (
b140aa92, paths undermooncake-store/)AppendOpLogVisibleBeforeDurable)src/master_service.cpp:5756-5764)REMOVEDand freed in the durable callbackBatchEvict):11150-11157,:11268-11335):11284-11291)The writer has one etcd transaction in flight at a time. While it waits, it accepts up to
oplog_batch_max_entries(default 1,024) reserved entries (src/ha/oplog/ordered_oplog_writer.cpp:279-300). PutEnd, Remove, and eviction share these entries.Two more facts from the code:
ha_oplog_dropped_put_end_totalis defined but never incremented (src/ha_metric_manager.cpp:128).offload_on_evictbranch is bypassed (src/master_service.cpp:11205-11207).B.3 Results
The latency rows are the per-second p50 and p99 over t = 5–140 s, the window before HA memory filled up.
Before and during eviction. Each cell is the range over 3 runs, in µs.
B.4 Interpretation
The OpLog record itself is cheap. Before eviction starts, only PutEnd is slower, by about 0.33 ms. This is serialization inside the shard lock plus the writer mutex. Nothing waits for etcd.
The large cost comes from how eviction interacts with the reservation.
BatchEvict: OpLog reservation failed ... stopping eviction scan.Raising
oplog_batch_max_entries. A larger value raises the limit. At about 316 B per entry, the default etcd request limit of 1.5 MiB allows about 4,900 entries. We estimate that this would give about 2,400 evictions/s, still below the 9,300 puts/s. We did not measure this.Skipped PutEnd records happen in the same seconds as eviction cycles. During the 40k items/s fill, 6–7% of PutEnd records were skipped (1.0–1.19M).
What would remain if eviction were fixed:
B.5 Pre-registered criteria
B.6 Limits
MOONCAKE_ENABLE_OPLOG_PERF_METRICS=ON, which adds one histogram observation per entry in HA.oplog_batch_max_entriesor the production peak rate (22k items/s).§C Restore path cost (M4)
Restoring a batch snapshot file at 25M keys (upstream
245e7106, same hardware as §A.3, median of 3):StandbyMetadataStorefill, thenRestoreFromBatchOpLogPromotion, 1 threadWhere the 279 s goes.
RestoreFromStandbyState, which also runs at promotion. That loop takes a per-object shard lock, builds a per-chunk hash map, and inserts every replica into a per-segmentstd::map.StandbyMetadataStore.DrainChunk.ValidateReplicaIds.Changes on
mainsince245e7106. Three commits touch the promotion restore: #3806, #4553, and #4559. We timed the restore call alone at 10M keys on both commits. This used the same input and a small harness that runsStandbyMetadataStoreand thenRestoreFromBatchOpLogPromotion, on one AMD EPYC 9454P node (5 CPU), alternating 3 runs per commit.245e7106b140aa92The extra memory matches two things: #4559 keeps the standby copy (3.8 GiB) until serving starts, and #3806 keeps
installed_keysfor rollback. So the 279 s at 25M is conservative formain, and the 38.2 GiB is now an underestimate. We did not re-measure at 25M.Remount scan. On
245e7106, oneReMountSegmentafter a restore took 26.5 s at 25M keys, and 317 s for 12 stores in sequence. The call holdsclient_mutex_and the snapshot mutex exclusively. #4291, which would change this, is still open. With segment identity persisted and the allocator rebuilt at restore, the remount is an identity comparison of about 0.01 ms (§A.3).§D Safety mechanisms and correctness argument
D.1 Mechanisms
discarded_replicas_), for the remaining hold time (up toput_start_release_timeout)READY; rename it toCLAIMEDbefore any allocation; lock file;POISONif two masters share the directory; maximum age (120 s by default)client_idmatch. Otherwise every restored entry of the segment is dropped. Restored disk replicas stay closed until an RPC from the sameclient_idarrivesdefault_kv_lease_ttl)D.2 Proof summary
The claim: every replica descriptor that the master hands out for key
Kleads a reader who follows the read protocol to the bytes written by the put that created the current object, or to an error (a miss). Anything else counts as wrong: an older or newer value, another key's bytes, or a removed key coming back.Handoff steps (memory replicas).
BackgroundWorkQuiesced(), so the export sees one consistent stateS, and every put or remove acknowledged to a client happened beforeS.Sor quarantined.Segmentopens it.Replica::next_id_continues from the old master's value.Assumptions that code cannot enforce:
put_start_release_timeout.default_kv_lease_ttl.put_start_release_timeout.The L4 part of the proof (steps L0–L7, the task-id uniqueness assumption, and the mixed-version analysis) belongs to track B and A6. HA mode is out of the proof's scope; the handoff refuses to run with HA.
Verification history.
145c50db, found no remaining hole. The fork port (35ef7acb) has its own proof revision and verification.§E Failure cases and observability
Tested with the real binary and real stores over TCP. Every case below started empty and safely, with 0 wrong reads:
Further tested cases.
Metrics:
master_handoff_last_snapshot_result,_bytes,_objects,_workersmaster_handoff_last_shutdown_duration_secondsmaster_handoff_restore_result,_duration_seconds,_objectsmaster_handoff_snapshot_age_secondsmaster_handoff_segments_confirmed,_dropped,_pendingLog lines:
handoff_shutdown=start|serving_stopped|quiesced|deadline|...handoff_write=ok|...handoff_startup=ok|claim_failed|poison_failed|deadline|...Result reasons include:
skipped_restore_memory,skipped_processing,skipped_not_owner,skipped_poisoned,stale,deadline,poisoned.Flags in the fork. Upstream names and defaults are open (RFC Q2).
--handoff_dir""(off)--handoff_shutdown_deadline_sec--handoff_restore_deadline_sec--handoff_max_age_sec--handoff_max_workers--handoff_restore_workers--handoff_restore_margin_percent§F Upstream observations we plan to report separately
We will file each of these separately from this RFC discussion, as its own issue with a reproduction. They are bug reports, not arguments in favor of the handoff. They are listed here only so that maintainers know they are coming.
Status on 2026-10-22: items 1–3 and 5 are reported as linked below. Items 2 and 3 were already under discussion elsewhere, so we added our data to those threads instead of opening new issues. Item 4 is addressed by #4802 instead of a separate issue.
ha_oplog_dropped_put_end_totalis defined but never incremented. A skipped PutEnd record is visible only in a warning log. #4546 already covers this counter, and #4582 would delete it; we added our data there: https://github.com/kvcache-ai/Mooncake/issues/4546#issuecomment-6308811788offload_on_evictwith OpLog. Theoffload_on_evictbranch is bypassed when OpLog is enabled. We may have missed a documented reason. This was already raised in #4723; we added our reading there: https://github.com/kvcache-ai/Mooncake/pull/4723#issuecomment-6308812084ValidateStandbyRemountSegmentcompares segment name, endpoint, and capacity only. We have not demonstrated a failure on the promotion path. A3 proposes adding segment id and base; this is now #4802.mainat8c19b728: SIGTERM to exit took 20.7 s with 12M objects and 48.2 s with 25M objects (median of 3 runs; after SIGKILL the kernel's teardown took 1.1 s and 2.4 s). Reported in #4856. The first measurement (21.9 s with a 12M-object index) was a single run on build828b43d7, in the cold-restart phase.§G Unplanned failovers observed in the M3 rig (facts only)
This section depends on our etcd configuration: 5-minute periodic compaction, chosen by us. We do not draw conclusions about rates from it.
@github-actions[bot] commented on GitHub (Oct 15, 2026):
Thanks for opening this issue, @Juhyun-Kim-Memphis!
32131411A maintainer will triage this when possible. To help us respond faster, please include:
Useful links: Documentation · Contributing guide
@Juhyun-Kim-Memphis commented on GitHub (Oct 20, 2026):
The implementation is now up for review as three PRs: #4802 (parallel promotion restore and a segment identity check on remount, useful for HA on its own), #4804 (the shutdown handoff itself, including its L4 state), and #4687 (the L4 wrong-read fix that the L4 part depends on). #4804 is open for review in parallel with the discussion here; the failure policy in Section 5 is still the main open question for it. I would be glad to hear the maintainers' view on that and on the questions in Section 9 whenever you have time.