[GH-ISSUE #2756] [RFC]: OpLog Batch Record Ordered Writer and Durable Prefix #719

New issue

Closed

opened 2026-10-29 21:18:39 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented 2026-10-29 21:18:39 +00:00

Owner

Copy link

Originally created by @Icedcoco on GitHub (Jul 6, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2756

Originally assigned to: @Icedcoco on GitHub.

Introduction

Mooncake Store HA should do more than restart the Master process quickly. After a primary failure and failover, the new Master should preserve as much useful KV cache metadata, segment registry state, and object-location index state as possible.

The existing HA work has two complementary recovery paths:

1. Snapshot provides a coarse-grained baseline for restoring complete Master state.
2. OpLog provides incremental changes after the snapshot, so a standby can keep following the primary and catch up before promotion.

RFC #1200 proposed an etcd-backed hot-standby mechanism for Master metadata HA. The primary writes metadata mutations as OpLog entries, and the standby reads/watches those entries from etcd and applies them locally. RFC #1650 then introduced an OpLogStore / OpLogChangeNotifier abstraction to decouple the upper-layer replication logic from the etcd-specific implementation.

PR #2331 is the immediate baseline for this RFC. Although that PR was submitted by @Icedcoco, most of the HA hot-standby and OpLog implementation came from earlier work by @Libotry and @00fish0 in the dev/snapshot_etcd_storage_integration_testing_fixbug branch. The work in #2331 was mainly to rebase that legacy implementation onto the current kvcache-ai/Mooncake:main, adapt it to the current Mooncake Store architecture, and integrate recently added Store features such as tenant-aware object metadata, segment lifecycle replication, and the updated standby promotion / snapshot handoff paths.

This RFC builds on #2331 and proposes the next step for the OpLog write and replication protocol: batch records, an ordered writer, and a durable prefix. The goal is not to replace the business semantics that #2331 already adapted. Instead, this RFC keeps the current logical OpLogEntry semantics and replaces the underlying write and replay protocol to address lock-held durable waits, per-entry backend key amplification, and promotion boundary ambiguity.

Relevant links:

• Roadmap: https://github.com/kvcache-ai/Mooncake/issues/1035
• Hot Standby RFC: https://github.com/kvcache-ai/Mooncake/issues/1200
• OpLog Store Abstraction RFC: https://github.com/kvcache-ai/Mooncake/issues/1650
• Snapshot RFC: https://github.com/kvcache-ai/Mooncake/issues/1150
• etcd wrapper for HA: https://github.com/kvcache-ai/Mooncake/pull/1451
• initial hot-standby / OpLog implementation: https://github.com/kvcache-ai/Mooncake/pull/1515
• HA backend abstraction: https://github.com/kvcache-ai/Mooncake/pull/1678
• Redis leadership backend: https://github.com/kvcache-ai/Mooncake/pull/1722
• HA runtime / standby lifecycle refactor: https://github.com/kvcache-ai/Mooncake/pull/1777
• current rebase and adaptation PR: https://github.com/kvcache-ai/Mooncake/pull/2331

Background

Roadmap Context

The Mooncake Store V3 Roadmap (#1035) tracks HA under Milestone 2: Master Service Enhancements, specifically Metadata Adaptation - HA. That item has three related lines of work:

• Snapshot: #1150, #1431, #1465, #1381 and related work provide persistence and recovery for Master metadata, segment state, allocator state, and task-manager state.
• OpLog: #1200, #1451, #1515 and related work provide incremental metadata replication after a snapshot baseline.
• HA refactoring: #1648, #1678, #1722, #1777 and related work move leader coordination, standby runtime, snapshot bootstrap, and OpLog following into clearer runtime boundaries.

The relationship is straightforward: snapshot restores a complete baseline, OpLog catches the standby up from that baseline to a later primary state, and the HA runtime orchestrates snapshot bootstrap, OpLog following, promotion, and serving.

Snapshot Baseline

RFC #1150 was motivated by a practical limitation of early HA mode: fast Master restart alone does not recover KV cache location metadata. If all metadata is lost after failover, upper-layer inference must rebuild cache state, and the recovery time can be much longer than the Master process restart time.

PR #1431 implemented Master snapshot and restore:

• fork + copy-on-write to reduce blocking during snapshot creation;
• background serialization and compression of metadata, segment state, allocator state, and task-manager state;
• local filesystem and S3 snapshot object stores;
• restore-time cleanup of incomplete metadata, non-ready segments, and expired leases.

PR #1465 added task-manager serialization/deserialization. PR #1381 designed an etcd snapshot object store so that snapshot data can also use the HA etcd deployment.

Snapshot is still periodic full-state persistence. It cannot recover mutations that happen after the latest snapshot. OpLog is therefore needed to reduce that loss window and let the standby catch up before promotion.

RFC #1200: Initial OpLog Hot Standby

RFC #1200 proposed using etcd as a reliable intermediate component for primary-standby OpLog synchronization:

• On the primary, OpLogManager assigns a globally increasing sequence_id and creates OpLogEntry records for metadata mutations.
• EtcdOpLogStore persists each entry to etcd, for example under /oplog/{cluster}/{sequence_id}.
• /oplog/{cluster}/latest records the latest sequence.
• On the standby, OpLogWatcher receives real-time changes through etcd watch and reads historical entries during startup or reconnect.
• OpLogApplier checks ordering and applies entries to standby metadata.
• HotStandbyService manages standby lifecycle and promotion.

The main design points were correctness and RPO/RTO: global sequence ordering, etcd persistence, etcd watch, snapshot sequence recording, and OpLog cleanup.

PR #1451 and PR #1515: Early Implementation

PR #1451 wrapped the basic etcd interface needed by Master HA.

PR #1515 implemented the early hot-standby and OpLog interface:

• EtcdOpLogStore
• OpLogManager
• OpLogWatcher
• OpLogApplier
• HotStandbyService
• StandbyStateMachine
• OpLog types such as PUT_END, PUT_REVOKE, and REMOVE
• unit tests for standby metadata store behavior, gap handling, watch/reconnect, and promotion

This moved the OpLog design from RFC into a runnable implementation. However, the early code was tightly coupled to etcd, and the Mooncake Store metadata schema, tenant model, segment lifecycle, and promotion runtime have changed since then.

RFC #1650: OpLog Store Abstraction

RFC #1650 addressed the etcd coupling in the early OpLog subsystem. Several upper-layer components directly held or constructed EtcdOpLogStore, while the old watcher mixed watch, reconnect, deserialization, and apply orchestration in one class.

The RFC proposed:

• an OpLogStore interface for write, read, read-since, latest-sequence, snapshot-sequence, and cleanup operations;
• an OpLogChangeNotifier interface for backend-specific notification;
• an OpLogReplicator that replaces the monolithic etcd-centric watcher;
• shared OpLog serialization;
• centralized store creation through a factory;
• mock stores/notifiers for deterministic unit tests.

That RFC intentionally kept behavior unchanged. It extracted interfaces and clarified boundaries, but did not change the legacy per-entry OpLog layout or logical entry semantics.

HA Runtime Refactoring

PR #1678 introduced an HA backend abstraction and moved leader coordination from the legacy etcd helper path to LeaderCoordinator. It also kept OpLogStore and SnapshotStore as backend boundaries for follow-up work. PR #1722 added a Redis leader coordinator, proving that the leadership path can be backend-independent.

RFC #1648 discussed Kubernetes Lease-based HA. That work is about leader coordination and master-view discovery. It does not by itself provide OpLog durability or replay semantics.

PR #1777 split the always-on Master process runtime from the leader-only serving runtime, introduced standby-controller composition, and made snapshot bootstrap plus OpLog following part of the standby runtime. This is the runtime foundation needed for switching the standby between legacy per-entry replay and batch-record replay.

PR #2331: Current Foundation

PR #2331 ([Store] add HA OpLog replication and tenant-aware standby promotion) rebased and adapted the earlier HA hot-standby work to current main. The key changes include:

• HA OpLog configuration: oplog_store_type, oplog_store_root_dir, and oplog_poll_interval_ms;
• primary-side OpLog publishing for Store metadata mutation paths;
• durable OpLog guarding for HA-enabled mutations where dropping the update would make primary and standby diverge;
• SEGMENT_MOUNT, SEGMENT_UNMOUNT, and SEGMENT_UPDATE OpLog entries for standby segment registry replay;
• tenant_id in OpLogEntry, tenant-aware metadata store APIs, and tenant-aware standby snapshot/restore paths;
• backward-compatible OpLog deserialization when old entries do not contain tenant_id;
• standby snapshot export and promoted-primary restore for object metadata, group membership, segment state, and the OpLog sequence baseline;
• tests for OpLog apply, standby promotion, tenant isolation, segment lifecycle replay, and MasterService HA mutation behavior.

After #2331, current main has a much more complete OpLog HA baseline. The remaining issue is the storage and write model: the legacy path still stores one backend key per logical OpLogEntry, and /latest remains the latest-sequence pointer. EtcdOpLogStore::WriteOpLog(entry, true) already uses an internal batch writer, so the business thread is not directly writing each single key to etcd. But from the Master mutation path, synchronous waiting can still happen while holding snapshot_mutex_ in shared mode and a metadata shard lock in exclusive mode.

Motivation

The current #2331-based HA OpLog path covers many business cases, but the write protocol still has several structural problems.

1. Lock-held Durable Wait

Several paths can wait for durable OpLog persistence while holding snapshot_mutex_ and a metadata shard lock:

• Remove()
• BatchRemove()
• RemoveAll()
• BatchReplicaClear()
• BatchEvict()
• NoFBatchEvict()
• BatchEvictDiskReplica()
• stale-handle cleanup
• expired processing/replication cleanup
• segment lifecycle updates
• offload/promotion success paths

The current sync write path enqueues entries into an internal pending batch and waits for the flush result. etcd latency, batch-flush delay, retry, or sync timeout can therefore block unrelated metadata mutations in the same shard.

2. Per-entry Backend Key Amplification

The legacy layout is:

/oplog/{cluster}/{sequence_id}
/oplog/{cluster}/latest
/oplog/{cluster}/snapshot/{snapshot_id}/sequence_id

Each logical OpLog entry becomes one backend key. Under high mutation concurrency, the backend must handle many keys, watch events, MVCC items, and compaction items. Internal group commit reduces the number of flush calls, but it does not reduce the number of persisted keys.

3. Split Safety Boundary

In the legacy model, an entry key means the entry exists, while /latest means the primary has advanced the latest sequence pointer. For standby apply and promotion, the important concept is a verified, continuous, no-gap durable boundary.

Batch-record mode should make this explicit. A batch record and the durable boundary should advance together through a CAS/txn protocol, and standby/promotion should trust only that durable boundary.

4. Need for a Post-#2331 Evolution Path

#2331 adapted the legacy HA OpLog implementation to current main and added tenant, segment, and promotion compatibility. The next step should not rewrite those semantics. It should keep:

• current logical OpLogEntry semantics;
• current PUT_END, REMOVE, and SEGMENT_* apply behavior;
• tenant-aware metadata;
• standby snapshot export and promoted-primary restore;
• legacy compatibility during upgrade.

The part to replace is the underlying durable write and replay protocol.

Goals

1. Replace new per-entry backend keys with batch records. One batch record is one backend KV value containing multiple logical OpLogEntry records over a continuous sequence range.
2. Use {batch_id, last_seq} as the durable prefix and the only safe boundary for callbacks, standby apply, and promotion.
3. Move OpLog admission, sequence assignment, batching, retry, and durable callbacks out of metadata shard lock hold time, or reduce the lock-held path to a short enqueue.
4. Keep logical sequence globally monotonic and continuous.
5. Keep the existing logical meaning of OpLogEntry.
6. Support mixed reading of legacy per-entry OpLog and new batch-record OpLog in the same cluster, without double-writing.
7. Require txn/CAS for the first version. Non-txn backends remain a future extension and must not be claimed as HA-correct in this version.
8. On backend failure, fail closed: stop new write admission, do not skip the stuck batch, do not report success before durability, and retry until backend recovery or leadership change.

Non-Goals

1. Do not introduce a new delta OpLog schema in the first version. Batch entries still use the current full OpLogEntry format.
2. Do not double-write legacy per-entry OpLog and batch-record OpLog.
3. Do not require old standbys to read new batch records. Operators must upgrade standby/readers first, then switch primary to batch-record mode.
4. Do not claim HA correctness for non-txn backends in the first version.
5. Do not put leadership lease/watch/election semantics into the minimal KV backend interface.
6. Do not fully redesign UpsertStart in this RFC. That path needs a separate design because its semantics are more complex.

Proposal

1. Architecture

Introduce three layers:

class OrderedOpLogWriter;
class OpLogBatchStorage;
class HaKvBackend;

OrderedOpLogWriter

OrderedOpLogWriter is the only OpLog admission, sequence assignment, batching, retry, and callback-dispatch entry point in a serving primary.

Business code uses:

1. Reserve() before metadata mutation.
2. Commit() inside the metadata shard lock to attach the final OpLogEntry and durable callback.
3. Abort() if the precondition check fails or no OpLog is needed.

The writer owns:

• writer health and accepting;
• open waiting-batch slot accounting;
• logical sequence assignment;
• work-conserving batch seal;
• single in-flight writing batch;
• backend txn/CAS write;
• durable prefix advancement;
• retry and unknown-outcome handling;
• callback dispatch in logical sequence order.

OpLogBatchStorage

OpLogBatchStorage is a layout/storage helper between the writer and the KV backend. It owns:

• batch-record key construction;
• durable-prefix key construction;
• batch-record encoding/decoding;
• durable-prefix encoding/decoding;
• checksum validation;
• etcd txn/CAS write;
• range reading of batch records;
• legacy /latest read for cutover initialization.

It does not own admission, sequence assignment, retry policy, callback dispatch, or Master business semantics.

HaKvBackend

HaKvBackend is a reusable HA KV adapter layer. It provides only generic KV/range/transaction capabilities. It does not know about OpLog, durable prefixes, leadership, snapshot, or metadata semantics.

The first production implementation should be an etcd txn/CAS backend. Future leadership reuse should use sidecar interfaces such as HaKvLeaseBackend or HaKvWatchBackend, instead of adding lease/watch/election methods to the minimal KV interface.

2. Core Data Structures

struct DurablePrefix {
    uint64_t batch_id{0};
    uint64_t last_seq{0};
};

struct OpLogBatchRecord {
    uint32_t schema_version{1};
    uint64_t batch_id{0};
    uint64_t first_seq{0};
    uint64_t last_seq{0};
    std::vector<OpLogEntry> entries;
    uint32_t checksum{0};
};

DurablePrefix means every logical entry in [1, last_seq] is durable and safe to apply/promote from. batch_id identifies the physical batch record that contains last_seq.

OpLogBatchRecord must satisfy:

• entries is non-empty;
• entries are sorted by increasing sequence_id;
• first_seq == entries.front().sequence_id;
• last_seq == entries.back().sequence_id;
• sequence IDs are continuous inside a batch;
• sequence IDs are continuous across batches;
• batch_id is monotonic;
• checksum covers the encoded batch content except the checksum field itself.

3. Minimal KV Backend Interface

struct KvPair {
    std::string key;
    std::string value;
};

enum class KvCompareKind {
    kValueEquals,
    kKeyNotExists,
};

struct KvCompare {
    std::string key;
    KvCompareKind kind{KvCompareKind::kValueEquals};
    std::string expected_value;
};

struct KvTxn {
    std::vector<KvCompare> compares;
    std::vector<KvPair> puts;
};

class HaKvBackend {
public:
    virtual ~HaKvBackend() = default;

    virtual ErrorCode Get(std::string_view key, std::string& value) = 0;
    virtual ErrorCode Put(std::string_view key, std::string_view value) = 0;
    virtual ErrorCode Range(std::string_view begin_key,
                            std::string_view end_key,
                            size_t limit,
                            std::vector<KvPair>& kvs) = 0;
    virtual bool SupportsTxn() const = 0;
    virtual ErrorCode Txn(const KvTxn& txn) = 0;
};

Batch-record HA initialization must check SupportsTxn(). If the backend does not support txn/CAS, the first version should fail initialization or return unsupported.

4. Storage Layout

Keep legacy keys unchanged:

/oplog/{cluster}/{seq_padded}
/oplog/{cluster}/latest
/oplog/{cluster}/snapshot/{snapshot_id}/sequence_id

Add batch-record keys under the same prefix:

/oplog/{cluster}/batches/{batch_id_padded}
/oplog/{cluster}/durable_prefix

Use fixed-width decimal encoding for batch_id, preferably the same 20-digit padding used by legacy sequence keys. This keeps lexicographic range scan order equal to numeric order.

Since the new keys remain under /oplog/{cluster}/, readers must explicitly filter:

/latest
/snapshot/
/batches/
/durable_prefix

5. Durable Prefix Value

/oplog/{cluster}/durable_prefix should contain at least:

schema_version=<version>
batch_id=<id>
last_seq=<seq>
checksum=<optional>

The actual encoding can use an existing project serialization format, but it must include schema_version so later versions can add leader epoch, term, primary id, or layout version.

Semantics:

• [1, last_seq] is the only safe apply/promotion boundary.
• batch_id locates the batch containing last_seq.
• standby and promotion must use durable prefix, not /latest, as the safety boundary.
• batch-record mode does not write new /latest values.
• legacy /latest is used only for cutover initialization and legacy reader compatibility.

6. Cutover and Legacy Compatibility

Upgrade order:

1. Upgrade standby/readers so they can read both legacy per-entry OpLog and batch-record OpLog.
2. Switch the primary to batch-record writer.
3. After the switch, the primary stops writing new legacy per-entry keys and stops updating /latest.

If a batch-record writer starts and durable prefix does not exist:

legacy_latest = Read(/oplog/{cluster}/latest)
durable_prefix = { batch_id = 0, last_seq = legacy_latest }
next_sequence = legacy_latest + 1
next_batch_id = 1

Create durable prefix with CAS/create-if-absent:

compare durable_prefix key does not exist
put durable_prefix = { batch_id = 0, last_seq = legacy_latest }

If the compare fails, another initializer has created the durable prefix. The current writer must read the existing prefix and use it as the source of truth.

This design intentionally trusts /latest during cutover:

• do not scan legacy max sequence;
• do not check for legacy keys with seq > latest;
• if such keys exist, they are treated as outside the new durable boundary.

This is an operational tradeoff. It assumes the old write path has been stopped during cutover, or that /latest is the accepted legacy safety boundary.

7. Writer Admission Protocol

Add:

oplog_batch_max_entries = 1024

This limits the number of entries in the current open waiting batch. It is an admission/backpressure limit, not a target batch size.

Business code uses:

tl::expected<Reservation, ErrorCode> Reserve();

tl::expected<PendingHandle, ErrorCode> Commit(Reservation&& reservation,
                                              OpLogEntry entry,
                                              DurableCallback callback);

void Abort(Reservation&& reservation);

Reserve

Reserve() runs before metadata mutation:

if (!accepting) return OPLOG_UNAVAILABLE;
if (open_waiting_reserved_or_committed >= oplog_batch_max_entries)
    return QUEUE_FULL;
++open_waiting_reserved_or_committed;
return Reservation{};

Reserve() does not allocate a sequence ID. Therefore, Abort() does not create a gap.

Commit

Commit() runs inside the metadata shard lock, but must stay short:

• consume an existing reservation;
• assign the next continuous sequence_id;
• fill the entry;
• append it to the waiting queue;
• attach the durable callback;
• notify the writer;
• do no backend I/O;
• do not wait for queue space;
• do not serialize the full batch record.

If the caller already has a reservation and the entry uses a known serializable OpLog format, normal Commit() should not fail.

Abort

Abort() releases a reservation if the precondition check fails or no OpLog is needed:

• release the slot;
• do not allocate sequence;
• do not create a gap;
• do not trigger callbacks.

8. Batch Seal and Flush Policy

The first version uses a work-conserving single in-flight writer:

1. If writing_batch is idle and waiting_queue has entries, seal the current open waiting batch immediately.
2. Even one entry is enough to write a batch.
3. If writing_batch is in progress, business threads can keep reserve/commit into a new open waiting batch.
4. After writing_batch succeeds:
   - if waiting batch has entries, seal and write the next batch immediately;
   - otherwise sleep until commit notification.

Implications:

• oplog_batch_max_entries is a backpressure limit, not a flush threshold.
• Low traffic may produce one-entry batches.
• High traffic or slower backend writes will naturally coalesce entries.
• The design optimizes latency first and does not intentionally wait to fill a batch.

Seal does not wait for reservations that have not yet committed or aborted. Already committed entries can be drained and sealed. Slow business threads keep their reserved slots and their later commits go into a later seal.

9. Sequence and Batch ID

Logical sequence is assigned in Commit():

next_sequence starts from durable_prefix.last_seq + 1
Commit() assigns current next_sequence
next_sequence++

Guarantees:

• Abort() creates no gap;
• committed entries are continuous;
• entries inside a batch are continuous;
• batches are continuous;
• standby strict no-gap validation can hold.

batch_id is assigned at seal time:

next_batch_id starts from durable_prefix.batch_id + 1
seal waiting batch assigns current next_batch_id
next_batch_id++

batch_id does not replace logical sequence. It locates batch records, describes the physical durable prefix position, supports standby range reads, and lets promotion verify batch continuity.

10. Backend txn/CAS Protocol

For one writing batch:

expected_prefix = current durable prefix before this batch
new_prefix = { batch_id = batch.batch_id, last_seq = batch.last_seq }

txn:
  compare /oplog/{cluster}/durable_prefix == Encode(expected_prefix)
  put /oplog/{cluster}/batches/{batch_id_padded} = Encode(batch)
  put /oplog/{cluster}/durable_prefix = Encode(new_prefix)

After txn success:

• batch record is readable;
• durable prefix points to this batch;
• callbacks can be dispatched;
• standby can apply through new_prefix.last_seq.

On txn failure or network timeout:

1. Re-read durable prefix.
2. If durable prefix equals new_prefix and the batch record is readable and valid, treat the previous unknown outcome as success.
3. If durable prefix still equals expected_prefix, retry the txn.
4. If durable prefix is older than expected, or newer but not equal to this batch's new_prefix, treat it as stale writer / concurrent primary / data anomaly. Keep writer unhealthy and do not skip.

Durable prefix must be monotonic. Exact-value compare prevents overwrite and rollback.

11. Backend Failure Policy

The first version uses:

fail-closed admission + indefinite retry + success auto-recover

On the first backend txn or durable-prefix advancement failure:

keep writing_batch in place
accepting=false
new Reserve() fails fast
do not dispatch success callbacks
do not write later batches
retry current writing_batch indefinitely

For committed but not durable mutations:

• the writer does not actively return an error;
• RPC/finalize waiters remain pending;
• if retry eventually succeeds, callbacks complete normally;
• if leadership changes or the service shuts down, upper-layer RPC deadline, connection close, or shutdown handling ends client waiting.

After a successful retry:

durable prefix advances
callbacks are dispatched in sequence order
writing_batch is cleared
accepting=true
writer immediately checks the waiting batch

Later batches must never pass a stuck batch. Passing it would create a durable-prefix gap and break standby/promotion safety.

12. Durable Callback and Finalize

Callbacks fire only after durable prefix has advanced:

batch record readable
batch checksum valid
durable_prefix points to this batch
=> callbacks in this batch may be dispatched in logical sequence order

Callback must not fire just because the batch record was written, and must not depend on legacy /latest.

The writer thread should only enqueue callback tasks. It should not perform heavy business logic, resource release, quota accounting, or metadata relocking.

Remove/evict/clear finalize callbacks may perform:

• EraseMetadata();
• replica erase/finalize;
• quota release;
• local disk usage release;
• cache total accounting;
• disk_object_count updates;
• group-member unregister;
• release of moved-out replica/resource state;
• RPC waiter wakeup.

The callback context must include enough information to finalize safely:

• tenant id;
• object key;
• sequence id;
• operation kind;
• affected replica ids;
• expected replica status;
• quota/resource release mode.

Finalize must re-enter the shard lock and verify that current metadata still matches the pending mutation. If the object has been recreated by a higher-sequence write, the callback must not delete the new metadata.

13. Visibility Semantics

Operations That Reduce Readability

Operations that reduce readable replica/object state may make the state invisible on the primary before durability:

• Remove()
• BatchRemove()
• RemoveAll()
• BatchReplicaClear()
• BatchEvict()
• NoFBatchEvict()
• BatchEvictDiskReplica()
• stale-handle cleanup
• expired processing/replication cleanup

Target flow:

1. Reserve().
2. Acquire snapshot_mutex_ shared lock and metadata shard lock.
3. Check object, lease, pin, refcnt, task, and other preconditions.
4. Build the current-format OpLog entry.
5. Mark affected replicas as ReplicaStatus::REMOVED, or otherwise stop returning them as COMPLETE.
6. Commit(reservation, entry, durable callback).
7. Release locks.
8. After durable callback, perform real erase/finalize and resource release.

Resource release must not happen before durable callback. This includes buffer/resource reuse, quota release, local disk usage release, cache accounting, disk object count, and group membership cleanup.

ReplicaStatus::REMOVED means:

• logically unreadable;
• not returned by GetReplicaList();
• resources are not finalized yet;
• physical erase/release happens after durable callback.

If other COMPLETE replicas remain, reads return those readable replicas. If all readable replicas are removed, reads return OBJECT_NOT_FOUND.

Operations That Increase Readability

Operations that add readable replica/object state may remain visible on the primary before durability. If the primary crashes before durability, the new primary may lose that added readable state.

This applies to:

• PutEnd
• AddReplica
• NotifyOffloadSuccess() fallback
• NotifyPromotionSuccess()
• copy/move target complete
• other paths that mark replicas COMPLETE

This keeps the first version close to current behavior and avoids introducing a pending-visible state for every readability-increasing path.

No New PendingVisibilityState

The first version does not add a new object-level PendingVisibilityState. Read-path distinction should use existing ReplicaStatus:

Has COMPLETE replica
    => OK, return COMPLETE descriptors
No COMPLETE replica, all remaining replicas are REMOVED or metadata only represents removed state
    => OBJECT_NOT_FOUND
No COMPLETE replica, but has PROCESSING / INITIALIZED / other in-progress status
    => REPLICA_IS_NOT_READY
Metadata entry absent / invalid
    => OBJECT_NOT_FOUND

14. Standby Apply

The first version uses polling, not batch-key watch:

1. Periodically read /oplog/{cluster}/durable_prefix.
2. If durable_prefix does not exist, continue using the legacy path.
3. If durable_prefix exists, treat it as the only safety boundary.
4. Range-read /oplog/{cluster}/batches/ after the last applied batch id.
5. Validate schema, checksum, batch_id continuity, and sequence continuity.
6. Expand entries.
7. Apply entries in logical sequence order.

Legacy and batch path transition:

• If durable prefix does not exist, standby uses legacy /latest plus per-entry keys.
• If durable prefix exists and batch_id == 0, it is a cutover boundary. Legacy [1,last_seq] is the safe boundary.
• If durable prefix exists and batch_id > 0, standby must first apply legacy entries through the cutover last_seq, then read batch records.

Batch-record mode must be strict no-gap. If durable prefix is unreadable, a referenced batch is unreadable, batch IDs are discontinuous, first_seq != local_last_applied_seq + 1, entries are discontinuous, checksum fails, schema version is unsupported, or deserialization fails, standby must become unhealthy/fail-closed. It must not skip the gap, and it must not be promoted.

Batch entries still use the current OpLogEntry:

• PUT_END: apply full metadata post-state;
• REMOVE: delete object metadata;
• SEGMENT_*: update standby segment registry;
• tenant_id: keep #2331 tenant-aware semantics, defaulting missing legacy values to default.

As follow-up hardening, PUT_END apply should consistently use object last_sequence_id stale checks. Strict no-gap reduces reordering risk, but stale checks still help against duplicates, retries, and legacy/batch mixed anomalies.

15. Promotion

Before promotion, standby must catch up to durable prefix:

local_last_applied_seq == durable_prefix.last_seq

Promotion must fail closed if:

• durable prefix cannot be read;
• the batch referenced by durable prefix cannot be read;
• batch checksum or schema validation fails;
• batch_id or sequence gap remains unresolved;
• local applied sequence is behind durable prefix and cannot catch up;
• batch-record mode has any unresolved gap.

In batch-record mode, /latest is not a promotion safety boundary.

16. Relation to Current Code

The current write entry points should be gradually replaced or routed through the new writer:

• AppendOpLogAndNotify()
• AppendOpLogAndNotifyDurable()
• AppendOpLogAndNotifyDurableOrAbort()
• PersistOpLogEntryWithSyncRetries()
• AppendOrPersistOrEnqueue()
• PersistRemoveForHA()
• PersistSegmentOpForHAOrEnqueue()

Legacy EtcdOpLogStore can still be used for:

• legacy per-entry reads;
• reading /latest during cutover;
• legacy standby path when durable prefix does not exist;
• compatibility with old snapshot-boundary reads.

The batch-record writer should not write new entries through EtcdOpLogStore::WriteOpLog().

Implementation Plan

Phase 1: Batch-record Writer Foundation

• Add oplog_batch_max_entries, default 1024.
• Define DurablePrefix and OpLogBatchRecord.
• Define ha/kv/HaKvBackend and an etcd txn/CAS backend.
• Define OpLogBatchStorage.
• Implement batch key, durable-prefix key, and value codecs.
• Implement checksum and schema-version validation.
• Implement OrderedOpLogWriter:
  • Reserve / Commit / Abort;
  • sequence assignment in Commit();
  • work-conserving seal;
  • single in-flight writing batch;
  • accepting=false after first backend failure;
  • indefinite retry for stuck batch;
  • accepting=true after retry success;
  • callback dispatch in sequence order.
• Do not write new /latest values in batch-record mode.

Phase 2: Standby Dual-format Reader

• Use legacy path when durable prefix does not exist.
• Enter batch-record path when durable prefix exists.
• If standby is behind the cutover legacy latest, catch up through legacy per-entry replay first.
• Poll durable prefix.
• Range-read batches.
• Expand and apply batch entries.
• Disable normal gap skip in batch-record mode.
• Require final catch-up to durable prefix before promotion.

Phase 3: Basic Write-path Integration

Priority paths:

• PutEnd / AddReplica
• segment lifecycle
• Remove / BatchRemove / RemoveAll
• BatchReplicaClear
• BatchEvict / NoFBatchEvict / BatchEvictDiskReplica
• offload success
• promotion success

The initial integration can keep most current visibility behavior and only replace the durable write entry point. This keeps the first PR reviewable and avoids changing all resource-release semantics at once.

Phase 4: Remove/Evict Logical Remove and Durable Finalize

• In the short lock-held path, mark affected replicas as REMOVED.
• Release locks after Commit().
• Perform real erase/finalize in the durable callback.
• Adjust read path using existing ReplicaStatus:
  • at least one COMPLETE: OK;
  • all readable replicas removed or absent: OBJECT_NOT_FOUND;
  • only in-progress replicas: REPLICA_IS_NOT_READY.

Phase 5: Health and Operations

• Expose writer accepting=false / unhealthy state to runtime/admin state.
• Fail fast for new mutating requests.
• Let leadership/supervisor decide whether to stop serving or trigger failover.
• Add metrics for durable prefix lag, retry count, stuck batch age, queue size, callback latency, and admission failures.
• Later add durable-prefix watch, byte limits, non-txn backend exploration, and ha/kv lease/watch sidecar interfaces.

Phase 6: UpsertStart Follow-up

UpsertStart has more complex semantics:

• same-size in-place rewrite;
• size-changing replacement;
• old PROCESSING preemption;
• quota replacement;
• discarded replicas.

This RFC does not make UpsertStart a blocker for the first batch-record writer version. A separate design should define old-readable visibility, new-readable visibility before durability, failure handling, quota/resource handling, and compatibility with current-format full-state entries.

Phase 7: Delta Schema Follow-up

Delta schema is not part of the first version. Future work can introduce:

• OBJECT_MARK_REMOVED
• REPLICA_MARK_REMOVED
• OBJECT_CLEAR_REPLICAS
• OBJECT_MARK_UNAVAILABLE
• REPLICA_ADD
• REPLICA_UPDATE
• REPLICA_MARK_COMPLETE

Apply rules, idempotency, legacy mixed apply, snapshot relation, and compaction relation should be designed separately.

Testing Strategy

Storage and Codec

• batch key padding preserves numeric range order;
• durable prefix encode/decode works;
• batch record encode/decode works;
• checksum detects corrupted values;
• unsupported schema version fails closed;
• reader filters /latest, /snapshot/, /batches/, and /durable_prefix correctly.

txn/CAS

• txn puts batch record and durable prefix together;
• expected durable-prefix compare advances prefix on success;
• compare failure does not trigger callback;
• unknown outcome retry recognizes success when prefix already advanced and batch validates;
• durable prefix cannot roll back;
• non-txn backend returns unsupported for first-version batch-record HA.

Writer Admission

• Reserve() does not allocate sequence;
• Abort() creates no gap;
• Commit() assigns continuous sequence;
• Reserve() limits open waiting batch only, not writing batch;
• reaching oplog_batch_max_entries makes new Reserve() fail fast;
• seal releases committed slots so new reservations can enter the next open waiting batch;
• Commit() does no backend I/O, no waiting, and no full batch serialization.

Writer Flush / Retry / Callback

• an idle writer writes even a one-entry batch immediately;
• single in-flight writer prevents later batches from passing a stuck batch;
• first backend failure sets accepting=false;
• new Reserve() fails while accepting=false;
• already reserved callers may still commit or abort;
• stuck batch retries indefinitely;
• retry success restores accepting=true;
• callback does not fire before durable prefix advancement;
• callbacks dispatch in logical sequence order;
• slow callback execution does not block the next backend write.

Legacy Compatibility

• durable prefix absent means standby uses legacy path;
• cutover initializes {batch_id=0,last_seq=latest} from /latest;
• cutover does not scan max sequence;
• standby behind legacy latest catches up with legacy per-entry replay first;
• first batch satisfies first_seq == legacy_latest + 1;
• batch-record mode does not write new /latest.

Standby and Promotion

• poll durable prefix and range-read batches;
• validate batch_id continuity;
• validate first_seq/last_seq continuity;
• validate entry sequence continuity;
• checksum/schema/deserialize failures fail closed;
• batch-record mode does not use normal gap skip;
• promotion requires catch-up to durable prefix;
• promotion fails closed if durable prefix cannot be reached.

MasterService Integration

• PutEnd / AddReplica remain visible on primary according to current semantics;
• Remove / BatchRemove / RemoveAll keep durable guard correctness;
• BatchReplicaClear, BatchEvict, NoFBatchEvict, and BatchEvictDiskReplica preserve pin/refcnt/lease preconditions;
• segment mount/unmount/update replay into standby segment registry;
• tenant-aware keys remain isolated in standby metadata;
• promotion restore preserves tenant, group_id, data_type, and segment registry state;
• durable finalize does not delete metadata recreated by a higher sequence.

Key Design Points Summary

1. Build on #2331: keep current OpLog semantics, tenant-aware metadata, segment lifecycle replay, and promotion handoff.
2. Batch record is the new durable unit: reduce per-entry backend key amplification.
3. Durable prefix is the only safety boundary: callbacks, standby apply, and promotion use {batch_id,last_seq}.
4. Sequence is assigned in Commit: Reserve() does not allocate sequence and Abort() creates no gap.
5. Work-conserving writer: no artificial wait to fill a batch; high concurrency naturally coalesces entries.
6. Fail closed: first backend failure stops new admission, the stuck batch is not skipped, and retry success recovers automatically.
7. Legacy compatible without double-write: new readers support both formats; new primary stops writing legacy per-entry keys after cutover.
8. First version requires txn/CAS: non-txn backends are future work and are not HA-correct in this RFC.
9. No delta schema in the first version: batch entries still use current full OpLogEntry.
10. Strict no-gap in batch-record mode: standby cannot skip batch gaps, and promotion must catch up to durable prefix.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Icedcoco on GitHub (Jul 6, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2756 Originally assigned to: @Icedcoco on GitHub. ## Introduction Mooncake Store HA should do more than restart the Master process quickly. After a primary failure and failover, the new Master should preserve as much useful KV cache metadata, segment registry state, and object-location index state as possible. The existing HA work has two complementary recovery paths: 1. **Snapshot** provides a coarse-grained baseline for restoring complete Master state. 2. **OpLog** provides incremental changes after the snapshot, so a standby can keep following the primary and catch up before promotion. RFC #1200 proposed an etcd-backed hot-standby mechanism for Master metadata HA. The primary writes metadata mutations as OpLog entries, and the standby reads/watches those entries from etcd and applies them locally. RFC #1650 then introduced an `OpLogStore` / `OpLogChangeNotifier` abstraction to decouple the upper-layer replication logic from the etcd-specific implementation. PR #2331 is the immediate baseline for this RFC. Although that PR was submitted by @Icedcoco, most of the HA hot-standby and OpLog implementation came from earlier work by @Libotry and @00fish0 in the `dev/snapshot_etcd_storage_integration_testing_fixbug` branch. The work in #2331 was mainly to rebase that legacy implementation onto the current `kvcache-ai/Mooncake:main`, adapt it to the current Mooncake Store architecture, and integrate recently added Store features such as tenant-aware object metadata, segment lifecycle replication, and the updated standby promotion / snapshot handoff paths. This RFC builds on #2331 and proposes the next step for the OpLog write and replication protocol: **batch records**, an **ordered writer**, and a **durable prefix**. The goal is not to replace the business semantics that #2331 already adapted. Instead, this RFC keeps the current logical `OpLogEntry` semantics and replaces the underlying write and replay protocol to address lock-held durable waits, per-entry backend key amplification, and promotion boundary ambiguity. Relevant links: - Roadmap: https://github.com/kvcache-ai/Mooncake/issues/1035 - Hot Standby RFC: https://github.com/kvcache-ai/Mooncake/issues/1200 - OpLog Store Abstraction RFC: https://github.com/kvcache-ai/Mooncake/issues/1650 - Snapshot RFC: https://github.com/kvcache-ai/Mooncake/issues/1150 - etcd wrapper for HA: https://github.com/kvcache-ai/Mooncake/pull/1451 - initial hot-standby / OpLog implementation: https://github.com/kvcache-ai/Mooncake/pull/1515 - HA backend abstraction: https://github.com/kvcache-ai/Mooncake/pull/1678 - Redis leadership backend: https://github.com/kvcache-ai/Mooncake/pull/1722 - HA runtime / standby lifecycle refactor: https://github.com/kvcache-ai/Mooncake/pull/1777 - current rebase and adaptation PR: https://github.com/kvcache-ai/Mooncake/pull/2331 ## Background ### Roadmap Context The Mooncake Store V3 Roadmap (#1035) tracks HA under `Milestone 2: Master Service Enhancements`, specifically `Metadata Adaptation - HA`. That item has three related lines of work: - **Snapshot**: #1150, #1431, #1465, #1381 and related work provide persistence and recovery for Master metadata, segment state, allocator state, and task-manager state. - **OpLog**: #1200, #1451, #1515 and related work provide incremental metadata replication after a snapshot baseline. - **HA refactoring**: #1648, #1678, #1722, #1777 and related work move leader coordination, standby runtime, snapshot bootstrap, and OpLog following into clearer runtime boundaries. The relationship is straightforward: snapshot restores a complete baseline, OpLog catches the standby up from that baseline to a later primary state, and the HA runtime orchestrates snapshot bootstrap, OpLog following, promotion, and serving. ### Snapshot Baseline RFC #1150 was motivated by a practical limitation of early HA mode: fast Master restart alone does not recover KV cache location metadata. If all metadata is lost after failover, upper-layer inference must rebuild cache state, and the recovery time can be much longer than the Master process restart time. PR #1431 implemented Master snapshot and restore: - fork + copy-on-write to reduce blocking during snapshot creation; - background serialization and compression of metadata, segment state, allocator state, and task-manager state; - local filesystem and S3 snapshot object stores; - restore-time cleanup of incomplete metadata, non-ready segments, and expired leases. PR #1465 added task-manager serialization/deserialization. PR #1381 designed an etcd snapshot object store so that snapshot data can also use the HA etcd deployment. Snapshot is still periodic full-state persistence. It cannot recover mutations that happen after the latest snapshot. OpLog is therefore needed to reduce that loss window and let the standby catch up before promotion. ### RFC #1200: Initial OpLog Hot Standby RFC #1200 proposed using etcd as a reliable intermediate component for primary-standby OpLog synchronization: - On the primary, `OpLogManager` assigns a globally increasing `sequence_id` and creates `OpLogEntry` records for metadata mutations. - `EtcdOpLogStore` persists each entry to etcd, for example under `/oplog/{cluster}/{sequence_id}`. - `/oplog/{cluster}/latest` records the latest sequence. - On the standby, `OpLogWatcher` receives real-time changes through etcd watch and reads historical entries during startup or reconnect. - `OpLogApplier` checks ordering and applies entries to standby metadata. - `HotStandbyService` manages standby lifecycle and promotion. The main design points were correctness and RPO/RTO: global sequence ordering, etcd persistence, etcd watch, snapshot sequence recording, and OpLog cleanup. ### PR #1451 and PR #1515: Early Implementation PR #1451 wrapped the basic etcd interface needed by Master HA. PR #1515 implemented the early hot-standby and OpLog interface: - `EtcdOpLogStore` - `OpLogManager` - `OpLogWatcher` - `OpLogApplier` - `HotStandbyService` - `StandbyStateMachine` - OpLog types such as `PUT_END`, `PUT_REVOKE`, and `REMOVE` - unit tests for standby metadata store behavior, gap handling, watch/reconnect, and promotion This moved the OpLog design from RFC into a runnable implementation. However, the early code was tightly coupled to etcd, and the Mooncake Store metadata schema, tenant model, segment lifecycle, and promotion runtime have changed since then. ### RFC #1650: OpLog Store Abstraction RFC #1650 addressed the etcd coupling in the early OpLog subsystem. Several upper-layer components directly held or constructed `EtcdOpLogStore`, while the old watcher mixed watch, reconnect, deserialization, and apply orchestration in one class. The RFC proposed: - an `OpLogStore` interface for write, read, read-since, latest-sequence, snapshot-sequence, and cleanup operations; - an `OpLogChangeNotifier` interface for backend-specific notification; - an `OpLogReplicator` that replaces the monolithic etcd-centric watcher; - shared OpLog serialization; - centralized store creation through a factory; - mock stores/notifiers for deterministic unit tests. That RFC intentionally kept behavior unchanged. It extracted interfaces and clarified boundaries, but did not change the legacy per-entry OpLog layout or logical entry semantics. ### HA Runtime Refactoring PR #1678 introduced an HA backend abstraction and moved leader coordination from the legacy etcd helper path to `LeaderCoordinator`. It also kept `OpLogStore` and `SnapshotStore` as backend boundaries for follow-up work. PR #1722 added a Redis leader coordinator, proving that the leadership path can be backend-independent. RFC #1648 discussed Kubernetes Lease-based HA. That work is about leader coordination and master-view discovery. It does not by itself provide OpLog durability or replay semantics. PR #1777 split the always-on Master process runtime from the leader-only serving runtime, introduced standby-controller composition, and made snapshot bootstrap plus OpLog following part of the standby runtime. This is the runtime foundation needed for switching the standby between legacy per-entry replay and batch-record replay. ### PR #2331: Current Foundation PR #2331 (`[Store] add HA OpLog replication and tenant-aware standby promotion`) rebased and adapted the earlier HA hot-standby work to current main. The key changes include: - HA OpLog configuration: `oplog_store_type`, `oplog_store_root_dir`, and `oplog_poll_interval_ms`; - primary-side OpLog publishing for Store metadata mutation paths; - durable OpLog guarding for HA-enabled mutations where dropping the update would make primary and standby diverge; - `SEGMENT_MOUNT`, `SEGMENT_UNMOUNT`, and `SEGMENT_UPDATE` OpLog entries for standby segment registry replay; - `tenant_id` in `OpLogEntry`, tenant-aware metadata store APIs, and tenant-aware standby snapshot/restore paths; - backward-compatible OpLog deserialization when old entries do not contain `tenant_id`; - standby snapshot export and promoted-primary restore for object metadata, group membership, segment state, and the OpLog sequence baseline; - tests for OpLog apply, standby promotion, tenant isolation, segment lifecycle replay, and MasterService HA mutation behavior. After #2331, current main has a much more complete OpLog HA baseline. The remaining issue is the storage and write model: the legacy path still stores one backend key per logical `OpLogEntry`, and `/latest` remains the latest-sequence pointer. `EtcdOpLogStore::WriteOpLog(entry, true)` already uses an internal batch writer, so the business thread is not directly writing each single key to etcd. But from the Master mutation path, synchronous waiting can still happen while holding `snapshot_mutex_` in shared mode and a metadata shard lock in exclusive mode. ## Motivation The current #2331-based HA OpLog path covers many business cases, but the write protocol still has several structural problems. ### 1. Lock-held Durable Wait Several paths can wait for durable OpLog persistence while holding `snapshot_mutex_` and a metadata shard lock: - `Remove()` - `BatchRemove()` - `RemoveAll()` - `BatchReplicaClear()` - `BatchEvict()` - `NoFBatchEvict()` - `BatchEvictDiskReplica()` - stale-handle cleanup - expired processing/replication cleanup - segment lifecycle updates - offload/promotion success paths The current sync write path enqueues entries into an internal pending batch and waits for the flush result. etcd latency, batch-flush delay, retry, or sync timeout can therefore block unrelated metadata mutations in the same shard. ### 2. Per-entry Backend Key Amplification The legacy layout is: ```text /oplog/{cluster}/{sequence_id} /oplog/{cluster}/latest /oplog/{cluster}/snapshot/{snapshot_id}/sequence_id ``` Each logical OpLog entry becomes one backend key. Under high mutation concurrency, the backend must handle many keys, watch events, MVCC items, and compaction items. Internal group commit reduces the number of flush calls, but it does not reduce the number of persisted keys. ### 3. Split Safety Boundary In the legacy model, an entry key means the entry exists, while `/latest` means the primary has advanced the latest sequence pointer. For standby apply and promotion, the important concept is a verified, continuous, no-gap durable boundary. Batch-record mode should make this explicit. A batch record and the durable boundary should advance together through a CAS/txn protocol, and standby/promotion should trust only that durable boundary. ### 4. Need for a Post-#2331 Evolution Path #2331 adapted the legacy HA OpLog implementation to current main and added tenant, segment, and promotion compatibility. The next step should not rewrite those semantics. It should keep: - current logical `OpLogEntry` semantics; - current `PUT_END`, `REMOVE`, and `SEGMENT_*` apply behavior; - tenant-aware metadata; - standby snapshot export and promoted-primary restore; - legacy compatibility during upgrade. The part to replace is the underlying durable write and replay protocol. ## Goals 1. Replace new per-entry backend keys with **batch records**. One batch record is one backend KV value containing multiple logical `OpLogEntry` records over a continuous sequence range. 2. Use `{batch_id, last_seq}` as the **durable prefix** and the only safe boundary for callbacks, standby apply, and promotion. 3. Move OpLog admission, sequence assignment, batching, retry, and durable callbacks out of metadata shard lock hold time, or reduce the lock-held path to a short enqueue. 4. Keep logical sequence globally monotonic and continuous. 5. Keep the existing logical meaning of `OpLogEntry`. 6. Support mixed reading of legacy per-entry OpLog and new batch-record OpLog in the same cluster, without double-writing. 7. Require txn/CAS for the first version. Non-txn backends remain a future extension and must not be claimed as HA-correct in this version. 8. On backend failure, fail closed: stop new write admission, do not skip the stuck batch, do not report success before durability, and retry until backend recovery or leadership change. ## Non-Goals 1. Do not introduce a new delta OpLog schema in the first version. Batch entries still use the current full `OpLogEntry` format. 2. Do not double-write legacy per-entry OpLog and batch-record OpLog. 3. Do not require old standbys to read new batch records. Operators must upgrade standby/readers first, then switch primary to batch-record mode. 4. Do not claim HA correctness for non-txn backends in the first version. 5. Do not put leadership lease/watch/election semantics into the minimal KV backend interface. 6. Do not fully redesign `UpsertStart` in this RFC. That path needs a separate design because its semantics are more complex. ## Proposal ### 1. Architecture Introduce three layers: ```cpp class OrderedOpLogWriter; class OpLogBatchStorage; class HaKvBackend; ``` #### OrderedOpLogWriter `OrderedOpLogWriter` is the only OpLog admission, sequence assignment, batching, retry, and callback-dispatch entry point in a serving primary. Business code uses: 1. `Reserve()` before metadata mutation. 2. `Commit()` inside the metadata shard lock to attach the final `OpLogEntry` and durable callback. 3. `Abort()` if the precondition check fails or no OpLog is needed. The writer owns: - writer health and `accepting`; - open waiting-batch slot accounting; - logical sequence assignment; - work-conserving batch seal; - single in-flight writing batch; - backend txn/CAS write; - durable prefix advancement; - retry and unknown-outcome handling; - callback dispatch in logical sequence order. #### OpLogBatchStorage `OpLogBatchStorage` is a layout/storage helper between the writer and the KV backend. It owns: - batch-record key construction; - durable-prefix key construction; - batch-record encoding/decoding; - durable-prefix encoding/decoding; - checksum validation; - etcd txn/CAS write; - range reading of batch records; - legacy `/latest` read for cutover initialization. It does not own admission, sequence assignment, retry policy, callback dispatch, or Master business semantics. #### HaKvBackend `HaKvBackend` is a reusable HA KV adapter layer. It provides only generic KV/range/transaction capabilities. It does not know about OpLog, durable prefixes, leadership, snapshot, or metadata semantics. The first production implementation should be an etcd txn/CAS backend. Future leadership reuse should use sidecar interfaces such as `HaKvLeaseBackend` or `HaKvWatchBackend`, instead of adding lease/watch/election methods to the minimal KV interface. ### 2. Core Data Structures ```cpp struct DurablePrefix { uint64_t batch_id{0}; uint64_t last_seq{0}; }; struct OpLogBatchRecord { uint32_t schema_version{1}; uint64_t batch_id{0}; uint64_t first_seq{0}; uint64_t last_seq{0}; std::vector<OpLogEntry> entries; uint32_t checksum{0}; }; ``` `DurablePrefix` means every logical entry in `[1, last_seq]` is durable and safe to apply/promote from. `batch_id` identifies the physical batch record that contains `last_seq`. `OpLogBatchRecord` must satisfy: - `entries` is non-empty; - entries are sorted by increasing `sequence_id`; - `first_seq == entries.front().sequence_id`; - `last_seq == entries.back().sequence_id`; - sequence IDs are continuous inside a batch; - sequence IDs are continuous across batches; - `batch_id` is monotonic; - checksum covers the encoded batch content except the checksum field itself. ### 3. Minimal KV Backend Interface ```cpp struct KvPair { std::string key; std::string value; }; enum class KvCompareKind { kValueEquals, kKeyNotExists, }; struct KvCompare { std::string key; KvCompareKind kind{KvCompareKind::kValueEquals}; std::string expected_value; }; struct KvTxn { std::vector<KvCompare> compares; std::vector<KvPair> puts; }; class HaKvBackend { public: virtual ~HaKvBackend() = default; virtual ErrorCode Get(std::string_view key, std::string& value) = 0; virtual ErrorCode Put(std::string_view key, std::string_view value) = 0; virtual ErrorCode Range(std::string_view begin_key, std::string_view end_key, size_t limit, std::vector<KvPair>& kvs) = 0; virtual bool SupportsTxn() const = 0; virtual ErrorCode Txn(const KvTxn& txn) = 0; }; ``` Batch-record HA initialization must check `SupportsTxn()`. If the backend does not support txn/CAS, the first version should fail initialization or return unsupported. ### 4. Storage Layout Keep legacy keys unchanged: ```text /oplog/{cluster}/{seq_padded} /oplog/{cluster}/latest /oplog/{cluster}/snapshot/{snapshot_id}/sequence_id ``` Add batch-record keys under the same prefix: ```text /oplog/{cluster}/batches/{batch_id_padded} /oplog/{cluster}/durable_prefix ``` Use fixed-width decimal encoding for `batch_id`, preferably the same 20-digit padding used by legacy sequence keys. This keeps lexicographic range scan order equal to numeric order. Since the new keys remain under `/oplog/{cluster}/`, readers must explicitly filter: ```text /latest /snapshot/ /batches/ /durable_prefix ``` ### 5. Durable Prefix Value `/oplog/{cluster}/durable_prefix` should contain at least: ```text schema_version=<version> batch_id=<id> last_seq=<seq> checksum=<optional> ``` The actual encoding can use an existing project serialization format, but it must include `schema_version` so later versions can add leader epoch, term, primary id, or layout version. Semantics: - `[1, last_seq]` is the only safe apply/promotion boundary. - `batch_id` locates the batch containing `last_seq`. - standby and promotion must use durable prefix, not `/latest`, as the safety boundary. - batch-record mode does not write new `/latest` values. - legacy `/latest` is used only for cutover initialization and legacy reader compatibility. ### 6. Cutover and Legacy Compatibility Upgrade order: ```text 1. Upgrade standby/readers so they can read both legacy per-entry OpLog and batch-record OpLog. 2. Switch the primary to batch-record writer. 3. After the switch, the primary stops writing new legacy per-entry keys and stops updating /latest. ``` If a batch-record writer starts and durable prefix does not exist: ```text legacy_latest = Read(/oplog/{cluster}/latest) durable_prefix = { batch_id = 0, last_seq = legacy_latest } next_sequence = legacy_latest + 1 next_batch_id = 1 ``` Create durable prefix with CAS/create-if-absent: ```text compare durable_prefix key does not exist put durable_prefix = { batch_id = 0, last_seq = legacy_latest } ``` If the compare fails, another initializer has created the durable prefix. The current writer must read the existing prefix and use it as the source of truth. This design intentionally trusts `/latest` during cutover: - do not scan legacy max sequence; - do not check for legacy keys with `seq > latest`; - if such keys exist, they are treated as outside the new durable boundary. This is an operational tradeoff. It assumes the old write path has been stopped during cutover, or that `/latest` is the accepted legacy safety boundary. ### 7. Writer Admission Protocol Add: ```text oplog_batch_max_entries = 1024 ``` This limits the number of entries in the current open waiting batch. It is an admission/backpressure limit, not a target batch size. Business code uses: ```cpp tl::expected<Reservation, ErrorCode> Reserve(); tl::expected<PendingHandle, ErrorCode> Commit(Reservation&& reservation, OpLogEntry entry, DurableCallback callback); void Abort(Reservation&& reservation); ``` #### Reserve `Reserve()` runs before metadata mutation: ```text if (!accepting) return OPLOG_UNAVAILABLE; if (open_waiting_reserved_or_committed >= oplog_batch_max_entries) return QUEUE_FULL; ++open_waiting_reserved_or_committed; return Reservation{}; ``` `Reserve()` does not allocate a sequence ID. Therefore, `Abort()` does not create a gap. #### Commit `Commit()` runs inside the metadata shard lock, but must stay short: - consume an existing reservation; - assign the next continuous `sequence_id`; - fill the entry; - append it to the waiting queue; - attach the durable callback; - notify the writer; - do no backend I/O; - do not wait for queue space; - do not serialize the full batch record. If the caller already has a reservation and the entry uses a known serializable OpLog format, normal `Commit()` should not fail. #### Abort `Abort()` releases a reservation if the precondition check fails or no OpLog is needed: - release the slot; - do not allocate sequence; - do not create a gap; - do not trigger callbacks. ### 8. Batch Seal and Flush Policy The first version uses a **work-conserving single in-flight writer**: ```text 1. If writing_batch is idle and waiting_queue has entries, seal the current open waiting batch immediately. 2. Even one entry is enough to write a batch. 3. If writing_batch is in progress, business threads can keep reserve/commit into a new open waiting batch. 4. After writing_batch succeeds: - if waiting batch has entries, seal and write the next batch immediately; - otherwise sleep until commit notification. ``` Implications: - `oplog_batch_max_entries` is a backpressure limit, not a flush threshold. - Low traffic may produce one-entry batches. - High traffic or slower backend writes will naturally coalesce entries. - The design optimizes latency first and does not intentionally wait to fill a batch. Seal does not wait for reservations that have not yet committed or aborted. Already committed entries can be drained and sealed. Slow business threads keep their reserved slots and their later commits go into a later seal. ### 9. Sequence and Batch ID Logical sequence is assigned in `Commit()`: ```text next_sequence starts from durable_prefix.last_seq + 1 Commit() assigns current next_sequence next_sequence++ ``` Guarantees: - `Abort()` creates no gap; - committed entries are continuous; - entries inside a batch are continuous; - batches are continuous; - standby strict no-gap validation can hold. `batch_id` is assigned at seal time: ```text next_batch_id starts from durable_prefix.batch_id + 1 seal waiting batch assigns current next_batch_id next_batch_id++ ``` `batch_id` does not replace logical sequence. It locates batch records, describes the physical durable prefix position, supports standby range reads, and lets promotion verify batch continuity. ### 10. Backend txn/CAS Protocol For one writing batch: ```text expected_prefix = current durable prefix before this batch new_prefix = { batch_id = batch.batch_id, last_seq = batch.last_seq } txn: compare /oplog/{cluster}/durable_prefix == Encode(expected_prefix) put /oplog/{cluster}/batches/{batch_id_padded} = Encode(batch) put /oplog/{cluster}/durable_prefix = Encode(new_prefix) ``` After txn success: - batch record is readable; - durable prefix points to this batch; - callbacks can be dispatched; - standby can apply through `new_prefix.last_seq`. On txn failure or network timeout: 1. Re-read durable prefix. 2. If durable prefix equals `new_prefix` and the batch record is readable and valid, treat the previous unknown outcome as success. 3. If durable prefix still equals `expected_prefix`, retry the txn. 4. If durable prefix is older than expected, or newer but not equal to this batch's `new_prefix`, treat it as stale writer / concurrent primary / data anomaly. Keep writer unhealthy and do not skip. Durable prefix must be monotonic. Exact-value compare prevents overwrite and rollback. ### 11. Backend Failure Policy The first version uses: > fail-closed admission + indefinite retry + success auto-recover On the first backend txn or durable-prefix advancement failure: ```text keep writing_batch in place accepting=false new Reserve() fails fast do not dispatch success callbacks do not write later batches retry current writing_batch indefinitely ``` For committed but not durable mutations: - the writer does not actively return an error; - RPC/finalize waiters remain pending; - if retry eventually succeeds, callbacks complete normally; - if leadership changes or the service shuts down, upper-layer RPC deadline, connection close, or shutdown handling ends client waiting. After a successful retry: ```text durable prefix advances callbacks are dispatched in sequence order writing_batch is cleared accepting=true writer immediately checks the waiting batch ``` Later batches must never pass a stuck batch. Passing it would create a durable-prefix gap and break standby/promotion safety. ### 12. Durable Callback and Finalize Callbacks fire only after durable prefix has advanced: ```text batch record readable batch checksum valid durable_prefix points to this batch => callbacks in this batch may be dispatched in logical sequence order ``` Callback must not fire just because the batch record was written, and must not depend on legacy `/latest`. The writer thread should only enqueue callback tasks. It should not perform heavy business logic, resource release, quota accounting, or metadata relocking. Remove/evict/clear finalize callbacks may perform: - `EraseMetadata()`; - replica erase/finalize; - quota release; - local disk usage release; - cache total accounting; - `disk_object_count` updates; - group-member unregister; - release of moved-out replica/resource state; - RPC waiter wakeup. The callback context must include enough information to finalize safely: - tenant id; - object key; - sequence id; - operation kind; - affected replica ids; - expected replica status; - quota/resource release mode. Finalize must re-enter the shard lock and verify that current metadata still matches the pending mutation. If the object has been recreated by a higher-sequence write, the callback must not delete the new metadata. ### 13. Visibility Semantics #### Operations That Reduce Readability Operations that reduce readable replica/object state may make the state invisible on the primary before durability: - `Remove()` - `BatchRemove()` - `RemoveAll()` - `BatchReplicaClear()` - `BatchEvict()` - `NoFBatchEvict()` - `BatchEvictDiskReplica()` - stale-handle cleanup - expired processing/replication cleanup Target flow: ```text 1. Reserve(). 2. Acquire snapshot_mutex_ shared lock and metadata shard lock. 3. Check object, lease, pin, refcnt, task, and other preconditions. 4. Build the current-format OpLog entry. 5. Mark affected replicas as ReplicaStatus::REMOVED, or otherwise stop returning them as COMPLETE. 6. Commit(reservation, entry, durable callback). 7. Release locks. 8. After durable callback, perform real erase/finalize and resource release. ``` Resource release must not happen before durable callback. This includes buffer/resource reuse, quota release, local disk usage release, cache accounting, disk object count, and group membership cleanup. `ReplicaStatus::REMOVED` means: - logically unreadable; - not returned by `GetReplicaList()`; - resources are not finalized yet; - physical erase/release happens after durable callback. If other `COMPLETE` replicas remain, reads return those readable replicas. If all readable replicas are removed, reads return `OBJECT_NOT_FOUND`. #### Operations That Increase Readability Operations that add readable replica/object state may remain visible on the primary before durability. If the primary crashes before durability, the new primary may lose that added readable state. This applies to: - `PutEnd` - `AddReplica` - `NotifyOffloadSuccess()` fallback - `NotifyPromotionSuccess()` - copy/move target complete - other paths that mark replicas `COMPLETE` This keeps the first version close to current behavior and avoids introducing a pending-visible state for every readability-increasing path. #### No New PendingVisibilityState The first version does not add a new object-level `PendingVisibilityState`. Read-path distinction should use existing `ReplicaStatus`: ```text Has COMPLETE replica => OK, return COMPLETE descriptors No COMPLETE replica, all remaining replicas are REMOVED or metadata only represents removed state => OBJECT_NOT_FOUND No COMPLETE replica, but has PROCESSING / INITIALIZED / other in-progress status => REPLICA_IS_NOT_READY Metadata entry absent / invalid => OBJECT_NOT_FOUND ``` ### 14. Standby Apply The first version uses polling, not batch-key watch: ```text 1. Periodically read /oplog/{cluster}/durable_prefix. 2. If durable_prefix does not exist, continue using the legacy path. 3. If durable_prefix exists, treat it as the only safety boundary. 4. Range-read /oplog/{cluster}/batches/ after the last applied batch id. 5. Validate schema, checksum, batch_id continuity, and sequence continuity. 6. Expand entries. 7. Apply entries in logical sequence order. ``` Legacy and batch path transition: - If durable prefix does not exist, standby uses legacy `/latest` plus per-entry keys. - If durable prefix exists and `batch_id == 0`, it is a cutover boundary. Legacy `[1,last_seq]` is the safe boundary. - If durable prefix exists and `batch_id > 0`, standby must first apply legacy entries through the cutover `last_seq`, then read batch records. Batch-record mode must be strict no-gap. If durable prefix is unreadable, a referenced batch is unreadable, batch IDs are discontinuous, `first_seq != local_last_applied_seq + 1`, entries are discontinuous, checksum fails, schema version is unsupported, or deserialization fails, standby must become unhealthy/fail-closed. It must not skip the gap, and it must not be promoted. Batch entries still use the current `OpLogEntry`: - `PUT_END`: apply full metadata post-state; - `REMOVE`: delete object metadata; - `SEGMENT_*`: update standby segment registry; - `tenant_id`: keep #2331 tenant-aware semantics, defaulting missing legacy values to `default`. As follow-up hardening, `PUT_END` apply should consistently use object `last_sequence_id` stale checks. Strict no-gap reduces reordering risk, but stale checks still help against duplicates, retries, and legacy/batch mixed anomalies. ### 15. Promotion Before promotion, standby must catch up to durable prefix: ```text local_last_applied_seq == durable_prefix.last_seq ``` Promotion must fail closed if: - durable prefix cannot be read; - the batch referenced by durable prefix cannot be read; - batch checksum or schema validation fails; - batch_id or sequence gap remains unresolved; - local applied sequence is behind durable prefix and cannot catch up; - batch-record mode has any unresolved gap. In batch-record mode, `/latest` is not a promotion safety boundary. ### 16. Relation to Current Code The current write entry points should be gradually replaced or routed through the new writer: - `AppendOpLogAndNotify()` - `AppendOpLogAndNotifyDurable()` - `AppendOpLogAndNotifyDurableOrAbort()` - `PersistOpLogEntryWithSyncRetries()` - `AppendOrPersistOrEnqueue()` - `PersistRemoveForHA()` - `PersistSegmentOpForHAOrEnqueue()` Legacy `EtcdOpLogStore` can still be used for: - legacy per-entry reads; - reading `/latest` during cutover; - legacy standby path when durable prefix does not exist; - compatibility with old snapshot-boundary reads. The batch-record writer should not write new entries through `EtcdOpLogStore::WriteOpLog()`. ## Implementation Plan ### Phase 1: Batch-record Writer Foundation - Add `oplog_batch_max_entries`, default 1024. - Define `DurablePrefix` and `OpLogBatchRecord`. - Define `ha/kv/HaKvBackend` and an etcd txn/CAS backend. - Define `OpLogBatchStorage`. - Implement batch key, durable-prefix key, and value codecs. - Implement checksum and schema-version validation. - Implement `OrderedOpLogWriter`: - `Reserve / Commit / Abort`; - sequence assignment in `Commit()`; - work-conserving seal; - single in-flight writing batch; - `accepting=false` after first backend failure; - indefinite retry for stuck batch; - `accepting=true` after retry success; - callback dispatch in sequence order. - Do not write new `/latest` values in batch-record mode. ### Phase 2: Standby Dual-format Reader - Use legacy path when durable prefix does not exist. - Enter batch-record path when durable prefix exists. - If standby is behind the cutover legacy latest, catch up through legacy per-entry replay first. - Poll durable prefix. - Range-read batches. - Expand and apply batch entries. - Disable normal gap skip in batch-record mode. - Require final catch-up to durable prefix before promotion. ### Phase 3: Basic Write-path Integration Priority paths: - `PutEnd` / `AddReplica` - segment lifecycle - `Remove` / `BatchRemove` / `RemoveAll` - `BatchReplicaClear` - `BatchEvict` / `NoFBatchEvict` / `BatchEvictDiskReplica` - offload success - promotion success The initial integration can keep most current visibility behavior and only replace the durable write entry point. This keeps the first PR reviewable and avoids changing all resource-release semantics at once. ### Phase 4: Remove/Evict Logical Remove and Durable Finalize - In the short lock-held path, mark affected replicas as `REMOVED`. - Release locks after `Commit()`. - Perform real erase/finalize in the durable callback. - Adjust read path using existing `ReplicaStatus`: - at least one `COMPLETE`: OK; - all readable replicas removed or absent: `OBJECT_NOT_FOUND`; - only in-progress replicas: `REPLICA_IS_NOT_READY`. ### Phase 5: Health and Operations - Expose writer `accepting=false` / unhealthy state to runtime/admin state. - Fail fast for new mutating requests. - Let leadership/supervisor decide whether to stop serving or trigger failover. - Add metrics for durable prefix lag, retry count, stuck batch age, queue size, callback latency, and admission failures. - Later add durable-prefix watch, byte limits, non-txn backend exploration, and `ha/kv` lease/watch sidecar interfaces. ### Phase 6: UpsertStart Follow-up `UpsertStart` has more complex semantics: - same-size in-place rewrite; - size-changing replacement; - old `PROCESSING` preemption; - quota replacement; - discarded replicas. This RFC does not make `UpsertStart` a blocker for the first batch-record writer version. A separate design should define old-readable visibility, new-readable visibility before durability, failure handling, quota/resource handling, and compatibility with current-format full-state entries. ### Phase 7: Delta Schema Follow-up Delta schema is not part of the first version. Future work can introduce: - `OBJECT_MARK_REMOVED` - `REPLICA_MARK_REMOVED` - `OBJECT_CLEAR_REPLICAS` - `OBJECT_MARK_UNAVAILABLE` - `REPLICA_ADD` - `REPLICA_UPDATE` - `REPLICA_MARK_COMPLETE` Apply rules, idempotency, legacy mixed apply, snapshot relation, and compaction relation should be designed separately. ## Testing Strategy ### Storage and Codec - batch key padding preserves numeric range order; - durable prefix encode/decode works; - batch record encode/decode works; - checksum detects corrupted values; - unsupported schema version fails closed; - reader filters `/latest`, `/snapshot/`, `/batches/`, and `/durable_prefix` correctly. ### txn/CAS - txn puts batch record and durable prefix together; - expected durable-prefix compare advances prefix on success; - compare failure does not trigger callback; - unknown outcome retry recognizes success when prefix already advanced and batch validates; - durable prefix cannot roll back; - non-txn backend returns unsupported for first-version batch-record HA. ### Writer Admission - `Reserve()` does not allocate sequence; - `Abort()` creates no gap; - `Commit()` assigns continuous sequence; - `Reserve()` limits open waiting batch only, not writing batch; - reaching `oplog_batch_max_entries` makes new `Reserve()` fail fast; - seal releases committed slots so new reservations can enter the next open waiting batch; - `Commit()` does no backend I/O, no waiting, and no full batch serialization. ### Writer Flush / Retry / Callback - an idle writer writes even a one-entry batch immediately; - single in-flight writer prevents later batches from passing a stuck batch; - first backend failure sets `accepting=false`; - new `Reserve()` fails while `accepting=false`; - already reserved callers may still commit or abort; - stuck batch retries indefinitely; - retry success restores `accepting=true`; - callback does not fire before durable prefix advancement; - callbacks dispatch in logical sequence order; - slow callback execution does not block the next backend write. ### Legacy Compatibility - durable prefix absent means standby uses legacy path; - cutover initializes `{batch_id=0,last_seq=latest}` from `/latest`; - cutover does not scan max sequence; - standby behind legacy latest catches up with legacy per-entry replay first; - first batch satisfies `first_seq == legacy_latest + 1`; - batch-record mode does not write new `/latest`. ### Standby and Promotion - poll durable prefix and range-read batches; - validate batch_id continuity; - validate first_seq/last_seq continuity; - validate entry sequence continuity; - checksum/schema/deserialize failures fail closed; - batch-record mode does not use normal gap skip; - promotion requires catch-up to durable prefix; - promotion fails closed if durable prefix cannot be reached. ### MasterService Integration - `PutEnd` / `AddReplica` remain visible on primary according to current semantics; - `Remove` / `BatchRemove` / `RemoveAll` keep durable guard correctness; - `BatchReplicaClear`, `BatchEvict`, `NoFBatchEvict`, and `BatchEvictDiskReplica` preserve pin/refcnt/lease preconditions; - segment mount/unmount/update replay into standby segment registry; - tenant-aware keys remain isolated in standby metadata; - promotion restore preserves tenant, `group_id`, `data_type`, and segment registry state; - durable finalize does not delete metadata recreated by a higher sequence. ## Key Design Points Summary 1. **Build on #2331**: keep current OpLog semantics, tenant-aware metadata, segment lifecycle replay, and promotion handoff. 2. **Batch record is the new durable unit**: reduce per-entry backend key amplification. 3. **Durable prefix is the only safety boundary**: callbacks, standby apply, and promotion use `{batch_id,last_seq}`. 4. **Sequence is assigned in Commit**: `Reserve()` does not allocate sequence and `Abort()` creates no gap. 5. **Work-conserving writer**: no artificial wait to fill a batch; high concurrency naturally coalesces entries. 6. **Fail closed**: first backend failure stops new admission, the stuck batch is not skipped, and retry success recovers automatically. 7. **Legacy compatible without double-write**: new readers support both formats; new primary stops writing legacy per-entry keys after cutover. 8. **First version requires txn/CAS**: non-txn backends are future work and are not HA-correct in this RFC. 9. **No delta schema in the first version**: batch entries still use current full `OpLogEntry`. 10. **Strict no-gap in batch-record mode**: standby cannot skip batch gaps, and promotion must catch up to durable prefix. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:18:39 +00:00

• closed this issue
• added the
  RFC
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:18:40 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 6, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#2756
GitHub user ID	102317026
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4890108209 --> @github-actions[bot] commented on GitHub (Jul 6, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #2756 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:18:40 +00:00

Author

Owner

Copy link

@00fish0 commented on GitHub (Jul 6, 2026):

Thanks for writing this up. Overall, the proposal makes sense to me, and I'm supportive of moving forward in this direction. Happy to discuss any remaining details.

<!-- gh-comment-id:4894254205 --> @00fish0 commented on GitHub (Jul 6, 2026): Thanks for writing this up. Overall, the proposal makes sense to me, and I'm supportive of moving forward in this direction. Happy to discuss any remaining details.

bsakaguchi commented 2026-10-29 21:18:40 +00:00

Author

Owner

Copy link

@RuixiangMa commented on GitHub (Jul 7, 2026):

This is a well-designed RFC，have a few questions I'd like to ask：
1. Cutover baseline from /latest
FlushBatch() writes entry keys separately from /latest, which is updated afterward potentially via a background thread with 1s interval. The codebase even has GetMaxSequenceId() to work around /latest staleness. Would it be worth considering GetMaxSequenceId() as the cutover baseline, or adding a guard to ensure the old write path is quiesced before cutover?

2. Resource behavior during extended backend unavailability
The current code is fail-open, OpLogManager::Append() ) logs WARNING and continues on failure, and EraseMetadata() releases resources immediately. The RFC's move to fail-closed with deferred release is a great improvement, but if the backend is down for long, REMOVED-but-not-finalized replicas would accumulate while accepting=false blocks new writes. Would a backoff strategy, resource ceiling, or forced-failover trigger when stuck-batch age exceeds a threshold make sense?

<!-- gh-comment-id:4903567158 --> @RuixiangMa commented on GitHub (Jul 7, 2026): This is a well-designed RFC，have a few questions I'd like to ask： **1. Cutover baseline from /latest** FlushBatch() writes entry keys separately from /latest, which is updated afterward potentially via a background thread with 1s interval. The codebase even has GetMaxSequenceId() to work around /latest staleness. Would it be worth considering GetMaxSequenceId() as the cutover baseline, or adding a guard to ensure the old write path is quiesced before cutover? **2. Resource behavior during extended backend unavailability** The current code is fail-open, OpLogManager::Append() ) logs WARNING and continues on failure, and EraseMetadata() releases resources immediately. The RFC's move to fail-closed with deferred release is a great improvement, but if the backend is down for long, REMOVED-but-not-finalized replicas would accumulate while accepting=false blocks new writes. Would a backoff strategy, resource ceiling, or forced-failover trigger when stuck-batch age exceeds a threshold make sense?

bsakaguchi commented 2026-10-29 21:18:41 +00:00

Author

Owner

Copy link

@Icedcoco commented on GitHub (Jul 7, 2026):

@RuixiangMa Thanks, these are exactly the two points I was still hesitating about.

For the cutover baseline, I agree /latest is not obviously the only correct choice. My original reason for using /latest was to keep the cutover boundary aligned with the legacy published latest sequence, instead of treating any existing per-entry key as committed state. But as you pointed out, FlushBatch() and /latest are not advanced atomically today, and GetMaxSequenceId() already exists partly because /latest can be stale.

So I think there are two reasonable options:

1. keep /latest, but require an explicit quiesce + final flush + latest == max_seq guard before creating durable_prefix;
2. use GetMaxSequenceId() as the cutover baseline, and treat all existing per-entry keys up to that max sequence as the legacy boundary.

I do not have a strong final preference yet. Option 1 keeps the boundary closer to the old published state; option 2 may be more robust against /latest lag. I will think more about this and update the RFC once the tradeoff is clearer.

For extended backend unavailability, my current assumption was:

• the existing writing batch keeps retrying;
• the waiting/open queue stops accepting new entries;
• new mutating requests fail fast and return error to clients;
• the system then waits for backend recovery or HA failover.

This also implicitly assumes that leadership and OpLog both depend on etcd in the first version, so if etcd is unavailable, both write durability and HA coordination may be affected. That part definitely needs a more careful failure-mode discussion.

Your point about backoff strategy is good. We should at least define retry backoff to avoid hot-looping and log pressure, and probably expose stuck-batch age / pending-finalize resource usage so the supervisor can decide whether to stop serving or trigger failover. I still think we should not release REMOVED resources before durable prefix advances, but the operational policy around long unavailability needs more detail.

If you have a better preference between /latest vs GetMaxSequenceId(), or a cleaner failover policy for the long-backend-outage case, please share it. I am still open to adjusting this part of the design.

<!-- gh-comment-id:4904151633 --> @Icedcoco commented on GitHub (Jul 7, 2026): @RuixiangMa Thanks, these are exactly the two points I was still hesitating about. For the cutover baseline, I agree `/latest` is not obviously the only correct choice. My original reason for using `/latest` was to keep the cutover boundary aligned with the legacy published latest sequence, instead of treating any existing per-entry key as committed state. But as you pointed out, `FlushBatch()` and `/latest` are not advanced atomically today, and `GetMaxSequenceId()` already exists partly because `/latest` can be stale. So I think there are two reasonable options: 1. keep `/latest`, but require an explicit quiesce + final flush + `latest == max_seq` guard before creating `durable_prefix`; 2. use `GetMaxSequenceId()` as the cutover baseline, and treat all existing per-entry keys up to that max sequence as the legacy boundary. I do not have a strong final preference yet. Option 1 keeps the boundary closer to the old published state; option 2 may be more robust against `/latest` lag. I will think more about this and update the RFC once the tradeoff is clearer. For extended backend unavailability, my current assumption was: - the existing writing batch keeps retrying; - the waiting/open queue stops accepting new entries; - new mutating requests fail fast and return error to clients; - the system then waits for backend recovery or HA failover. This also implicitly assumes that leadership and OpLog both depend on etcd in the first version, so if etcd is unavailable, both write durability and HA coordination may be affected. That part definitely needs a more careful failure-mode discussion. Your point about backoff strategy is good. We should at least define retry backoff to avoid hot-looping and log pressure, and probably expose stuck-batch age / pending-finalize resource usage so the supervisor can decide whether to stop serving or trigger failover. I still think we should not release `REMOVED` resources before durable prefix advances, but the operational policy around long unavailability needs more detail. If you have a better preference between `/latest` vs `GetMaxSequenceId()`, or a cleaner failover policy for the long-backend-outage case, please share it. I am still open to adjusting this part of the design.

bsakaguchi commented 2026-10-29 21:18:41 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 6, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6008604726 --> @github-actions[bot] commented on GitHub (Oct 6, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:18:41 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 13, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:6137576963 --> @github-actions[bot] commented on GitHub (Oct 13, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:30:21 +00:00

[PR #740] [MERGED] add ascend direct transport #1787

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#719

No description provided.