bsakaguchi commented

Owner

Originally created by @yzhan1 on GitHub (May 4, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2031

Changes proposed

Motivation

Mooncake Store maintains a two-tier KVCache hierarchy: MEMORY replicas (DRAM, RDMA-direct, sub-millisecond reads) and LOCAL_DISK replicas (per-client SSD, multi-millisecond reads via a peer RPC plus disk I/O). Today, once a key has been evicted from DRAM down to LOCAL_DISK, it stays on LOCAL_DISK indefinitely. Every read for that key pays the SSD cost forever.

For inference-serving workloads this is a TTFT-shaped problem. A KV block that gets reused dozens of times after its first SSD-only lookup keeps generating SSD reads instead of being warmed back into DRAM.

This RFC proposes the mirror direction of #1899: when a Get hits a key whose only complete replicas are LOCAL_DISK, asynchronously create a new MEMORY replica so subsequent Gets for that key go fast. The structure mirrors offload-on-evict exactly — opt-in gflag, deferred per-shard queue, heartbeat-pull on the owning client, NotifyPromotionSuccess push back, refcnt-pinned source replica.

In Scope

Master-coordinated, opt-in promotion of LOCAL_DISK replicas to MEMORY, triggered by Get / GetReplicaList access.
Frequency-gated admission (master-side CountMinSketch) so one-shot reads don't waste DRAM.
High-watermark interlock so promotion never fires when DRAM is already under eviction pressure.
Refcnt pinning on the source LOCAL_DISK replica so it cannot be evicted while a promotion is in flight.
Symmetry with #1899: same gflag style, same per-shard task struct shape, parallel heartbeat RPC, parallel Notify…Success completion path.
C++ unit test + Python integration test.

Out of Scope

Promotion from non-LOCAL_DISK sources (DISK, remote DFS, NVMe-oF). LOCAL_DISK only in v1.
Master-driven proactive scan (e.g., periodic "promote all keys above frequency T"). v1 is access-triggered only.
Removing the source LOCAL_DISK replica after promotion. v1 retains it (cheap insurance against MEMORY eviction).

Background

Lifecycle today, and why this RFC is needed

A traced walkthrough of an object's lifecycle in main confirms there is no master-coordinated promotion today. Every relevant code path was inspected.

PutStart           → allocate buffer; insert ObjectMetadata with PROCESSING
                     MEMORY replica (master_service.cpp:854).
                     Returns OBJECT_ALREADY_EXISTS if key already exists. *
PutEnd             → mark replica COMPLETE; if enable_offload &&
                     !offload_on_evict, immediately PushOffloadingQueue;
                     GrantLease (master_service.cpp:910).
GetReplicaList     → return COMPLETE replicas in INSERTION ORDER. No
                     master-side preference for MEMORY over LOCAL_DISK
                     (master_service.cpp:758). Lease granted.

Eviction at high-watermark (one of three modes):
  Mode A  enable_offload=false             → erase MEMORY replicas; key
                                              persists with empty replica
                                              list; future Get returns
                                              REPLICA_IS_NOT_READY.
  Mode B  enable_offload=true,             → LOCAL_DISK replica was
          offload_on_evict=false             already created at PutEnd; just
                                              erase MEMORY.
  Mode C  enable_offload=true,             → pin one MEMORY replica via
          offload_on_evict=true (#1899)      refcnt, queue for deferred
                                              offload, evict redundant MEMORY
                                              copies; LOCAL_DISK appears via
                                              NotifyOffloadSuccess later.

Get on a LOCAL_DISK-only key:
  master.GetReplicaList → returns LOCAL_DISK descriptor.
  client.batch_get_into_offload_object_internal (real_client.cpp:4213):
       - RPC the LOCAL_DISK holder
       - holder reads its own SSD via file_storage_->BatchGet
       - requester does RDMA read into its destination buffer
       - release_offload_buffer  (fire-and-forget)
  *** No master_->PutStart, no AddReplica, no new MEMORY replica created. ***
  *** The metadata stays LOCAL_DISK-only forever; every subsequent Get  ***
  *** repeats the same SSD path.                                        ***

* Re-Put is rejected: PutStart on an existing key returns OBJECT_ALREADY_EXISTS. Upsert is the only path that succeeds, and it replaces the entire metadata (discards old replicas + reallocates) — it cannot add a MEMORY replica next to an existing LOCAL_DISK.

Background threads that mutate replica state (eviction, client monitor, task cleanup, snapshot, job dispatch) were each examined; none scan for hot LOCAL_DISK keys or perform any promotion-shaped work.

This is the gap Roadmap #1883 Milestone 13 names. Mooncake has the primitives in place — allocation strategy, RDMA write, replica add, refcnt pinning — but no policy that wires them together for the access-triggered case.

Design choices considered

Two coherent ways to close the gap exist; this RFC commits to (a).

(a) Master-coordinated, access-triggered promotion (this RFC). Master observes "Get on LOCAL_DISK-only key", applies frequency + watermark + dedup gates, enqueues a deferred promotion. Owning client picks up via heartbeat, reads its local SSD, allocates DRAM via the existing AllocationStrategy path, writes via Transfer Engine, master commits the new MEMORY replica. Mirrors #1899 seam-for-seam.
(b) Make Upsert compose-friendly so the engine re-Puts. Today's UpsertStart discards old replicas before reallocating; a variant that adds a new MEMORY replica next to existing LOCAL_DISK without dropping the old would let the engine Re-Put on every Get-from-disk. The engine becomes responsible for the policy.

Recommendation: (a). Three reasons:

Globally aware. Master sees access frequency and DRAM watermark across all callers. The engine sees neither. (a) makes admission decisions on the right signal; (b) shifts policy to a layer that doesn't have the data.
No engine cooperation needed. SGLang HiCache, vLLM HiCache, and the various other consumers don't need to be modified. (a) ships once and benefits every existing integration.
Symmetric with the offload-on-evict. The repo's mental model already accepts "master coordinates tier movement on internal triggers" — (a) extends that mental model in the obvious direction. (b) introduces a different ownership story (engine-driven tier policy) that the codebase does not currently use anywhere.

(b) is left available as a future option if a need ever arises for engine-driven cache control over the same surface; this RFC does not preclude it.

Why a deferred queue (not synchronous promotion at Get time)

A Get RPC must return fast. Synchronous DRAM allocation + SSD read + Transfer Engine write would add multi-ms to every cold-path Get and gate the response on writeable DRAM space. Deferring is also what offload-on-evict chose for the same reason. The Get path is unchanged from today's behavior; promotion happens out of band.

Proposal

Architecture

Two phases, mirroring #1899:

PHASE 1 — TRIGGER (synchronous, on the Get path)

   client.BatchGet(keys)
       │
       ▼ master.GetReplicaList(key)
           │
           ├─ has any complete MEMORY replica?  ── yes ─► return list, done
           │
           ├─ only LOCAL_DISK replicas?
           │     │
           │     ├─ promotion_sketch_.increment(key)  → count
           │     │
           │     ├─ count < admission_threshold?     ── yes ─► return list, done
           │     │
           │     ├─ DRAM used_ratio > high_watermark? ── yes ─► return list, done
           │     │
           │     ├─ promotion_tasks already has key?  ── yes ─► return list, done (dedup)
           │     │
           │     ├─ promotion_queue at limit?         ── yes ─► return list, done (cap)
           │     │
           │     └─ PushPromotionQueue(key, source_local_disk_replica)
           │           │
           │           ├─ source.inc_refcnt()        # pin so disk eviction can't drop it
           │           ├─ shard.promotion_tasks[key] = PromotionTask{source_id, size, now}
           │           └─ pending[client_owning_disk].objects[key] = size
           │
           ▼ return replica list  (unchanged — caller still reads from LOCAL_DISK)


PHASE 2 — DEFERRED PROMOTION (async, driven by the owning client's heartbeat)

   client (the LOCAL_DISK holder)
       │
       ▼ master.PromotionObjectHeartbeat(client_id) → map<key, size>
       │
       ▼ for each (key, size):
       │     bytes = file_storage_->BatchLoad(key)         # client reads its own disk
       │     mem_desc = master.PromotionAllocStart(key, size)
       │                  └─ master allocates DRAM via AllocationStrategy
       │                     and stages a PROCESSING MEMORY replica
       │     transfer_engine.WriteAt(mem_desc, bytes)      # bytes now in DRAM
       │     master.NotifyPromotionSuccess(client_id, key, mem_desc)
       │
       ▼ master:
           ├─ commits PROCESSING replica → COMPLETE
           ├─ source LOCAL_DISK replica.dec_refcnt()       # unpin
           └─ shard.promotion_tasks.erase(key)

After Phase 2 completes, the next Get for that key sees the MEMORY replica first and avoids the SSD path.

Config

New gflags in mooncake-store/src/master.cpp, added directly after the existing --offload_force_evict block:

DEFINE_bool(promotion_on_hit, false,
            "Promote LOCAL_DISK-only keys to MEMORY on read access");
DEFINE_uint32(promotion_admission_threshold, 2,
              "Min CountMinSketch count for a key before promotion fires "
              "(set 1 to disable second-touch gating)");
DEFINE_uint32(promotion_queue_limit, 50000,
              "Max in-flight promotion tasks across all shards; "
              "additional triggers are dropped silently");

Mirrored fields on MasterConfig, MasterServiceSupervisorConfig, MasterServiceConfig, and MasterServiceConfigBuilder — same propagation pattern as offload_on_evict_ / offload_force_evict_.

Master-side internals

// In include/master_service.h, alongside OffloadingTask (line ~842).
struct PromotionTask {
    ReplicaID source_id;           // the LOCAL_DISK replica being promoted
    uint64_t object_size;
    std::chrono::system_clock::time_point start_time;
};

// Per-shard map, parallel to offloading_tasks.
std::unordered_map<std::string, const PromotionTask> promotion_tasks
    GUARDED_BY(mutex);

// Per-master frequency sketch. Increments on every "only-LOCAL_DISK" Get.
// 4096 width × 4 depth, decayed periodically by the existing eviction
// thread tick.
std::unique_ptr<CountMinSketch> promotion_sketch_;

// Per-master config knobs read once at startup.
const bool promotion_on_hit_;
const uint32_t promotion_admission_threshold_;
const uint32_t promotion_queue_limit_;

Per-client pending state

Mirrors LocalDiskSegment::offloading_objects. Add a sibling map:

// In LocalDiskSegment (segment.h), alongside offloading_objects.
std::unordered_map<std::string, int64_t> GUARDED_BY(offloading_mutex_)
    promotion_objects;

The promoting client is always the same client that holds the LOCAL_DISK replica (it is the only node that can read the SSD bytes), so the existing LocalDiskSegment keyed by client UUID is the natural home.

New RPCs

Three new master-service RPCs, mirroring OffloadObjectHeartbeat / NotifyOffloadSuccess:

// 1) Heartbeat pull — the client owning LOCAL_DISK replicas asks master
//    for outstanding promotion work for itself.
auto PromotionObjectHeartbeat(const UUID& client_id)
    -> tl::expected<std::unordered_map<std::string, int64_t>, ErrorCode>;

// 2) Allocation — client requests a DRAM slot for the promoted key.
//    Master allocates via the existing AllocationStrategy path and stages
//    a PROCESSING MEMORY replica that is invisible to readers.
struct PromotionAllocStartResponse {
    Replica::Descriptor memory_descriptor;
};
auto PromotionAllocStart(const std::string& key, uint64_t size)
    -> tl::expected<PromotionAllocStartResponse, ErrorCode>;

// 3) Completion — client tells master the bytes are in DRAM.
//    Master commits the staged replica to COMPLETE, decrements the
//    source LOCAL_DISK replica's refcnt, and erases the task.
auto NotifyPromotionSuccess(const UUID& client_id, const std::string& key,
                            const Replica::Descriptor& memory_descriptor)
    -> tl::expected<void, ErrorCode>;

PromotionAllocStart reuses AllocationStrategy::AllocateFrom (already used by MasterService::PutStart and CopyStart), so DRAM allocation goes through the same admission path as normal Puts.

Get-path trigger

A small block added to MasterService::GetReplicaList (master_service.cpp:758-794), executed only when promotion_on_hit_ is true:

if (promotion_on_hit_) {
    bool any_memory = obj_metadata.HasReplica(&Replica::fn_is_memory_replica);
    bool any_local_disk = obj_metadata.HasReplica(
        &Replica::fn_is_local_disk_replica);
    if (!any_memory && any_local_disk) {
        TryPushPromotionQueue(key, obj_metadata, accessor.GetShard());
    }
}

TryPushPromotionQueue is a private method that does the gating: promotion_sketch_->increment(key) >= threshold, watermark check via MasterMetricManager::instance().get_global_mem_used_ratio(), dedup against shard->promotion_tasks, queue-limit check, then enqueue.

Race and edge-case handling

Source LOCAL_DISK replica evicted mid-flight. Refcnt pin in PushPromotionQueue prevents disk-side eviction (mirrors PushOffloadingQueue). If the disk replica is already pinned for unrelated reasons, refcnt simply increments and decrements normally.
Object evicted/deleted entirely. NotifyPromotionSuccess looks up metadata; if the key is gone, the staged MEMORY replica is freed via the existing PutRevoke-style cleanup and no replica is committed.
Multiple Gets in rapid succession. Dedup via shard->promotion_tasks.contains(key). Frequency-sketch increments are still cheap.
High DRAM watermark. Promotion is skipped at trigger time. The Get path is unchanged. The next Get re-evaluates the watermark.
Conflict with offload-on-evict. If both flags are on, an item could theoretically oscillate (DRAM evict → SSD → frequent reads → promote → DRAM → ... ). Mitigation: promotion_admission_threshold defaults to 2, so single-shot reads don't trigger; combined with the natural delay of the heartbeat tick this is empirically not a concern but tested explicitly.
Allocation failure. PromotionAllocStart returns an error; client logs and drops the task. Master's task entry is reaped on a configurable timeout (suggest 30 s), with refcnt decremented on the source replica.
Client owning LOCAL_DISK is gone. The promotion task in the master's shard becomes orphan; reaped by the same client-monitor thread that already cleans LocalDiskSegment on client expiry (per c58d1f9).

Backward compatibility

--promotion_on_hit defaults to false. Existing deployments see zero behavior change.
Snapshot/serialize unchanged. The new promotion_tasks map is in-flight state, not persisted (consistent with how offloading_tasks is treated). Restoring from a snapshot starts with an empty promotion queue; in-flight tasks are simply re-queued by the next Get on the same key.
No public-API changes. The new RPCs are master-internal and reachable only by clients that already speak the heartbeat protocol.

Metrics

Surfaced via the existing MasterMetricManager in the same shape as the offload counters that #1899 added:

promotion_attempts_total (counter) — every Get that found only LOCAL_DISK replicas, regardless of whether the promotion fired.
promotion_queued_total (counter) — successful PushPromotionQueue calls.
promotion_completed_total (counter) — NotifyPromotionSuccess calls.
promotion_failed_total (counter, by reason) — admission_gated, watermark_gated, queue_full, alloc_failed.
promotion_in_flight (gauge) — sum of shard->promotion_tasks.size() across shards.

These are minimal but enough to validate the feature in production telemetry without further per-class plumbing.

Tests

**mooncake-store/tests/promotion_on_hit_test.cpp** (new) — mirrors offload_on_evict_test.cpp. Cases:
- Default (flag off): no promotion ever fires; identical behavior to main.
- Single-shot Get on a LOCAL_DISK-only key: no promotion (admission gate at threshold=2).
- Repeated Gets above threshold: promotion fires; subsequent Get sees MEMORY replica.
- High-watermark gate: pre-fill DRAM, then trigger Gets — no promotion.
- Queue cap: synthetic burst on promotion_queue_limit distinct keys.
- Race: source LOCAL_DISK forcibly evicted while promotion task is queued.
**mooncake-wheel/tests/test_promotion_on_hit.py** (new) — mirrors test_offload_on_eviction.py. End-to-end:
- Put 100 keys, fill DRAM, observe LOCAL_DISK replicas appear.
- Hot-loop Get on 10 of those keys for threshold + 1 rounds.
- Verify those 10 keys now have MEMORY replicas; the other 90 do not.

Potential Next Steps

Stage	Scope
v1 (this RFC)	Master-coordinated, frequency-gated, per-shard queue. Opt-in via `--promotion_on_hit`. Self-contained PR.
v2	Master-driven proactive promotion based on access frequency tracking, decoupled from the Get path (so even passive holders can be promoted).
v3	Optional removal of the source LOCAL_DISK replica after promotion (configurable by `--promotion_drop_source`).

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @yzhan1 on GitHub (May 4, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2031 # Changes proposed ## Motivation Mooncake Store maintains a two-tier KVCache hierarchy: **MEMORY** replicas (DRAM, RDMA-direct, sub-millisecond reads) and **LOCAL_DISK** replicas (per-client SSD, multi-millisecond reads via a peer RPC plus disk I/O). Today, once a key has been evicted from DRAM down to LOCAL_DISK, it stays on LOCAL_DISK indefinitely. Every read for that key pays the SSD cost forever. For inference-serving workloads this is a TTFT-shaped problem. A KV block that gets reused dozens of times after its first SSD-only lookup keeps generating SSD reads instead of being warmed back into DRAM. This RFC proposes the **mirror direction of #1899**: when a Get hits a key whose only complete replicas are LOCAL_DISK, asynchronously create a new MEMORY replica so subsequent Gets for that key go fast. The structure mirrors offload-on-evict exactly — opt-in gflag, deferred per-shard queue, heartbeat-pull on the owning client, NotifyPromotionSuccess push back, refcnt-pinned source replica. ### In Scope - Master-coordinated, opt-in promotion of LOCAL_DISK replicas to MEMORY, triggered by Get / GetReplicaList access. - Frequency-gated admission (master-side CountMinSketch) so one-shot reads don't waste DRAM. - High-watermark interlock so promotion never fires when DRAM is already under eviction pressure. - Refcnt pinning on the source LOCAL_DISK replica so it cannot be evicted while a promotion is in flight. - Symmetry with #1899: same gflag style, same per-shard task struct shape, parallel heartbeat RPC, parallel `Notify…Success` completion path. - C++ unit test + Python integration test. ### Out of Scope - Promotion from non-LOCAL_DISK sources (DISK, remote DFS, NVMe-oF). LOCAL_DISK only in v1. - Master-driven proactive scan (e.g., periodic "promote all keys above frequency T"). v1 is access-triggered only. - Removing the source LOCAL_DISK replica after promotion. v1 retains it (cheap insurance against MEMORY eviction). ## Background ### Lifecycle today, and why this RFC is needed A traced walkthrough of an object's lifecycle in `main` confirms there is **no master-coordinated promotion** today. Every relevant code path was inspected. ``` PutStart → allocate buffer; insert ObjectMetadata with PROCESSING MEMORY replica (master_service.cpp:854). Returns OBJECT_ALREADY_EXISTS if key already exists. * PutEnd → mark replica COMPLETE; if enable_offload && !offload_on_evict, immediately PushOffloadingQueue; GrantLease (master_service.cpp:910). GetReplicaList → return COMPLETE replicas in INSERTION ORDER. No master-side preference for MEMORY over LOCAL_DISK (master_service.cpp:758). Lease granted. Eviction at high-watermark (one of three modes): Mode A enable_offload=false → erase MEMORY replicas; key persists with empty replica list; future Get returns REPLICA_IS_NOT_READY. Mode B enable_offload=true, → LOCAL_DISK replica was offload_on_evict=false already created at PutEnd; just erase MEMORY. Mode C enable_offload=true, → pin one MEMORY replica via offload_on_evict=true (#1899) refcnt, queue for deferred offload, evict redundant MEMORY copies; LOCAL_DISK appears via NotifyOffloadSuccess later. Get on a LOCAL_DISK-only key: master.GetReplicaList → returns LOCAL_DISK descriptor. client.batch_get_into_offload_object_internal (real_client.cpp:4213): - RPC the LOCAL_DISK holder - holder reads its own SSD via file_storage_->BatchGet - requester does RDMA read into its destination buffer - release_offload_buffer (fire-and-forget) *** No master_->PutStart, no AddReplica, no new MEMORY replica created. *** *** The metadata stays LOCAL_DISK-only forever; every subsequent Get *** *** repeats the same SSD path. *** ``` `*` Re-Put is rejected: `PutStart` on an existing key returns `OBJECT_ALREADY_EXISTS`. `Upsert` is the only path that succeeds, and it *replaces* the entire metadata (discards old replicas + reallocates) — it cannot add a MEMORY replica next to an existing LOCAL_DISK. Background threads that mutate replica state (eviction, client monitor, task cleanup, snapshot, job dispatch) were each examined; none scan for hot LOCAL_DISK keys or perform any promotion-shaped work. This is the gap Roadmap #1883 Milestone 13 names. Mooncake has the primitives in place — allocation strategy, RDMA write, replica add, refcnt pinning — but no policy that wires them together for the access-triggered case. ### Design choices considered Two coherent ways to close the gap exist; this RFC commits to (a). - **(a) Master-coordinated, access-triggered promotion (this RFC).** Master observes "Get on LOCAL_DISK-only key", applies frequency + watermark + dedup gates, enqueues a deferred promotion. Owning client picks up via heartbeat, reads its local SSD, allocates DRAM via the existing `AllocationStrategy` path, writes via Transfer Engine, master commits the new MEMORY replica. Mirrors #1899 seam-for-seam. - **(b) Make `Upsert` compose-friendly so the engine re-Puts.** Today's `UpsertStart` discards old replicas before reallocating; a variant that adds a new MEMORY replica next to existing LOCAL_DISK without dropping the old would let the engine Re-Put on every Get-from-disk. The engine becomes responsible for the policy. **Recommendation: (a).** Three reasons: - **Globally aware.** Master sees access frequency and DRAM watermark across all callers. The engine sees neither. (a) makes admission decisions on the right signal; (b) shifts policy to a layer that doesn't have the data. - **No engine cooperation needed.** SGLang HiCache, vLLM HiCache, and the various other consumers don't need to be modified. (a) ships once and benefits every existing integration. - **Symmetric with the offload-on-evict.** The repo's mental model already accepts "master coordinates tier movement on internal triggers" — (a) extends that mental model in the obvious direction. (b) introduces a different ownership story (engine-driven tier policy) that the codebase does not currently use anywhere. (b) is left available as a future option if a need ever arises for engine-driven cache control over the same surface; this RFC does not preclude it. ### Why a deferred queue (not synchronous promotion at Get time) A Get RPC must return fast. Synchronous DRAM allocation + SSD read + Transfer Engine write would add multi-ms to every cold-path Get and gate the response on writeable DRAM space. Deferring is also what offload-on-evict chose for the same reason. The Get path is unchanged from today's behavior; promotion happens out of band. ## Proposal ### Architecture Two phases, mirroring #1899: ``` PHASE 1 — TRIGGER (synchronous, on the Get path) client.BatchGet(keys) │ ▼ master.GetReplicaList(key) │ ├─ has any complete MEMORY replica? ── yes ─► return list, done │ ├─ only LOCAL_DISK replicas? │ │ │ ├─ promotion_sketch_.increment(key) → count │ │ │ ├─ count < admission_threshold? ── yes ─► return list, done │ │ │ ├─ DRAM used_ratio > high_watermark? ── yes ─► return list, done │ │ │ ├─ promotion_tasks already has key? ── yes ─► return list, done (dedup) │ │ │ ├─ promotion_queue at limit? ── yes ─► return list, done (cap) │ │ │ └─ PushPromotionQueue(key, source_local_disk_replica) │ │ │ ├─ source.inc_refcnt() # pin so disk eviction can't drop it │ ├─ shard.promotion_tasks[key] = PromotionTask{source_id, size, now} │ └─ pending[client_owning_disk].objects[key] = size │ ▼ return replica list (unchanged — caller still reads from LOCAL_DISK) PHASE 2 — DEFERRED PROMOTION (async, driven by the owning client's heartbeat) client (the LOCAL_DISK holder) │ ▼ master.PromotionObjectHeartbeat(client_id) → map<key, size> │ ▼ for each (key, size): │ bytes = file_storage_->BatchLoad(key) # client reads its own disk │ mem_desc = master.PromotionAllocStart(key, size) │ └─ master allocates DRAM via AllocationStrategy │ and stages a PROCESSING MEMORY replica │ transfer_engine.WriteAt(mem_desc, bytes) # bytes now in DRAM │ master.NotifyPromotionSuccess(client_id, key, mem_desc) │ ▼ master: ├─ commits PROCESSING replica → COMPLETE ├─ source LOCAL_DISK replica.dec_refcnt() # unpin └─ shard.promotion_tasks.erase(key) ``` After Phase 2 completes, the next Get for that key sees the MEMORY replica first and avoids the SSD path. ### Config New gflags in `mooncake-store/src/master.cpp`, added directly after the existing `--offload_force_evict` block: ```cpp DEFINE_bool(promotion_on_hit, false, "Promote LOCAL_DISK-only keys to MEMORY on read access"); DEFINE_uint32(promotion_admission_threshold, 2, "Min CountMinSketch count for a key before promotion fires " "(set 1 to disable second-touch gating)"); DEFINE_uint32(promotion_queue_limit, 50000, "Max in-flight promotion tasks across all shards; " "additional triggers are dropped silently"); ``` Mirrored fields on `MasterConfig`, `MasterServiceSupervisorConfig`, `MasterServiceConfig`, and `MasterServiceConfigBuilder` — same propagation pattern as `offload_on_evict_` / `offload_force_evict_`. ### Master-side internals ```cpp // In include/master_service.h, alongside OffloadingTask (line ~842). struct PromotionTask { ReplicaID source_id; // the LOCAL_DISK replica being promoted uint64_t object_size; std::chrono::system_clock::time_point start_time; }; // Per-shard map, parallel to offloading_tasks. std::unordered_map<std::string, const PromotionTask> promotion_tasks GUARDED_BY(mutex); // Per-master frequency sketch. Increments on every "only-LOCAL_DISK" Get. // 4096 width × 4 depth, decayed periodically by the existing eviction // thread tick. std::unique_ptr<CountMinSketch> promotion_sketch_; // Per-master config knobs read once at startup. const bool promotion_on_hit_; const uint32_t promotion_admission_threshold_; const uint32_t promotion_queue_limit_; ``` ### Per-client pending state Mirrors `LocalDiskSegment::offloading_objects`. Add a sibling map: ```cpp // In LocalDiskSegment (segment.h), alongside offloading_objects. std::unordered_map<std::string, int64_t> GUARDED_BY(offloading_mutex_) promotion_objects; ``` The promoting client is always the same client that holds the LOCAL_DISK replica (it is the only node that can read the SSD bytes), so the existing `LocalDiskSegment` keyed by client UUID is the natural home. ### New RPCs Three new master-service RPCs, mirroring `OffloadObjectHeartbeat` / `NotifyOffloadSuccess`: ```cpp // 1) Heartbeat pull — the client owning LOCAL_DISK replicas asks master // for outstanding promotion work for itself. auto PromotionObjectHeartbeat(const UUID& client_id) -> tl::expected<std::unordered_map<std::string, int64_t>, ErrorCode>; // 2) Allocation — client requests a DRAM slot for the promoted key. // Master allocates via the existing AllocationStrategy path and stages // a PROCESSING MEMORY replica that is invisible to readers. struct PromotionAllocStartResponse { Replica::Descriptor memory_descriptor; }; auto PromotionAllocStart(const std::string& key, uint64_t size) -> tl::expected<PromotionAllocStartResponse, ErrorCode>; // 3) Completion — client tells master the bytes are in DRAM. // Master commits the staged replica to COMPLETE, decrements the // source LOCAL_DISK replica's refcnt, and erases the task. auto NotifyPromotionSuccess(const UUID& client_id, const std::string& key, const Replica::Descriptor& memory_descriptor) -> tl::expected<void, ErrorCode>; ``` `PromotionAllocStart` reuses `AllocationStrategy::AllocateFrom` (already used by `MasterService::PutStart` and `CopyStart`), so DRAM allocation goes through the same admission path as normal Puts. ### Get-path trigger A small block added to `MasterService::GetReplicaList` (`master_service.cpp:758-794`), executed only when `promotion_on_hit_` is true: ```cpp if (promotion_on_hit_) { bool any_memory = obj_metadata.HasReplica(&Replica::fn_is_memory_replica); bool any_local_disk = obj_metadata.HasReplica( &Replica::fn_is_local_disk_replica); if (!any_memory && any_local_disk) { TryPushPromotionQueue(key, obj_metadata, accessor.GetShard()); } } ``` `TryPushPromotionQueue` is a private method that does the gating: `promotion_sketch_->increment(key) >= threshold`, watermark check via `MasterMetricManager::instance().get_global_mem_used_ratio()`, dedup against `shard->promotion_tasks`, queue-limit check, then enqueue. ### Race and edge-case handling - **Source LOCAL_DISK replica evicted mid-flight.** Refcnt pin in `PushPromotionQueue` prevents disk-side eviction (mirrors `PushOffloadingQueue`). If the disk replica is already pinned for unrelated reasons, refcnt simply increments and decrements normally. - **Object evicted/deleted entirely.** `NotifyPromotionSuccess` looks up metadata; if the key is gone, the staged MEMORY replica is freed via the existing `PutRevoke`-style cleanup and no replica is committed. - **Multiple Gets in rapid succession.** Dedup via `shard->promotion_tasks.contains(key)`. Frequency-sketch increments are still cheap. - **High DRAM watermark.** Promotion is skipped at trigger time. The Get path is unchanged. The next Get re-evaluates the watermark. - **Conflict with offload-on-evict.** If both flags are on, an item could theoretically oscillate (DRAM evict → SSD → frequent reads → promote → DRAM → ... ). Mitigation: `promotion_admission_threshold` defaults to 2, so single-shot reads don't trigger; combined with the natural delay of the heartbeat tick this is empirically not a concern but tested explicitly. - **Allocation failure.** `PromotionAllocStart` returns an error; client logs and drops the task. Master's task entry is reaped on a configurable timeout (suggest 30 s), with refcnt decremented on the source replica. - **Client owning LOCAL_DISK is gone.** The promotion task in the master's shard becomes orphan; reaped by the same client-monitor thread that already cleans `LocalDiskSegment` on client expiry (per `c58d1f9`). ### Backward compatibility - `--promotion_on_hit` defaults to `false`. Existing deployments see zero behavior change. - Snapshot/serialize unchanged. The new `promotion_tasks` map is in-flight state, not persisted (consistent with how `offloading_tasks` is treated). Restoring from a snapshot starts with an empty promotion queue; in-flight tasks are simply re-queued by the next Get on the same key. - No public-API changes. The new RPCs are master-internal and reachable only by clients that already speak the heartbeat protocol. ### Metrics Surfaced via the existing `MasterMetricManager` in the same shape as the offload counters that #1899 added: - `promotion_attempts_total` (counter) — every Get that found only LOCAL_DISK replicas, regardless of whether the promotion fired. - `promotion_queued_total` (counter) — successful PushPromotionQueue calls. - `promotion_completed_total` (counter) — NotifyPromotionSuccess calls. - `promotion_failed_total` (counter, by reason) — admission_gated, watermark_gated, queue_full, alloc_failed. - `promotion_in_flight` (gauge) — sum of `shard->promotion_tasks.size()` across shards. These are minimal but enough to validate the feature in production telemetry without further per-class plumbing. ## Tests - `**mooncake-store/tests/promotion_on_hit_test.cpp`** (new) — mirrors `offload_on_evict_test.cpp`. Cases: - Default (flag off): no promotion ever fires; identical behavior to main. - Single-shot Get on a LOCAL_DISK-only key: no promotion (admission gate at threshold=2). - Repeated Gets above threshold: promotion fires; subsequent Get sees MEMORY replica. - High-watermark gate: pre-fill DRAM, then trigger Gets — no promotion. - Queue cap: synthetic burst on `promotion_queue_limit` distinct keys. - Race: source LOCAL_DISK forcibly evicted while promotion task is queued. - `**mooncake-wheel/tests/test_promotion_on_hit.py`** (new) — mirrors `test_offload_on_eviction.py`. End-to-end: - Put 100 keys, fill DRAM, observe LOCAL_DISK replicas appear. - Hot-loop Get on 10 of those keys for `threshold + 1` rounds. - Verify those 10 keys now have MEMORY replicas; the other 90 do not. ## Potential Next Steps | Stage | Scope | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **v1 (this RFC)** | Master-coordinated, frequency-gated, per-shard queue. Opt-in via `--promotion_on_hit`. Self-contained PR. | | v2 | Master-driven proactive promotion based on access frequency tracking, decoupled from the Get path (so even passive holders can be promoted). | | v3 | Optional removal of the source LOCAL_DISK replica after promotion (configurable by `--promotion_drop_source`). | ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:16:28 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:16:29 +00:00

Author

Owner

@ykwd commented on GitHub (May 6, 2026):

Thanks for this great RFC. While there is an RFC (https://github.com/kvcache-ai/Mooncake/issues/2025) working on a similar area, may we need some collaboration in case there will be some conflict or duplicated work. @LujhCoconut

@ykwd commented on GitHub (May 6, 2026): Thanks for this great RFC. While there is an RFC (https://github.com/kvcache-ai/Mooncake/issues/2025) working on a similar area, may we need some collaboration in case there will be some conflict or duplicated work. @LujhCoconut

bsakaguchi commented

2026-10-29 21:16:29 +00:00

Author

Owner

@yzhan1 commented on GitHub (May 6, 2026):

@ykwd Thanks for bringing that up. Besides that does the RFC look good to your or any other comments? I have some local diffs based on the RFC that's ready for PR.

Hi @Srinivasoo7, saw you created https://github.com/kvcache-ai/Mooncake/issues/2025 and wondering if you'd like to work on this together? Are you interested in taking v2/v3 mentioned in the RFC?

@yzhan1 commented on GitHub (May 6, 2026): @ykwd Thanks for bringing that up. Besides that does the RFC look good to your or any other comments? I have some local diffs based on the RFC that's ready for PR. Hi @Srinivasoo7, saw you created https://github.com/kvcache-ai/Mooncake/issues/2025 and wondering if you'd like to work on this together? Are you interested in taking v2/v3 mentioned in the RFC?

bsakaguchi commented

2026-10-29 21:16:29 +00:00

Author

Owner

@Srinivasoo7 commented on GitHub (May 7, 2026):

Hi @yzhan1 ,
Glad to see this design land so quickly this is exactly the feature I opened #2025 to explore.
@ykwd thanks for flagging the overlap.

I like several parts of your proposal: async master-coordinated promotion, source replica pinning, heartbeat-pull, and the completion notification path. The symmetry with #1899 offload-on-evict structure is also the right call.

For a clean V1 I'd suggest keeping scope minimal:
Default-off --promotion_on_hit flag
Async best-effort promotion only no blocking on Get
Master-side metadata only (promotion tasks tracked per-shard on master, not on client)
Dedup in-flight promotion by key
Memory watermark guard (reuse existing offload_mem_watermark_high)
Source LOCAL_DISK refcnt pinning for the task lifetime
Tests covering: promotion success, allocation failure, and no behavior change when flag is disabled

CountMinSketch/frequency-gated admission, NotifyPromotionFailure fast-path, richer metrics, and v2/v3 policy can all be clean follow-up PRs once V1 lands.

Happy to collaborate on the implementation, I can take up the V2 and V3.

Also,
Would you be open to consolidating this into #2025 so we have a single tracking thread for Milestone 13 / #1883? Happy to update the #2025 body with your architecture and co-own the implementation

@Srinivasoo7 commented on GitHub (May 7, 2026): Hi @yzhan1 , Glad to see this design land so quickly this is exactly the feature I opened #2025 to explore. @ykwd thanks for flagging the overlap. I like several parts of your proposal: async master-coordinated promotion, source replica pinning, heartbeat-pull, and the completion notification path. The symmetry with #1899 offload-on-evict structure is also the right call. For a clean V1 I'd suggest keeping scope minimal: Default-off --promotion_on_hit flag Async best-effort promotion only no blocking on Get Master-side metadata only (promotion tasks tracked per-shard on master, not on client) Dedup in-flight promotion by key Memory watermark guard (reuse existing offload_mem_watermark_high) Source LOCAL_DISK refcnt pinning for the task lifetime Tests covering: promotion success, allocation failure, and no behavior change when flag is disabled CountMinSketch/frequency-gated admission, NotifyPromotionFailure fast-path, richer metrics, and v2/v3 policy can all be clean follow-up PRs once V1 lands. Happy to collaborate on the implementation, I can take up the V2 and V3. Also, Would you be open to consolidating this into #2025 so we have a single tracking thread for Milestone 13 / #1883? Happy to update the #2025 body with your architecture and co-own the implementation

bsakaguchi commented

2026-10-29 21:16:29 +00:00

Author

Owner

@yzhan1 commented on GitHub (May 7, 2026):

@Srinivasoo7 For sure, feel free to copy over and consolidate into one issue list. I don't seem to have edit access.

I will publish my changes for v1 soon and will tag you for review and context needed (if any) for v2/v3.

@yzhan1 commented on GitHub (May 7, 2026): @Srinivasoo7 For sure, feel free to copy over and consolidate into one issue list. I don't seem to have edit access. I will publish my changes for v1 soon and will tag you for review and context needed (if any) for v2/v3.

bsakaguchi commented

2026-10-29 21:16:30 +00:00

Author

Owner

@Srinivasoo7 commented on GitHub (May 7, 2026):

@yzhan1 Sure, looking foward to the PR!

@Srinivasoo7 commented on GitHub (May 7, 2026): @yzhan1 Sure, looking foward to the PR!

bsakaguchi commented

2026-10-29 21:16:30 +00:00

Author

Owner

@yzhan1 commented on GitHub (May 10, 2026):

Hi @ykwd @Srinivasoo7 PTAL at #2071. Thanks!

@yzhan1 commented on GitHub (May 10, 2026): Hi @ykwd @Srinivasoo7 PTAL at #2071. Thanks!

bsakaguchi commented

2026-10-29 21:16:31 +00:00

Author

Owner

@yzhan1 commented on GitHub (May 24, 2026):

@Srinivasoo7 Hi, now that v1 is complete, do you plan to work on v2/3? Let me know if you are busy working on other RFCs, I can continue to work on them if that's the case

@yzhan1 commented on GitHub (May 24, 2026): @Srinivasoo7 Hi, now that v1 is complete, do you plan to work on v2/3? Let me know if you are busy working on other RFCs, I can continue to work on them if that's the case

bsakaguchi commented

2026-10-29 21:16:31 +00:00

Author

Owner

@Srinivasoo7 commented on GitHub (May 24, 2026):

@yzhan1
Thanks for the headsup, I'll work on the v2 and let's collaborate on v3?
Let me know your thoughts!

@Srinivasoo7 commented on GitHub (May 24, 2026): @yzhan1 Thanks for the headsup, I'll work on the v2 and let's collaborate on v3? Let me know your thoughts!

bsakaguchi commented

2026-10-29 21:16:31 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Aug 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:16:31 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 31, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Aug 31, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:29:13 +00:00

[PR #562] [MERGED] refactor: introduce expected pattern for error handling in master service #1653

Rows
Columns

[GH-ISSUE #2031] [RFC]: Promote data from L2 -> L1 when it's not in L1 #562

Changes proposed

Motivation

In Scope

Out of Scope

Background

Lifecycle today, and why this RFC is needed

Design choices considered

Why a deferred queue (not synchronous promotion at Get time)

Proposal

Architecture

Config

Master-side internals

Per-client pending state

New RPCs

Get-path trigger

Race and edge-case handling

Backward compatibility

Metrics

Tests

Potential Next Steps

Before submitting a new issue...