kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4499] [RFC]: Add KVCacheStore as a shared capacity backend for Mooncake Store #1144

New issue

Open

opened 2026-10-29 21:23:19 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:19 +00:00

Owner

Originally created by @zxpdemonio on GitHub (Oct 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4499

Changes proposed

This RFC proposes an optional KVCacheStore Low Level adapter for Mooncake Store. The adapter uses the existing distributed-storage integration point, keeps Mooncake Master authoritative for logical object lifecycle, and accesses KVCacheStore through the node-local EFC data plane.

PR1 covers the smallest functional Low Level read/write/delete contract: one configured shared target and one provider value per object. The larger prototype remains Draft PR #4500 and is split into staged child PRs below; Standard mode is deferred until its public control-plane contract and validation environment are established.

The design requires explicit configuration and remains buildable when the KVCacheStore SDK is absent. It does not start EFC, infer enablement from a Unix socket, or copy provider placement state into Mooncake metadata.

Why this needs an RFC

Mooncake's contribution guide asks architectural changes larger than 500 lines to start with a GitHub RFC. This change affects storage metadata, request routing, failure handling, and the distributed-storage abstraction. It also overlaps with two existing proposals:

The driver itself is not the hard part. This RFC asks where a shared, provider-managed cache should attach to Mooncake and which parts of the lifecycle Mooncake should own.

Motivation

Mooncake already moves KV cache data between memory and external capacity tiers. KVCacheStore adds a provider-managed G3.5 capacity tier with a user-space KV interface. Its public deployment model places an EFC process beside the application; the SDK talks to EFC over a Unix socket and shared memory, while EFC handles the remote storage data path. The storage service can be attached to another compute node without copying data into that node first.

That model is useful for disaggregated inference, but it does not fit perfectly into Mooncake's current OSS offload path. The current OSS offload design treats the external object as a LOCAL_DISK replica owned by a real Mooncake client. Readers contact that owner, and the owner accesses OSS through FileStorage and DistributedStorageBackend.

KVCacheStore is different in one important respect: every correctly configured compute node has a local EFC endpoint for the same attached storage pool. Sending a read through the original writer adds an unnecessary owner hop and makes the writer part of the read availability path. The proposed adapter lets the requesting Mooncake client read the shared provider directly, after Mooncake Master has confirmed that the logical replica is valid.

The integration must preserve a clear metadata boundary:

Mooncake decides whether a logical object exists and whether its replica is committed.
KVCacheStore and EFC decide where provider records live and may reclaim them under capacity pressure.
The adapter translates between those two views without turning transient provider details into durable Mooncake metadata.

Goals

Add an optional KVCacheStore Low Level backend without changing existing deployments.
Reuse Mooncake's existing distributed-storage and object-adapter path where practical.
Keep Mooncake Master authoritative for logical object state and commit ordering.
Allow any configured Mooncake client to read the shared provider through its local EFC.
Define a portable, versioned layout for objects larger than one KVCacheStore value.
Treat provider eviction as a cache miss, never as valid data.
Bound Low Level provider operations and make partial outcomes explicit.
Preserve tenant separation in provider keys.
Keep the SDK optional at build time.
Split the implementation into reviewable pull requests with SDK-free tests.

Non-goals

Starting, supervising, or reconfiguring EFC from Mooncake.
Reimplementing provider capacity management, watermarks, compaction, or device GC.
Storing EFC mountpoints, device identities, or provider placement in Mooncake Master.
Adding Mooncake-side replication or migration between Low Level targets in the first release.
Making delete or upsert atomic across Mooncake Master and KVCacheStore.
Supporting multiple writers that bypass the Mooncake Master lifecycle.
Landing KVCacheStore Standard in the first community PR.
Changing the separate Mooncake Store RS cold-tier implementation.

Public KVCacheStore architecture

The following diagram is limited to behavior described in the public KVCacheStore documentation. It does not assume an internal control-plane topology.

flowchart LR
    subgraph Node[Compute node]
        App[Inference process]
        SDK[KVCacheStore Low Level SDK]
        EFC[kvcs-efc data-plane proxy]
        SHM[Shared-memory rings]

        App --> SDK
        SDK -->|Unix socket| EFC
        SDK <--> SHM
        EFC <--> SHM
    end

    EFC -->|record operations| G35[Attached KVCacheStore G3.5 tier]
    G35 --> Capacity[Provider-managed capacity and LRU]

The public integration guide establishes the constraints that matter to Mooncake:

the application and kvcs-efc run on the same node and under the same OS user;
the SDK connects to /var/run/kvcs/efc-grpc.sock by default;
data transfer uses shared-memory rings sized from value size, batch size, and worker counts;
Low Level provides batch put, query, get, and delete operations;
LRU eviction is part of the service and cannot be disabled;
applications own key-prefix isolation and must handle a miss by recomputing or fetching data elsewhere.

The broader product overview describes KVCacheStore as a capacity layer between compute-local memory and remote shared storage. Mooncake should use that capacity service, not duplicate it.

Relationship to existing Mooncake work

Design	Primary abstraction	Placement owner	Read path	Relationship to this RFC
Current OSS offload	`ObjectStorageAdapter` under `DistributedStorageBackend`	OSS/provider	reader contacts the offload-owner Mooncake client	Provides the closest existing integration point, but retains an owner hop.
#1957	NVMe KV backend, connector, executor, and control plane	Mooncake control plane	Mooncake routes to selected devices	Useful for Mooncake-owned devices. KVCacheStore already owns capacity placement and LRU, so duplicating that control plane would create two authorities.
#2993	`DistributedKVStorageBackend` and `IDistributedKVClient`	external distributed KV service	currently follows the `FileStorage`/owner model	Architecturally close. The community should decide whether KVCacheStore becomes its first driver or whether direct-reader semantics belong in the existing object-adapter path.
This RFC	shared provider adapter plus Master-authoritative reconciliation	KVCacheStore for physical records; Mooncake for logical lifecycle	requesting client uses its local EFC	Adds the direct-reader behavior and defines eviction, metadata, and chunk-layout semantics.

The prototype uses ObjectStorageAdapter because that path already carries an adapter identity and object size in the DFS replica descriptor. This RFC does not treat that choice as irreversible. If #2993 becomes the accepted common interface, the KVCacheStore driver should implement IDistributedKVClient, while the lifecycle and failure semantics below remain unchanged.

Proposed architecture

flowchart TB
    App[Application]
    Client[Mooncake client]
    Master[Mooncake Master]

    subgraph Requester[Requesting compute node]
        DSB[DistributedStorageBackend]
        Adapter[KVCacheStore adapter]
        SDK[Low Level SDK]
        EFC[Node-local EFC]
    end

    KVCS[Shared KVCacheStore capacity tier]

    App --> Client
    Client <-->|logical metadata and replica state| Master
    Client --> DSB --> Adapter --> SDK --> EFC --> KVCS

    Master -. stores .-> Logical["logical key, object size,<br/>state, adapter identity"]
    Adapter -. request-local only .-> Context["target and layout context"]
    KVCS -. owns .-> Physical["physical records, placement,<br/>capacity and eviction"]

There are three independent authorities:

Concern	Authority	Durable in Mooncake Master?
Logical key existence	Mooncake Master	Yes
Put state and commit point	Mooncake Master	Yes
Object size and adapter identity	Mooncake Master	Yes
Physical record keys and chunk layout	Versioned adapter contract	No provider locations are stored
Device placement and capacity	KVCacheStore/EFC	No
Provider query result	Requesting client	No; request-local only
LRU eviction	KVCacheStore	No; reconciled as a miss

This split avoids two bad states: the Master must not advertise provider data that was never committed, and it must not mistake provider placement details for stable metadata.

Access mode and initial scope

The first community landing contains only kvcs-lowlevel.

Property	Low Level, first landing	Standard, follow-up
Public integration contract	Documented	Not sufficiently documented for this RFC
Object layout	Defined by the Mooncake adapter	Defined by KVCacheStore
Provider query deadline	Can be bounded by the adapter	Prototype cannot bound every fallback RPC
Placement	Provider-managed below the selected EFC target	Provider control plane
Community status	Proposed	Experimental prototype only

Keeping Standard out of the first PR is a review and operability decision, not a claim that the mode is unworkable. It can follow once its public deployment contract, failure model, and validation environment are available.

Metadata model

Mooncake Master stores one committed DFS/object-storage replica descriptor containing:

the logical object size;
the adapter identity, kvcs-lowlevel;
the normal Mooncake replica state and ownership fields.

It does not store a KVCacheStore target, mountpoint index, EFC address, provider manifest, or device location. On snapshot restore, Mooncake must reject a provider descriptor whose adapter identity does not match the configured adapter. Reopening the same logical key with different physical-key rules is unsafe.

The provider context returned by query is attached only to the client request. It may be reused by the immediately following read for the same logical key and size. It must not be cached across unrelated requests or persisted by the Master.

Provider key and value layout

Tenant-safe physical keys

The physical root key is an injective printable encoding:

mc1:<tenant-id-hex>.<logical-key-hex>

The adapter validates the encoded root and every derived suffix against the configured provider key limit before issuing any SDK operation. Hex encoding is intentionally simple: it avoids delimiter ambiguity and keeps different tenants from mapping the same logical key to one provider record.

Inline and chunked values

flowchart TD
    Object[Logical Mooncake object]
    Check{"size <= configured<br/>max value size?"}
    Root["One record<br/>key = root<br/>value = full object"]
    Parts["Chunk records<br/>root.part_0 ... root.part_N"]
    Manifest["Manifest record<br/>root.manifest"]

    Object --> Check
    Check -->|yes| Root
    Check -->|no| Parts
    Check -->|no| Manifest
    Manifest -. describes .-> Parts

Small values use the root key directly. Larger values use deterministic .part_N keys plus a .manifest record. The manifest is a fixed 32-byte, versioned, little-endian wire format:

Offset	Size	Field
0	4	magic `MCMF`
4	2	format version
6	2	header size
8	4	chunk count
12	4	reserved, must be zero
16	8	total logical size
24	8	configured chunk size

The implementation must encode and decode fields explicitly. Serializing a C++ structure is not part of the format. Readers reject an unknown version, inconsistent size, zero chunk count, non-zero reserved field, missing chunk, or chunk whose size does not match the manifest.

The manifest is written last and acts as the provider-side visibility point. Its existence does not commit the Mooncake object; PutEnd at the Master remains the logical commit point.

Request flows

Put

sequenceDiagram
    participant A as Application
    participant C as Mooncake client
    participant M as Mooncake Master
    participant K as KVCS adapter
    participant E as Local EFC / KVCacheStore

    A->>C: Put(key, buffers)
    C->>M: PutStart(key, size, kvcs-lowlevel)
    M-->>C: PROCESSING replica descriptor
    C->>K: BatchPut(logical key, buffers)
    K->>E: put root OR chunks
    opt chunked object
        K->>E: put manifest last
    end
    alt provider write succeeds
        K-->>C: success
        C->>M: PutEnd(key)
        M-->>C: COMPLETE
        C-->>A: success
    else provider write fails
        K->>E: delete only records known to be inserted
        K-->>C: failure or indeterminate outcome
        C->>M: PutRevoke(key)
        C-->>A: error
    end

Put remains insert-only. Mooncake creates logical metadata first, writes the provider, and commits only after every required provider record is visible. Rollback deletes only records whose insertion is known to have succeeded. A timed-out SDK call is not proof of failure and is handled as an ambiguous outcome, described below.

Query and Get

sequenceDiagram
    participant A as Application
    participant C as Mooncake client
    participant M as Mooncake Master
    participant K as KVCS adapter
    participant E as Local EFC / KVCacheStore

    A->>C: Query/Get(key)
    par authoritative metadata query
        C->>M: Query(key)
        M-->>C: logical state and replicas
    and bounded provider probe
        C->>K: QueryProvider(key, deadline)
        K->>E: batch query root/manifest
        E-->>K: hit, miss, or error
        K-->>C: request-local provider context
    end

    alt complete memory replica exists
        C-->>A: use memory replica
    else Master confirms complete KVCS replica and provider hit
        C->>K: Get(key, expected size, context)
        K->>E: batch get root or chunks
        E-->>K: values
        K-->>C: verified object
        C-->>A: object
    else provider miss or incomplete layout
        C-->>A: cache miss / replica unavailable
    else Master has no complete KVCS replica
        C-->>A: ignore provider result
    end

The Master result is always authoritative. A provider hit cannot create a Mooncake object, resurrect a deleted key, or override a memory replica. Conversely, a complete Master descriptor cannot turn an EFC miss into data. The read succeeds only when both views agree.

Low Level provider query may run in parallel with the Master query because the SDK call has an absolute deadline. Speculation is an optimization: its result is ignored when the Master returns a usable memory replica or no complete KVCacheStore replica.

Remove and retry

flowchart TD
    Start[Remove logical key]
    Master[Remove Mooncake metadata]
    Provider[Delete provider records]
    Done[Done]
    Retry[Retry cleanup with same key]
    Orphan["Provider records may remain,<br/>but are no longer readable through Mooncake"]

    Start --> Master
    Master -->|removed or already absent| Provider
    Provider -->|success or confirmed absent| Done
    Provider -->|transient failure| Orphan
    Orphan --> Retry --> Provider

The first implementation removes Master metadata before deleting provider records. This ordering prefers an unreachable orphan over metadata that points to missing data. A retry is allowed to continue provider deletion when the Master already reports OBJECT_NOT_FOUND.

KVCacheStore Low Level does not provide key enumeration for Mooncake-owned prefixes, so the first landing cannot promise background orphan discovery. Durable cleanup tasks are an open design item, not an implicit capability.

Upsert

Upsert is Remove followed by Put. It is not atomic. There is an interval in which the key is absent, and a failed second step leaves it absent. Callers that require compare-and-swap or uninterrupted replacement need a future generation-aware API.

Eviction and stale metadata

Provider eviction is expected behavior in Low Level mode. It creates a specific consistency case: Mooncake Master may still contain a complete KVCacheStore replica descriptor after its provider records have been evicted.

The adapter handles this conservatively:

provider query reports a miss or incomplete layout;
the client does not advertise the KVCacheStore replica as readable;
if no other complete replica exists, the operation returns a cache miss or replica-unavailable result;
the caller may recompute the object and replace it through the normal Remove/Put lifecycle.

The first PR must also define how stale provider replicas are invalidated at the Master. The preferred design is a conditional invalidation tied to the queried replica identity or epoch, so a late miss cannot remove a newer Put. If the existing Master API cannot express that safely, automatic invalidation remains disabled and the limitation must be documented before landing.

Returning a miss is safe; silently claiming that the provider still contains the object is not. This point is a release blocker because provider LRU cannot be disabled.

Target selection and availability

An EFC configuration may expose more than one Low Level target. The prototype computes a stable target order from the encoded root key, tries the first target, and falls back to other healthy targets for the same operation.

Fallback is not replication. The first release writes one target and must not advertise several targets as independent Mooncake replicas. A read fallback is useful only when the object can legitimately exist on another configured target; otherwise it adds latency without durability. The initial community implementation should therefore start with one explicit shared target unless maintainers confirm that multi-target lookup is part of the supported deployment contract.

Mooncake does not persist target indices. A configuration change may reorder mountpoints, so any multi-target design must identify targets by stable provider identity rather than array position before it can be considered durable.

Failure semantics

Situation	Required behavior
Provider miss with no Master replica	Return miss; do not create metadata.
Provider hit with no complete Master replica	Ignore the hit.
Complete Master replica plus provider miss	Treat the provider replica as unavailable; never return uninitialized data.
Missing or malformed manifest/chunk	Return incomplete/corrupt result; do not assemble a partial object.
Put fails after known chunk inserts	Delete only the known inserted records, then revoke the Master Put.
Put outcome is ambiguous	Quarantine affected physical keys in-process and fail closed.
Delete outcome is ambiguous	Do not issue an unsafe blind retry against a potentially newer value.
Master commit fails after provider Put	Attempt provider cleanup and return failure.
Adapter identity differs during restore	Fail closed.
SDK or EFC unavailable at startup	Fail explicit KVCacheStore configuration; never fall back silently to another backend.

Low Level has no generation token or conditional delete. A timeout therefore cannot distinguish “the provider did nothing” from “the provider completed but the reply was lost.” The prototype quarantines affected physical keys in the current process until restart. This prevents immediate unsafe reuse, but it is not durable fencing. Cross-process recovery requires provider generations or a Mooncake-owned generation in every physical key.

Configuration and runtime contract

KVCacheStore is enabled only by explicit configuration. Socket presence is not an activation signal.

export MOONCAKE_KVCS_MODE=low-level
export MOONCAKE_KVCS_EFC_SOCKET=/var/run/kvcs/efc-grpc.sock
export MOONCAKE_KVCS_SINGLE_TENANT=true
export MOONCAKE_KVCS_MAX_VALUE_SIZE=4194304
export MOONCAKE_KVCS_MAX_KEYS_PER_BATCH=8
export MOONCAKE_KVCS_GET_WORKERS=1
export MOONCAKE_KVCS_SET_WORKERS=1
export MOONCAKE_KVCS_SIMPLE_WORKERS=1
export MOONCAKE_KVCS_OPERATION_TIMEOUT_MS=30000

The values above are safe starting points, not universal tuning recommendations. The KVCacheStore integration guide sizes shared memory approximately from:

max_value_size * max_keys_per_batch *
    (get_workers + set_workers + simple_workers)

Mooncake validates the effective configuration before creating the adapter. It does not create the socket, start EFC, modify /dev/shm, or rewrite provider configuration. Runtime packaging must mount the EFC socket and allocate enough shared memory inside the application container.

The initial scope is single-tenant because the current Mooncake distributed-storage configuration is single-tenant. The physical key still contains the tenant ID so the format does not bake in a cross-tenant collision.

Build and dependency model

The KVCacheStore SDK remains optional. CMake discovers it through pkg-config and compiles a NOT_SUPPORTED factory when it is absent. Existing Mooncake builds and deployments acquire no new runtime dependency.

The community repository should not vendor KVCacheStore binaries or implementation sources. SDK-independent behavior—key encoding, manifests, chunking, reconciliation, deadlines, rollback planning, and error mapping—must be testable through an injected driver. Tests that call the public C ABI can run in an optional provider-enabled job.

Published validation evidence must come from the containerized toolchain used for the prototype and provider SDK tests. Host-only compilation or network checks are not acceptance evidence.

Compatibility

Existing filesystem and OSS adapters keep their current configuration and behavior.
Regular DFS replica serialization remains readable without KVCacheStore support.
Provider descriptors carry an explicit adapter identity and object size.
Snapshot restore fails closed on an unknown or mismatched adapter.
A build without the SDK continues to compile and returns NOT_SUPPORTED only when KVCacheStore is explicitly requested.
Automatic socket-based activation is excluded.
The legacy prototype value MOONCAKE_KVCS_MODE=kvcs may map to low-level for one transition period, but new documentation must not use it.

Security and isolation

Logical keys are encoded before crossing the provider boundary; raw binary keys are not used as path-like strings.
Key size, value size, batch dimensions, buffer ranges, and aggregate lengths are checked before an SDK call.
EFC configuration and environment values are untrusted input and are rejected on invalid type, duplicate target identity, or out-of-range index.
Provider credentials, if introduced by a later mode, remain runtime secrets and are neither persisted in Master metadata nor logged.
The adapter opens no listener and adds no authentication surface to Mooncake. Access to the EFC socket and shared memory remains a deployment responsibility.

Observability

The adapter should expose, at minimum:

operation count, bytes, and latency by operation;
query hit, miss, incomplete, timeout, and provider error;
root versus chunked layout;
rollback attempts and failures;
quarantined-key count;
stale-replica invalidation attempts and outcomes;
selected target identity when multi-target support is enabled.

Misses are expected cache events and should not be logged as transport failures. Malformed layouts and ambiguous outcomes deserve separate error labels because they require different operational responses.

Validation plan

No implementation PR should be marked ready until the rebased commit has the following evidence from that container environment.

Build matrix

Build without the KVCacheStore SDK.
Build with the supported SDK and official mock library.
Run changed-line formatting and the repository's normal pre-commit checks.
Verify that OSS and filesystem-only builds are unchanged.

Deterministic tests

physical-key encoding and length boundaries;
32-byte manifest golden vectors and malformed-input rejection;
inline and multi-chunk put/query/get/delete;
missing chunk, wrong size, and corrupt manifest;
provider hit with absent or incomplete Master metadata;
memory-replica priority over a speculative provider result;
operation deadline and cancellation behavior;
known-partial rollback and ambiguous-outcome quarantine;
snapshot restore with matching and mismatched adapter identity;
provider eviction followed by stale-replica invalidation;
SDK-absent factory behavior.

Real EFC validation

Put, Query, Get, and Remove through the node-local EFC socket.
Payload verification across process and container restart.
Values below, at, and above the configured chunk threshold.
Multi-buffer reads.
Forced provider eviction and the subsequent Mooncake miss path.
EFC unavailable, delayed, and restarted during an operation.
Shared-memory sizing documented with the exact test configuration.

Performance numbers should be reported separately from correctness evidence, with hardware, EFC configuration, object-size distribution, concurrency, and baseline commit recorded.

Revised staged delivery (supersedes the earlier all-in-one sequence)

This section supersedes the earlier all-in-one implementation plan. The
existing Draft PR #4500 is a full prototype and remains available for
reference; PR 1 is a separate, smaller branch based on community main. Each
stage below should be reviewed and validated on its own commit.

PR 1: minimal Low Level read/write/delete (the first child PR of #4500). One configured, shared EFC target;
one KVCacheStore value per object; 4 MiB default per-value limit. Keep the
existing distributed-storage adapter and Master-authoritative placement
path. Implement insert-only Put, Query, Get and Delete and quarantine ambiguous
writes. Query Master first;
perform any needed provider query within a bounded deadline only for
matching Master KVCS replicas. Query reports existence, not a manifest;
Get validates its bytes against the Master object size. Reject oversize
values before calling the SDK, and fail
initialization on multi-target or incompatible mixed deployments. Bulk
RemoveByRegex and RemoveAll explicitly report unsupported until
provider-wide deletion has a safe contract. Without the optional SDK,
other backends still build
and explicit KVCS initialization reports NOT_SUPPORTED. Exit criterion:
focused SDK/no-SDK builds and correctness tests, plus the existing
single-value real-EFC correctness evidence. Single-key Remove and
BatchRemove are included. Because PR 1 is insert-only, they delete the
physical value before releasing the Master name; a concurrent Put therefore
cannot publish a new value that this Remove later deletes. A provider miss
is idempotent. If Master rejects the removal after the provider delete, the
error is returned and the logical metadata remains for retry; generation
IDs or a Master-side reservation can provide atomic cross-system deletion
in a later protocol. Bulk RemoveByRegex and RemoveAll remain unsupported.
No chunks, multi-target router, eviction repair or metrics.
PR 2: large objects and, subject to a defined deployment contract,
multi-target routing. A separate stacked child PR based on PR 1 adds
versioned manifests, deterministic chunks, bounded batch handling,
cross-chunk rollback and uncertain-outcome isolation. Only add stable
multi-target routing once target identity and mixed-deployment semantics
are agreed. Validate boundaries, failed/partial operations and recovery
against the actual deployment. If these responsibilities are too large for
one review, submit them as two sequential child PRs; PR 1 remains usable.
Eviction repair and operational hardening. Conditionally invalidate
stale Master replicas after provider misses, with race tests and operator
recovery guidance. PR 1 fails closed on a provider miss; it does not
automatically correct stale Master metadata.
Dedicated metrics PR. Bring back the operation metrics preserved in
the separate metrics backup branch. Keep it out of PR 1 and do not publish
performance figures in PR descriptions, comments or logs.
Standard mode follow-up. Scope this separately once the public API
contract and validation environment have been established.

The earlier full prototype has already passed real-EFC correctness
validation. Retain that result as historical evidence. The cleanup head has
separate focused SDK/no-SDK evidence; do not turn the historical run into a
claim about a new live-EFC execution of this cleanup head.

Corresponding corrections to the existing RFC sections

Architecture and data layout: the PR 1 data path is one SDK value per
object on a single shared mountpoint. The chunk format, manifest and target
selection are PR 2 designs, not claims about the PR 1 code.
Read path: complete the Master lookup before any bounded provider query;
only Master-advertised KVCS replicas can be read from the provider. There is
no speculative parallel Master/provider query in PR 1. Provider query
only checks presence. The Master descriptor supplies the expected
size; Get checks the actual provider response. There is no second manifest
or per-request shard context to serialize.
Failure and recovery: PR 1 rejects unsupported topology and oversize
writes, preserves ambiguous-outcome isolation, treats a missing provider
value as an idempotent delete, and leaves Master metadata for retry if its
removal is rejected after the physical delete. Automatic stale-replica
cleanup is stage 3; bulk deletion remains unsupported.
Validation: separate PR 1 single-value read/write, error-path,
SDK-optional build and real-EFC checks from PR 2 chunk/multi-target and
stage 3 eviction-race cases. The prototype's completed real-EFC run and the
PR 1 head's run must be labeled independently.

Alternatives considered

Implement KVCacheStore directly under #2993

This is a viable outcome and may be the cleanest community direction. The missing piece is direct-reader semantics: the current owner-oriented path should not require every shared-provider read to traverse the original writer. If #2993 accepts request-local provider queries and direct reads, this RFC's adapter can live there.

Reuse the current OSS owner path unchanged

This avoids client changes but keeps the original writer in the data path. It weakens availability and adds a network hop even though the requesting node already has local access to the shared provider.

Reuse the #1957 Mooncake control plane

That design suits Mooncake-owned NVMe devices. Applying it unchanged to KVCacheStore would duplicate provider placement, capacity, LRU, and health decisions. Mooncake would hold state it cannot keep authoritative.

Persist provider target locations in Master metadata

Rejected. EFC topology and provider placement may change independently of Mooncake snapshots. Persisting them would convert a runtime routing hint into stale protocol state.

Enable KVCacheStore whenever the EFC socket exists

Rejected. A mounted socket does not prove that every client and the Master agree on adapter mode, key format, tenant, or target set. Explicit configuration makes mismatches visible.

Land Standard and Low Level together

Deferred. Low Level has a public contract and bounded record operations. Standard introduces a second metadata authority and currently lacks the same public failure and deadline model. Combining both would make the first review broader without proving the shared path better.

Open questions

Should KVCacheStore attach through the existing ObjectStorageAdapter, or become the first concrete driver under #2993?
What is the preferred Master API for conditionally invalidating an evicted provider replica without racing a newer Put?
Should the first release allow only one shared Low Level target?
Is a durable provider-cleanup queue required before landing, or is explicit retry plus observability acceptable initially?
Which SDK version and real-EFC environment can be used for community CI or periodic validation?
Which Mooncake error should represent “Master replica exists, provider has evicted it” so upper layers can treat it as a cache miss?

Requested decision

This RFC asks maintainers to agree on five points before implementation review begins:

KVCacheStore Low Level is useful as an optional shared capacity backend.
Mooncake Master owns logical lifecycle; KVCacheStore owns physical capacity and placement.
Reads may go directly through the requesting client's local EFC after Master-authoritative reconciliation.
Provider eviction must have a safe stale-metadata invalidation path.
The first landing is Low Level only, with Standard handled separately.

Once those boundaries are settled, the implementation can be rebased and split without asking reviewers to infer architecture from a large prototype diff.

Before submitting

I searched existing issues and read the Mooncake documentation. This proposal is related to #1957 and #2993 and does not duplicate either design.

Originally created by @zxpdemonio on GitHub (Oct 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4499 ## Changes proposed This RFC proposes an optional KVCacheStore Low Level adapter for Mooncake Store. The adapter uses the existing distributed-storage integration point, keeps Mooncake Master authoritative for logical object lifecycle, and accesses KVCacheStore through the node-local EFC data plane. PR1 covers the smallest functional Low Level read/write/delete contract: one configured shared target and one provider value per object. The larger prototype remains Draft PR #4500 and is split into staged child PRs below; Standard mode is deferred until its public control-plane contract and validation environment are established. The design requires explicit configuration and remains buildable when the KVCacheStore SDK is absent. It does not start EFC, infer enablement from a Unix socket, or copy provider placement state into Mooncake metadata. ## Why this needs an RFC Mooncake's [contribution guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) asks architectural changes larger than 500 lines to start with a GitHub RFC. This change affects storage metadata, request routing, failure handling, and the distributed-storage abstraction. It also overlaps with two existing proposals: - [#1957: Mooncake Store NVMe KV Backend](https://github.com/kvcache-ai/Mooncake/issues/1957) - [#2993: Distributed KV Storage Backend for Mooncake Store](https://github.com/kvcache-ai/Mooncake/issues/2993) The driver itself is not the hard part. This RFC asks where a shared, provider-managed cache should attach to Mooncake and which parts of the lifecycle Mooncake should own. ## Motivation Mooncake already moves KV cache data between memory and external capacity tiers. KVCacheStore adds a provider-managed G3.5 capacity tier with a user-space KV interface. Its public deployment model places an EFC process beside the application; the SDK talks to EFC over a Unix socket and shared memory, while EFC handles the remote storage data path. The storage service can be attached to another compute node without copying data into that node first. That model is useful for disaggregated inference, but it does not fit perfectly into Mooncake's current OSS offload path. The current [OSS offload design](https://kvcache-ai.github.io/Mooncake/deployment/oss-offload.html) treats the external object as a `LOCAL_DISK` replica owned by a real Mooncake client. Readers contact that owner, and the owner accesses OSS through `FileStorage` and `DistributedStorageBackend`. KVCacheStore is different in one important respect: every correctly configured compute node has a local EFC endpoint for the same attached storage pool. Sending a read through the original writer adds an unnecessary owner hop and makes the writer part of the read availability path. The proposed adapter lets the requesting Mooncake client read the shared provider directly, after Mooncake Master has confirmed that the logical replica is valid. The integration must preserve a clear metadata boundary: - Mooncake decides whether a logical object exists and whether its replica is committed. - KVCacheStore and EFC decide where provider records live and may reclaim them under capacity pressure. - The adapter translates between those two views without turning transient provider details into durable Mooncake metadata. ## Goals - Add an optional KVCacheStore Low Level backend without changing existing deployments. - Reuse Mooncake's existing distributed-storage and object-adapter path where practical. - Keep Mooncake Master authoritative for logical object state and commit ordering. - Allow any configured Mooncake client to read the shared provider through its local EFC. - Define a portable, versioned layout for objects larger than one KVCacheStore value. - Treat provider eviction as a cache miss, never as valid data. - Bound Low Level provider operations and make partial outcomes explicit. - Preserve tenant separation in provider keys. - Keep the SDK optional at build time. - Split the implementation into reviewable pull requests with SDK-free tests. ## Non-goals - Starting, supervising, or reconfiguring EFC from Mooncake. - Reimplementing provider capacity management, watermarks, compaction, or device GC. - Storing EFC mountpoints, device identities, or provider placement in Mooncake Master. - Adding Mooncake-side replication or migration between Low Level targets in the first release. - Making delete or upsert atomic across Mooncake Master and KVCacheStore. - Supporting multiple writers that bypass the Mooncake Master lifecycle. - Landing KVCacheStore Standard in the first community PR. - Changing the separate Mooncake Store RS cold-tier implementation. ## Public KVCacheStore architecture The following diagram is limited to behavior described in the public KVCacheStore documentation. It does not assume an internal control-plane topology. ```mermaid flowchart LR subgraph Node[Compute node] App[Inference process] SDK[KVCacheStore Low Level SDK] EFC[kvcs-efc data-plane proxy] SHM[Shared-memory rings] App --> SDK SDK -->|Unix socket| EFC SDK <--> SHM EFC <--> SHM end EFC -->|record operations| G35[Attached KVCacheStore G3.5 tier] G35 --> Capacity[Provider-managed capacity and LRU] ``` The public [integration guide](https://www.alibabacloud.com/help/en/kvcachestore/best-practices-for-integrating-the-kvcachestore-sdk) establishes the constraints that matter to Mooncake: - the application and `kvcs-efc` run on the same node and under the same OS user; - the SDK connects to `/var/run/kvcs/efc-grpc.sock` by default; - data transfer uses shared-memory rings sized from value size, batch size, and worker counts; - Low Level provides batch put, query, get, and delete operations; - LRU eviction is part of the service and cannot be disabled; - applications own key-prefix isolation and must handle a miss by recomputing or fetching data elsewhere. The broader [product overview](https://www.alibabacloud.com/help/en/kvcachestore/product-overview/what-is-kvcachestore) describes KVCacheStore as a capacity layer between compute-local memory and remote shared storage. Mooncake should use that capacity service, not duplicate it. ## Relationship to existing Mooncake work | Design | Primary abstraction | Placement owner | Read path | Relationship to this RFC | | --- | --- | --- | --- | --- | | Current OSS offload | `ObjectStorageAdapter` under `DistributedStorageBackend` | OSS/provider | reader contacts the offload-owner Mooncake client | Provides the closest existing integration point, but retains an owner hop. | | [#1957](https://github.com/kvcache-ai/Mooncake/issues/1957) | NVMe KV backend, connector, executor, and control plane | Mooncake control plane | Mooncake routes to selected devices | Useful for Mooncake-owned devices. KVCacheStore already owns capacity placement and LRU, so duplicating that control plane would create two authorities. | | [#2993](https://github.com/kvcache-ai/Mooncake/issues/2993) | `DistributedKVStorageBackend` and `IDistributedKVClient` | external distributed KV service | currently follows the `FileStorage`/owner model | Architecturally close. The community should decide whether KVCacheStore becomes its first driver or whether direct-reader semantics belong in the existing object-adapter path. | | This RFC | shared provider adapter plus Master-authoritative reconciliation | KVCacheStore for physical records; Mooncake for logical lifecycle | requesting client uses its local EFC | Adds the direct-reader behavior and defines eviction, metadata, and chunk-layout semantics. | The prototype uses `ObjectStorageAdapter` because that path already carries an adapter identity and object size in the DFS replica descriptor. This RFC does not treat that choice as irreversible. If #2993 becomes the accepted common interface, the KVCacheStore driver should implement `IDistributedKVClient`, while the lifecycle and failure semantics below remain unchanged. ## Proposed architecture ```mermaid flowchart TB App[Application] Client[Mooncake client] Master[Mooncake Master] subgraph Requester[Requesting compute node] DSB[DistributedStorageBackend] Adapter[KVCacheStore adapter] SDK[Low Level SDK] EFC[Node-local EFC] end KVCS[Shared KVCacheStore capacity tier] App --> Client Client <-->|logical metadata and replica state| Master Client --> DSB --> Adapter --> SDK --> EFC --> KVCS Master -. stores .-> Logical["logical key, object size, state, adapter identity"] Adapter -. request-local only .-> Context["target and layout context"] KVCS -. owns .-> Physical["physical records, placement, capacity and eviction"] ``` There are three independent authorities: | Concern | Authority | Durable in Mooncake Master? | | --- | --- | --- | | Logical key existence | Mooncake Master | Yes | | Put state and commit point | Mooncake Master | Yes | | Object size and adapter identity | Mooncake Master | Yes | | Physical record keys and chunk layout | Versioned adapter contract | No provider locations are stored | | Device placement and capacity | KVCacheStore/EFC | No | | Provider query result | Requesting client | No; request-local only | | LRU eviction | KVCacheStore | No; reconciled as a miss | This split avoids two bad states: the Master must not advertise provider data that was never committed, and it must not mistake provider placement details for stable metadata. ## Access mode and initial scope The first community landing contains only `kvcs-lowlevel`. | Property | Low Level, first landing | Standard, follow-up | | --- | --- | --- | | Public integration contract | Documented | Not sufficiently documented for this RFC | | Object layout | Defined by the Mooncake adapter | Defined by KVCacheStore | | Provider query deadline | Can be bounded by the adapter | Prototype cannot bound every fallback RPC | | Placement | Provider-managed below the selected EFC target | Provider control plane | | Community status | Proposed | Experimental prototype only | Keeping Standard out of the first PR is a review and operability decision, not a claim that the mode is unworkable. It can follow once its public deployment contract, failure model, and validation environment are available. ## Metadata model Mooncake Master stores one committed DFS/object-storage replica descriptor containing: - the logical object size; - the adapter identity, `kvcs-lowlevel`; - the normal Mooncake replica state and ownership fields. It does not store a KVCacheStore target, mountpoint index, EFC address, provider manifest, or device location. On snapshot restore, Mooncake must reject a provider descriptor whose adapter identity does not match the configured adapter. Reopening the same logical key with different physical-key rules is unsafe. The provider context returned by query is attached only to the client request. It may be reused by the immediately following read for the same logical key and size. It must not be cached across unrelated requests or persisted by the Master. ## Provider key and value layout ### Tenant-safe physical keys The physical root key is an injective printable encoding: ```text mc1:<tenant-id-hex>.<logical-key-hex> ``` The adapter validates the encoded root and every derived suffix against the configured provider key limit before issuing any SDK operation. Hex encoding is intentionally simple: it avoids delimiter ambiguity and keeps different tenants from mapping the same logical key to one provider record. ### Inline and chunked values ```mermaid flowchart TD Object[Logical Mooncake object] Check{"size <= configured max value size?"} Root["One record key = root value = full object"] Parts["Chunk records root.part_0 ... root.part_N"] Manifest["Manifest record root.manifest"] Object --> Check Check -->|yes| Root Check -->|no| Parts Check -->|no| Manifest Manifest -. describes .-> Parts ``` Small values use the root key directly. Larger values use deterministic `.part_N` keys plus a `.manifest` record. The manifest is a fixed 32-byte, versioned, little-endian wire format: | Offset | Size | Field | | ---: | ---: | --- | | 0 | 4 | magic `MCMF` | | 4 | 2 | format version | | 6 | 2 | header size | | 8 | 4 | chunk count | | 12 | 4 | reserved, must be zero | | 16 | 8 | total logical size | | 24 | 8 | configured chunk size | The implementation must encode and decode fields explicitly. Serializing a C++ structure is not part of the format. Readers reject an unknown version, inconsistent size, zero chunk count, non-zero reserved field, missing chunk, or chunk whose size does not match the manifest. The manifest is written last and acts as the provider-side visibility point. Its existence does not commit the Mooncake object; `PutEnd` at the Master remains the logical commit point. ## Request flows ### Put ```mermaid sequenceDiagram participant A as Application participant C as Mooncake client participant M as Mooncake Master participant K as KVCS adapter participant E as Local EFC / KVCacheStore A->>C: Put(key, buffers) C->>M: PutStart(key, size, kvcs-lowlevel) M-->>C: PROCESSING replica descriptor C->>K: BatchPut(logical key, buffers) K->>E: put root OR chunks opt chunked object K->>E: put manifest last end alt provider write succeeds K-->>C: success C->>M: PutEnd(key) M-->>C: COMPLETE C-->>A: success else provider write fails K->>E: delete only records known to be inserted K-->>C: failure or indeterminate outcome C->>M: PutRevoke(key) C-->>A: error end ``` Put remains insert-only. Mooncake creates logical metadata first, writes the provider, and commits only after every required provider record is visible. Rollback deletes only records whose insertion is known to have succeeded. A timed-out SDK call is not proof of failure and is handled as an ambiguous outcome, described below. ### Query and Get ```mermaid sequenceDiagram participant A as Application participant C as Mooncake client participant M as Mooncake Master participant K as KVCS adapter participant E as Local EFC / KVCacheStore A->>C: Query/Get(key) par authoritative metadata query C->>M: Query(key) M-->>C: logical state and replicas and bounded provider probe C->>K: QueryProvider(key, deadline) K->>E: batch query root/manifest E-->>K: hit, miss, or error K-->>C: request-local provider context end alt complete memory replica exists C-->>A: use memory replica else Master confirms complete KVCS replica and provider hit C->>K: Get(key, expected size, context) K->>E: batch get root or chunks E-->>K: values K-->>C: verified object C-->>A: object else provider miss or incomplete layout C-->>A: cache miss / replica unavailable else Master has no complete KVCS replica C-->>A: ignore provider result end ``` The Master result is always authoritative. A provider hit cannot create a Mooncake object, resurrect a deleted key, or override a memory replica. Conversely, a complete Master descriptor cannot turn an EFC miss into data. The read succeeds only when both views agree. Low Level provider query may run in parallel with the Master query because the SDK call has an absolute deadline. Speculation is an optimization: its result is ignored when the Master returns a usable memory replica or no complete KVCacheStore replica. ### Remove and retry ```mermaid flowchart TD Start[Remove logical key] Master[Remove Mooncake metadata] Provider[Delete provider records] Done[Done] Retry[Retry cleanup with same key] Orphan["Provider records may remain, but are no longer readable through Mooncake"] Start --> Master Master -->|removed or already absent| Provider Provider -->|success or confirmed absent| Done Provider -->|transient failure| Orphan Orphan --> Retry --> Provider ``` The first implementation removes Master metadata before deleting provider records. This ordering prefers an unreachable orphan over metadata that points to missing data. A retry is allowed to continue provider deletion when the Master already reports `OBJECT_NOT_FOUND`. KVCacheStore Low Level does not provide key enumeration for Mooncake-owned prefixes, so the first landing cannot promise background orphan discovery. Durable cleanup tasks are an open design item, not an implicit capability. ### Upsert Upsert is `Remove` followed by `Put`. It is not atomic. There is an interval in which the key is absent, and a failed second step leaves it absent. Callers that require compare-and-swap or uninterrupted replacement need a future generation-aware API. ## Eviction and stale metadata Provider eviction is expected behavior in Low Level mode. It creates a specific consistency case: Mooncake Master may still contain a complete KVCacheStore replica descriptor after its provider records have been evicted. The adapter handles this conservatively: 1. provider query reports a miss or incomplete layout; 2. the client does not advertise the KVCacheStore replica as readable; 3. if no other complete replica exists, the operation returns a cache miss or replica-unavailable result; 4. the caller may recompute the object and replace it through the normal Remove/Put lifecycle. The first PR must also define how stale provider replicas are invalidated at the Master. The preferred design is a conditional invalidation tied to the queried replica identity or epoch, so a late miss cannot remove a newer Put. If the existing Master API cannot express that safely, automatic invalidation remains disabled and the limitation must be documented before landing. Returning a miss is safe; silently claiming that the provider still contains the object is not. This point is a release blocker because provider LRU cannot be disabled. ## Target selection and availability An EFC configuration may expose more than one Low Level target. The prototype computes a stable target order from the encoded root key, tries the first target, and falls back to other healthy targets for the same operation. Fallback is not replication. The first release writes one target and must not advertise several targets as independent Mooncake replicas. A read fallback is useful only when the object can legitimately exist on another configured target; otherwise it adds latency without durability. The initial community implementation should therefore start with one explicit shared target unless maintainers confirm that multi-target lookup is part of the supported deployment contract. Mooncake does not persist target indices. A configuration change may reorder mountpoints, so any multi-target design must identify targets by stable provider identity rather than array position before it can be considered durable. ## Failure semantics | Situation | Required behavior | | --- | --- | | Provider miss with no Master replica | Return miss; do not create metadata. | | Provider hit with no complete Master replica | Ignore the hit. | | Complete Master replica plus provider miss | Treat the provider replica as unavailable; never return uninitialized data. | | Missing or malformed manifest/chunk | Return incomplete/corrupt result; do not assemble a partial object. | | Put fails after known chunk inserts | Delete only the known inserted records, then revoke the Master Put. | | Put outcome is ambiguous | Quarantine affected physical keys in-process and fail closed. | | Delete outcome is ambiguous | Do not issue an unsafe blind retry against a potentially newer value. | | Master commit fails after provider Put | Attempt provider cleanup and return failure. | | Adapter identity differs during restore | Fail closed. | | SDK or EFC unavailable at startup | Fail explicit KVCacheStore configuration; never fall back silently to another backend. | Low Level has no generation token or conditional delete. A timeout therefore cannot distinguish “the provider did nothing” from “the provider completed but the reply was lost.” The prototype quarantines affected physical keys in the current process until restart. This prevents immediate unsafe reuse, but it is not durable fencing. Cross-process recovery requires provider generations or a Mooncake-owned generation in every physical key. ## Configuration and runtime contract KVCacheStore is enabled only by explicit configuration. Socket presence is not an activation signal. ```bash export MOONCAKE_KVCS_MODE=low-level export MOONCAKE_KVCS_EFC_SOCKET=/var/run/kvcs/efc-grpc.sock export MOONCAKE_KVCS_SINGLE_TENANT=true export MOONCAKE_KVCS_MAX_VALUE_SIZE=4194304 export MOONCAKE_KVCS_MAX_KEYS_PER_BATCH=8 export MOONCAKE_KVCS_GET_WORKERS=1 export MOONCAKE_KVCS_SET_WORKERS=1 export MOONCAKE_KVCS_SIMPLE_WORKERS=1 export MOONCAKE_KVCS_OPERATION_TIMEOUT_MS=30000 ``` The values above are safe starting points, not universal tuning recommendations. The KVCacheStore integration guide sizes shared memory approximately from: ```text max_value_size * max_keys_per_batch * (get_workers + set_workers + simple_workers) ``` Mooncake validates the effective configuration before creating the adapter. It does not create the socket, start EFC, modify `/dev/shm`, or rewrite provider configuration. Runtime packaging must mount the EFC socket and allocate enough shared memory inside the application container. The initial scope is single-tenant because the current Mooncake distributed-storage configuration is single-tenant. The physical key still contains the tenant ID so the format does not bake in a cross-tenant collision. ## Build and dependency model The KVCacheStore SDK remains optional. CMake discovers it through `pkg-config` and compiles a `NOT_SUPPORTED` factory when it is absent. Existing Mooncake builds and deployments acquire no new runtime dependency. The community repository should not vendor KVCacheStore binaries or implementation sources. SDK-independent behavior—key encoding, manifests, chunking, reconciliation, deadlines, rollback planning, and error mapping—must be testable through an injected driver. Tests that call the public C ABI can run in an optional provider-enabled job. Published validation evidence must come from the containerized toolchain used for the prototype and provider SDK tests. Host-only compilation or network checks are not acceptance evidence. ## Compatibility - Existing filesystem and OSS adapters keep their current configuration and behavior. - Regular DFS replica serialization remains readable without KVCacheStore support. - Provider descriptors carry an explicit adapter identity and object size. - Snapshot restore fails closed on an unknown or mismatched adapter. - A build without the SDK continues to compile and returns `NOT_SUPPORTED` only when KVCacheStore is explicitly requested. - Automatic socket-based activation is excluded. - The legacy prototype value `MOONCAKE_KVCS_MODE=kvcs` may map to `low-level` for one transition period, but new documentation must not use it. ## Security and isolation - Logical keys are encoded before crossing the provider boundary; raw binary keys are not used as path-like strings. - Key size, value size, batch dimensions, buffer ranges, and aggregate lengths are checked before an SDK call. - EFC configuration and environment values are untrusted input and are rejected on invalid type, duplicate target identity, or out-of-range index. - Provider credentials, if introduced by a later mode, remain runtime secrets and are neither persisted in Master metadata nor logged. - The adapter opens no listener and adds no authentication surface to Mooncake. Access to the EFC socket and shared memory remains a deployment responsibility. ## Observability The adapter should expose, at minimum: - operation count, bytes, and latency by operation; - query hit, miss, incomplete, timeout, and provider error; - root versus chunked layout; - rollback attempts and failures; - quarantined-key count; - stale-replica invalidation attempts and outcomes; - selected target identity when multi-target support is enabled. Misses are expected cache events and should not be logged as transport failures. Malformed layouts and ambiguous outcomes deserve separate error labels because they require different operational responses. ## Validation plan No implementation PR should be marked ready until the rebased commit has the following evidence from that container environment. ### Build matrix - Build without the KVCacheStore SDK. - Build with the supported SDK and official mock library. - Run changed-line formatting and the repository's normal pre-commit checks. - Verify that OSS and filesystem-only builds are unchanged. ### Deterministic tests - physical-key encoding and length boundaries; - 32-byte manifest golden vectors and malformed-input rejection; - inline and multi-chunk put/query/get/delete; - missing chunk, wrong size, and corrupt manifest; - provider hit with absent or incomplete Master metadata; - memory-replica priority over a speculative provider result; - operation deadline and cancellation behavior; - known-partial rollback and ambiguous-outcome quarantine; - snapshot restore with matching and mismatched adapter identity; - provider eviction followed by stale-replica invalidation; - SDK-absent factory behavior. ### Real EFC validation - Put, Query, Get, and Remove through the node-local EFC socket. - Payload verification across process and container restart. - Values below, at, and above the configured chunk threshold. - Multi-buffer reads. - Forced provider eviction and the subsequent Mooncake miss path. - EFC unavailable, delayed, and restarted during an operation. - Shared-memory sizing documented with the exact test configuration. Performance numbers should be reported separately from correctness evidence, with hardware, EFC configuration, object-size distribution, concurrency, and baseline commit recorded. ## Revised staged delivery (supersedes the earlier all-in-one sequence) This section supersedes the earlier all-in-one implementation plan. The existing Draft PR #4500 is a full prototype and remains available for reference; PR 1 is a separate, smaller branch based on community main. Each stage below should be reviewed and validated on its own commit. 1. **PR 1: minimal Low Level read/write/delete (the first child PR of #4500).** One configured, shared EFC target; one KVCacheStore value per object; 4 MiB default per-value limit. Keep the existing distributed-storage adapter and Master-authoritative placement path. Implement insert-only Put, Query, Get and Delete and quarantine ambiguous writes. Query Master first; perform any needed provider query within a bounded deadline only for matching Master KVCS replicas. Query reports existence, not a manifest; Get validates its bytes against the Master object size. Reject oversize values before calling the SDK, and fail initialization on multi-target or incompatible mixed deployments. Bulk `RemoveByRegex` and `RemoveAll` explicitly report unsupported until provider-wide deletion has a safe contract. Without the optional SDK, other backends still build and explicit KVCS initialization reports NOT_SUPPORTED. Exit criterion: focused SDK/no-SDK builds and correctness tests, plus the existing single-value real-EFC correctness evidence. Single-key Remove and BatchRemove are included. Because PR 1 is insert-only, they delete the physical value before releasing the Master name; a concurrent Put therefore cannot publish a new value that this Remove later deletes. A provider miss is idempotent. If Master rejects the removal after the provider delete, the error is returned and the logical metadata remains for retry; generation IDs or a Master-side reservation can provide atomic cross-system deletion in a later protocol. Bulk RemoveByRegex and RemoveAll remain unsupported. No chunks, multi-target router, eviction repair or metrics. 2. **PR 2: large objects and, subject to a defined deployment contract, multi-target routing.** A separate stacked child PR based on PR 1 adds versioned manifests, deterministic chunks, bounded batch handling, cross-chunk rollback and uncertain-outcome isolation. Only add stable multi-target routing once target identity and mixed-deployment semantics are agreed. Validate boundaries, failed/partial operations and recovery against the actual deployment. If these responsibilities are too large for one review, submit them as two sequential child PRs; PR 1 remains usable. 3. **Eviction repair and operational hardening.** Conditionally invalidate stale Master replicas after provider misses, with race tests and operator recovery guidance. PR 1 fails closed on a provider miss; it does not automatically correct stale Master metadata. 4. **Dedicated metrics PR.** Bring back the operation metrics preserved in the separate metrics backup branch. Keep it out of PR 1 and do not publish performance figures in PR descriptions, comments or logs. 5. **Standard mode follow-up.** Scope this separately once the public API contract and validation environment have been established. The earlier full prototype **has already passed real-EFC correctness validation**. Retain that result as historical evidence. The cleanup head has separate focused SDK/no-SDK evidence; do not turn the historical run into a claim about a new live-EFC execution of this cleanup head. ## Corresponding corrections to the existing RFC sections - **Architecture and data layout:** the PR 1 data path is one SDK value per object on a single shared mountpoint. The chunk format, manifest and target selection are PR 2 designs, not claims about the PR 1 code. - **Read path:** complete the Master lookup before any bounded provider query; only Master-advertised KVCS replicas can be read from the provider. There is no speculative parallel Master/provider query in PR 1. Provider query only checks presence. The Master descriptor supplies the expected size; Get checks the actual provider response. There is no second manifest or per-request shard context to serialize. - **Failure and recovery:** PR 1 rejects unsupported topology and oversize writes, preserves ambiguous-outcome isolation, treats a missing provider value as an idempotent delete, and leaves Master metadata for retry if its removal is rejected after the physical delete. Automatic stale-replica cleanup is stage 3; bulk deletion remains unsupported. - **Validation:** separate PR 1 single-value read/write, error-path, SDK-optional build and real-EFC checks from PR 2 chunk/multi-target and stage 3 eviction-race cases. The prototype's completed real-EFC run and the PR 1 head's run must be labeled independently. ## Alternatives considered ### Implement KVCacheStore directly under #2993 This is a viable outcome and may be the cleanest community direction. The missing piece is direct-reader semantics: the current owner-oriented path should not require every shared-provider read to traverse the original writer. If #2993 accepts request-local provider queries and direct reads, this RFC's adapter can live there. ### Reuse the current OSS owner path unchanged This avoids client changes but keeps the original writer in the data path. It weakens availability and adds a network hop even though the requesting node already has local access to the shared provider. ### Reuse the #1957 Mooncake control plane That design suits Mooncake-owned NVMe devices. Applying it unchanged to KVCacheStore would duplicate provider placement, capacity, LRU, and health decisions. Mooncake would hold state it cannot keep authoritative. ### Persist provider target locations in Master metadata Rejected. EFC topology and provider placement may change independently of Mooncake snapshots. Persisting them would convert a runtime routing hint into stale protocol state. ### Enable KVCacheStore whenever the EFC socket exists Rejected. A mounted socket does not prove that every client and the Master agree on adapter mode, key format, tenant, or target set. Explicit configuration makes mismatches visible. ### Land Standard and Low Level together Deferred. Low Level has a public contract and bounded record operations. Standard introduces a second metadata authority and currently lacks the same public failure and deadline model. Combining both would make the first review broader without proving the shared path better. ## Open questions 1. Should KVCacheStore attach through the existing `ObjectStorageAdapter`, or become the first concrete driver under #2993? 2. What is the preferred Master API for conditionally invalidating an evicted provider replica without racing a newer Put? 3. Should the first release allow only one shared Low Level target? 4. Is a durable provider-cleanup queue required before landing, or is explicit retry plus observability acceptable initially? 5. Which SDK version and real-EFC environment can be used for community CI or periodic validation? 6. Which Mooncake error should represent “Master replica exists, provider has evicted it” so upper layers can treat it as a cache miss? ## Requested decision This RFC asks maintainers to agree on five points before implementation review begins: 1. KVCacheStore Low Level is useful as an optional shared capacity backend. 2. Mooncake Master owns logical lifecycle; KVCacheStore owns physical capacity and placement. 3. Reads may go directly through the requesting client's local EFC after Master-authoritative reconciliation. 4. Provider eviction must have a safe stale-metadata invalidation path. 5. The first landing is Low Level only, with Standard handled separately. Once those boundaries are settled, the implementation can be rebased and split without asking reviewers to infer architecture from a large prototype diff. ## Before submitting - [x] I searched existing issues and read the [Mooncake documentation](https://kvcache-ai.github.io/Mooncake/). This proposal is related to #1957 and #2993 and does not duplicate either design.

bsakaguchi added the

RFC

label

2026-10-29 21:23:19 +00:00

bsakaguchi commented

2026-10-29 21:23:20 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Thanks for opening this issue, @zxpdemonio!

Field	Value
Issue	#4499
GitHub user ID	`16206312`
Reporter	@zxpdemonio

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 8, 2026): Thanks for opening this issue, @zxpdemonio! | Field | Value | |-------|-------| | **Issue** | #4499 | | **GitHub user ID** | `16206312` | | **Reporter** | @zxpdemonio | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.