[GH-ISSUE #1957] [RFC]: Mooncake Store NVMe KV Backend #546

New issue

Open

opened 2026-10-29 21:16:13 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented 2026-10-29 21:16:13 +00:00

Owner

Copy link

Originally created by @zxpdemonio on GitHub (Apr 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1957

Changes proposed

Summary

This RFC proposes a standards-aligned NVMe KV backend for Mooncake Store.

The design keeps Mooncake’s backend-neutral object interface, adds a dedicated NVMe KV backend and connector/executor stack, separates logical identity from physical key lookup, and keeps device state, placement state, and cleanup orchestration in the control plane.

The backend supports both inline objects and manifest-plus-chunk objects, works across multiple devices, and leaves a clean path for stronger multi-tenant isolation and richer scheduling policies.

Motivation

Mooncake serves AI storage paths such as KV cache offload, layered storage, and persistence of immutable or append-mostly objects. These workloads need storage that is cheaper and larger than DRAM, more object-native than file or block abstractions, fast enough for latency-sensitive runtime paths, and manageable through Mooncake’s control plane.

NVMe KV is a strong fit because it exposes key-value semantics directly at the device layer. Compared with filesystem-oriented backends, it better matches object-style access, reduces software overhead, enables natural multi-device pooling, and integrates cleanly into Mooncake’s scheduling, observability, and cleanup model.

Representative scenarios include KV cache offload, memory-to-NVMe layered storage, persistent model artifacts, and unified control-plane management of NVMe KV devices.

Goals / Non-goals

Goals

• integrate NVMe KV through the same backend abstraction used by other storage backends
• support single-device and multi-device deployment
• define clear backend, connector, and executor responsibilities
• use deterministic physical keys bounded by NVMe KV key-size limits
• preserve correctness under collision, corruption, stale state, and partial write visibility
• support both inline objects and manifest-plus-chunk objects
• keep placement and cleanup authority in the control plane
• distinguish device failure from capacity-full conditions
• support manual cleanup and control-plane-driven automatic fallback cleanup
• reserve identity fields for tenant-, domain-, and namespace-aware isolation

Non-goals

• hardware-brand-specific backend architecture
• separate backend stacks for KV cache and model weights
• full replica management or global rebalancing in the first version
• complete orphan or foreign-data reconciliation in the initial proposal
• mandatory external console or full OpenAPI surface in phase 1
• mandatory multi-tenant policy engine before backend usability

Proposal

The proposal adds an NVMe KV backend with four main roles:

Storage backend

The backend adapts to Mooncake’s shared storage interface and owns object semantics:

• mapping logical objects to physical placement
• choosing object layout
• validating reads
• maintaining a local recovery catalog
• handling runtime decisions for failure, full-state, and retry behavior

Connector

The connector owns device access semantics:

• resolving configured device selectors
• initializing device access
• exposing device capability metadata
• providing uniform store/retrieve operations by physical key

Executor

The executor owns low-level transport submission:

• issuing NVMe KV commands
• translating transport/device errors into Mooncake error semantics
• allowing different transport implementations without changing backend semantics

Control plane

The control plane owns cluster-visible authority:

• device state and schedulability
• object placement state
• pending device-control operations
• cleanup orchestration
• stale-state convergence

The control plane does not proxy the data plane. Object I/O remains in the runtime backend path.

Data model

Object identity

The design uses an explicit logical identity abstraction:

• NvmeKvObjectIdentity
• NvmeKvIsolationContext

The isolation context is structured to support:

• tenant_id
• domain_id
• namespace_id

Even if early deployments leave these empty, the model reserves them now so future multi-tenant isolation does not require redesigning object identity.

Physical key model

NVMe KV physical keys are bounded by device key-size limits. The design therefore requires:

• arbitrary logical keys at the Mooncake layer
• deterministic mapping from logical identity to bounded physical key
• collision safety so wrong data is never returned

The design uses:

• one deterministic physical key for the root object
• deterministic physical keys for child chunks of large objects
• a separate verify hash stored in object metadata/header

Representative interfaces include:

• EncodeNvmeKvPhysicalKey(...)
• EncodeNvmeKvChunkPhysicalKey(...)
• ComputeNvmeKvVerifyHash(...)

A physical key is a placement key, not proof of identity. Reads must validate header structure, verify hash, payload checksum, and object-type-specific constraints before
returning data.

Object layout

Inline object

Small objects that fit under the effective single-value limit are stored as one object blob:

[ObjectHeader | optional identity metadata | payload]

This is the preferred fast path.

Manifest-plus-chunk object

Large objects are stored as:

• one root manifest object
• multiple chunk objects

The manifest records enough metadata to reconstruct the full object safely, including chunk placement and integrity information.

Object header fields

Representative fields include:

• magic
• version
• object_type
• payload_size
• verify_hash
• payload_checksum
• header_checksum
• flags
• identity_metadata_size

Representative identity-related fields include:

• NvmeKvObjectHeader::kFlagHasIsolationContext
• NvmeKvStoredIdentityMetadata

Visibility model

Chunked objects must not become visible until the manifest is valid. More generally, the design uses a two-step visibility rule:

1. write device data
2. expose the object only when the committed object is valid

Metadata and catalog model

The proposal separates two metadata scopes:

• Local recovery catalog: persisted by the backend for restart recovery, local lookup, and re-reporting state after restart
• Control-plane placement authority: shared authoritative state for device registry, placement records, and pending controls

This split is intentional: backend catalog is a local recovery cache, while the control plane is the shared authority.

Control plane semantics

The control plane defines cluster-wide semantics for NVMe KV resources.

Device-state authority

Operator-facing states should include:

• REGISTERING
• HEALTHY
• DEGRADED
• DRAINING
• DISABLED
• FAILED
• FULL

Placement authority

The control plane owns authoritative answers for:

• whether a logical object exists
• which device holds the committed placement
• which physical key is authoritative
• whether the object is committed, deleting, corrupted, or otherwise restricted

This prevents stale on-device data from being treated as live data.

Shared control core

HTTP and RPC may both be used as entrypoints, but they must share one control core:

• HTTP/RPC adapt protocol
• shared control service owns semantics

Stale convergence

If a backend stops reporting heartbeat, the control plane must converge stale device state into failed and unschedulable state rather than leaving stale healthy state visible indefinitely.

Device controls

The control plane may issue explicit operations such as enable or disable. The runtime backend applies them during heartbeat, and subsequent placement follows the updated schedulable set.

Cleanup authority

Cleanup is control-plane-orchestrated rather than hidden local deletion in the backend. This keeps reclaim observable, auditable, and aligned with placement authority.

Scheduling and cleanup semantics

Device scheduling

The initial proposal uses a simple stable policy:

• choose from the enabled-device set
• use hash-based selection over enabled devices
• allow bounded retry across candidates when a write path must move away from a failed target

This is intentionally modest and can evolve later.

Runtime device states

The backend distinguishes:

• ENABLED
• DISABLED_BY_CONFIG
• DISABLED_BY_FAILURE
• DISABLED_BY_CAPACITY

A full device is not the same as a failed device.

Failure vs capacity-full semantics

Transport and executor layers should map capacity-full conditions to DEVICE_CAPACITY_FULL instead of generic write failure.

This preserves the difference between:

• failed device: service/path problem
• full device: healthy but temporarily not writable

That distinction improves scheduling, operator visibility, cleanup behavior, and retry behavior.

Cleanup modes

The proposal supports two cleanup strategies:

Manual cleanup

The caller provides an explicit selector, such as:

• list of keys
• regex selector
• remove-all scope

LRU cleanup

The control plane selects reclaim candidates from authoritative placement state, ordered by a recency signal.

Cleanup safety rules

Cleanup must be conservative and authority-driven. It should not delete objects that are:

• missing from authoritative placement state
• not committed
• still in processing
• lease-protected
• tied to active replication tasks

Single active cleanup per device

Only one cleanup should run on a device at a time to reduce races and duplicated reclaim work.

Low-watermark reclaim

Cleanup should reclaim toward a low-watermark target rather than deleting an arbitrary small batch, to avoid repeated full-trigger and tiny-reclaim loops.

Automatic cleanup fallback

When a batch offload ends in DEVICE_CAPACITY_FULL, the runtime may:

1. report fresh backend state to the control plane
2. identify full devices
3. request LRU cleanup for those devices
4. retry the batch once within bounded limits

This keeps recovery control-plane-driven, explicit, bounded, and observable.

Priority order during full recovery

The cleanup model should remain conservative:

read > cleanup > new write on the full device

That means:

• reads should continue for valid committed objects
• cleanup should happen before reusing the full device for new writes
• new writes should prefer other healthy devices when available

Compatibility

Runtime compatibility

• upper layers continue using a backend-neutral object interface
• FileStorage remains the runtime hub for offload/load execution
• heartbeat remains the mechanism for applying control-plane decisions and reporting runtime state

Semantic compatibility

The proposal preserves familiar Mooncake semantics:

• committed objects are returned only when valid
• cleanup is consistent with control-plane authority
• capacity exhaustion is treated differently from path failure

Forward compatibility

The design leaves room for:

• richer placement strategies
• stronger hotplug handling
• tenant-aware isolation semantics
• asynchronous control-plane task models
• more complete reconcile and orphan-handling flows

Alternatives considered

Treat NVMe KV as a special-purpose side path

Rejected because it would leak device-specific logic into upper layers, fragment scheduling behavior, and weaken consistent control-plane integration.

Collapse connector and backend into one layer

Rejected because it would mix object semantics, placement logic, capability discovery, and transport-specific execution in one component, making the design harder to evolve and less vendor-neutral.

Use physical key alone as object identity

Rejected because collision or stale device data could cause wrong-object reads. The proposal instead requires verify-hash and checksum validation.

Let the backend perform hidden local cleanup on full

Rejected because it would bypass control-plane authority, reduce observability, and increase reclaim risk under transient runtime conditions.

Build a complete reconcile/orphan framework first

Rejected because it would delay delivery of the useful core backend. The first version instead prioritizes correct live-path behavior and correct cleanup of authority-known objects.

Drawbacks / Risks

• more system complexity than a local-disk backend
• dual metadata surfaces require careful restart and reconciliation behavior
• verify-hash and checksum validation add metadata and read-path cost
• full recovery quality depends on cleanup candidate quality and recency signals
• multi-tenant policy semantics are not complete yet
• some hotplug and shared-device edge cases remain future work

Open questions

1. Should future placement become capacity-weighted, topology-aware, or replica-aware?
2. What should be the canonical recency signal for LRU cleanup?
3. What is the desired contract for hotplug discovery, disappearance, reappearance, and requalification?
4. What guarantees are needed when multiple hosts access the same NVMe KV device or namespace?
5. What minimal scan/reconcile interface should be standardized after the initial backend lands?
6. When tenant_id, domain_id, and namespace_id become active routing inputs, how should placement, cleanup, and observability be partitioned?
7. Should cleanup, reconcile, drain, and rebuild converge into one asynchronous task framework?
8. What is the community contract for corruption handling, quarantine, and repair after validation failure?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @zxpdemonio on GitHub (Apr 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1957 ### Changes proposed ## Summary This RFC proposes a standards-aligned NVMe KV backend for Mooncake Store. The design keeps Mooncake’s backend-neutral object interface, adds a dedicated NVMe KV backend and connector/executor stack, separates logical identity from physical key lookup, and keeps device state, placement state, and cleanup orchestration in the control plane. The backend supports both inline objects and manifest-plus-chunk objects, works across multiple devices, and leaves a clean path for stronger multi-tenant isolation and richer scheduling policies. ## Motivation Mooncake serves AI storage paths such as KV cache offload, layered storage, and persistence of immutable or append-mostly objects. These workloads need storage that is cheaper and larger than DRAM, more object-native than file or block abstractions, fast enough for latency-sensitive runtime paths, and manageable through Mooncake’s control plane. NVMe KV is a strong fit because it exposes key-value semantics directly at the device layer. Compared with filesystem-oriented backends, it better matches object-style access, reduces software overhead, enables natural multi-device pooling, and integrates cleanly into Mooncake’s scheduling, observability, and cleanup model. Representative scenarios include KV cache offload, memory-to-NVMe layered storage, persistent model artifacts, and unified control-plane management of NVMe KV devices. ## Goals / Non-goals ### Goals - integrate NVMe KV through the same backend abstraction used by other storage backends - support single-device and multi-device deployment - define clear backend, connector, and executor responsibilities - use deterministic physical keys bounded by NVMe KV key-size limits - preserve correctness under collision, corruption, stale state, and partial write visibility - support both inline objects and manifest-plus-chunk objects - keep placement and cleanup authority in the control plane - distinguish device failure from capacity-full conditions - support manual cleanup and control-plane-driven automatic fallback cleanup - reserve identity fields for tenant-, domain-, and namespace-aware isolation ### Non-goals - hardware-brand-specific backend architecture - separate backend stacks for KV cache and model weights - full replica management or global rebalancing in the first version - complete orphan or foreign-data reconciliation in the initial proposal - mandatory external console or full OpenAPI surface in phase 1 - mandatory multi-tenant policy engine before backend usability ## Proposal The proposal adds an NVMe KV backend with four main roles: ### Storage backend The backend adapts to Mooncake’s shared storage interface and owns object semantics: - mapping logical objects to physical placement - choosing object layout - validating reads - maintaining a local recovery catalog - handling runtime decisions for failure, full-state, and retry behavior ### Connector The connector owns device access semantics: - resolving configured device selectors - initializing device access - exposing device capability metadata - providing uniform store/retrieve operations by physical key ### Executor The executor owns low-level transport submission: - issuing NVMe KV commands - translating transport/device errors into Mooncake error semantics - allowing different transport implementations without changing backend semantics ### Control plane The control plane owns cluster-visible authority: - device state and schedulability - object placement state - pending device-control operations - cleanup orchestration - stale-state convergence The control plane does not proxy the data plane. Object I/O remains in the runtime backend path. ## Data model ### Object identity The design uses an explicit logical identity abstraction: - `NvmeKvObjectIdentity` - `NvmeKvIsolationContext` The isolation context is structured to support: - `tenant_id` - `domain_id` - `namespace_id` Even if early deployments leave these empty, the model reserves them now so future multi-tenant isolation does not require redesigning object identity. ### Physical key model NVMe KV physical keys are bounded by device key-size limits. The design therefore requires: - arbitrary logical keys at the Mooncake layer - deterministic mapping from logical identity to bounded physical key - collision safety so wrong data is never returned The design uses: - one deterministic physical key for the root object - deterministic physical keys for child chunks of large objects - a separate verify hash stored in object metadata/header Representative interfaces include: - `EncodeNvmeKvPhysicalKey(...)` - `EncodeNvmeKvChunkPhysicalKey(...)` - `ComputeNvmeKvVerifyHash(...)` A physical key is a placement key, not proof of identity. Reads must validate header structure, verify hash, payload checksum, and object-type-specific constraints before returning data. ### Object layout #### Inline object Small objects that fit under the effective single-value limit are stored as one object blob: `[ObjectHeader | optional identity metadata | payload]` This is the preferred fast path. #### Manifest-plus-chunk object Large objects are stored as: - one root manifest object - multiple chunk objects The manifest records enough metadata to reconstruct the full object safely, including chunk placement and integrity information. ### Object header fields Representative fields include: - `magic` - `version` - `object_type` - `payload_size` - `verify_hash` - `payload_checksum` - `header_checksum` - `flags` - `identity_metadata_size` Representative identity-related fields include: - `NvmeKvObjectHeader::kFlagHasIsolationContext` - `NvmeKvStoredIdentityMetadata` ### Visibility model Chunked objects must not become visible until the manifest is valid. More generally, the design uses a two-step visibility rule: 1. write device data 2. expose the object only when the committed object is valid ### Metadata and catalog model The proposal separates two metadata scopes: - **Local recovery catalog**: persisted by the backend for restart recovery, local lookup, and re-reporting state after restart - **Control-plane placement authority**: shared authoritative state for device registry, placement records, and pending controls This split is intentional: backend catalog is a local recovery cache, while the control plane is the shared authority. ## Control plane semantics The control plane defines cluster-wide semantics for NVMe KV resources. ### Device-state authority Operator-facing states should include: - `REGISTERING` - `HEALTHY` - `DEGRADED` - `DRAINING` - `DISABLED` - `FAILED` - `FULL` ### Placement authority The control plane owns authoritative answers for: - whether a logical object exists - which device holds the committed placement - which physical key is authoritative - whether the object is committed, deleting, corrupted, or otherwise restricted This prevents stale on-device data from being treated as live data. ### Shared control core HTTP and RPC may both be used as entrypoints, but they must share one control core: - HTTP/RPC adapt protocol - shared control service owns semantics ### Stale convergence If a backend stops reporting heartbeat, the control plane must converge stale device state into failed and unschedulable state rather than leaving stale healthy state visible indefinitely. ### Device controls The control plane may issue explicit operations such as enable or disable. The runtime backend applies them during heartbeat, and subsequent placement follows the updated schedulable set. ### Cleanup authority Cleanup is control-plane-orchestrated rather than hidden local deletion in the backend. This keeps reclaim observable, auditable, and aligned with placement authority. ## Scheduling and cleanup semantics ### Device scheduling The initial proposal uses a simple stable policy: - choose from the enabled-device set - use hash-based selection over enabled devices - allow bounded retry across candidates when a write path must move away from a failed target This is intentionally modest and can evolve later. ### Runtime device states The backend distinguishes: - `ENABLED` - `DISABLED_BY_CONFIG` - `DISABLED_BY_FAILURE` - `DISABLED_BY_CAPACITY` A full device is not the same as a failed device. ### Failure vs capacity-full semantics Transport and executor layers should map capacity-full conditions to `DEVICE_CAPACITY_FULL` instead of generic write failure. This preserves the difference between: - failed device: service/path problem - full device: healthy but temporarily not writable That distinction improves scheduling, operator visibility, cleanup behavior, and retry behavior. ### Cleanup modes The proposal supports two cleanup strategies: #### Manual cleanup The caller provides an explicit selector, such as: - list of keys - regex selector - remove-all scope #### LRU cleanup The control plane selects reclaim candidates from authoritative placement state, ordered by a recency signal. ### Cleanup safety rules Cleanup must be conservative and authority-driven. It should not delete objects that are: - missing from authoritative placement state - not committed - still in processing - lease-protected - tied to active replication tasks ### Single active cleanup per device Only one cleanup should run on a device at a time to reduce races and duplicated reclaim work. ### Low-watermark reclaim Cleanup should reclaim toward a low-watermark target rather than deleting an arbitrary small batch, to avoid repeated full-trigger and tiny-reclaim loops. ### Automatic cleanup fallback When a batch offload ends in `DEVICE_CAPACITY_FULL`, the runtime may: 1. report fresh backend state to the control plane 2. identify full devices 3. request LRU cleanup for those devices 4. retry the batch once within bounded limits This keeps recovery control-plane-driven, explicit, bounded, and observable. ### Priority order during full recovery The cleanup model should remain conservative: **read > cleanup > new write on the full device** That means: - reads should continue for valid committed objects - cleanup should happen before reusing the full device for new writes - new writes should prefer other healthy devices when available ## Compatibility ### Runtime compatibility - upper layers continue using a backend-neutral object interface - FileStorage remains the runtime hub for offload/load execution - heartbeat remains the mechanism for applying control-plane decisions and reporting runtime state ### Semantic compatibility The proposal preserves familiar Mooncake semantics: - committed objects are returned only when valid - cleanup is consistent with control-plane authority - capacity exhaustion is treated differently from path failure ### Forward compatibility The design leaves room for: - richer placement strategies - stronger hotplug handling - tenant-aware isolation semantics - asynchronous control-plane task models - more complete reconcile and orphan-handling flows ## Alternatives considered ### Treat NVMe KV as a special-purpose side path Rejected because it would leak device-specific logic into upper layers, fragment scheduling behavior, and weaken consistent control-plane integration. ### Collapse connector and backend into one layer Rejected because it would mix object semantics, placement logic, capability discovery, and transport-specific execution in one component, making the design harder to evolve and less vendor-neutral. ### Use physical key alone as object identity Rejected because collision or stale device data could cause wrong-object reads. The proposal instead requires verify-hash and checksum validation. ### Let the backend perform hidden local cleanup on full Rejected because it would bypass control-plane authority, reduce observability, and increase reclaim risk under transient runtime conditions. ### Build a complete reconcile/orphan framework first Rejected because it would delay delivery of the useful core backend. The first version instead prioritizes correct live-path behavior and correct cleanup of authority-known objects. ## Drawbacks / Risks - more system complexity than a local-disk backend - dual metadata surfaces require careful restart and reconciliation behavior - verify-hash and checksum validation add metadata and read-path cost - full recovery quality depends on cleanup candidate quality and recency signals - multi-tenant policy semantics are not complete yet - some hotplug and shared-device edge cases remain future work ## Open questions 1. Should future placement become capacity-weighted, topology-aware, or replica-aware? 2. What should be the canonical recency signal for LRU cleanup? 3. What is the desired contract for hotplug discovery, disappearance, reappearance, and requalification? 4. What guarantees are needed when multiple hosts access the same NVMe KV device or namespace? 5. What minimal scan/reconcile interface should be standardized after the initial backend lands? 6. When `tenant_id`, `domain_id`, and `namespace_id` become active routing inputs, how should placement, cleanup, and observability be partitioned? 7. Should cleanup, reconcile, drain, and rebuild converge into one asynchronous task framework? 8. What is the community contract for corruption handling, quarantine, and repair after validation failure? ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

stale

label 2026-10-29 21:16:13 +00:00

bsakaguchi commented 2026-10-29 21:16:14 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Apr 22, 2026):

Architecture

  flowchart LR                                                                                                                                                                       
      A[Application / Runtime] --> B[FileStorage]                                                                                                                                    
      B --> C[StorageBackendInterface]                                                                                                                                               
      C --> D[NvmeKvStorageBackend]                                                                                                                                                  
      D --> E[NvmeKvConnector]                                                                                                                                                       
      E --> F[NvmeKvCommandExecutor]                                                                                                                                                 
      F --> G[NVMe KV Device]                                                                                                                                                        
                                                                                                                                                                                     
      B -. state report / cleanup request .-> H[Control-plane client]                                                                                                                
      H --> I[MasterService]                                                                                                                                                         
      I --> J[NvmeKvControlService]       

Data model

  flowchart TB                                                                                                                                                                       
      A[Logical object identity] --> B[Physical key encoding]                                                                                                                        
      A --> C[Verify hash]                                                                                                                                                           
      A --> D[Isolation context]                                                                                                                                                     
                                                                                                                                                                                     
      D --> D1[tenant_id]                                                                                                                                                            
      D --> D2[domain_id]                                                                                                                                                            
      D --> D3[namespace_id]                                                                                                                                                         
                                                                                                                                                                                     
      B --> E[Root object]                                                                                                                                                           
      B --> F[Chunk objects]                                                                                                                                                         
      C --> E                                                                                                                                                                        
      C --> F                                                                                                                                                                        

Object layout

  flowchart TB                                                                                                                                                                       
      A[Logical object] --> B{Fits single-value limit?}                                                                                                                              
                                                                                                                                                                                     
      B -->|Yes| C[Inline object]                                                                                                                                                    
      B -->|No| D[Manifest object]                                                                                                                                                   
                                                                                                                                                                                     
      C --> C1[ObjectHeader]                                                                                                                                                         
      C --> C2[Optional identity metadata]                                                                                                                                           
      C --> C3[Payload]                                                                                                                                                              
                                                                                                                                                                                     
      D --> D1[ObjectHeader]                                                                                                                                                         
      D --> D2[Identity metadata]                                                                                                                                                    
      D --> D3[Manifest payload]                                                                                                                                                     
      D3 --> E[Chunk record 0..N]                                                                                                                                                    
      E --> F[Chunk objects 0..N]   

Full-device recovery

  sequenceDiagram                                                                                                                                                                    
      participant R as Runtime / FileStorage                                                                                                                                         
      participant B as NvmeKvStorageBackend                                                                                                                                          
      participant M as MasterService                                                                                                                                                 
      participant C as Control plane                                                                                                                                                 
                                                                                                                                                                                     
      R->>B: BatchOffload                                                                                                                                                            
      B-->>R: DEVICE_CAPACITY_FULL                                                                                                                                                   
      R->>M: Report fresh backend state                                                                                                                                              
      M->>C: Mark device FULL                                                                                                                                                        
      R->>M: Request LRU cleanup                                                                                                                                                     
      M->>C: Select cleanup candidates                                                                                                                                               
      C-->>M: Cleanup decision                                                                                                                                                       
      M-->>R: Cleanup result                                                                                                                                                         
      R->>B: Bounded retry  

<!-- gh-comment-id:4296923435 --> @zxpdemonio commented on GitHub (Apr 22, 2026): Architecture ```mermaid flowchart LR A[Application / Runtime] --> B[FileStorage] B --> C[StorageBackendInterface] C --> D[NvmeKvStorageBackend] D --> E[NvmeKvConnector] E --> F[NvmeKvCommandExecutor] F --> G[NVMe KV Device] B -. state report / cleanup request .-> H[Control-plane client] H --> I[MasterService] I --> J[NvmeKvControlService] ``` Data model ```mermaid flowchart TB A[Logical object identity] --> B[Physical key encoding] A --> C[Verify hash] A --> D[Isolation context] D --> D1[tenant_id] D --> D2[domain_id] D --> D3[namespace_id] B --> E[Root object] B --> F[Chunk objects] C --> E C --> F ``` Object layout ```mermaid flowchart TB A[Logical object] --> B{Fits single-value limit?} B -->|Yes| C[Inline object] B -->|No| D[Manifest object] C --> C1[ObjectHeader] C --> C2[Optional identity metadata] C --> C3[Payload] D --> D1[ObjectHeader] D --> D2[Identity metadata] D --> D3[Manifest payload] D3 --> E[Chunk record 0..N] E --> F[Chunk objects 0..N] ``` Full-device recovery ```mermaid sequenceDiagram participant R as Runtime / FileStorage participant B as NvmeKvStorageBackend participant M as MasterService participant C as Control plane R->>B: BatchOffload B-->>R: DEVICE_CAPACITY_FULL R->>M: Report fresh backend state M->>C: Mark device FULL R->>M: Request LRU cleanup M->>C: Select cleanup candidates C-->>M: Cleanup decision M-->>R: Cleanup result R->>B: Bounded retry ```

bsakaguchi commented 2026-10-29 21:16:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:5041833791 --> @github-actions[bot] commented on GitHub (Jul 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:16:14 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Jul 24, 2026):

https://github.com/kvcache-ai/Mooncake/pull/2167 pr updated

<!-- gh-comment-id:5067373973 --> @zxpdemonio commented on GitHub (Jul 24, 2026): https://github.com/kvcache-ai/Mooncake/pull/2167 pr updated

bsakaguchi commented 2026-10-29 21:16:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 24, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6379232781 --> @github-actions[bot] commented on GitHub (Oct 24, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#546

No description provided.