[GH-ISSUE #4591] [RFC]: Shared NVMe KV Backend for Mooncake Store #1168

New issue

Open

opened 2026-10-29 21:23:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 21:23:33 +00:00

Owner

Copy link

Originally created by @Javen-Ke on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4591

Changes proposed

1. Introduction

This proposal introduces a SHARED_NVME_KV replica type that allows multiple Clients to access the same external KV storage pool directly over NVMe-oF. The Master manages allocation identities, replica publication, LRU eviction, and cleanup. The Target executes KV commands and manages physical storage space. Published replicas remain available independently of the writing Client, and other Clients can read them without forwarding requests through that Client.

All Clients connect to the same set of namespaces using an identical, ordered namespace list. Each Client selects a namespace using hash(logical_key) % N, where N is the number of namespaces. This distributes large sets of keys approximately evenly across namespaces. Applications continue to use Put/Get: a write selects the shared replica type, obtains an allocation ID from the Master, writes directly to the Target, and publishes the replica. Other Clients query the descriptor and read directly using the same routing rule. This balances key counts, not necessarily storage usage or access load.

2. Background and Motivation

SPDK 26.09 adds support for exposing KV-capable bdevs as NVMe KV namespaces through its NVMe-oF Target. This provides a Target implementation on which multiple Clients can build direct access to an external KV pool. The release was published on September 24, 2026; see the SPDK 26.09 release notes.

The existing NvmeKvStorageBackend publishes LOCAL_DISK replicas through FileStorage, with reads served by the Client that owns the replica. PR #3513 proposes NVMe-oF connectivity for that backend while retaining this replica model. Even when multiple Clients can access the underlying device, Client B still reads data written by Client A through A. Once A exits, that proxy path can no longer serve the data, even if the data remains intact on the Target.

A shared pool needs data access and replica lifetime to be independent of the writer. The relationship to existing work is:

Related work	Relationship to this proposal
NVMe KV RFC #1957	Defines object layouts, executors, and control-plane cleanup responsibilities, while leaving guarantees for multiple hosts sharing a namespace open. This proposal builds on those foundations and specifies allocation, publication, and deletion for shared replicas.
NVMe KV PR #2167 and NVMe-oF extension #3513	Integrate through FileStorage / LOCAL_DISK. This proposal reuses object encoding and command execution, lets the requesting Client access the Target directly, and assigns physical identities and lifecycle management to the Master.
Distributed KV RFC #2993	Accesses an external distributed KV service but still reports LOCAL_DISK replicas through FileStorage. This proposal represents data as independent shared replicas, without requiring an owning Client to load and forward it.

3. Technical Design

3.1 Components and Responsibilities

flowchart LR
    C["Client (extended)"] -. "Allocate, publish, query" .-> M["Master (extended)"]
    C --> O["SharedNvmeKv (new)"]
    O --> E["KV executor (reused interface, SPDK implementation)"]
    E --> T["External NVMe KV Target"]
    M -. "Cleanup tasks" .-> G["Cleanup worker (proposed within Master)"]
    G --> E
    classDef changed fill:#fff2cc,stroke:#b7950b;
    classDef external fill:#eeeeee,stroke:#777777;
    class C,M,O,E,G changed;
    class T external;

Dashed arrows represent control requests; solid arrows represent data operations.

• Master: Extend replica metadata and PutStart / PutEnd to manage allocation identities, publication state, access recency, and pending cleanup records. A writing Client's exit does not invalidate its published shared replicas.
• Client and SharedNvmeKv: The Client selects the shared path from the replica descriptor. The new SharedNvmeKv object layer handles physical key encoding, object framing, idempotent writes, and read validation.
• KV executor: Use NvmeKvCommandExecutor as the reuse boundary and adapt the SPDK KV implementation from #3513. The executor owns device connections, KV commands, DMA buffers, and completion handling.
• Cleanup worker: Proposed as a background worker within the Master, issuing Delete through the KV executor independently of the original writing Client. Section 5 describes the implementation approach.

3.2 Replica Descriptors and Physical Addressing

The shared descriptor is returned through the existing replica query interface. Its core fields are:

struct SharedKvDescriptor {
    uint64_t object_size;    // Payload size in bytes.
    uint64_t allocation_id;  // Unique identity of this physical allocation.
};

struct SharedKvReplicaData {
    SharedKvDescriptor descriptor;
};

The descriptor does not need the writing Client's address. The request carries the logical key, which selects the namespace using the routing rule above; allocation_id identifies the physical allocation within that namespace. Initially, namespace membership, ordering, and the hash function must remain unchanged while shared allocations exist. Reads, writes, and cleanup use the same mapping.

Each new allocation receives a new allocation_id; retries of the same device write reuse that ID. Physical keys use a fixed 16-byte encoding, matching the maximum key length allowed by the NVMe Key Value Command Set specification:

physical_key = BE64(allocation_id) || BE64(part_index)

BE64 denotes a 64-bit big-endian encoding. Part 0 stores an inline object or a manifest; parts 1 through N store data chunks.

Using unique, non-reused allocation IDs avoids accidental deletion and physical-key hash collisions:

• Avoid accidental deletion: Hashing the logical key reuses the same physical key after an object is recreated. A new allocation ID prevents a delayed Delete from affecting the new object.
• Avoid hash collisions: Different logical keys can have the same hash; unique allocation IDs give each allocation a distinct physical key.

IDs must never be reused within the managed storage domain, including across Master restarts. We propose integrating ID range reservations with the Master's existing persistence infrastructure and issuing IDs only from durably reserved ranges, as described in Section 5. Stored objects include their full logical and physical identities, lengths, and checksums, all of which are validated on reads.

The Master separately maintains control state that is not copied into the descriptor:

• Publication state, read leases, and pin protection.
• The current write attempt's allocation_id, used to match completion and revocation requests. This identity is retained even if a write to multiple replica types falls back to memory only.
• The storage location and accounted size of allocations awaiting cleanup, plus layout version and chunk count once chunking is supported.

3.3 Put and Get

Put:

1. PutStart creates an in-progress record on the Master, allocates an ID, and returns a SharedKvDescriptor.
2. The Client writes directly to the Target using atomic store-if-not-exists. The entire encoded value must fit the device's value-size limit.
3. After a successful write, the Client includes the allocation identity in PutEnd. The Master publishes the replica only if it matches the current write attempt, preserving the existing success criteria for writes involving multiple replica types.

The first write to a new ID should not encounter an existing entry. However, if a Store succeeds and its completion result is lost, retrying the same allocation may return an already-exists result. The Client then reads back the entry and validates its identity, length, and checksums before treating the retry as successful. Retries must use the same ID and contents.

PutRevoke also matches the write identity so that delayed requests cannot alter a later write attempt. Clients do not delete physical data merely because a write or publication result is unknown. The Master is responsible for reclaiming abandoned allocations.

Get:

1. The requesting Client obtains a published descriptor and read protection from the Master.
2. The Client directly retrieves the physical key, validates identity, length, and checksums, and returns the payload. Read protection must remain valid throughout the operation.
3. If the Target explicitly returns NotFound, the Client asks the Master to invalidate only the shared replica matching the allocation ID from that query. Other replicas are retained. Timeouts and disconnections are returned as read errors without invalidating the descriptor.

Normal LRU eviction must not cause NotFound for a published replica under valid read protection. This path handles Target data loss or stale descriptors, such as published data lost after restarting a nonpersistent Target.

3.4 Master LRU and Cleanup

The Master may use an NVMe-oF initiator to periodically query capacity metrics exposed by the Target. Capacity watermarks trigger LRU eviction, with candidates selected by access recency. Monitoring is used to trigger reclamation; the Master does not use it to restrict new writes. The Target decides whether space can be allocated during Store and reports insufficient capacity. The supported capacity metrics and monitoring interface remain to be determined.

The Master coordinates eviction and deletion:

1. Check pins and in-progress operations, withdraw the candidate replica's visibility, and record a cleanup task.
2. Wait for existing read protection to end and confirm that no writes from the old attempt can still take effect.
3. Delete all KV entries belonging to the allocation. Once deletion is confirmed, update capacity accounting and remove the cleanup task.

The Target must not independently evict managed entries through LRU or TTL expiration. The Master controls their eviction and cleanup.

Never reusing IDs does not prevent a delayed Store from recreating an old entry after Delete. Until old writes are confirmed to have stopped, retain the pending cleanup task and recheck it later. A timeout alone is not proof that a write has stopped. Section 5 describes write termination and cleanup recovery.

4. Implementation and Evaluation

4.1 Implementation Phases

This proposal supports Put/Get for complete, immutable objects. Upsert, range writes, and session writes are outside its scope and must be explicitly rejected.

Phase 1: Shared Reads and Writes

The initial deployment uses a single Master, dedicated namespaces, fixed routing, and the default tenant.

• Connect to Targets supporting 16-byte keys and atomic store-if-not-exists, and implement inline object reads and writes.
• Implement ID allocation and persistence that prevent reuse. Recovery of object mappings is outside this phase.
• Extend the Master and Client with the new replica descriptor and write identities in completion and revocation requests.
• Support direct reads from other Clients, including after the writing Client exits.

Phase 2: Reclamation

• Add capacity monitoring and LRU eviction decisions to the Master.
• Implement cleanup execution, capacity accounting, and retries after failures.

Phase 3: Chunking and Multiple Tenants

• Chunking: Extend the object layout while retaining the same allocation and publication protocol. Publish and evict complete objects.

  Register the allocation ID, layout version, and chunk count before writing. Write chunks first, then the manifest, and finally publish through the Master. The registered chunk count allows cleanup to enumerate every physical key even if the manifest is missing. Both chunks and the manifest must fit the device's value-size limit.

• Multiple tenants: Reuse the existing logical identity model and add capacity accounting, quotas, and eviction policies per tenant.

  All tenants sharing a physical namespace use one ID allocation domain. Mutually untrusted tenants also require access isolation at the Target; logical key prefixes do not enforce device access permissions.

4.2 Validation Plan

• Sharing and publication: Verify reads from other Clients after the writer exits, ID uniqueness across restarts, rejection of delayed completion and revocation requests for old allocations, existing success criteria for writes to multiple replica types, and regression coverage for existing backends.
• Reclamation and extensions: Test protection against deletion during reads, delayed Store operations, deletion retries, partial chunk-write failures, sustained capacity pressure, tenant quotas, and eviction fairness.
• Performance: Compare direct KV commands, LOCAL_DISK over NVMe-oF, and the shared path using the same devices, network, object sizes, and concurrency. Report throughput, P99 latency, CPU usage, and data-copy overhead. Results will be added after implementation.

5. Implementation Approach and Remaining Validation

1. Executor reuse: Local and shared backends use the same KV command interface, adapting the SPDK implementation from #3513. Manage the SPDK environment centrally within each process, with executors owning connections, queues, and DMA buffers. Shutdown stops new submissions and waits for in-flight I/O to finish before releasing resources. Validate coexistence of both backends and resource lifetime during failure handling and shutdown.
2. ID persistence: Record the upper bound of each reserved ID range through the Master's persistence infrastructure. Activate a range only after it is durable; normal allocations then update an in-memory counter. Reserve the next range asynchronously in advance, waiting only when the current range is exhausted and the next is not yet durable. On restart, skip all previously reserved IDs, including unused ones. Validate crashes during reservation, persistence failures, and restarts. This avoids a separate synchronous persistence operation for every object write.
3. Cleanup execution and recovery: Run cleanup in a Master background worker. Completion and revocation requests distinguish confirmed write termination from an unknown outcome. Termination confirmation requires the Client to stop new Store submissions and retries for that ID and establish that in-flight I/O has ended. Keep allocations without that confirmation in a queue for later rechecking. When metadata recovery is added, use the operation log's durable-completion callback to issue Delete only after the visibility withdrawal and pending cleanup record are durable, and retry unfinished tasks after recovery. If a disconnected Client cannot confirm termination, whether the chosen transport can provide reliable evidence that old writes have stopped remains to be validated. A fixed delay is not a safety guarantee.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Javen-Ke on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4591 ### Changes proposed ## 1. Introduction This proposal introduces a `SHARED_NVME_KV` replica type that allows multiple Clients to access the same external KV storage pool directly over NVMe-oF. The Master manages allocation identities, replica publication, LRU eviction, and cleanup. The Target executes KV commands and manages physical storage space. Published replicas remain available independently of the writing Client, and other Clients can read them without forwarding requests through that Client. All Clients connect to the same set of namespaces using an identical, ordered namespace list. Each Client selects a namespace using `hash(logical_key) % N`, where `N` is the number of namespaces. This distributes large sets of keys approximately evenly across namespaces. Applications continue to use Put/Get: a write selects the shared replica type, obtains an allocation ID from the Master, writes directly to the Target, and publishes the replica. Other Clients query the descriptor and read directly using the same routing rule. This balances key counts, not necessarily storage usage or access load. ## 2. Background and Motivation SPDK 26.09 adds support for exposing KV-capable bdevs as NVMe KV namespaces through its NVMe-oF Target. This provides a Target implementation on which multiple Clients can build direct access to an external KV pool. The release was published on September 24, 2026; see the [SPDK 26.09 release notes](https://spdk.io/release/2026/09/24/v26.09_release/). The existing `NvmeKvStorageBackend` publishes `LOCAL_DISK` replicas through `FileStorage`, with reads served by the Client that owns the replica. [PR #3513](https://github.com/kvcache-ai/Mooncake/pull/3513) proposes NVMe-oF connectivity for that backend while retaining this replica model. Even when multiple Clients can access the underlying device, Client B still reads data written by Client A through A. Once A exits, that proxy path can no longer serve the data, even if the data remains intact on the Target. A shared pool needs data access and replica lifetime to be independent of the writer. The relationship to existing work is: | Related work | Relationship to this proposal | | --- | --- | | [NVMe KV RFC #1957](https://github.com/kvcache-ai/Mooncake/issues/1957) | Defines object layouts, executors, and control-plane cleanup responsibilities, while leaving guarantees for multiple hosts sharing a namespace open. This proposal builds on those foundations and specifies allocation, publication, and deletion for shared replicas. | | [NVMe KV PR #2167](https://github.com/kvcache-ai/Mooncake/pull/2167) and [NVMe-oF extension #3513](https://github.com/kvcache-ai/Mooncake/pull/3513) | Integrate through `FileStorage` / `LOCAL_DISK`. This proposal reuses object encoding and command execution, lets the requesting Client access the Target directly, and assigns physical identities and lifecycle management to the Master. | | [Distributed KV RFC #2993](https://github.com/kvcache-ai/Mooncake/issues/2993) | Accesses an external distributed KV service but still reports `LOCAL_DISK` replicas through `FileStorage`. This proposal represents data as independent shared replicas, without requiring an owning Client to load and forward it. | ## 3. Technical Design ### 3.1 Components and Responsibilities ```mermaid flowchart LR C["Client (extended)"] -. "Allocate, publish, query" .-> M["Master (extended)"] C --> O["SharedNvmeKv (new)"] O --> E["KV executor (reused interface, SPDK implementation)"] E --> T["External NVMe KV Target"] M -. "Cleanup tasks" .-> G["Cleanup worker (proposed within Master)"] G --> E classDef changed fill:#fff2cc,stroke:#b7950b; classDef external fill:#eeeeee,stroke:#777777; class C,M,O,E,G changed; class T external; ``` Dashed arrows represent control requests; solid arrows represent data operations. - **Master:** Extend replica metadata and `PutStart` / `PutEnd` to manage allocation identities, publication state, access recency, and pending cleanup records. A writing Client's exit does not invalidate its published shared replicas. - **Client and `SharedNvmeKv`:** The Client selects the shared path from the replica descriptor. The new `SharedNvmeKv` object layer handles physical key encoding, object framing, idempotent writes, and read validation. - **KV executor:** Use `NvmeKvCommandExecutor` as the reuse boundary and adapt the SPDK KV implementation from #3513. The executor owns device connections, KV commands, DMA buffers, and completion handling. - **Cleanup worker:** Proposed as a background worker within the Master, issuing `Delete` through the KV executor independently of the original writing Client. Section 5 describes the implementation approach. ### 3.2 Replica Descriptors and Physical Addressing The shared descriptor is returned through the existing replica query interface. Its core fields are: ```cpp struct SharedKvDescriptor { uint64_t object_size; // Payload size in bytes. uint64_t allocation_id; // Unique identity of this physical allocation. }; struct SharedKvReplicaData { SharedKvDescriptor descriptor; }; ``` The descriptor does not need the writing Client's address. The request carries the logical key, which selects the namespace using the routing rule above; `allocation_id` identifies the physical allocation within that namespace. Initially, namespace membership, ordering, and the hash function must remain unchanged while shared allocations exist. Reads, writes, and cleanup use the same mapping. Each new allocation receives a new `allocation_id`; retries of the same device write reuse that ID. Physical keys use a fixed 16-byte encoding, matching the maximum key length allowed by the [NVMe Key Value Command Set specification](https://nvmexpress.org/wp-content/uploads/NVM-Express-Key-Value-Command-Set-Specification-Revision-1.2-2025.03.11-Ratified.pdf): ```text physical_key = BE64(allocation_id) || BE64(part_index) ``` `BE64` denotes a 64-bit big-endian encoding. Part 0 stores an inline object or a manifest; parts 1 through N store data chunks. Using unique, non-reused allocation IDs avoids accidental deletion and physical-key hash collisions: - **Avoid accidental deletion:** Hashing the logical key reuses the same physical key after an object is recreated. A new allocation ID prevents a delayed `Delete` from affecting the new object. - **Avoid hash collisions:** Different logical keys can have the same hash; unique allocation IDs give each allocation a distinct physical key. IDs must never be reused within the managed storage domain, including across Master restarts. We propose integrating ID range reservations with the Master's existing persistence infrastructure and issuing IDs only from durably reserved ranges, as described in Section 5. Stored objects include their full logical and physical identities, lengths, and checksums, all of which are validated on reads. The Master separately maintains control state that is not copied into the descriptor: - Publication state, read leases, and pin protection. - The current write attempt's `allocation_id`, used to match completion and revocation requests. This identity is retained even if a write to multiple replica types falls back to memory only. - The storage location and accounted size of allocations awaiting cleanup, plus layout version and chunk count once chunking is supported. ### 3.3 Put and Get **Put:** 1. `PutStart` creates an in-progress record on the Master, allocates an ID, and returns a `SharedKvDescriptor`. 2. The Client writes directly to the Target using atomic store-if-not-exists. The entire encoded value must fit the device's value-size limit. 3. After a successful write, the Client includes the allocation identity in `PutEnd`. The Master publishes the replica only if it matches the current write attempt, preserving the existing success criteria for writes involving multiple replica types. The first write to a new ID should not encounter an existing entry. However, if a `Store` succeeds and its completion result is lost, retrying the same allocation may return an already-exists result. The Client then reads back the entry and validates its identity, length, and checksums before treating the retry as successful. Retries must use the same ID and contents. `PutRevoke` also matches the write identity so that delayed requests cannot alter a later write attempt. Clients do not delete physical data merely because a write or publication result is unknown. The Master is responsible for reclaiming abandoned allocations. **Get:** 1. The requesting Client obtains a published descriptor and read protection from the Master. 2. The Client directly retrieves the physical key, validates identity, length, and checksums, and returns the payload. Read protection must remain valid throughout the operation. 3. If the Target explicitly returns `NotFound`, the Client asks the Master to invalidate only the shared replica matching the allocation ID from that query. Other replicas are retained. Timeouts and disconnections are returned as read errors without invalidating the descriptor. Normal LRU eviction must not cause `NotFound` for a published replica under valid read protection. This path handles Target data loss or stale descriptors, such as published data lost after restarting a nonpersistent Target. ### 3.4 Master LRU and Cleanup The Master may use an NVMe-oF initiator to periodically query capacity metrics exposed by the Target. Capacity watermarks trigger LRU eviction, with candidates selected by access recency. Monitoring is used to trigger reclamation; the Master does not use it to restrict new writes. The Target decides whether space can be allocated during `Store` and reports insufficient capacity. The supported capacity metrics and monitoring interface remain to be determined. The Master coordinates eviction and deletion: 1. Check pins and in-progress operations, withdraw the candidate replica's visibility, and record a cleanup task. 2. Wait for existing read protection to end and confirm that no writes from the old attempt can still take effect. 3. Delete all KV entries belonging to the allocation. Once deletion is confirmed, update capacity accounting and remove the cleanup task. The Target must not independently evict managed entries through LRU or TTL expiration. The Master controls their eviction and cleanup. Never reusing IDs does not prevent a delayed `Store` from recreating an old entry after `Delete`. Until old writes are confirmed to have stopped, retain the pending cleanup task and recheck it later. A timeout alone is not proof that a write has stopped. Section 5 describes write termination and cleanup recovery. ## 4. Implementation and Evaluation ### 4.1 Implementation Phases This proposal supports Put/Get for complete, immutable objects. Upsert, range writes, and session writes are outside its scope and must be explicitly rejected. **Phase 1: Shared Reads and Writes** The initial deployment uses a single Master, dedicated namespaces, fixed routing, and the default tenant. - [ ] Connect to Targets supporting 16-byte keys and atomic store-if-not-exists, and implement inline object reads and writes. - [ ] Implement ID allocation and persistence that prevent reuse. Recovery of object mappings is outside this phase. - [ ] Extend the Master and Client with the new replica descriptor and write identities in completion and revocation requests. - [ ] Support direct reads from other Clients, including after the writing Client exits. **Phase 2: Reclamation** - [ ] Add capacity monitoring and LRU eviction decisions to the Master. - [ ] Implement cleanup execution, capacity accounting, and retries after failures. **Phase 3: Chunking and Multiple Tenants** - [ ] **Chunking:** Extend the object layout while retaining the same allocation and publication protocol. Publish and evict complete objects. Register the allocation ID, layout version, and chunk count before writing. Write chunks first, then the manifest, and finally publish through the Master. The registered chunk count allows cleanup to enumerate every physical key even if the manifest is missing. Both chunks and the manifest must fit the device's value-size limit. - [ ] **Multiple tenants:** Reuse the existing logical identity model and add capacity accounting, quotas, and eviction policies per tenant. All tenants sharing a physical namespace use one ID allocation domain. Mutually untrusted tenants also require access isolation at the Target; logical key prefixes do not enforce device access permissions. ### 4.2 Validation Plan - **Sharing and publication:** Verify reads from other Clients after the writer exits, ID uniqueness across restarts, rejection of delayed completion and revocation requests for old allocations, existing success criteria for writes to multiple replica types, and regression coverage for existing backends. - **Reclamation and extensions:** Test protection against deletion during reads, delayed `Store` operations, deletion retries, partial chunk-write failures, sustained capacity pressure, tenant quotas, and eviction fairness. - **Performance:** Compare direct KV commands, `LOCAL_DISK` over NVMe-oF, and the shared path using the same devices, network, object sizes, and concurrency. Report throughput, P99 latency, CPU usage, and data-copy overhead. Results will be added after implementation. ## 5. Implementation Approach and Remaining Validation 1. **Executor reuse:** Local and shared backends use the same KV command interface, adapting the SPDK implementation from [#3513](https://github.com/kvcache-ai/Mooncake/pull/3513). Manage the SPDK environment centrally within each process, with executors owning connections, queues, and DMA buffers. Shutdown stops new submissions and waits for in-flight I/O to finish before releasing resources. Validate coexistence of both backends and resource lifetime during failure handling and shutdown. 2. **ID persistence:** Record the upper bound of each reserved ID range through the Master's persistence infrastructure. Activate a range only after it is durable; normal allocations then update an in-memory counter. Reserve the next range asynchronously in advance, waiting only when the current range is exhausted and the next is not yet durable. On restart, skip all previously reserved IDs, including unused ones. Validate crashes during reservation, persistence failures, and restarts. This avoids a separate synchronous persistence operation for every object write. 3. **Cleanup execution and recovery:** Run cleanup in a Master background worker. Completion and revocation requests distinguish confirmed write termination from an unknown outcome. Termination confirmation requires the Client to stop new `Store` submissions and retries for that ID and establish that in-flight I/O has ended. Keep allocations without that confirmation in a queue for later rechecking. When metadata recovery is added, use the operation log's durable-completion callback to issue `Delete` only after the visibility withdrawal and pending cleanup record are durable, and retry unfinished tasks after recovery. If a disconnected Client cannot confirm termination, whether the chosen transport can provide reliable evidence that old writes have stopped remains to be validated. A fixed delay is not a safety guarantee. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label 2026-10-29 21:23:33 +00:00

bsakaguchi commented 2026-10-29 21:23:34 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 12, 2026):

Thanks for opening this issue, @Javen-Ke!

Field	Value
Issue	#4591
GitHub user ID	288857397
Reporter	@Javen-Ke

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6121477417 --> @github-actions[bot] commented on GitHub (Oct 12, 2026): Thanks for opening this issue, @Javen-Ke! | Field | Value | |-------|-------| | **Issue** | #4591 | | **GitHub user ID** | `288857397` | | **Reporter** | @Javen-Ke | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1168

No description provided.