kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4719] [RFC]: Mooncake Store KV Cache Reshard #1202

New issue

Open

opened 2026-10-29 21:23:54 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:23:54 +00:00

Owner

Originally created by @LLLL114 on GitHub (Oct 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4719

Originally assigned to: @Bo-Vincent on GitHub.

Changes proposed

This proposal adds KV cache reuse to Mooncake Store without tying cached data to the topology that wrote it. One complete KV page is identified by one logical key. Writers retain their own sharding and physical layouts. Readers use the source layout manifest stored in the Store, together with their current target layout, to plan the required byte ranges. This allows inference instances with different tensor parallelism (TP), pipeline parallelism (PP), and physical layouts to share cached KV data.

The intended audience is maintainers of Mooncake Store, the Reshard SDK, and inference frameworks. The main changes are Store reshard planning in Python and framework integration. The underlying object capabilities depend on [multipart RFC #4366](https://github.com/kvcache-ai/Mooncake/issues/4366). Implementation begins with a minimal multipart demo that provides the capabilities required here, followed by Store reshard. The internal multipart design is covered by its separate RFC and is not repeated here.

Motivation and goals

When a KV page is stored as separate rank-local shards and its data keys encode TP or PP ranks, the cached KV for a token sequence becomes tied to the topology that wrote it. An instance with a different topology must locate multiple source keys, identify the source layout, and reconstruct its target shards. Once the writer exits, readers must also be able to operate without its runtime addresses or participation in communication.

The proposal separates logical content identity, the source layout in the Store, and target runtime memory:

Logical content identity determines whether data represents the same reusable KV content. It excludes TP and PP ranks and physical layout.
The source layout manifest describes the layers, KV heads, and byte layout covered by each Part in the Store.
The target memory binding describes the reading rank's registered buffers, page locations, and strides. It is valid only for the current runtime.

The goals are to support reads across TP, PP, and physical layouts; preserve HiCache's existence filtering, prefetch, cancellation, hit accounting, and fallback behavior; and allow reads after the source inference instance exits. The initial implementation does not require writers to anticipate future reader topologies or convert all KV data to a single physical layout.

Scope and dependencies

[Multipart RFC #4366](https://github.com/kvcache-ai/Mooncake/issues/4366) only needs the object capabilities used by this proposal: independent writes to designated Parts, existence and metadata queries for complete objects, an immutable reference to application metadata, and batched byte-range reads from Parts.
This proposal includes:

A model semantic domain, page keys, and shared source layout manifests.
Logical planning and byte-range generation driven by the target rank's requirements.
Local caching of source manifests and read templates that contain no live addresses.
Initial SGLang HiCache integration and functional validation.

The initial scope excludes dtype conversion, page-size reshaping, KV conversion between models, automatic layout rewriting, selecting among multiple stored layouts for the same page, and full multipart persistence, migration, and high-availability extensions. Receiver-side auto staging, #4117, #4502, page-unified layouts, and new CUDA transfer kernels are separate optimization topics, not prerequisites for this proposal.

The existing runtime-to-runtime (R2R) reshard implementation and snapshot contracts remain unchanged. The Store path uses a separate memory binding and does not relax R2R identity constraints to enable reuse of stored KV data.

Responsibilities

Layer	Responsibilities
Inference framework	Provide actual layer and head ownership, host-pool layouts, and registered memory; coordinate TP and PP ranks; manage the prefetch lifecycle; and transfer data from host memory to the GPU.
Mooncake Python Reshard	Manage source manifests, intersect source coverage with target requirements, generate and reuse byte-range templates, and validate target addresses and read results.
Mooncake Store and Transfer Engine	Provide multipart objects, metadata, valid replica information, and general byte-transfer capabilities.

Logical planning runs in the Python SDK inside the reader process, not in the Store Master. The Master does not need to understand model layers, attention heads, TP, PP, or SGLang page slots. The framework does not need to implement its own algorithm for intersecting KV layer and head ranges.

flowchart LR
    K[Logical page key] --> O[Store object metadata]
    O --> M[Source manifest reference]
    M --> C[Reader manifest cache]
    C --> P[Python planning and range templates]
    T[Target placement and memory binding] --> P
    P --> R[Store range reads]
    R --> H[HiCache host pool]
    H --> G[Existing framework H2D path]

Logical keys and source manifests

Object keys

A data key identifies the logical content of a complete page, rather than a rank-local shard. The SGLang prototype reuses the framework's token-prefix hash and adds the model semantic domain and a protocol suffix:

page_key = <backend/model prefix>_<token prefix hash>_<model_domain>

model_domain includes the namespace, model identity and revision, semantic configuration that affects KV content, and descriptors such as dtype, page size, the global layer set, total KV-head count, and K/V head dimensions. TP, PP, rank identities, and physical axis order are excluded.

This gives the same page key to the same model semantics and token prefix across different topologies. A shared model name alone is not sufficient for KV reuse: revisions and settings such as RoPE that affect the content must be isolated by the semantic domain. A physical format change leaves the logical key unchanged; a change in model semantics must change the key.

Manifest contents and storage

A manifest describes a source layout for a fixed page size and is shared by all pages with that layout. It includes at least the model's semantic identity, the global KV descriptor, the physical layout, and the global layer set and head interval covered by each Part.

Part sizes, K/V starting offsets, and strides are derived from this description, avoiding two potentially inconsistent representations of byte geometry. A manifest contains no page token hash, GPU pointer, process-local address, or replica address intended for long-term reuse.

manifest_key = kv/<model_domain>/manifests/<layout_digest>

page ObjectMetadata
    manifest_key -> shared metadata object in the Store

Manifests are stored as immutable, content-addressed metadata objects. Each page's ObjectMetadata stores a reference to its manifest instead of duplicating the complete manifest. Readers can therefore resolve the actual stored layout directly from the page key, without scanning a catalog of source layouts.

The prototype marks manifests as metadata, hard-pins them, and reads them back after publication to verify that their contents match. Reclaiming manifests that are no longer referenced by any page remains a separate productionization task.

Write flow

Manifest initialization

During backend initialization, the framework reports the global layers and KV heads actually owned by each rank, along with its host layout. The SGLang prototype aggregates these descriptions within TP groups and then across PP groups to construct a consistent, complete placement.

All ranks derive the Part ordering and their own Part indices from the same placement. TP rank 0 / PP rank 0 publishes the shared manifest. The framework completes initialization coordination before accepting requests that use the backend. This step gathers layout descriptions only, not KV payloads.

For the initially supported topologies without replicated KV-head shards, each rank corresponds to one Part, so N = TP × PP. A Part contains all K/V data owned by that rank for the current page. Each rank may supply multiple source memory fragments for writing its Part, without gathering all ranks' KV payloads onto rank 0.

Writing a page

Construct the page key from the model semantic domain and token-prefix hash.
Check in a batch whether the complete objects already exist. Treat existing pages as successful skips without rewriting their payloads.
For pages that are not yet complete, each rank writes its own Part and supplies the same manifest reference.
Only a complete object can become a subsequent cache hit. A successful write by one rank does not mean that the entire page is readable.

The existence check is a fast path for avoiding unnecessary writes, not a lock for resolving races. If a race occurs between the check and the write, the Store's Part write protocol still determines whether the write can proceed.
If another topology has already completed the same logical page, the current topology also skips the write. The source manifest continues to describe the stored layout, and readers perform resharding as needed. The prototype neither stores another copy for the current topology nor automatically rewrites the same key into the target layout after a read.
If an object is incomplete and its existing writes belong to an incompatible source layout, the current writer must not mix its Parts into that object. The operation is handled as a cache-backup failure and relies on Store cleanup or a later retry. This RFC does not introduce a separate protocol for taking over an incomplete object across layouts.

Reads and target-driven planning

Resolving source layouts for a request

The reader queries object metadata for candidate page keys in batches, obtaining the available pages and their manifest references. When a reference is first encountered, the reader fetches the manifest from the Store and validates its content digest, model semantic domain, layout, and Part count. Subsequent requests can reuse the local manifest cache.
Different pages in one request may come from different writer topologies or physical layouts. The read context therefore contains an ordered list of page keys, their corresponding layout IDs, and the required set of manifests, rather than a single current source layout. When a page is missing or incomplete, HiCache uses the available contiguous prefix and follows its existing fallback policy.

Computing logical intersections

Planning is driven by the target placement. For each global layer and K/V component required by a target rank, the planner intersects the target KV-head interval with the coverage described by the source manifest:

Layers and heads required by the target rank
     corresponding source Parts
     token/head byte ranges within those Parts
     byte ranges in the target's registered memory

Source byte positions are derived from the manifest; target positions come from the local memory binding. Each side uses its own token stride, head stride, and base offset, so their physical layouts need not match. Ranges are merged only when contiguous on both sides, avoiding reads or overwrites of unrelated data merely to increase I/O size.
The planner must verify complete logical coverage of the target's requirements and reject undeclared overlap. Byte-range generation must also validate registered-memory bounds, alignment, disjoint target addresses, operation-count limits, and total-byte limits.

Caching and concurrent requests

Cache or context	Location	Contents	Lifetime
Source manifest cache	Reader process	Validated, immutable model and layout descriptions	Bounded cache, reused by manifest reference
Read template cache	Each reading rank's SDK reader	Relative source and target offsets, lengths, and Part mappings	While the target placement and host-pool binding remain valid
Prefetch context	Current read operation	Ordered page keys, layout IDs, manifests, and operation identity	One prefetch operation
Replica snapshot	Current native read operation	Current replica descriptors and read leases	Subject to Store validity and refresh rules

A local manifest or template cache hit does not eliminate the need to check current object readability and replica leases. Templates do not retain source replica addresses for long-term reuse across requests. Target page locations are not fixed to the previous request's addresses: each read supplies the current page-slot offsets and validates their bounds. Reallocating the host pool or changing its layout requires rebuilding the binding and reader.Within a reader with a fixed target configuration, the prototype caches templates by source layout_id. The reader already fixes the target layout and buffer geometry. Multiple pages with the same source layout can therefore reuse planning results, while each request retains its own page sequence, destination locations, and cancellation state.

SGLang HiCache integration

During initialization, existing CPU communication groups aggregate TP/PP ownership, and a coordinator rank publishes the manifest. During prefetch, the coordinator queries the hit prefix and creates a read context, which is distributed through the TP/PP groups. Each rank verifies that its local key order matches the context, then independently plans and reads its own target shard. Writes retain HiCache's existing flow: check existence first, then submit pages that are not already cached. Reads continue to respect prefetch budgets, cancellation, hit coordination across ranks, and existing cache-miss fallback behavior. Object and template reuse must not depend on globally mutable state shared between requests.
The current host-layout mapping is:

SGLang host layout	Layout within a Store Part
page_first	PLHD
page_first_direct, layer_first	LPHD
page_head	HPLD

The logical key is a reuse contract: instances writing the same token prefix in the same semantic domain must provide semantically interchangeable KV data. The prototype does not compare independently computed KV results byte for byte across topologies, nor does it promise identical floating-point values under different parallel reduction orders. Model configuration, explicit revisions, and runtime validation must establish the conditions for cache reuse.

Error handling and compatibility

Condition	Handling
The same page already exists as a complete object	Skip the write and retain the stored source layout.
A page is missing or incomplete	Treat it as a cache miss; do not consume partial KV data.
A manifest is missing, corrupted, or semantically incompatible	Reject the data and use HiCache's error and fallback paths.
Target ranges exceed bounds, overlap, or lack complete coverage	Reject the plan before submitting transfers.
An object becomes invalid after the query, or a read fails	Apply Store replica-validity checks and report only the prefix that was actually read in full.
The source inference process has exited	Continue reading as long as the Store replicas and manifest remain valid.

Validation and performance expectations

Functional acceptance focuses on resharding: logical coverage across TP/PP, uneven PP layer assignments, all 3×3 conversions among PLHD/LPHD/HPLD, different source manifests across pages, correct addresses when templates are reused, and fallback on missing data, cancellation, or invalid bounds. Tests against a real Store must verify target bytes exactly and cover KV restoration after the uploader process exits. Multipart completeness, reclamation, and lease tests belong to the Store layer.

The existing prototype has passed byte-level validation against a real Store for these layout combinations, as well as SGLang validation of reads in both directions between TP1/PP3 and TP2/PP2 and restoration after the source process exits.

Performance should be evaluated as two separate questions:

Store object-management cost.
End-to-end HiCache reshard cost.

The prototype is available in [Mooncake Store and the Reshard SDK](https://github.com/LLLL114/Mooncake/tree/experiment/kv-reshard-multipart) and the [SGLang HiCache integration](https://github.com/LLLL114/sglang/tree/experiment/kv-reshard-multipart), both on the experiment/kv-reshard-multipart branch. The experimental implementation referenced by this draft consists of Store commit [02a125e1](github.com/LLLL114/Mooncake@02a125e112), Store reshard commit [b89af266](github.com/LLLL114/Mooncake@b89af26644), and SGLang integration commit [bf7373f1](github.com/LLLL114/sglang@bf7373f197), all on experimental branches.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @LLLL114 on GitHub (Oct 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4719 Originally assigned to: @Bo-Vincent on GitHub. ### Changes proposed This proposal adds KV cache reuse to Mooncake Store without tying cached data to the topology that wrote it. One complete KV page is identified by one logical key. Writers retain their own sharding and physical layouts. Readers use the source layout manifest stored in the Store, together with their current target layout, to plan the required byte ranges. This allows inference instances with different tensor parallelism (TP), pipeline parallelism (PP), and physical layouts to share cached KV data. The intended audience is maintainers of Mooncake Store, the Reshard SDK, and inference frameworks. The main changes are Store reshard planning in Python and framework integration. The underlying object capabilities depend on [[multipart RFC #4366](https://github.com/kvcache-ai/Mooncake/issues/4366)](https://github.com/kvcache-ai/Mooncake/issues/4366). Implementation begins with a minimal multipart demo that provides the capabilities required here, followed by Store reshard. The internal multipart design is covered by its separate RFC and is not repeated here. ## Motivation and goals When a KV page is stored as separate rank-local shards and its data keys encode TP or PP ranks, the cached KV for a token sequence becomes tied to the topology that wrote it. An instance with a different topology must locate multiple source keys, identify the source layout, and reconstruct its target shards. Once the writer exits, readers must also be able to operate without its runtime addresses or participation in communication. The proposal separates logical content identity, the source layout in the Store, and target runtime memory: - **Logical content identity** determines whether data represents the same reusable KV content. It excludes TP and PP ranks and physical layout. - **The source layout manifest** describes the layers, KV heads, and byte layout covered by each Part in the Store. - **The target memory binding** describes the reading rank's registered buffers, page locations, and strides. It is valid only for the current runtime. The goals are to support reads across TP, PP, and physical layouts; preserve HiCache's existence filtering, prefetch, cancellation, hit accounting, and fallback behavior; and allow reads after the source inference instance exits. The initial implementation does not require writers to anticipate future reader topologies or convert all KV data to a single physical layout. ## Scope and dependencies [[Multipart RFC #4366](https://github.com/kvcache-ai/Mooncake/issues/4366)](https://github.com/kvcache-ai/Mooncake/issues/4366) only needs the object capabilities used by this proposal: independent writes to designated Parts, existence and metadata queries for complete objects, an immutable reference to application metadata, and batched byte-range reads from Parts. This proposal includes: 1. A model semantic domain, page keys, and shared source layout manifests. 2. Logical planning and byte-range generation driven by the target rank's requirements. 3. Local caching of source manifests and read templates that contain no live addresses. 4. Initial SGLang HiCache integration and functional validation. The initial scope excludes dtype conversion, page-size reshaping, KV conversion between models, automatic layout rewriting, selecting among multiple stored layouts for the same page, and full multipart persistence, migration, and high-availability extensions. Receiver-side auto staging, #4117, #4502, page-unified layouts, and new CUDA transfer kernels are separate optimization topics, not prerequisites for this proposal. The existing runtime-to-runtime (R2R) reshard implementation and snapshot contracts remain unchanged. The Store path uses a separate memory binding and does not relax R2R identity constraints to enable reuse of stored KV data. ## Responsibilities Layer | Responsibilities -- | -- Inference framework | Provide actual layer and head ownership, host-pool layouts, and registered memory; coordinate TP and PP ranks; manage the prefetch lifecycle; and transfer data from host memory to the GPU. Mooncake Python Reshard | Manage source manifests, intersect source coverage with target requirements, generate and reuse byte-range templates, and validate target addresses and read results. Mooncake Store and Transfer Engine | Provide multipart objects, metadata, valid replica information, and general byte-transfer capabilities. **Logical planning runs in the Python SDK inside the reader process, not in the Store Master.** The Master does not need to understand model layers, attention heads, TP, PP, or SGLang page slots. The framework does not need to implement its own algorithm for intersecting KV layer and head ranges. ```mermaid flowchart LR K[Logical page key] --> O[Store object metadata] O --> M[Source manifest reference] M --> C[Reader manifest cache] C --> P[Python planning and range templates] T[Target placement and memory binding] --> P P --> R[Store range reads] R --> H[HiCache host pool] H --> G[Existing framework H2D path] ``` ## Logical keys and source manifests ### Object keys A data key identifies the logical content of a complete page, rather than a rank-local shard. The SGLang prototype reuses the framework's token-prefix hash and adds the model semantic domain and a protocol suffix: ```text page_key = <backend/model prefix>_<token prefix hash>_<model_domain> ``` `model_domain` includes the namespace, model identity and revision, semantic configuration that affects KV content, and descriptors such as dtype, page size, the global layer set, total KV-head count, and K/V head dimensions. TP, PP, rank identities, and physical axis order are excluded. This gives the same page key to the same model semantics and token prefix across different topologies. A shared model name alone is not sufficient for KV reuse: revisions and settings such as RoPE that affect the content must be isolated by the semantic domain. A physical format change leaves the logical key unchanged; a change in model semantics must change the key. ### Manifest contents and storage A manifest describes a source layout for a fixed page size and is shared by all pages with that layout. It includes at least the model's semantic identity, the global KV descriptor, the physical layout, and the global layer set and head interval covered by each Part. Part sizes, K/V starting offsets, and strides are derived from this description, avoiding two potentially inconsistent representations of byte geometry. A manifest contains no page token hash, GPU pointer, process-local address, or replica address intended for long-term reuse. ```text manifest_key = kv/<model_domain>/manifests/<layout_digest> page ObjectMetadata manifest_key -> shared metadata object in the Store ``` Manifests are stored as immutable, content-addressed metadata objects. Each page's ObjectMetadata stores a reference to its manifest instead of duplicating the complete manifest. Readers can therefore resolve the actual stored layout directly from the page key, without scanning a catalog of source layouts. The prototype marks manifests as metadata, hard-pins them, and reads them back after publication to verify that their contents match. Reclaiming manifests that are no longer referenced by any page remains a separate productionization task. ## Write flow ### Manifest initialization During backend initialization, the framework reports the global layers and KV heads actually owned by each rank, along with its host layout. The SGLang prototype aggregates these descriptions within TP groups and then across PP groups to construct a consistent, complete placement. All ranks derive the Part ordering and their own Part indices from the same placement. `TP rank 0 / PP rank 0` publishes the shared manifest. The framework completes initialization coordination before accepting requests that use the backend. This step gathers layout descriptions only, not KV payloads. For the initially supported topologies without replicated KV-head shards, each rank corresponds to one Part, so `N = TP × PP`. A Part contains all K/V data owned by that rank for the current page. Each rank may supply multiple source memory fragments for writing its Part, without gathering all ranks' KV payloads onto rank 0. ### Writing a page 1. Construct the page key from the model semantic domain and token-prefix hash. 2. Check in a batch whether the complete objects already exist. Treat existing pages as successful skips without rewriting their payloads. 3. For pages that are not yet complete, each rank writes its own Part and supplies the same manifest reference. 4. Only a complete object can become a subsequent cache hit. A successful write by one rank does not mean that the entire page is readable. The existence check is a fast path for avoiding unnecessary writes, not a lock for resolving races. If a race occurs between the check and the write, the Store's Part write protocol still determines whether the write can proceed. If another topology has already completed the same logical page, the current topology also skips the write. The source manifest continues to describe the stored layout, and readers perform resharding as needed. The prototype neither stores another copy for the current topology nor automatically rewrites the same key into the target layout after a read. If an object is incomplete and its existing writes belong to an incompatible source layout, the current writer must not mix its Parts into that object. The operation is handled as a cache-backup failure and relies on Store cleanup or a later retry. This RFC does not introduce a separate protocol for taking over an incomplete object across layouts. ## Reads and target-driven planning ### Resolving source layouts for a request The reader queries object metadata for candidate page keys in batches, obtaining the available pages and their manifest references. When a reference is first encountered, the reader fetches the manifest from the Store and validates its content digest, model semantic domain, layout, and Part count. Subsequent requests can reuse the local manifest cache. Different pages in one request may come from different writer topologies or physical layouts. The read context therefore contains an ordered list of page keys, their corresponding layout IDs, and the required set of manifests, rather than a single current source layout. When a page is missing or incomplete, HiCache uses the available contiguous prefix and follows its existing fallback policy. ### Computing logical intersections Planning is driven by the target placement. For each global layer and K/V component required by a target rank, the planner intersects the target KV-head interval with the coverage described by the source manifest: ``` Layers and heads required by the target rank corresponding source Parts token/head byte ranges within those Parts byte ranges in the target's registered memory ``` Source byte positions are derived from the manifest; target positions come from the local memory binding. Each side uses its own token stride, head stride, and base offset, so their physical layouts need not match. Ranges are merged only when contiguous on both sides, avoiding reads or overwrites of unrelated data merely to increase I/O size. The planner must verify complete logical coverage of the target's requirements and reject undeclared overlap. Byte-range generation must also validate registered-memory bounds, alignment, disjoint target addresses, operation-count limits, and total-byte limits. ## Caching and concurrent requests Cache or context | Location | Contents | Lifetime -- | -- | -- | -- Source manifest cache | Reader process | Validated, immutable model and layout descriptions | Bounded cache, reused by manifest reference Read template cache | Each reading rank's SDK reader | Relative source and target offsets, lengths, and Part mappings | While the target placement and host-pool binding remain valid Prefetch context | Current read operation | Ordered page keys, layout IDs, manifests, and operation identity | One prefetch operation Replica snapshot | Current native read operation | Current replica descriptors and read leases | Subject to Store validity and refresh rules A local manifest or template cache hit does not eliminate the need to check current object readability and replica leases. Templates do not retain source replica addresses for long-term reuse across requests. Target page locations are not fixed to the previous request's addresses: each read supplies the current page-slot offsets and validates their bounds. Reallocating the host pool or changing its layout requires rebuilding the binding and reader.Within a reader with a fixed target configuration, the prototype caches templates by source layout_id. The reader already fixes the target layout and buffer geometry. Multiple pages with the same source layout can therefore reuse planning results, while each request retains its own page sequence, destination locations, and cancellation state. ## SGLang HiCache integration During initialization, existing CPU communication groups aggregate TP/PP ownership, and a coordinator rank publishes the manifest. During prefetch, the coordinator queries the hit prefix and creates a read context, which is distributed through the TP/PP groups. Each rank verifies that its local key order matches the context, then independently plans and reads its own target shard. Writes retain HiCache's existing flow: check existence first, then submit pages that are not already cached. Reads continue to respect prefetch budgets, cancellation, hit coordination across ranks, and existing cache-miss fallback behavior. Object and template reuse must not depend on globally mutable state shared between requests. The current host-layout mapping is: SGLang host layout | Layout within a Store Part -- | -- page_first | PLHD page_first_direct, layer_first | LPHD page_head | HPLD The logical key is a reuse contract: instances writing the same token prefix in the same semantic domain must provide semantically interchangeable KV data. The prototype does not compare independently computed KV results byte for byte across topologies, nor does it promise identical floating-point values under different parallel reduction orders. Model configuration, explicit revisions, and runtime validation must establish the conditions for cache reuse. ## Error handling and compatibility Condition | Handling -- | -- The same page already exists as a complete object | Skip the write and retain the stored source layout. A page is missing or incomplete | Treat it as a cache miss; do not consume partial KV data. A manifest is missing, corrupted, or semantically incompatible | Reject the data and use HiCache's error and fallback paths. Target ranges exceed bounds, overlap, or lack complete coverage | Reject the plan before submitting transfers. An object becomes invalid after the query, or a read fails | Apply Store replica-validity checks and report only the prefix that was actually read in full. The source inference process has exited | Continue reading as long as the Store replicas and manifest remain valid. The logical key is a reuse contract: instances writing the same token prefix in the same semantic domain must provide semantically interchangeable KV data. The prototype does not compare independently computed KV results byte for byte across topologies, nor does it promise identical floating-point values under different parallel reduction orders. Model configuration, explicit revisions, and runtime validation must establish the conditions for cache reuse. ## Validation and performance expectations Functional acceptance focuses on resharding: logical coverage across TP/PP, uneven PP layer assignments, all 3×3 conversions among PLHD/LPHD/HPLD, different source manifests across pages, correct addresses when templates are reused, and fallback on missing data, cancellation, or invalid bounds. Tests against a real Store must verify target bytes exactly and cover KV restoration after the uploader process exits. Multipart completeness, reclamation, and lease tests belong to the Store layer. The existing prototype has passed byte-level validation against a real Store for these layout combinations, as well as SGLang validation of reads in both directions between TP1/PP3 and TP2/PP2 and restoration after the source process exits. Performance should be evaluated as two separate questions: - **Store object-management cost.** - **End-to-end HiCache reshard cost.** The prototype is available in [[Mooncake Store and the Reshard SDK](https://github.com/LLLL114/Mooncake/tree/experiment/kv-reshard-multipart)](https://github.com/LLLL114/Mooncake/tree/experiment/kv-reshard-multipart) and the [[SGLang HiCache integration](https://github.com/LLLL114/sglang/tree/experiment/kv-reshard-multipart)](https://github.com/LLLL114/sglang/tree/experiment/kv-reshard-multipart), both on the `experiment/kv-reshard-multipart` branch. The experimental implementation referenced by this draft consists of Store commit [[02a125e1](https://github.com/LLLL114/Mooncake/commit/02a125e112774c6972920b90568ff16db8ccb100)](https://github.com/LLLL114/Mooncake/commit/02a125e112774c6972920b90568ff16db8ccb100), Store reshard commit [[b89af266](https://github.com/LLLL114/Mooncake/commit/b89af2664416950d3c98ba3c864aac87f2ed2ca9)](https://github.com/LLLL114/Mooncake/commit/b89af2664416950d3c98ba3c864aac87f2ed2ca9), and SGLang integration commit [[bf7373f1](https://github.com/LLLL114/sglang/commit/bf7373f197a801b3b1ca020f10ad854d3be9c15c)](https://github.com/LLLL114/sglang/commit/bf7373f197a801b3b1ca020f10ad854d3be9c15c), all on experimental branches. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:23:54 +00:00

bsakaguchi commented

2026-10-29 21:23:55 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 16, 2026):

Thanks for opening this issue, @LLLL114!

Field	Value
Issue	#4719
GitHub user ID	`121648372`
Reporter	@LLLL114

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 16, 2026): Thanks for opening this issue, @LLLL114! | Field | Value | |-------|-------| | **Issue** | #4719 | | **GitHub user ID** | `121648372` | | **Reporter** | @LLLL114 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:55 +00:00

Author

Owner

@kunkunblueberry commented on GitHub (Oct 17, 2026):

Thanks for writing this up — the split between logical content identity, the stored source layout, and the target runtime binding is the part that makes this tractable, and keeping logical planning in the reader process (rather than the Master) is the right boundary.

I would like to contribute a first slice rather than take over the whole RFC: the logical planning half, as a new pure-Python subpackage next to the existing R2R planner — model semantic domain + logical page keys, content-addressed source layout manifests (with derived byte geometry), target-driven byte-range generation, and the read-template cache. Draft PR: #4741.

It deliberately does not touch the Store write protocol (that stays with multipart #4366) or the SGLang HiCache integration, so it can be reviewed on its own. It currently covers PLHD/LPHD/HPLD with the full 3x3 conversion matrix verified element-by-element, uneven PP layer subsets, head-sharded sources, and requirements whose head interval spans Part boundaries.

Two things I would like the RFC author's opinion on, since they change the shape of the plan:

Part byte geometry: I derive part length / K-V offsets / strides from the manifest layout + logical descriptor (K region first, then V) rather than storing them. If the prototype stores an explicit geometry, I would rather match it than invent a second representation.
Manifest lifetime: the RFC leaves reclaiming unreferenced manifests to productionization. For the first slice I treat manifests as immutable and content-addressed (readers verify the digest on decode), which is compatible with any later reclamation policy — but if you expect manifests to be mutated in place (e.g. re-published under the same key), digest verification would need to become advisory.

@kunkunblueberry commented on GitHub (Oct 17, 2026): Thanks for writing this up — the split between logical content identity, the stored source layout, and the target runtime binding is the part that makes this tractable, and keeping logical planning in the reader process (rather than the Master) is the right boundary. I would like to contribute a first slice rather than take over the whole RFC: the logical planning half, as a new pure-Python subpackage next to the existing R2R planner — model semantic domain + logical page keys, content-addressed source layout manifests (with derived byte geometry), target-driven byte-range generation, and the read-template cache. Draft PR: #4741. It deliberately does not touch the Store write protocol (that stays with multipart #4366) or the SGLang HiCache integration, so it can be reviewed on its own. It currently covers PLHD/LPHD/HPLD with the full 3x3 conversion matrix verified element-by-element, uneven PP layer subsets, head-sharded sources, and requirements whose head interval spans Part boundaries. Two things I would like the RFC author's opinion on, since they change the shape of the plan: 1. Part byte geometry: I derive part length / K-V offsets / strides from the manifest layout + logical descriptor (K region first, then V) rather than storing them. If the prototype stores an explicit geometry, I would rather match it than invent a second representation. 2. Manifest lifetime: the RFC leaves reclaiming unreferenced manifests to productionization. For the first slice I treat manifests as immutable and content-addressed (readers verify the digest on decode), which is compatible with any later reclamation policy — but if you expect manifests to be mutated in place (e.g. re-published under the same key), digest verification would need to become advisory.

bsakaguchi commented

2026-10-29 21:23:55 +00:00

Author

Owner

@kunkunblueberry commented on GitHub (Oct 22, 2026):

No problem — thanks for the update, and for confirming the direction.

I read the merged mooncake.reshard.kv_cache._store package before replying, and it answers the two questions I raised when I opened the slice, in the same direction I had prototyped:

Part byte geometry is derived, not persisted. KVCacheStoreLayout documents "Persist only shard geometry and axis order. Sizes and strides are derived", and fragments() derives the affine offsets without enumerating tokens or bytes. That was exactly the representation I was arguing for (one representation of byte geometry, derived from coverage + axis order), so no divergence to reconcile.
Manifests are content-addressed and verified on decode. kv_cache_store_layout_from_json validates exact fields, full shard coverage and the canonical (layout_id, digest) pair, which also settles the lifetime question for now: immutable + digest-checked is compatible with adding a reclamation policy later.

Since the merged work also covers the full PLHD/LPHD/HPLD matrix with byte-equality tests and adds upload planning, binding, serde and TP/PP placement, I have closed my draft PR (#4741) rather than keeping a parallel store_reshard package — a second implementation of the same abstraction would only cost review time.

Two things I would be glad to help with, if useful:

The multi-part write protocol (#4366) follow-up. My slice deliberately stayed on the read/plan side and left part_count/part_index and per-part completion alone; if it helps, I can pick up a piece there once the direction you and the author settle on is public.
A read-side guard for the case the plan does not cover: a reader whose own layout is not one of the three formats, or whose head interval spans parts the manifest no longer covers. The merged planner is strict about coverage, so this may already be handled — happy to check rather than assume.

Slack works for me too if that is easier than the issue thread.

@kunkunblueberry commented on GitHub (Oct 22, 2026): No problem — thanks for the update, and for confirming the direction. I read the merged `mooncake.reshard.kv_cache._store` package before replying, and it answers the two questions I raised when I opened the slice, in the same direction I had prototyped: 1. **Part byte geometry is derived, not persisted.** `KVCacheStoreLayout` documents "Persist only shard geometry and axis order. Sizes and strides are derived", and `fragments()` derives the affine offsets without enumerating tokens or bytes. That was exactly the representation I was arguing for (one representation of byte geometry, derived from coverage + axis order), so no divergence to reconcile. 2. **Manifests are content-addressed and verified on decode.** `kv_cache_store_layout_from_json` validates exact fields, full shard coverage and the canonical `(layout_id, digest)` pair, which also settles the lifetime question for now: immutable + digest-checked is compatible with adding a reclamation policy later. Since the merged work also covers the full PLHD/LPHD/HPLD matrix with byte-equality tests and adds upload planning, binding, serde and TP/PP placement, I have closed my draft PR (#4741) rather than keeping a parallel `store_reshard` package — a second implementation of the same abstraction would only cost review time. Two things I would be glad to help with, if useful: - **The multi-part write protocol (#4366) follow-up.** My slice deliberately stayed on the read/plan side and left `part_count`/`part_index` and per-part completion alone; if it helps, I can pick up a piece there once the direction you and the author settle on is public. - **A read-side guard for the case the plan does not cover**: a reader whose own layout is not one of the three formats, or whose head interval spans parts the manifest no longer covers. The merged planner is strict about coverage, so this may already be handled — happy to check rather than assume. Slack works for me too if that is easier than the issue thread.