kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3111] [RFC]: Manifest-driven heterogeneous model weight resharding and storage #797

New issue

Open

opened 2026-10-29 21:19:38 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:19:38 +00:00

Owner

Originally created by @Bo-Vincent on GitHub (Jul 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3111

Changes proposed

Summary

This RFC proposes manifest-driven heterogeneous model-weight resharding and storage for Mooncake. A source and target may use different TP, PP, EP, and DP layouts while sharing one model-semantic manifest, N-D logical-box planner, and backend-neutral transfer-region contract.

The proposed contract supports three paths:

live runtime GPU-to-GPU (G2G) transfer;
runtime-to-Store upload; and
Store-to-runtime load.

Live G2G and Store-to-runtime use a LogicalTransferPlan. Runtime-to-Store uses a separate WeightUploadPlan, while sharing tensor identity, DP replica selection, coverage, and fragment semantics with the load path.

The framework runtime remains the source of truth for model semantics and live GPU addresses. Mooncake consumes explicit placement and runtime-binding data, computes N-D overlaps in global logical-tensor coordinates, and lowers the result into Transfer Engine (TE), Store, or future executor operations.

The first version only accepts byte-compatible layouts. A mismatch in dtype, shape, item size, or layout_fingerprint fails closed.

Motivation

RL rollouts, elastic scale-out, and inference replacement can require a conversion such as:

TP4 / PP2 / EP8 / DP2  ->  TP8 / PP4 / EP2 / DP4

The existing one-dimensional TP overlap model cannot express all of the following together:

source and target sharding different tensor dimensions;
pipeline-parallel layer or tensor ownership changes;
expert-parallel and tensor-parallel changes in the same tensor; and
a common load contract for live GPU and persistent Store sources.

Mooncake already provides TE, Store, Group, leases, memory registration, and ranged transfer. What is missing is a stable contract between framework-owned model semantics and Mooncake-owned physical transfer and storage operations.

Goals and non-goals

Goals:

Support joint TP, PP, EP, and DP changes, including different source and target shard dimensions.
Persist source-native TP/PP/EP fragments as an immutable weight revision that can serve future target layouts.
Avoid model-wide, tensor-wide, or all-expert all-gathers.
Avoid a model-sized temporary GPU or CPU buffer.
Bind one logical load plan to either live-runtime or Store fragments; keep a dedicated upload plan for the write path.
Preserve legacy manifests, one-dimensional TP behavior, and normal Store/TE/KVCache paths.
Preserve lease, generation, coverage, layout, and address-bounds checks.

Non-goals:

Inferring tensor, layer, or expert semantics from parameter names in Mooncake core.
Owning source discovery, Kubernetes scheduling, or target traffic activation.
Requiring NCCL M2N, ncclMemAlloc, or any particular framework.

Proposed design

flowchart TB
    P["Source Placement + Target Placement"]
    L["LogicalTransferPlan<br/>address-independent"]
    B["Source Binding + Target Binding"]
    T["TransferPlan<br/>executable"]
    E["TE / Store executor"]

    P -->|"N-D planning"| L
    L --> T
    B -->|"bind and validate"| T
    T --> E

Placement is address-independent and is the only input to N-D planning. Runtime
Binding adds the current addresses, endpoints, generation, and lease immediately
before execution. Address changes therefore require rebinding, not replanning.
For a Store source, the stored fragment/object-range mapping acts as the source
binding; only the target requires a live Runtime Binding.

Manifest contract

Manifest	Purpose	Key fields
Placement	Address-independent planning	tensor identity, global/local shape, logical offset, dtype/layout, layer/expert owner, TP/PP/EP/DP rank
Runtime Binding	Binds one live allocation	placement ID, instance, generation, lease, address, size, device, endpoint
Store Weight Manifest	Publishes one persistent revision	model/revision, group, stored fragments, object key/offset, coverage, optional checksum

Placement + Runtime Binding = executable runtime manifest

Frameworks export Placement and Runtime Binding. Mooncake neither imports SGLang, vLLM, or slime nor maintains model-specific naming rules. placement_id is derived only from canonical logical content, never from an address, endpoint, or lease. A legacy partition_dim=d remains compatible as shard_dims=[d].

The Store Weight Manifest never stores a live address. Group provides co-routing and resource coordination; the Weight Manifest defines a complete revision and indexes its payloads. A production multi-writer implementation requires revision compare-and-swap plus OPEN -> COMMITTED/ABORTED arbitration. The current prototype uses a single-writer commit protocol.

N-D reshard planner

Each fragment is an N-D box in a global logical tensor. The planner directly computes source/target box intersections:

TP: source and target may shard different tensor dimensions.
PP: routing follows the real layer/tensor owner recorded by Placement.
EP: an expert is a leading logical coordinate; independent per-expert allocations remain valid fragments.
DP: G2G selects one complete source replica; Store persists one complete DP replica.

The four axes are planned jointly: ownership and replica selection are followed by one N-D intersection calculation. The planner never creates an intermediate TP, PP, or EP model layout.

Each logical region contains the overlap offset and shape, source and target base offsets, inner_bytes, outer-loop counts, and source/target strides. Physical addresses, object keys, leases, and registrations are attached only during binding.

Lowering must be bounded. The prototype caps per-region segments and lazily batches operations with max_batch_operations; the production contract must also cap total-plan operations and per-batch bytes. It must not expand a region into one operation per row or element.

Weight storage

Store retains source-native TP/PP/EP fragments and their logical coordinates. It does not all-gather or rewrite them into a fixed parallel layout before persisting. One generation-consistent, complete DP replica is selected. Consequently, stored payload is approximately one logical model: TP, PP, and EP alter fragment count and shape, while DP does not multiply storage capacity.

weights/<namespace>/<model_id>/<revision>/
  payload/<upload_id>/<fragment_id>  source-native bytes
  manifest                           immutable revision visibility point
  sessions/<upload_id>/decision      commit or abort

The write path is:

prepare_upload() validates model/revision, tensor descriptors, and full coverage; selects one complete DP replica; and returns a WeightUploadPlan.
Workers call batch_put_from() to upload registered GPU fragments as independent ObjectDataType.WEIGHT payloads. Independent experts remain independent objects.
The coordinator collects complete receipts, verifies payload existence, and commits the session decision.
It publishes an immutable WeightManifest. Payloads without a published manifest are not readable as a weight revision.

The read path combines WeightManifest + Target Placement into an N-D load plan and uses get_into_ranges() to fill target final buffers directly:

TP4/PP2/EP8/DP2 runtime
  -> Store: source-native TP4/PP2/EP8 fragments, one DP replica
  -> TP8/PP4/EP2/DP4 runtime

Plans and manifests contain metadata, not model bytes. Upload reads registered source-GPU ranges; load writes target final buffers. max_range_bytes and max_ranges_per_request bound staging and submission, so no model-scale temporary buffer is required.

group_id groups a revision's payloads, manifest, and session metadata. Group infrastructure provides physical co-routing, lease refresh, hard pinning, and eviction coordination. The manifest provides the logical tensor directory and revision visibility. A Group is neither a cross-object transaction nor a model serving state machine.

The prototype implements upload, commit, abort, load, session decisions, conflict detection, coverage validation, and an optional checksum field. It does not yet enforce checksum production/verification, create-only revision CAS, cross-node READY/SERVING state, serving leases, or a tier-residency catalog. Those lifecycle states should extend the Model Weight Management (WML) control plane proposed by RFC #2282:

IMPORTING -> READY -> SERVING -> IDLE -> EVICTING/EVICTED

RAM/L3 paths are covered by the prototype. SSD/OSS still require ranged-read, bounded-staging, and recovery validation.

Execution paths

Live G2G. The source is snapshotted, target final buffers are bound, and TE executes the plan. The target activates only after completion validation and a world barrier. The source may continue read-only inference while snapshotted, but cannot mutate, free, or unregister the weights.

Runtime-to-Store. One complete source DP replica uploads source-native fragments to immutable payloads. A WeightManifest becomes visible only after all payload checks pass. This path uses WeightUploadPlan and does not assume that upload and load share the same physical operation form.

Store-to-runtime. A Store Weight Manifest and a target Placement produce a logical load plan. Ranged reads write target final buffers directly. RAM/L3 can serve ranges directly; SSD/OSS may use fixed chunks or bounded staging.

A checkpoint or object file can later be another persistent source, but its model semantics and post-load transforms remain framework-loader work.

Compatibility and safety

Only an explicit Weight Transfer API enters this planner. Existing KVCache and ordinary TE operations retain their current paths.

Before any data operation, Mooncake validates model/revision, schema, placement digest, tensor layout, generation, lease, registered address ranges, selected source replica, and target coverage. A stale generation, revoked lease, layout mismatch, coverage gap, or out-of-bounds address fails closed.

When DMA completion is uncertain, source ownership, registration, and the snapshot remain retained. Legacy runtime_v1, one-dimensional TP, and existing Store/TE paths remain available through compatibility adapters. Unknown capabilities must not silently fall back to an unchecked copy.

Prototype status and validation

Reference implementation: vin/heterogeneous-weight-transfer-v3at a2467783ed01556227bee25956ca47a590ec6384.

Companion SGLang integration: vin/runtime-weight-manifest-v3at 288705ffdd0ea16d38b98676dac3412984cd3583.

The prototype includes Placement/Binding, v1 compatibility, a TP/PP/EP/DP N-D planner, bounded TE lazy lowering, runtime G2G, Store upload/load, and lease, generation, bounds, and completion protection. Mooncake core contains no model naming rules.

"Supported" below means that a framework has an adapter, live runtime binding, and a validation loop. It does not mean that the generic planner alone supports every model.

Framework	Prototype status	Boundary
SGLang	Personal branch; live G2G serving E2E completed; Store contract/library tests exist	Not upstream; production revision barrier, discovery, and traffic activation remain open
vLLM	Not integrated	Needs Placement/Binding export, snapshot lifecycle, and target loader
slime / verl	Not directly integrated	Can orchestrate an SGLang rollout but still need discovery, barrier, rollback, and activation integration
Other frameworks	Protocol-level only	Require Placement/Binding export and conformance validation

Model/layout in SGLang	Prototype status	Evidence
Qwen3 dense / MoE	Adapter implemented	Unit/contract tests; Qwen3 MoE 235B FP8 TP4/PP1/EP4/DP1 -> TP2/PP2/EP2/DP1 serving E2E
Qwen3.5 dense / MoE text	Adapter implemented	Qwen3.5-0.8B TP1 -> TP2 serving E2E; MoE cross-repository contract
Qwen3.5 multimodal / multimodal MoE	Adapter implemented	Unit/manifest contract only; no VL serving E2E yet
Qwen3-Next	Adapter implemented; canonical Triton MoE runner required	Unit/contract and TP4/PP1/EP4/DP1 -> TP2/PP1/EP2/DP1 serving E2E
DeepSeek, Llama, Mistral/Mixtral, and others	Not adapted

The Mooncake test matrix covers bidirectional TP/PP/EP changes, EP/TP cross-dimension cases, a four-axis combination, Store/G2G sources, and legacy manifest regression. Five dual-host H20 RDMA 4/8 GiB cases passed content verification. A dim0-to-dim2 case currently produces 2,097,152 CopyRange operations. Scatter batching from PR #3000 can reduce submission overhead but cannot remove this region count; compressed strided lowering needs a separate design.

Alternatives considered

Framework-only conversion. Frameworks may retain their own planners, but Store could not independently reshard-load a revision for a target Placement.
Model adapters in Mooncake core. This duplicates and drifts model rules; semantics must come from a framework manifest.
All-gather into one storage layout before upload. This increases peak memory and network traffic; Store instead preserves source-native fragments.
Hard dependency on NCCL M2N. It may be an optimized executor when its topology and allocation constraints hold, but it is not a correctness dependency.

Phasing

Phase	Scope
1. Core contract	Stabilize Placement, Binding, N-D regions, legacy-manifest compatibility, and TE fallback
2. Weight storage	Integrate `WeightStore` with Group/WML; add revision CAS, mandatory integrity, READY/serving lifecycle, and SSD/OSS recovery
3. Executor	Integrate scatter batching from PR #3000, design compressed strided lowering, and evaluate an optional M2N executor
4. L3 weight write	Add the Weight Write API, GPU-range direct write, bounded batching/concurrency, receipt recovery, and revision publish/abort semantics
5. Frameworks	Upstream SGLang integration, then vLLM; add discovery, barriers, rollback, and activation integration for slime/verl
6. Models	Keep Qwen3/Qwen3.5/Qwen3-Next as the baseline; add DeepSeek-V2/V3/R1, Llama/Mistral, Mixtral, and multimodal conformance/E2E

Model-semantic adapters belong in SGLang, vLLM, and other framework repositories. Mooncake owns the schema, planner, Store/TE executors, and cross-framework conformance fixtures; it must not duplicate model rules in core.

Open questions

Which component should persist and publish Store Weight Manifest and the RFC #2282 WML revision metadata, and what should the complete weight lifecycle look like?
Should M2N be added as an optional fast path? Which topology and allocation conditions should enable an NCCL M2N executor?
Can this architecture later support additional forms of conversion, for example FP16-to-FP8 conversion?

Unified KVCache and Model Weight Management RFC #2282
Scatter transfer batching PR #3000
NVIDIA NCCL M2N
SGLang Runtime Weight Manifest companion RFC (issue link to be added)

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Bo-Vincent on GitHub (Jul 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3111 ### Changes proposed ## Changes proposed ### Summary This RFC proposes manifest-driven heterogeneous model-weight resharding and storage for Mooncake. A source and target may use different `TP`, `PP`, `EP`, and `DP` layouts while sharing one model-semantic manifest, N-D logical-box planner, and backend-neutral transfer-region contract. The proposed contract supports three paths: * live runtime GPU-to-GPU (G2G) transfer; * runtime-to-Store upload; and * Store-to-runtime load. Live G2G and Store-to-runtime use a `LogicalTransferPlan`. Runtime-to-Store uses a separate `WeightUploadPlan`, while sharing tensor identity, DP replica selection, coverage, and fragment semantics with the load path. The framework runtime remains the source of truth for model semantics and live GPU addresses. Mooncake consumes explicit placement and runtime-binding data, computes N-D overlaps in global logical-tensor coordinates, and lowers the result into Transfer Engine (TE), Store, or future executor operations. The first version only accepts byte-compatible layouts. A mismatch in dtype, shape, item size, or `layout_fingerprint` fails closed. ### Motivation RL rollouts, elastic scale-out, and inference replacement can require a conversion such as: ```text TP4 / PP2 / EP8 / DP2 -> TP8 / PP4 / EP2 / DP4 ``` The existing one-dimensional TP overlap model cannot express all of the following together: * source and target sharding different tensor dimensions; * pipeline-parallel layer or tensor ownership changes; * expert-parallel and tensor-parallel changes in the same tensor; and * a common load contract for live GPU and persistent Store sources. Mooncake already provides TE, Store, Group, leases, memory registration, and ranged transfer. What is missing is a stable contract between framework-owned model semantics and Mooncake-owned physical transfer and storage operations. ### Goals and non-goals Goals: * Support joint TP, PP, EP, and DP changes, including different source and target shard dimensions. * Persist source-native TP/PP/EP fragments as an immutable weight revision that can serve future target layouts. * Avoid model-wide, tensor-wide, or all-expert all-gathers. * Avoid a model-sized temporary GPU or CPU buffer. * Bind one logical load plan to either live-runtime or Store fragments; keep a dedicated upload plan for the write path. * Preserve legacy manifests, one-dimensional TP behavior, and normal Store/TE/KVCache paths. * Preserve lease, generation, coverage, layout, and address-bounds checks. Non-goals: * Inferring tensor, layer, or expert semantics from parameter names in Mooncake core. * Owning source discovery, Kubernetes scheduling, or target traffic activation. * Requiring NCCL M2N, `ncclMemAlloc`, or any particular framework. ### Proposed design ```mermaid flowchart TB P["Source Placement + Target Placement"] L["LogicalTransferPlan<br/>address-independent"] B["Source Binding + Target Binding"] T["TransferPlan<br/>executable"] E["TE / Store executor"] P -->|"N-D planning"| L L --> T B -->|"bind and validate"| T T --> E ``` Placement is address-independent and is the only input to N-D planning. Runtime Binding adds the current addresses, endpoints, generation, and lease immediately before execution. Address changes therefore require rebinding, not replanning. For a Store source, the stored fragment/object-range mapping acts as the source binding; only the target requires a live Runtime Binding. #### Manifest contract | Manifest | Purpose | Key fields | | --- | --- | --- | | Placement | Address-independent planning | tensor identity, global/local shape, logical offset, dtype/layout, layer/expert owner, TP/PP/EP/DP rank | | Runtime Binding | Binds one live allocation | placement ID, instance, generation, lease, address, size, device, endpoint | | Store Weight Manifest | Publishes one persistent revision | model/revision, group, stored fragments, object key/offset, coverage, optional checksum | ```text Placement + Runtime Binding = executable runtime manifest ``` Frameworks export Placement and Runtime Binding. Mooncake neither imports SGLang, vLLM, or slime nor maintains model-specific naming rules. `placement_id` is derived only from canonical logical content, never from an address, endpoint, or lease. A legacy `partition_dim=d` remains compatible as `shard_dims=[d]`. The Store Weight Manifest never stores a live address. Group provides co-routing and resource coordination; the Weight Manifest defines a complete revision and indexes its payloads. A production multi-writer implementation requires revision compare-and-swap plus `OPEN -> COMMITTED/ABORTED` arbitration. The current prototype uses a single-writer commit protocol. #### N-D reshard planner Each fragment is an N-D box in a global logical tensor. The planner directly computes source/target box intersections: * **TP:** source and target may shard different tensor dimensions. * **PP:** routing follows the real layer/tensor owner recorded by Placement. * **EP:** an expert is a leading logical coordinate; independent per-expert allocations remain valid fragments. * **DP:** G2G selects one complete source replica; Store persists one complete DP replica. The four axes are planned jointly: ownership and replica selection are followed by one N-D intersection calculation. The planner never creates an intermediate TP, PP, or EP model layout. Each logical region contains the overlap offset and shape, source and target base offsets, `inner_bytes`, outer-loop counts, and source/target strides. Physical addresses, object keys, leases, and registrations are attached only during binding. Lowering must be bounded. The prototype caps per-region segments and lazily batches operations with `max_batch_operations`; the production contract must also cap total-plan operations and per-batch bytes. It must not expand a region into one operation per row or element. #### Weight storage Store retains source-native TP/PP/EP fragments and their logical coordinates. It does not all-gather or rewrite them into a fixed parallel layout before persisting. One generation-consistent, complete DP replica is selected. Consequently, stored payload is approximately one logical model: TP, PP, and EP alter fragment count and shape, while DP does not multiply storage capacity. ```text weights/<namespace>/<model_id>/<revision>/ payload/<upload_id>/<fragment_id> source-native bytes manifest immutable revision visibility point sessions/<upload_id>/decision commit or abort ``` The write path is: 1. `prepare_upload()` validates model/revision, tensor descriptors, and full coverage; selects one complete DP replica; and returns a `WeightUploadPlan`. 2. Workers call `batch_put_from()` to upload registered GPU fragments as independent `ObjectDataType.WEIGHT` payloads. Independent experts remain independent objects. 3. The coordinator collects complete receipts, verifies payload existence, and commits the session decision. 4. It publishes an immutable `WeightManifest`. Payloads without a published manifest are not readable as a weight revision. The read path combines `WeightManifest + Target Placement` into an N-D load plan and uses `get_into_ranges()` to fill target final buffers directly: ```text TP4/PP2/EP8/DP2 runtime -> Store: source-native TP4/PP2/EP8 fragments, one DP replica -> TP8/PP4/EP2/DP4 runtime ``` Plans and manifests contain metadata, not model bytes. Upload reads registered source-GPU ranges; load writes target final buffers. `max_range_bytes` and `max_ranges_per_request` bound staging and submission, so no model-scale temporary buffer is required. `group_id` groups a revision's payloads, manifest, and session metadata. Group infrastructure provides physical co-routing, lease refresh, hard pinning, and eviction coordination. The manifest provides the logical tensor directory and revision visibility. A Group is neither a cross-object transaction nor a model serving state machine. The prototype implements upload, commit, abort, load, session decisions, conflict detection, coverage validation, and an optional checksum field. It does not yet enforce checksum production/verification, create-only revision CAS, cross-node `READY/SERVING` state, serving leases, or a tier-residency catalog. Those lifecycle states should extend the Model Weight Management (WML) control plane proposed by RFC #2282: ```text IMPORTING -> READY -> SERVING -> IDLE -> EVICTING/EVICTED ``` RAM/L3 paths are covered by the prototype. SSD/OSS still require ranged-read, bounded-staging, and recovery validation. #### Execution paths **Live G2G.** The source is snapshotted, target final buffers are bound, and TE executes the plan. The target activates only after completion validation and a world barrier. The source may continue read-only inference while snapshotted, but cannot mutate, free, or unregister the weights. **Runtime-to-Store.** One complete source DP replica uploads source-native fragments to immutable payloads. A `WeightManifest` becomes visible only after all payload checks pass. This path uses `WeightUploadPlan` and does not assume that upload and load share the same physical operation form. **Store-to-runtime.** A Store Weight Manifest and a target Placement produce a logical load plan. Ranged reads write target final buffers directly. RAM/L3 can serve ranges directly; SSD/OSS may use fixed chunks or bounded staging. A checkpoint or object file can later be another persistent source, but its model semantics and post-load transforms remain framework-loader work. ### Compatibility and safety Only an explicit Weight Transfer API enters this planner. Existing KVCache and ordinary TE operations retain their current paths. Before any data operation, Mooncake validates model/revision, schema, placement digest, tensor layout, generation, lease, registered address ranges, selected source replica, and target coverage. A stale generation, revoked lease, layout mismatch, coverage gap, or out-of-bounds address fails closed. When DMA completion is uncertain, source ownership, registration, and the snapshot remain retained. Legacy `runtime_v1`, one-dimensional TP, and existing Store/TE paths remain available through compatibility adapters. Unknown capabilities must not silently fall back to an unchecked copy. ### Prototype status and validation Reference implementation: [vin/heterogeneous-weight-transfer-v3](https://github.com/Bo-Vincent/Mooncake/tree/vin/heterogeneous-weight-transfer-v3)at `a2467783ed01556227bee25956ca47a590ec6384`. Companion SGLang integration: [vin/runtime-weight-manifest-v3](https://github.com/Bo-Vincent/vin-sglang/tree/vin/runtime-weight-manifest-v3)at `288705ffdd0ea16d38b98676dac3412984cd3583`. The prototype includes Placement/Binding, v1 compatibility, a TP/PP/EP/DP N-D planner, bounded TE lazy lowering, runtime G2G, Store upload/load, and lease, generation, bounds, and completion protection. Mooncake core contains no model naming rules. "Supported" below means that a framework has an adapter, live runtime binding, and a validation loop. It does not mean that the generic planner alone supports every model. | Framework | Prototype status | Boundary | | --- | --- | --- | | SGLang | Personal branch; live G2G serving E2E completed; Store contract/library tests exist | Not upstream; production revision barrier, discovery, and traffic activation remain open | | vLLM | Not integrated | Needs Placement/Binding export, snapshot lifecycle, and target loader | | slime / verl | Not directly integrated | Can orchestrate an SGLang rollout but still need discovery, barrier, rollback, and activation integration | | Other frameworks | Protocol-level only | Require Placement/Binding export and conformance validation | | Model/layout in SGLang | Prototype status | Evidence | | --- | --- | --- | | Qwen3 dense / MoE | Adapter implemented | Unit/contract tests; Qwen3 MoE 235B FP8 TP4/PP1/EP4/DP1 -> TP2/PP2/EP2/DP1 serving E2E | | Qwen3.5 dense / MoE text | Adapter implemented | Qwen3.5-0.8B TP1 -> TP2 serving E2E; MoE cross-repository contract | | Qwen3.5 multimodal / multimodal MoE | Adapter implemented | Unit/manifest contract only; no VL serving E2E yet | | Qwen3-Next | Adapter implemented; canonical Triton MoE runner required | Unit/contract and TP4/PP1/EP4/DP1 -> TP2/PP1/EP2/DP1 serving E2E | | DeepSeek, Llama, Mistral/Mixtral, and others | Not adapted | | The Mooncake test matrix covers bidirectional TP/PP/EP changes, EP/TP cross-dimension cases, a four-axis combination, Store/G2G sources, and legacy manifest regression. Five dual-host H20 RDMA 4/8 GiB cases passed content verification. A dim0-to-dim2 case currently produces 2,097,152 `CopyRange` operations. Scatter batching from PR #3000 can reduce submission overhead but cannot remove this region count; compressed strided lowering needs a separate design. ### Alternatives considered * **Framework-only conversion.** Frameworks may retain their own planners, but Store could not independently reshard-load a revision for a target Placement. * **Model adapters in Mooncake core.** This duplicates and drifts model rules; semantics must come from a framework manifest. * **All-gather into one storage layout before upload.** This increases peak memory and network traffic; Store instead preserves source-native fragments. * **Hard dependency on NCCL M2N.** It may be an optimized executor when its topology and allocation constraints hold, but it is not a correctness dependency. ### Phasing | Phase | Scope | | --- | --- | | 1. Core contract | Stabilize Placement, Binding, N-D regions, legacy-manifest compatibility, and TE fallback | | 2. Weight storage | Integrate `WeightStore` with Group/WML; add revision CAS, mandatory integrity, READY/serving lifecycle, and SSD/OSS recovery | | 3. Executor | Integrate scatter batching from PR #3000, design compressed strided lowering, and evaluate an optional M2N executor | | 4. L3 weight write | Add the Weight Write API, GPU-range direct write, bounded batching/concurrency, receipt recovery, and revision publish/abort semantics | | 5. Frameworks | Upstream SGLang integration, then vLLM; add discovery, barriers, rollback, and activation integration for slime/verl | | 6. Models | Keep Qwen3/Qwen3.5/Qwen3-Next as the baseline; add DeepSeek-V2/V3/R1, Llama/Mistral, Mixtral, and multimodal conformance/E2E | Model-semantic adapters belong in SGLang, vLLM, and other framework repositories. Mooncake owns the schema, planner, Store/TE executors, and cross-framework conformance fixtures; it must not duplicate model rules in core. ### Open questions 1. Which component should persist and publish Store Weight Manifest and the RFC #2282 WML revision metadata, and what should the complete weight lifecycle look like? 2. Should M2N be added as an optional fast path? Which topology and allocation conditions should enable an NCCL M2N executor? 3. Can this architecture later support additional forms of conversion, for example FP16-to-FP8 conversion? ### Related work * [Unified KVCache and Model Weight Management RFC #2282](https://github.com/kvcache-ai/Mooncake/issues/2282) * [Scatter transfer batching PR #3000](https://github.com/kvcache-ai/Mooncake/pull/3000) * [NVIDIA NCCL M2N](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_m2n) * SGLang Runtime Weight Manifest companion RFC (issue link to be added) ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

stale

labels

2026-10-29 21:19:38 +00:00

bsakaguchi commented

2026-10-29 21:19:39 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 24, 2026):

Thanks for opening this issue, @Bo-Vincent!

Field	Value
Issue	#3111
GitHub user ID	`205891898`
Reporter	@Bo-Vincent

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 24, 2026): Thanks for opening this issue, @Bo-Vincent! | Field | Value | |-------|-------| | **Issue** | #3111 | | **GitHub user ID** | `205891898` | | **Reporter** | @Bo-Vincent | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.