kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1883] [RoadMap][Call For Contribution] Mooncake Project Overall Roadmap #535

New issue

Open

opened 2026-10-29 21:16:04 +00:00 by bsakaguchi · 14 comments

bsakaguchi commented

2026-10-29 21:16:04 +00:00

Owner

Originally created by @stmatengss on GitHub (Apr 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1883

Milestone 1: Inference & RL Framework Integration

Deepen integration with mainstream frameworks and enhance support for large-scale architectures.

(Agentic-aware): Support agentic-aware KVCache System in inference frameworks.
(PD Disaggregation): Optimize PD with more features and support SOTA models.
(Hybrid Model): Optimize performance for New Models.
(RL Disaggregation): Separate Rollout (sampling) and Update (weight updating) using a globally shared KVCache pool and efficient sample flow for higher PPO/RLHF throughput.

Milestone 2: Store V3 Evolution

Enhance the core storage system for better modularity, elasticity, and high concurrency.

(TE & Store Decoupling): Completely separate Transfer Engine (TE) and Store into independent packages.
(Tensor-Native & Multi-Replica): Integrate TP rank and model info into APIs; support replica lists for high-frequency KVCache items.
(Client/Worker Decoupling): Separate dummy client from real client for flexible, highly available deployment topologies.
(Engram-like Backend): Introduce an optimized storage abstraction for N-gram key-value operations to handle highly sparse and concurrent random access.
(Object Grouping): Allow specifying multiple objects as part of the same group during put operations, with visibility and eviction handled at the group level
(Visibility Enhancement): Further improve visibility by providing more flexible and comprehensive metrics querying capabilities
(Memory Allocation Optimization): OffsetAllocator is currently optimized for uniform size distributions, but there are now cases with two or more size classes; targeted optimizations are needed to improve memory utilization

Store V3 RFCs:

Mooncake #1035 — [RoadMap] Mooncake Store V3 Roadmap
Mooncake #1836 — [RFC] Mooncake Store Multi-Tenant Isolation
Mooncake #1732 — [RFC] Support Radix Tree Index in Store Master for Prefix Query
Mooncake #1772 — [RFC] KVPool eviction: evict suffix keys when prefix keys are missing
Mooncake #1645 — [RFC] Hard Pin and Upsert for Mooncake Store
Mooncake #1648 — [RFC] Kubernetes Lease-Based HA Without External etcd
Mooncake #1650 — [RFC] HA OpLog Store Abstraction Layer
Mooncake #1673 — [RFC] Data Type Support in Mooncake Store
Mooncake #1372 — [RFC] Support Engram as a storage backend in Mooncake
Mooncake #1150 — [RFC] Persistence and Recovery Solution for KV Metadata and Segment Information
Mooncake #1131 — [RFC] Refactor allocation strategy for better performance and flexibility
Mooncake #1100 — [RFC] Dynamic replica management
Mooncake #1062 — [RFC] Add Local Cache Mechanism for Mooncake Store Client
Mooncake #1054 — [RFC] Contribute Local Storage to Distributed Pool
Mooncake #972 — [RFC] More KVCache metrics in both master/client side
Mooncake #961 — [RFC] Fake Client for Mooncake Client
Mooncake #1809 — [RFC] Integrate Mooncake Store into Dynamo KVCM with Native Rust API (Closed)
Mooncake #1920 — [RFC] Rolling Upgrade Support for Mooncake Store
Mooncake #1957 — [RFC] Mooncake Store NVMe KV Backend
Mooncake #1979 — [RFC] Dynamo KVBM + Mooncake Store Integration Design
Mooncake #1998 — [RFC] Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction
Mooncake #2081 — [RFC] Simplify Mooncake Store for Maintainability
Mooncake #2127 — [RFC] Lightweight Best-Effort Group Semantics for Mooncake Store Objects
Mooncake #2153 — [RFC] Dynamic Tenant Quota and Scoped-Key Isolation

Store V3 Key PRs (recent):

Mooncake #2080 — [Store] Add DSA-like workload allocation strategy (Merged)
Mooncake #2112 — [Store] feat: introduce per-tenant storage quota in master
Mooncake #2180 — [Store] Add opt-in grouped object routing semantics

Milestone 3: Transfer Engine (TENT) & Networking

Focus on declarative data orchestration and pushing hardware limits.

Release TENT V1.1 at 26Q2 @alogfans

(Declarative Orchestration): Implement a plugin-based multi-transport architecture; enable dynamic path selection via hardware auto-discovery and declarative routing policies.
(QoS-aware Slice Spraying): Introduce cross-process priority-based slice spraying, minimizing the interference between different tasks; aiming to reduce P99 latency in 8x 400 Gbps network.
(Ease of Maintenance): Decouple heterogeneous hardware support from transport protocols via a unified GPU abstraction layer and modular plugin architecture to streamline cross-platform deployment. Improve the self-healing ability.
(Transfer Queue): Implement centralized/distributed transfer queue with priority scheduling and backpressure for high-throughput, low-latency data movement under heavy load.
(NCCL Support): Use NCCL device API as an optional backend

Transfer Engine RFCs:

Mooncake #1058 — [RoadMap] Mooncake Transfer Engine NEXT
Mooncake #1850 — [RFC] End-to-End Transfer Tracing for Mooncake
Mooncake #1749 — [RFC] Optimizing Mooncake TCP Transport for High-Concurrency Throughput
Mooncake #1773 — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
Mooncake #2117 — [RFC] Hierarchical Arch for Intra/Inter Data Center Deployment
Mooncake #2123 — [RFC] Device Transport — GPU-Initiated Communication for TENT
Mooncake #2132 — [RFC] TENT Local Transfer Admission Queue
Mooncake #2132 — [RFC] [RFC]: NCCL device API in Mooncake TENT

Transfer Engine Key PRs (recent):

Mooncake #2079 — [TENT] Add rule-based transport and device selection
Mooncake #2090 — [TENT] Add cross-platform dynamic library adapter with GPU vendor abstraction
Mooncake #2155 — [TE] Improving the RDMA transport failure handling
Mooncake #2160 — [TE] Add explicit progressBatch API

Milestone 4: OS-Level Optimizations

Improve throughput and reduce latency through low-level system integrations.

(Decentralized Metadata): Migrate SSD metadata management from clients to a dedicated Master to resolve bottleneck issues.
(Multi-Level Resource Pooling): Aggregate VRAM, DRAM, NVMe, and Remote resources into a transparent cache pool using CXL and SHM.

Related RFCs:

Mooncake #1123 — [RFC] Support hugepage memory allocation via mmap to bypass RDMA memory registration limits (Closed)
Mooncake #2106 — [RFC] GPU VRAM (HBM) Pool Support for Mooncake Store

Milestone 5: Routing & Checkpoint Engine

Enhance global scheduling and model weight handling.

(KVCache-Aware Routing): Route Decode requests based on underlying KVCache distribution (History Cache) to maximize hit rates and minimize TTFT.
(Checkpoint Engine): Strengthen parallel distribution of weights via P2P Store; support native Torch format offload for fast MoE loading.

Conductor & Routing RFCs:

Mooncake #977 — [RFC] Mooncake-Conductor: Design and Implementation of a Global Scheduler
Mooncake #1767 — [RFC] Mooncake Data Plane Model Distribution, Phase 1
Mooncake #2189 — [Call For Contributions] Mooncake Conductor Roadmap

Milestone 6: Heterogeneous & Cross-Platform Inference (Domestic GPU Support)

Break hardware silos for collaborative deployment. Expand support for domestic accelerator platforms.

(Cross-Architecture Collaboration): Enable tasks to flow across different vendors (e.g., Vendor A for high-throughput Prefill, Vendor B for low-latency Decode).
(Ascend 910B / Atlas 800T): Adapt Transfer Engine and Store for Huawei Ascend NPU, including Ascend Direct protocol and HCCL integration.
(Kunpeng SuperNode UB Transport): Enable UB transport via Kunpeng UMDK URMA APIs for Kunpeng 950 platforms.
(Moore Threads MACA/MUSA): Align MACA GPU paths with Mooncake build system; integrate MUSA AI Tensor Engine (MATE) for FA3 attention backend.
(MetaX / Yunsilicon): Resolve RDMA compatibility with Yunsilicon XScale NIC and ensure stable deployment.
(AMD ROCm / HIP): Add ROCm HIP support to the Mooncake Python package for AMD GPU transport.
(AWS EFA): Enhance EFA transport with fi_read, endpoint LRU eviction, and multi-NIC striping.

Related RFCs:

Mooncake #1773 — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs
Mooncake #1058 — [RoadMap] TENT — Full Heterogeneous Deployment section (Ascend ↔ NVIDIA)
Mooncake #1035 — [RoadMap] Store V3 — Hardware mentions: AMD, Moore Threads, AWS EFA, Ascend
Mooncake #1975 — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design
Mooncake #1965 — [Feature Request] MACA Support for Mooncake EP/PG

Tracked issues (Ascend):

Mooncake #1854 — [Ascend 910B] rtsHostRegister failed in mooncake-store
Mooncake #1107 — Mooncake RDMA error on Ascend 910B
Mooncake #1766 — Failed to modify QP to RTR with Yunsilicon XScale (MetaX)

Heterogeneous Key PRs (recent):

Mooncake #2118 — feat: add Hygon DCU/DTK and Iluvatar CoreX platform support (Merged)
Mooncake #2196 — [Store] enables the Ubtransport for Mooncake Store (Kunpeng)
Mooncake #2113 — [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin

Milestone 7: Unified Resource Management & Multi-Model Serving

Manage KVCache and model weights cohesively on GPUs.

(Tiered Weight Storage): Manage weights across VRAM (hot), DRAM (warm), and SSD (cold).
(Fine-Grained GPU Sharing): Implement CUDA MPS + custom memory isolation for multi-model sharing.
(Dynamic Weight Swapping): Use Mooncake's TE for fast RDMA-based weight migration.
(Unified View): Manage both KVCache and weight pools globally.

https://github.com/kvcache-ai/Mooncake/issues/2282

Milestone 8: Fast Instance Startup

Utilize data transfer capabilities to accelerate initialization.

(Fast Model Distribution): Pull weights via RDMA from neighboring nodes instead of remote storage.
(CUDA Context Restore): Snapshot and restore initialized inference processes to skip warmup.
(Warm Pool): Maintain a pool of pre-warmed processes for stateful scaling.
(Lazy Pulling): Optimize container images to only pull needed layers.

Related RFCs:

Mooncake #1767 — [RFC] Mooncake Data Plane Model Distribution, Phase 1

Milestone 9: Elasticity & Self-Healing

Leverage decoupled architecture for robust operations.

(LLM-Aware Autoscaler): Scale based on global KVCache pressure and Prefix Tree hit rates.
(Fast GPU Recovery): Recover failed nodes in <10s by loading weights, utilizing the intact KVCache pool.
(sched_ext BPF Scheduler): Ensure lowest-latency CPU scheduling for GPU feeder threads.
(TE Fault Tolerance): Automatic fallback to TCP and re-routing on RDMA/node failures: Already supported by TENT.

Related RFCs:

Mooncake #975 — [RFC] Cleanup processing replicas if the transferring client is down (Closed)
Mooncake #1920 — [RFC] Rolling Upgrade Support for Mooncake Store

Build the foundation for complex Agent and Multi-Modal workflows.

(Agent State Cloning): Fast A2A state sharing and cross-node data transport.
(Agent PD Disaggregation): Separate resources for "thinking" vs. "interactive" agents.
(EPD Disaggregation): Separate Vision Encoder, Prefill, and Decode for multi-modal scenarios. (Qwen-series supported in sglang)
(Diffusion Serving): Explore disaggregated architectures for Diffusion models.

Omni Collaboration

(Omni Transfer Engine Integration): Integrate Mooncake Transfer Engine as a connector for inter-stage data transfer in the omni multi-modal pipeline (AR → Generation → Diffusion).
(Worker-Level Inter-Stage Transfer): Optimize worker-level data movement between omni pipeline stages via high-speed RDMA or shared memory.
(Omni Prefix Caching): Support Hidden State Prefix Caching for multi-modal omni models.

Related RFC

https://github.com/vllm-project/vllm-omni/issues/2904
https://github.com/vllm-project/vllm-omni/issues/3635
Mooncake #2098 — [RFC] Agent-Aware KV Cache Support in Mooncake (Phase 1)

Diffusion Collaboration

(Disaggregated Diffusion Deployment): Support decentralized disaggregated deployment for video generation models, decoupling transformer and decoder stages.
(Queue-based Orchestration): Implement queue-based scheduling for multi-stage diffusion serving requests.

Milestone 11: KVCache Dynamic Migration

(Live Migration): Support lossless, dynamic hot-migration of massive KVCache states across clients/nodes for Spot Instances and elastic scaling.
(Migration on Store Instance Exit): Support migrating hot data to other nodes during scale-down or rolling upgrades (optional, enabled manually).
(Load Balancing)

Related RFCs:

Mooncake #1159 — [RFC] Replica copy and move support

Migration Key PRs (recent):

Mooncake #2170 — [Store] add copy/move execution api support in master side (Merged)

Milestone 12: Hardware Support & ICMS

(New Hardware): Support Supernodes, Rubin, PPU, Matrix384.
(ICMS): Follow BF4 developments and integrate related capabilities.

Milestone 13: SSD Offload Support

(SSD offload): Support offloading KV cache to SSD, ensuring higher hit ratio in limited DRAM scenarios.
(Bandwidth optimization): Optimized bandwidth of SSD loading using io_uring lib.
(L2→L1 data promotion): If no memory replica and the object is used, promote it to memory.
(Smart eviction strategy): e.g., Selectively evict cold data with LRU-like strategy.

Related RFCs:

Mooncake #971 — [RFC] Support distributed read and write operations to SSDs
Mooncake #952 — [RFC] Support Mooncake file eviction in DFS/3FS (Closed)
Mooncake #1940 — [RFC] Implement SSD Pool over NVMe-oF in Mooncake Store
Mooncake #1957 — [RFC] Mooncake Store NVMe KV Backend
Mooncake #1990 — [RFC] Native SSD KV Storage Backend for Mooncake Store
Mooncake #2025 — [RFC] L2→L1 data promotion: If no memory replica and the object is used, promote it to memory
Mooncake #2031 — [RFC] Promote data from L2 -> L1 when it's not in L1

SSD Offload Key PRs (recent):

Mooncake #2084 — feat(nof): add NVMe-oF SSD cache support
Mooncake #2167 — [Store] introduce NVMe KV backend core
Mooncake #2172 — feat(store): add SPDK NoF worker pool
Mooncake #2176 — [Store] L2->L1 promotion-on-hit: observability metrics + max_per_heartbeat knob

Milestone 14: Mooncake PG

(Transfer Engine): Integrate TENT and refactor transfer engine management.
(Fault-Tolerance Backend): Improve FT-backend fault-handling and recovery, explore elastic backend design.
(PyTorch c10d API): Lazy initialization & Split group for faster initialization.
(Collective Communication): Support ring- and tree-topology algorithm.
(Hardware Adaption): Support Ascend, Moore Threads, and AWS EFA.
(Reliability): (1) Establish comprehensive unit-testing and benchmark suites (2) Agentic Specification for robust AI-Coding generation.

Mooncake PG Key PRs & Issues (recent):

Mooncake #2157 — [Feature Request] Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan
Mooncake #2149 — [PG] support external TE (Merged)
Mooncake #2182 — [PG] Add rank fault tolerance and degraded recovery

Milestone 15: Mooncake EP

(High Throughput Kernel): Implement high throughput kernel for high batch size inference.
(Elastic EP): Support elastically EP extension.
(TMA Engine): TMA engine optimization for Hopper-specific architecture.
(Benchmark): Comprehensive MoE kernel benchmark under incast traffic.

Mooncake EP RFCs:

Mooncake #1975 — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design

—

Milestone 16: Documentation

(Update and Refactor Docs): The documentation has become extensive; restructure it and update outdated content
(AI-Friendly Knowledge Base): Organize Mooncake’s knowledge into a format that is easy for AI systems to read and consume

To contribute features or provide feedback, please edit this issue directly (if you have write access) or comment below. I will update the roadmap accordingly.

Originally created by @stmatengss on GitHub (Apr 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1883 ### Milestone 1: Inference & RL Framework Integration Deepen integration with mainstream frameworks and enhance support for large-scale architectures. - [ ] (Agentic-aware): Support agentic-aware KVCache System in inference frameworks. - [ ] (PD Disaggregation): Optimize PD with more features and support SOTA models. - [ ] (Hybrid Model): Optimize performance for New Models. - [ ] (RL Disaggregation): Separate Rollout (sampling) and Update (weight updating) using a globally shared KVCache pool and efficient sample flow for higher PPO/RLHF throughput. --- ### Milestone 2: Store V3 Evolution Enhance the core storage system for better modularity, elasticity, and high concurrency. - [ ] (TE & Store Decoupling): Completely separate Transfer Engine (TE) and Store into independent packages. - [ ] (Tensor-Native & Multi-Replica): Integrate TP rank and model info into APIs; support replica lists for high-frequency KVCache items. - [ ] (Client/Worker Decoupling): Separate dummy client from real client for flexible, highly available deployment topologies. - [ ] (Engram-like Backend): Introduce an optimized storage abstraction for N-gram key-value operations to handle highly sparse and concurrent random access. - [ ] (Object Grouping): Allow specifying multiple objects as part of the same group during put operations, with visibility and eviction handled at the group level - [ ] (Visibility Enhancement): Further improve visibility by providing more flexible and comprehensive metrics querying capabilities - [ ] (Memory Allocation Optimization): OffsetAllocator is currently optimized for uniform size distributions, but there are now cases with two or more size classes; targeted optimizations are needed to improve memory utilization **Store V3 RFCs:** - Mooncake [#1035](https://github.com/kvcache-ai/Mooncake/issues/1035) — [RoadMap] Mooncake Store V3 Roadmap - Mooncake [#1836](https://github.com/kvcache-ai/Mooncake/issues/1836) — [RFC] Mooncake Store Multi-Tenant Isolation - Mooncake [#1732](https://github.com/kvcache-ai/Mooncake/issues/1732) — [RFC] Support Radix Tree Index in Store Master for Prefix Query - Mooncake [#1772](https://github.com/kvcache-ai/Mooncake/issues/1772) — [RFC] KVPool eviction: evict suffix keys when prefix keys are missing - Mooncake [#1645](https://github.com/kvcache-ai/Mooncake/issues/1645) — [RFC] Hard Pin and Upsert for Mooncake Store - Mooncake [#1648](https://github.com/kvcache-ai/Mooncake/issues/1648) — [RFC] Kubernetes Lease-Based HA Without External etcd - Mooncake [#1650](https://github.com/kvcache-ai/Mooncake/issues/1650) — [RFC] HA OpLog Store Abstraction Layer - Mooncake [#1673](https://github.com/kvcache-ai/Mooncake/issues/1673) — [RFC] Data Type Support in Mooncake Store - Mooncake [#1372](https://github.com/kvcache-ai/Mooncake/issues/1372) — [RFC] Support Engram as a storage backend in Mooncake - Mooncake [#1150](https://github.com/kvcache-ai/Mooncake/issues/1150) — [RFC] Persistence and Recovery Solution for KV Metadata and Segment Information - Mooncake [#1131](https://github.com/kvcache-ai/Mooncake/issues/1131) — [RFC] Refactor allocation strategy for better performance and flexibility - Mooncake [#1100](https://github.com/kvcache-ai/Mooncake/issues/1100) — [RFC] Dynamic replica management - Mooncake [#1062](https://github.com/kvcache-ai/Mooncake/issues/1062) — [RFC] Add Local Cache Mechanism for Mooncake Store Client - Mooncake [#1054](https://github.com/kvcache-ai/Mooncake/issues/1054) — [RFC] Contribute Local Storage to Distributed Pool - Mooncake [#972](https://github.com/kvcache-ai/Mooncake/issues/972) — [RFC] More KVCache metrics in both master/client side - Mooncake [#961](https://github.com/kvcache-ai/Mooncake/issues/961) — [RFC] Fake Client for Mooncake Client - Mooncake [#1809](https://github.com/kvcache-ai/Mooncake/issues/1809) — [RFC] Integrate Mooncake Store into Dynamo KVCM with Native Rust API *(Closed)* - Mooncake [#1920](https://github.com/kvcache-ai/Mooncake/issues/1920) — [RFC] Rolling Upgrade Support for Mooncake Store - Mooncake [#1957](https://github.com/kvcache-ai/Mooncake/issues/1957) — [RFC] Mooncake Store NVMe KV Backend - Mooncake [#1979](https://github.com/kvcache-ai/Mooncake/issues/1979) — [RFC] Dynamo KVBM + Mooncake Store Integration Design - Mooncake [#1998](https://github.com/kvcache-ai/Mooncake/issues/1998) — [RFC] Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction - Mooncake [#2081](https://github.com/kvcache-ai/Mooncake/issues/2081) — [RFC] Simplify Mooncake Store for Maintainability - Mooncake [#2127](https://github.com/kvcache-ai/Mooncake/issues/2127) — [RFC] Lightweight Best-Effort Group Semantics for Mooncake Store Objects - Mooncake [#2153](https://github.com/kvcache-ai/Mooncake/issues/2153) — [RFC] Dynamic Tenant Quota and Scoped-Key Isolation **Store V3 Key PRs (recent):** - Mooncake [#2080](https://github.com/kvcache-ai/Mooncake/pull/2080) — [Store] Add DSA-like workload allocation strategy *(Merged)* - Mooncake [#2112](https://github.com/kvcache-ai/Mooncake/pull/2112) — [Store] feat: introduce per-tenant storage quota in master - Mooncake [#2180](https://github.com/kvcache-ai/Mooncake/pull/2180) — [Store] Add opt-in grouped object routing semantics --- ### Milestone 3: Transfer Engine (TENT) & Networking Focus on declarative data orchestration and pushing hardware limits. Release TENT V1.1 at 26Q2 @alogfans - [ ] (Declarative Orchestration): Implement a plugin-based multi-transport architecture; enable dynamic path selection via hardware auto-discovery and declarative routing policies. - [ ] (QoS-aware Slice Spraying): Introduce cross-process priority-based slice spraying, minimizing the interference between different tasks; aiming to reduce P99 latency in 8x 400 Gbps network. - [ ] (Ease of Maintenance): Decouple heterogeneous hardware support from transport protocols via a unified GPU abstraction layer and modular plugin architecture to streamline cross-platform deployment. Improve the self-healing ability. - [ ] (Transfer Queue): Implement centralized/distributed transfer queue with priority scheduling and backpressure for high-throughput, low-latency data movement under heavy load. - [ ] (NCCL Support): Use NCCL device API as an optional backend **Transfer Engine RFCs:** - Mooncake [#1058](https://github.com/kvcache-ai/Mooncake/issues/1058) — [RoadMap] Mooncake Transfer Engine NEXT - Mooncake [#1850](https://github.com/kvcache-ai/Mooncake/issues/1850) — [RFC] End-to-End Transfer Tracing for Mooncake - Mooncake [#1749](https://github.com/kvcache-ai/Mooncake/issues/1749) — [RFC] Optimizing Mooncake TCP Transport for High-Concurrency Throughput - Mooncake [#1773](https://github.com/kvcache-ai/Mooncake/issues/1773) — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs - Mooncake [#2117](https://github.com/kvcache-ai/Mooncake/issues/2117) — [RFC] Hierarchical Arch for Intra/Inter Data Center Deployment - Mooncake [#2123](https://github.com/kvcache-ai/Mooncake/issues/2123) — [RFC] Device Transport — GPU-Initiated Communication for TENT - Mooncake [#2132](https://github.com/kvcache-ai/Mooncake/issues/2132) — [RFC] TENT Local Transfer Admission Queue - Mooncake [#2132](https://github.com/kvcache-ai/Mooncake/issues/2630) — [RFC] [RFC]: NCCL device API in Mooncake TENT **Transfer Engine Key PRs (recent):** - Mooncake [#2079](https://github.com/kvcache-ai/Mooncake/pull/2079) — [TENT] Add rule-based transport and device selection - Mooncake [#2090](https://github.com/kvcache-ai/Mooncake/pull/2090) — [TENT] Add cross-platform dynamic library adapter with GPU vendor abstraction - Mooncake [#2155](https://github.com/kvcache-ai/Mooncake/pull/2155) — [TE] Improving the RDMA transport failure handling - Mooncake [#2160](https://github.com/kvcache-ai/Mooncake/pull/2160) — [TE] Add explicit progressBatch API --- ### Milestone 4: OS-Level Optimizations Improve throughput and reduce latency through low-level system integrations. - [ ] (Decentralized Metadata): Migrate SSD metadata management from clients to a dedicated Master to resolve bottleneck issues. - [ ] (Multi-Level Resource Pooling): Aggregate VRAM, DRAM, NVMe, and Remote resources into a transparent cache pool using CXL and SHM. **Related RFCs:** - Mooncake [#1123](https://github.com/kvcache-ai/Mooncake/issues/1123) — [RFC] Support hugepage memory allocation via mmap to bypass RDMA memory registration limits *(Closed)* - Mooncake [#2106](https://github.com/kvcache-ai/Mooncake/issues/2106) — [RFC] GPU VRAM (HBM) Pool Support for Mooncake Store --- ### Milestone 5: Routing & Checkpoint Engine Enhance global scheduling and model weight handling. - [ ] (KVCache-Aware Routing): Route Decode requests based on underlying KVCache distribution (History Cache) to maximize hit rates and minimize TTFT. - [ ] (Checkpoint Engine): Strengthen parallel distribution of weights via P2P Store; support native Torch format offload for fast MoE loading. **Conductor & Routing RFCs:** - Mooncake [#977](https://github.com/kvcache-ai/Mooncake/issues/977) — [RFC] Mooncake-Conductor: Design and Implementation of a Global Scheduler - Mooncake [#1767](https://github.com/kvcache-ai/Mooncake/issues/1767) — [RFC] Mooncake Data Plane Model Distribution, Phase 1 - Mooncake [#2189](https://github.com/kvcache-ai/Mooncake/issues/2189) — [Call For Contributions] Mooncake Conductor Roadmap --- ### Milestone 6: Heterogeneous & Cross-Platform Inference (Domestic GPU Support) Break hardware silos for collaborative deployment. Expand support for domestic accelerator platforms. - [ ] (Cross-Architecture Collaboration): Enable tasks to flow across different vendors (e.g., Vendor A for high-throughput Prefill, Vendor B for low-latency Decode). - [ ] (Ascend 910B / Atlas 800T): Adapt Transfer Engine and Store for Huawei Ascend NPU, including Ascend Direct protocol and HCCL integration. - [ ] (Kunpeng SuperNode UB Transport): Enable UB transport via Kunpeng UMDK URMA APIs for Kunpeng 950 platforms. - [ ] (Moore Threads MACA/MUSA): Align MACA GPU paths with Mooncake build system; integrate MUSA AI Tensor Engine (MATE) for FA3 attention backend. - [ ] (MetaX / Yunsilicon): Resolve RDMA compatibility with Yunsilicon XScale NIC and ensure stable deployment. - [ ] (AMD ROCm / HIP): Add ROCm HIP support to the Mooncake Python package for AMD GPU transport. - [ ] (AWS EFA): Enhance EFA transport with fi_read, endpoint LRU eviction, and multi-NIC striping. **Related RFCs:** - Mooncake [#1773](https://github.com/kvcache-ai/Mooncake/issues/1773) — [RFC] Enable UB transport via Kunpeng UMDK URMA APIs - Mooncake [#1058](https://github.com/kvcache-ai/Mooncake/issues/1058) — [RoadMap] TENT — Full Heterogeneous Deployment section (Ascend ↔ NVIDIA) - Mooncake [#1035](https://github.com/kvcache-ai/Mooncake/issues/1035) — [RoadMap] Store V3 — Hardware mentions: AMD, Moore Threads, AWS EFA, Ascend - Mooncake [#1975](https://github.com/kvcache-ai/Mooncake/issues/1975) — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design - Mooncake [#1965](https://github.com/kvcache-ai/Mooncake/issues/1965) — [Feature Request] MACA Support for Mooncake EP/PG **Tracked issues (Ascend):** - Mooncake [#1854](https://github.com/kvcache-ai/Mooncake/issues/1854) — [Ascend 910B] rtsHostRegister failed in mooncake-store - Mooncake [#1107](https://github.com/kvcache-ai/Mooncake/issues/1107) — Mooncake RDMA error on Ascend 910B - Mooncake [#1766](https://github.com/kvcache-ai/Mooncake/issues/1766) — Failed to modify QP to RTR with Yunsilicon XScale (MetaX) **Heterogeneous Key PRs (recent):** - Mooncake [#2118](https://github.com/kvcache-ai/Mooncake/pull/2118) — feat: add Hygon DCU/DTK and Iluvatar CoreX platform support *(Merged)* - Mooncake [#2196](https://github.com/kvcache-ai/Mooncake/pull/2196) — [Store] enables the Ubtransport for Mooncake Store (Kunpeng) - Mooncake [#2113](https://github.com/kvcache-ai/Mooncake/pull/2113) — [EFA] Add MC_EFA_CQ_THREADS env var and reduce idle CPU spin --- ### Milestone 7: Unified Resource Management & Multi-Model Serving Manage KVCache and model weights cohesively on GPUs. - [ ] (Tiered Weight Storage): Manage weights across VRAM (hot), DRAM (warm), and SSD (cold). - [ ] (Fine-Grained GPU Sharing): Implement CUDA MPS + custom memory isolation for multi-model sharing. - [ ] (Dynamic Weight Swapping): Use Mooncake's TE for fast RDMA-based weight migration. - [ ] (Unified View): Manage both KVCache and weight pools globally. https://github.com/kvcache-ai/Mooncake/issues/2282 --- ### Milestone 8: Fast Instance Startup Utilize data transfer capabilities to accelerate initialization. - [ ] (Fast Model Distribution): Pull weights via RDMA from neighboring nodes instead of remote storage. - [ ] (CUDA Context Restore): Snapshot and restore initialized inference processes to skip warmup. - [ ] (Warm Pool): Maintain a pool of pre-warmed processes for stateful scaling. - [ ] (Lazy Pulling): Optimize container images to only pull needed layers. **Related RFCs:** - Mooncake [#1767](https://github.com/kvcache-ai/Mooncake/issues/1767) — [RFC] Mooncake Data Plane Model Distribution, Phase 1 --- ### Milestone 9: Elasticity & Self-Healing Leverage decoupled architecture for robust operations. - [ ] (LLM-Aware Autoscaler): Scale based on global KVCache pressure and Prefix Tree hit rates. - [ ] (Fast GPU Recovery): Recover failed nodes in <10s by loading weights, utilizing the intact KVCache pool. - [ ] (sched_ext BPF Scheduler): Ensure lowest-latency CPU scheduling for GPU feeder threads. - [ ] (TE Fault Tolerance): Automatic fallback to TCP and re-routing on RDMA/node failures: Already supported by TENT. **Related RFCs:** - Mooncake [#975](https://github.com/kvcache-ai/Mooncake/issues/975) — [RFC] Cleanup processing replicas if the transferring client is down *(Closed)* - Mooncake [#1920](https://github.com/kvcache-ai/Mooncake/issues/1920) — [RFC] Rolling Upgrade Support for Mooncake Store --- ### Milestone 10: Agent & Multi-Modal Support Build the foundation for complex Agent and Multi-Modal workflows. - [ ] (Agent State Cloning): Fast A2A state sharing and cross-node data transport. - [ ] (Agent PD Disaggregation): Separate resources for "thinking" vs. "interactive" agents. - [ ] (EPD Disaggregation): Separate Vision Encoder, Prefill, and Decode for multi-modal scenarios. (Qwen-series supported in sglang) - [ ] (Diffusion Serving): Explore disaggregated architectures for Diffusion models. #### Omni Collaboration - [ ] (Omni Transfer Engine Integration): Integrate Mooncake Transfer Engine as a connector for inter-stage data transfer in the omni multi-modal pipeline (AR → Generation → Diffusion). - [ ] (Worker-Level Inter-Stage Transfer): Optimize worker-level data movement between omni pipeline stages via high-speed RDMA or shared memory. - [ ] (Omni Prefix Caching): Support Hidden State Prefix Caching for multi-modal omni models. **Related RFC** - https://github.com/vllm-project/vllm-omni/issues/2904 - https://github.com/vllm-project/vllm-omni/issues/3635 - Mooncake [#2098](https://github.com/kvcache-ai/Mooncake/issues/2098) — [RFC] Agent-Aware KV Cache Support in Mooncake (Phase 1) #### Diffusion Collaboration - [ ] (Disaggregated Diffusion Deployment): Support decentralized disaggregated deployment for video generation models, decoupling transformer and decoder stages. - [ ] (Queue-based Orchestration): Implement queue-based scheduling for multi-stage diffusion serving requests. --- ### Milestone 11: KVCache Dynamic Migration - [ ] (Live Migration): Support lossless, dynamic hot-migration of massive KVCache states across clients/nodes for Spot Instances and elastic scaling. - [ ] (Migration on Store Instance Exit): Support migrating hot data to other nodes during scale-down or rolling upgrades (optional, enabled manually). - [ ] (Load Balancing) **Related RFCs:** - Mooncake [#1159](https://github.com/kvcache-ai/Mooncake/issues/1159) — [RFC] Replica copy and move support **Migration Key PRs (recent):** - Mooncake [#2170](https://github.com/kvcache-ai/Mooncake/pull/2170) — [Store] add copy/move execution api support in master side *(Merged)* --- ### Milestone 12: Hardware Support & ICMS - [ ] (New Hardware): Support Supernodes, Rubin, PPU, Matrix384. - [ ] (ICMS): Follow BF4 developments and integrate related capabilities. --- ### Milestone 13: SSD Offload Support - [x] (SSD offload): Support offloading KV cache to SSD, ensuring higher hit ratio in limited DRAM scenarios. - [x] (Bandwidth optimization): Optimized bandwidth of SSD loading using io_uring lib. - [ ] (L2→L1 data promotion): If no memory replica and the object is used, promote it to memory. - [ ] (Smart eviction strategy): e.g., Selectively evict cold data with LRU-like strategy. **Related RFCs:** - Mooncake [#971](https://github.com/kvcache-ai/Mooncake/issues/971) — [RFC] Support distributed read and write operations to SSDs - Mooncake [#952](https://github.com/kvcache-ai/Mooncake/issues/952) — [RFC] Support Mooncake file eviction in DFS/3FS *(Closed)* - Mooncake [#1940](https://github.com/kvcache-ai/Mooncake/issues/1940) — [RFC] Implement SSD Pool over NVMe-oF in Mooncake Store - Mooncake [#1957](https://github.com/kvcache-ai/Mooncake/issues/1957) — [RFC] Mooncake Store NVMe KV Backend - Mooncake [#1990](https://github.com/kvcache-ai/Mooncake/issues/1990) — [RFC] Native SSD KV Storage Backend for Mooncake Store - Mooncake [#2025](https://github.com/kvcache-ai/Mooncake/issues/2025) — [RFC] L2→L1 data promotion: If no memory replica and the object is used, promote it to memory - Mooncake [#2031](https://github.com/kvcache-ai/Mooncake/issues/2031) — [RFC] Promote data from L2 -> L1 when it's not in L1 **SSD Offload Key PRs (recent):** - Mooncake [#2084](https://github.com/kvcache-ai/Mooncake/pull/2084) — feat(nof): add NVMe-oF SSD cache support - Mooncake [#2167](https://github.com/kvcache-ai/Mooncake/pull/2167) — [Store] introduce NVMe KV backend core - Mooncake [#2172](https://github.com/kvcache-ai/Mooncake/pull/2172) — feat(store): add SPDK NoF worker pool - Mooncake [#2176](https://github.com/kvcache-ai/Mooncake/pull/2176) — [Store] L2->L1 promotion-on-hit: observability metrics + max_per_heartbeat knob --- ### Milestone 14: Mooncake PG - [ ] **(Transfer Engine): Integrate TENT and refactor transfer engine management.** - [ ] **(Fault-Tolerance Backend): Improve FT-backend fault-handling and recovery, explore elastic backend design.** - [ ] (PyTorch c10d API): Lazy initialization & Split group for faster initialization. - [ ] (Collective Communication): Support ring- and tree-topology algorithm. - [ ] (Hardware Adaption): Support Ascend, Moore Threads, and AWS EFA. - [ ] (Reliability): (1) Establish comprehensive unit-testing and benchmark suites (2) Agentic Specification for robust AI-Coding generation. **Mooncake PG Key PRs & Issues (recent):** - Mooncake [#2157](https://github.com/kvcache-ai/Mooncake/issues/2157) — [Feature Request] Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan - Mooncake [#2149](https://github.com/kvcache-ai/Mooncake/pull/2149) — [PG] support external TE *(Merged)* - Mooncake [#2182](https://github.com/kvcache-ai/Mooncake/pull/2182) — [PG] Add rank fault tolerance and degraded recovery --- ### Milestone 15: Mooncake EP - [ ] **(High Throughput Kernel): Implement high throughput kernel for high batch size inference.** - [ ] (Elastic EP): Support elastically EP extension. - [ ] (TMA Engine): TMA engine optimization for Hopper-specific architecture. - [ ] (Benchmark): Comprehensive MoE kernel benchmark under incast traffic. **Mooncake EP RFCs:** - Mooncake [#1975](https://github.com/kvcache-ai/Mooncake/issues/1975) — [RFC] Mooncake EP Multi-Vendor GPU Adaptation Design — ### Milestone 16: Documentation - [ ] (Update and Refactor Docs): The documentation has become extensive; restructure it and update outdated content - [ ] (AI-Friendly Knowledge Base): Organize Mooncake’s knowledge into a format that is easy for AI systems to read and consume ------------ To contribute features or provide feedback, please edit this issue directly (if you have write access) or comment below. I will update the roadmap accordingly.

bsakaguchi commented

2026-10-29 21:16:05 +00:00

Author

Owner

@Colors-111 commented on GitHub (Apr 15, 2026):

Hi, regarding the roadmap item:
(Agentic-aware): Support agentic-aware KVCache System in inference frameworks.
Could you share more details on what this specifically includes? For example:

Is there a design doc or RFC for the API (e.g., session management, state cloning)?
What integration points are planned for vLLM/SGLang?
Is this related to the "Agent State Cloning" mentioned in Milestone 10?

Thanks!

@Colors-111 commented on GitHub (Apr 15, 2026): Hi, regarding the roadmap item: `(Agentic-aware): Support agentic-aware KVCache System in inference frameworks.` Could you share more details on what this specifically includes? For example: - Is there a design doc or RFC for the API (e.g., session management, state cloning)? - What integration points are planned for vLLM/SGLang? - Is this related to the "Agent State Cloning" mentioned in Milestone 10? Thanks!

bsakaguchi commented

2026-10-29 21:16:05 +00:00

Author

Owner

@llc-kc commented on GitHub (Apr 17, 2026):

does the "(Agentic-aware): Support agentic-aware KVCache System in inference frameworks" refer to [Roadmap]: SGLang Distributed KVCache System For Agentic Workload （https://github.com/sgl-project/sglang/issues/21846） ?

@llc-kc commented on GitHub (Apr 17, 2026): does the "(Agentic-aware): Support agentic-aware KVCache System in inference frameworks" refer to [Roadmap]: SGLang Distributed KVCache System For Agentic Workload （https://github.com/sgl-project/sglang/issues/21846） ?

bsakaguchi commented

2026-10-29 21:16:05 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 17, 2026):

does the "(Agentic-aware): Support agentic-aware KVCache System in inference frameworks" refer to [Roadmap]: SGLang Distributed KVCache System For Agentic Workload （https://github.com/sgl-project/sglang/issues/21846） ?

Yes, the "Agentic-aware KVCache" roadmap item is closely related to the SGLang distributed KVCache initiative you linked. That effort is one of the primary integration targets we have in mind.

@stmatengss commented on GitHub (Apr 17, 2026): > does the "(Agentic-aware): Support agentic-aware KVCache System in inference frameworks" refer to [Roadmap]: SGLang Distributed KVCache System For Agentic Workload （[https://github.com/sgl-project/sglang/issues/21846）](https://github.com/sgl-project/sglang/issues/21846%EF%BC%89) ? Yes, the "Agentic-aware KVCache" roadmap item is closely related to the SGLang distributed KVCache initiative you linked. That effort is one of the primary integration targets we have in mind.

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 17, 2026):

Hi, regarding the roadmap item: (Agentic-aware): Support agentic-aware KVCache System in inference frameworks. Could you share more details on what this specifically includes? For example:

Is there a design doc or RFC for the API (e.g., session management, state cloning)?

What integration points are planned for vLLM/SGLang?

Is this related to the "Agent State Cloning" mentioned in Milestone 10?

Thanks!

Design doc / RFC: There is no public RFC yet. The feature is still in early planning. We plan to open a GitHub issue for design discussion before implementation begins.
API scope: The current thinking includes session-level KVCache management (e.g., associating KV blocks with an agent session ID), and potentially state cloning to support branching agent trajectories without redundant recomputation.
Integration points: vLLM and SGLang are both planned targets. The integration will likely go through the existing KV
Connector interface.
Milestone 10 / Agent State Cloning: Yes, this is related. State cloning is one of the key primitives needed to efficiently support tree-structured agent execution (e.g., multi-branch tool-use or beam search over agent steps).

We'll update this thread when a design RFC is opened. Feel free to share use cases or requirements you have — community input will help shape the design. If you have interests, join the Slack channel and let us discuss the best details!

@stmatengss commented on GitHub (Apr 17, 2026): > Hi, regarding the roadmap item: `(Agentic-aware): Support agentic-aware KVCache System in inference frameworks.` Could you share more details on what this specifically includes? For example: > > * Is there a design doc or RFC for the API (e.g., session management, state cloning)? > * What integration points are planned for vLLM/SGLang? > * Is this related to the "Agent State Cloning" mentioned in Milestone 10? > > Thanks! - Design doc / RFC: There is no public RFC yet. The feature is still in early planning. We plan to open a GitHub issue for design discussion before implementation begins. - API scope: The current thinking includes session-level KVCache management (e.g., associating KV blocks with an agent session ID), and potentially state cloning to support branching agent trajectories without redundant recomputation. - Integration points: vLLM and SGLang are both planned targets. The integration will likely go through the existing KV Connector interface. - Milestone 10 / Agent State Cloning: Yes, this is related. State cloning is one of the key primitives needed to efficiently support tree-structured agent execution (e.g., multi-branch tool-use or beam search over agent steps). We'll update this thread when a design RFC is opened. Feel free to share use cases or requirements you have — community input will help shape the design. If you have interests, join the Slack channel and let us discuss the best details!

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@stmatengss commented on GitHub (May 25, 2026):

📋 Roadmap Tracker Update (2026-04-05 → 2026-05-25)

Linked new RFCs and key implementation PRs from the past 50 days to the corresponding milestones above.

New RFCs Added

#	Title	Milestone
#1920	Rolling Upgrade Support for Mooncake Store	M2, M9
#1940	Implement SSD Pool over NVMe-oF in Mooncake Store	M13
#1957	Mooncake Store NVMe KV Backend	M2, M13
#1975	Mooncake EP Multi-Vendor GPU Adaptation Design	M6, M15
#1979	Dynamo KVBM + Mooncake Store Integration Design	M2
#1990	Native SSD KV Storage Backend for Mooncake Store	M13
#1998	Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction	M2
#2025	L2→L1 data promotion	M13
#2031	Promote data from L2 -> L1 when it's not in L1	M13
#2081	Simplify Mooncake Store for Maintainability	M2
#2098	Agent-Aware KV Cache Support in Mooncake (Phase 1)	M10
#2106	GPU VRAM (HBM) Pool Support for Mooncake Store	M4
#2117	Hierarchical Arch for Intra/Inter Data Center Deployment	M3
#2123	Device Transport — GPU-Initiated Communication for TENT	M3
#2127	Lightweight Best-Effort Group Semantics for Mooncake Store Objects	M2
#2132	TENT Local Transfer Admission Queue	M3
#2153	Dynamic Tenant Quota and Scoped-Key Isolation	M2

New Roadmap/Feature Issues

#	Title	Milestone
#2189	[Call For Contributions] Mooncake Conductor Roadmap	M5
#2157	Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan	M14
#1965	MACA Support for Mooncake EP/PG	M6, M15

Key Implementation PRs (recent 50 days)

Store V3 (M2):

#2080 — DSA-like workload allocation strategy ✅
#2112 — Per-tenant storage quota in master
#2180 — Opt-in grouped object routing semantics

Transfer Engine (M3):

#2079 — Rule-based transport and device selection
#2090 — Cross-platform dynamic library adapter with GPU vendor abstraction
#2155 — Improving RDMA transport failure handling
#2160 — Explicit progressBatch API

Heterogeneous (M6):

#2118 — Hygon DCU/DTK and Iluvatar CoreX platform support ✅
#2196 — UB Transport for Mooncake Store (Kunpeng)
#2113 — EFA CQ_THREADS env var and reduced idle CPU spin

KVCache Migration (M11):

#2170 — Copy/move execution API support in master side ✅

SSD Offload (M13):

#2084 — NVMe-oF SSD cache support
#2167 — NVMe KV backend core
#2172 — SPDK NoF worker pool
#2176 — L2→L1 promotion-on-hit with observability metrics

Mooncake PG (M14):

#2149 — PG support external TE ✅
#2182 — PG rank fault tolerance and degraded recovery

@stmatengss commented on GitHub (May 25, 2026): ## 📋 Roadmap Tracker Update (2026-04-05 → 2026-05-25) Linked new RFCs and key implementation PRs from the past 50 days to the corresponding milestones above. ### New RFCs Added | # | Title | Milestone | |---|-------|-----------| | #1920 | Rolling Upgrade Support for Mooncake Store | M2, M9 | | #1940 | Implement SSD Pool over NVMe-oF in Mooncake Store | M13 | | #1957 | Mooncake Store NVMe KV Backend | M2, M13 | | #1975 | Mooncake EP Multi-Vendor GPU Adaptation Design | M6, M15 | | #1979 | Dynamo KVBM + Mooncake Store Integration Design | M2 | | #1990 | Native SSD KV Storage Backend for Mooncake Store | M13 | | #1998 | Radix Tree Index — Prefix-Aware KV Cache Management and Cascading Eviction | M2 | | #2025 | L2→L1 data promotion | M13 | | #2031 | Promote data from L2 -> L1 when it's not in L1 | M13 | | #2081 | Simplify Mooncake Store for Maintainability | M2 | | #2098 | Agent-Aware KV Cache Support in Mooncake (Phase 1) | M10 | | #2106 | GPU VRAM (HBM) Pool Support for Mooncake Store | M4 | | #2117 | Hierarchical Arch for Intra/Inter Data Center Deployment | M3 | | #2123 | Device Transport — GPU-Initiated Communication for TENT | M3 | | #2127 | Lightweight Best-Effort Group Semantics for Mooncake Store Objects | M2 | | #2132 | TENT Local Transfer Admission Queue | M3 | | #2153 | Dynamic Tenant Quota and Scoped-Key Isolation | M2 | ### New Roadmap/Feature Issues | # | Title | Milestone | |---|-------|-----------| | #2189 | [Call For Contributions] Mooncake Conductor Roadmap | M5 | | #2157 | Mooncake PG Fault-Tolerance: Diagnosis & Improvement Plan | M14 | | #1965 | MACA Support for Mooncake EP/PG | M6, M15 | ### Key Implementation PRs (recent 50 days) **Store V3 (M2):** - #2080 — DSA-like workload allocation strategy ✅ - #2112 — Per-tenant storage quota in master - #2180 — Opt-in grouped object routing semantics **Transfer Engine (M3):** - #2079 — Rule-based transport and device selection - #2090 — Cross-platform dynamic library adapter with GPU vendor abstraction - #2155 — Improving RDMA transport failure handling - #2160 — Explicit progressBatch API **Heterogeneous (M6):** - #2118 — Hygon DCU/DTK and Iluvatar CoreX platform support ✅ - #2196 — UB Transport for Mooncake Store (Kunpeng) - #2113 — EFA CQ_THREADS env var and reduced idle CPU spin **KVCache Migration (M11):** - #2170 — Copy/move execution API support in master side ✅ **SSD Offload (M13):** - #2084 — NVMe-oF SSD cache support - #2167 — NVMe KV backend core - #2172 — SPDK NoF worker pool - #2176 — L2→L1 promotion-on-hit with observability metrics **Mooncake PG (M14):** - #2149 — PG support external TE ✅ - #2182 — PG rank fault tolerance and degraded recovery

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@xianzhiT commented on GitHub (May 29, 2026):

@stmatengss Hi! We are very interested in Milestone 7. Internally, we have been working on a distributed cache pool for model weights, which aligns well with unified management of KVCache and model weights. We would be happy to participate in the design and implementation, and share our experience in distributed weight caching and multi-model serving.

@xianzhiT commented on GitHub (May 29, 2026): @stmatengss Hi! We are very interested in Milestone 7. Internally, we have been working on a distributed cache pool for model weights, which aligns well with unified management of KVCache and model weights. We would be happy to participate in the design and implementation, and share our experience in distributed weight caching and multi-model serving.

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@stmatengss commented on GitHub (Jun 5, 2026):

@stmatengss Hi! We are very interested in Milestone 7. Internally, we have been working on a distributed cache pool for model weights, which aligns well with unified management of KVCache and model weights. We would be happy to participate in the design and implementation, and share our experience in distributed weight caching and multi-model serving.

@xianzhiT Hi! Thanks for your interest and willingness to contribute!

Unified management of KVCache and model weights (Milestone 7) is a crucial part of our roadmap, and your experience in distributed weight caching and multi-model serving sounds like a perfect fit.

Since the design for Milestone 7 is still in its early stages, we would love to collaborate with you. To kick things off, you can join our discussion through the following ways:

Open a dedicated RFC/Issue: Feel free to open a new Issue or RFC in this repo to share your current architecture, insights, or a preliminary proposal for the unified management design.
Slack Channel: You are also very welcome to join our Slack channel for more real-time and deep technical discussions.

Looking forward to building this together!

@stmatengss commented on GitHub (Jun 5, 2026): > [@stmatengss](https://github.com/stmatengss) Hi! We are very interested in Milestone 7. Internally, we have been working on a distributed cache pool for model weights, which aligns well with unified management of KVCache and model weights. We would be happy to participate in the design and implementation, and share our experience in distributed weight caching and multi-model serving. @xianzhiT Hi! Thanks for your interest and willingness to contribute! Unified management of KVCache and model weights (Milestone 7) is a crucial part of our roadmap, and your experience in distributed weight caching and multi-model serving sounds like a perfect fit. Since the design for Milestone 7 is still in its early stages, we would love to collaborate with you. To kick things off, you can join our discussion through the following ways: 1. **Open a dedicated RFC/Issue:** Feel free to open a new Issue or RFC in this repo to share your current architecture, insights, or a preliminary proposal for the unified management design. 2. **Slack Channel:** You are also very welcome to join our Slack channel for more real-time and deep technical discussions. Looking forward to building this together!

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@CAICAIIs commented on GitHub (Jun 5, 2026):

Hi~ @stmatengss I’d like to work on the Milestone 10 Agent-Aware KV Cache slice, starting from the concrete Phase 1 RFC in #2098.

I have posted my proposed implementation plan in #2098

Could you please confirm whether this is an acceptable starting scope, and whether I can begin the Mooncake-side Phase 1 implementation?

@CAICAIIs commented on GitHub (Jun 5, 2026): Hi~ @stmatengss I’d like to work on the Milestone 10 Agent-Aware KV Cache slice, starting from the concrete Phase 1 RFC in #2098. I have posted my proposed implementation plan in #2098 Could you please confirm whether this is an acceptable starting scope, and whether I can begin the Mooncake-side Phase 1 implementation?

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@stmatengss commented on GitHub (Jun 7, 2026):

Hi~ @stmatengss I’d like to work on the Milestone 10 Agent-Aware KV Cache slice, starting from the concrete Phase 1 RFC in #2098.

I have posted my proposed implementation plan in #2098

Could you please confirm whether this is an acceptable starting scope, and whether I can begin the Mooncake-side Phase 1 implementation?

Sure. Feel free to ping me.

@stmatengss commented on GitHub (Jun 7, 2026): > Hi~ [@stmatengss](https://github.com/stmatengss) I’d like to work on the Milestone 10 Agent-Aware KV Cache slice, starting from the concrete Phase 1 RFC in [#2098](https://github.com/kvcache-ai/Mooncake/issues/2098). > > I have posted my proposed implementation plan in [#2098](https://github.com/kvcache-ai/Mooncake/issues/2098) > > Could you please confirm whether this is an acceptable starting scope, and whether I can begin the Mooncake-side Phase 1 implementation? Sure. Feel free to ping me.

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@Srinivasoo7 commented on GitHub (Jun 17, 2026):

@stmatengss
Can you help me with sharing the slack channel we coordinate in?
It'd be really helpful.

Thanks!

@Srinivasoo7 commented on GitHub (Jun 17, 2026): @stmatengss Can you help me with sharing the slack channel we coordinate in? It'd be really helpful. Thanks!

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@stmatengss commented on GitHub (Jun 20, 2026):

@stmatengss
Can you help me with sharing the slack channel we coordinate in?
It'd be really helpful.

Thanks!

https://join.slack.com/t/mooncake-project/shared_invite/zt-3ig4fjai8-KH1zIm3x8Vm8WqyH0i_JaA

You can also find it in homepage

@stmatengss commented on GitHub (Jun 20, 2026): > @stmatengss > Can you help me with sharing the slack channel we coordinate in? > It'd be really helpful. > > > Thanks! > https://join.slack.com/t/mooncake-project/shared_invite/zt-3ig4fjai8-KH1zIm3x8Vm8WqyH0i_JaA You can also find it in homepage

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@stmatengss commented on GitHub (Jun 27, 2026):

📋 Roadmap Tracker Update (2026-05-25 → 2026-06-27)

Linked new RFCs and key implementation PRs from the past 33 days to the corresponding milestones above.

New RFCs Added

RFC	Title	Milestone
#2633	[RFC]: Host-aware Local-first Allocation for Mooncake Store	M2: Store V3
#2639	[RFC]: Deprecate and remove the test-only initAll / init_all initialization API	M2: Store V3
#2582	[RFC]: Mooncake Store AcceleratorDevice Abstraction	M2: Store V3
#2581	[RFC]: Mooncake Store Hardware Abstraction	M2: Store V3
#2509	[RFC]: L2→L1 proactive promotion via bounded master-side scan V2	M2: Store V3
#2426	[RFC]: Refactor and Improve DistributedStorageBackend (and 3FS Adapter)	M2: Store V3
#2306	[RFC]: Master-Side LOCAL_DISK Replica Warm Re-Adoption Across Client Restart	M2: Store V3
#2233	[RFC]: L2→L1 promotion background retry for hot LOCAL_DISK-only keys V1.1	M2: Store V3
#2223	[Store] RFC: CROSS_PROCESS_MEMCPY strategy for same-host different-process transfers	M2: Store V3
#2213	[RFC]: Prefetch SSD-Only Objects to DRAM on Exist	M2: Store V3
#2630	[RFC]: NCCL device API in Mooncake TENT	M3: TENT
#2572	[RFC]: Streaming Transfer for Mooncake	M3: TENT
#2568	[RFC]: Traffic class hint API for TransferEngine	M3: TENT
#2519	[RFC]: Deadline-aware admission & adaptive degradation for TENT transfers	M3: TENT
#2479	[RFC]: Zero-Copy Read/Write: Eliminate Redundant Data Copies	M3: TENT
#2516	[Feature Request]: Topology- and load-aware remote replica selection in SelectBestReplica	M3: TENT / M5: Routing
#2444	[Feature Request]: 集成绿算技术 GP 系列 NVMe-oF 存储后端	M4: OS-Level
#2608	[RFC]: RL Disaggregation via Rollout / Update Separation and Shared Mooncake Data Plane	M1: Inference & RL
#2607	[RFC]: Comprehensive MoE / EP Kernel Benchmark Under Incast Traffic	M3: TENT / M6: Heterogeneous
#2518	[RFC]: AscendRdmaTransport — NPU↔RDMA Protocol Bridge (Closed)	M6: Heterogeneous
#2454	[RFC]: Coordinator-Based Control Plane for Mooncake PG	M6: Heterogeneous
#2282	[RFC]: Unified KVCache and Model Weight Management in Mooncake Store	M7: Unified Resource
#2609	[RFC]: AI-Friendly Knowledge Base for Mooncake	General
#2400	[RFC]: Refactor MasterAdminServer out of rpc_service (Closed)	M2: Store V3

Key Implementation PRs (recent, merged since 2026-05-25)

Store V3 — Multi-Tenant Quota (full stack)

[Store] feat: add tenant quota core #2358
[Store] dynamic tenant quota master admission #2492
[Store] Complete dynamic tenant quota admin, eviction, metrics, and snapshots #2549

Store V3 — Structured Object & Tensor Batch

[Store] add remote tensor batch interfaces #2050
[Store] add structured object flat mvp #2600
[Store] add structured object copy-mode policy #2472
[Store] add codec inference and recursive structure expansion #2521

Store V3 — P2P HA & OpLog

[Store] Port OpLog storage layer from main for P2P HA #2485
[Store] Port OpLogApplier and OpLogReplicator from main #2496
[Store] Add P2P OpLog types and payload serialization #2512
[Store] recover etcd client after leader hang (SIGSTOP) #2383
[Store] Make etcd master-view watch event-driven instead of polling #2484

Store V3 — Core Improvements

Store Extract 3FS logic into DistributedStorageBackend #2234
[Store] Introduce buffer pool for zero copy interfaces #2095
[Store] Event-Driven Tiered-Cache Scheduler with Pluggable Policy #2550
[Store] Add SSD free-ratio-first allocation strategy #2450
[Store] Reduce lookup eviction contention #2405
[Store] Introduce cached batch query result #1834
[Store] Add runtime read/write config with HTTP API for dynamic tuning #2360
[Store][Refactor]: Refactor master admin service #2422
[Store][K8s-Native][3/N] Label-based routing #2537
[Store] master log journal #2297
[Store] Implement dual rdma forward path #2092

TENT & Networking — Admission, QoS, Transport

[TENT] Add local admission queue prototype #2341
[TENT] add local runtime queue dispatch #2562
[TENT] Add optional Request.deadline_ns and MLU observability metric (RFC #2519 step 1) #2618
[TENT] Reduce SegmentDesc fetch overhead on cold receive path #2482
[TENT] feat: add per-request transport_hint for fine-grained transport selection #2339
[TENT] Add policy name binding to transport selector #2295
[TransferEngine] Make InfiniBand Service Level configurable via MC_IB_SL #2525
[TransferEngine] Apply configured SL/TC to the TENT notification QP #2526
[TE] add device API support #2333
[TE] Universal TCP Force Mechanism for All Environments #2593
[TE] Enforce one-shot lifecycle for connected RDMA endpoints #2588
[TE] Make TransferEngine movable #2567
Update BatchAsync for MMNVL #2384

OS-Level / NVMe-oF

feat(store): route NoF replicas through put and get #2247
feat(store): add NoF SSD deployment tools and e2e coverage #2273

Heterogeneous & Cross-Platform (EP / PG / Sunrise / ROCm / Ascend)

[EP] support MUSA platform #2480
[EP] integrate Device API (P2pTransport/RdmaTransport) — CUDA only #2382
[EP] Add MACA standalone device EP support #2309
[EP] Add MACA portable EP support with CUDA intrinsic fallbacks #2292
[PG] Add experimental MACA PG support #2365
[PG] Add MUSA build support #2329
[MUSA] Add release-musa github workflow #2576
[TE][Sunrise][Feat] Enable Sunrise support in the classic transfer engine path #2290
[TE][Sunrise][Feat] Enable Sunrise VRAM support in tebench for the TENT backend #2452
[TransferEngine][ROCm] Add HIP dmabuf MR registration for AMD GPUs #2225
[TE] Support per-role Ascend protocol for co-located Transfer Engines #2499
[PG] Fix elastic P2P after max_world_size recovery #2623

@stmatengss commented on GitHub (Jun 27, 2026): ## 📋 Roadmap Tracker Update (2026-05-25 → 2026-06-27) Linked new RFCs and key implementation PRs from the past 33 days to the corresponding milestones above. ### New RFCs Added | RFC | Title | Milestone | |-----|-------|-----------| | #2633 | [RFC]: Host-aware Local-first Allocation for Mooncake Store | M2: Store V3 | | #2639 | [RFC]: Deprecate and remove the test-only initAll / init_all initialization API | M2: Store V3 | | #2582 | [RFC]: Mooncake Store AcceleratorDevice Abstraction | M2: Store V3 | | #2581 | [RFC]: Mooncake Store Hardware Abstraction | M2: Store V3 | | #2509 | [RFC]: L2→L1 proactive promotion via bounded master-side scan V2 | M2: Store V3 | | #2426 | [RFC]: Refactor and Improve DistributedStorageBackend (and 3FS Adapter) | M2: Store V3 | | #2306 | [RFC]: Master-Side LOCAL_DISK Replica Warm Re-Adoption Across Client Restart | M2: Store V3 | | #2233 | [RFC]: L2→L1 promotion background retry for hot LOCAL_DISK-only keys V1.1 | M2: Store V3 | | #2223 | [Store] RFC: CROSS_PROCESS_MEMCPY strategy for same-host different-process transfers | M2: Store V3 | | #2213 | [RFC]: Prefetch SSD-Only Objects to DRAM on Exist | M2: Store V3 | | #2630 | [RFC]: NCCL device API in Mooncake TENT | M3: TENT | | #2572 | [RFC]: Streaming Transfer for Mooncake | M3: TENT | | #2568 | [RFC]: Traffic class hint API for TransferEngine | M3: TENT | | #2519 | [RFC]: Deadline-aware admission & adaptive degradation for TENT transfers | M3: TENT | | #2479 | [RFC]: Zero-Copy Read/Write: Eliminate Redundant Data Copies | M3: TENT | | #2516 | [Feature Request]: Topology- and load-aware remote replica selection in SelectBestReplica | M3: TENT / M5: Routing | | #2444 | [Feature Request]: 集成绿算技术 GP 系列 NVMe-oF 存储后端 | M4: OS-Level | | #2608 | [RFC]: RL Disaggregation via Rollout / Update Separation and Shared Mooncake Data Plane | M1: Inference & RL | | #2607 | [RFC]: Comprehensive MoE / EP Kernel Benchmark Under Incast Traffic | M3: TENT / M6: Heterogeneous | | #2518 | [RFC]: AscendRdmaTransport — NPU↔RDMA Protocol Bridge (Closed) | M6: Heterogeneous | | #2454 | [RFC]: Coordinator-Based Control Plane for Mooncake PG | M6: Heterogeneous | | #2282 | [RFC]: Unified KVCache and Model Weight Management in Mooncake Store | M7: Unified Resource | | #2609 | [RFC]: AI-Friendly Knowledge Base for Mooncake | General | | #2400 | [RFC]: Refactor MasterAdminServer out of rpc_service (Closed) | M2: Store V3 | ### Key Implementation PRs (recent, merged since 2026-05-25) **Store V3 — Multi-Tenant Quota (full stack)** - [Store] feat: add tenant quota core #2358 - [Store] dynamic tenant quota master admission #2492 - [Store] Complete dynamic tenant quota admin, eviction, metrics, and snapshots #2549 **Store V3 — Structured Object & Tensor Batch** - [Store] add remote tensor batch interfaces #2050 - [Store] add structured object flat mvp #2600 - [Store] add structured object copy-mode policy #2472 - [Store] add codec inference and recursive structure expansion #2521 **Store V3 — P2P HA & OpLog** - [Store] Port OpLog storage layer from main for P2P HA #2485 - [Store] Port OpLogApplier and OpLogReplicator from main #2496 - [Store] Add P2P OpLog types and payload serialization #2512 - [Store] recover etcd client after leader hang (SIGSTOP) #2383 - [Store] Make etcd master-view watch event-driven instead of polling #2484 **Store V3 — Core Improvements** - [Store](refactor) Extract 3FS logic into DistributedStorageBackend #2234 - [Store] Introduce buffer pool for zero copy interfaces #2095 - [Store] Event-Driven Tiered-Cache Scheduler with Pluggable Policy #2550 - [Store] Add SSD free-ratio-first allocation strategy #2450 - [Store] Reduce lookup eviction contention #2405 - [Store] Introduce cached batch query result #1834 - [Store] Add runtime read/write config with HTTP API for dynamic tuning #2360 - [Store][Refactor]: Refactor master admin service #2422 - [Store][K8s-Native][3/N] Label-based routing #2537 - [Store] master log journal #2297 - [Store] Implement dual rdma forward path #2092 **TENT & Networking — Admission, QoS, Transport** - [TENT] Add local admission queue prototype #2341 - [TENT] add local runtime queue dispatch #2562 - [TENT] Add optional Request.deadline_ns and MLU observability metric (RFC #2519 step 1) #2618 - [TENT] Reduce SegmentDesc fetch overhead on cold receive path #2482 - [TENT] feat: add per-request transport_hint for fine-grained transport selection #2339 - [TENT] Add policy name binding to transport selector #2295 - [TransferEngine] Make InfiniBand Service Level configurable via MC_IB_SL #2525 - [TransferEngine] Apply configured SL/TC to the TENT notification QP #2526 - [TE] add device API support #2333 - [TE] Universal TCP Force Mechanism for All Environments #2593 - [TE] Enforce one-shot lifecycle for connected RDMA endpoints #2588 - [TE] Make TransferEngine movable #2567 - Update BatchAsync for MMNVL #2384 **OS-Level / NVMe-oF** - feat(store): route NoF replicas through put and get #2247 - feat(store): add NoF SSD deployment tools and e2e coverage #2273 **Heterogeneous & Cross-Platform (EP / PG / Sunrise / ROCm / Ascend)** - [EP] support MUSA platform #2480 - [EP] integrate Device API (P2pTransport/RdmaTransport) — CUDA only #2382 - [EP] Add MACA standalone device EP support #2309 - [EP] Add MACA portable EP support with CUDA intrinsic fallbacks #2292 - [PG] Add experimental MACA PG support #2365 - [PG] Add MUSA build support #2329 - [MUSA] Add release-musa github workflow #2576 - [TE][Sunrise][Feat] Enable Sunrise support in the classic transfer engine path #2290 - [TE][Sunrise][Feat] Enable Sunrise VRAM support in tebench for the TENT backend #2452 - [TransferEngine][ROCm] Add HIP dmabuf MR registration for AMD GPUs #2225 - [TE] Support per-role Ascend protocol for co-located Transfer Engines #2499 - [PG] Fix elastic P2P after max_world_size recovery #2623

bsakaguchi commented

2026-10-29 21:16:08 +00:00

Author

Owner

@catyans commented on GitHub (Jul 11, 2026):

For Milestone 3: Transfer Engine (TENT) & Networking, I opened #2856 as a proposed QoS control-plane foundation. The RFC defines a tenant -> intent -> request QoS Contract schema and effective policy model. It is meant to provide the contract layer for future priority authorization, inflight policing, bandwidth caps, receiver-credit allocation, and weighted scheduling. Initial implementation scope is deliberately small: schema, validation, effective-policy resolution, and explain/debug output. Runtime enforcement would be split into follow-up PRs.

@catyans commented on GitHub (Jul 11, 2026): For Milestone 3: Transfer Engine (TENT) & Networking, I opened #2856 as a proposed QoS control-plane foundation. The RFC defines a `tenant -> intent -> request` QoS Contract schema and effective policy model. It is meant to provide the contract layer for future priority authorization, inflight policing, bandwidth caps, receiver-credit allocation, and weighted scheduling. Initial implementation scope is deliberately small: schema, validation, effective-policy resolution, and explain/debug output. Runtime enforcement would be split into follow-up PRs.

bsakaguchi commented

2026-10-29 21:16:08 +00:00

Author

Owner

@everettxwzhu commented on GitHub (Sep 1, 2026):

Hi @stmatengss, I’m a first-time Mooncake contributor, and I’d like to work on #2223 under Milestone 2: Store V3.

My plan is to carry the owner process identity in buffer metadata, detect same-host cross-process transfers, and add a CROSS_PROCESS_MEMCPY path using process_vm_readv() / process_vm_writev(). I’ll also cover error fallback, multi-process tests, and a small benchmark.

Could this work be tracked under Milestone 2? Also, the Slack invite shared earlier appears to have expired. Could you please share an updated link?

@everettxwzhu commented on GitHub (Sep 1, 2026): Hi @stmatengss, I’m a first-time Mooncake contributor, and I’d like to work on #2223 under Milestone 2: Store V3. My plan is to carry the owner process identity in buffer metadata, detect same-host cross-process transfers, and add a `CROSS_PROCESS_MEMCPY` path using `process_vm_readv()` / `process_vm_writev()`. I’ll also cover error fallback, multi-process tests, and a small benchmark. Could this work be tracked under Milestone 2? Also, the Slack invite shared earlier appears to have expired. Could you please share an updated link?

bsakaguchi referenced this issue

2026-10-29 21:29:04 +00:00

[PR #535] [MERGED] [TransferEngine] fix segfault when create cq failed #1633