kvcache-ai/Mooncake

Fork

You've already forked Mooncake

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #378] [RoadMap] Mooncake Store V2 #132

New issue

Closed

opened 2026-10-29 21:09:53 +00:00 by bsakaguchi · 17 comments

bsakaguchi commented

2026-10-29 21:09:53 +00:00

Owner

Copy link

Originally created by @stmatengss on GitHub (May 19, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/378

This is a roadmap for Mooncake Store V2! If you're interested in contributing to any item, please let us know!

If any items you’re interested in are missing from the roadmap, your suggestions and input are highly encouraged! Please don’t hesitate to comment in this thread, submit a feature request, or establish an RFC.

Mooncake Store V2

LMCache Adaptor
- KVCache Reuse between prefill nodes
- HTTP notification protocols (KVAdmitMsg/KVEvictMsg) with LMCache Controller @xiaguan
- Mooncake-based BufferAllocator in LMCache layer to allocate MemoryObj (zero-copy) https://github.com/LMCache/LMCache/pull/642
SGLang Adaptor https://github.com/sgl-project/sglang/pull/7211
HiCache support https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md
Mooncake Client
- Client-side Buffer to reduce metadata footprint
- Batch Put/Get interfaces for layer-wise. @zhaoyongke @xinranwang17 #380
- Soft-pin mechanism to pin KVCache in the indicated node/layers. #587
- Flexible APIs: removeAll, migrate #355
- Implementing the basic metrics system on the client side. #733 #738
- User-friendly configuration method (using global yaml config or avoid configurations)
Mooncake Master
- Lease designs to avoid conflicts between get/remove #374
- High availability service with extra protocols #587 @ykwd
- KVCache eviction mechanism #374 #287
- 3FS backend for KVCache persistence. #437 #610 #690 @SgtPepperr
Multi-layer Storage
- VRAM storage support (GDR-based zero-copy)
- GPU-based VRAM pool @XucSh #710
- SSD storage support #968
Benchmark and CI/CD
- KVCache workload emulation benchmark
- Stress benchmark for high concurrency
Revise Website Documentation
- Create the Python binding documentation.
- Create the Mooncake Master documentation.
- Create documentation to assist users in integrating with LMCache. @XucSh #385
Deployment
- User-friendly configuration method (using global yaml config or avoiding configurations)
- native K8S helm chart for running store (including P, D and master). https://github.com/sgl-project/rbg/pull/75
- Integrating etcd service into mooncake store
Fault-tolerance and High Availability (HA)
- Master failover: multiple backup instances, elect new leader when old one fails #451
- Master failover: kv metadata persistency #760
- Client failover: tolerant client crash and network partition #501
- Ensure correctness of get and put operations #374 #778 #993
- Move stable HA-features (that do not depend on etcd) to non-HA mode #845

Originally created by @stmatengss on GitHub (May 19, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/378 This is a roadmap for Mooncake Store V2! If you're interested in contributing to any item, please let us know! If any items you’re interested in are missing from the roadmap, your suggestions and input are highly encouraged! Please don’t hesitate to comment in this thread, submit a feature request, or establish an RFC. ------------- ### Mooncake Store V2 1. LMCache Adaptor - [ ] KVCache Reuse between prefill nodes - [x] HTTP notification protocols (`KVAdmitMsg/KVEvictMsg`) with LMCache Controller @xiaguan - [x] Mooncake-based BufferAllocator in LMCache layer to allocate MemoryObj (zero-copy) https://github.com/LMCache/LMCache/pull/642 2. SGLang Adaptor https://github.com/sgl-project/sglang/pull/7211 3. HiCache support https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md 4. Mooncake Client - [ ] Client-side Buffer to reduce metadata footprint - [x] Batch Put/Get interfaces for layer-wise. @zhaoyongke @xinranwang17 #380 - [x] Soft-pin mechanism to pin KVCache in the indicated node/layers. #587 - [x] Flexible APIs: removeAll, migrate #355 - [x] Implementing the basic metrics system on the client side. #733 #738 - [ ] User-friendly configuration method (using global yaml config or avoid configurations) 4. Mooncake Master - [x] Lease designs to avoid conflicts between get/remove #374 - [x] High availability service with extra protocols #587 @ykwd - [x] KVCache eviction mechanism #374 #287 - [x] 3FS backend for KVCache persistence. #437 #610 #690 @SgtPepperr 5. Multi-layer Storage - [ ] VRAM storage support (GDR-based zero-copy) - [x] GPU-based VRAM pool @XucSh #710 - [ ] SSD storage support #968 6. Benchmark and CI/CD - [ ] KVCache workload emulation benchmark - [ ] Stress benchmark for high concurrency 7. Revise Website Documentation - [x] Create the Python binding documentation. - [x] Create the Mooncake Master documentation. - [x] Create documentation to assist users in integrating with LMCache. @XucSh #385 8. Deployment - [ ] User-friendly configuration method (using global yaml config or avoiding configurations) - [x] native K8S helm chart for running store (including P, D and master). https://github.com/sgl-project/rbg/pull/75 - [x] Integrating etcd service into mooncake store 9. Fault-tolerance and High Availability (HA) - [x] Master failover: multiple backup instances, elect new leader when old one fails #451 - [ ] Master failover: kv metadata persistency #760 - [x] Client failover: tolerant client crash and network partition #501 - [x] Ensure correctness of get and put operations #374 #778 #993 - [x] Move stable HA-features (that do not depend on etcd) to non-HA mode #845

bsakaguchi

2026-10-29 21:09:53 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:09:53 +00:00

Author

Owner

Copy link

@zhaoyongke commented on GitHub (May 20, 2025):

Great!
BTW, is cache-aware scheduler in the roadmap?

@zhaoyongke commented on GitHub (May 20, 2025): Great! BTW, is cache-aware scheduler in the roadmap?

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@stmatengss commented on GitHub (May 20, 2025):

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

@stmatengss commented on GitHub (May 20, 2025): > Great! BTW, is cache-aware scheduler in the roadmap? Good idea! Is the Cache-aware scheduler on the master side or another implementation?

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@zhaoyongke commented on GitHub (May 20, 2025):

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later.

@zhaoyongke commented on GitHub (May 20, 2025): > > Great! BTW, is cache-aware scheduler in the roadmap? > > Good idea! Is the Cache-aware scheduler on the master side or another implementation? Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later.

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@zhaoyongke commented on GitHub (May 20, 2025):

Great! BTW, is cache-aware scheduler in the roadmap?

Good idea! Is the Cache-aware scheduler on the master side or another implementation?

Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later.

support cache aware scheduler

@zhaoyongke commented on GitHub (May 20, 2025): > > > Great! BTW, is cache-aware scheduler in the roadmap? > > > > > > Good idea! Is the Cache-aware scheduler on the master side or another implementation? > > Both, mooncake master provides key location information for global scheduler. I'll provide another RFC later. [support cache aware scheduler](https://github.com/kvcache-ai/Mooncake/issues/381)

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@hzh0425 commented on GitHub (May 21, 2025):

KVCache Reuse between prefill nodes

@stmatengss Hi, I wanna try it, thanks~

@hzh0425 commented on GitHub (May 21, 2025): > * KVCache Reuse between prefill nodes @stmatengss Hi, I wanna try it, thanks~

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@stmatengss commented on GitHub (May 21, 2025):

KVCache Reuse between prefill nodes

@stmatengss Hi, I wanna try it, thanks~

Cool! Hope to see your PR soon!

@stmatengss commented on GitHub (May 21, 2025): > > * KVCache Reuse between prefill nodes > > @stmatengss Hi, I wanna try it, thanks~ Cool! Hope to see your PR soon!

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (May 23, 2025):

High availability service with extra protocols
Integrating etcd service into mooncake store

Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR.

@ykwd commented on GitHub (May 23, 2025): > High availability service with extra protocols > Integrating etcd service into mooncake store Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR.

bsakaguchi commented

2026-10-29 21:09:54 +00:00

Author

Owner

Copy link

@zhaoyongke commented on GitHub (Jun 10, 2025):

SGLang Adaptor

We'll take this @huangtingwei9988

@zhaoyongke commented on GitHub (Jun 10, 2025): > 2. SGLang Adaptor We'll take this @huangtingwei9988

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@wwq2333 commented on GitHub (Aug 7, 2025):

High availability service with extra protocols
Integrating etcd service into mooncake store

Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR.

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

@wwq2333 commented on GitHub (Aug 7, 2025): > > High availability service with extra protocols > > Integrating etcd service into mooncake store > > Hi, I wanna try the High availability feature. And as this feature may depend on ETCD server, these two could be solved in the same PR. Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Aug 7, 2025):

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

@ykwd commented on GitHub (Aug 7, 2025): > > Hi [@ykwd](https://github.com/ykwd) [@stmatengss](https://github.com/stmatengss), in the master's high availability design, is there a plan to implement persistence of various meta information? Yes. Supporting KV metadata persistence is on our roadmap.

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@SpecterCipher commented on GitHub (Aug 11, 2025):

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? @ykwd

@SpecterCipher commented on GitHub (Aug 11, 2025): > > Hi [@ykwd](https://github.com/ykwd) [@stmatengss](https://github.com/stmatengss), in the master's high availability design, is there a plan to implement persistence of various meta information? > > Yes. Supporting KV metadata persistence is on our roadmap. Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? @ykwd

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Aug 12, 2025):

Hi @ykwd @stmatengss, in the master's high availability design, is there a plan to implement persistence of various meta information?

Yes. Supporting KV metadata persistence is on our roadmap.

Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? @ykwd

@SpecterCipher There’s no official documentation yet. In the meantime, we’ve had some informal discussions about the design. Using etcd to store KV metadata is not preferred due to performance concerns. Instead, we’re considering having each client store the metadata for the KVs it holds. This would make metadata persistence independent of etcd or any other external service. However, this approach hasn’t been finalized yet.

@ykwd commented on GitHub (Aug 12, 2025): > > > Hi [@ykwd](https://github.com/ykwd) [@stmatengss](https://github.com/stmatengss), in the master's high availability design, is there a plan to implement persistence of various meta information? > > > > > > Yes. Supporting KV metadata persistence is on our roadmap. > > Hi, may I ask whether there has been any discussion or documentation concerning this approach? For example, would the metadata be stored in etcd? [@ykwd](https://github.com/ykwd) @SpecterCipher There’s no official documentation yet. In the meantime, we’ve had some informal discussions about the design. Using etcd to store KV metadata is not preferred due to performance concerns. Instead, we’re considering having each client store the metadata for the KVs it holds. This would make metadata persistence independent of etcd or any other external service. However, this approach hasn’t been finalized yet.

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@Alan-D-Chen commented on GitHub (Oct 14, 2025):

Type A

This is a one-to-one PD disaggregated inference framework. Under this framework, the KVcache generated by GPU-a in the prefill node can only be transmitted to GPU-a in the decode node; in other words, GPU-a in the decode node can only access the memory of GPU-a in the prefill node.

Type B

This is a PD inference framework with a KVcache store. In this framework, the KVcache generated by the GPUs in the prefill node must first be transmitted to the KVcache store, and the framework will prioritize allocating the KVcache from the KVcache store to idle decode GPUs based on the busyness of the GPUs in the decode node.

May I ask which type of inference framework the mooncake belongs to?

In the P4D4 framework structure, the cache of prefill node 0 can only be accessed by decode node 0, rather than by "any arbitrary" decode node that can directly access the cache of prefill node 0, and it is not "shared across the entire cluster". Is my understanding correct?

@Alan-D-Chen commented on GitHub (Oct 14, 2025): <img width="1120" height="758" alt="Image" src="https://github.com/user-attachments/assets/d28af12b-9682-44d8-b155-709fac0bd0d9" /> ### **Type A** This is a one-to-one PD disaggregated inference framework. Under this framework, the KVcache generated by GPU-a in the prefill node can only be transmitted to GPU-a in the decode node; in other words, GPU-a in the decode node can only access the memory of GPU-a in the prefill node. <img width="1587" height="727" alt="Image" src="https://github.com/user-attachments/assets/da648d5a-cb3c-4bde-9f70-158afb0885d6" /> ### **Type B** This is a PD inference framework with a KVcache store. In this framework, the KVcache generated by the GPUs in the prefill node must first be transmitted to the KVcache store, and the framework will prioritize allocating the KVcache from the KVcache store to idle decode GPUs based on the busyness of the GPUs in the decode node. May I ask which type of inference framework the mooncake belongs to? In the P4D4 framework structure, the cache of prefill node 0 can only be accessed by decode node 0, rather than by "any arbitrary" decode node that can directly access the cache of prefill node 0, and it is not "shared across the entire cluster". Is my understanding correct?

bsakaguchi commented

2026-10-29 21:09:55 +00:00

Author

Owner

Copy link

@Alan-D-Chen commented on GitHub (Oct 14, 2025):

I hope my question hasn't caused you too much trouble. We students are fascinated by the wisdom of the mooncake authors and have developed a great interest in it.

@tchaikov @misterwilliam @simpx @karya0

@Alan-D-Chen commented on GitHub (Oct 14, 2025): I hope my question hasn't caused you too much trouble. We students are fascinated by the wisdom of the mooncake authors and have developed a great interest in it. @tchaikov @misterwilliam @simpx @karya0

bsakaguchi commented

2026-10-29 21:09:56 +00:00

Author

Owner

Copy link

@tianlang-wq commented on GitHub (Oct 24, 2025):

@Alan-D-Chen 目前两种方式都支持
sglang +hicache + mooncake 这个方案属于Type A

vllm + lmcache + mooncake 这个方案我们实现出来的是Type B

可以通过 vllm + mooncake_connector 实现Type A 相关iss https://github.com/kvcache-ai/Mooncake/pull/865

我的理解是 P4D4 的场景无论是TypeA 还是TypeB 都不是一一对应的，而是由proxy 来决定 prefill-1 处理完后接下来该有那个decode处理，比如 prefill-1处理后可能会有decode-2处理，说白了这是两个工作池，由proxy实现调度。

@tianlang-wq commented on GitHub (Oct 24, 2025): @Alan-D-Chen 目前两种方式都支持 sglang +hicache + mooncake 这个方案属于Type A vllm + lmcache + mooncake 这个方案我们实现出来的是Type B 可以通过 vllm + mooncake_connector 实现Type A 相关iss https://github.com/kvcache-ai/Mooncake/pull/865 我的理解是 P4D4 的场景无论是TypeA 还是TypeB 都不是一一对应的，而是由proxy 来决定 prefill-1 处理完后接下来该有那个decode处理，比如 prefill-1处理后可能会有decode-2处理，说白了这是两个工作池，由proxy实现调度。

bsakaguchi commented

2026-10-29 21:09:56 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 12, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 12, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:09:56 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 20, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 20, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

No Branch/Tag specified

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels

Mirrored from GitHub Pull Request

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#132

Reference in a new issue

Repository

kvcache-ai/Mooncake

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns