[GH-ISSUE #171] [RFC] Mooncake-Store KVCache SSD Offload #80

New issue

Closed

opened 2026-10-29 21:08:52 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:08:52 +00:00

Owner

Originally created by @yuan-luo on GitHub (Mar 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/171

Motivation.

Hello Mooncake community,
We're from Ant Group team and are actively working on P/D disaggregation inference. We'd like to share our initial proposal for achieving SSD Offload Mooncake-Store, the distributed object store for vLLM XpYd scenarios. With this feature, KVCache store is expected to be more robust, achieve better performance (e.g., long documents) in case of hitting cache in LLM inference.

Proposed Change.

The plan is to implement SSD offload within Mooncake-Store. The entire solution can be categorized into three parts:

Local Store
In the Local Store solution, the SSD buffer coexists with the DRAM buffer on the same node. The SSD stores only the data from the DRAM locally. The DRAM buffer will be periodically persisted to the SSD at arbitrary intervals. Once the DRAM buffer is flushed to the SSD, it can be removed from the node to free up storage resources.
In this solution, once the data can't be fetched in the DRAM, it needs to further look up the SSD meta data on the node. The latency is expected to be slightly longer than the legacy non-SSD solution, our best guess the penalty can be less than several micro-seconds depending on the local buffer indexing design.

Global Store
In the Global Store solution, the location's relationship between SSD buffer and DRAM buffer is orthogonal, one SSD can receive data buffer from any node's DRAM. The centralized Meta Service is responsible to lookup the SSD buffer address.
One open question is whether to write the buffer to a remote node in one or two steps, which means the buffer transfer is via directly one-hop DRAM-SSD or via two-hops DRAM-DRAM-SSD:

In the one-step case, the current TransferEngine, which is a DRAM-to-DRAM oriented mechanism, is insufficient. We need to adopt mechanisms such as NVMeoF as the transfer channel. Alternatively, we could extend the TransferEngine's functionality on the server side - using a mechanism similar to the DeepSeek 3FS ChunkEngine - to perform write operations by conducting a reverse "RDMA_READ" from the server to the client and harvesting the result.
In the two-step case, the write can be composed of firstly use DRAM-DRAM transfer which is already supported, and then use Local Store way to persist the buffer into SSD and update the Meta Service.

In terms of write engine, we propose to write data into the SSD in several methods:
POSIX API - the advantage is relatively easier to implement and deploy, the disadvantage is latency is longer due to context switch and data copy between user space and kernel space.
SPDK NVMe (Local Store) - the advantage is latency is shorter, the deployment is complicated and the implementation impacts source side.
SPDK NVMeoF (Global Store) - the advantage is the latency is shorter, but the deployment is more complicated and the implementation impacts both initiator and target.

3rd-party Distributed Store
In the 3rd-party Distributed Store solution, the data buffer is stored in an independent, third-party distributed file system, such as DeepSeek 3FS. Mooncake-Store maintains the indexing between the KVCache object and the store file in the distributed file system. This approach allows the capacity of the persisted KVCache to be enormous while also reducing implementation and deployment complexity.
Meanwhile, managing cache eviction policies in a distributed file system might be more complex than in an in-memory or local store-based system. We need to consider how to efficiently purge stale data and keep cache consistency across the distributed system.
Maintaining an index (as Mooncake-Store does) between the KVCache object and the store file can introduce additional complexity. As the dataset grows, the indexing itself may become a bottleneck, requiring extra resources to maintain and scale it effectively.

Feedback Period.

No response

CC List.

@ShangmingCai @stmatengss @james0zan @doujiang24 @alogfans

Any Other Things.

No response

Originally created by @yuan-luo on GitHub (Mar 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/171 ### Motivation. Hello Mooncake community, We're from Ant Group team and are actively working on P/D disaggregation inference. We'd like to share our initial proposal for achieving SSD Offload Mooncake-Store, the distributed object store for vLLM XpYd scenarios. With this feature, KVCache store is expected to be more robust, achieve better performance (e.g., long documents) in case of hitting cache in LLM inference. ### Proposed Change. The plan is to implement SSD offload within Mooncake-Store. The entire solution can be categorized into three parts: **Local Store** In the _Local Store_ solution, the SSD buffer coexists with the DRAM buffer on the same node. The SSD stores only the data from the DRAM locally. The DRAM buffer will be periodically persisted to the SSD at arbitrary intervals. Once the DRAM buffer is flushed to the SSD, it can be removed from the node to free up storage resources. In this solution, once the data can't be fetched in the DRAM, it needs to further look up the SSD meta data on the node. The latency is expected to be slightly longer than the legacy non-SSD solution, our best guess the penalty can be less than several micro-seconds depending on the local buffer indexing design. **Global Store** In the _Global Store_ solution, the location's relationship between SSD buffer and DRAM buffer is orthogonal, one SSD can receive data buffer from any node's DRAM. The centralized Meta Service is responsible to lookup the SSD buffer address. One open question is whether to write the buffer to a remote node in one or two steps, which means the buffer transfer is via directly one-hop DRAM-SSD or via two-hops DRAM-DRAM-SSD: * In the one-step case, the current TransferEngine, which is a DRAM-to-DRAM oriented mechanism, is insufficient. We need to adopt mechanisms such as NVMeoF as the transfer channel. Alternatively, we could extend the TransferEngine's functionality on the server side - using a mechanism similar to the DeepSeek 3FS ChunkEngine - to perform write operations by conducting a reverse "RDMA_READ" from the server to the client and harvesting the result. * In the two-step case, the write can be composed of firstly use DRAM-DRAM transfer which is already supported, and then use _Local Store_ way to persist the buffer into SSD and update the Meta Service. In terms of write engine, we propose to write data into the SSD in several methods: * POSIX API - the advantage is relatively easier to implement and deploy, the disadvantage is latency is longer due to context switch and data copy between user space and kernel space. * SPDK NVMe (Local Store) - the advantage is latency is shorter, the deployment is complicated and the implementation impacts source side. * SPDK NVMeoF (Global Store) - the advantage is the latency is shorter, but the deployment is more complicated and the implementation impacts both initiator and target. **3rd-party Distributed Store** In the _3rd-party Distributed Store_ solution, the data buffer is stored in an independent, third-party distributed file system, such as DeepSeek 3FS. Mooncake-Store maintains the indexing between the KVCache object and the store file in the distributed file system. This approach allows the capacity of the persisted KVCache to be enormous while also reducing implementation and deployment complexity. Meanwhile, managing cache eviction policies in a distributed file system might be more complex than in an in-memory or local store-based system. We need to consider how to efficiently purge stale data and keep cache consistency across the distributed system. Maintaining an index (as Mooncake-Store does) between the KVCache object and the store file can introduce additional complexity. As the dataset grows, the indexing itself may become a bottleneck, requiring extra resources to maintain and scale it effectively. ### Feedback Period. _No response_ ### CC List. [@ShangmingCai](https://github.com/ShangmingCai) [@stmatengss](https://github.com/stmatengss) [@james0zan](https://github.com/james0zan) [@doujiang24](https://github.com/doujiang24) [@alogfans](https://github.com/alogfans) ### Any Other Things. _No response_

bsakaguchi

2026-10-29 21:08:52 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@stmatengss commented on GitHub (Mar 26, 2025):

Thank you for your contribution! @yuan-luo Local store is a high priority. LMCache/InfiniStore supports local persistent KVCache Store. Flushing KVCache to storage is already on Mooncake's roadmap. SSD Offload should be (1) optional, because DRAM is enough for storing KV currently. (2) not in critical path: asynchronously flushing to SSD could reduce the influence on performance.

@stmatengss commented on GitHub (Mar 26, 2025): Thank you for your contribution! @yuan-luo Local store is a high priority. LMCache/InfiniStore supports local persistent KVCache Store. Flushing KVCache to storage is already on Mooncake's roadmap. SSD Offload should be (1) optional, because DRAM is enough for storing KV currently. (2) not in critical path: asynchronously flushing to SSD could reduce the influence on performance.

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 20, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 20, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 27, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 27, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:17 +00:00

[PR #80] [MERGED] Fix P2P store bugs and improvement #1314