[GH-ISSUE #159] KV cache SSD Offloading #77

New issue

Closed

opened 2026-10-29 21:08:48 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:08:48 +00:00

Owner

Originally created by @ekaynar on GitHub (Mar 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/159

Is it possible to offload the KV cache to SSDs when we use vllm disaggregated inference?
If so, could you please let me know how to configure the path and size of the drive that I want to use for caching?

Thank you

Originally created by @ekaynar on GitHub (Mar 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/159 Is it possible to offload the KV cache to SSDs when we use vllm disaggregated inference? If so, could you please let me know how to configure the path and size of the drive that I want to use for caching? Thank you

bsakaguchi closed this issue

2026-10-29 21:08:48 +00:00

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 21, 2025):

I think this solution has no performance gain for disaggregated PD, it may be for the state preservation and recovery of the inference engine. I think some team is working on it. Please check https://github.com/vllm-project/vllm/pull/11532 https://github.com/vllm-project/vllm/pull/8018

@ShangmingCai commented on GitHub (Mar 21, 2025): I think this solution has no performance gain for disaggregated PD, it may be for the state preservation and recovery of the inference engine. I think some team is working on it. Please check https://github.com/vllm-project/vllm/pull/11532 https://github.com/vllm-project/vllm/pull/8018

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 21, 2025):

@ekaynar You can joint the #feat-kvcache-offloading slack channel of vllm to get more info.

@ShangmingCai commented on GitHub (Mar 21, 2025): @ekaynar You can joint the #feat-kvcache-offloading slack channel of vllm to get more info.

bsakaguchi commented

2026-10-29 21:08:49 +00:00

Author

Owner

@ekaynar commented on GitHub (Mar 21, 2025):

@ShangmingCai Thank you for your response! To clarify, are you saying that the Mooncake-store KV Cache pool can only be created using a CPU or GPU? Does this mean that Mooncake store cannot utilize other storage mediums, such as SSDs, NVMes, or remote storage, as a KV Cache?

I noticed in the document that the transfer engine supports NVMeOF. Could you explain what that is used for?

@ekaynar commented on GitHub (Mar 21, 2025): @ShangmingCai Thank you for your response! To clarify, are you saying that the Mooncake-store KV Cache pool can only be created using a CPU or GPU? Does this mean that Mooncake store cannot utilize other storage mediums, such as SSDs, NVMes, or remote storage, as a KV Cache? I noticed in the document that the transfer engine supports NVMeOF. Could you explain what that is used for?

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 21, 2025):

@ekaynar Mooncake transfer engine supports various backends, but MooncakeStore is specially designed for high-performance KVCache transmission, which utilizes the Mooncake transfer engine to do the transfer. Potentially, we may make some modifications to the backend of MooncakeStore to make it work, but performance is our top priority, so SSDs-based solutions will not be considered at least recently.

@ShangmingCai commented on GitHub (Mar 21, 2025): @ekaynar Mooncake transfer engine supports various backends, but MooncakeStore is specially designed for high-performance KVCache transmission, which utilizes the Mooncake transfer engine to do the transfer. Potentially, we may make some modifications to the backend of MooncakeStore to make it work, but performance is our top priority, so SSDs-based solutions will not be considered at least recently.

bsakaguchi commented

2026-10-29 21:08:50 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 26, 2025):

Since https://github.com/kvcache-ai/Mooncake/issues/171 will address it in the near future, I will close this issue. Feel free to reopen it again.

@ShangmingCai commented on GitHub (Mar 26, 2025): Since https://github.com/kvcache-ai/Mooncake/issues/171 will address it in the near future, I will close this issue. Feel free to reopen it again.

bsakaguchi referenced this issue

2026-10-29 21:26:16 +00:00

[PR #77] [CLOSED] [TransferEngine] Implement Linux Shared Memory Transport #issue48 #1312