kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #368] How to use Mooncake in vLLM V1? #127

New issue

Closed

opened 2026-10-29 21:09:47 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented

2026-10-29 21:09:47 +00:00

Owner

Originally created by @zhenwei-intel on GitHub (May 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/368

Originally created by @zhenwei-intel on GitHub (May 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/368

bsakaguchi closed this issue

2026-10-29 21:09:47 +00:00

bsakaguchi commented

2026-10-29 21:09:48 +00:00

Author

Owner

@stmatengss commented on GitHub (May 15, 2025):

Currently, lmcache + mooncake supports deploying vllm v1.
Here are some references:
https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/lmcache-integration.md
https://blog.lmcache.ai/2025-05-08-mooncake/

@stmatengss commented on GitHub (May 15, 2025): Currently, ` lmcache + mooncake` supports deploying vllm v1. Here are some references: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/lmcache-integration.md https://blog.lmcache.ai/2025-05-08-mooncake/

bsakaguchi commented

2026-10-29 21:09:48 +00:00

Author

Owner

@tensorflowt commented on GitHub (May 16, 2025):

Currently, lmcache + mooncake supports deploying vllm v1. Here are some references: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/lmcache-integration.md https://blog.lmcache.ai/2025-05-08-mooncake/

How to use it? Currently I only see the performance indicators！

@tensorflowt commented on GitHub (May 16, 2025): > Currently, ` lmcache + mooncake` supports deploying vllm v1. Here are some references: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/lmcache-integration.md https://blog.lmcache.ai/2025-05-08-mooncake/ How to use it? Currently I only see the performance indicators！

bsakaguchi commented

2026-10-29 21:09:48 +00:00

Author

Owner

@zhenwei-intel commented on GitHub (May 16, 2025):

I found a doc here: github.com/LMCache/LMCache@635eefb7ff/examples/experimental/storage_backends/mooncakestore/README.md

However, the doc uses V0, so I'm not sure if it will work on V1.
VLLM_USE_V1=0

@zhenwei-intel commented on GitHub (May 16, 2025): I found a doc here: https://github.com/LMCache/LMCache/blob/635eefb7ff5360df579ef6258710307c133036e6/examples/experimental/storage_backends/mooncakestore/README.md However, the doc uses V0, so I'm not sure if it will work on V1. `VLLM_USE_V1=0`

bsakaguchi commented

2026-10-29 21:09:48 +00:00

Author

Owner

@stmatengss commented on GitHub (May 18, 2025):

Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake.

chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://192.168.0.137:50051/"
remote_serde: "naive"
pipelined_backend: False
max_local_cpu_size: 100

We will release a deployment document ASAP. thx!

@stmatengss commented on GitHub (May 18, 2025): Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake. ``` chunk_size: 256 local_device: "cpu" remote_url: "mooncakestore://192.168.0.137:50051/" remote_serde: "naive" pipelined_backend: False max_local_cpu_size: 100 ``` We will release a deployment document ASAP. thx!

bsakaguchi commented

2026-10-29 21:09:48 +00:00

Author

Owner

@zhenwei-intel commented on GitHub (May 21, 2025):

[Doc] Add document for integrating Mooncake Store to LMCache V1 https://github.com/kvcache-ai/Mooncake/pull/385

@zhenwei-intel commented on GitHub (May 21, 2025): [Doc] Add document for integrating Mooncake Store to LMCache V1 https://github.com/kvcache-ai/Mooncake/pull/385

bsakaguchi commented

2026-10-29 21:09:49 +00:00

Author

Owner

@zhenwei-intel commented on GitHub (May 21, 2025):

Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake.
chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://192.168.0.137:50051/"
remote_serde: "naive"
pipelined_backend: False
max_local_cpu_size: 100
We will release a deployment document ASAP. thx!

Here it's also using a KV pool instead of P2P for KV transmission, right?

this solution doesn't rely on CUDA, right?

@zhenwei-intel commented on GitHub (May 21, 2025): > Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake. > > ``` > chunk_size: 256 > local_device: "cpu" > remote_url: "mooncakestore://192.168.0.137:50051/" > remote_serde: "naive" > pipelined_backend: False > max_local_cpu_size: 100 > ``` > > We will release a deployment document ASAP. thx! Here it's also using a KV pool instead of P2P for KV transmission, right? this solution doesn't rely on CUDA, right?

bsakaguchi commented

2026-10-29 21:09:49 +00:00

Author

Owner

@stmatengss commented on GitHub (May 21, 2025):

Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake.
chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://192.168.0.137:50051/"
remote_serde: "naive"
pipelined_backend: False
max_local_cpu_size: 100
We will release a deployment document ASAP. thx!
Here it's also using a KV pool instead of P2P for KV transmission, right?

this solution doesn't rely on CUDA, right?

Yes, it is a CPU-offload KVCache pool. We will release a VRAM-based implementation

@stmatengss commented on GitHub (May 21, 2025): > > Mooncake is one of the LMCache backends. You should follow the user guidance in LMCache (https://docs.lmcache.ai/getting_started/installation.html#install-from-source-v1) and set it as a backend. Here is an example YAML file for mooncake. > > ``` > > chunk_size: 256 > > local_device: "cpu" > > remote_url: "mooncakestore://192.168.0.137:50051/" > > remote_serde: "naive" > > pipelined_backend: False > > max_local_cpu_size: 100 > > ``` > > > > > > > > > > > > > > > > > > > > > > > > We will release a deployment document ASAP. thx! > > Here it's also using a KV pool instead of P2P for KV transmission, right? > > this solution doesn't rely on CUDA, right? Yes, it is a CPU-offload KVCache pool. We will release a VRAM-based implementation

bsakaguchi commented

2026-10-29 21:09:49 +00:00

Author

Owner

@mugglew commented on GitHub (May 28, 2025):

@stmatengss is that means we can not use mooncake + vllm v1 on cuda env such as H20?

@mugglew commented on GitHub (May 28, 2025): @stmatengss is that means we can not use mooncake + vllm v1 on cuda env such as H20?