kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #928] [Usage]: how to understand mooncake? #295

New issue

Closed

opened 2026-10-29 21:12:11 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:12:11 +00:00

Owner

Originally created by @Alan-D-Chen on GitHub (Oct 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/928

Describe your usage question

Type A

This is a one-to-one PD disaggregated inference framework. Under this framework, the KVcache generated by GPU-a in the prefill node can only be transmitted to GPU-a in the decode node; in other words, GPU-a in the decode node can only access the memory of GPU-a in the prefill node.

Type B

This is a PD inference framework with a KVcache store. In this framework, the KVcache generated by the GPUs in the prefill node must first be transmitted to the KVcache store, and the framework will prioritize allocating the KVcache from the KVcache store to idle decode GPUs based on the busyness of the GPUs in the decode node.

May I ask which type of inference framework the mooncake belongs to?

In the P4D4 framework structure, the cache of prefill node 0 can only be accessed by decode node 0, rather than by "any arbitrary" decode node that can directly access the cache of prefill node 0, and it is not "shared across the entire cluster". Is my understanding correct?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Alan-D-Chen on GitHub (Oct 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/928 ### Describe your usage question <img width="1120" height="758" alt="Image" src="https://github.com/user-attachments/assets/d28af12b-9682-44d8-b155-709fac0bd0d9" /> ### **Type A** This is a one-to-one PD disaggregated inference framework. Under this framework, the KVcache generated by GPU-a in the prefill node can only be transmitted to GPU-a in the decode node; in other words, GPU-a in the decode node can only access the memory of GPU-a in the prefill node. <img width="1587" height="727" alt="Image" src="https://github.com/user-attachments/assets/da648d5a-cb3c-4bde-9f70-158afb0885d6" /> ### **Type B** This is a PD inference framework with a KVcache store. In this framework, the KVcache generated by the GPUs in the prefill node must first be transmitted to the KVcache store, and the framework will prioritize allocating the KVcache from the KVcache store to idle decode GPUs based on the busyness of the GPUs in the decode node. May I ask which type of inference framework the mooncake belongs to? In the P4D4 framework structure, the cache of prefill node 0 can only be accessed by decode node 0, rather than by "any arbitrary" decode node that can directly access the cache of prefill node 0, and it is not "shared across the entire cluster". Is my understanding correct? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:12:11 +00:00

closed this issue
added the
stale

auto-closed

question
labels

bsakaguchi commented

2026-10-29 21:12:12 +00:00

Author

Owner

@Alan-D-Chen commented on GitHub (Oct 14, 2025):

I hope my question hasn't caused you too much trouble. We students are fascinated by the wisdom of the mooncake authors and have developed a great interest in it.

@tchaikov @misterwilliam @simpx @karya0

@Alan-D-Chen commented on GitHub (Oct 14, 2025): I hope my question hasn't caused you too much trouble. We students are fascinated by the wisdom of the mooncake authors and have developed a great interest in it. @tchaikov @misterwilliam @simpx @karya0

bsakaguchi commented

2026-10-29 21:12:12 +00:00

Author

Owner

@stmatengss commented on GitHub (Oct 14, 2025):

The answer should be both, plz ref to this figure:

@stmatengss commented on GitHub (Oct 14, 2025): The answer should be both, plz ref to this figure: <img width="2042" height="526" alt="Image" src="https://github.com/user-attachments/assets/674c7ea0-297a-44c2-82ae-007d998e4f23" />

bsakaguchi commented

2026-10-29 21:12:12 +00:00

Author

Owner

@Alan-D-Chen commented on GitHub (Oct 15, 2025):

@stmatengss Hello, thank you very much for your reply. May I ask where you saw these two pictures? In my understanding, mooncake is a PD disaggregated inference framework. However, from what it seems in the picture on your right, mooncake appears to be a plugin, and the mooncake store is integrated into the vLLM framework.

@Alan-D-Chen commented on GitHub (Oct 15, 2025): @stmatengss Hello, thank you very much for your reply. May I ask where you saw these two pictures? In my understanding, mooncake is a PD disaggregated inference framework. However, from what it seems in the picture on your right, mooncake appears to be a plugin, and the mooncake store is integrated into the vLLM framework.

bsakaguchi commented

2026-10-29 21:12:12 +00:00

Author

Owner

@stmatengss commented on GitHub (Oct 15, 2025):

You can understand each inference instance has a co-located client which occupies memory resources, and all these clients combined together can be abstracted as a mooncake store.

@stmatengss commented on GitHub (Oct 15, 2025): You can understand each inference instance has a co-located client which occupies memory resources, and all these clients combined together can be abstracted as a mooncake store.

bsakaguchi commented

2026-10-29 21:12:13 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:12:13 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 29, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jun 29, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.