[GH-ISSUE #17] Why is there a transfer from prefill instance to decode instance inside a distributed kvcache? #15

New issue

Closed

opened 2026-10-29 21:07:53 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:07:53 +00:00

Owner

Originally created by @uniqueZt on GitHub (Dec 4, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/17

hello，Why is there a transfer from prefill instance to decode instance inside a distributed kvcache? Are you worried about a lot of traffic crossing?

Originally created by @uniqueZt on GitHub (Dec 4, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/17 hello，Why is there a transfer from prefill instance to decode instance inside a distributed kvcache? Are you worried about a lot of traffic crossing?

bsakaguchi closed this issue

2026-10-29 21:07:54 +00:00

bsakaguchi commented

2026-10-29 21:07:54 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Dec 5, 2024):

hello，Why is there a transfer from prefill instance to decode instance inside a distributed kvcache? Are you worried about a lot of traffic crossing?

I'm sorry. I'm not sure about your real question, but I'll do my best to answer it. Currently, we have open-sourced the underlying transfer engine. But the mooncake store is still in progress. Therefore, in the disaggregated prefill/decode demo, the KVCache needs to be transferred from the prefill instance to the decode instance.

Also, although data transmission is mainly from the prefill instance to the decode instance, communication signals and metadata are still required between them.

@ShangmingCai commented on GitHub (Dec 5, 2024): > hello，Why is there a transfer from prefill instance to decode instance inside a distributed kvcache? Are you worried about a lot of traffic crossing? I'm sorry. I'm not sure about your real question, but I'll do my best to answer it. Currently, we have open-sourced the underlying transfer engine. But the mooncake store is still in progress. Therefore, in the disaggregated prefill/decode demo, the KVCache needs to be transferred from the prefill instance to the decode instance. Also, although data transmission is mainly from the prefill instance to the decode instance, communication signals and metadata are still required between them.

bsakaguchi commented

2026-10-29 21:07:54 +00:00

Author

Owner

@uniqueZt commented on GitHub (Dec 5, 2024):

Okay, I mean in the architecture diagram, the DARM and SSD managed by the mooncake store should be similar to a distributed storage. Whether the internal transfer is to meet the network read overhead of the Prefill and decode stages as much as possible, that is, the local cache near the GPU, waiting for the open source of the overall system

@uniqueZt commented on GitHub (Dec 5, 2024): Okay, I mean in the architecture diagram, the DARM and SSD managed by the mooncake store should be similar to a distributed storage. Whether the internal transfer is to meet the network read overhead of the Prefill and decode stages as much as possible, that is, the local cache near the GPU, waiting for the open source of the overall system ![Uploading image.png…]()

bsakaguchi referenced this issue

2026-10-29 21:17:35 +00:00

[GH-ISSUE #2440] [Bug]: Execute mooncake+hixl, final destruction phase core #646