[GH-ISSUE #50 ] Inquiry Regarding &#34;Mooncake- A KVCache-centric Disaggregated Architecture for LLM Serving&#34; #31

Author

Owner

@chestnut-Q commented on GitHub (Dec 28, 2024):

Hello @VegetaPn,

Thank you for your interest in Mooncake and for sharing these questions. Below are the detailed answers:

These blocks are the logical IDs of memory blocks. For example, if a machine allocates 10,000 blocks at startup, the block IDs would range from 0 to 9999.
Conductor estimates processing time based on the requests currently being processed or queued on each instance (e.g., prompt_len, prefix_prompt_len, batch_size). For prefill instances, Conductor estimates the TTFT based on the last request in the queue; for decoding instances, it estimates the TBT for the requests in a batch. Conductor handles multiple concurrent requests efficiently, so any additional latency is negligible in practice.
Currently, there is only one Conductor in each Mooncake cluster, running on a dedicated node.
Conductor adopts an LRU eviction strategy for KVCache, with carefully designed concurrency and locking mechanisms to ensure that eviction does not disrupt the selection of a prefill node. Specifically, there is no KVCache eviction during prefix hash matching; moreover, once a block is matched, it will not be evicted until its ref_count drops to 0.

I hope this helps clarify your questions. Please let me know if there's anything else I can assist you with.

@chestnut-Q commented on GitHub (Dec 28, 2024): Hello @VegetaPn, Thank you for your interest in Mooncake and for sharing these questions. Below are the detailed answers: 1. These blocks are the logical IDs of memory blocks. For example, if a machine allocates 10,000 blocks at startup, the block IDs would range from `0` to `9999`. 2. Conductor estimates processing time based on the requests currently being processed or queued on each instance (e.g., `prompt_len`, `prefix_prompt_len`, `batch_size`). For prefill instances, Conductor estimates the TTFT based on the last request in the queue; for decoding instances, it estimates the TBT for the requests in a batch. Conductor handles multiple concurrent requests efficiently, so any additional latency is negligible in practice. 3. Currently, there is only one Conductor in each Mooncake cluster, running on a dedicated node. 4. Conductor adopts an LRU eviction strategy for KVCache, with carefully designed concurrency and locking mechanisms to ensure that eviction does not disrupt the selection of a prefill node. Specifically, there is no KVCache eviction during prefix hash matching; moreover, once a block is matched, it will not be evicted until its `ref_count` drops to 0. I hope this helps clarify your questions. Please let me know if there's anything else I can assist you with.

bsakaguchi commented

Author

Owner

@VegetaPn commented on GitHub (Dec 28, 2024):

@chestnut-Q Thank you very much for your reply!

I have a few more questions I'd like to ask:

Regarding the second point, data such as queue_duration and batch_size belong to the inference instances, How are they synchronized to Conductor—by carrying them in the inference results or by synchronizing them at a fixed frequency?
Regarding the third point, since Conductor is a single-instance service, is there currently a fallback mechanism if Conductor unavailable/down? If it evolves into a multi-instance service, are there any good solutions in place? What I can think of is to create replicas of Conductor, and if the primary node fails, switch one of the replicas to become the primary node. However, this would require maintaining a very strong consistency between the primary and secondary nodes.
What is the call chain between Conductor and inference instances? 1) Does Conductor first call Prefill, get the initial token, and then call Decoding, or 2) call Prefill and Decoding simultaneously, or 3) Conductor only calls Decoding, and then Decoding initiates a call to Prefill? Although it seems to be the first method in vLLM, I would like to understand the calling relationship in Kimi. Because it seems that the latter two methods are more friendly for KVCache transmission.
Looking at the current open-source code of TransferEngine, if two nodes are to perform KVCache transmission, the initiating node needs to know the virtual address of the target node and the rkey of the MemoryRegion. Will these address details also be managed by Conductor? If Conductor does not manage the address information of Blocks, then it is necessary to pre-communicate the address information of Blocks between the two nodes? Does this imply that there is communication between P and D?
If there is communication between P and D, is the metadata management service of TransferEngine unnecessary? Wouldn't it be better to simply convey the metadata needed for RDMA in the communication information between P and D?

Your response has been very helpful to me, thank you again!

@VegetaPn commented on GitHub (Dec 28, 2024): @chestnut-Q Thank you very much for your reply! I have a few more questions I'd like to ask: 1. Regarding the second point, data such as queue_duration and batch_size belong to the inference instances, How are they synchronized to Conductor—by carrying them in the inference results or by synchronizing them at a fixed frequency? 2. Regarding the third point, since Conductor is a single-instance service, is there currently a fallback mechanism if Conductor unavailable/down? If it evolves into a multi-instance service, are there any good solutions in place? What I can think of is to create replicas of Conductor, and if the primary node fails, switch one of the replicas to become the primary node. However, this would require maintaining a very strong consistency between the primary and secondary nodes. 3. What is the call chain between Conductor and inference instances? 1) Does Conductor first call Prefill, get the initial token, and then call Decoding, or 2) call Prefill and Decoding simultaneously, or 3) Conductor only calls Decoding, and then Decoding initiates a call to Prefill? Although it seems to be the first method in vLLM, I would like to understand the calling relationship in Kimi. Because it seems that the latter two methods are more friendly for KVCache transmission. 4. Looking at the current open-source code of TransferEngine, if two nodes are to perform KVCache transmission, the initiating node needs to know the virtual address of the target node and the rkey of the MemoryRegion. Will these address details also be managed by Conductor? If Conductor does not manage the address information of Blocks, then it is necessary to pre-communicate the address information of Blocks between the two nodes? Does this imply that there is communication between P and D? 5. If there is communication between P and D, is the metadata management service of TransferEngine unnecessary? Wouldn't it be better to simply convey the metadata needed for RDMA in the communication information between P and D? Your response has been very helpful to me, thank you again!

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 28, 2024):

Hello @VegetaPn, here are the answers:

Conductor has information on all requests on an instance when scheduling requests, and it maintains this information based on responses of instances.
Because Conductor has a very low chance of being unavailable, we generally treat it as though there is only a single instance.
We choose (1).
&5. There is no communication between P and D. Instead, an additional metadata server is set up outside of Conductor, aimed at simplifying the interface and facilitating scalability.

If you have any more questions, please feel free to ask.

@chestnut-Q commented on GitHub (Dec 28, 2024): Hello @VegetaPn, here are the answers: 1. Conductor has information on all requests on an instance when scheduling requests, and it maintains this information based on responses of instances. 2. Because Conductor has a very low chance of being unavailable, we generally treat it as though there is only a single instance. 3. We choose (1). 4. &5. There is no communication between P and D. Instead, an additional metadata server is set up outside of Conductor, aimed at simplifying the interface and facilitating scalability. If you have any more questions, please feel free to ask.

bsakaguchi commented

Author

Owner

@VegetaPn commented on GitHub (Dec 28, 2024):

@chestnut-Q Thank you very much! I have a few more questions I'd like to ask:

Based on your response, Conductor manages the memory allocation for Prefill instances; does Conductor also manage the memory allocation for Decoding? The length of the response generated during the Decode phase is uncertain, making it challenging if Conductor is to manage the memory allocation for D.
Metadata server, can it be understood that: the metaserver stores static information, such as the address of each block, rather than frequently changing dynamic information. During KVCache transmission, the blockIds have already been obtained from the requests, and it is only necessary to query the actual virtual address/Rkey from the information in the metaserver. Even the data in the metaserver can be cached locally.
If Conductor requests the Prefill instance first, will the Prefill instance transmit the KVCache layer by layer to the Decoding instance simultaneously with its computation? Or will it wait until Conductor requests the Decoding instance to then perform the KVCache transfer?
In a multi-round dialogue scenario, when initiating a second session request for Prefill, the longest reusable KVCache is on the Decoding instance (which stores the KVCache generated during the first session). At this point, will this Prefill instance pull the KVCache from the Decoding instance?

@VegetaPn commented on GitHub (Dec 28, 2024): @chestnut-Q Thank you very much! I have a few more questions I'd like to ask: 1. Based on your response, Conductor manages the memory allocation for Prefill instances; does Conductor also manage the memory allocation for Decoding? The length of the response generated during the Decode phase is uncertain, making it challenging if Conductor is to manage the memory allocation for D. 2. Metadata server, can it be understood that: the metaserver stores static information, such as the address of each block, rather than frequently changing dynamic information. During KVCache transmission, the blockIds have already been obtained from the requests, and it is only necessary to query the actual virtual address/Rkey from the information in the metaserver. Even the data in the metaserver can be cached locally. 3. If Conductor requests the Prefill instance first, will the Prefill instance transmit the KVCache layer by layer to the Decoding instance simultaneously with its computation? Or will it wait until Conductor requests the Decoding instance to then perform the KVCache transfer? 4. In a multi-round dialogue scenario, when initiating a second session request for Prefill, the longest reusable KVCache is on the Decoding instance (which stores the KVCache generated during the first session). At this point, will this Prefill instance pull the KVCache from the Decoding instance?

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 28, 2024):

@VegetaPn, here is a straightforward and easy-to-implement approach: Conductor does not manage the KVCache of the decoding instance, which means the decoding node does not cache the KVCache. There is no layer-by-layer KVCache transfer between the prefill and decoding nodes. These optimizations are not discussed in the paper but can be implemented on top of it. Your understanding of point 2 is correct.

@chestnut-Q commented on GitHub (Dec 28, 2024): @VegetaPn, here is a straightforward and easy-to-implement approach: Conductor does not manage the KVCache of the decoding instance, which means the decoding node does not cache the KVCache. There is no layer-by-layer KVCache transfer between the prefill and decoding nodes. These optimizations are not discussed in the paper but can be implemented on top of it. Your understanding of point 2 is correct.

bsakaguchi commented

Author

Owner

@VegetaPn commented on GitHub (Dec 29, 2024):

@chestnut-Q, if it's not a layer-by-layer transfer, then the entire KVCache can only be transferred after the prefill execution is completed, which would likely be inefficient. How have you considered the load and efficiency issues regarding the transfer? Have there been any related optimizations implemented in Kimi?

@VegetaPn commented on GitHub (Dec 29, 2024): @chestnut-Q, if it's not a layer-by-layer transfer, then the entire KVCache can only be transferred after the prefill execution is completed, which would likely be inefficient. How have you considered the load and efficiency issues regarding the transfer? Have there been any related optimizations implemented in Kimi?

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 29, 2024):

@VegetaPn Yes, but this asynchronous transfer does not interfere with the model inference on GPUs.

@chestnut-Q commented on GitHub (Dec 29, 2024): @VegetaPn Yes, but this asynchronous transfer does not interfere with the model inference on GPUs.

bsakaguchi commented

Author

Owner

@woodyji commented on GitHub (Dec 29, 2024):

The transfer of these KVCache blocks across CPUs and GPUs is handled by a separate (GPUDirect) RDMA-based component called Messenger. This archtecture also enables us to provide the context caching API to outside users for a higher reusement of KVCache.

Based on your previous response and your paper, does the prefill instance write KVCache to the decode instance using a GPUDirect-RDMA WRITE operation? Is the source address of the prefill instance in GPU space while the target address of the decode instance is in CPU space?
The prefill instance transfers KVCache layer-by-layer. Once the prefill instance has completed all computations and transfers, the conductor begins scheduling the decode instance. Therefore, does the decode instance need to copy KVCache from CPU to GPU without any overlap during the request's life cycle?

@VegetaPn

@woodyji commented on GitHub (Dec 29, 2024): `The transfer of these KVCache blocks across CPUs and GPUs is handled by a separate (GPUDirect) RDMA-based component called Messenger. This archtecture also enables us to provide the context caching API to outside users for a higher reusement of KVCache.` 1. Based on your previous response and your paper, does the prefill instance write KVCache to the decode instance using a GPUDirect-RDMA `WRITE` operation? Is the source address of the prefill instance in GPU space while the target address of the decode instance is in CPU space? 2. The prefill instance transfers KVCache layer-by-layer. Once the prefill instance has completed all computations and transfers, the `conductor` begins scheduling the decode instance. Therefore, does the decode instance need to copy KVCache from CPU to GPU without any overlap during the request's life cycle? @VegetaPn

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 29, 2024):

We believe that using CPU-based two-hop transfers is easier to implement and scale, as it requires fewer modifications to the existing inference framework. Moreover, KVCache transfers will not affect the inference of other requests, which is advantageous for systems where throughput is a priority. The choice of transfer method depends on the storage medium (e.g., HBM, DRAM, etc.) and the hardware setup. You can refer to our mooncake-transfer-engine implementation for more details.

@woodyji

@chestnut-Q commented on GitHub (Dec 29, 2024): We believe that using CPU-based two-hop transfers is easier to implement and scale, as it requires fewer modifications to the existing inference framework. Moreover, KVCache transfers will not affect the inference of other requests, which is advantageous for systems where throughput is a priority. The choice of transfer method depends on the storage medium (e.g., HBM, DRAM, etc.) and the hardware setup. You can refer to our `mooncake-transfer-engine` implementation for more details. @woodyji

bsakaguchi commented

Author

Owner

@woodyji commented on GitHub (Dec 29, 2024):

I noticed that the mooncake-transfer-engine supports multiple types of mediums. However, I'm confused about the actual choices made by mooncake, as the paper mentions both GPU-to-CPU and CPU-to-CPU transfers. @chestnut-Q

@woodyji commented on GitHub (Dec 29, 2024): I noticed that the mooncake-transfer-engine supports multiple types of mediums. However, I'm confused about the actual choices made by mooncake, as the paper mentions both GPU-to-CPU and CPU-to-CPU transfers. @chestnut-Q

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 29, 2024):

@woodyji Mooncake supports multiple types of transfers to accommodate different hardware conditions. However, you can think of it as GPU -> CPU -> GPU because this modification is the simplest, as I mentioned above.

@chestnut-Q commented on GitHub (Dec 29, 2024): @woodyji Mooncake supports multiple types of transfers to accommodate different hardware conditions. However, you can think of it as GPU -> CPU -> GPU because this modification is the simplest, as I mentioned above.

bsakaguchi commented

Author

Owner

@woodyji commented on GitHub (Dec 30, 2024):

Thank you for your response. I have two additional questions regarding KVCache transfer:

In the GPU -> CPU -> GPU mode, should the prefill process first offload the KVCache from GPU to CPU, followed by the Messenger handling the CPU to CPU transfer via RDMA? Once the transfer is complete, the decode process perform an async load from CPU to GPU?
As mentioned in your paper, the Messenger operates as an independent process. Does it share the same CPU address space with the inference process? How do you implement the sharing of such a large address space across multiple processes?

@chestnut-Q

@woodyji commented on GitHub (Dec 30, 2024): Thank you for your response. I have two additional questions regarding KVCache transfer: 1. In the GPU -> CPU -> GPU mode, should the prefill process first offload the KVCache from GPU to CPU, followed by the `Messenger` handling the CPU to CPU transfer via RDMA? Once the transfer is complete, the decode process perform an async load from CPU to GPU? 2. As mentioned in your paper, the `Messenger` operates as an independent process. Does it share the same CPU address space with the inference process? How do you implement the sharing of such a large address space across multiple processes? @chestnut-Q

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 30, 2024):

@woodyji

In the GPU -> CPU -> GPU mode, should the prefill process first offload the KVCache from GPU to CPU, followed by the Messenger handling the CPU to CPU transfer via RDMA? Once the transfer is complete, the decode process perform an async load from CPU to GPU?

Yes.

As mentioned in your paper, the Messenger operates as an independent process. Does it share the same CPU address space with the inference process? How do you implement the sharing of such a large address space across multiple processes?

You can use shared memory to implement this.

@chestnut-Q commented on GitHub (Dec 30, 2024): @woodyji > In the GPU -> CPU -> GPU mode, should the prefill process first offload the KVCache from GPU to CPU, followed by the `Messenger` handling the CPU to CPU transfer via RDMA? Once the transfer is complete, the decode process perform an async load from CPU to GPU? Yes. > As mentioned in your paper, the Messenger operates as an independent process. Does it share the same CPU address space with the inference process? How do you implement the sharing of such a large address space across multiple processes? You can use shared memory to implement this.

bsakaguchi commented

Author

Owner

@woodyji commented on GitHub (Dec 30, 2024):

The workflow makes sense. However, will the decode instance transfer the incremental KVCache to the prefill instance for better KVCache reuse? The paper does not seem to clearly state this. @chestnut-Q

@woodyji commented on GitHub (Dec 30, 2024): The workflow makes sense. However, will the decode instance transfer the incremental KVCache to the prefill instance for better KVCache reuse? The paper does not seem to clearly state this. @chestnut-Q

bsakaguchi commented

Author

Owner

@chestnut-Q commented on GitHub (Dec 30, 2024):

The workflow makes sense. However, will the decode instance transfer the incremental KVCache to the prefill instance for better KVCache reuse? The paper does not seem to clearly state this. @chestnut-Q

Please refer to my previous reply. @woodyji

@chestnut-Q commented on GitHub (Dec 30, 2024): > The workflow makes sense. However, will the decode instance transfer the incremental KVCache to the prefill instance for better KVCache reuse? The paper does not seem to clearly state this. @chestnut-Q Please refer to my previous reply. @woodyji

bsakaguchi commented

Author

Owner

@woodyji commented on GitHub (Jan 2, 2025):

In the GPU -> CPU -> GPU mode, will offloading and data transfer simultaneously lead to PCIe bandwidth contention?
@chestnut-Q

@woodyji commented on GitHub (Jan 2, 2025): In the GPU -> CPU -> GPU mode, will offloading and data transfer simultaneously lead to PCIe bandwidth contention? @chestnut-Q

bsakaguchi commented

Author

Owner

@github-actions[bot] commented on GitHub (Jul 23, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 23, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented