[GH-ISSUE #131] Potential Cache Hit Rate Reduction in Cache-Aware Scheduling #68

New issue

Closed

opened 2026-10-29 21:08:41 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:08:41 +00:00

Owner

Originally created by @cy953708688 on GitHub (Mar 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/131

In the cache-aware scheduling algorithm, there is a concern about whether the cache hit rate might decrease under certain conditions. Specifically, since the scheduling criterion is to select the node with the shortest TTFT (Time to First Token), if a worker node with a cache hit has a long queuetime, the scheduler might choose another node with a shorter queuetime but a lower cache hit rate. This could potentially lead to a reduction in the overall cache hit rate.

However, based on my understanding, the scheduling algorithm also takes into account the KV Cache transfer time and the computation time saved by cache reuse. If the sum of the KV Cache transfer time and the time saved by cache reuse is shorter than the additional queuetime of choosing another node, the system can still maintain a high cache hit rate even under high concurrency. This suggests that the scheduler makes a trade-off between the computation savings from cache reuse and the queuetime, ensuring that cache-aware scheduling can effectively utilize cache resources and improve overall system performance even under high load.

Could you please clarify whether this understanding is correct? Additionally, are there any specific mechanisms in place to further mitigate the potential reduction in cache hit rate under high concurrency scenarios?

Originally created by @cy953708688 on GitHub (Mar 7, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/131 In the cache-aware scheduling algorithm, there is a concern about whether the cache hit rate might decrease under certain conditions. Specifically, since the scheduling criterion is to select the node with the shortest TTFT (Time to First Token), if a worker node with a cache hit has a long queuetime, the scheduler might choose another node with a shorter queuetime but a lower cache hit rate. This could potentially lead to a reduction in the overall cache hit rate. However, based on my understanding, the scheduling algorithm also takes into account the KV Cache transfer time and the computation time saved by cache reuse. If the sum of the KV Cache transfer time and the time saved by cache reuse is shorter than the additional queuetime of choosing another node, the system can still maintain a high cache hit rate even under high concurrency. This suggests that the scheduler makes a trade-off between the computation savings from cache reuse and the queuetime, ensuring that cache-aware scheduling can effectively utilize cache resources and improve overall system performance even under high load. Could you please clarify whether this understanding is correct? Additionally, are there any specific mechanisms in place to further mitigate the potential reduction in cache hit rate under high concurrency scenarios?

bsakaguchi

2026-10-29 21:08:41 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:08:42 +00:00

Author

Owner

@chestnut-Q commented on GitHub (Mar 8, 2025):

@cy953708688 Yes, your understanding is correct. There is indeed a trade-off between cache locality and instance load. We achieve a balance by estimating transfer, queuing, and compute (with prefix cache) times to minimize the TTFT. Additionally, we have implemented a cache load balancing mechanism to enhance the hit rate of the hot caches. You can find a detailed discussion of these techniques in our paper.

@chestnut-Q commented on GitHub (Mar 8, 2025): @cy953708688 Yes, your understanding is correct. There is indeed a trade-off between cache locality and instance load. We achieve a balance by estimating transfer, queuing, and compute (with prefix cache) times to minimize the TTFT. Additionally, we have implemented a cache load balancing mechanism to enhance the hit rate of the hot caches. You can find a detailed discussion of these techniques in our paper.

bsakaguchi commented

2026-10-29 21:08:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:08:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 29, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 29, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.