[GH-ISSUE #749] [Paper Question] this cache-aware prefill scheduling strategy is a little bit confused me #235

New issue

Closed

opened 2026-10-29 21:11:19 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:11:19 +00:00

Owner

Originally created by @Huixxi on GitHub (Aug 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/749

Originally assigned to: @chestnut-Q on GitHub.

Here, what if the time of prefill calculate this transfer_len prompt T_prefill_transfer_len on this current instance is shorter than the time T_transfer transfer this part of kvcache from the best_match_instance? Should there need compare these two time cost first then do the following TTFT comparison?

Originally created by @Huixxi on GitHub (Aug 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/749 Originally assigned to: @chestnut-Q on GitHub. <img width="1098" height="944" alt="Image" src="https://github.com/user-attachments/assets/4a55e951-ec9d-47a9-8406-9f453b229666" /> Here, what if the time of prefill calculate this transfer_len prompt T_prefill_transfer_len on this current instance is shorter than the time T_transfer transfer this part of kvcache from the best_match_instance? Should there need compare these two time cost first then do the following TTFT comparison?

bsakaguchi closed this issue

2026-10-29 21:11:19 +00:00

bsakaguchi commented

2026-10-29 21:11:20 +00:00

Author

Owner

@Huixxi commented on GitHub (Aug 15, 2025):

And it seems the selection of P doesn't consider the load of P, it that ok?

@Huixxi commented on GitHub (Aug 15, 2025): And it seems the selection of P doesn't consider the load of P, it that ok?

bsakaguchi commented

2026-10-29 21:11:20 +00:00

Author

Owner

@chestnut-Q commented on GitHub (Aug 17, 2025):

Hi @Huixxi. We don’t directly compare the transfer time (T_transfer) and the prefill time (T_prefill_transfer_len). Instead, we use the parameter kvcache_balancing_threshold to decide whether to transfer the KVCache. In long-context requests, where this algorithm is intended to be applied, transfer time is almost always lower than recomputation, so once best_prefix_len / prefix_len exceeds kvcache_balancing_threshold, we no longer consider recomputation.

Regarding the load on prefill instances, there is indeed an implicit consideration: we treat the sum of queued prefill execution times on an instance (T_queue) as its load.

@chestnut-Q commented on GitHub (Aug 17, 2025): Hi @Huixxi. We don’t directly compare the transfer time (T_transfer) and the prefill time (T_prefill_transfer_len). Instead, we use the parameter **kvcache_balancing_threshold** to decide whether to transfer the KVCache. In long-context requests, where this algorithm is intended to be applied, transfer time is almost always lower than recomputation, so once best_prefix_len / prefix_len exceeds **kvcache_balancing_threshold**, we no longer consider recomputation. Regarding the load on prefill instances, there is indeed an implicit consideration: we treat the sum of queued prefill execution times on an instance (T_queue) as its load.

bsakaguchi referenced this issue

2026-10-29 21:27:16 +00:00

[PR #235] [MERGED] [Bugfix]: fix redis access error during multi-threaded runtime. #1416