[GH-ISSUE #492] Support priority to evict the longer part of the prompt token list #171

New issue

Closed

opened 2026-10-29 21:10:28 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:10:28 +00:00

Owner

Originally created by @maobaolong on GitHub (Jun 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/492

It would be better, if we support evict the key(it is chunk in lmcache which contains a part of kvcache of prompt) which is the last tail chunk first.

If without this, we could evict the first n chunks first, then the following chunks of that prompt would never be used and become to garbage.

Originally created by @maobaolong on GitHub (Jun 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/492 It would be better, if we support evict the key(it is chunk in lmcache which contains a part of kvcache of prompt) which is the last tail chunk first. If without this, we could evict the first n chunks first, then the following chunks of that prompt would never be used and become to garbage.

bsakaguchi

2026-10-29 21:10:28 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:29 +00:00

Author

Owner

@ykwd commented on GitHub (Jun 16, 2025):

Hi, @maobaolong , glad to hear from you. Could you be more specific about the proposal?

@ykwd commented on GitHub (Jun 16, 2025): Hi, @maobaolong , glad to hear from you. Could you be more specific about the proposal?

bsakaguchi commented

2026-10-29 21:10:29 +00:00

Author

Owner

@maobaolong commented on GitHub (Jul 12, 2025):

@ykwd Sorry the description is not complete before, I just finish it.

@maobaolong commented on GitHub (Jul 12, 2025): @ykwd Sorry the description is not complete before, I just finish it.

bsakaguchi commented

2026-10-29 21:10:29 +00:00

Author

Owner

@ykwd commented on GitHub (Jul 14, 2025):

Thanks for the suggestion! We're currently preparing a series of optimizations around allocation and eviction.

As for prefix-aware eviction, I'm not yet sure what the optimal strategy would be. For example, regarding your idea of evicting from the tail of a prefix — one possible scenario is that the head KV entries of a prefix may already be cached locally (on device or host), while the tail entries are actually more likely to hit Mooncake.

@ykwd commented on GitHub (Jul 14, 2025): Thanks for the suggestion! We're currently preparing a series of optimizations around allocation and eviction. As for prefix-aware eviction, I'm not yet sure what the optimal strategy would be. For example, regarding your idea of evicting from the tail of a prefix — one possible scenario is that the head KV entries of a prefix may already be cached locally (on device or host), while the tail entries are actually more likely to hit Mooncake.

bsakaguchi commented

2026-10-29 21:10:29 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 4, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 4, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:29 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 12, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 12, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:12:23 +00:00

[GH-ISSUE #952] [RFC] [Store] support Mooncake file eviction in DFS/3FS #308