kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3317] [Usage]: Drop‑off in Cache Hit Rate #840

New issue

Closed

opened 2026-10-29 21:20:04 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:20:04 +00:00

Owner

Originally created by @Yong-ee on GitHub (Aug 6, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3317

Describe your usage question

I have been running model inference with a HiCache‑L1 + L2 + Mooncake KV cache storage path backed by SSD. During benchmarking, I encountered the same issue reported by the official team in this article: https://www.kvcache.ai/blog/scaling-kv-cache-beyond-memory/?f_link_type=f_linkinlinenote&flow_extra=eyJpbmxpbmVfZGlzcGxheV9wb3NpdGlvbiI6MCwiZG9jX3Bvc2l0aW9uIjowLCJkb2NfaWQiOiI2ZDIyZjczYTQ3ZWIyNWU1LTBiYTRlODZmMmNkYWQ2YzUifQ%3D%3D.

The cache hit rate drops in the final rounds compared with previous rounds：
Round 0: Average TTFT = 2.53s, Cache Hit Rate = 0.000000 (20 requests, 20 clients)
Round 1: Average TTFT = 3.05s, Cache Hit Rate = 0.499939 (20 requests, 20 clients)
Round 2: Average TTFT = 3.41s, Cache Hit Rate = 0.666558 (20 requests, 20 clients)
Round 3: Average TTFT = 3.82s, Cache Hit Rate = 0.749863 (20 requests, 20 clients)
Round 4: Average TTFT = 4.22s, Cache Hit Rate = 0.799844 (20 requests, 20 clients)
Round 5: Average TTFT = 4.59s, Cache Hit Rate = 0.833164 (20 requests, 20 clients)
Round 6: Average TTFT = 4.85s, Cache Hit Rate = 0.856964 (20 requests, 20 clients)
Round 7: Average TTFT = 5.33s, Cache Hit Rate = 0.874813 (20 requests, 20 clients)
Round 8: Average TTFT = 7.41s, Cache Hit Rate = 0.844261 (20 requests, 20 clients)

My understanding is that if all KV cache is offloaded to SSD, the hit rate should not degrade. Could anyone shed light on why this phenomenon occurs?
Thank you for your guidance.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Yong-ee on GitHub (Aug 6, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3317 ### Describe your usage question I have been running model inference with a HiCache‑L1 + L2 + Mooncake KV cache storage path backed by SSD. During benchmarking, I encountered the same issue reported by the official team in this article: [https://www.kvcache.ai/blog/scaling-kv-cache-beyond-memory/?f_link_type=f_linkinlinenote&flow_extra=eyJpbmxpbmVfZGlzcGxheV9wb3NpdGlvbiI6MCwiZG9jX3Bvc2l0aW9uIjowLCJkb2NfaWQiOiI2ZDIyZjczYTQ3ZWIyNWU1LTBiYTRlODZmMmNkYWQ2YzUifQ%3D%3D](https://link.wtturl.cn/?target=https%3A%2F%2Fwww.kvcache.ai%2Fblog%2Fscaling-kv-cache-beyond-memory%2F%3Ff_link_type%3Df_linkinlinenote%26flow_extra%3DeyJpbmxpbmVfZGlzcGxheV9wb3NpdGlvbiI6MCwiZG9jX3Bvc2l0aW9uIjowLCJkb2NfaWQiOiI2ZDIyZjczYTQ3ZWIyNWU1LTBiYTRlODZmMmNkYWQ2YzUifQ%253D%253D&scene=im&aid=497858&lang=zh). <img width="826" height="534" alt="Image" src="https://github.com/user-attachments/assets/9c0f2f98-8915-4852-9351-6845ff51e0ba" /> The cache hit rate drops in the final rounds compared with previous rounds： Round 0: Average TTFT = 2.53s, Cache Hit Rate = 0.000000 (20 requests, 20 clients) Round 1: Average TTFT = 3.05s, Cache Hit Rate = 0.499939 (20 requests, 20 clients) Round 2: Average TTFT = 3.41s, Cache Hit Rate = 0.666558 (20 requests, 20 clients) Round 3: Average TTFT = 3.82s, Cache Hit Rate = 0.749863 (20 requests, 20 clients) Round 4: Average TTFT = 4.22s, Cache Hit Rate = 0.799844 (20 requests, 20 clients) Round 5: Average TTFT = 4.59s, Cache Hit Rate = 0.833164 (20 requests, 20 clients) Round 6: Average TTFT = 4.85s, Cache Hit Rate = 0.856964 (20 requests, 20 clients) Round 7: Average TTFT = 5.33s, Cache Hit Rate = 0.874813 (20 requests, 20 clients) Round 8: Average TTFT = 7.41s, Cache Hit Rate = 0.844261 (20 requests, 20 clients) My understanding is that if all KV cache is offloaded to SSD, the hit rate should not degrade. Could anyone shed light on why this phenomenon occurs? Thank you for your guidance. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:20:04 +00:00

bsakaguchi commented

2026-10-29 21:20:06 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 6, 2026):

Thanks for opening this issue, @Yong-ee!

Field	Value
Issue	#3317
GitHub user ID	`15357666`
Reporter	@Yong-ee

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 6, 2026): Thanks for opening this issue, @Yong-ee! | Field | Value | |-------|-------| | **Issue** | #3317 | | **GitHub user ID** | `15357666` | | **Reporter** | @Yong-ee | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:20:06 +00:00

Author

Owner

@ykwd commented on GitHub (Aug 7, 2026):

This is a feature of the multi-turn benchmark. In each turn, all of the prefix cache from previous turns is reused, so the lifetime of all KV cache entries extends until the end of the benchmark. The main purpose of this benchmark is to make it easy to compare the performance difference between KV cache hits and misses.

In real-world production workloads, however, KV cache entries have a limited lifetime. You can refer to the analysis in the first section of this blog for more details.

@ykwd commented on GitHub (Aug 7, 2026): This is a feature of the multi-turn benchmark. In each turn, all of the prefix cache from previous turns is reused, so the lifetime of all KV cache entries extends until the end of the benchmark. The main purpose of this benchmark is to make it easy to compare the performance difference between KV cache hits and misses. In real-world production workloads, however, KV cache entries have a limited lifetime. You can refer to the analysis in the first section of this blog for more details.

bsakaguchi commented

2026-10-29 21:20:06 +00:00

Author

Owner

@Yong-ee commented on GitHub (Aug 7, 2026):

This is a feature of the multi-turn benchmark. In each turn, all of the prefix cache from previous turns is reused, so the lifetime of all KV cache entries extends until the end of the benchmark. The main purpose of this benchmark is to make it easy to compare the performance difference between KV cache hits and misses.

In real-world production workloads, however, KV cache entries have a limited lifetime. You can refer to the analysis in the first section of this blog for more details.

Thank you for your reply. My observations are different. I noticed that when the hicache ratio is set to 2, there is no drop‑off point in the cache hit rate. However, when the ratio is 1.6, a drop‑off occurs at the penultimate turn, and when the ratio is 1, the drop‑off appears at the 3rd turn. This behavior seems counter‑intuitive, and I would like to understand how this relates to the L2 cache size.

@Yong-ee commented on GitHub (Aug 7, 2026): > This is a feature of the multi-turn benchmark. In each turn, all of the prefix cache from previous turns is reused, so the lifetime of all KV cache entries extends until the end of the benchmark. The main purpose of this benchmark is to make it easy to compare the performance difference between KV cache hits and misses. > > In real-world production workloads, however, KV cache entries have a limited lifetime. You can refer to the analysis in the first section of this blog for more details. Thank you for your reply. My observations are different. I noticed that when the hicache ratio is set to 2, there is no drop‑off point in the cache hit rate. However, when the ratio is 1.6, a drop‑off occurs at the penultimate turn, and when the ratio is 1, the drop‑off appears at the 3rd turn. This behavior seems counter‑intuitive, and I would like to understand how this relates to the L2 cache size.

bsakaguchi commented

2026-10-29 21:20:06 +00:00

Author

Owner

@Yong-ee commented on GitHub (Aug 10, 2026):

I've found the root cause. It's caused by the SGLang framework and has nothing to do with Mooncake.

@Yong-ee commented on GitHub (Aug 10, 2026): I've found the root cause. It's caused by the SGLang framework and has nothing to do with Mooncake.

bsakaguchi referenced this issue

2026-10-29 21:30:56 +00:00

[PR #840] [MERGED] refactor(store): use dedicated thread for signal handling #1862