[GH-ISSUE #4020] [RFC]: Reproducing Table 1 hit rates on the released trace, consistent +4-6pp offset #1020

New issue

Closed

opened 2026-10-29 21:22:03 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:22:03 +00:00

Owner

Originally created by @gauravapiscean on GitHub (Sep 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4020

Originally assigned to: @chestnut-Q on GitHub.

Changes proposed

I've been reproducing your Table 1 cache hit rates on arxiv-trace/mooncake_trace.jsonl using LRU, and I get the same curve shape but a systematic offset at every capacity:

blocks	1k	10k	30k	50k	100k	infiniti
paper	0.30	0.40	0.48	0.50	0.51	0.51
mine	0.341	0.460	0.537	0.551	0.552	0.553

I compute hit rate as (sum of longest-resident-prefix blocks) / (sum of all blocks requested), replaying in timestamp order. Since the infinite-cache case is policy-free, I'd expect it to be reproducible from the trace alone, but I get 0.553 against your 0.51.

I've tried five denominator definitions (blocks vs input_length tokens, excluding the partial tail block, per-request averaging) and none closes the gap. Is the released trace the same one used for Table 1, or is there a detail in the hit-rate definition I'm missing?

Repro: https://github.com/gauravapiscean/agentic-kv-cache

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @gauravapiscean on GitHub (Sep 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4020 Originally assigned to: @chestnut-Q on GitHub. ### Changes proposed I've been reproducing your Table 1 cache hit rates on arxiv-trace/mooncake_trace.jsonl using LRU, and I get the same curve shape but a systematic offset at every capacity: | blocks | 1k | 10k | 30k | 50k | 100k | infiniti | |--------|-------|-------|-------|-------|-------|-------| | paper | 0.30 | 0.40 | 0.48 | 0.50 | 0.51 | 0.51 | | mine | 0.341 | 0.460 | 0.537 | 0.551 | 0.552 | 0.553 | I compute hit rate as (sum of longest-resident-prefix blocks) / (sum of all blocks requested), replaying in timestamp order. Since the infinite-cache case is policy-free, I'd expect it to be reproducible from the trace alone, but I get 0.553 against your 0.51. I've tried five denominator definitions (blocks vs input_length tokens, excluding the partial tail block, per-request averaging) and none closes the gap. Is the released trace the same one used for Table 1, or is there a detail in the hit-rate definition I'm missing? Repro: https://github.com/gauravapiscean/agentic-kv-cache ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:22:03 +00:00

closed this issue
added the
RFC
label

bsakaguchi commented

2026-10-29 21:22:04 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 10, 2026):

Thanks for opening this issue, @gauravapiscean!

Field	Value
Issue	#4020
GitHub user ID	`10956383`
Reporter	@gauravapiscean

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 10, 2026): Thanks for opening this issue, @gauravapiscean! | Field | Value | |-------|-------| | **Issue** | #4020 | | **GitHub user ID** | `10956383` | | **Reporter** | @gauravapiscean | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:22:04 +00:00

Author

Owner

@chestnut-Q commented on GitHub (Oct 8, 2026):

Hi @gauravapiscean, thank you for pointing this. When preparing the trace for release, we filtered out requests marked as rejected, as their output lengths are 0, which could be confusing. However, the results in Table 1 were computed on the trace before filtering, which explains the gap.

I’ll update the paper to correct this inconsistency as soon as possible. Thanks!

@chestnut-Q commented on GitHub (Oct 8, 2026): Hi @gauravapiscean, thank you for pointing this. When preparing the trace for release, we filtered out requests marked as `rejected`, as their output lengths are 0, which could be confusing. However, the results in Table 1 were computed on the trace **before filtering**, which explains the gap. I’ll update the paper to correct this inconsistency as soon as possible. Thanks!

bsakaguchi referenced this issue

2026-10-29 21:31:41 +00:00

[PR #1020] [MERGED] [feat]More KVCache metrics in both master/client side #1958