[GH-ISSUE #452] Request for reproducible benchmark setup for section 5.3 (MOONCAKE Store) #157

New issue

Closed

opened 2026-10-29 21:10:16 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:10:16 +00:00

Owner

Originally created by @Jeffwan on GitHub (Jun 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/452

Thank you for the insightful work on MoonCake! I’m particularly interested in reproducing the experimental results described in mooncake paper https://www.usenix.org/system/files/fast25-qin.pdf Section 5.3 (MoonCake Store), especially the evaluations comparing global and local KVCache designs.

To ensure alignment and minimize discrepancies, could you please share more details about the setup used in the experiments? I see the traces have been public, besides that, if you have any scripts, configuration files, or internal benchmarks that could be shared, I’d greatly appreciate it.

My goal is to build a comparable evaluation to better understand the trade-offs of global caching in LLM inference. Thanks a lot!

Originally created by @Jeffwan on GitHub (Jun 5, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/452 Thank you for the insightful work on MoonCake! I’m particularly interested in reproducing the experimental results described in mooncake paper https://www.usenix.org/system/files/fast25-qin.pdf Section 5.3 (MoonCake Store), especially the evaluations comparing global and local KVCache designs. To ensure alignment and minimize discrepancies, could you please share more details about the setup used in the experiments? I see the traces have been public, besides that, if you have any scripts, configuration files, or internal benchmarks that could be shared, I’d greatly appreciate it. My goal is to build a comparable evaluation to better understand the trade-offs of global caching in LLM inference. Thanks a lot!

bsakaguchi

2026-10-29 21:10:16 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:17 +00:00

Author

Owner

@ae86zhizhi commented on GitHub (Jun 5, 2025):

I am also interested in building a comparable evaluation to better understand the trade-offs of global caching in LLM inference. In particular, I found the experiments in Section 5.3 very relevant to my current research. I would like to kindly ask if you could share more details about the experimental setup used in that section. Specifically, I am interested in understanding:

The number and bandwidth of network interface cards (NICs) on each prefill node
Whether the Mooncake store instances were deployed on the same machines as the prefill nodes
And, if possible, whether the experiment scripts used in your evaluation could be shared

@ae86zhizhi commented on GitHub (Jun 5, 2025): I am also interested in building a comparable evaluation to better understand the trade-offs of global caching in LLM inference. In particular, I found the experiments in Section 5.3 very relevant to my current research. I would like to kindly ask if you could share more details about the experimental setup used in that section. Specifically, I am interested in understanding: - The number and bandwidth of network interface cards (NICs) on each prefill node - Whether the Mooncake store instances were deployed on the same machines as the prefill nodes - And, if possible, whether the experiment scripts used in your evaluation could be shared

bsakaguchi commented

2026-10-29 21:10:17 +00:00

Author

Owner

@chestnut-Q commented on GitHub (Jun 6, 2025):

Hi @Jeffwan and @ae86zhizhi, thank you for your interest!

Unfortunately, I’m unable to provide an exact reproduction process, as the experimental environment—including Conductor, our in-house inference engine, and other components—is part of our internal production system and not publicly accessible.

That said, I’m happy to provide more context on the local-cache vs. global-cache experiments to help you build a comparable evaluation:

Both the local and global designs use the same underlying implementation. The key difference is that the local cache does not allow KVCache to be shared across nodes, whereas the global cache pool does.
The global cache follows the scheduler described in the paper. The local cache simply selects the instance with the longest prefix-cache match. However, under high load, it abandons the longest-match heuristic and switches to load balancing. This fallback mechanism is the main reason the local cache generally shows a lower hit rate compared to the global cache. PS. In our system, an instance’s prefill load is defined as: \sum exec time of queued requests / max queuing-time threshold (e.g., 30s).
Each node is equipped with 8 A800 GPUs and interconnected via 4 × 200 Gbps RDMA NICs.
"Whether the Mooncake store instances were deployed on the same machines as the prefill nodes": Yes.

Hope this provides some clarity.

@chestnut-Q commented on GitHub (Jun 6, 2025): Hi @Jeffwan and @ae86zhizhi, thank you for your interest! Unfortunately, I’m unable to provide an exact reproduction process, as the experimental environment—including Conductor, our in-house inference engine, and other components—is part of our internal production system and not publicly accessible. That said, I’m happy to provide more context on the local-cache vs. global-cache experiments to help you build a comparable evaluation: * Both the local and global designs use the same underlying implementation. The key difference is that the local cache **does not** allow KVCache to be shared across nodes, whereas the global cache pool **does**. * The global cache follows the scheduler described in the paper. The local cache simply selects the instance with the longest prefix-cache match. However, under high load, it abandons the longest-match heuristic and switches to load balancing. This fallback mechanism is the main reason the local cache generally shows a lower hit rate compared to the global cache. PS. In our system, an instance’s prefill load is defined as: $$\sum$$ exec time of queued requests / max queuing-time threshold (e.g., 30s). * Each node is equipped with 8 A800 GPUs and interconnected via 4 × 200 Gbps RDMA NICs. * "Whether the Mooncake store instances were deployed on the same machines as the prefill nodes": Yes. Hope this provides some clarity.

bsakaguchi commented

2026-10-29 21:10:17 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 8, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 8, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:17 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:46 +00:00

[PR #157] [MERGED] [FEATURE] catch exceptions during allocator creation #1361