[GH-ISSUE #380] [RFC][Store]Provide BatchPut() and BatchGet() interfaces for fine-grained KV Slices #135

New issue

Closed

opened 2026-10-29 21:09:57 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:09:57 +00:00

Owner

Originally created by @zhaoyongke on GitHub (May 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/380

Mooncake Store provides put()/get() interfaces with single key for now, it's ok for a couple of tokens' KV save/load.

While in vLLM and SGLang, an optimization is transfering KVs layer-by-layer for better compute-communication overlapping.

vllm offloading

sglang cache controller

However, when KVs split into layers, the value size will decreased sharply, and the number of requests increased accordingly.

To reduce the overhead of single key put()/get, we may provide batched python interfaces as:

def batch_put(key : list[str], value : list[byte]):
    ...
def batch_get(key : list[str]) -> list[byte]
    ...

To add batch_put(), batch_get() interfaces, some considerations are needed:

Should be called asyncly
Should provide detailed status for each key/value
Should update metrics correctly
(Optional)Better storage and network-bandwidth utilizing
(Optional)Auto-batching, keep higher-level api unchanged

Welcome to codesign & review & contribue & PR!

Originally created by @zhaoyongke on GitHub (May 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/380 Mooncake Store provides put()/get() interfaces with single key for now, it's ok for a couple of tokens' KV save/load. While in vLLM and SGLang, an optimization is transfering KVs layer-by-layer for better compute-communication overlapping. [vllm offloading](https://github.com/vllm-project/vllm/issues/15123) [sglang cache controller](https://github.com/sgl-project/sglang/blob/b146555749f84a684c7cf5e9d2950ca474b82de2/python/sglang/srt/managers/cache_controller.py#L312) However, when KVs split into layers, the value size will decreased sharply, and the number of requests increased accordingly. To reduce the overhead of single key put()/get, we may provide batched python interfaces as: ``` def batch_put(key : list[str], value : list[byte]): ... def batch_get(key : list[str]) -> list[byte] ... ``` To add batch_put(), batch_get() interfaces, some considerations are needed: - Should be called asyncly - Should provide detailed status for each key/value - Should update metrics correctly - (Optional)Better storage and network-bandwidth utilizing - (Optional)Auto-batching, keep higher-level api unchanged Welcome to codesign & review & contribue & PR!

bsakaguchi closed this issue

2026-10-29 21:09:57 +00:00

bsakaguchi commented

2026-10-29 21:09:57 +00:00

Author

Owner

@stmatengss commented on GitHub (May 20, 2025):

Layer-wise KVCache has the potential to reduce TTFT as the transfer operations can be pipelined with prefill. However, careful consideration of batch size is required: processing all layers in a single batch would revert to a block-based put/get interface pattern. As an extension, is it feasible to implement cross-request layer batching (i.e., grouping layers from different requests into batches)?

@stmatengss commented on GitHub (May 20, 2025): Layer-wise KVCache has the potential to reduce TTFT as the transfer operations can be pipelined with prefill. However, careful consideration of batch size is required: processing all layers in a single batch would revert to a block-based put/get interface pattern. As an extension, is it feasible to implement cross-request layer batching (i.e., grouping layers from different requests into batches)?

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 20, 2025):

Layer-wise KVCache has the potential to reduce TTFT as the transfer operations can be pipelined with prefill. However, careful consideration of batch size is required: processing all layers in a single batch would revert to a block-based put/get interface pattern. As an extension, is it feasible to implement cross-request layer batching (i.e., grouping layers from different requests into batches)?

At the first step, inference engines will choose a dynamic batch size as it needed, auto-batching in the mooncake store is preferred in the next step.

@zhaoyongke commented on GitHub (May 20, 2025): > Layer-wise KVCache has the potential to reduce TTFT as the transfer operations can be pipelined with prefill. However, careful consideration of batch size is required: processing all layers in a single batch would revert to a block-based put/get interface pattern. As an extension, is it feasible to implement cross-request layer batching (i.e., grouping layers from different requests into batches)? At the first step, inference engines will choose a dynamic batch size as it needed, auto-batching in the mooncake store is preferred in the next step.

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 21, 2025):

@xinranwang17

@zhaoyongke commented on GitHub (May 21, 2025): @xinranwang17

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@xinranwang17 commented on GitHub (May 21, 2025):