kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4941] [RFC]: Range reads in Mooncake Store Get — transfer only the shared span of a KV object #1256

New issue

Open

opened 2026-10-29 21:24:29 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:24:29 +00:00

Owner

Originally created by @huangxiaobai-ydmy on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4941

Originally assigned to: @Aionw on GitHub.

Summary

This RFC proposes optional byte-range parameters for Mooncake Store's client-side Get/BatchGet read path, so a reader can transfer only the span of a pooled KV object it actually shares, instead of the whole object. No master-side wire or metadata change is required for v1; objects stay format-opaque; lease semantics are unchanged.

Baseline for triage

Note: this is an RFC / design proposal, not a bug report. The fields below map the issue-bot checklist onto the API discussion.

Field	Value
Mooncake version	Proposal based on `main` at `7d0ff6c65c34` (read path audited: `GetReplicaList` → `allocateSlices` / transfer submit; no master wire change in v1).
Environment (for planned benches)	Ascend 910B + RDMA (Ascend Direct / TE). Discussion itself is code-path / API shape and does not depend on a specific cluster.
Current vs proposed behavior	Today: `Get`/`BatchGet` always transfer `[0, object_size)`. Proposed: optional `(offset, length)` ranges; omit ranges → same as today (backward compatible). Concrete waste case: 10-block object (1280 tokens), shared prefix 5 blocks (640) → transfer 1280 today, 640 with ranges.

Motivation

Get is whole-object today: the read path resolves GetReplicaList(key, tenant_id) and reads from byte 0 to object_size (allocateSlices in client_buffer.cpp covers the full buffer for memory replicas and the full file for disk replicas).

For prefix-cache consumers this is wasteful whenever the hit is partial. Concrete example: a request pools its KV as a 10-block object (1280 tokens, block size 128); the next request shares the first 5 blocks. The radix layer reports a 640-token hit, but the reader still transfers all 1280 tokens — the extra 50% of RDMA bandwidth, staging buffer, and host-to-device copy is pure waste. The longer the pooled prefix and the shorter the shared span, the worse the ratio. This is the same retrieval-span mismatch we worked through in llm-d/llm-d-router#2899.

Current integrations work around it by choosing fine-grained objects (vLLM keys KV in per-block batches, SGLang HiCache offloads per radix node). That avoids partial-read waste but pays a different cost: object-count amplification in master metadata, many small RPCs, and staging pressure on batched loads (vllm-project/vllm#63465 shows a split load over fine-grained keys exhausting the staging budget). Today granularity is a trade-off: coarse objects waste bandwidth on partial hits; fine objects bloat metadata and RPC counts. A range read dissolves the trade-off — clients keep coarse objects (fewer keys, fewer RPCs, larger sequential transfers) and still pay only for the shared span.

#380 (BatchGet) batches independent objects and does not address ranges within one object; the two are orthogonal.

Existing foundation (why this is cheap now)

The design is already range-friendly at every layer except the API:

Metadata: ObjectPart (object_part.h) is "one byte range of an object's content, written independently; an object's content is an ordered sequence of parts; each part carries its own length and checksum". Every object has one part today, so v1 needs no multi-part resolution — and the abstraction a range read composes with already exists.
Addressing: the replica descriptor returned to readers is buffer_address_ + size_ (AllocatedBuffer::Descriptor, allocator.h). The transport can address any sub-range of that buffer as-is — no master-side wire change for v1.
Consistency: objects are immutable after PutEnd; the read lease binds the whole object and stays unchanged. A ranged read adds no new consistency surface relative to a full read.
Choke point: allocateSlices() plus the transfer submit loop are the only places that apply an (offset, length) — offset arithmetic, not structural change.

Proposed changes

Add optional byte ranges to the client read API: Get(key, ranges) / BatchGet(keys, ranges_per_key); each range is (offset, length) in bytes within the object. The store stays format-opaque; converting a radix/token span to byte ranges is the client's job (it owns the layout).
A list of ranges, not one range: KV blobs are commonly layer-major, so "tokens 0–640, all layers" is L byte-discontiguous ranges, one per layer. Each range maps onto one transfer task in the existing batch-submit model.
Ranges are clamped to object size; omitting ranges falls back to today's full-object read — fully backward compatible.
Checksum verification for ranged reads is per covered part (with one part today, that is the object checksum).
Local-disk and DFS replica read paths get the same offset arithmetic.
Lease, eviction, and replica selection unchanged; the lease continues to cover the whole object during the read.
Suggested observability: ranged-Get share, average sum(range)/object_size (this ratio is the observed bandwidth saving), fallback-to-full-read count.

Non-goals for v1

No master metadata or RPC changes.
No multi-part objects (the parts_ container is the future evolution path; ranges and multi-part compose naturally, but v1 needs neither).
No block/token semantics inside the store.

Relationship to existing work

#380 (BatchPut / BatchGet) batches independent objects. Ranges here are within a single object; the two are orthogonal and compose naturally.
#1732 (prefix query in the store master) and #1403 (KV-store indexer API standardization) concern how a prefix hit is found. This RFC does not touch indexer APIs: the hit span (from Conductor / HiCache radix) is an upstream signal that the client converts into byte ranges.
#3477 (Store RPC correctness, compatibility, and execution model) argues for RPC-level stability. v1 of this proposal deliberately changes no RPC signature — ranges are a client-side read-path parameter — which keeps it consistent with that direction.
#3553 (Prometheus metrics standardization): the suggested observability counters (ranged-Get share, average range/object ratio, fallback count) would follow that framework rather than introduce ad-hoc gauges.

Alternatives considered

Single range + token-major layout contract on clients: simpler API, but forces engine-side layout changes; rejected for v1 in favor of multi-range.
Separate GetRange RPC: keeps the existing signature untouched but forks the read path; extending Get with an optional argument keeps one code path.
Status quo (fine-grained objects): already costs metadata amplification and staging pressure; the referenced issues are evidence the community is paying this cost today.

Test plan

Unit tests for range slicing/clamping/verification per replica type (memory, local-disk, DFS).
Benchmarks on an Ascend 910B server (Ascend + RDMA): synthetic prefix traffic with zipf-shaped share lengths, reporting transferred bytes, staging peak, and Get wall time for full-object vs ranged reads at several granularity settings; engine-level measurements on the Ascend backend if useful for the discussion.

Open questions

Multi-range acceptable for v1, or single-range plus a token-major layout contract?
Any concern with per-part checksum verification for partial reads?
Would the HiCache / vLLM integrations move to coarser objects once range reads land?
API shape: extend Get/BatchGet with an optional argument, or a separate GetRange RPC?

Context

Motivated by the retrieval-span work in llm-d/llm-d-router#2899 and by the split-load staging behavior in vllm-project/vllm#63465. I have a concrete implementation sketch in mind (optional ranges in the client API, applied at allocateSlices() and the transfer submit loop, master-side wire untouched for v1) and would be glad to send a PR with unit tests and benchmark numbers once the API shape is agreed.

Originally created by @huangxiaobai-ydmy on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4941 Originally assigned to: @Aionw on GitHub. ### Summary This RFC proposes optional byte-range parameters for Mooncake Store's client-side `Get`/`BatchGet` read path, so a reader can transfer only the span of a pooled KV object it actually shares, instead of the whole object. No master-side wire or metadata change is required for v1; objects stay format-opaque; lease semantics are unchanged. ### Baseline for triage > Note: this is an **RFC / design proposal**, not a bug report. The fields below map the issue-bot checklist onto the API discussion. | Field | Value | |-------|-------| | **Mooncake version** | Proposal based on `main` at `7d0ff6c65c34` (read path audited: `GetReplicaList` → `allocateSlices` / transfer submit; no master wire change in v1). | | **Environment (for planned benches)** | Ascend 910B + RDMA (Ascend Direct / TE). Discussion itself is code-path / API shape and does not depend on a specific cluster. | | **Current vs proposed behavior** | **Today:** `Get`/`BatchGet` always transfer `[0, object_size)`. **Proposed:** optional `(offset, length)` ranges; omit ranges → same as today (backward compatible). Concrete waste case: 10-block object (1280 tokens), shared prefix 5 blocks (640) → transfer 1280 today, 640 with ranges. | ### Motivation `Get` is whole-object today: the read path resolves `GetReplicaList(key, tenant_id)` and reads from byte 0 to `object_size` (`allocateSlices` in `client_buffer.cpp` covers the full buffer for memory replicas and the full file for disk replicas). For prefix-cache consumers this is wasteful whenever the hit is partial. Concrete example: a request pools its KV as a 10-block object (1280 tokens, block size 128); the next request shares the first 5 blocks. The radix layer reports a 640-token hit, but the reader still transfers all 1280 tokens — the extra 50% of RDMA bandwidth, staging buffer, and host-to-device copy is pure waste. The longer the pooled prefix and the shorter the shared span, the worse the ratio. This is the same retrieval-span mismatch we worked through in llm-d/llm-d-router#2899. Current integrations work around it by choosing fine-grained objects (vLLM keys KV in per-block batches, SGLang HiCache offloads per radix node). That avoids partial-read waste but pays a different cost: object-count amplification in master metadata, many small RPCs, and staging pressure on batched loads (vllm-project/vllm#63465 shows a split load over fine-grained keys exhausting the staging budget). Today granularity is a trade-off: coarse objects waste bandwidth on partial hits; fine objects bloat metadata and RPC counts. A range read dissolves the trade-off — clients keep coarse objects (fewer keys, fewer RPCs, larger sequential transfers) and still pay only for the shared span. #380 (`BatchGet`) batches independent objects and does not address ranges within one object; the two are orthogonal. ### Existing foundation (why this is cheap now) The design is already range-friendly at every layer except the API: 1. **Metadata**: `ObjectPart` (`object_part.h`) is "one byte range of an object's content, written independently; an object's content is an ordered sequence of parts; each part carries its own length and checksum". Every object has one part today, so v1 needs no multi-part resolution — and the abstraction a range read composes with already exists. 2. **Addressing**: the replica descriptor returned to readers is `buffer_address_ + size_` (`AllocatedBuffer::Descriptor`, `allocator.h`). The transport can address any sub-range of that buffer as-is — no master-side wire change for v1. 3. **Consistency**: objects are immutable after `PutEnd`; the read lease binds the whole object and stays unchanged. A ranged read adds no new consistency surface relative to a full read. 4. **Choke point**: `allocateSlices()` plus the transfer submit loop are the only places that apply an `(offset, length)` — offset arithmetic, not structural change. ### Proposed changes - Add optional byte ranges to the client read API: `Get(key, ranges)` / `BatchGet(keys, ranges_per_key)`; each range is `(offset, length)` in bytes within the object. The store stays format-opaque; converting a radix/token span to byte ranges is the client's job (it owns the layout). - **A list of ranges, not one range**: KV blobs are commonly layer-major, so "tokens 0–640, all layers" is L byte-discontiguous ranges, one per layer. Each range maps onto one transfer task in the existing batch-submit model. - Ranges are clamped to object size; omitting ranges falls back to today's full-object read — fully backward compatible. - Checksum verification for ranged reads is per covered part (with one part today, that is the object checksum). - Local-disk and DFS replica read paths get the same offset arithmetic. - Lease, eviction, and replica selection unchanged; the lease continues to cover the whole object during the read. - Suggested observability: ranged-Get share, average `sum(range)/object_size` (this ratio is the observed bandwidth saving), fallback-to-full-read count. ### Non-goals for v1 - No master metadata or RPC changes. - No multi-part objects (the `parts_` container is the future evolution path; ranges and multi-part compose naturally, but v1 needs neither). - No block/token semantics inside the store. ### Relationship to existing work - #380 (`BatchPut` / `BatchGet`) batches independent objects. Ranges here are within a single object; the two are orthogonal and compose naturally. - #1732 (prefix query in the store master) and #1403 (KV-store indexer API standardization) concern how a prefix hit is *found*. This RFC does not touch indexer APIs: the hit span (from Conductor / HiCache radix) is an upstream signal that the client converts into byte ranges. - #3477 (Store RPC correctness, compatibility, and execution model) argues for RPC-level stability. v1 of this proposal deliberately changes no RPC signature — ranges are a client-side read-path parameter — which keeps it consistent with that direction. - #3553 (Prometheus metrics standardization): the suggested observability counters (ranged-Get share, average range/object ratio, fallback count) would follow that framework rather than introduce ad-hoc gauges. ### Alternatives considered - **Single range + token-major layout contract on clients**: simpler API, but forces engine-side layout changes; rejected for v1 in favor of multi-range. - **Separate `GetRange` RPC**: keeps the existing signature untouched but forks the read path; extending `Get` with an optional argument keeps one code path. - **Status quo (fine-grained objects)**: already costs metadata amplification and staging pressure; the referenced issues are evidence the community is paying this cost today. ### Test plan - Unit tests for range slicing/clamping/verification per replica type (memory, local-disk, DFS). - Benchmarks on an Ascend 910B server (Ascend + RDMA): synthetic prefix traffic with zipf-shaped share lengths, reporting transferred bytes, staging peak, and Get wall time for full-object vs ranged reads at several granularity settings; engine-level measurements on the Ascend backend if useful for the discussion. ### Open questions 1. Multi-range acceptable for v1, or single-range plus a token-major layout contract? 2. Any concern with per-part checksum verification for partial reads? 3. Would the HiCache / vLLM integrations move to coarser objects once range reads land? 4. API shape: extend `Get`/`BatchGet` with an optional argument, or a separate `GetRange` RPC? ### Context Motivated by the retrieval-span work in llm-d/llm-d-router#2899 and by the split-load staging behavior in vllm-project/vllm#63465. I have a concrete implementation sketch in mind (optional ranges in the client API, applied at `allocateSlices()` and the transfer submit loop, master-side wire untouched for v1) and would be glad to send a PR with unit tests and benchmark numbers once the API shape is agreed.

bsakaguchi commented

2026-10-29 21:24:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 26, 2026):

Thanks for opening this issue, @huangxiaobai-ydmy!

Field	Value
Issue	#4941
GitHub user ID	`207993101`
Reporter	@huangxiaobai-ydmy

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 26, 2026): Thanks for opening this issue, @huangxiaobai-ydmy! | Field | Value | |-------|-------| | **Issue** | #4941 | | **GitHub user ID** | `207993101` | | **Reporter** | @huangxiaobai-ydmy | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:30 +00:00

Author

Owner

@thrower01 commented on GitHub (Oct 27, 2026):

Thanks for the clear write-up. A few quick points I want to confirm before I dig deeper:

Checksum on partial reads — You mention per-part checksum verification, but with a single part today (the whole object), how does this work for a range read? Do we recompute checksum over the covered sub-range, or is it a full-part checksum that only validates if the entire part was read?
Staging buffer savings for memory replicas — When we say "transfer only the sub-range", does that mean RDMA reads directly from allocator_buffer_base + offset (no extra staging allocation), or does it still stage into a smaller intermediate buffer? The former is where the real bandwidth + staging win lives, just want to make sure that's the intended path.

Excited to see this land.

@thrower01 commented on GitHub (Oct 27, 2026): Thanks for the clear write-up. A few quick points I want to confirm before I dig deeper: 1. **Checksum on partial reads** — You mention per-part checksum verification, but with a single part today (the whole object), how does this work for a range read? Do we recompute checksum over the covered sub-range, or is it a full-part checksum that only validates if the entire part was read? 2. **Staging buffer savings for memory replicas** — When we say "transfer only the sub-range", does that mean RDMA reads directly from `allocator_buffer_base + offset` (no extra staging allocation), or does it still stage into a smaller intermediate buffer? The former is where the real bandwidth + staging win lives, just want to make sure that's the intended path. Excited to see this land.

bsakaguchi commented

2026-10-29 21:24:30 +00:00

Author

Owner

@huangxiaobai-ydmy commented on GitHub (Oct 28, 2026):

Thanks for the questions — clarifying both:

1. Checksum on partial reads

Today every object is a single ObjectPart, and the stored checksum is over that entire part. A ranged Get that only transfers a sub-span cannot recompute or match that full-part checksum from the bytes it received.

So for v1 the rule is: verify a part's checksum only when the requested ranges fully cover that part. With one part today, that means:

ranges omit / cover [0, object_size) → same full-object checksum path as today
true partial ranges → skip part checksum for that read (lease + transport integrity still apply; we do not invent a sub-range checksum that is not stored)

When multi-part objects land later, the same rule composes: verify each fully covered part, skip partially covered ones. Happy to call this out more explicitly in the RFC text if useful.

2. Staging for memory replicas

Intended path is the former: RDMA (or the local memcpy path) addresses buffer_address_ + offset with length equal to the range — i.e. allocateSlices() / transfer submit emit sub-range tasks against the replica descriptor as-is, without allocating a full-object staging buffer and without an intermediate “copy whole object then slice” step.

A smaller destination buffer sized to sum(ranges) (or the caller's provided buffer) is fine; what we want to avoid is staging/Copy* proportional to object_size when only a prefix/span is needed. Disk/DFS replica paths get the same offset arithmetic into their read windows.

Happy to dig into either point further if anything is still ambiguous.

@huangxiaobai-ydmy commented on GitHub (Oct 28, 2026): Thanks for the questions — clarifying both: **1. Checksum on partial reads** Today every object is a single `ObjectPart`, and the stored checksum is over that **entire** part. A ranged Get that only transfers a sub-span cannot recompute or match that full-part checksum from the bytes it received. So for v1 the rule is: verify a part's checksum **only when the requested ranges fully cover that part**. With one part today, that means: - ranges omit / cover `[0, object_size)` → same full-object checksum path as today - true partial ranges → **skip** part checksum for that read (lease + transport integrity still apply; we do **not** invent a sub-range checksum that is not stored) When multi-part objects land later, the same rule composes: verify each fully covered part, skip partially covered ones. Happy to call this out more explicitly in the RFC text if useful. **2. Staging for memory replicas** Intended path is the former: RDMA (or the local memcpy path) addresses `buffer_address_ + offset` with `length` equal to the range — i.e. `allocateSlices()` / transfer submit emit sub-range tasks against the replica descriptor as-is, **without** allocating a full-object staging buffer and without an intermediate “copy whole object then slice” step. A smaller destination buffer sized to `sum(ranges)` (or the caller's provided buffer) is fine; what we want to avoid is staging/`Copy*` proportional to `object_size` when only a prefix/span is needed. Disk/DFS replica paths get the same offset arithmetic into their read windows. Happy to dig into either point further if anything is still ambiguous.

bsakaguchi referenced this issue

2026-10-29 21:32:52 +00:00

[PR #1256] [MERGED] [CI] Disable EP's test_mooncake_backend_p2p_cpu in CI workflow #2111