kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4766] [RFC]: Pipeline SSD restore sub-batches with DRAM-to-HBM transfers #1215

New issue

Closed

opened 2026-10-29 21:24:02 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:02 +00:00

Owner

Originally created by @ckeyever on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4766

Changes proposed

Summary

Add an opt-in, bounded sub-batch pipeline to Mooncake Store's LOCAL_DISK restore path. While the SSD owner reads sub-batch B(i+1) into its DRAM staging arena, the requesting client transfers the already loaded sub-batch B(i) into its final destination buffers, including accelerator HBM.

The first implementation operates on complete objects and reuses the existing offload-read and buffer-release RPCs. It does not introduce intra-object chunking or a streaming RPC protocol. The initial target workload is batched KV-cache restoration on Ascend; scheduling should remain transport-independent.

This is a design proposal. No performance improvement has been measured yet.

Motivation and current behavior

The current restore path contains a barrier between SSD loading and destination transfer:

The requester selects LOCAL_DISK replicas and groups reads by the SSD owner's RPC endpoint.
The owner allocates DRAM buffers and calls FileStorage::BatchGet / LoadBatch.
LoadBatch waits for the complete batch's storage-backend BatchLoad before returning buffer addresses.
The requester calls Client::BatchGetOffloadObject, which submits Transfer Engine READs and waits for the TransferFuture.
The requester releases the owner's offload buffers.

Relevant source at the inspected baseline:

The batch barrier prevents transfers for ready objects from overlapping the remaining objects' disk reads within that request. This proposal targets end-to-end SSD-hit restoration latency, measured until all requested destination data is usable.

Scope

In scope:

Whole-object sub-batches for batched LOCAL_DISK restores.
Overlap between SSD-to-DRAM loading and DRAM-to-destination transfer.
Bounded staging occupancy, completion-driven release, and explicit failure handling.
Existing same-process and remote-owner transfer strategies.
Unchanged public result ordering, per-key error behavior, and checksum semantics.

Deferred:

Splitting one object into disk-read chunks.
Per-chunk READY notifications or BeginRestore/PollReady streaming protocols.
Changing the on-disk format, replica selection, SSD promotion policy, or master scheduling.
Extending the new pipeline to NOF_SSD, DFS, and other disk-replica paths.
Automatically consuming partially restored KV objects from the model execution stream.

A request with a single object has no intra-object overlap benefit in this version.

Proposed design

1. Plan whole-object sub-batches

After the existing replica query and selection, group LOCAL_DISK objects by owner endpoint. Partition each owner group using both a target byte size and a maximum object count.

Each sub-batch contains full objects with their original result indices, destination slices, selected-replica metadata, sizes, and expected checksums. Preserve the caller's current duplicate-key behavior and restore results to their original positions.

Use byte-based partitioning because KV object sizes vary. An object larger than the target size is handled as a standalone sub-batch; account for its actual allocation size. If it exceeds the pipeline admission budget, execute the existing path exclusively after draining pipeline work, rather than rejecting a previously valid request.

Start with two sub-batches in the lifecycle window: one being loaded and one being transferred. Increase the window only after measurement. Keep useful batching within each SSD request so that the change does not collapse NVMe queue depth or turn small objects into one-RPC-per-key requests.

2. Run independent load and transfer stages

Use shared executors/coroutines rather than creating threads for every request.

Load stage:

Reserve bounded staging and lifecycle-window credits before issuing an offload read.
Use the existing owner RPC, or the existing same-process load path.
Once that sub-batch is fully loaded, publish its returned buffers to a bounded READY queue.
Continue with the next eligible sub-batch while transfers run independently.

Transfer stage:

Consume READY sub-batches and submit the existing Transfer Engine READ operation or same-process copy strategy.
Retain the resulting Future and buffer owner in an in-flight context.
Observe terminal completion independently of the load stage.
Run the existing checksum verification at the appropriate completion boundary, release the owner buffer, and return credits.

Conceptual steady state:

Interval	SSD load stage	Destination-transfer stage
0	Load B0	Idle
1	Load B1	Transfer B0
2	Load B2	Transfer B1
3	Load B3	Transfer B2
4	Idle	Transfer B3

The table assumes equal stage durations for illustration; the actual scheduler is completion-driven.

A dedicated transfer worker may initially use the existing blocking transfer wrapper while the independent load worker continues. Exposing the existing submitter's TransferFuture through a nonblocking Client helper can subsequently support multiple in-flight transfers. Asynchronous execution in every low-level API is not a prerequisite for the two stages to overlap.

3. Bound memory and scheduling

Track the full lifecycle:

LOADING -> READY -> TRANSFERRING -> VERIFYING -> RELEASED
                   \-> FAILED/DRAINING -> RELEASED only after safe completion

Admission counts all allocated or reserved bytes, including reads in progress, READY buffers, alignment overhead, and transfers/checks still holding buffers.

Use:

A small per-request lifecycle window.
A per-owner limit shared across that request's sub-batches.
An owner-side process-wide budget across requesting clients, using the existing allocator capacity as a hard bound.
Fair scheduling so one large restore does not monopolize an SSD owner or transfer endpoint.

The bounded READY queue is essential: when destination transfers slow down, loading must stop before unbounded DRAM accumulation. Requester-local accounting alone is insufficient to bound concurrent remote clients.

4. Preserve ownership and lease safety

The current synchronous ReleaseGuard must not release buffers immediately after a new asynchronous submit helper returns. Ownership moves to the in-flight operation context and lasts through confirmed transfer termination and any verification that still accesses the staging buffer.

For the same-process path, retain the LocalBatchResult owner. For the remote path, retain the returned batch_id and issue the existing release RPC only after the source is no longer accessed.

The owner's existing TTL-based GC is a specific issue to settle before enabling this feature: prefetching introduces additional residence time before transfer. Track deadlines conservatively from before each load RPC, reserve transfer capacity before prefetching, and stop admitting work that cannot meet its budget. These checks reduce exposure but do not prove that a timed-out DMA has stopped.

The required safety contract is:

Owner GC must not recycle memory while a transfer can still access it.
An expiry or cancellation enters a draining/quarantine state until outstanding I/O/transfer access is confirmed quiescent.
A reported timeout alone is not confirmation of quiescence.
Release and error cleanup must tolerate duplicate notifications without double freeing.

Prefer a minimal integration with existing buffer ownership and release handling. If safe lifetime handling requires an owner capability or lease extension, include that small change and gate the opt-in path accordingly; it does not require a streaming data protocol. Mixed-version deployments must retain the existing path unless the required owner behavior is known to be available.

5. Preserve API and failure semantics

Keep synchronous public batch APIs synchronous: return only when their existing completion conditions are satisfied. Internally, individual objects can finish earlier.

Map load, transfer, and checksum errors back to the affected keys. Do not report a partially written destination as a successful object. Preserve the current behavior for independent successful keys.

On cancellation or fatal request failure, stop new loads, drain/cancel outstanding operations using supported transport semantics, release buffers after quiescence, and retain the caller's destination-buffer lifetime until those accesses have ended. Retries must follow the existing replica/version rules; do not overwrite a destination while an earlier attempt can still write to it.

Implementation touch points

Component	Expected change
RealClient LOCAL_DISK batch restore orchestration	Partition owner groups, schedule sub-batches, preserve original result mapping.
Client / TransferSubmitter	Reuse the existing transfer strategy; optionally expose a Future-returning offload-transfer helper.
FileStorage buffer lifecycle	Reuse whole-object BatchGet/BatchGetLocal; integrate bounded ownership and safe lease/GC handling.
StorageBackendInterface / on-disk format	No new range-read API or format changes in this version.
Ascend transport	Reuse the existing path initially; measure local-copy synchronization and executor capacity before changing it.

Existing Ascend constraints remain applicable: buffer-pool mode and the async executor cannot simply be enabled together, and the optional pinned restore arena is disabled with io_uring in the inspected code. Enabling an async environment variable alone does not remove the Store batch barrier. Reuse the existing protocol-aware registered staging allocator and validate the intended deployment's memory/transfer compatibility.

Tentative opt-in tuning controls, with final names to follow maintainer guidance:

Enable/disable pipelined LOCAL_DISK restore; disabled by default initially.
Target bytes and maximum keys per sub-batch.
Maximum lifecycle window and maximum transfer concurrency.
Per-request and owner-wide staging-byte limits.

Initial experiments should sweep 1-4 MiB target sub-batches and windows of 2 and 4. These are experimental settings, not selected production defaults.

Expected benefit and trade-offs

For N equal sub-batches with effective load time tr and transfer time tx, an ideal two-stage pipeline takes:

T_pipeline ~= tr + tx + (N - 1) * max(tr, tx)

The corresponding serialized work takes N * (tr + tx). This model assumes independent resources and excludes additional RPC and scheduling costs. The real baseline is the current one-large-batch implementation, so benchmark against it directly; extra sub-batch RPCs and reduced disk batching can erase the overlap benefit.

Benefits should be largest when both stages contribute materially to restoration time and requests contain multiple objects. Costs include more RPCs, executor work, staging pressure, and lifetime-management complexity. Use the current path for small requests or unsupported configurations.

Validation and rollout

Correctness:

Multi-key reads across one and multiple owners; original ordering and duplicate-key behavior.
Small, mixed-size, oversized, and non-aligned objects; multiple destination slices.
Exact data/checksum validation for host buffers and Ascend HBM destinations.
Disk read failure, missing/evicted keys, allocation failure, transfer failure, cancellation, lost release responses, and TTL expiry during queued/in-flight work.
Delayed completions to prove buffers and destination ranges are not reused while still accessed.
Disabled and unsupported paths preserve existing behavior.

Performance:

Compare the existing large-batch path, serialized sub-batches, and pipelined sub-batches.
Hold I/O mode, total bytes, hardware topology, cache state, and client concurrency constant.
Test same-process, same-host different-process, and remote SSD owners.
Sweep object sizes/counts, byte targets, windows, and request concurrency.
Record p50/p95/p99 end-to-end destination-ready latency, throughput, RPC count, CPU use, peak staging occupancy, and queue wait.
Trace load/READY/transfer/release timestamps and demonstrate that load(B1) begins before transfer(B0) ends.
Measure isolated stage throughput and joint execution to identify shared PCIe/NUMA/network bottlenecks.
Include small-request and single-object controls, without claiming an internal-pipeline benefit for one object.

Roll out as opt-in on deployments with validated buffer-lifetime support. Change defaults only after correctness and workload-level performance results are available. Report measurements before claiming a speedup.

Alternatives considered

Async Transfer Engine only: retains the whole-batch SSD completion barrier.
More concurrent complete restore requests: may overlap existing operations, but does not explicitly bound or schedule overlap within one large batch.
Intra-object chunk streaming: can accelerate a single large object but requires range-read support and additional control/lifetime machinery; defer to a separate RFC.
Owner-push transfers: changes the current requester-READ direction and destination publication; unnecessary for this first step.

The following existing discussions and open PRs were reviewed before submission:

Existing work	Relationship to this proposal
#4268: Read a batch's LOCAL_DISK endpoints in parallel	Parallelizes independent owner endpoints. This RFC overlaps loading and transfer across sub-batches within an owner group, including a request hitting only one owner. Compose with that work and reuse its persistent scheduling/restore-ownership helpers where appropriate, rather than duplicating endpoint fan-out machinery.
#2502: Batch SSD reads in BatchLoad	Improves storage-stage NVMe queue depth. Preserve useful disk batching inside each sub-batch and evaluate both optimizations together.
#3417: Explicit SSD-to-DRAM prefetch trigger and #4272: Prefetch SSD-only objects to DRAM on exist	Move SSD-to-DRAM promotion ahead of demand reads. This RFC schedules an already issued demand restore into the caller's destination without adding a MEMORY replica or a promotion trigger. The approaches can complement each other.
#3786: GDS offload using TENT and #4395: GDS abstraction for the local-SSD tier	Provide direct storage/accelerator paths. This RFC overlaps the existing DRAM-staged fallback, initially targeting Ascend, without introducing GDS replica metadata or a direct-storage engine.

Searches included SSD pipeline, SSD overlap, offload sub-batch/subbatch, and open offload PRs. The distinct question here is whether to add bounded, whole-object load/transfer overlap inside a LOCAL_DISK demand batch.

Questions for maintainers

Is requester-side, whole-object sub-batch scheduling the preferred first integration point for LOCAL_DISK batch restores?
Which existing owner-buffer lease/GC mechanism should provide the required safe asynchronous lifetime contract?
Should the first implementation expose a Future-returning Client helper, or start with independent workers around the current blocking wrappers?
Which backend/transport combinations and workload benchmarks should be required before enabling the feature by default?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation.

AI assistance disclosure: this RFC was drafted with AI assistance at the submitter's request.

Originally created by @ckeyever on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4766 ### Changes proposed ## Summary Add an opt-in, bounded sub-batch pipeline to Mooncake Store's LOCAL_DISK restore path. While the SSD owner reads sub-batch B(i+1) into its DRAM staging arena, the requesting client transfers the already loaded sub-batch B(i) into its final destination buffers, including accelerator HBM. The first implementation operates on complete objects and reuses the existing offload-read and buffer-release RPCs. It does not introduce intra-object chunking or a streaming RPC protocol. The initial target workload is batched KV-cache restoration on Ascend; scheduling should remain transport-independent. This is a design proposal. No performance improvement has been measured yet. ## Motivation and current behavior The current restore path contains a barrier between SSD loading and destination transfer: 1. The requester selects LOCAL_DISK replicas and groups reads by the SSD owner's RPC endpoint. 2. The owner allocates DRAM buffers and calls FileStorage::BatchGet / LoadBatch. 3. LoadBatch waits for the complete batch's storage-backend BatchLoad before returning buffer addresses. 4. The requester calls Client::BatchGetOffloadObject, which submits Transfer Engine READs and waits for the TransferFuture. 5. The requester releases the owner's offload buffers. Relevant source at the inspected baseline: - [FileStorage::LoadBatch and BatchGet](https://github.com/kvcache-ai/Mooncake/blob/292e8391b7df0a385092474374660f599958f6c4/mooncake-store/src/file_storage.cpp#L233). - [RealClient::batch_get_into_offload_object_internal](https://github.com/kvcache-ai/Mooncake/blob/292e8391b7df0a385092474374660f599958f6c4/mooncake-store/src/real_client.cpp#L6816). - [Client::BatchGetOffloadObject](https://github.com/kvcache-ai/Mooncake/blob/292e8391b7df0a385092474374660f599958f6c4/mooncake-store/src/client_service.cpp#L3793). - [SSD offload design](https://github.com/kvcache-ai/Mooncake/blob/292e8391b7df0a385092474374660f599958f6c4/docs/source/design/store/ssd-offload.md). The batch barrier prevents transfers for ready objects from overlapping the remaining objects' disk reads within that request. This proposal targets end-to-end SSD-hit restoration latency, measured until all requested destination data is usable. ## Scope In scope: - Whole-object sub-batches for batched LOCAL_DISK restores. - Overlap between SSD-to-DRAM loading and DRAM-to-destination transfer. - Bounded staging occupancy, completion-driven release, and explicit failure handling. - Existing same-process and remote-owner transfer strategies. - Unchanged public result ordering, per-key error behavior, and checksum semantics. Deferred: - Splitting one object into disk-read chunks. - Per-chunk READY notifications or BeginRestore/PollReady streaming protocols. - Changing the on-disk format, replica selection, SSD promotion policy, or master scheduling. - Extending the new pipeline to NOF_SSD, DFS, and other disk-replica paths. - Automatically consuming partially restored KV objects from the model execution stream. A request with a single object has no intra-object overlap benefit in this version. ## Proposed design ### 1. Plan whole-object sub-batches After the existing replica query and selection, group LOCAL_DISK objects by owner endpoint. Partition each owner group using both a target byte size and a maximum object count. Each sub-batch contains full objects with their original result indices, destination slices, selected-replica metadata, sizes, and expected checksums. Preserve the caller's current duplicate-key behavior and restore results to their original positions. Use byte-based partitioning because KV object sizes vary. An object larger than the target size is handled as a standalone sub-batch; account for its actual allocation size. If it exceeds the pipeline admission budget, execute the existing path exclusively after draining pipeline work, rather than rejecting a previously valid request. Start with two sub-batches in the lifecycle window: one being loaded and one being transferred. Increase the window only after measurement. Keep useful batching within each SSD request so that the change does not collapse NVMe queue depth or turn small objects into one-RPC-per-key requests. ### 2. Run independent load and transfer stages Use shared executors/coroutines rather than creating threads for every request. Load stage: - Reserve bounded staging and lifecycle-window credits before issuing an offload read. - Use the existing owner RPC, or the existing same-process load path. - Once that sub-batch is fully loaded, publish its returned buffers to a bounded READY queue. - Continue with the next eligible sub-batch while transfers run independently. Transfer stage: - Consume READY sub-batches and submit the existing Transfer Engine READ operation or same-process copy strategy. - Retain the resulting Future and buffer owner in an in-flight context. - Observe terminal completion independently of the load stage. - Run the existing checksum verification at the appropriate completion boundary, release the owner buffer, and return credits. Conceptual steady state: | Interval | SSD load stage | Destination-transfer stage | | --- | --- | --- | | 0 | Load B0 | Idle | | 1 | Load B1 | Transfer B0 | | 2 | Load B2 | Transfer B1 | | 3 | Load B3 | Transfer B2 | | 4 | Idle | Transfer B3 | The table assumes equal stage durations for illustration; the actual scheduler is completion-driven. A dedicated transfer worker may initially use the existing blocking transfer wrapper while the independent load worker continues. Exposing the existing submitter's TransferFuture through a nonblocking Client helper can subsequently support multiple in-flight transfers. Asynchronous execution in every low-level API is not a prerequisite for the two stages to overlap. ### 3. Bound memory and scheduling Track the full lifecycle: LOADING -> READY -> TRANSFERRING -> VERIFYING -> RELEASED \-> FAILED/DRAINING -> RELEASED only after safe completion Admission counts all allocated or reserved bytes, including reads in progress, READY buffers, alignment overhead, and transfers/checks still holding buffers. Use: - A small per-request lifecycle window. - A per-owner limit shared across that request's sub-batches. - An owner-side process-wide budget across requesting clients, using the existing allocator capacity as a hard bound. - Fair scheduling so one large restore does not monopolize an SSD owner or transfer endpoint. The bounded READY queue is essential: when destination transfers slow down, loading must stop before unbounded DRAM accumulation. Requester-local accounting alone is insufficient to bound concurrent remote clients. ### 4. Preserve ownership and lease safety The current synchronous ReleaseGuard must not release buffers immediately after a new asynchronous submit helper returns. Ownership moves to the in-flight operation context and lasts through confirmed transfer termination and any verification that still accesses the staging buffer. For the same-process path, retain the LocalBatchResult owner. For the remote path, retain the returned batch_id and issue the existing release RPC only after the source is no longer accessed. The owner's existing TTL-based GC is a specific issue to settle before enabling this feature: prefetching introduces additional residence time before transfer. Track deadlines conservatively from before each load RPC, reserve transfer capacity before prefetching, and stop admitting work that cannot meet its budget. These checks reduce exposure but do not prove that a timed-out DMA has stopped. The required safety contract is: - Owner GC must not recycle memory while a transfer can still access it. - An expiry or cancellation enters a draining/quarantine state until outstanding I/O/transfer access is confirmed quiescent. - A reported timeout alone is not confirmation of quiescence. - Release and error cleanup must tolerate duplicate notifications without double freeing. Prefer a minimal integration with existing buffer ownership and release handling. If safe lifetime handling requires an owner capability or lease extension, include that small change and gate the opt-in path accordingly; it does not require a streaming data protocol. Mixed-version deployments must retain the existing path unless the required owner behavior is known to be available. ### 5. Preserve API and failure semantics Keep synchronous public batch APIs synchronous: return only when their existing completion conditions are satisfied. Internally, individual objects can finish earlier. Map load, transfer, and checksum errors back to the affected keys. Do not report a partially written destination as a successful object. Preserve the current behavior for independent successful keys. On cancellation or fatal request failure, stop new loads, drain/cancel outstanding operations using supported transport semantics, release buffers after quiescence, and retain the caller's destination-buffer lifetime until those accesses have ended. Retries must follow the existing replica/version rules; do not overwrite a destination while an earlier attempt can still write to it. ## Implementation touch points | Component | Expected change | | --- | --- | | RealClient LOCAL_DISK batch restore orchestration | Partition owner groups, schedule sub-batches, preserve original result mapping. | | Client / TransferSubmitter | Reuse the existing transfer strategy; optionally expose a Future-returning offload-transfer helper. | | FileStorage buffer lifecycle | Reuse whole-object BatchGet/BatchGetLocal; integrate bounded ownership and safe lease/GC handling. | | StorageBackendInterface / on-disk format | No new range-read API or format changes in this version. | | Ascend transport | Reuse the existing path initially; measure local-copy synchronization and executor capacity before changing it. | Existing Ascend constraints remain applicable: buffer-pool mode and the async executor cannot simply be enabled together, and the optional pinned restore arena is disabled with io_uring in the inspected code. Enabling an async environment variable alone does not remove the Store batch barrier. Reuse the existing protocol-aware registered staging allocator and validate the intended deployment's memory/transfer compatibility. Tentative opt-in tuning controls, with final names to follow maintainer guidance: - Enable/disable pipelined LOCAL_DISK restore; disabled by default initially. - Target bytes and maximum keys per sub-batch. - Maximum lifecycle window and maximum transfer concurrency. - Per-request and owner-wide staging-byte limits. Initial experiments should sweep 1-4 MiB target sub-batches and windows of 2 and 4. These are experimental settings, not selected production defaults. ## Expected benefit and trade-offs For N equal sub-batches with effective load time tr and transfer time tx, an ideal two-stage pipeline takes: T_pipeline ~= tr + tx + (N - 1) * max(tr, tx) The corresponding serialized work takes N * (tr + tx). This model assumes independent resources and excludes additional RPC and scheduling costs. The real baseline is the current one-large-batch implementation, so benchmark against it directly; extra sub-batch RPCs and reduced disk batching can erase the overlap benefit. Benefits should be largest when both stages contribute materially to restoration time and requests contain multiple objects. Costs include more RPCs, executor work, staging pressure, and lifetime-management complexity. Use the current path for small requests or unsupported configurations. ## Validation and rollout Correctness: - Multi-key reads across one and multiple owners; original ordering and duplicate-key behavior. - Small, mixed-size, oversized, and non-aligned objects; multiple destination slices. - Exact data/checksum validation for host buffers and Ascend HBM destinations. - Disk read failure, missing/evicted keys, allocation failure, transfer failure, cancellation, lost release responses, and TTL expiry during queued/in-flight work. - Delayed completions to prove buffers and destination ranges are not reused while still accessed. - Disabled and unsupported paths preserve existing behavior. Performance: - Compare the existing large-batch path, serialized sub-batches, and pipelined sub-batches. - Hold I/O mode, total bytes, hardware topology, cache state, and client concurrency constant. - Test same-process, same-host different-process, and remote SSD owners. - Sweep object sizes/counts, byte targets, windows, and request concurrency. - Record p50/p95/p99 end-to-end destination-ready latency, throughput, RPC count, CPU use, peak staging occupancy, and queue wait. - Trace load/READY/transfer/release timestamps and demonstrate that load(B1) begins before transfer(B0) ends. - Measure isolated stage throughput and joint execution to identify shared PCIe/NUMA/network bottlenecks. - Include small-request and single-object controls, without claiming an internal-pipeline benefit for one object. Roll out as opt-in on deployments with validated buffer-lifetime support. Change defaults only after correctness and workload-level performance results are available. Report measurements before claiming a speedup. ## Alternatives considered - **Async Transfer Engine only:** retains the whole-batch SSD completion barrier. - **More concurrent complete restore requests:** may overlap existing operations, but does not explicitly bound or schedule overlap within one large batch. - **Intra-object chunk streaming:** can accelerate a single large object but requires range-read support and additional control/lifetime machinery; defer to a separate RFC. - **Owner-push transfers:** changes the current requester-READ direction and destination publication; unnecessary for this first step. ## Related work The following existing discussions and open PRs were reviewed before submission: | Existing work | Relationship to this proposal | | --- | --- | | [#4268: Read a batch's LOCAL_DISK endpoints in parallel](https://github.com/kvcache-ai/Mooncake/pull/4268) | Parallelizes independent owner endpoints. This RFC overlaps loading and transfer across sub-batches within an owner group, including a request hitting only one owner. Compose with that work and reuse its persistent scheduling/restore-ownership helpers where appropriate, rather than duplicating endpoint fan-out machinery. | | [#2502: Batch SSD reads in BatchLoad](https://github.com/kvcache-ai/Mooncake/pull/2502) | Improves storage-stage NVMe queue depth. Preserve useful disk batching inside each sub-batch and evaluate both optimizations together. | | [#3417: Explicit SSD-to-DRAM prefetch trigger](https://github.com/kvcache-ai/Mooncake/issues/3417) and [#4272: Prefetch SSD-only objects to DRAM on exist](https://github.com/kvcache-ai/Mooncake/pull/4272) | Move SSD-to-DRAM promotion ahead of demand reads. This RFC schedules an already issued demand restore into the caller's destination without adding a MEMORY replica or a promotion trigger. The approaches can complement each other. | | [#3786: GDS offload using TENT](https://github.com/kvcache-ai/Mooncake/issues/3786) and [#4395: GDS abstraction for the local-SSD tier](https://github.com/kvcache-ai/Mooncake/issues/4395) | Provide direct storage/accelerator paths. This RFC overlaps the existing DRAM-staged fallback, initially targeting Ascend, without introducing GDS replica metadata or a direct-storage engine. | Searches included SSD pipeline, SSD overlap, offload sub-batch/subbatch, and open offload PRs. The distinct question here is whether to add bounded, whole-object load/transfer overlap inside a LOCAL_DISK demand batch. ## Questions for maintainers 1. Is requester-side, whole-object sub-batch scheduling the preferred first integration point for LOCAL_DISK batch restores? 2. Which existing owner-buffer lease/GC mechanism should provide the required safe asynchronous lifetime contract? 3. Should the first implementation expose a Future-returning Client helper, or start with independent workers around the current blocking wrappers? 4. Which backend/transport combinations and workload benchmarks should be required before enabling the feature by default? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/). AI assistance disclosure: this RFC was drafted with AI assistance at the submitter's request.

bsakaguchi closed this issue

2026-10-29 21:24:02 +00:00

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Thanks for opening this issue, @ckeyever!

Field	Value
Issue	#4766
GitHub user ID	`11692486`
Reporter	@ckeyever

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Thanks for opening this issue, @ckeyever! | Field | Value | |-------|-------| | **Issue** | #4766 | | **GitHub user ID** | `11692486` | | **Reporter** | @ckeyever | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:36 +00:00

[PR #1215] [CLOSED] Reset static singleton MasterMetricManager after Leader Re-election #2079