[GH-ISSUE #2572] [RFC]: Streaming Transfer for Mooncake #676

New issue

Closed

opened 2026-10-29 21:18:00 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 21:18:00 +00:00

Owner

Copy link

Originally created by @zxpdemonio on GitHub (Jun 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2572

Changes proposed

This RFC proposes adding streaming transfer support to Mooncake Store so that large training or inference payloads can be produced, transferred, and consumed progressively instead of only through object-level blocking Put / Get calls.

Motivation

Mooncake Store currently exposes object-level transfer APIs. A caller starts a Get or Put and observes completion only when the whole object transfer finishes. This works well for bulk transfer bandwidth, but it leaves no way for upper layers to overlap transfer with compute when payloads are large.

This matters for future asynchronous and pipelined workloads:

• Rollout / transfer overlap: an actor can publish generated samples progressively while later samples are still being generated.
• Sample-level pipelining: reward, log-prob, or learner-side work can start on completed chunks without waiting for a full batch.
• veRL fully_async_policy: sample-granularity execution can use chunk completion as the synchronization unit without storing one key per sample.
• Lower peak memory: producers and consumers can stream through fixed-size buffers instead of materializing the full payload at every stage.

This is not primarily intended to improve the latency of today’s fully synchronous PPO path. In that path, each stage waits for the whole batch for operations such as advantage normalization and mini-batch splitting, and large RDMA transfers already achieve high utilization. For the near term, Mooncake’s direct benefit over TransferQueue comes from reducing key fragmentation and supporting field-level selective reads. Streaming transfer is the foundation for future async and pipelined execution.

Goals

• Add a common chunk-completion model that can be reused by progressive read, streaming scatter read, and progressive put.
• Allow callers to wait for or poll individual chunks while preserving a simple wait-all path.
• Support bounded/windowed submission so large streaming transfers do not exhaust transfer-engine resources.
• Keep the public API small and focused on high-level transfer intent.
• Split the implementation into small PRs that are easier to review than PR #1864.

Non-goals

• Replace the existing synchronous Get / Put APIs.
• Require all users to adopt streaming semantics.
• Expose transfer-engine scheduling details directly to Python users.
• Optimize every low-level synchronization mechanism in the RFC itself. Implementation PRs can choose the safest primitive and evolve it after review.

High-level architecture

Streaming transfer adds a handle-oriented layer on top of the existing Mooncake Store transfer path. The store still uses the Master for metadata and the Transfer Engine for data movement. The new part is that a large logical transfer can be split into chunks, and completion of each chunk becomes visible to the caller.

flowchart LR
    App[Application / veRL pipeline]
    Py[Python / C++ Store API]
    Handle[Streaming Handle\nprogress, wait_chunk, wait_all]
    Session[Streaming Session\nchunk state + windowed submission]
    Store[Mooncake Store Client]
    Master[Master Service\nmetadata only]
    TE[Transfer Engine]
    Remote[Remote Store Client / Segment]

    App --> Py
    Py --> Handle
    Handle --> Session
    Session --> Store
    Store --> Master
    Store --> TE
    TE --> Remote
    Remote --> TE
    TE --> Session
    Session --> Handle
    Handle --> App

The key abstraction is a streaming session. A session owns the chunk state for one logical transfer. The caller receives a handle that exposes progress-oriented operations such as polling completed chunks, waiting for a specific chunk, or waiting for the entire transfer.

Synchronous APIs remain as convenience wrappers: they can submit the same underlying transfer and then call wait_all() internally.

API model

The full streaming-transfer series is expected to expose three families of operations.

Progressive read

Progressive read transfers one object into a caller-provided destination buffer and reports completion by chunk.

Conceptually:

handle = progressive_get(key, destination, size, chunk_size);
handle.wait_chunk(i);
handle.wait_all();

This is useful when a consumer can start processing the first chunks before the whole object arrives.

Streaming scatter read

Streaming scatter read generalizes progressive read to multiple keys and ranges. It allows a structured or batched object to be read into selected destinations while still exposing chunk-level progress.

Conceptually:

handle = streaming_batch_get_buffer_ranges(keys, ranges, destination);
handle.wait_chunk(i);
handle.wait_all();

This is the read-side primitive that can serve structured-object selective reads and TransferQueue-like batched access without creating one key per sample-field pair.

Progressive put

Progressive put lets a producer publish chunks as they become available and then seal the object when no more chunks will be written.

Conceptually:

handle = progressive_put(key, total_size, chunk_size);
handle.put_chunk(i, source);
handle.seal();

This is the write-side primitive needed for rollout/transfer overlap: the producer does not need to wait until the whole batch is generated before publishing data.

Dataflow in an async pipeline

flowchart LR
    A0[Actor generates chunk 0]
    M0[Mooncake marks chunk 0 ready]
    R0[Reward processes partial batch]
    A1[Actor generates chunk 1]
    M1[Mooncake marks selected fields / ranges ready]
    L0[Learner consumes available chunks]
    S[Actor seals object]
    Done[wait_all completes]

    A0 -->|progressive_put| M0
    M0 -->|chunk ready| R0
    R0 --> A1
    A1 -->|progressive_put| M1
    M1 -->|streaming read| L0
    L0 --> S
    S --> Done

The important behavior is not that each individual chunk is faster than a bulk transfer. The benefit is that independent stages can overlap in time when the upper-layer algorithm can operate on partial data.

Relationship to TransferQueue replacement

TransferQueue commonly represents a training batch as many small KV entries, roughly batch_size × field_count. This creates metadata and scheduling overhead from many small keys.

Mooncake’s preferred model is to store each stage output as a small number of larger structured objects:

• fewer object keys,
• field-level selective reads,
• optional streaming within large fields or batches.

Streaming transfer complements structured objects. It avoids returning to one-key-per-sample just to gain progressive visibility.

Implementation plan

The complete code path exists in PR #1864. To make review tractable, it should be landed as a series:

1. Chunked read infrastructure

   • Shared chunk completion/session model.
   • Handle operations such as num_chunks, is_chunk_ready, wait_chunk, and wait_all.
   • Batch lifecycle support for incremental submission and completion.

2. Progressive read and streaming scatter read submission

   • Build read requests from object/range metadata.
   • Submit transfer windows incrementally.
   • Reuse the common chunked read session.

3. Client and Python integration

   • Expose progressive/streaming read APIs through real client, dummy client, and pybind layers.
   • Add Python tests and examples.

4. Progressive put

   • Add producer-side streaming writes.
   • Add object sealing semantics.
   • Add tests that cover producer/consumer overlap.

Each PR should preserve the existing synchronous API behavior and introduce only the minimum public surface needed for that layer.

Design considerations

Completion and waiting

A streaming handle should provide both polling and blocking APIs. The implementation must avoid lost wakeups and must not hold unrelated state locks while waiting for long periods. The exact synchronization primitive can be refined in implementation review, but the external contract should stay simple: a chunk is either pending, completed successfully, or failed.

Backpressure

Large transfers should not submit all chunks at once. A session should maintain a submission window so resource usage stays bounded while still keeping the transfer engine busy.

Failure handling

Failures should be visible through the same handle used for progress. If one chunk fails, later wait_chunk() / wait_all() calls should return an error instead of hanging or silently reporting success.

API compatibility

Existing object-level APIs remain unchanged. Streaming APIs are additive. Synchronous reads/writes can continue to use the existing path or internally wait on streaming handles after the streaming path is mature.

Testing plan

• Unit tests for precompleted sessions, chunk bounds, move semantics, and error propagation.
• Concurrency tests for wait/poll behavior and failure wakeup behavior.
• Integration tests for progressive read and streaming scatter read across Mooncake services.
• Python tests for public bindings.
• End-to-end tests or benchmarks for async/pipelined workloads once upper-layer integration is available.

Open questions

• Which synchronization primitive should be used for the first landed version: futex-based waiting or a simpler condition-variable implementation?
• What default chunk size and window size should be exposed, and which should stay internal?
• Should progressive put sealing be visible as a separate public operation, or should it be wrapped by a higher-level object writer?
• What is the minimum Python API needed for veRL integration without overfitting to one training pipeline?

Summary

Streaming transfer should be viewed as a foundation for future async and pipelined Mooncake Store workloads. The design keeps today’s synchronous APIs intact while adding handles that expose progressive completion for large transfers.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @zxpdemonio on GitHub (Jun 23, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2572 ## Changes proposed This RFC proposes adding streaming transfer support to Mooncake Store so that large training or inference payloads can be produced, transferred, and consumed progressively instead of only through object-level blocking `Put` / `Get` calls. ## Motivation Mooncake Store currently exposes object-level transfer APIs. A caller starts a `Get` or `Put` and observes completion only when the whole object transfer finishes. This works well for bulk transfer bandwidth, but it leaves no way for upper layers to overlap transfer with compute when payloads are large. This matters for future asynchronous and pipelined workloads: - **Rollout / transfer overlap**: an actor can publish generated samples progressively while later samples are still being generated. - **Sample-level pipelining**: reward, log-prob, or learner-side work can start on completed chunks without waiting for a full batch. - **veRL `fully_async_policy`**: sample-granularity execution can use chunk completion as the synchronization unit without storing one key per sample. - **Lower peak memory**: producers and consumers can stream through fixed-size buffers instead of materializing the full payload at every stage. This is not primarily intended to improve the latency of today’s fully synchronous PPO path. In that path, each stage waits for the whole batch for operations such as advantage normalization and mini-batch splitting, and large RDMA transfers already achieve high utilization. For the near term, Mooncake’s direct benefit over TransferQueue comes from reducing key fragmentation and supporting field-level selective reads. Streaming transfer is the foundation for future async and pipelined execution. ## Goals - Add a common chunk-completion model that can be reused by progressive read, streaming scatter read, and progressive put. - Allow callers to wait for or poll individual chunks while preserving a simple wait-all path. - Support bounded/windowed submission so large streaming transfers do not exhaust transfer-engine resources. - Keep the public API small and focused on high-level transfer intent. - Split the implementation into small PRs that are easier to review than PR #1864. ## Non-goals - Replace the existing synchronous `Get` / `Put` APIs. - Require all users to adopt streaming semantics. - Expose transfer-engine scheduling details directly to Python users. - Optimize every low-level synchronization mechanism in the RFC itself. Implementation PRs can choose the safest primitive and evolve it after review. ## High-level architecture Streaming transfer adds a handle-oriented layer on top of the existing Mooncake Store transfer path. The store still uses the Master for metadata and the Transfer Engine for data movement. The new part is that a large logical transfer can be split into chunks, and completion of each chunk becomes visible to the caller. ```mermaid flowchart LR App[Application / veRL pipeline] Py[Python / C++ Store API] Handle[Streaming Handle\nprogress, wait_chunk, wait_all] Session[Streaming Session\nchunk state + windowed submission] Store[Mooncake Store Client] Master[Master Service\nmetadata only] TE[Transfer Engine] Remote[Remote Store Client / Segment] App --> Py Py --> Handle Handle --> Session Session --> Store Store --> Master Store --> TE TE --> Remote Remote --> TE TE --> Session Session --> Handle Handle --> App ``` The key abstraction is a **streaming session**. A session owns the chunk state for one logical transfer. The caller receives a **handle** that exposes progress-oriented operations such as polling completed chunks, waiting for a specific chunk, or waiting for the entire transfer. Synchronous APIs remain as convenience wrappers: they can submit the same underlying transfer and then call `wait_all()` internally. ## API model The full streaming-transfer series is expected to expose three families of operations. ### Progressive read Progressive read transfers one object into a caller-provided destination buffer and reports completion by chunk. Conceptually: ```cpp handle = progressive_get(key, destination, size, chunk_size); handle.wait_chunk(i); handle.wait_all(); ``` This is useful when a consumer can start processing the first chunks before the whole object arrives. ### Streaming scatter read Streaming scatter read generalizes progressive read to multiple keys and ranges. It allows a structured or batched object to be read into selected destinations while still exposing chunk-level progress. Conceptually: ```cpp handle = streaming_batch_get_buffer_ranges(keys, ranges, destination); handle.wait_chunk(i); handle.wait_all(); ``` This is the read-side primitive that can serve structured-object selective reads and TransferQueue-like batched access without creating one key per sample-field pair. ### Progressive put Progressive put lets a producer publish chunks as they become available and then seal the object when no more chunks will be written. Conceptually: ```cpp handle = progressive_put(key, total_size, chunk_size); handle.put_chunk(i, source); handle.seal(); ``` This is the write-side primitive needed for rollout/transfer overlap: the producer does not need to wait until the whole batch is generated before publishing data. ## Dataflow in an async pipeline ```mermaid flowchart LR A0[Actor generates chunk 0] M0[Mooncake marks chunk 0 ready] R0[Reward processes partial batch] A1[Actor generates chunk 1] M1[Mooncake marks selected fields / ranges ready] L0[Learner consumes available chunks] S[Actor seals object] Done[wait_all completes] A0 -->|progressive_put| M0 M0 -->|chunk ready| R0 R0 --> A1 A1 -->|progressive_put| M1 M1 -->|streaming read| L0 L0 --> S S --> Done ``` The important behavior is not that each individual chunk is faster than a bulk transfer. The benefit is that independent stages can overlap in time when the upper-layer algorithm can operate on partial data. ## Relationship to TransferQueue replacement TransferQueue commonly represents a training batch as many small KV entries, roughly `batch_size × field_count`. This creates metadata and scheduling overhead from many small keys. Mooncake’s preferred model is to store each stage output as a small number of larger structured objects: - fewer object keys, - field-level selective reads, - optional streaming within large fields or batches. Streaming transfer complements structured objects. It avoids returning to one-key-per-sample just to gain progressive visibility. ## Implementation plan The complete code path exists in PR #1864. To make review tractable, it should be landed as a series: 1. **Chunked read infrastructure** - Shared chunk completion/session model. - Handle operations such as `num_chunks`, `is_chunk_ready`, `wait_chunk`, and `wait_all`. - Batch lifecycle support for incremental submission and completion. 2. **Progressive read and streaming scatter read submission** - Build read requests from object/range metadata. - Submit transfer windows incrementally. - Reuse the common chunked read session. 3. **Client and Python integration** - Expose progressive/streaming read APIs through real client, dummy client, and pybind layers. - Add Python tests and examples. 4. **Progressive put** - Add producer-side streaming writes. - Add object sealing semantics. - Add tests that cover producer/consumer overlap. Each PR should preserve the existing synchronous API behavior and introduce only the minimum public surface needed for that layer. ## Design considerations ### Completion and waiting A streaming handle should provide both polling and blocking APIs. The implementation must avoid lost wakeups and must not hold unrelated state locks while waiting for long periods. The exact synchronization primitive can be refined in implementation review, but the external contract should stay simple: a chunk is either pending, completed successfully, or failed. ### Backpressure Large transfers should not submit all chunks at once. A session should maintain a submission window so resource usage stays bounded while still keeping the transfer engine busy. ### Failure handling Failures should be visible through the same handle used for progress. If one chunk fails, later `wait_chunk()` / `wait_all()` calls should return an error instead of hanging or silently reporting success. ### API compatibility Existing object-level APIs remain unchanged. Streaming APIs are additive. Synchronous reads/writes can continue to use the existing path or internally wait on streaming handles after the streaming path is mature. ## Testing plan - Unit tests for precompleted sessions, chunk bounds, move semantics, and error propagation. - Concurrency tests for wait/poll behavior and failure wakeup behavior. - Integration tests for progressive read and streaming scatter read across Mooncake services. - Python tests for public bindings. - End-to-end tests or benchmarks for async/pipelined workloads once upper-layer integration is available. ## Open questions - Which synchronization primitive should be used for the first landed version: futex-based waiting or a simpler condition-variable implementation? - What default chunk size and window size should be exposed, and which should stay internal? - Should progressive put sealing be visible as a separate public operation, or should it be wrapped by a higher-level object writer? - What is the minimum Python API needed for veRL integration without overfitting to one training pipeline? ## Summary Streaming transfer should be viewed as a foundation for future async and pipelined Mooncake Store workloads. The design keeps today’s synchronous APIs intact while adding handles that expose progressive completion for large transfers. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue 2026-10-29 21:18:01 +00:00

bsakaguchi commented 2026-10-29 21:18:01 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 23, 2026):

Thanks for opening this issue, @zxpdemonio!

Field	Value
Issue	#2572
GitHub user ID	16206312
Reporter	@zxpdemonio

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4776622499 --> @github-actions[bot] commented on GitHub (Jun 23, 2026): Thanks for opening this issue, @zxpdemonio! | Field | Value | |-------|-------| | **Issue** | #2572 | | **GitHub user ID** | `16206312` | | **Reporter** | @zxpdemonio | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:18:01 +00:00

Author

Owner

Copy link

@zxpdemonio commented on GitHub (Aug 12, 2026):

Closing this RFC because the implementation direction has evolved. #2881 has been merged and provides session-based ranged multi-buffer transfers, while the Transfer Engine batching optimization is being handled separately in #3310. This RFC also included broader progressive read/put and chunk-completion semantics that were not adopted as one combined design; if those capabilities are needed later, they should be proposed again around a concrete upper-layer use case and the current APIs.

<!-- gh-comment-id:5261361310 --> @zxpdemonio commented on GitHub (Aug 12, 2026): Closing this RFC because the implementation direction has evolved. #2881 has been merged and provides session-based ranged multi-buffer transfers, while the Transfer Engine batching optimization is being handled separately in #3310. This RFC also included broader progressive read/put and chunk-completion semantics that were not adopted as one combined design; if those capabilities are needed later, they should be proposed again around a concrete upper-layer use case and the current APIs.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#676

No description provided.