mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #2608] [RFC]: RL Disaggregation via Rollout / Update Separation and Shared Mooncake Data Plane #684
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#684
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Srinivasoo7 on GitHub (Jun 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2608
Changes proposed
Summary
This RFC proposes a design for RL disaggregation in Mooncake from Issue #1883 Milestone 1 by separating the rollout phase from the update phase and using Mooncake's data plane to move rollout artifacts, training samples, and versioned state between independently scaled worker pools.
In RL post-training workflows such as PPO, GRPO, and RLHF, rollout and update have different resource profiles:
rollout is inference-heavy and KV-cache-heavy,
update is training/optimizer-heavy and collective-heavy.
Co-locating rollout and update on the same GPU pool can lead to poor utilization because one phase may idle while the other phase is active. The proposed design introduces a clearer boundary between:
Rollout workers, which generate samples,
Update workers, which consume samples and perform training steps,
Mooncake Store / Transfer Engine, which provides the shared data plane.
The first version should be design-first and prototype-oriented. It should define the data contracts and versioning semantics before attempting a full RL framework integration.
Motivation
RL post-training pipelines often alternate between generation and training. These phases stress the system differently:
A single colocated setup is simple, but it can waste resources when rollout and update do not need the same number or type of GPUs.
Mooncake already focuses on disaggregated serving and KV-cache-centric infrastructure. RL disaggregation is a natural extension: rollout and update should be able to scale independently while sharing a reliable data plane for samples, metadata, and versioned state.
The current RL sample flow demonstrates the idea of passing data through Mooncake Store, but it does not yet define a production-oriented rollout/update boundary.
Goals
Non-Goals
Key Design Principle: Versioned State
KV cache and rollout artifacts must be associated with a policy or weight version.
A KV cache produced under policy version
v1should not be treated as valid for exact generation under policy versionv2unless the system explicitly chooses an off-policy or stale-policy mode.Default behavior should be conservative:
This avoids silently mixing model states across update boundaries.
Proposed Architecture
Core Concepts
RolloutSession
A
RolloutSessionrepresents a rollout-side unit of generation.Possible fields:
The exact implementation does not need to use this dataclass. The purpose is to define the conceptual boundary.
UpdateSession
An
UpdateSessionrepresents an update-side training step or mini-batch consumption unit.Possible fields:
PolicyVersion
A
PolicyVersionidentifies the model/weight version used to generate samples.Possible fields:
Data Plane Contract
The rollout-to-update boundary should include:
Suggested logical record:
KV Reuse Semantics
Safe Default
KV reuse is allowed only when:
If the policy version differs, the rollout worker should recompute or ignore the cached KV by default.
Optional Staleness Mode
Some RL algorithms may tolerate stale samples or off-policy data. If maintainers want to support this later, the mode should be explicit.
Example:
This should apply to sample consumption semantics, not silently to exact KV reuse.
Why This Matters
KV tensors are produced by a specific model state. Reusing KV across weight updates can be incorrect if the new model weights differ from the weights that generated the cache.
Therefore, this RFC treats versioning as part of the core design rather than an implementation detail.
Proposed API Shape
This section is illustrative only.
Rollout Side
Update Side
Weight Version Flow
The update side should publish new policy versions after optimizer steps.
Possible flow:
Scaling Model
Rollout and update worker pools should scale independently.
Example:
The exact scheduler is outside the scope of this RFC. The RFC only requires that Mooncake's data plane can support the separation cleanly.
Implementation Plan
Phase 1 — Design and Interfaces
mooncake-rl/ordocs/.Phase 2 — Minimal Prototype
Phase 3 — Framework Selection
Ask maintainers which RL framework should be the first integration target.
Candidates may include:
Phase 4 — Reference Integration
After framework selection:
Phase 5 — Optimization
Potential later optimizations:
Benchmark Plan
A useful benchmark should compare:
Possible metrics:
CI Strategy
Initial CI should only validate lightweight behavior:
Full RL benchmarks should remain manual or nightly until a stable test environment exists.
Alternatives Considered
Status Quo: Colocated Rollout and Update
This is simpler but can waste resources because rollout and update have different hardware and scheduling characteristics.
Plain File/Object Store Without Mooncake
A generic store can move samples but does not exercise Mooncake's intended disaggregated data-plane role and does not integrate naturally with KV/cache-aware infrastructure.
Immediate Full Framework Integration
Rejected for first step.
A full framework integration before agreeing on the data contract may lead to the wrong abstraction. The safer path is to define the boundary first, then integrate.
Reuse KV Across Weight Updates by Default
Rejected.
KV cache is produced by a specific policy/weight version. Reusing it across weight updates can be incorrect unless a specific stale-policy mode is explicitly designed and accepted.
Risks
Open Questions
Expected Outcome
This RFC should produce a clear rollout/update boundary for RL disaggregation and a minimal prototype that uses Mooncake as the shared data plane. The design should be conservative about KV reuse and explicit about policy versioning.
Before submitting a new issue...
@github-actions[bot] commented on GitHub (Jun 24, 2026):
Thanks for opening this issue, @Srinivasoo7!
194645829A maintainer will triage this when possible. To help us respond faster, please include:
Useful links: Documentation · Contributing guide