[GH-ISSUE #2875] [RFC]: DrainJob bandwidth rate limiting #749

New issue

Closed

opened 2026-10-29 21:19:04 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:04 +00:00

Owner

Originally created by @liangxu2000 on GitHub (Jul 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2875

Changes proposed

1. Current State: DrainJob Performance Parameters

After reading through the mooncake-store source, here are the existing control parameters related to DrainJob:

Parameter	Location	Default	Purpose
`max_concurrency`	`CreateDrainJobRequest`	4	Max concurrent MoveTasks per DrainJob
`task_thread_pool_`	Client constructor	4	Client-side task execution thread pool
`kJobDispatchThreadSleepMs`	Master dispatch loop	500ms	Master DrainJob scheduling interval
TaskPollThreadMain interval	Client poll loop	1000ms	Client task fetch interval
`kTaskBatchSize`	Client poll loop	16	Max tasks per `FetchTasks` RPC
`kMaxDrainUnitRetries`	Master drain logic	3	Max retries per key move

All existing controls operate on the "count" dimension. There is zero rate/bandwidth limiting.

Key observations from the transfer path:

In ExecuteReplicaTransfer, the entire object is split_into_slices and submitted to Transfer Engine in a single batch via submitTransfer. No pacing between slices.
Drain MoveTasks share the same RDMA link and client thread pool as regular read/write operations. No priority isolation.
A single object can be several GB. With max_concurrency=4, tens of GB could be in-flight on the RDMA link simultaneously.

2. Discussion Topic

Does the community see a need for DrainJob bandwidth/rate limiting? Are there any existing plans?

From a usage perspective, an aggressive DrainJob could saturate shared RDMA bandwidth and impact read latency for online workloads. I'd like to understand:

Has anyone encountered this as a pain point in production?
Any thoughts on the design direction from the community?

3. Possible Directions (high-level)

Master-side: Track recent throughput via a sliding window (e.g., 5s) over completed task byte counts. Skip scheduling new MoveTasks when the current rate meets the configured limit. Leverages the existing 500ms dispatch loop as a natural pacing mechanism.
Client-side: Split large object transfers into sub-batches (e.g., 64MB each). Insert a token bucket between sub-batches for precise pacing.
API: Add an optional bandwidth_limit_mbps field to CreateDrainJobRequest (0 = unlimited, backward compatible).

The concrete design choice — single-layer vs. dual-layer, where to place it, whether to go adaptive — depends on real community needs. This RFC is intended to start that discussion.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @liangxu2000 on GitHub (Jul 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2875 ### Changes proposed ### Changes proposed ## 1. Current State: DrainJob Performance Parameters After reading through the `mooncake-store` source, here are the existing control parameters related to DrainJob: | Parameter | Location | Default | Purpose | |-----------|----------|---------|---------| | `max_concurrency` | `CreateDrainJobRequest` | 4 | Max concurrent MoveTasks per DrainJob | | `task_thread_pool_` | Client constructor | 4 | Client-side task execution thread pool | | `kJobDispatchThreadSleepMs` | Master dispatch loop | 500ms | Master DrainJob scheduling interval | | TaskPollThreadMain interval | Client poll loop | 1000ms | Client task fetch interval | | `kTaskBatchSize` | Client poll loop | 16 | Max tasks per `FetchTasks` RPC | | `kMaxDrainUnitRetries` | Master drain logic | 3 | Max retries per key move | **All existing controls operate on the "count" dimension. There is zero rate/bandwidth limiting.** Key observations from the transfer path: - In `ExecuteReplicaTransfer`, the entire object is `split_into_slices` and submitted to Transfer Engine in a single batch via `submitTransfer`. No pacing between slices. - Drain MoveTasks share the same RDMA link and client thread pool as regular read/write operations. No priority isolation. - A single object can be several GB. With `max_concurrency=4`, tens of GB could be in-flight on the RDMA link simultaneously. ## 2. Discussion Topic Does the community see a need for DrainJob bandwidth/rate limiting? Are there any existing plans? From a usage perspective, an aggressive DrainJob could saturate shared RDMA bandwidth and impact read latency for online workloads. I'd like to understand: - Has anyone encountered this as a pain point in production? - Any thoughts on the design direction from the community? ## 3. Possible Directions (high-level) - **Master-side**: Track recent throughput via a sliding window (e.g., 5s) over completed task byte counts. Skip scheduling new MoveTasks when the current rate meets the configured limit. Leverages the existing 500ms dispatch loop as a natural pacing mechanism. - **Client-side**: Split large object transfers into sub-batches (e.g., 64MB each). Insert a token bucket between sub-batches for precise pacing. - **API**: Add an optional `bandwidth_limit_mbps` field to `CreateDrainJobRequest` (0 = unlimited, backward compatible). The concrete design choice — single-layer vs. dual-layer, where to place it, whether to go adaptive — depends on real community needs. This RFC is intended to start that discussion. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:19:04 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 13, 2026):

Thanks for opening this issue, @liangxu2000!

Field	Value
Issue	#2875
GitHub user ID	`258133064`
Reporter	@liangxu2000

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 13, 2026): Thanks for opening this issue, @liangxu2000! | Field | Value | |-------|-------| | **Issue** | #2875 | | **GitHub user ID** | `258133064` | | **Reporter** | @liangxu2000 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 12, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 12, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:05 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 20, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 20, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.