kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2911] [RFC]: Add optional Store transport connection warmup #754

New issue

Closed

opened 2026-10-29 21:19:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:08 +00:00

Owner

Originally created by @zchuango on GitHub (Jul 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2911

Changes proposed

Motivation

In a multi-node Mooncake Store deployment, data may remain cold for a period of time before a burst of GET or write traffic arrives.

When transport connections to remote segments have not yet been established, many requests may concurrently enter the connection setup path and repeatedly poll the connection state. A sudden wave of these requests can create a connection storm, consume a large amount of client and transport resources, and in severe cases make the affected processes unstable or unresponsive.

The latency and resource pressure are especially visible when:

multiple clients become active at approximately the same time;
remote data has not been accessed recently;
many requests target endpoints without reusable connections;
first-access latency is important.

Proposed change

Add an optional, best-effort Store transport connection warmup during client setup.

The proposed flow is:

The Store client asks the Master for a bounded list of eligible remote segment endpoints.
The Master filters unavailable segments, the requester's own segments, incompatible protocols, empty endpoints, and duplicate endpoints.
The client submits a small READ-only probe to each selected target through the existing Transfer Engine.
The probe triggers the normal metadata lookup, handshake, endpoint creation, and connection caching paths before business traffic arrives.
Failures remain best-effort and do not fail normal client setup.

This is transport connection warmup rather than KV data or cache preloading. It does not write remote memory.

Safety and scalability

The feature should have the following safeguards:

disabled by default;
finite default target count;
a defensive Master-side target limit;
bounded client-side concurrency;
READ-only probes;
exact protocol-token matching;
endpoint deduplication;
safe ownership of probe buffers while transfers are still pending;
batch-ID cleanup retries;
warmup shutdown before probe memory is unregistered.

If a probe times out, its destination buffer must remain owned until the Transfer Engine reports that the transfer can no longer write to it.

Scope

This proposal only adds Store-level connection warmup.

It does not:

change the general Transfer Engine handshake concurrency model;
add receiver-side handshake worker pools;
guarantee successful warmup for every target;
guarantee connection reuse for transports that do not cache connections;
add periodic warmup;
add a new scheduling framework.

Configuration

The initial implementation provides configuration for:

enabling Store warmup;
READ probe size;
per-target timeout;
client-side concurrency;
maximum requested targets.

The default requested target count is finite, and the Master applies an additional hard cap to each request.

Expected benefit

For warmup targets that are later accessed by business traffic, the first request can avoid part of the metadata lookup, handshake, and endpoint initialization latency.

The expected benefit is primarily:

lower first-access latency;
reduced concurrent connection-state polling during traffic bursts;
reduced risk of a connection-establishment storm.

This proposal is not expected to increase steady-state transfer bandwidth.

Validation plan

The implementation should include:

Master target-selection unit tests;
protocol filtering and endpoint deduplication tests;
maximum-target and Master hard-cap tests;
pending-transfer buffer-lifetime tests;
batch cleanup fairness and retry tests;
TCP loopback integration tests covering real READ probes;
repeated integration-test runs for basic stability.

Real cross-machine, RDMA/UB hardware, large-scale stress, and explicit connection-reuse measurements can be validated separately.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @zchuango on GitHub (Jul 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2911 ### Changes proposed ## Motivation In a multi-node Mooncake Store deployment, data may remain cold for a period of time before a burst of GET or write traffic arrives. When transport connections to remote segments have not yet been established, many requests may concurrently enter the connection setup path and repeatedly poll the connection state. A sudden wave of these requests can create a connection storm, consume a large amount of client and transport resources, and in severe cases make the affected processes unstable or unresponsive. The latency and resource pressure are especially visible when: - multiple clients become active at approximately the same time; - remote data has not been accessed recently; - many requests target endpoints without reusable connections; - first-access latency is important. ## Proposed change Add an optional, best-effort Store transport connection warmup during client setup. The proposed flow is: 1. The Store client asks the Master for a bounded list of eligible remote segment endpoints. 2. The Master filters unavailable segments, the requester's own segments, incompatible protocols, empty endpoints, and duplicate endpoints. 3. The client submits a small READ-only probe to each selected target through the existing Transfer Engine. 4. The probe triggers the normal metadata lookup, handshake, endpoint creation, and connection caching paths before business traffic arrives. 5. Failures remain best-effort and do not fail normal client setup. This is transport connection warmup rather than KV data or cache preloading. It does not write remote memory. ## Safety and scalability The feature should have the following safeguards: - disabled by default; - finite default target count; - a defensive Master-side target limit; - bounded client-side concurrency; - READ-only probes; - exact protocol-token matching; - endpoint deduplication; - safe ownership of probe buffers while transfers are still pending; - batch-ID cleanup retries; - warmup shutdown before probe memory is unregistered. If a probe times out, its destination buffer must remain owned until the Transfer Engine reports that the transfer can no longer write to it. ## Scope This proposal only adds Store-level connection warmup. It does not: - change the general Transfer Engine handshake concurrency model; - add receiver-side handshake worker pools; - guarantee successful warmup for every target; - guarantee connection reuse for transports that do not cache connections; - add periodic warmup; - add a new scheduling framework. ## Configuration The initial implementation provides configuration for: - enabling Store warmup; - READ probe size; - per-target timeout; - client-side concurrency; - maximum requested targets. The default requested target count is finite, and the Master applies an additional hard cap to each request. ## Expected benefit For warmup targets that are later accessed by business traffic, the first request can avoid part of the metadata lookup, handshake, and endpoint initialization latency. The expected benefit is primarily: - lower first-access latency; - reduced concurrent connection-state polling during traffic bursts; - reduced risk of a connection-establishment storm. This proposal is not expected to increase steady-state transfer bandwidth. ## Validation plan The implementation should include: - Master target-selection unit tests; - protocol filtering and endpoint deduplication tests; - maximum-target and Master hard-cap tests; - pending-transfer buffer-lifetime tests; - batch cleanup fairness and retry tests; - TCP loopback integration tests covering real READ probes; - repeated integration-test runs for basic stability. Real cross-machine, RDMA/UB hardware, large-scale stress, and explicit connection-reuse measurements can be validated separately. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:19:08 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 14, 2026):

Thanks for opening this issue, @zchuango!

Field	Value
Issue	#2911
GitHub user ID	`237737874`
Reporter	@zchuango

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 14, 2026): Thanks for opening this issue, @zchuango! | Field | Value | |-------|-------| | **Issue** | #2911 | | **GitHub user ID** | `237737874` | | **Reporter** | @zchuango | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 13, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 13, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:10 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 21, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 21, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:26 +00:00

[PR #754] [MERGED] [TransferEngine] Add guide in testing Transfer Engine, and remove confusing output in transfer engine #1795