kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #3446] [RFC]: TCP Transport Optimization for Non-RDMA Fallback #881

New issue

Open

opened 2026-10-29 21:20:29 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:29 +00:00

Owner

Originally created by @zhtshr on GitHub (Aug 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3446

Changes proposed

Summary

Make the classic TE tcp transport a usable high-performance fallback on
clusters without RDMA, not just a functional last resort.

Today's bottlenecks are structural, not protocol-level: grouped transfers are
strictly serialized (one slice in flight), the transport runs on a single
io_context thread, and connection setup blocks that thread. This RFC fixes
those three plus socket tuning and connection-pool hardening.

No wire-format change — all work is scheduling, threading, and socket
config; a patched side interoperates with an unpatched peer (v1 and v2, both
directions). Split into 2 PRs. Does not overlap rdma_twosided (#3377), which
targets RDMA-equipped clusters.

Motivation

tcp is the lowest-priority transport (selectTransport: tcp=1 < rdma=2),
so it only carries traffic where RDMA is absent — exactly where it must not be
a cliff. Four problems:

RTT-bound, not bandwidth-bound. startTransferSequence advances one
slice at a time; under v2 the next slice waits for the peer's ack. Effective
throughput is slice_size / RTT regardless of link speed — 128KB over 100μs
caps at ~1.3GB/s.
Single-threaded. One worker handles accept, every inbound readBody,
and every outbound transfer. One core tops out at ~5–8GB/s.
Blocking connect (defect-grade). getConnection calls resolve() /
asio::connect() synchronously on the io thread; one DNS/SYN timeout
(1s/3s) freezes the whole transport, accept included.
No congestion tuning. Only TCP_NODELAY is set, so pipelining depends
entirely on SO_SNDBUF ≥ BDP, which default autotuning misses on high-BDP
links (100Gbps × 2ms = 25MB).

Two cheap side-fixes: the 17B SessionHeader is a separate async_write (a
tiny packet per slice under TCP_NODELAY), and CUDA staging does a
new/delete per chunk in all four directions.

Not a problem: the CPU path is already zero-copy (writeBody hands
addr + offset straight to async_write). Only the CUDA branch stages through
host memory. So "zero-copy" here means dropping unnecessary chunking and
reusing staging — not removing a copy.

Non-goals

Replacing RDMA as the high-bandwidth path — this is fallback QoS.
Changing the wire protocol (v1/v2 framing, status frame, kOpcodeV2Flag
untouched); v3 batching is deferred, see Open questions.
SO_ZEROCOPY — needs source pages valid until the error-queue completion,
conflicting with releasing the buffer at COMPLETED; also a loss <256KB.
Migrating off asio (io_uring / coro_io) — bottleneck is concurrency and
buffer sizing, not syscall cost.
GPU copy pipelining — orthogonal, see Future work.

Architecture

Current: single thread, window = 1, blocking connect.
Proposed: bounded concurrent window over N streams on an io_context pool.

submit* ─ StreamGroup{slices, atomic cursor}   window = min(pending, N)
            ├─ runner_0 ─ socket_0 ─ slice_0 → slice_4 → ...
            ├─ runner_1 ─ socket_1 ─ slice_1 → slice_5 → ...
            └─ runner_N ─ socket_N ─ ...
                 ▼
          io_context pool (M threads, lazy)   each socket pinned to one ctx
          acquireConnection: pool hit → sync; miss → async_resolve/connect

Two invariants keep it cheap:

Per-socket serialization preserved. Each socket binds to one
io_context; sessions already use socket_->get_executor() (not a global
context), so Client/ServerSession need no locks and no logic change. The
only global-context reference — the grouped continuation post — moves to the
socket executor.
One request per socket preserved. Concurrency comes from N sockets, not
pipelining one socket. The server sees the same "header then payload"
sequence, so ServerSession is untouched and old peers interoperate.

Adaptive window (no workload mode): window = min(pending, N). Large
transfers have few slices → narrows to 1–2 streams (avoids send-buffer
contention/incast); small transfers have many → fills N to hide ack RTT.
Chunking varies only by memory type (CPU: none; CUDA: staged), not by size.
Default N=4 (4 × 4MB ≈ 16MB in flight, spans 0.3MB–12.5MB BDP). Dispatch is a
shared atomic cursor (work-stealing) to avoid stranding on uneven slices.

Roadmap (2 PRs)

Benchmarking goes first: there is currently no benchmark, so without a
baseline PR2's gains are unquantifiable and regressions invisible. PR1 is also
low-risk (tuning), while PR2 bundles the concurrency work under one review.

PR1 — Baseline + socket/pool tuning.

tcp_transport_bench (configurable slice size/count/streams; throughput +
P50/P99; tc netem RTT).
Socket options: SO_SNDBUF/SO_RCVBUF (MC_TCP_SOCK_BUF, 4MB,
0=autotune), TCP_QUICKACK, optional TCP_CONGESTION.
Pool on by default; MC_TCP_POOL_MAX_PER_PEER cap; cleanupIdleConnections
off the locked path onto a timer.
Validation: existing suite + pool-saturation case; baseline at 4KB/128KB/8MB.

PR2 — Concurrency: multi-stream, threading, async connect.

N-runner window (shared atomic cursor), with submitTransfer/Task routed
through it to bound their unlimited fire-and-forget loop.
Header + first chunk coalesced into one async_write (wire bytes identical);
CPU path stops chunking; CUDA staging reused per session.
io_context pool (MC_TCP_IO_THREADS, lazy), each socket pinned to one ctx;
getConnection→async acquireConnection (resolve cache; fixes the blocking-
connect defect); session->initiate() on the socket executor.
Validation: v1↔v2 mixed-version matrix (preserve the
readWriteAck()/writeBody() order verbatim); failing resolve/connect
no longer stalls others; disconnect / in-flight-destruction cases; throughput
vs PR1 — small slices scale ~N, no regression at 8MB.

Alternatives considered

Option	Why not now
`SO_ZEROCOPY`	Source-page lifetime conflicts with `COMPLETED`; loss <256KB.
`io_uring` backend	Bottleneck isn't syscall cost; transport rewrite.
`coro_io` rebuild	Same, larger blast radius; asio yields these wins as-is.
`v3` batch header	Real for tiny slices but needs v1/v2/v3 compat; PR2's window covers most of it — decide from data.
Pipeline one socket	Needs on-wire request IDs + stateful server; N sockets gets the window with zero protocol change.
Everything in one PR	Benchmark must land first as the baseline; folding it in leaves the optimization unmeasured.

Resolved decisions

Defaults: connection pooling defaults on (hazard closed by the PR1
cap; reuse removes per-transfer reconnect). MC_TCP_SLICE_SIZE stays
64KB — post-PR2 it only governs CUDA staging, so unchanged is a CPU no-op
and CUDA no-regression; opt into 1MB when wanted. Only the pool flip is a
documented behavior change.
MC_TCP_STREAMS_PER_PEER=4 fixed; no BDP autoscaling this round.
Config via getenv, aligning with existing TCP style and avoiding merge
churn against active rdma_* work in config.h.

Open questions

Is v3 batching worth it? Deferred until PR2 data shows a workload the
multi-stream window can't cover; unscheduled otherwise.

Future work

Pinned staging pool + copy/send double buffering (today's CUDA path is
synchronous cudaMemcpy→async_write→cudaMemcpy, no overlap, ~½ device
bandwidth). Orthogonal; ~150 LOC.
v3 multi-descriptor batching, if PR2 justifies it.

Ask

Review (a) TCP-as-fallback being worth this investment, (b) the no-wire-change
boundary (v3 deferred, not proposed here), and (c) the 2-PR order putting
benchmarking before the concurrency work.

Related: #3377 (rdma_twosided, a separate transport targeting RDMA-equipped clusters).

Originally created by @zhtshr on GitHub (Aug 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3446 ### Changes proposed ## Summary Make the classic TE `tcp` transport a usable high-performance fallback on clusters without RDMA, not just a functional last resort. Today's bottlenecks are structural, not protocol-level: grouped transfers are strictly serialized (one slice in flight), the transport runs on a single `io_context` thread, and connection setup blocks that thread. This RFC fixes those three plus socket tuning and connection-pool hardening. **No wire-format change** — all work is scheduling, threading, and socket config; a patched side interoperates with an unpatched peer (v1 and v2, both directions). Split into 2 PRs. Does not overlap `rdma_twosided` (#3377), which targets RDMA-equipped clusters. ## Motivation `tcp` is the lowest-priority transport (`selectTransport`: `tcp=1 < rdma=2`), so it only carries traffic where RDMA is absent — exactly where it must not be a cliff. Four problems: 1. **RTT-bound, not bandwidth-bound.** `startTransferSequence` advances one slice at a time; under v2 the next slice waits for the peer's ack. Effective throughput is `slice_size / RTT` regardless of link speed — 128KB over 100μs caps at ~1.3GB/s. 2. **Single-threaded.** One worker handles `accept`, every inbound `readBody`, and every outbound transfer. One core tops out at ~5–8GB/s. 3. **Blocking connect (defect-grade).** `getConnection` calls `resolve()` / `asio::connect()` synchronously on the io thread; one DNS/SYN timeout (1s/3s) freezes the whole transport, `accept` included. 4. **No congestion tuning.** Only `TCP_NODELAY` is set, so pipelining depends entirely on `SO_SNDBUF ≥ BDP`, which default autotuning misses on high-BDP links (100Gbps × 2ms = 25MB). Two cheap side-fixes: the 17B `SessionHeader` is a separate `async_write` (a tiny packet per slice under `TCP_NODELAY`), and CUDA staging does a `new/delete` per chunk in all four directions. **Not a problem:** the CPU path is already zero-copy (`writeBody` hands `addr + offset` straight to `async_write`). Only the CUDA branch stages through host memory. So "zero-copy" here means dropping unnecessary chunking and reusing staging — not removing a copy. ## Non-goals - Replacing RDMA as the high-bandwidth path — this is fallback QoS. - Changing the wire protocol (v1/v2 framing, status frame, `kOpcodeV2Flag` untouched); `v3` batching is deferred, see Open questions. - `SO_ZEROCOPY` — needs source pages valid until the error-queue completion, conflicting with releasing the buffer at `COMPLETED`; also a loss <256KB. - Migrating off asio (`io_uring` / `coro_io`) — bottleneck is concurrency and buffer sizing, not syscall cost. - GPU copy pipelining — orthogonal, see Future work. ## Architecture Current: single thread, window = 1, blocking connect. Proposed: bounded concurrent window over N streams on an `io_context` pool. ``` submit* ─ StreamGroup{slices, atomic cursor} window = min(pending, N) ├─ runner_0 ─ socket_0 ─ slice_0 → slice_4 → ... ├─ runner_1 ─ socket_1 ─ slice_1 → slice_5 → ... └─ runner_N ─ socket_N ─ ... ▼ io_context pool (M threads, lazy) each socket pinned to one ctx acquireConnection: pool hit → sync; miss → async_resolve/connect ``` Two invariants keep it cheap: - **Per-socket serialization preserved.** Each socket binds to one `io_context`; sessions already use `socket_->get_executor()` (not a global context), so `Client/ServerSession` need no locks and no logic change. The only global-context reference — the grouped continuation post — moves to the socket executor. - **One request per socket preserved.** Concurrency comes from N sockets, not pipelining one socket. The server sees the same "header then payload" sequence, so `ServerSession` is untouched and old peers interoperate. **Adaptive window** (no workload mode): `window = min(pending, N)`. Large transfers have few slices → narrows to 1–2 streams (avoids send-buffer contention/incast); small transfers have many → fills N to hide ack RTT. Chunking varies only by memory type (CPU: none; CUDA: staged), not by size. Default `N=4` (4 × 4MB ≈ 16MB in flight, spans 0.3MB–12.5MB BDP). Dispatch is a shared atomic cursor (work-stealing) to avoid stranding on uneven slices. ## Roadmap (2 PRs) Benchmarking goes first: there is currently **no benchmark**, so without a baseline PR2's gains are unquantifiable and regressions invisible. PR1 is also low-risk (tuning), while PR2 bundles the concurrency work under one review. **PR1 — Baseline + socket/pool tuning.** - `tcp_transport_bench` (configurable slice size/count/streams; throughput + P50/P99; `tc netem` RTT). - Socket options: `SO_SNDBUF`/`SO_RCVBUF` (`MC_TCP_SOCK_BUF`, 4MB, `0`=autotune), `TCP_QUICKACK`, optional `TCP_CONGESTION`. - Pool on by default; `MC_TCP_POOL_MAX_PER_PEER` cap; `cleanupIdleConnections` off the locked path onto a timer. - *Validation:* existing suite + pool-saturation case; baseline at 4KB/128KB/8MB. **PR2 — Concurrency: multi-stream, threading, async connect.** - N-runner window (shared atomic cursor), with `submitTransfer`/`Task` routed through it to **bound** their unlimited fire-and-forget loop. - Header + first chunk coalesced into one `async_write` (wire bytes identical); CPU path stops chunking; CUDA staging reused per session. - `io_context` pool (`MC_TCP_IO_THREADS`, lazy), each socket pinned to one ctx; `getConnection`→async `acquireConnection` (resolve cache; fixes the blocking- connect defect); `session->initiate()` on the socket executor. - *Validation:* v1↔v2 mixed-version matrix (preserve the `readWriteAck()`/`writeBody()` order verbatim); failing `resolve`/`connect` no longer stalls others; disconnect / in-flight-destruction cases; throughput vs PR1 — small slices scale ~N, no regression at 8MB. ## Alternatives considered | Option | Why not now | | --- | --- | | `SO_ZEROCOPY` | Source-page lifetime conflicts with `COMPLETED`; loss <256KB. | | `io_uring` backend | Bottleneck isn't syscall cost; transport rewrite. | | `coro_io` rebuild | Same, larger blast radius; asio yields these wins as-is. | | `v3` batch header | Real for tiny slices but needs v1/v2/v3 compat; PR2's window covers most of it — decide from data. | | Pipeline one socket | Needs on-wire request IDs + stateful server; N sockets gets the window with zero protocol change. | | Everything in one PR | Benchmark must land first as the baseline; folding it in leaves the optimization unmeasured. | ## Resolved decisions - **Defaults:** connection pooling defaults **on** (hazard closed by the PR1 cap; reuse removes per-transfer reconnect). `MC_TCP_SLICE_SIZE` **stays 64KB** — post-PR2 it only governs CUDA staging, so unchanged is a CPU no-op and CUDA no-regression; opt into 1MB when wanted. Only the pool flip is a documented behavior change. - **`MC_TCP_STREAMS_PER_PEER=4`** fixed; no BDP autoscaling this round. - **Config via `getenv`**, aligning with existing TCP style and avoiding merge churn against active `rdma_*` work in `config.h`. ## Open questions - **Is `v3` batching worth it?** Deferred until PR2 data shows a workload the multi-stream window can't cover; unscheduled otherwise. ## Future work - Pinned staging pool + copy/send double buffering (today's CUDA path is synchronous `cudaMemcpy`→`async_write`→`cudaMemcpy`, no overlap, ~½ device bandwidth). Orthogonal; ~150 LOC. - `v3` multi-descriptor batching, if PR2 justifies it. ## Ask Review (a) TCP-as-fallback being worth this investment, (b) the no-wire-change boundary (`v3` deferred, not proposed here), and (c) the 2-PR order putting benchmarking before the concurrency work. --- Related: #3377 (`rdma_twosided`, a separate transport targeting RDMA-equipped clusters).

bsakaguchi commented

2026-10-29 21:20:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 14, 2026):

Thanks for opening this issue, @zhtshr!

Field	Value
Issue	#3446
GitHub user ID	`44193225`
Reporter	@zhtshr

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 14, 2026): Thanks for opening this issue, @zhtshr! | Field | Value | |-------|-------| | **Issue** | #3446 | | **GitHub user ID** | `44193225` | | **Reporter** | @zhtshr | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:05 +00:00

[PR #881] [MERGED] fix(store): fix memory leak in client_integration_test.cpp #1883