[GH-ISSUE #4995] [RFC]: Faster io_uring SSD reads for offload: sliding-window submission, cross-bucket batching and dedicated read threads #1267

New issue

Open

opened 2026-10-29 21:24:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:39 +00:00

Owner

Originally created by @SnowLegend-star on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4995

Changes proposed

Title: [RFC]: Faster io_uring SSD reads for offload: sliding-window submission, cross-bucket batching and dedicated read threads

Changes proposed

Motivation

We serve GLM-style MLA models with SGLang HiCache + Mooncake standalone (SSD offload as L3). Prefill throughput is bounded by how fast one real_client can load cold KV from SSD. With MOONCAKE_OFFLOAD_USE_URING=true on a single NVMe drive, one reader gets ~3.0 GB/s, while the drive can do ~6 GB/s. Profiling the read path (FileStorage::BatchGet → BucketStorageBackend::BatchLoad → UringFile::batch_read) shows three limits:

Fixed-size groups. SharedUringRing::batch_read submits up to QUEUE_DEPTH (32) reads, then waits for the whole group before it submits the next one. So the slowest read in each group sets the pace, and the queue drains between groups.
Per-bucket submission. BatchLoad calls batch_read once per bucket file, one after another. A batch whose keys span many buckets becomes many small serial submissions.
Fixed-buffer registration on every caller thread. Reads run on the caller's thread, and each thread's ring registers the whole staging arena the first time it does I/O. In standalone mode the callers are coro_io pool threads (hardware_concurrency() of them; 128 on our host). After splitting the arena into <=1 GiB iovecs (same idea as #4862), we saw ~200 registrations of a 10 GiB arena at cold start. Each one pins ~2.6M pages, so early reads time out (-706), and single-reader throughput drops from 3.0 to 1.3 GB/s until the threads are warm. #4788 fixes this with io_uring_clone_buffers(), but that needs kernel 6.12+; many production hosts (ours included) run 5.10.

Proposal

All changes are inside the io_uring path. POSIX reads and the write path are unchanged.

Sliding-window batch read. New SharedUringRing::batch_read_multi(items, window). It fills the ring up to window in-flight SQEs, then submits one more each time a CQE completes. Each item records its own result. On a ring-level error it drains the ring and resets it, as today. The window and ring size come from MOONCAKE_OFFLOAD_URING_QUEUE_DEPTH (default 32, ring capped at 4096).
Cross-bucket batching. BucketStorageBackend::BatchLoad opens every bucket file the batch needs, collects all ReadDescs with their file, and issues a single UringFile::batch_read_multi call. The completion, short-read and pointer-offset checks after the read stay the same.
Dedicated read threads. New UringFile::start_read_workers(threads, queue_depth), controlled by MOONCAKE_OFFLOAD_URING_READ_THREADS (proposed default 8; 0 keeps the current behavior of reading on the caller thread). When enabled:
- batch reads are handed to a small pool of read threads (a job queue plus a promise per job);
- the fixed buffer is registered only on those threads' rings, right after register_global_buffer publishes it, and one thread at a time (a mutex), because concurrent pinning of the same arena was much slower than serial on 5.10;
- other threads do not register, so cold-start cost is threads × arena, not pool size × arena.
<=1 GiB fixed-buffer chunks. Needed so that arenas larger than 1 GiB work on 5.10. This overlaps with #4862, and I'm happy to rebase onto it instead of carrying a duplicate.

Results

Setup: kernel 5.10, one NVMe drive, staging arena 10 GiB, lease 5 s, 16 client threads, ~1.8 MB per object, ~430 MB per BatchGet. One reader with N concurrent batch reads. Unique bytes / wall time, GB/s:

Build	N=1	N=2	N=4	Cold start
uring, current	3.01	4.78	6.11	OK
+ <=1 GiB chunks only	1.30 (3.52 warm)	5.15	5.92	`-706` until all pool threads registered
+ this RFC (8 read threads, QD 32)	3.6–3.8	5.7–6.2	6.2–6.4	OK, 8 registrations, done ~3 s after start

With 8 read threads, average SSD load per batch is ~70 ms at N=1. At N=4 the drive is close to saturated. Every run verifies the data it reads back.

Plan

I'd split the code into separate PRs (about 450 non-test lines in total, plus tests):

Sliding-window batch_read_multi + cross-bucket batching in BatchLoad. This works without the other two parts and is useful on its own.
Dedicated read threads + serialized registration, on top of #4862 (or including the chunking if #4862 changes direction).

Each PR adds unit tests for >1 GiB fixed buffers, multi-file batch reads, concurrent callers on read workers, and multi-bucket BatchLoad over uring.

Questions for maintainers

Is a small dedicated read-thread pool acceptable in UringFile, or would you prefer the reads to stay on caller threads? One alternative is to fall back to #4788's clone approach when the kernel supports it.
Default for MOONCAKE_OFFLOAD_URING_READ_THREADS: should it be 8 when uring is enabled, or 0 (opt-in)?
Related but out of scope here: duplicate reads of the same keys from several TP ranks are covered by #2357 (see my comment there).

：）

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @SnowLegend-star on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4995 ### Changes proposed Title: [RFC]: Faster io_uring SSD reads for offload: sliding-window submission, cross-bucket batching and dedicated read threads ### Changes proposed #### Motivation We serve GLM-style MLA models with SGLang HiCache + Mooncake standalone (SSD offload as L3). Prefill throughput is bounded by how fast one `real_client` can load cold KV from SSD. With `MOONCAKE_OFFLOAD_USE_URING=true` on a single NVMe drive, one reader gets ~3.0 GB/s, while the drive can do ~6 GB/s. Profiling the read path (`FileStorage::BatchGet` → `BucketStorageBackend::BatchLoad` → `UringFile::batch_read`) shows three limits: 1. **Fixed-size groups.** `SharedUringRing::batch_read` submits up to `QUEUE_DEPTH` (32) reads, then waits for the whole group before it submits the next one. So the slowest read in each group sets the pace, and the queue drains between groups. 2. **Per-bucket submission.** `BatchLoad` calls `batch_read` once per bucket file, one after another. A batch whose keys span many buckets becomes many small serial submissions. 3. **Fixed-buffer registration on every caller thread.** Reads run on the caller's thread, and each thread's ring registers the whole staging arena the first time it does I/O. In standalone mode the callers are `coro_io` pool threads (`hardware_concurrency()` of them; 128 on our host). After splitting the arena into <=1 GiB iovecs (same idea as #4862), we saw ~200 registrations of a 10 GiB arena at cold start. Each one pins ~2.6M pages, so early reads time out (`-706`), and single-reader throughput drops from 3.0 to 1.3 GB/s until the threads are warm. #4788 fixes this with `io_uring_clone_buffers()`, but that needs kernel 6.12+; many production hosts (ours included) run 5.10. #### Proposal All changes are inside the io_uring path. POSIX reads and the write path are unchanged. - **Sliding-window batch read.** New `SharedUringRing::batch_read_multi(items, window)`. It fills the ring up to `window` in-flight SQEs, then submits one more each time a CQE completes. Each item records its own result. On a ring-level error it drains the ring and resets it, as today. The window and ring size come from `MOONCAKE_OFFLOAD_URING_QUEUE_DEPTH` (default 32, ring capped at 4096). - **Cross-bucket batching.** `BucketStorageBackend::BatchLoad` opens every bucket file the batch needs, collects all `ReadDesc`s with their file, and issues a single `UringFile::batch_read_multi` call. The completion, short-read and pointer-offset checks after the read stay the same. - **Dedicated read threads.** New `UringFile::start_read_workers(threads, queue_depth)`, controlled by `MOONCAKE_OFFLOAD_URING_READ_THREADS` (proposed default 8; `0` keeps the current behavior of reading on the caller thread). When enabled: - batch reads are handed to a small pool of read threads (a job queue plus a promise per job); - the fixed buffer is registered only on those threads' rings, right after `register_global_buffer` publishes it, and one thread at a time (a mutex), because concurrent pinning of the same arena was much slower than serial on 5.10; - other threads do not register, so cold-start cost is `threads × arena`, not `pool size × arena`. - **<=1 GiB fixed-buffer chunks.** Needed so that arenas larger than 1 GiB work on 5.10. This overlaps with #4862, and I'm happy to rebase onto it instead of carrying a duplicate. #### Results Setup: kernel 5.10, one NVMe drive, staging arena 10 GiB, lease 5 s, 16 client threads, ~1.8 MB per object, ~430 MB per `BatchGet`. One reader with N concurrent batch reads. Unique bytes / wall time, GB/s: | Build | N=1 | N=2 | N=4 | Cold start | |---|---|---|---|---| | uring, current | 3.01 | 4.78 | 6.11 | OK | | + <=1 GiB chunks only | 1.30 (3.52 warm) | 5.15 | 5.92 | `-706` until all pool threads registered | | + this RFC (8 read threads, QD 32) | **3.6–3.8** | **5.7–6.2** | **6.2–6.4** | OK, 8 registrations, done ~3 s after start | With 8 read threads, average SSD load per batch is ~70 ms at N=1. At N=4 the drive is close to saturated. Every run verifies the data it reads back. #### Plan I'd split the code into separate PRs (about 450 non-test lines in total, plus tests): 1. Sliding-window `batch_read_multi` + cross-bucket batching in `BatchLoad`. This works without the other two parts and is useful on its own. 2. Dedicated read threads + serialized registration, on top of #4862 (or including the chunking if #4862 changes direction). Each PR adds unit tests for >1 GiB fixed buffers, multi-file batch reads, concurrent callers on read workers, and multi-bucket `BatchLoad` over uring. #### Questions for maintainers - Is a small dedicated read-thread pool acceptable in `UringFile`, or would you prefer the reads to stay on caller threads? One alternative is to fall back to #4788's clone approach when the kernel supports it. - Default for `MOONCAKE_OFFLOAD_URING_READ_THREADS`: should it be 8 when uring is enabled, or 0 (opt-in)? - Related but out of scope here: duplicate reads of the same keys from several TP ranks are covered by #2357 (see my comment there). ：） ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:24:39 +00:00

bsakaguchi commented

2026-10-29 21:24:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 27, 2026):

Thanks for opening this issue, @SnowLegend-star!

Field	Value
Issue	#4995
GitHub user ID	`60999643`
Reporter	@SnowLegend-star

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 27, 2026): Thanks for opening this issue, @SnowLegend-star! | Field | Value | |-------|-------| | **Issue** | #4995 | | **GitHub user ID** | `60999643` | | **Reporter** | @SnowLegend-star | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.