[GH-ISSUE #4497] [RFC]: [Store] Batch 3FS adapter I/O through USRBIO rings #1143

New issue

Open

opened 2026-10-29 21:23:17 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:17 +00:00

Owner

Originally created by @2xdevv on GitHub (Oct 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4497

At baseline 0d1a8040, the descriptor-based DistributedStorageBackend::BatchWrite and BatchRead paths issue individual WriteAt/ReadAt calls. The HF3FS adapter prepares, submits, and waits for one chunk at a time. As a result, a batch of independent object requests does not keep multiple requests in flight on the USRBIO ring.

I propose preserving the batch through FileSystemAdapter and processing it in bounded waves in Hf3fsAdapter. The caller-facing API remains synchronous and returns one result per request in input order.

The implementation is split into two PRs:

Pass descriptor batches through the adapter. Add FdIoRequest, BatchWriteAt, BatchReadAt, and an opt-in SupportsBatchIo capability. Default batch methods call the existing positional operations sequentially. For the HF3FS path, reuse bucket handles within groups bounded to 64 open files and allow concurrent positional I/O on cached shard descriptors. POSIX retains its existing execution behavior. Implementation: PR #4498.
Submit multiple requests through USRBIO rings. Build on the first change and use the per-thread read/write rings and shared staging buffer. Give each in-flight request a distinct buffer slice, split large requests and batches into bounded waves, and map completions back to their original requests. Preserve full-transfer-or-error behavior, drain prepared requests on submission failure, and retire thread resources after unrecoverable completion errors. Single positional operations use the same implementation. Implementation: feat/hf3fs-adapter-batching, commit 58cb4452.

Correctness checks completed:

All 11 fake-USRBIO tests passed under ASan/UBSan, with leak checking disabled. These cover completion reordering, ring pressure, chunking, short transfers, error handling, and concurrent callers.
All five native integration tests on the updated batching stack passed against a local 3FS mount. Coverage includes scatter/gather I/O, requests exceeding ring and staging-buffer capacities, 65 bucket files with bounded handle reuse, and concurrent batches sharing a shard.
Verified writes, reads, and concurrent reads passed for POSIX and HF3FS with both allocation layouts. Separate correctness runs on the measurement cluster also passed.

The backend-only performance comparison uses the original production code at 0d1a8040 versus the combined batching implementation at the earlier commit 791f2046. Both binaries used byte-identical benchmark source and the same 3FS SDK library. These measurements cover the combined effect of both PRs.

Measurements use storage_backend_bench --backend=dfs --dfs_adapter=hf3fs, with --dfs_allocator=shard or bucket and --test=offload, load, or concurrent_load. The benchmark allocates DFS descriptors and directly invokes DistributedStorageBackend::BatchWrite and BatchRead, which perform data I/O through Hf3fsAdapter and the native USRBIO API. It does not start a Mooncake master, perform client replica selection, or read from Mooncake MEMORY replicas.

Initialization and descriptor allocation are outside the measurement window. For load and concurrent_load, the benchmark first writes the dataset on one thread, then completes read warmup before starting the timed reads. Backend timing covers the BatchWrite/BatchRead calls; wall timing covers the measured worker loops, including request preparation and result checking, across all participating threads. Each original/candidate pair uses the same workload settings.

These tests do not force cold caches inside 3FS. Prefill may warm storage-side caches, and using the same prefill procedure does not establish identical cache residency between runs. The reported read gains therefore describe read-after-prefill behavior, not reads served from Mooncake memory replicas or a controlled cold-storage comparison.

Measurement environment: one client host with an AMD EPYC 9534, 128 logical CPUs, 755 GiB RAM, and a mounted 3FS cluster. The binaries were built on a separate host with GCC 13.3.0.

Workload	Values	Batch sizes	Threads	Paired repetitions
Shard and bucket, write and read	1, 4, 8, 16 MiB	1, 4, 16, 64	1	3–5 per configuration
Shard and bucket, concurrent read	4, 16 MiB	16	2, 4, 8	3 per configuration

There are 76 configurations and 688 runs in total. Every run exited successfully with zero reported I/O errors. Timing runs used --verify=false; payload verification was performed separately in the correctness gate and verification rechecks. The new 4- and 8-thread timing cases were not individually rerun with payload verification enabled.

Single-thread results below use one thread and batch size 16, matching the batch size used for concurrent reads. Each cell is the median of 3–5 paired candidate/original wall-throughput ratios for that value size, operation, and layout. Read corresponds to load; write corresponds to offload. These measurements cover both batching changes together.

Value size	Shard read	Shard write	Bucket read	Bucket write
1 MiB	4.20×	2.71×	10.96×	4.96×
4 MiB	2.12×	2.69×	4.86×	3.39×
8 MiB	1.51×	2.71×	2.98×	2.61×
16 MiB	1.18×	1.55×	2.01×	1.71×

Across the full single-thread sweep (batch sizes 4, 16, and 64), all 230 paired comparisons improved wall throughput. The separate batch-1 control has a pooled median wall-throughput ratio of 0.996×, but it does not establish absence of regressions: shard writes at 1 MiB and bucket writes at 8 MiB have per-configuration median wall ratios of 0.931× and 0.942×, respectively.

For concurrent reads at batch 16, each entry below is the median of three paired wall-throughput ratios:

Layout	Value size	2 threads	4 threads	8 threads
Shard	4 MiB	2.09×	2.34×	3.14×
Shard	16 MiB	1.31×	1.45×	2.09×
Bucket	4 MiB	3.87×	3.32×	3.19×
Bucket	16 MiB	1.88×	1.86×	1.69×

All 36 concurrent paired comparisons improved wall throughput, with a pooled median of 2.09× and a range of 1.29–3.94×. The relative gain increases with thread count for the tested shard cases and decreases for the tested bucket cases.

Reads are measured after a serial dataset prefill outside the timing window. These results describe that workload; they do not establish cold-storage performance or a storage bandwidth ceiling. Large-batch cases also have limited latency samples, so the throughput results are stronger evidence than tail-latency estimates.

Real-client benchmark: `store_kv_bench`

Additional measurements use the Python client benchmark mooncake-store/benchmarks/store_kv_bench.py, with DFS support from PR #4465, pinned at 08868466. Both variants use the same unmodified benchmark script. The comparison is original production code at 0d1a8040 versus the combined batching implementation at 58cb4452, so these results cover both batching changes together.

The client runs write_perf and read_perf with --io-api=zcopy, one worker (--numjobs=1), --iodepth=1, and batch size 16. Each phase processes 8 GiB: 8,192 / 2,048 / 1,024 / 512 objects for 1 / 4 / 8 / 16 MiB values, respectively. There are three paired repetitions per configuration, alternating original/candidate execution order by repetition. Both Release builds were prepared on the same build host with GCC 12.4.0 and load the same runtime 3FS SDK on the measurement host described above.

Each case starts a local Mooncake master and uses --protocol=tcp; DFS data I/O uses the native 3FS USRBIO API. Each write requests one memory replica and one DFS replica. After the writer exits, a fresh reader with no global memory segment checks that every key has exactly one complete DFS replica and zero memory replicas, then uses the existing benchmark read loop on those keys. NoF, automatic offload, promotion, and DFS eviction are disabled. Writes therefore include the work of creating both replicas, while reads exercise the DFS path. Throughput counts logical payload bytes once.

Each cell below is the median of three paired candidate/original logical wall-throughput ratios:

Value size	Shard read	Shard write	Bucket read	Bucket write
1 MiB	2.48×	2.04×	7.17×	3.75×
4 MiB	1.45×	2.16×	3.14×	2.89×
8 MiB	1.54×	2.10×	2.24×	2.54×
16 MiB	1.15×	1.37×	1.83×	1.63×

All 48 paired batch-16 read/write comparisons improved throughput. The full reduced suite contains 60 timing runs, including the two batch-1 controls, plus 20 separate correctness-gate runs. All completed successfully. Timing runs disable payload verification and tracing; the gates verify 32 objects per case (640 verified reads in total). Native gate traces show 16 prepared I/Os per submit for the candidate's batch-16 cases versus one for the original. Larger values require multiple waves of chunks.

The batch-1 controls do not establish absence of regressions: shard 1-MiB read/write median ratios are 0.976× / 1.024×, and bucket 8-MiB read/write ratios are 1.019× / 0.990×. The shard-read ratios range from 0.816× to 0.980× over three pairs; other batch-1 sizes were not included in this client suite.

Reads follow prefill and may benefit from 3FS caches. All eight primary read configurations include phases below five seconds (shortest 2.28 seconds), limiting tail-latency conclusions. The benchmark's zcopy path still copies payloads into registered write buffers and materializes read bytes; those costs remain in the timings. This is a CPU client benchmark, not a vLLM/GPU workload.

Related work: RFC #2426 discusses broader DFS architecture, allocation, and replica semantics. This proposal focuses on executing existing descriptor batches through the adapter and USRBIO rings.

Feedback on the adapter capability contract, bounded resource handling, and the two-step PR split would be helpful.

Originally created by @2xdevv on GitHub (Oct 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4497 At baseline `0d1a8040`, the descriptor-based `DistributedStorageBackend::BatchWrite` and `BatchRead` paths issue individual `WriteAt`/`ReadAt` calls. The HF3FS adapter prepares, submits, and waits for one chunk at a time. As a result, a batch of independent object requests does not keep multiple requests in flight on the USRBIO ring. I propose preserving the batch through `FileSystemAdapter` and processing it in bounded waves in `Hf3fsAdapter`. The caller-facing API remains synchronous and returns one result per request in input order. The implementation is split into two PRs: 1. **Pass descriptor batches through the adapter.** Add `FdIoRequest`, `BatchWriteAt`, `BatchReadAt`, and an opt-in `SupportsBatchIo` capability. Default batch methods call the existing positional operations sequentially. For the HF3FS path, reuse bucket handles within groups bounded to 64 open files and allow concurrent positional I/O on cached shard descriptors. POSIX retains its existing execution behavior. Implementation: [PR #4498](https://github.com/kvcache-ai/Mooncake/pull/4498). 2. **Submit multiple requests through USRBIO rings.** Build on the first change and use the per-thread read/write rings and shared staging buffer. Give each in-flight request a distinct buffer slice, split large requests and batches into bounded waves, and map completions back to their original requests. Preserve full-transfer-or-error behavior, drain prepared requests on submission failure, and retire thread resources after unrecoverable completion errors. Single positional operations use the same implementation. Implementation: [`feat/hf3fs-adapter-batching`](https://github.com/2xdevv/Mooncake/tree/feat/hf3fs-adapter-batching), commit `58cb4452`. Correctness checks completed: - All 11 fake-USRBIO tests passed under ASan/UBSan, with leak checking disabled. These cover completion reordering, ring pressure, chunking, short transfers, error handling, and concurrent callers. - All five native integration tests on the updated batching stack passed against a local 3FS mount. Coverage includes scatter/gather I/O, requests exceeding ring and staging-buffer capacities, 65 bucket files with bounded handle reuse, and concurrent batches sharing a shard. - Verified writes, reads, and concurrent reads passed for POSIX and HF3FS with both allocation layouts. Separate correctness runs on the measurement cluster also passed. The backend-only performance comparison uses the original production code at `0d1a8040` versus the combined batching implementation at the earlier commit `791f2046`. Both binaries used byte-identical benchmark source and the same 3FS SDK library. These measurements cover the combined effect of both PRs. Measurements use `storage_backend_bench --backend=dfs --dfs_adapter=hf3fs`, with `--dfs_allocator=shard` or `bucket` and `--test=offload`, `load`, or `concurrent_load`. The benchmark allocates DFS descriptors and directly invokes `DistributedStorageBackend::BatchWrite` and `BatchRead`, which perform data I/O through `Hf3fsAdapter` and the native USRBIO API. It does not start a Mooncake master, perform client replica selection, or read from Mooncake `MEMORY` replicas. Initialization and descriptor allocation are outside the measurement window. For `load` and `concurrent_load`, the benchmark first writes the dataset on one thread, then completes read warmup before starting the timed reads. Backend timing covers the `BatchWrite`/`BatchRead` calls; wall timing covers the measured worker loops, including request preparation and result checking, across all participating threads. Each original/candidate pair uses the same workload settings. These tests do not force cold caches inside 3FS. Prefill may warm storage-side caches, and using the same prefill procedure does not establish identical cache residency between runs. The reported read gains therefore describe read-after-prefill behavior, not reads served from Mooncake memory replicas or a controlled cold-storage comparison. Measurement environment: one client host with an AMD EPYC 9534, 128 logical CPUs, 755 GiB RAM, and a mounted 3FS cluster. The binaries were built on a separate host with GCC 13.3.0. | Workload | Values | Batch sizes | Threads | Paired repetitions | |---|---|---|---|---| | Shard and bucket, write and read | 1, 4, 8, 16 MiB | 1, 4, 16, 64 | 1 | 3–5 per configuration | | Shard and bucket, concurrent read | 4, 16 MiB | 16 | 2, 4, 8 | 3 per configuration | There are 76 configurations and 688 runs in total. Every run exited successfully with zero reported I/O errors. Timing runs used `--verify=false`; payload verification was performed separately in the correctness gate and verification rechecks. The new 4- and 8-thread timing cases were not individually rerun with payload verification enabled. Single-thread results below use **one thread and batch size 16**, matching the batch size used for concurrent reads. Each cell is the median of 3–5 paired candidate/original **wall-throughput ratios** for that value size, operation, and layout. Read corresponds to `load`; write corresponds to `offload`. These measurements cover both batching changes together. | Value size | Shard read | Shard write | Bucket read | Bucket write | |---|---:|---:|---:|---:| | 1 MiB | 4.20× | 2.71× | 10.96× | 4.96× | | 4 MiB | 2.12× | 2.69× | 4.86× | 3.39× | | 8 MiB | 1.51× | 2.71× | 2.98× | 2.61× | | 16 MiB | 1.18× | 1.55× | 2.01× | 1.71× | Across the full single-thread sweep (batch sizes 4, 16, and 64), all 230 paired comparisons improved wall throughput. The separate batch-1 control has a pooled median wall-throughput ratio of 0.996×, but it does not establish absence of regressions: shard writes at 1 MiB and bucket writes at 8 MiB have per-configuration median wall ratios of 0.931× and 0.942×, respectively. For concurrent reads at batch 16, each entry below is the median of three paired wall-throughput ratios: | Layout | Value size | 2 threads | 4 threads | 8 threads | |---|---|---:|---:|---:| | Shard | 4 MiB | 2.09× | 2.34× | 3.14× | | Shard | 16 MiB | 1.31× | 1.45× | 2.09× | | Bucket | 4 MiB | 3.87× | 3.32× | 3.19× | | Bucket | 16 MiB | 1.88× | 1.86× | 1.69× | All 36 concurrent paired comparisons improved wall throughput, with a pooled median of 2.09× and a range of 1.29–3.94×. The relative gain increases with thread count for the tested shard cases and decreases for the tested bucket cases. Reads are measured after a serial dataset prefill outside the timing window. These results describe that workload; they do not establish cold-storage performance or a storage bandwidth ceiling. Large-batch cases also have limited latency samples, so the throughput results are stronger evidence than tail-latency estimates. ### Real-client benchmark: `store_kv_bench` Additional measurements use the Python client benchmark [`mooncake-store/benchmarks/store_kv_bench.py`](https://github.com/2xdevv/Mooncake/blob/08868466a0438bb9518d310016219ed2063a30cd/mooncake-store/benchmarks/store_kv_bench.py), with DFS support from [PR #4465](https://github.com/kvcache-ai/Mooncake/pull/4465), pinned at `08868466`. Both variants use the same unmodified benchmark script. The comparison is original production code at `0d1a8040` versus the combined batching implementation at `58cb4452`, so these results cover both batching changes together. The client runs `write_perf` and `read_perf` with **`--io-api=zcopy`, one worker (`--numjobs=1`), `--iodepth=1`, and batch size 16**. Each phase processes 8 GiB: 8,192 / 2,048 / 1,024 / 512 objects for 1 / 4 / 8 / 16 MiB values, respectively. There are three paired repetitions per configuration, alternating original/candidate execution order by repetition. Both Release builds were prepared on the same build host with GCC 12.4.0 and load the same runtime 3FS SDK on the measurement host described above. Each case starts a local Mooncake master and uses `--protocol=tcp`; DFS data I/O uses the native 3FS USRBIO API. Each write requests one memory replica and one DFS replica. After the writer exits, a fresh reader with no global memory segment checks that every key has exactly one complete DFS replica and zero memory replicas, then uses the existing benchmark read loop on those keys. NoF, automatic offload, promotion, and DFS eviction are disabled. Writes therefore include the work of creating both replicas, while reads exercise the DFS path. Throughput counts logical payload bytes once. Each cell below is the **median of three paired candidate/original logical wall-throughput ratios**: | Value size | Shard read | Shard write | Bucket read | Bucket write | |---|---:|---:|---:|---:| | 1 MiB | 2.48× | 2.04× | 7.17× | 3.75× | | 4 MiB | 1.45× | 2.16× | 3.14× | 2.89× | | 8 MiB | 1.54× | 2.10× | 2.24× | 2.54× | | 16 MiB | 1.15× | 1.37× | 1.83× | 1.63× | All 48 paired batch-16 read/write comparisons improved throughput. The full reduced suite contains 60 timing runs, including the two batch-1 controls, plus 20 separate correctness-gate runs. All completed successfully. Timing runs disable payload verification and tracing; the gates verify 32 objects per case (640 verified reads in total). Native gate traces show 16 prepared I/Os per submit for the candidate's batch-16 cases versus one for the original. Larger values require multiple waves of chunks. The batch-1 controls do not establish absence of regressions: shard 1-MiB read/write median ratios are 0.976× / 1.024×, and bucket 8-MiB read/write ratios are 1.019× / 0.990×. The shard-read ratios range from 0.816× to 0.980× over three pairs; other batch-1 sizes were not included in this client suite. Reads follow prefill and may benefit from 3FS caches. All eight primary read configurations include phases below five seconds (shortest 2.28 seconds), limiting tail-latency conclusions. The benchmark's `zcopy` path still copies payloads into registered write buffers and materializes read bytes; those costs remain in the timings. This is a CPU client benchmark, not a vLLM/GPU workload. Related work: [RFC #2426](https://github.com/kvcache-ai/Mooncake/issues/2426) discusses broader DFS architecture, allocation, and replica semantics. This proposal focuses on executing existing descriptor batches through the adapter and USRBIO rings. Feedback on the adapter capability contract, bounded resource handling, and the two-step PR split would be helpful.

bsakaguchi commented

2026-10-29 21:23:18 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Thanks for opening this issue, @2xdevv!

Field	Value
Issue	#4497
GitHub user ID	`261649551`
Reporter	@2xdevv

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 8, 2026): Thanks for opening this issue, @2xdevv! | Field | Value | |-------|-------| | **Issue** | #4497 | | **GitHub user ID** | `261649551` | | **Reporter** | @2xdevv | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.