[PR #13] [MERGED] feat(overlaybd): concurrent coalesced zfile read path with read-path metrics #117

New issue

Closed

opened 2026-10-29 14:19:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/13
Author: @huajq
Created: 7/27/2026
Status: ✅ Merged
Merged: 7/27/2026
Merged by: @huajq

Base: main ← Head: zfile

📝 Commits (1)

a3318e3 feat(overlaybd): concurrent coalesced zfile read path with read-path metrics

📊 Changes

8 files changed (+2244 additions, -326 deletions)

View changed files

📝 Cargo.lock (+14 -1)
📝 crates/observability/src/lib.rs (+25 -1)
📝 storage/overlaybd/Cargo.toml (+8 -1)
➕ storage/overlaybd/benches/zfile_read_benchmark.rs (+778 -0)
📝 storage/overlaybd/src/backend/cache/meta.rs (+2 -5)
📝 storage/overlaybd/src/compression/zfile.rs (+1097 -309)
📝 storage/overlaybd/src/metrics.rs (+314 -1)
📝 storage/overlaybd/src/prefetch.rs (+6 -8)

📄 Description

Summary

ZFileRO reads no longer serialize on a whole-request mutex: pread takes &self with a Sync compressor, so reads on one shared handle overlap in the backing file.
Encoded blocks are fetched in coalesced batches (up to 64 KiB) with one merged backing read per batch; a block failing CRC or decode is evicted and re-read on its own (at most 3 retries).
The merged batch buffer is pooled per thread and no longer zero-filled before being fully overwritten.
Per-pread exact counters (output/encoded bytes, blocks, batches, retries) plus 1-in-1024 sampled duration histograms, recorded through lazily pre-registered metric handles; the two duration histograms get fine-grained Prometheus buckets.
CRC-32C moves from crc to crc32c (SSE4.2) in zfile, cache meta, and prefetch traces.
New zfile_read criterion benchmark (read throughput, block size, compression throughput), using the rand 0.10 API (RngExt) to match this repo's dev-dependency.

Benchmark

Xeon Platinum 8269CY, kernel 6.8, in-memory instrumented backing, criterion mean.

Read throughput

case	time	throughput
`lz4_random4k_qd1`	1.254 µs	3.04 GiB/s
`zstd_random4k_qd1`	1.304 µs	2.93 GiB/s
`lz4_compressible_random4k_qd1`	494 ns	7.75 GiB/s
`zstd_compressible_random4k_qd1`	642 ns	5.95 GiB/s
`lz4_seq128k_qd1`	35.85 µs	3.41 GiB/s
`zstd_seq128k_qd1`	37.90 µs	3.22 GiB/s
`lz4_random4k_qd16_shared`	20.6 µs/op (1.29 µs per 4 KiB page)	2.95 GiB/s
`zstd_random4k_qd16_shared`	21.7 µs/op (1.36 µs per 4 KiB page)	2.80 GiB/s

QD16 on one shared handle costs the same per page as QD1 (1.29 vs 1.25 µs), i.e. 16 concurrent reads overlap in the backing file instead of serializing.

Block size (4 KiB random read)

case	time
`lz4_random4k_qd1_bs4k`	1.60 µs
`lz4_random4k_qd1_bs64k`	20.1 µs (reads a whole 64 KiB block per 4 KiB request)

Memory compression ratio

Compressed memory image size relative to the original (lower is better):

Memory state	LZ4	ZSTD
Idle	32.68%	21.78%
Zero-page / random-page mix	34.63%	32.52%

Verification

cargo test -p overlaybd --lib: 302 passed, 0 failed
cargo clippy -p overlaybd --all-targets -- -D warnings: clean

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/13 **Author:** [@huajq](https://github.com/huajq) **Created:** 7/27/2026 **Status:** ✅ Merged **Merged:** 7/27/2026 **Merged by:** [@huajq](https://github.com/huajq) **Base:** `main` ← **Head:** `zfile` --- ### 📝 Commits (1) - [`a3318e3`](https://github.com/kvcache-ai/AgentENV/commit/a3318e372689a40fd89e701b406f42b949684cc8) feat(overlaybd): concurrent coalesced zfile read path with read-path metrics ### 📊 Changes **8 files changed** (+2244 additions, -326 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+14 -1) 📝 `crates/observability/src/lib.rs` (+25 -1) 📝 `storage/overlaybd/Cargo.toml` (+8 -1) ➕ `storage/overlaybd/benches/zfile_read_benchmark.rs` (+778 -0) 📝 `storage/overlaybd/src/backend/cache/meta.rs` (+2 -5) 📝 `storage/overlaybd/src/compression/zfile.rs` (+1097 -309) 📝 `storage/overlaybd/src/metrics.rs` (+314 -1) 📝 `storage/overlaybd/src/prefetch.rs` (+6 -8) </details> ### 📄 Description ## Summary - `ZFileRO` reads no longer serialize on a whole-request mutex: `pread` takes `&self` with a `Sync` compressor, so reads on one shared handle overlap in the backing file. - Encoded blocks are fetched in coalesced batches (up to 64 KiB) with one merged backing read per batch; a block failing CRC or decode is evicted and re-read on its own (at most 3 retries). - The merged batch buffer is pooled per thread and no longer zero-filled before being fully overwritten. - Per-pread exact counters (output/encoded bytes, blocks, batches, retries) plus 1-in-1024 sampled duration histograms, recorded through lazily pre-registered metric handles; the two duration histograms get fine-grained Prometheus buckets. - CRC-32C moves from `crc` to `crc32c` (SSE4.2) in zfile, cache meta, and prefetch traces. - New `zfile_read` criterion benchmark (read throughput, block size, compression throughput), using the rand 0.10 API (`RngExt`) to match this repo's dev-dependency. ## Benchmark Xeon Platinum 8269CY, kernel 6.8, in-memory instrumented backing, criterion mean. ### Read throughput | case | time | throughput | |---|---:|---:| | `lz4_random4k_qd1` | 1.254 µs | 3.04 GiB/s | | `zstd_random4k_qd1` | 1.304 µs | 2.93 GiB/s | | `lz4_compressible_random4k_qd1` | 494 ns | 7.75 GiB/s | | `zstd_compressible_random4k_qd1` | 642 ns | 5.95 GiB/s | | `lz4_seq128k_qd1` | 35.85 µs | 3.41 GiB/s | | `zstd_seq128k_qd1` | 37.90 µs | 3.22 GiB/s | | `lz4_random4k_qd16_shared` | 20.6 µs/op (1.29 µs per 4 KiB page) | 2.95 GiB/s | | `zstd_random4k_qd16_shared` | 21.7 µs/op (1.36 µs per 4 KiB page) | 2.80 GiB/s | QD16 on one shared handle costs the same per page as QD1 (1.29 vs 1.25 µs), i.e. 16 concurrent reads overlap in the backing file instead of serializing. ### Block size (4 KiB random read) | case | time | |---|---:| | `lz4_random4k_qd1_bs4k` | 1.60 µs | | `lz4_random4k_qd1_bs64k` | 20.1 µs (reads a whole 64 KiB block per 4 KiB request) | ### Memory compression ratio Compressed memory image size relative to the original (lower is better): | Memory state | LZ4 | ZSTD | |---|---:|---:| | Idle | 32.68% | 21.78% | | Zero-page / random-page mix | 34.63% | 32.52% | ## Verification - `cargo test -p overlaybd --lib`: 302 passed, 0 failed - `cargo clippy -p overlaybd --all-targets -- -D warnings`: clean --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:19:53 +00:00