[PR #244] [MERGED] fix(overlaybd): restore premerged index prune offload lost in a rebase #283

New issue

Closed

opened 2026-10-29 14:21:48 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:48 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/244
Author: @YXalix
Created: 9/3/2026
Status: ✅ Merged
Merged: 9/3/2026
Merged by: @yingdi-shan

Base: main ← Head: fix/restore-premerged-prune-offload

📝 Commits (1)

a90934d fix(overlaybd): restore premerged index prune offload lost in a rebase

📊 Changes

2 files changed (+224 additions, -24 deletions)

View changed files

📝 storage/overlaybd/src/lsmt/file/helper.rs (+164 -24)
📝 storage/overlaybd/src/lsmt/file/tests.rs (+60 -0)

📄 Description

Problem

uvm-ublk-daemon RSS keeps climbing (observed close to 1 TiB) once the premerged-index cache directory accumulates hundreds of thousands of artifacts, and it stays high long after every sandbox has exited. The directory itself is only ~4 GiB.

Root cause: the prune offload from #179 was silently reverted by #123. The auth branch in #123 was authored before #179 merged; when it was rebased onto main before merging, the rebase conflict in storage/overlaybd/src/lsmt/file/helper.rs was resolved with the branch's stale copy. The revert is visible in #123's own diff as helper.rs +24/-156 and tests.rs -54 — an exact mirror of #179 — hidden inside a 24-file auth PR. Current main therefore runs the pre-#179 code.

Why the old code leaks memory

The per-key lock only deduplicates writes for the same cache key. Every pause seals a new upper layer (new UUID), so every layer stack gets a new key — hundreds of thousands of artifacts means hundreds of thousands of distinct keys, each spawning its own detached write task with no global bound:

Each spawned task captures the whole merged Arc<ReadOnlyIndex> (~24 B/mapping; page-granular memory-snapshot stacks can reach hundreds of MB) and pins it until the task actually finishes.
Each task then runs a full-directory prune via async tokio::fs: one blocking-pool round-trip per directory entry. With N tasks and N entries that is O(N²) blocking work, which the 512-thread blocking pool can never drain.
Tasks are detached, so sandbox teardown cancels nothing; the backlog drains at a crawl after all sandboxes exit, so RSS stays elevated. The directory overshooting the max_dir_bytes cap (≤1 GiB configured, 4 GiB observed) confirms prune was no longer keeping up.

Fix

Restore the #179 implementation on top of current main (conflict-free reverse-apply of the #123 hunks for the two lsmt/file files):

Prune scan runs as a single spawn_blocking instead of one async round-trip per entry.
Prune is triggered once per 1/8 of max_dir_bytes of new writes, at most one scan per directory at a time; growth charged during a scan chains a follow-up scan.
The merged index is dropped before the lock-release/prune tail, so queued writes stop pinning it.

Later changes to the same file from #3522df8 / #e97064f (sys-module refactor, unaligned-extent rejection) are preserved — the restore only touches the premerged sections.

Verification

cargo test -p overlaybd premerged — 8 passed, including the restored prune-accounting tests from #179.
cargo fmt -p overlaybd -- --check and cargo clippy -p overlaybd --all-targets clean.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/244 **Author:** [@YXalix](https://github.com/YXalix) **Created:** 9/3/2026 **Status:** ✅ Merged **Merged:** 9/3/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `fix/restore-premerged-prune-offload` --- ### 📝 Commits (1) - [`a90934d`](https://github.com/kvcache-ai/AgentENV/commit/a90934de2797178d15849a5453400ca35498a82a) fix(overlaybd): restore premerged index prune offload lost in a rebase ### 📊 Changes **2 files changed** (+224 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `storage/overlaybd/src/lsmt/file/helper.rs` (+164 -24) 📝 `storage/overlaybd/src/lsmt/file/tests.rs` (+60 -0) </details> ### 📄 Description ## Problem `uvm-ublk-daemon` RSS keeps climbing (observed close to 1 TiB) once the premerged-index cache directory accumulates hundreds of thousands of artifacts, and it stays high long after every sandbox has exited. The directory itself is only ~4 GiB. Root cause: the prune offload from #179 was **silently reverted by #123**. The auth branch in #123 was authored before #179 merged; when it was rebased onto main before merging, the rebase conflict in `storage/overlaybd/src/lsmt/file/helper.rs` was resolved with the branch's stale copy. The revert is visible in #123's own diff as `helper.rs +24/-156` and `tests.rs -54` — an exact mirror of #179 — hidden inside a 24-file auth PR. Current `main` therefore runs the pre-#179 code. ## Why the old code leaks memory The per-key lock only deduplicates writes for the *same* cache key. Every pause seals a new upper layer (new UUID), so every layer stack gets a *new* key — hundreds of thousands of artifacts means hundreds of thousands of distinct keys, each spawning its own detached write task with no global bound: 1. Each spawned task captures the whole merged `Arc<ReadOnlyIndex>` (~24 B/mapping; page-granular memory-snapshot stacks can reach hundreds of MB) and pins it until the task actually finishes. 2. Each task then runs a full-directory prune via async `tokio::fs`: one blocking-pool round-trip **per directory entry**. With N tasks and N entries that is O(N²) blocking work, which the 512-thread blocking pool can never drain. 3. Tasks are detached, so sandbox teardown cancels nothing; the backlog drains at a crawl after all sandboxes exit, so RSS stays elevated. The directory overshooting the `max_dir_bytes` cap (≤1 GiB configured, 4 GiB observed) confirms prune was no longer keeping up. ## Fix Restore the #179 implementation on top of current main (conflict-free reverse-apply of the #123 hunks for the two `lsmt/file` files): - Prune scan runs as a single `spawn_blocking` instead of one async round-trip per entry. - Prune is triggered once per 1/8 of `max_dir_bytes` of new writes, at most one scan per directory at a time; growth charged during a scan chains a follow-up scan. - The merged index is dropped before the lock-release/prune tail, so queued writes stop pinning it. Later changes to the same file from #3522df8 / #e97064f (sys-module refactor, unaligned-extent rejection) are preserved — the restore only touches the premerged sections. ## Verification - `cargo test -p overlaybd premerged` — 8 passed, including the restored prune-accounting tests from #179. - `cargo fmt -p overlaybd -- --check` and `cargo clippy -p overlaybd --all-targets` clean. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:48 +00:00