[PR #57] [MERGED] fix(snapshot): parallelize OSS layer validation #145

New issue

Closed

opened 2026-10-29 14:20:11 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:11 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/57
Author: @huajq
Created: 7/29/2026
Status: ✅ Merged
Merged: 7/29/2026
Merged by: @yingdi-shan

Base: main ← Head: perf/parallel-oss-layer-validation

📝 Commits (1)

86b6770 fix(snapshot): parallelize OSS layer validation

📊 Changes

1 file changed (+126 additions, -14 deletions)

View changed files

📝 src/snapshot/repository/backends/oss/resolver.rs (+126 -14)

📄 Description

Summary

validate OSS-managed snapshot layers with up to 16 concurrent HEAD requests
preserve eager validation before runtime image configuration is materialized
preserve existing missing-layer and backend error semantics
keep generated OverlayBD layer ordering unchanged

Motivation

The OSS runtime resolver previously validated every managed layer sequentially.

Snapshots with many rootfs and attached-drive layers therefore spent most of
their cold resolve time waiting for serialized OSS HEAD requests. A snapshot
with approximately 100 managed layers took 1.47–1.87 seconds in
load_snapshot before sandbox startup could begin.

Implementation

The OSS resolver now:

collects the original indexes and digests of managed layers;
validates their existence with buffer_unordered(16);
returns the same ArtifactNotFound error with the original layer index and
digest when an object is missing;
invokes the existing image materializer only after all validations succeed.

The shared runtime materializer and POSIX repository backend are unchanged.

Performance

Measured through the existing
agentenv_sandbox_stage_duration_seconds{operation="create_warm",stage="load_snapshot"}
metric with a fresh local snapshot cache:

Version	Cold `load_snapshot`
Before	1.47–1.87 s
After	201.6–234.3 ms

Warm-cache resolution was approximately 10 ms.

Validation

cargo test -p agentenv --lib snapshot::repository::backends::oss::resolver -- --nocapture
- 2 passed, 0 failed
cargo clippy -p agentenv --lib -- -D warnings
cargo build --release
cargo fmt --all -- --check
git diff --check

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/57 **Author:** [@huajq](https://github.com/huajq) **Created:** 7/29/2026 **Status:** ✅ Merged **Merged:** 7/29/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `perf/parallel-oss-layer-validation` --- ### 📝 Commits (1) - [`86b6770`](https://github.com/kvcache-ai/AgentENV/commit/86b67709f62f463e8b0cf3b6f6b7a31b44cb076c) fix(snapshot): parallelize OSS layer validation ### 📊 Changes **1 file changed** (+126 additions, -14 deletions) <details> <summary>View changed files</summary> 📝 `src/snapshot/repository/backends/oss/resolver.rs` (+126 -14) </details> ### 📄 Description ## Summary - validate OSS-managed snapshot layers with up to 16 concurrent `HEAD` requests - preserve eager validation before runtime image configuration is materialized - preserve existing missing-layer and backend error semantics - keep generated OverlayBD layer ordering unchanged ## Motivation The OSS runtime resolver previously validated every managed layer sequentially. Snapshots with many rootfs and attached-drive layers therefore spent most of their cold resolve time waiting for serialized OSS `HEAD` requests. A snapshot with approximately 100 managed layers took 1.47–1.87 seconds in `load_snapshot` before sandbox startup could begin. ## Implementation The OSS resolver now: 1. collects the original indexes and digests of managed layers; 2. validates their existence with `buffer_unordered(16)`; 3. returns the same `ArtifactNotFound` error with the original layer index and digest when an object is missing; 4. invokes the existing image materializer only after all validations succeed. The shared runtime materializer and POSIX repository backend are unchanged. ## Performance Measured through the existing `agentenv_sandbox_stage_duration_seconds{operation="create_warm",stage="load_snapshot"}` metric with a fresh local snapshot cache: | Version | Cold `load_snapshot` | | --- | ---: | | Before | 1.47–1.87 s | | After | 201.6–234.3 ms | Warm-cache resolution was approximately 10 ms. ## Validation - `cargo test -p agentenv --lib snapshot::repository::backends::oss::resolver -- --nocapture` - 2 passed, 0 failed - `cargo clippy -p agentenv --lib -- -D warnings` - `cargo build --release` - `cargo fmt --all -- --check` - `git diff --check` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:11 +00:00