[PR #39] [MERGED] perf: isolate benchmark and e2e dependencies #134

New issue

Closed

opened 2026-10-29 14:20:04 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:04 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/39
Author: @NickNYU
Created: 7/28/2026
Status: ✅ Merged
Merged: 7/30/2026
Merged by: @LSX-s-Software

Base: main ← Head: autoresearch/issue-8-p1

📝 Commits (2)

8e325df perf: keep heavyweight tooling out of unit-test builds
eaef527 Update crates/benchmarks/Cargo.toml

📊 Changes

12 files changed (+105 additions, -36 deletions)

View changed files

📝 Cargo.lock (+33 -2)
📝 Cargo.toml (+2 -22)
📝 Makefile (+6 -6)
➕ crates/benchmarks/Cargo.toml (+41 -0)
📝 crates/benchmarks/benches/oci_conversion_pipeline_benchmark.rs (+0 -0)
📝 crates/benchmarks/benches/orchestrator_store_benchmark.rs (+0 -0)
📝 crates/benchmarks/benches/snapshot_benchmark.rs (+0 -0)
📝 crates/benchmarks/benches/ublk_overlaybd_benchmark.rs (+0 -0)
➕ crates/e2e-tests/Cargo.toml (+18 -0)
📝 crates/e2e-tests/tests/snapshot_oss_e2e_test.rs (+3 -4)
📝 docs/src/internals/sandbox-testing.md (+1 -1)
📝 docs/src/internals/template-builder-testing.md (+1 -1)

📄 Description

Summary

move the four Criterion benchmarks into a dedicated agentenv-benchmarks workspace package
move the Docker/MinIO OSS snapshot test into agentenv-e2e-tests
keep the existing Makefile entry points while removing Criterion and test-support from the root package dev-dependencies
update internal documentation for the relocated OSS test

Part of #8 (P1).

Why

Cargo activates a package's dev-dependencies for library test builds even when the test target does not use them. Keeping Criterion and agentenv-test-support on the root package caused cargo test -p agentenv --lib --no-run to compile benchmark tooling, AWS SDK crates, and testcontainers.

Dedicated workspace packages preserve the benchmark/e2e targets while moving those costs behind explicit package selection.

Measured result

Same isolated VM for both runs: Ubuntu 22.04 ARM64, 8 vCPU, 16 GiB RAM, rustc 1.97.1. Commands used a fresh target directory and a warm Cargo source cache.

`cargo test -p agentenv --lib --no-run --locked`	Before	After	Change
wall time	243.41s	221.34s	-22.07s (-9.1%)
target size	7.4 GiB	6.6 GiB	about -0.8 GiB
compiler artifacts	749	649	-100 (-13.4%)

The compiler-artifact gate found all six P1 dependencies before the change and none afterward:

criterion
agentenv-test-support
aws-sdk-s3
aws-config
testcontainers
testcontainers-modules

The ARM64 timings are screening data rather than a replacement for the issue's x86_64 baseline; the dependency isolation result itself comes from Cargo's compiled artifact stream and dependency tree.

Verification

cargo fmt --all -- --check
cargo test -p agentenv --lib --locked — 633 passed, 0 failed, 2 ignored
cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test --no-run --locked
cargo test -p agentenv-benchmarks --benches --no-run --locked — 4 executables
cargo clippy -p agentenv -p agentenv-benchmarks -p agentenv-e2e-tests --all-targets --locked -- -D warnings
root dependency-tree gate — 0 forbidden dependencies
Makefile dry-run for benchmark and OSS e2e entry points

Not run

Docker/MinIO-backed ignored e2e tests at runtime
KVM/capability-dependent benchmark execution

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/39 **Author:** [@NickNYU](https://github.com/NickNYU) **Created:** 7/28/2026 **Status:** ✅ Merged **Merged:** 7/30/2026 **Merged by:** [@LSX-s-Software](https://github.com/LSX-s-Software) **Base:** `main` ← **Head:** `autoresearch/issue-8-p1` --- ### 📝 Commits (2) - [`8e325df`](https://github.com/kvcache-ai/AgentENV/commit/8e325df1a781ca289fbf9b6de6b9d0c3057575f0) perf: keep heavyweight tooling out of unit-test builds - [`eaef527`](https://github.com/kvcache-ai/AgentENV/commit/eaef527d3a7b6671cbbc05316f9f65c5c02fdd4c) Update crates/benchmarks/Cargo.toml ### 📊 Changes **12 files changed** (+105 additions, -36 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+33 -2) 📝 `Cargo.toml` (+2 -22) 📝 `Makefile` (+6 -6) ➕ `crates/benchmarks/Cargo.toml` (+41 -0) 📝 `crates/benchmarks/benches/oci_conversion_pipeline_benchmark.rs` (+0 -0) 📝 `crates/benchmarks/benches/orchestrator_store_benchmark.rs` (+0 -0) 📝 `crates/benchmarks/benches/snapshot_benchmark.rs` (+0 -0) 📝 `crates/benchmarks/benches/ublk_overlaybd_benchmark.rs` (+0 -0) ➕ `crates/e2e-tests/Cargo.toml` (+18 -0) 📝 `crates/e2e-tests/tests/snapshot_oss_e2e_test.rs` (+3 -4) 📝 `docs/src/internals/sandbox-testing.md` (+1 -1) 📝 `docs/src/internals/template-builder-testing.md` (+1 -1) </details> ### 📄 Description ## Summary - move the four Criterion benchmarks into a dedicated `agentenv-benchmarks` workspace package - move the Docker/MinIO OSS snapshot test into `agentenv-e2e-tests` - keep the existing Makefile entry points while removing Criterion and test-support from the root package dev-dependencies - update internal documentation for the relocated OSS test Part of #8 (P1). ## Why Cargo activates a package's dev-dependencies for library test builds even when the test target does not use them. Keeping Criterion and `agentenv-test-support` on the root package caused `cargo test -p agentenv --lib --no-run` to compile benchmark tooling, AWS SDK crates, and testcontainers. Dedicated workspace packages preserve the benchmark/e2e targets while moving those costs behind explicit package selection. ## Measured result Same isolated VM for both runs: Ubuntu 22.04 ARM64, 8 vCPU, 16 GiB RAM, rustc 1.97.1. Commands used a fresh target directory and a warm Cargo source cache. | `cargo test -p agentenv --lib --no-run --locked` | Before | After | Change | | --- | ---: | ---: | ---: | | wall time | 243.41s | 221.34s | -22.07s (-9.1%) | | target size | 7.4 GiB | 6.6 GiB | about -0.8 GiB | | compiler artifacts | 749 | 649 | -100 (-13.4%) | The compiler-artifact gate found all six P1 dependencies before the change and none afterward: - `criterion` - `agentenv-test-support` - `aws-sdk-s3` - `aws-config` - `testcontainers` - `testcontainers-modules` The ARM64 timings are screening data rather than a replacement for the issue's x86_64 baseline; the dependency isolation result itself comes from Cargo's compiled artifact stream and dependency tree. ## Verification - `cargo fmt --all -- --check` - `cargo test -p agentenv --lib --locked` — 633 passed, 0 failed, 2 ignored - `cargo test -p agentenv-e2e-tests --test snapshot_oss_e2e_test --no-run --locked` - `cargo test -p agentenv-benchmarks --benches --no-run --locked` — 4 executables - `cargo clippy -p agentenv -p agentenv-benchmarks -p agentenv-e2e-tests --all-targets --locked -- -D warnings` - root dependency-tree gate — 0 forbidden dependencies - Makefile dry-run for benchmark and OSS e2e entry points ## Not run - Docker/MinIO-backed ignored e2e tests at runtime - KVM/capability-dependent benchmark execution --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:04 +00:00