[GH-ISSUE #8] Reduce Rust compile time and target artifact size #5

New issue

Open

opened 2026-10-29 14:18:16 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:18:16 +00:00

Owner

Copy link

Originally created by @LSX-s-Software on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/8

Background

Several common Rust build targets in AgentENV have become expensive to compile and produce very large artifacts. This affects local development feedback loops, CI runtime, disk usage, and the cost of switching between dev/test/release profiles.

A clean full rebuild is a useful way to expose the baseline cost, but the issue is broader than any particular local cleanup workflow. Even incremental builds can be slow when touching code inside the large root agentenv crate, and test builds produce multiple large binaries that quickly inflate target/.

Local measurements

Environment: 16 cores, 31 GiB RAM, rustc 1.97.0, current workspace.

The test targets below were measured with --no-run to isolate compile/link time from KVM, root permissions, network namespaces, or test runtime behavior.

Target	Command	Time
dev server	cargo build -p agentenv --bin server --timings	3m36s
unit tests	cargo test -p agentenv --lib --no-run --timings	3m16s
integration tests	cargo test -p agentenv --no-run --test integration --test orchestrator_integration --test snapshot_oss_e2e_test --timings	1m01s
release server	cargo build --release -p agentenv --bin server --timings	4m49s
no-op rebuild	same commands immediately repeated	~0.5s–0.8s
touch one root crate file	touch src/local_store.rs && cargo build -p agentenv --bin server	8.4s

target/ size after building the common dev/test/release targets:

target/                    21G
target/debug/              19G
target/debug/deps/         13G
target/debug/build/       1.1G
target/debug/incremental/ 5.0G
target/release/           2.1G
target/release/deps/      1.6G
target/release/build/     492M

Large artifacts observed:

852M target/debug/deps/agentenv-...             # unit test binary
813M target/debug/server                        # debug server
621M target/debug/deps/orchestrator_test-...
608M target/debug/deps/snapshot_test-...
606M target/debug/deps/snapshot_attached_drive_test-...
603M target/debug/deps/fc_test-...
602M target/debug/deps/ublk_test-...
598M target/debug/deps/process_test-...
598M target/debug/deps/envd_supervisor_test-...
463M target/debug/deps/libagentenv-....rlib
202M target/debug/deps/libaws_sdk_s3-....rlib
186M target/debug/deps/libiroh-....so
116M target/debug/deps/liboverlaybd-....rlib
111M target/debug/deps/libiroh_blobs-....rlib

Release server size:

target/release/server: 79M
stripped:              57M

Timing hotspots

Cold dev server build top items:

128.8s librocksdb-sys build-script
 57.9s tikv-jemalloc-sys build-script
 38.2s agentenv
 35.0s aws-lc-sys build-script
 28.2s overlaybd
 22.5s iroh-blobs
 21.2s iroh
 17.2s zstd-sys build-script
 16.7s lz4-sys build-script

Cold release server build top items:

154.7s librocksdb-sys build-script
 83.7s aws-lc-sys build-script
 51.2s zstd-sys build-script
 46.6s agentenv
 32.4s iroh
 31.8s tikv-jemalloc-sys build-script
 31.0s noq-proto
 28.9s iroh-blobs
 28.3s overlaybd
 27.9s rustls
 27.1s reqwest v0.13
 26.5s opendal
 25.0s reqwest v0.12

Initial analysis

This does not appear to be caused by a single crate. The cost is the result of several factors compounding:

1. The root agentenv crate is large and directly includes API, sandbox, snapshot, orchestrator, image, P2P, setup, and other subsystems. Touching code inside this crate can still require expensive incremental recompilation and relinking.
2. Heavy native build scripts dominate full rebuilds, especially librocksdb-sys, tikv-jemalloc-sys, aws-lc-sys, zstd-sys, and lz4-sys.
3. P2P/Iroh is currently an unconditional dependency, even though many dev and unit-test workflows do not need the Iroh backend.
4. Root agentenv dev-dependencies pull in benchmark and e2e dependencies. In particular, criterion and agentenv-test-support cause ordinary unit-test builds to compile AWS SDK, aws-config, testcontainers, and related dependencies.
5. The dependency graph is large and contains duplicated major/minor versions. Current metadata shows roughly 830 packages, about 791 transitive dependencies for agentenv, and around 75 duplicate crate names, including reqwest 0.12/0.13, hyper 0.14/1.x, rustls 0.21/0.23, bindgen 0.69/0.72, and netlink-packet-route 0.28/0.29/0.30.
6. dev/test/release profiles do not share artifacts, so each profile pays its own compilation and native build cost.
7. debug/test artifacts are very large due to debug info, large rlibs, repeated test binaries, and incremental caches.

Proposed direction

This should probably be handled as a build-performance and dependency-governance effort, split across several PRs.

P1: Isolate benchmark and e2e dev-dependencies (Done in #39)

Move Criterion benchmarks and OSS e2e/test-support code into separate workspace packages so that cargo test -p agentenv --lib does not compile AWS SDK, testcontainers, or Criterion.

Target: ordinary root-crate unit tests should not depend on:

• criterion
• agentenv-test-support
• aws-sdk-s3
• aws-config
• testcontainers
• testcontainers-modules

P2: Feature-gate P2P/Iroh

Make iroh and iroh-blobs optional dependencies behind a feature such as p2p-iroh. Default dev/unit-test builds should not compile the Iroh backend. If config selects Iroh but the binary was built without the feature, return a clear error.

P3: Feature-gate jemalloc

Make tikv-jemallocator optional. Release/server distribution builds can enable it, but regular dev/test builds should not need to compile jemalloc by default.

P4: Optimize or isolate RocksDB

Evaluate whether RocksDB can be built with fewer features or moved behind a feature/backend boundary. librocksdb-sys is currently the largest full-rebuild hotspot, so reducing its feature set or isolating it from ordinary tests could have a large impact.

P5: Consolidate HTTP/TLS dependencies

Review generated clients and root dependencies to avoid compiling both native-tls/OpenSSL and rustls/aws-lc/ring stacks where possible. Pay special attention to thirdparty/firecracker-client and thirdparty/envd/http-client default features.

P6: Split the root crate over time

Longer term, split the large root agentenv crate into smaller internal crates with clearer compile boundaries, e.g. config/types/image/snapshot/sandbox/orchestrator/p2p/server-app. This should improve incremental rebuild behavior and reduce the amount of code recompiled for localized changes.

Suggested acceptance criteria

Possible measurable goals:

• cargo test -p agentenv --lib --no-run no longer compiles AWS SDK, testcontainers, or Criterion.
• dev server full rebuild time is significantly reduced, with an initial target below 2 minutes on the measured machine.
• unit-test compile time is significantly reduced, with an initial target below 90 seconds on the measured machine.
• total debug integration-test binary size is reduced by at least 50%.
• target/debug/incremental after the common dev/test workflow is substantially below the current 5G range.
• release server builds either strip debuginfo by default or provide a documented stripped build path.

Notes

This is not a functional correctness bug, but it is a maintainability issue. It directly affects local iteration speed, CI cost, disk usage, and the ability to keep adding subsystems without making the workspace increasingly expensive to build.

Originally created by @LSX-s-Software on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/8 ## Background Several common Rust build targets in AgentENV have become expensive to compile and produce very large artifacts. This affects local development feedback loops, CI runtime, disk usage, and the cost of switching between dev/test/release profiles. A clean full rebuild is a useful way to expose the baseline cost, but the issue is broader than any particular local cleanup workflow. Even incremental builds can be slow when touching code inside the large root `agentenv` crate, and test builds produce multiple large binaries that quickly inflate `target/`. ## Local measurements Environment: 16 cores, 31 GiB RAM, `rustc 1.97.0`, current workspace. The test targets below were measured with `--no-run` to isolate compile/link time from KVM, root permissions, network namespaces, or test runtime behavior. | Target | Command | Time | | --- | --- | ---: | | dev server | `cargo build -p agentenv --bin server --timings` | 3m36s | | unit tests | `cargo test -p agentenv --lib --no-run --timings` | 3m16s | | integration tests | `cargo test -p agentenv --no-run --test integration --test orchestrator_integration --test snapshot_oss_e2e_test --timings` | 1m01s | | release server | `cargo build --release -p agentenv --bin server --timings` | 4m49s | | no-op rebuild | same commands immediately repeated | ~0.5s–0.8s | | touch one root crate file | `touch src/local_store.rs && cargo build -p agentenv --bin server` | 8.4s | `target/` size after building the common dev/test/release targets: ```text target/ 21G target/debug/ 19G target/debug/deps/ 13G target/debug/build/ 1.1G target/debug/incremental/ 5.0G target/release/ 2.1G target/release/deps/ 1.6G target/release/build/ 492M ``` Large artifacts observed: ```text 852M target/debug/deps/agentenv-... # unit test binary 813M target/debug/server # debug server 621M target/debug/deps/orchestrator_test-... 608M target/debug/deps/snapshot_test-... 606M target/debug/deps/snapshot_attached_drive_test-... 603M target/debug/deps/fc_test-... 602M target/debug/deps/ublk_test-... 598M target/debug/deps/process_test-... 598M target/debug/deps/envd_supervisor_test-... 463M target/debug/deps/libagentenv-....rlib 202M target/debug/deps/libaws_sdk_s3-....rlib 186M target/debug/deps/libiroh-....so 116M target/debug/deps/liboverlaybd-....rlib 111M target/debug/deps/libiroh_blobs-....rlib ``` Release server size: ```text target/release/server: 79M stripped: 57M ``` ## Timing hotspots Cold dev server build top items: ```text 128.8s librocksdb-sys build-script 57.9s tikv-jemalloc-sys build-script 38.2s agentenv 35.0s aws-lc-sys build-script 28.2s overlaybd 22.5s iroh-blobs 21.2s iroh 17.2s zstd-sys build-script 16.7s lz4-sys build-script ``` Cold release server build top items: ```text 154.7s librocksdb-sys build-script 83.7s aws-lc-sys build-script 51.2s zstd-sys build-script 46.6s agentenv 32.4s iroh 31.8s tikv-jemalloc-sys build-script 31.0s noq-proto 28.9s iroh-blobs 28.3s overlaybd 27.9s rustls 27.1s reqwest v0.13 26.5s opendal 25.0s reqwest v0.12 ``` ## Initial analysis This does not appear to be caused by a single crate. The cost is the result of several factors compounding: 1. The root `agentenv` crate is large and directly includes API, sandbox, snapshot, orchestrator, image, P2P, setup, and other subsystems. Touching code inside this crate can still require expensive incremental recompilation and relinking. 2. Heavy native build scripts dominate full rebuilds, especially `librocksdb-sys`, `tikv-jemalloc-sys`, `aws-lc-sys`, `zstd-sys`, and `lz4-sys`. 3. P2P/Iroh is currently an unconditional dependency, even though many dev and unit-test workflows do not need the Iroh backend. 4. Root `agentenv` dev-dependencies pull in benchmark and e2e dependencies. In particular, `criterion` and `agentenv-test-support` cause ordinary unit-test builds to compile AWS SDK, `aws-config`, `testcontainers`, and related dependencies. 5. The dependency graph is large and contains duplicated major/minor versions. Current metadata shows roughly 830 packages, about 791 transitive dependencies for `agentenv`, and around 75 duplicate crate names, including `reqwest 0.12/0.13`, `hyper 0.14/1.x`, `rustls 0.21/0.23`, `bindgen 0.69/0.72`, and `netlink-packet-route 0.28/0.29/0.30`. 6. dev/test/release profiles do not share artifacts, so each profile pays its own compilation and native build cost. 7. debug/test artifacts are very large due to debug info, large rlibs, repeated test binaries, and incremental caches. ## Proposed direction This should probably be handled as a build-performance and dependency-governance effort, split across several PRs. ### P1: Isolate benchmark and e2e dev-dependencies (Done in #39) Move Criterion benchmarks and OSS e2e/test-support code into separate workspace packages so that `cargo test -p agentenv --lib` does not compile AWS SDK, testcontainers, or Criterion. Target: ordinary root-crate unit tests should not depend on: - `criterion` - `agentenv-test-support` - `aws-sdk-s3` - `aws-config` - `testcontainers` - `testcontainers-modules` ### P2: Feature-gate P2P/Iroh Make `iroh` and `iroh-blobs` optional dependencies behind a feature such as `p2p-iroh`. Default dev/unit-test builds should not compile the Iroh backend. If config selects Iroh but the binary was built without the feature, return a clear error. ### P3: Feature-gate jemalloc Make `tikv-jemallocator` optional. Release/server distribution builds can enable it, but regular dev/test builds should not need to compile jemalloc by default. ### P4: Optimize or isolate RocksDB Evaluate whether RocksDB can be built with fewer features or moved behind a feature/backend boundary. `librocksdb-sys` is currently the largest full-rebuild hotspot, so reducing its feature set or isolating it from ordinary tests could have a large impact. ### P5: Consolidate HTTP/TLS dependencies Review generated clients and root dependencies to avoid compiling both native-tls/OpenSSL and rustls/aws-lc/ring stacks where possible. Pay special attention to `thirdparty/firecracker-client` and `thirdparty/envd/http-client` default features. ### P6: Split the root crate over time Longer term, split the large root `agentenv` crate into smaller internal crates with clearer compile boundaries, e.g. config/types/image/snapshot/sandbox/orchestrator/p2p/server-app. This should improve incremental rebuild behavior and reduce the amount of code recompiled for localized changes. ## Suggested acceptance criteria Possible measurable goals: - `cargo test -p agentenv --lib --no-run` no longer compiles AWS SDK, testcontainers, or Criterion. - dev server full rebuild time is significantly reduced, with an initial target below 2 minutes on the measured machine. - unit-test compile time is significantly reduced, with an initial target below 90 seconds on the measured machine. - total debug integration-test binary size is reduced by at least 50%. - `target/debug/incremental` after the common dev/test workflow is substantially below the current 5G range. - release server builds either strip debuginfo by default or provide a documented stripped build path. ## Notes This is not a functional correctness bug, but it is a maintainability issue. It directly affects local iteration speed, CI cost, disk usage, and the ability to keep adding subsystems without making the workspace increasingly expensive to build.

bsakaguchi commented 2026-10-29 14:18:17 +00:00

Author

Owner

Copy link

@NickNYU commented on GitHub (Jul 29, 2026):

Hi
I have sent a PR trying to partially solve the problem, please see if it works @LSX-s-Software
Thanks

<!-- gh-comment-id:5111779340 --> @NickNYU commented on GitHub (Jul 29, 2026): Hi I have sent a PR trying to partially solve the problem, please see if it works @LSX-s-Software Thanks

bsakaguchi referenced this issue 2026-10-29 14:18:21 +00:00

[GH-ISSUE #31] Support attached drives when creating a sandbox from a template #14

bsakaguchi referenced this issue 2026-10-29 14:18:50 +00:00

[GH-ISSUE #151] Gateway blocks sandbox-routed envd /metrics requests #50

bsakaguchi referenced this issue 2026-10-29 14:19:17 +00:00

[GH-ISSUE #294] Add lifecycle-aware per-sandbox resource metrics collection #78

bsakaguchi referenced this issue 2026-10-29 14:19:19 +00:00

[GH-ISSUE #305] bug: E2B TypeScript SDK 2.51.0 sandbox creation fails with HTTP 405 on /v2/sandboxes #79

bsakaguchi referenced this issue 2026-10-29 14:19:22 +00:00

[GH-ISSUE #312] bug(network): Debian ifupdown hotplug triggers ifquery process storm during large-scale sandbox churn, inflating pause/resume latency #83

bsakaguchi referenced this issue 2026-10-29 14:19:41 +00:00

[GH-ISSUE #443] Sandbox endpoints return 404 instead of 400 for malformed sandbox IDs #111

bsakaguchi referenced this issue 2026-10-29 14:21:06 +00:00

[PR #159] [MERGED] fix(gateway): proxy sandbox-routed envd metrics requests #216

bsakaguchi referenced this issue 2026-10-29 14:22:23 +00:00

[PR #311] [MERGED] feat: add sandbox metrics collection APIs #340

bsakaguchi referenced this issue 2026-10-29 14:22:26 +00:00

[PR #321] [MERGED] feat(templates): add E2B-compatible build status and logs #345

bsakaguchi referenced this issue 2026-10-29 14:22:35 +00:00

[PR #338] [MERGED] feat: add sandbox logs API #360

bsakaguchi referenced this issue 2026-10-29 14:22:42 +00:00

[PR #350] [CLOSED] feat: add webhook API #370

bsakaguchi referenced this issue 2026-10-29 14:22:45 +00:00

[PR #356] feat: add webhook API #375

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#5

No description provided.