[PR #337] [MERGED] feat: add concurrent sandbox throughput benchmarks #359

New issue

Closed

opened 2026-10-29 14:22:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/337
Author: @GxmzG
Created: 10/2/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/benchmark-concurrent-throughput

📝 Commits (4)

d2305fb feat: add concurrent sandbox throughput benchmarks
da71cdc feat: default throughput concurrency to 2x CPUs
5c7a0d7 fix: harden concurrent snapshot benchmark cleanup
83cf932 fix: bound benchmark concurrency and clean up failed starts

📊 Changes

1 file changed (+419 additions, -6 deletions)

View changed files

📝 crates/benchmarks/benches/snapshot_benchmark.rs (+419 -6)

📄 Description

Summary

The existing concurrent_resume benchmark reports latency. This adds throughput measurements for concurrent resume, running-sandbox snapshot capture, and pause in snapshot_benchmark.rs. Concurrency can be set with AENV_BENCH_CONCURRENCY.

Measurement

Each sample runs the operation concurrently on separate sandboxes. Throughput is completed operations divided by batch wall-clock time; sandbox setup and cleanup are excluded. This measures backend operations, not API or scheduler throughput.

Validation

I ran the benchmarks against real KVM/Firecracker sandboxes on a 6-vCPU Ubuntu 24.04 VM, using concurrency 6, 12, and 24 with 10 Criterion samples per setting.

In that concurrency order, resume measured 37.59, 48.17, and 52.69 ops/s. Running-sandbox snapshot capture measured 94.75, 138.41, and 99.97 ops/s. Pause measured 97.74, 147.12, and 106.40 ops/s.

Snapshot capture and pause were highest at concurrency 12 among the tested settings. For resume, Criterion increased at 24. These are preliminary results.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/337 **Author:** [@GxmzG](https://github.com/GxmzG) **Created:** 10/2/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/benchmark-concurrent-throughput` --- ### 📝 Commits (4) - [`d2305fb`](https://github.com/kvcache-ai/AgentENV/commit/d2305fb81f490ab96c264a24183a4ac722a2b7f3) feat: add concurrent sandbox throughput benchmarks - [`da71cdc`](https://github.com/kvcache-ai/AgentENV/commit/da71cdc7c08a62f2f5a039aba49a8f888968d7c1) feat: default throughput concurrency to 2x CPUs - [`5c7a0d7`](https://github.com/kvcache-ai/AgentENV/commit/5c7a0d795e613584439732e7668983b10eb567ea) fix: harden concurrent snapshot benchmark cleanup - [`83cf932`](https://github.com/kvcache-ai/AgentENV/commit/83cf932e0e06d372aa3ea69bcb41a316c4829b17) fix: bound benchmark concurrency and clean up failed starts ### 📊 Changes **1 file changed** (+419 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `crates/benchmarks/benches/snapshot_benchmark.rs` (+419 -6) </details> ### 📄 Description ## Summary The existing `concurrent_resume` benchmark reports latency. This adds throughput measurements for concurrent resume, running-sandbox snapshot capture, and pause in `snapshot_benchmark.rs`. Concurrency can be set with `AENV_BENCH_CONCURRENCY`. ## Measurement Each sample runs the operation concurrently on separate sandboxes. Throughput is completed operations divided by batch wall-clock time; sandbox setup and cleanup are excluded. This measures backend operations, not API or scheduler throughput. ## Validation I ran the benchmarks against real KVM/Firecracker sandboxes on a 6-vCPU Ubuntu 24.04 VM, using concurrency 6, 12, and 24 with 10 Criterion samples per setting. In that concurrency order, resume measured 37.59, 48.17, and 52.69 ops/s. Running-sandbox snapshot capture measured 94.75, 138.41, and 99.97 ops/s. Pause measured 97.74, 147.12, and 106.40 ops/s. Snapshot capture and pause were highest at concurrency 12 among the tested settings. For resume, Criterion increased at 24. These are preliminary results. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:35 +00:00