[PR #112] [CLOSED] chore(scripts): add API benchmark and smoke scripts #189

New issue

Closed

opened 2026-10-29 14:20:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:49 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/112
Author: @cleverhu
Created: 8/3/2026
Status: ❌ Closed

Base: main ← Head: chore/api-bench-scripts

📝 Commits (1)

a402b96 chore(scripts): add API benchmark and smoke scripts

📊 Changes

3 files changed (+954 additions, -1 deletions)

View changed files

📝 Makefile (+8 -1)
➕ scripts/bench.py (+727 -0)
➕ scripts/smoke.py (+219 -0)

📄 Description

What

Add scripts/bench.py (API-level lifecycle benchmark), scripts/smoke.py (post-deploy smoke test through the E2B SDK), and a make bench-api target.

Why

The repository has Criterion benchmarks for storage internals, but nothing that measures what a user actually experiences: the latency and throughput of create / pause / resume / snapshot / fork over the HTTP API, under concurrency.

That gap is not academic. Measuring through the E2B Python SDK, a create that the server completed in ~85 ms was reported as 137 ms, because the SDK lazily constructs an httpx.Client per subsystem and each construction spends ~37 ms building a default TLS context — on a plain-HTTP link. Without a tool that drives the API directly, that overhead is indistinguishable from server latency.

None; tooling addition with no runtime impact.

Scope and non-goals

Included: the two scripts and the make target.

Excluded:

Any change to server code or CI. Nothing runs these automatically.
A wrapper for third-party benchmark drivers. The internal branch had one; it hardcoded a cluster address, an API key and a template id and depended on an out-of-tree checkout, so it is not proposed here.
Assertions or thresholds. These report numbers; they do not gate.

Design and behavior changes

bench.py is standard library only and speaks HTTP directly, so no SDK sits in the measurement loop.

One concurrency tier per invocation, so tiers can be swept from a shell loop without cross-tier interference.
A keep-alive connection per worker. Without this every sample carries a TCP handshake, which is a significant error on an 80 ms operation.
Warmup rounds are discarded; create-only and create-delete modes; p50/p95/p99, wall time and amortized per-op time; JSON export retaining raw samples in completion order so drift over a run is visible.
With --metrics-url, it diffs a node's Prometheus endpoint across the measured window and prints the server-side stage breakdown (load_snapshot, create_sandbox, ublk operations) next to client latency. This is what makes it possible to say "the storage layer is 3 ms of the 85 ms" rather than guessing.

smoke.py runs the lifecycle through the official E2B SDK and times each step, which is the fastest way to confirm a deploy is healthy and E2B compatibility still holds.

Both default to http://127.0.0.1:8000 and read AENV_API_URL / AENV_API_KEY / AENV_TEMPLATE_ID (or the E2B_* equivalents) from the environment.

Compatibility and operations

Public API or generated protocol: N/A, the scripts are consumers of the existing API.
Configuration or defaults: N/A. BENCH_API_SCENARIO and BENCH_API_ARGS are Makefile variables only.
Snapshot manifest, artifact layout, or storage format: N/A.
Upgrade and rollback: N/A, additive files.
Host requirements, permissions, ports, or dependencies: bench.py needs only Python 3. smoke.py additionally needs pip install e2b. Neither is imported by the build.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

$ python3 -m py_compile scripts/bench.py scripts/smoke.py
(clean)

$ python3 scripts/bench.py --help
usage: bench.py [-h] [-c CONCURRENCY] [-n TOTAL] [--rounds ROUNDS] [-w WARMUP]
                [-m {create-delete,create-only}] [-t TEMPLATE]
                [--timeout TIMEOUT] [--settle SETTLE] [--keep] [-o OUTPUT]
                [--no-header] [--api-url API_URL] [--api-key API_KEY]
                [--metrics-url METRICS_URL]
                {create,fork,pause,pause-resume,resume,snapshot}

$ make -n bench-api
python3 scripts/bench.py create -c 1 -n 20 -w 1

An earlier revision of bench.py produced this on an internal 128-core host, template 2 vCPU / 1 GiB, create-only mode — included to show the shape of the output rather than as a claim about upstream main:

concurrency	requests	avg	p95	throughput
1	20	82.6 ms	105.5 ms	12.1 /s
10	100	427.8 ms	643.3 ms	20.9 /s
20	200	1571.6 ms	3404.8 ms	12.1 /s
50	200	5072.0 ms	8821.3 ms	8.9 /s

The degradation above ~10 concurrent creates tracks warm-pool exhaustion: with [pool] low_watermark = 2 the pool empties immediately and most requests fall to the cold path (fresh netns, fresh ublk device, cold Firecracker). That is a finding this tool surfaces, not something this PR changes.

Skipped checks and reasons:

No Rust target applies; the only non-Python change is a Makefile target verified with make -n.
No new tests: these are test tooling, and asserting on their output would require a provisioned server in CI.

Risks and reviewer notes

No runtime risk; nothing here is reachable from the server binary.

Reviewer notes:

smoke.py requires e2b from PyPI, which is not vendored. Please confirm an optional dependency in scripts/ is acceptable.
The Prometheus stage names in bench.py's metric table are coupled to the current SandboxStageTimer labels and will need updating if those are renamed.
Both scripts default to a local server and carry a placeholder API key; no cluster address or real credential is present.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/112 **Author:** [@cleverhu](https://github.com/cleverhu) **Created:** 8/3/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `chore/api-bench-scripts` --- ### 📝 Commits (1) - [`a402b96`](https://github.com/kvcache-ai/AgentENV/commit/a402b961d00891a7a7c5aa94ba200e94ce60ae28) chore(scripts): add API benchmark and smoke scripts ### 📊 Changes **3 files changed** (+954 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `Makefile` (+8 -1) ➕ `scripts/bench.py` (+727 -0) ➕ `scripts/smoke.py` (+219 -0) </details> ### 📄 Description ## What Add `scripts/bench.py` (API-level lifecycle benchmark), `scripts/smoke.py` (post-deploy smoke test through the E2B SDK), and a `make bench-api` target. ## Why The repository has Criterion benchmarks for storage internals, but nothing that measures what a user actually experiences: the latency and throughput of create / pause / resume / snapshot / fork over the HTTP API, under concurrency. That gap is not academic. Measuring through the E2B Python SDK, a create that the server completed in ~85 ms was reported as 137 ms, because the SDK lazily constructs an `httpx.Client` per subsystem and each construction spends ~37 ms building a default TLS context — on a plain-HTTP link. Without a tool that drives the API directly, that overhead is indistinguishable from server latency. ## Related issue None; tooling addition with no runtime impact. ## Scope and non-goals Included: the two scripts and the make target. Excluded: - Any change to server code or CI. Nothing runs these automatically. - A wrapper for third-party benchmark drivers. The internal branch had one; it hardcoded a cluster address, an API key and a template id and depended on an out-of-tree checkout, so it is not proposed here. - Assertions or thresholds. These report numbers; they do not gate. ## Design and behavior changes `bench.py` is standard library only and speaks HTTP directly, so no SDK sits in the measurement loop. - One concurrency tier per invocation, so tiers can be swept from a shell loop without cross-tier interference. - A keep-alive connection per worker. Without this every sample carries a TCP handshake, which is a significant error on an 80 ms operation. - Warmup rounds are discarded; `create-only` and `create-delete` modes; p50/p95/p99, wall time and amortized per-op time; JSON export retaining raw samples in completion order so drift over a run is visible. - With `--metrics-url`, it diffs a node's Prometheus endpoint across the measured window and prints the server-side stage breakdown (`load_snapshot`, `create_sandbox`, ublk operations) next to client latency. This is what makes it possible to say "the storage layer is 3 ms of the 85 ms" rather than guessing. `smoke.py` runs the lifecycle through the official E2B SDK and times each step, which is the fastest way to confirm a deploy is healthy and E2B compatibility still holds. Both default to `http://127.0.0.1:8000` and read `AENV_API_URL` / `AENV_API_KEY` / `AENV_TEMPLATE_ID` (or the `E2B_*` equivalents) from the environment. ## Compatibility and operations - Public API or generated protocol: N/A, the scripts are consumers of the existing API. - Configuration or defaults: N/A. `BENCH_API_SCENARIO` and `BENCH_API_ARGS` are Makefile variables only. - Snapshot manifest, artifact layout, or storage format: N/A. - Upgrade and rollback: N/A, additive files. - Host requirements, permissions, ports, or dependencies: `bench.py` needs only Python 3. `smoke.py` additionally needs `pip install e2b`. Neither is imported by the build. ## Validation - [ ] `make fmt` - [ ] `make clippy` - [ ] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [ ] Documentation updated - [x] Benchmarks or performance comparison completed Commands and results: ```text $ python3 -m py_compile scripts/bench.py scripts/smoke.py (clean) $ python3 scripts/bench.py --help usage: bench.py [-h] [-c CONCURRENCY] [-n TOTAL] [--rounds ROUNDS] [-w WARMUP] [-m {create-delete,create-only}] [-t TEMPLATE] [--timeout TIMEOUT] [--settle SETTLE] [--keep] [-o OUTPUT] [--no-header] [--api-url API_URL] [--api-key API_KEY] [--metrics-url METRICS_URL] {create,fork,pause,pause-resume,resume,snapshot} $ make -n bench-api python3 scripts/bench.py create -c 1 -n 20 -w 1 ``` An earlier revision of `bench.py` produced this on an internal 128-core host, template 2 vCPU / 1 GiB, `create-only` mode — included to show the shape of the output rather than as a claim about upstream `main`: | concurrency | requests | avg | p95 | throughput | | ---: | ---: | ---: | ---: | ---: | | 1 | 20 | 82.6 ms | 105.5 ms | 12.1 /s | | 10 | 100 | 427.8 ms | 643.3 ms | 20.9 /s | | 20 | 200 | 1571.6 ms | 3404.8 ms | 12.1 /s | | 50 | 200 | 5072.0 ms | 8821.3 ms | 8.9 /s | The degradation above ~10 concurrent creates tracks warm-pool exhaustion: with `[pool] low_watermark = 2` the pool empties immediately and most requests fall to the cold path (fresh netns, fresh ublk device, cold Firecracker). That is a finding this tool surfaces, not something this PR changes. Skipped checks and reasons: - No Rust target applies; the only non-Python change is a Makefile target verified with `make -n`. - No new tests: these *are* test tooling, and asserting on their output would require a provisioned server in CI. ## Risks and reviewer notes No runtime risk; nothing here is reachable from the server binary. Reviewer notes: - `smoke.py` requires `e2b` from PyPI, which is not vendored. Please confirm an optional dependency in `scripts/` is acceptable. - The Prometheus stage names in `bench.py`'s metric table are coupled to the current `SandboxStageTimer` labels and will need updating if those are renamed. - Both scripts default to a local server and carry a placeholder API key; no cluster address or real credential is present. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:49 +00:00