[GH-ISSUE #260] Sandbox create throughput halves as the serial log directory grows #70

New issue

Closed

opened 2026-10-29 14:19:09 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:09 +00:00

Owner

Originally created by @makhov on GitHub (Sep 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/260

Affected component or operation

Sandbox creation / boot

Regression summary

The persistent serial-output root, {serial_dir}/ (default $AENV_HOME/logs/serial), receives one subdirectory per sandbox and nothing ever prunes it. Sandbox create throughput on a node therefore decays with the node's uptime: every create links a new entry into a directory that only grows.

Measured on one node under sustained create load: at ~594k entries in the root it sustained 197–280 creates/sec. With the root emptied and nothing else changed, the same node sustained 460–475 creates/sec at 76–78% CPU — ~1.9×. Under that load the root grew at ~470 entries/sec.

Nearly every one of those subdirectories holds a single zero-byte firecracker-stderr.log: a guest resumed from a snapshot writes nothing to the console. So the cost is not in writing serial output, it is in linking the directory entry under an exclusive lock on the parent inode — and it falls on every create, whether or not anyone ever reads a log.

User impact: a node silently loses roughly half its create throughput as it ages, with no error, no log line, and no metric pointing at the cause. It presents as general degradation.

Compared versions

Baseline: 2d766447261a7252447fe8bccef2e87d48ece5a6
Current: 53e4061bed1e8a6d0205550e20274a68311e1fe5
First known bad: n/a
Local modifications: -

Host and workload

Linux x86_64, bare metal, KVM.
Sandboxes created from snapshots (hence the empty console output), ublk + OverlayBD storage path.
Sustained concurrent create load against a single node; same workload generator, concurrency, and duration on both sides of the comparison.
{serial_dir} set to a persistent host path, which is the configuration the docs describe; local ext4-family filesystem.

Benchmark command and procedure

1. Drive sustained concurrent sandbox creates at a single node; record creates/sec and host CPU once the rate is steady.
2. Count the serial root (`ls -U {serial_dir} | wc -l`) and note its directory inode size (`stat {serial_dir}`).
3. Move the root aside rather than deleting in place: `mv {serial_dir} {serial_dir}.parked && mkdir {serial_dir}`, then delete the parked copy out of band, after measuring. 
4. Repeat step 1 with the same generator, concurrency, and duration.

Both measurements are steady-state windows on the same node, minutes apart,
with no restart or configuration change between them.

Results

Serial root	Create throughput	Host CPU
~594k entries	197–280 /sec	—
empty	460–475 /sec	76–78%

Throughput was recorded as a steady-state range over the measurement window rather than as a percentile distribution, so treat these as ranges, not distributions.

The inverse correlation between root size and peak create rate held across all four nodes in the fleet at the time the problem was found (size column is disk usage of the tree):

Node	Entries in root	Tree size	Peak creates/sec
A	593,885	4.6 G	408
B	579,462	4.5 G	428
C	351,970	2.8 G	457
D	347,344	2.7 G	452

Raw data

Counts on the same four nodes a while later, showing that growth does not level off (this column is the directory inode size, not tree size):


node B   650,025 entries   41.6 MB directory inode
node C   411,348           26.5 MB
node D   408,246           26.4 MB
node A    58,596            3.8 MB   (root had been parked earlier)

Profiles and diagnostics

No response

Measurement checklist

I ran the baseline and current versions on the same host or documented the differences.
I repeated the measurement and reported the sample count.
I controlled or documented cold/warm caches and other relevant variables.
I searched existing issues and did not find a duplicate.

Originally created by @makhov on GitHub (Sep 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/260 ### Affected component or operation Sandbox creation / boot ### Regression summary The persistent serial-output root, `{serial_dir}/` (default `$AENV_HOME/logs/serial`), receives one subdirectory per sandbox and nothing ever prunes it. Sandbox create throughput on a node therefore decays with the node's uptime: every create links a new entry into a directory that only grows. Measured on one node under sustained create load: at ~594k entries in the root it sustained 197–280 creates/sec. With the root emptied and nothing else changed, the same node sustained 460–475 creates/sec at 76–78% CPU — **~1.9×**. Under that load the root grew at ~470 entries/sec. Nearly every one of those subdirectories holds a single zero-byte `firecracker-stderr.log`: a guest resumed from a snapshot writes nothing to the console. So the cost is not in writing serial output, it is in linking the directory entry under an exclusive lock on the parent inode — and it falls on every create, whether or not anyone ever reads a log. User impact: a node silently loses roughly half its create throughput as it ages, with no error, no log line, and no metric pointing at the cause. It presents as general degradation. ### Compared versions - Baseline: `2d766447261a7252447fe8bccef2e87d48ece5a6` - Current: `53e4061bed1e8a6d0205550e20274a68311e1fe5` - First known bad: n/a - Local modifications: - ### Host and workload - Linux x86_64, bare metal, KVM. - Sandboxes created from snapshots (hence the empty console output), ublk + OverlayBD storage path. - Sustained concurrent create load against a single node; same workload generator, concurrency, and duration on both sides of the comparison. - `{serial_dir}` set to a persistent host path, which is the configuration the docs describe; local ext4-family filesystem. ### Benchmark command and procedure ```shell 1. Drive sustained concurrent sandbox creates at a single node; record creates/sec and host CPU once the rate is steady. 2. Count the serial root (`ls -U {serial_dir} | wc -l`) and note its directory inode size (`stat {serial_dir}`). 3. Move the root aside rather than deleting in place: `mv {serial_dir} {serial_dir}.parked && mkdir {serial_dir}`, then delete the parked copy out of band, after measuring. 4. Repeat step 1 with the same generator, concurrency, and duration. Both measurements are steady-state windows on the same node, minutes apart, with no restart or configuration change between them. ``` ### Results | Serial root | Create throughput | Host CPU | |---------------|-------------------|----------| | ~594k entries | 197–280 /sec | — | | empty | 460–475 /sec | 76–78% | Throughput was recorded as a steady-state range over the measurement window rather than as a percentile distribution, so treat these as ranges, not distributions. The inverse correlation between root size and peak create rate held across all four nodes in the fleet at the time the problem was found (size column is disk usage of the tree): | Node | Entries in root | Tree size | Peak creates/sec | |------|-----------------|-----------|------------------| | A | 593,885 | 4.6 G | 408 | | B | 579,462 | 4.5 G | 428 | | C | 351,970 | 2.8 G | 457 | | D | 347,344 | 2.7 G | 452 | ### Raw data ```text Counts on the same four nodes a while later, showing that growth does not level off (this column is the directory inode size, not tree size): node B 650,025 entries 41.6 MB directory inode node C 411,348 26.5 MB node D 408,246 26.4 MB node A 58,596 3.8 MB (root had been parked earlier) ``` ### Profiles and diagnostics _No response_ ### Measurement checklist - [x] I ran the baseline and current versions on the same host or documented the differences. - [x] I repeated the measurement and reported the sample count. - [x] I controlled or documented cold/warm caches and other relevant variables. - [x] I searched existing issues and did not find a duplicate.

bsakaguchi closed this issue

2026-10-29 14:19:10 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:21 +00:00

[PR #70] [Parked] feat(snapshot): E2B alias rebuild semantics on template publish #155

bsakaguchi referenced this issue

2026-10-29 14:20:22 +00:00

[PR #71] [CLOSED] [1/4] feat(snapshot): template build-context archive store (posixfs + oss) #156

bsakaguchi referenced this issue

2026-10-29 14:20:22 +00:00

[PR #72] [CLOSED] [2/4] feat(api): E2B build-context upload endpoints and configuration #157

bsakaguchi referenced this issue