[GH-ISSUE #282] Compare Terminal-Bench 2.1 runtime and memory: AENV vs Docker #76

New issue

Open

opened 2026-10-29 14:19:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:17 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Sep 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/282

Workload runtime and memory comparison

This records the runtime and memory consumption of the same ten Terminal-Bench 2.1 workloads using the previous aenv configuration, tuned aenv configuration, and Docker.

Conditions

Workloads: Terminal-Bench 2.1 task solution scripts.
aenv: 2 GiB guest memory, 1 vCPU, with host C6 disabled during measurement. The previous column uses the previous DAMON configuration from issue #281; the tuned column uses the tuned DAMON profile from issue #281.
Docker: 2 GiB container memory limit, one pinned host CPU (--cpuset-cpus), images pre-pulled, with host C6 disabled during measurement.
Container startup and workload setup are included; image pull time is excluded.
Each workload was run once; all runs exited successfully.

Runtime

Workload	Previous aenv (s)	Docker (s)
train-fasttext	114.75	101.20
path-tracing	0.210	0.660
raman-fitting	15.87	14.43
largest-eigenval	11.27	10.34
count-dataset-tokens	75.59	71.71
build-cython-ext	95.95	91.84
dna-insert	23.66	22.66
write-compressor	54.71	53.25
regex-log	0.212	0.850
openssl-selfsigned-cert	0.412	1.100

The arithmetic mean is 39.26 s for previous aenv and 36.80 s for Docker. Short workloads are dominated by container or guest startup overhead; the longer workloads are generally faster in Docker in this run.

Memory consumption

The aenv columns retain the average occupied memory from #281: guest MemTotal - MemFree, including guest page cache and kernel memory.

Both Docker columns below come from the same fresh memory-measurement pass:

Docker cgroup avg: time-weighted average of cgroup v2 memory.current.
Docker excluding page cache avg: time-weighted average of memory.current - memory.stat[file]. This includes kernel memory charged to the container, along with anonymous memory; it is broader than process RSS. The excluded file counter covers filesystem cache, including tmpfs/shared memory, as defined in the kernel cgroup v2 documentation.

Workload	Previous aenv occupied avg (MiB)	Tuned aenv occupied avg (MiB)	Docker cgroup avg (MiB)	Docker excluding page cache avg (MiB)
train-fasttext	922.1	392.6	812.8	277.4
path-tracing	138.7	100.6	1.2	1.2
raman-fitting	298.3	202.7	214.6	73.2
largest-eigenval	219.3	163.9	146.4	56.2
count-dataset-tokens	585.8	274.7	503.9	166.3
build-cython-ext	402.0	272.8	256.6	166.5
dna-insert	318.1	183.7	235.0	77.8
write-compressor	540.4	315.5	293.0	280.1
regex-log	43.7	42.7	2.1	2.0
openssl-selfsigned-cert	60.7	61.5	7.0	6.8

The Docker memory pass used the same images (alexgshaw/<task>:20251031), solution scripts, 2 GiB limit, and one pinned logical CPU per container, with all ten containers running concurrently on distinct logical CPUs and host C6 disabled. Each script ran once and exited with code 0. No swap usage or OOM kills were observed. C6 settings were restored afterward.

Memory was sampled throughout each solution script, including its package installation/build steps, using a start gate and recorded completion time. The sampler requested 1 ms intervals during the first second and 20 ms afterward, and calculated time-weighted averages using actual timestamps. Container startup and waiting after completion are excluded from these memory averages. The two Docker memory columns replace the earlier coarse samples so they share the same measurement window; the runtime table above retains its original measurements.

path-tracing and regex-log only generate source/text files in these solution scripts and finished in about 8 ms and 4 ms respectively. Their memory estimates use only three and four samples and are sensitive to sampling resolution and kernel counter batching.

The accounting boundaries differ: aenv occupied memory covers the guest, while Docker values cover memory charged to the workload cgroup. Docker daemon memory and host allocations outside that cgroup are excluded.

Originally created by @yingdi-shan on GitHub (Sep 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/282 ## Workload runtime and memory comparison This records the runtime and memory consumption of the same ten Terminal-Bench 2.1 workloads using the previous aenv configuration, tuned aenv configuration, and Docker. ### Conditions - Workloads: Terminal-Bench 2.1 task solution scripts. - aenv: 2 GiB guest memory, 1 vCPU, with host C6 disabled during measurement. The previous column uses the previous DAMON configuration from issue #281; the tuned column uses the tuned DAMON profile from issue #281. - Docker: 2 GiB container memory limit, one pinned host CPU (`--cpuset-cpus`), images pre-pulled, with host C6 disabled during measurement. - Container startup and workload setup are included; image pull time is excluded. - Each workload was run once; all runs exited successfully. ### Runtime | Workload | Previous aenv (s) | Docker (s) | |---|---:|---:| | train-fasttext | 114.75 | 101.20 | | path-tracing | 0.210 | 0.660 | | raman-fitting | 15.87 | 14.43 | | largest-eigenval | 11.27 | 10.34 | | count-dataset-tokens | 75.59 | 71.71 | | build-cython-ext | 95.95 | 91.84 | | dna-insert | 23.66 | 22.66 | | write-compressor | 54.71 | 53.25 | | regex-log | 0.212 | 0.850 | | openssl-selfsigned-cert | 0.412 | 1.100 | The arithmetic mean is 39.26 s for previous aenv and 36.80 s for Docker. Short workloads are dominated by container or guest startup overhead; the longer workloads are generally faster in Docker in this run. ### Memory consumption The aenv columns retain the average occupied memory from #281: guest `MemTotal - MemFree`, including guest page cache and kernel memory. Both Docker columns below come from the same fresh memory-measurement pass: - **Docker cgroup avg:** time-weighted average of cgroup v2 `memory.current`. - **Docker excluding page cache avg:** time-weighted average of `memory.current - memory.stat[file]`. This **includes kernel memory charged to the container**, along with anonymous memory; it is broader than process RSS. The excluded `file` counter covers filesystem cache, including tmpfs/shared memory, as defined in the [kernel cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files). | Workload | Previous aenv occupied avg (MiB) | Tuned aenv occupied avg (MiB) | Docker cgroup avg (MiB) | Docker excluding page cache avg (MiB) | |---|---:|---:|---:|---:| | train-fasttext | 922.1 | 392.6 | 812.8 | 277.4 | | path-tracing | 138.7 | 100.6 | 1.2 | 1.2 | | raman-fitting | 298.3 | 202.7 | 214.6 | 73.2 | | largest-eigenval | 219.3 | 163.9 | 146.4 | 56.2 | | count-dataset-tokens | 585.8 | 274.7 | 503.9 | 166.3 | | build-cython-ext | 402.0 | 272.8 | 256.6 | 166.5 | | dna-insert | 318.1 | 183.7 | 235.0 | 77.8 | | write-compressor | 540.4 | 315.5 | 293.0 | 280.1 | | regex-log | 43.7 | 42.7 | 2.1 | 2.0 | | openssl-selfsigned-cert | 60.7 | 61.5 | 7.0 | 6.8 | The Docker memory pass used the same images (`alexgshaw/<task>:20251031`), solution scripts, 2 GiB limit, and one pinned logical CPU per container, with all ten containers running concurrently on distinct logical CPUs and host C6 disabled. Each script ran once and exited with code 0. No swap usage or OOM kills were observed. C6 settings were restored afterward. Memory was sampled throughout each solution script, including its package installation/build steps, using a start gate and recorded completion time. The sampler requested 1 ms intervals during the first second and 20 ms afterward, and calculated time-weighted averages using actual timestamps. Container startup and waiting after completion are excluded from these memory averages. The two Docker memory columns replace the earlier coarse samples so they share the same measurement window; the runtime table above retains its original measurements. `path-tracing` and `regex-log` only generate source/text files in these solution scripts and finished in about 8 ms and 4 ms respectively. Their memory estimates use only three and four samples and are sensitive to sampling resolution and kernel counter batching. The accounting boundaries differ: aenv occupied memory covers the guest, while Docker values cover memory charged to the workload cgroup. Docker daemon memory and host allocations outside that cgroup are excluded.

bsakaguchi referenced this issue

2026-10-29 14:20:26 +00:00

[PR #76] [1/4] feat(web): scaffold control-plane console and deploy wiring #160

bsakaguchi referenced this issue

2026-10-29 14:20:27 +00:00

[PR #77] [CLOSED] [2/4] feat(web): Gateway connection settings and session cookies #161

bsakaguchi referenced this issue

2026-10-29 14:20:28 +00:00

[PR #78] [CLOSED] [3/4] feat(web): sandbox and snapshot management views #162

bsakaguchi referenced this issue

2026-10-29 14:20:32 +00:00

[PR #79] [CLOSED] [4/4] feat(web): templates, dashboard, and node views #165