kvcache-ai/AgentENV
1
0
Fork 0
mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00
Code Issues 91 Projects Packages Wiki Activity

[GH-ISSUE #281] Document DAMON reclaim trade-offs with memory density, and runtime #77

New issue
Open
opened 2026-10-29 14:19:17 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-29 14:19:17 +00:00
Owner
Copy link

Originally created by @yingdi-shan on GitHub (Sep 13, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/281

Problem statement

AgentENV can reclaim file-backed pages aggressively with DAMON, but the settings affect both memory density and task runtime. Users need to know which workloads benefit, how much memory is saved, and when reclaim causes page refaults or write throttling.

Profiles being compared

Previous AgentENV profile:

damon_reclaim.min_age=60000000
damon_reclaim.quota_ms=100
damon_reclaim.quota_sz=1073741824
damon_reclaim.quota_reset_interval_ms=1000
damon_reclaim.wmarks_high=900
damon_reclaim.wmarks_mid=700
damon_reclaim.wmarks_low=200
damon_reclaim.skip_anon=Y
damon_reclaim.wmarks_interval=5000000

Tuned profile:

damon_reclaim.min_age=100000
damon_reclaim.quota_ms=20
damon_reclaim.quota_sz=1073741824
damon_reclaim.quota_reset_interval_ms=500
damon_reclaim.wmarks_high=990
damon_reclaim.wmarks_mid=990
damon_reclaim.wmarks_low=200
damon_reclaim.skip_anon=Y
damon_reclaim.wmarks_interval=1000000

The tuned profile considers pages cold after 100 ms, checks watermarks every second, and spends at most 20 ms reclaiming in each 500 ms quota period. DAMON parameter units and watermark behavior are defined by the kernel DAMON reclaim documentation.

Physical memory range used by DAMON

The range tuning used damon_reclaim.monitor_region_start and damon_reclaim.monitor_region_end, in guest physical-address bytes. The previous profile left both inputs at 0 (automatic selection). Linux 6.1 then selects the largest individual System RAM resource, which can leave other RAM ranges outside DAMON's coverage. See the kernel parameter documentation.

  • monitor_region_start = minimum start address of all System RAM entries.
  • monitor_region_end = maximum end address of all System RAM entries plus one, because /proc/iomem prints inclusive ends.

The saved 4 GiB tuning runs recorded:

Range parameter Previous effective value (automatic) Tuned effective value
monitor_region_start 1048576 (1 MiB) 4096 (4 KiB)
monitor_region_end 3221225471 (3 GiB − 1 byte) 5368709120 (5 GiB)

For that guest layout, the explicit settings were:

damon_reclaim.monitor_region_start=4096
damon_reclaim.monitor_region_end=5368709120

The 4 GiB guest had RAM below 3 GiB and another 1 GiB at physical addresses 4–5 GiB, separated by the 3–4 GiB address hole. Automatic selection missed the upper 1 GiB; expanding the span made it eligible for DAMON monitoring/reclaim. Increasing quota or lowering min_age alone cannot reclaim pages outside the monitored range. This fixes coverage; the measurements do not isolate the memory savings from this change alone.

Workloads and measurement

Each workload is the checked-in solution command for a Terminal-Bench 2.1 task. The commands run in a fresh Linux guest with the task's input files and dependencies:

Task Work performed
train-fasttext Convert a Parquet text dataset and train a FastText classifier
path-tracing Generate a C ray-tracing implementation for an image
raman-fitting Fit a Raman spectroscopy dataset and write fitted results
largest-eigenval Compute the largest eigenvalue and eigenvector of a matrix
count-dataset-tokens Count tokens in a dataset and write the count
build-cython-ext Build and install a Cython/Python extension
dna-insert Install sequence-analysis tools and design/check PCR primers
write-compressor Compile and run a data-compression program
regex-log Process a log with regular expressions and write the result
openssl-selfsigned-cert Generate and validate a self-signed certificate

Runtime is wall-clock time for the task command. Occupied memory is the time-weighted MemTotal - MemFree; page cache is reported separately as Cached. Every task was run once with each profile under the same guest size and task image.

Results

Task Previous runtime Tuned runtime Runtime change Previous occupied Tuned occupied Memory saved Previous cache Tuned cache
train-fasttext 114.75 s 122.71 s +6.9% 922.1 MiB 392.6 MiB 529.5 MiB (57.4%) 598.9 MiB 79.2 MiB
path-tracing 0.21 s 0.21 s -0.5% 138.7 MiB 100.6 MiB 38.1 MiB (27.5%) 91.1 MiB 9.7 MiB
raman-fitting 15.87 s 16.12 s +1.6% 298.3 MiB 202.7 MiB 95.6 MiB (32.1%) 187.9 MiB 80.6 MiB
largest-eigenval 11.27 s 10.87 s -3.5% 219.3 MiB 163.9 MiB 55.4 MiB (25.3%) 127.3 MiB 64.3 MiB
count-dataset-tokens 75.59 s 72.01 s -4.7% 585.8 MiB 274.7 MiB 311.1 MiB (53.1%) 380.8 MiB 75.2 MiB
build-cython-ext 95.95 s 102.25 s +6.6% 402.0 MiB 272.8 MiB 129.2 MiB (32.1%) 186.5 MiB 60.1 MiB
dna-insert 23.66 s 24.34 s +2.9% 318.1 MiB 183.7 MiB 134.4 MiB (42.2%) 207.0 MiB 73.1 MiB
write-compressor 54.71 s 56.55 s +3.4% 540.4 MiB 315.5 MiB 225.0 MiB (41.6%) 238.1 MiB 11.7 MiB
regex-log 0.21 s 0.22 s +2.8% 43.7 MiB 42.7 MiB 1.0 MiB (2.2%) 18.6 MiB 6.5 MiB
openssl-selfsigned-cert 0.41 s 0.61 s +47.8% 60.6 MiB 61.5 MiB -0.9 MiB (-1.4%) 25.0 MiB 11.7 MiB

Tuned reclaim reduced occupied memory and cache for the longer workloads. Runtime changes ranged from a 4.7% improvement to a 6.9% increase across the six longer tasks. Short tasks complete before reclaim can make a meaningful difference; their measurements are included for completeness but are dominated by startup and sampling resolution.

Dirty-page flushing interaction

A separate high-density experiment also used:

vm.dirty_background_bytes=16777216
vm.dirty_bytes=33554432
vm.dirty_writeback_centisecs=10
vm.dirty_expire_centisecs=100

These values reduced average Dirty memory during Parquet processing from about 77 MiB to about 5 MiB. They also reduced post-task cache, but dirty thresholds are global VM controls. In file-copy and archive stress tests, combining aggressive reclaim with these thresholds reduced cache while increasing runtime by roughly 2.1–2.5×.

Dirty-page flushing should therefore remain a separate opt-in profile until write-heavy workloads have been measured.

C6 and page-fault latency follow-up

The table above was run with host CPU C6 disabled for the duration of the measurements.

To isolate the source of the runtime gap, we warmed the host and OverlayBD cache, dropped only the guest page cache, and touched 1,536 individual 4 KiB pages from a NumPy shared library. Every cold access was a genuine major fault; no physical host-disk read was required.

Access path Median fault latency P95 P99
Guest cache hit (minor fault) 2.1 us 3.7 us 5.6 us
Guest miss, host/OverlayBD cache hit 826 us 970 us 988 us
Same path, separate I/O cores, C6 enabled 747 us 914 us 941 us
Same path, separate I/O cores, C6 disabled 111 us 123 us 169 us
Same path, all I/O threads on one core 72 us 87 us 118 us

A kernel trace measured roughly 10 us in the host file-cache read itself and about 115 us in the OverlayBD I/O worker operation. Most of the remaining latency came from waking and rescheduling Firecracker, ublk, and io_uring workers while CPUs entered the deep C6 idle state. This is why a host-cache hit can still approach 1 ms, while disabling C6 brings it close to the 100 us range.

Originally created by @yingdi-shan on GitHub (Sep 13, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/281 ## Problem statement AgentENV can reclaim file-backed pages aggressively with DAMON, but the settings affect both memory density and task runtime. Users need to know which workloads benefit, how much memory is saved, and when reclaim causes page refaults or write throttling. ## Profiles being compared Previous AgentENV profile: ```text damon_reclaim.min_age=60000000 damon_reclaim.quota_ms=100 damon_reclaim.quota_sz=1073741824 damon_reclaim.quota_reset_interval_ms=1000 damon_reclaim.wmarks_high=900 damon_reclaim.wmarks_mid=700 damon_reclaim.wmarks_low=200 damon_reclaim.skip_anon=Y damon_reclaim.wmarks_interval=5000000 ``` Tuned profile: ```text damon_reclaim.min_age=100000 damon_reclaim.quota_ms=20 damon_reclaim.quota_sz=1073741824 damon_reclaim.quota_reset_interval_ms=500 damon_reclaim.wmarks_high=990 damon_reclaim.wmarks_mid=990 damon_reclaim.wmarks_low=200 damon_reclaim.skip_anon=Y damon_reclaim.wmarks_interval=1000000 ``` The tuned profile considers pages cold after 100 ms, checks watermarks every second, and spends at most 20 ms reclaiming in each 500 ms quota period. DAMON parameter units and watermark behavior are defined by the [kernel DAMON reclaim documentation](https://docs.kernel.org/admin-guide/mm/damon/reclaim.html). ### Physical memory range used by DAMON The range tuning used **`damon_reclaim.monitor_region_start`** and **`damon_reclaim.monitor_region_end`**, in guest physical-address bytes. The previous profile left both inputs at `0` (automatic selection). Linux 6.1 then selects the largest individual `System RAM` resource, which can leave other RAM ranges outside DAMON's coverage. See the [kernel parameter documentation](https://docs.kernel.org/6.1/admin-guide/mm/damon/reclaim.html#monitor-region-start). - `monitor_region_start` = minimum start address of all `System RAM` entries. - `monitor_region_end` = maximum end address of all `System RAM` entries **plus one**, because `/proc/iomem` prints inclusive ends. The saved 4 GiB tuning runs recorded: | Range parameter | Previous effective value (automatic) | Tuned effective value | |---|---:|---:| | `monitor_region_start` | `1048576` (1 MiB) | `4096` (4 KiB) | | `monitor_region_end` | `3221225471` (3 GiB − 1 byte) | `5368709120` (5 GiB) | For that guest layout, the explicit settings were: ```text damon_reclaim.monitor_region_start=4096 damon_reclaim.monitor_region_end=5368709120 ``` The 4 GiB guest had RAM below 3 GiB and another 1 GiB at physical addresses 4–5 GiB, separated by the 3–4 GiB address hole. Automatic selection missed the upper 1 GiB; expanding the span made it eligible for DAMON monitoring/reclaim. Increasing quota or lowering `min_age` alone cannot reclaim pages outside the monitored range. This fixes coverage; the measurements do not isolate the memory savings from this change alone. ## Workloads and measurement Each workload is the checked-in solution command for a Terminal-Bench 2.1 task. The commands run in a fresh Linux guest with the task's input files and dependencies: | Task | Work performed | |---|---| | `train-fasttext` | Convert a Parquet text dataset and train a FastText classifier | | `path-tracing` | Generate a C ray-tracing implementation for an image | | `raman-fitting` | Fit a Raman spectroscopy dataset and write fitted results | | `largest-eigenval` | Compute the largest eigenvalue and eigenvector of a matrix | | `count-dataset-tokens` | Count tokens in a dataset and write the count | | `build-cython-ext` | Build and install a Cython/Python extension | | `dna-insert` | Install sequence-analysis tools and design/check PCR primers | | `write-compressor` | Compile and run a data-compression program | | `regex-log` | Process a log with regular expressions and write the result | | `openssl-selfsigned-cert` | Generate and validate a self-signed certificate | Runtime is wall-clock time for the task command. Occupied memory is the time-weighted `MemTotal - MemFree`; page cache is reported separately as `Cached`. Every task was run once with each profile under the same guest size and task image. ## Results | Task | Previous runtime | Tuned runtime | Runtime change | Previous occupied | Tuned occupied | Memory saved | Previous cache | Tuned cache | |---|---:|---:|---:|---:|---:|---:|---:|---:| | `train-fasttext` | 114.75 s | 122.71 s | +6.9% | 922.1 MiB | 392.6 MiB | 529.5 MiB (57.4%) | 598.9 MiB | 79.2 MiB | | `path-tracing` | 0.21 s | 0.21 s | -0.5% | 138.7 MiB | 100.6 MiB | 38.1 MiB (27.5%) | 91.1 MiB | 9.7 MiB | | `raman-fitting` | 15.87 s | 16.12 s | +1.6% | 298.3 MiB | 202.7 MiB | 95.6 MiB (32.1%) | 187.9 MiB | 80.6 MiB | | `largest-eigenval` | 11.27 s | 10.87 s | -3.5% | 219.3 MiB | 163.9 MiB | 55.4 MiB (25.3%) | 127.3 MiB | 64.3 MiB | | `count-dataset-tokens` | 75.59 s | 72.01 s | -4.7% | 585.8 MiB | 274.7 MiB | 311.1 MiB (53.1%) | 380.8 MiB | 75.2 MiB | | `build-cython-ext` | 95.95 s | 102.25 s | +6.6% | 402.0 MiB | 272.8 MiB | 129.2 MiB (32.1%) | 186.5 MiB | 60.1 MiB | | `dna-insert` | 23.66 s | 24.34 s | +2.9% | 318.1 MiB | 183.7 MiB | 134.4 MiB (42.2%) | 207.0 MiB | 73.1 MiB | | `write-compressor` | 54.71 s | 56.55 s | +3.4% | 540.4 MiB | 315.5 MiB | 225.0 MiB (41.6%) | 238.1 MiB | 11.7 MiB | | `regex-log` | 0.21 s | 0.22 s | +2.8% | 43.7 MiB | 42.7 MiB | 1.0 MiB (2.2%) | 18.6 MiB | 6.5 MiB | | `openssl-selfsigned-cert` | 0.41 s | 0.61 s | +47.8% | 60.6 MiB | 61.5 MiB | -0.9 MiB (-1.4%) | 25.0 MiB | 11.7 MiB | Tuned reclaim reduced occupied memory and cache for the longer workloads. Runtime changes ranged from a 4.7% improvement to a 6.9% increase across the six longer tasks. Short tasks complete before reclaim can make a meaningful difference; their measurements are included for completeness but are dominated by startup and sampling resolution. ## Dirty-page flushing interaction A separate high-density experiment also used: ```text vm.dirty_background_bytes=16777216 vm.dirty_bytes=33554432 vm.dirty_writeback_centisecs=10 vm.dirty_expire_centisecs=100 ``` These values reduced average `Dirty` memory during Parquet processing from about 77 MiB to about 5 MiB. They also reduced post-task cache, but dirty thresholds are global VM controls. In file-copy and archive stress tests, combining aggressive reclaim with these thresholds reduced cache while increasing runtime by roughly 2.1–2.5×. Dirty-page flushing should therefore remain a separate opt-in profile until write-heavy workloads have been measured. ## C6 and page-fault latency follow-up The table above was run with host CPU C6 disabled for the duration of the measurements. To isolate the source of the runtime gap, we warmed the host and OverlayBD cache, dropped only the guest page cache, and touched 1,536 individual 4 KiB pages from a NumPy shared library. Every cold access was a genuine major fault; no physical host-disk read was required. | Access path | Median fault latency | P95 | P99 | |---|---:|---:|---:| | Guest cache hit (minor fault) | 2.1 us | 3.7 us | 5.6 us | | Guest miss, host/OverlayBD cache hit | 826 us | 970 us | 988 us | | Same path, separate I/O cores, C6 enabled | 747 us | 914 us | 941 us | | Same path, separate I/O cores, C6 disabled | 111 us | 123 us | 169 us | | Same path, all I/O threads on one core | 72 us | 87 us | 118 us | A kernel trace measured roughly 10 us in the host file-cache read itself and about 115 us in the OverlayBD I/O worker operation. Most of the remaining latency came from waking and rescheduling Firecracker, ublk, and io_uring workers while CPUs entered the deep C6 idle state. This is why a host-cache hit can still approach 1 ms, while disabling C6 brings it close to the 100 us range.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
coverage-data
gh-pages
v0.3.0-rc.1
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
documentation

   
enhancement

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

No labels
bug
documentation
enhancement
enhancement
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/AgentENV#77
No description provided.