[PR #162] [MERGED] perf: reduce server idle memory retention (jemalloc decay + pool runtime threads) #220

New issue

Closed

opened 2026-10-29 14:21:09 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:09 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/162
Author: @huajq
Created: 8/12/2026
Status: ✅ Merged
Merged: 8/13/2026
Merged by: @yingdi-shan

Base: main ← Head: perf/jemalloc-decay-fc-pool-runtime

📝 Commits (2)

e247a1f perf(server): shorten jemalloc page decay via malloc_conf
56de617 perf(sandbox): run firecracker pool runtime on one worker thread

📊 Changes

2 files changed (+15 additions, -1 deletions)

View changed files

📝 src/bin/server.rs (+10 -0)
📝 src/sandbox/firecracker/pool.rs (+5 -1)

📄 Description

What

Two small changes that reduce the API server's idle/background memory
footprint:

Tune jemalloc through an embedded malloc_conf string
(dirty_decay_ms:1000,muzzy_decay_ms:1000,background_thread:true).
Pin the Firecracker warm pool's private Tokio runtime to a single
worker thread.

Why

A production memory audit showed two server-side hygiene issues:

After bursts (RocksDB opens, image resolution, template builds),
server RSS climbs and is slow to fall back: jemalloc's default
dirty-page decay is 10s (muzzy_decay_ms defaults to 0, i.e.
immediate), so burst pages linger as retained RSS well after the
burst. Shortening both to 1s returns them within ~2s.
Note on mechanism: background purging threads are already active —
the background_threads cargo feature makes tikv-jemalloc-sys
bake background_thread:true into the jemalloc build via
--with-malloc-conf (build.rs). It is included in the string only
for explicitness; the effective change here is the decay tuning.
The Firecracker pool's owned runtime uses the default multi-thread
builder, parking one worker thread per CPU core for the process
lifetime, although it only drives occasional I/O-bound block_on
calls from the maintenance thread.

N/A — small performance hygiene change from a direct memory audit; no
tracking issue.

Scope and non-goals

In scope: server allocator decay tuning; pool runtime thread count.
Out of scope: ublk-daemon memory (it does not use jemalloc), guest
memory, and any pool watermark/sizing changes.

Design and behavior changes

malloc_conf is jemalloc's link-time configuration mechanism and is
parsed at allocator init, before main(); the TOML config system
therefore cannot express these settings. Precedence (lowest first):
build-time config_malloc_conf (already carries
background_thread:true), link-time malloc_conf symbol (this
change), /etc/malloc.conf, MALLOC_CONF env var — later sources
win per key, so deployments keep an env-level override hatch.
The pool runtime keeps new_multi_thread() with worker_threads(1),
so the existing spawn-based fill logic behaves identically on one
worker; pool refill concurrency is unchanged (driven by
fill_concurrency, cooperative on the single worker).

Compatibility and operations

Public API or generated protocol: N/A — untouched.
Configuration or defaults: no config changes; MALLOC_CONF remains
available as a per-environment override.
Snapshot manifest, artifact layout, or storage format: N/A.
Upgrade and rollback: revert-safe; no state migration.
Host requirements, permissions, ports, or dependencies: none added.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests — skipped: no sandbox-lifecycle
behavior change; changes are allocator tuning and a runtime
worker-count setting covered by existing unit tests.
make -C services test — N/A: services/ untouched.
Generated clients/server regenerated — N/A: no generated code.
Documentation updated — N/A: no user-facing configuration added;
MALLOC_CONF is jemalloc's built-in override channel.
Benchmarks or performance comparison completed — skipped: change
affects memory retention timing and idle thread count, not a
latency/throughput path.

Commands and results:

$ make fmt          # cargo fmt --all -- --check — clean
$ make clippy       # workspace --all-targets --all-features -D warnings — clean
$ make test-unit    # 700 passed, 1 failed (pre-existing, see above)
$ nm target/release/server | grep malloc_conf
000000000397c378 d config_malloc_conf   # build-time baked conf (background_thread:true)
000000000396bc40 D malloc_conf          # this change (link-time symbol)
$ MALLOC_CONF="abort_conf:true" ./target/release/server --setup-only
# exit 0 — embedded config string parses; all options valid

Skipped checks and reasons: integration tests and benchmarks (no
lifecycle or latency-path change); test-unit failure is pre-existing on
the base commit in a root test environment, unrelated to these files.

Risks and reviewer notes

jemalloc: shorter decay means more frequent madvise purging after
bursts; the purging runs on jemalloc's background thread (already
enabled at build time), so the allocation path is unaffected. Config
validity verified with an abort_conf:true smoke run.
Pool runtime: single worker serializes maintenance block_on work;
refill stays I/O-bound and off the sandbox critical path.
Review focus: src/bin/server.rs (new exported symbol),
src/sandbox/firecracker/pool.rs (worker_threads).

Checklist

The PR contains one coherent change and no unrelated formatting or
refactoring.
New behavior is covered by tests, or I explained why testing is
impractical.
Logs and examples contain no credentials, tokens, or private
registry information.
I did not manually edit generated code without updating its
source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/162 **Author:** [@huajq](https://github.com/huajq) **Created:** 8/12/2026 **Status:** ✅ Merged **Merged:** 8/13/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `perf/jemalloc-decay-fc-pool-runtime` --- ### 📝 Commits (2) - [`e247a1f`](https://github.com/kvcache-ai/AgentENV/commit/e247a1f488d9284a33bc2c5ee7bbcb2c93ed9041) perf(server): shorten jemalloc page decay via malloc_conf - [`56de617`](https://github.com/kvcache-ai/AgentENV/commit/56de6171691b88718c7e84e615bb5a72bb225c1e) perf(sandbox): run firecracker pool runtime on one worker thread ### 📊 Changes **2 files changed** (+15 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `src/bin/server.rs` (+10 -0) 📝 `src/sandbox/firecracker/pool.rs` (+5 -1) </details> ### 📄 Description ## What Two small changes that reduce the API server's idle/background memory footprint: 1. Tune jemalloc through an embedded `malloc_conf` string (`dirty_decay_ms:1000,muzzy_decay_ms:1000,background_thread:true`). 2. Pin the Firecracker warm pool's private Tokio runtime to a single worker thread. ## Why A production memory audit showed two server-side hygiene issues: - After bursts (RocksDB opens, image resolution, template builds), server RSS climbs and is slow to fall back: jemalloc's default dirty-page decay is 10s (`muzzy_decay_ms` defaults to 0, i.e. immediate), so burst pages linger as retained RSS well after the burst. Shortening both to 1s returns them within ~2s. Note on mechanism: background purging threads are already active — the `background_threads` cargo feature makes `tikv-jemalloc-sys` bake `background_thread:true` into the jemalloc build via `--with-malloc-conf` (build.rs). It is included in the string only for explicitness; the effective change here is the decay tuning. - The Firecracker pool's owned runtime uses the default multi-thread builder, parking one worker thread per CPU core for the process lifetime, although it only drives occasional I/O-bound `block_on` calls from the maintenance thread. ## Related issue N/A — small performance hygiene change from a direct memory audit; no tracking issue. ## Scope and non-goals - In scope: server allocator decay tuning; pool runtime thread count. - Out of scope: ublk-daemon memory (it does not use jemalloc), guest memory, and any pool watermark/sizing changes. ## Design and behavior changes - `malloc_conf` is jemalloc's link-time configuration mechanism and is parsed at allocator init, before `main()`; the TOML config system therefore cannot express these settings. Precedence (lowest first): build-time `config_malloc_conf` (already carries `background_thread:true`), link-time `malloc_conf` symbol (this change), `/etc/malloc.conf`, `MALLOC_CONF` env var — later sources win per key, so deployments keep an env-level override hatch. - The pool runtime keeps `new_multi_thread()` with `worker_threads(1)`, so the existing spawn-based fill logic behaves identically on one worker; pool refill concurrency is unchanged (driven by `fill_concurrency`, cooperative on the single worker). ## Compatibility and operations - Public API or generated protocol: N/A — untouched. - Configuration or defaults: no config changes; `MALLOC_CONF` remains available as a per-environment override. - Snapshot manifest, artifact layout, or storage format: N/A. - Upgrade and rollback: revert-safe; no state migration. - Host requirements, permissions, ports, or dependencies: none added. ## Validation - [x] `make fmt` - [x] `make clippy` - [ ] `make test-unit` - [ ] Relevant Rust integration tests — skipped: no sandbox-lifecycle behavior change; changes are allocator tuning and a runtime worker-count setting covered by existing unit tests. - [ ] `make -C services test` — N/A: `services/` untouched. - [ ] Generated clients/server regenerated — N/A: no generated code. - [ ] Documentation updated — N/A: no user-facing configuration added; `MALLOC_CONF` is jemalloc's built-in override channel. - [ ] Benchmarks or performance comparison completed — skipped: change affects memory retention timing and idle thread count, not a latency/throughput path. Commands and results: ```text $ make fmt # cargo fmt --all -- --check — clean $ make clippy # workspace --all-targets --all-features -D warnings — clean $ make test-unit # 700 passed, 1 failed (pre-existing, see above) $ nm target/release/server | grep malloc_conf 000000000397c378 d config_malloc_conf # build-time baked conf (background_thread:true) 000000000396bc40 D malloc_conf # this change (link-time symbol) $ MALLOC_CONF="abort_conf:true" ./target/release/server --setup-only # exit 0 — embedded config string parses; all options valid ``` Skipped checks and reasons: integration tests and benchmarks (no lifecycle or latency-path change); test-unit failure is pre-existing on the base commit in a root test environment, unrelated to these files. ## Risks and reviewer notes - jemalloc: shorter decay means more frequent `madvise` purging after bursts; the purging runs on jemalloc's background thread (already enabled at build time), so the allocation path is unaffected. Config validity verified with an `abort_conf:true` smoke run. - Pool runtime: single worker serializes maintenance `block_on` work; refill stays I/O-bound and off the sandbox critical path. - Review focus: `src/bin/server.rs` (new exported symbol), `src/sandbox/firecracker/pool.rs` (`worker_threads`). ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:09 +00:00