[GH-ISSUE #180] Warm Firecracker pool never decays its refill target after a burst #60

New issue

Open

opened 2026-10-29 14:18:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:58 +00:00

Owner

Originally created by @epicvinny on GitHub (Aug 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/180

Affected component: Firecracker runtime

Problem statement

As an operator running AgentENV with the warm Firecracker process pool enabled,
I need the pool's refill target to decay after the burst that raised it has passed,
because the current behavior pins the excess warm processes — and the memory they
hold — for the entire lifetime of the server process.

Use case

Sandbox hosts serving bursty workloads (e.g. CI/agent sandbox creation in business
hours, idle overnight). A single burst of sandbox creations ratchets the geometric
fill target up; hours later the host still holds that many parked Firecracker
processes. On a shared host this is pure memory waste and also keeps more processes
parked with oom_score_adj=1000 than the steady state needs.

Current behavior and workarounds

WarmPool's maintenance worker computes its fill target from a geometric
high-watermark that only ever ratchets up on acquisitions. There is no time-based
decay: once raised, the target stays for the process lifetime. The only workaround
is restarting the server, which defeats the purpose of a warm pool.

Desired behavior

Given a configurable idle TTL (seconds),
When no warm-process acquisition happens within the TTL,
Then the maintenance worker decays the refill target back to the configured low
watermark and drains the excess warm processes.
Any acquisition resets the idle clock (busy pools never decay).
0 disables decay and preserves the current ratchet behavior.

Proposed approach

Add PoolConfig::idle_ttl to the warm-pool crate: the maintenance loop tracks
the last-acquisition instant, wakes on the TTL boundary, and when expired reduces
the fill target toward the low watermark and drains the excess entries. Wire it as
[pool.firecracker] idle_ttl_secs (default 600, 0 disables), documented in
config/default.toml and the configuration reference. The network-slot and ublk
pools keep idle_ttl: None (historical ratchet): they hold no guest processes, so
idle decay buys nothing there.

Implementation ready: https://github.com/ResultadosDigitais/AgentENV (branch
upstream/warm-pool-idle-ttl), with unit tests for decay, TTL wake-up, and
clock reset.

Compatibility and operational impact

API/config changes: new optional key [pool.firecracker] idle_ttl_secs (default
600). No breaking change; setting 0 restores exact previous behavior.

Snapshot or storage format changes: none.

New host/runtime requirements: none.

Upgrade and rollback considerations: rolling back simply ignores the new key;
behavior with the key absent on old builds is unchanged.

Alternatives considered

Time-based decay of parked entries themselves (per-entry TTL): wastes warm
processes that a steady-state workload still needs; decaying the refill target
preserves the configured low watermark.
External memory-pressure-driven scaling: couples pool sizing to host-level
signals the pool does not currently consume; a TTL is deterministic and
testable.
Doing nothing (restart to reset): operationally hostile for long-lived hosts.

Non-goals

Changing the geometric growth policy itself.
Per-entry idle TTL for parked processes.
Applying decay to the network-slot or ublk pools.

Contribution

I can implement it (implementation ready, PR to follow).

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @epicvinny on GitHub (Aug 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/180 **Affected component:** Firecracker runtime ## Problem statement As an operator running AgentENV with the warm Firecracker process pool enabled, I need the pool's refill target to decay after the burst that raised it has passed, because the current behavior pins the excess warm processes — and the memory they hold — for the entire lifetime of the server process. ## Use case Sandbox hosts serving bursty workloads (e.g. CI/agent sandbox creation in business hours, idle overnight). A single burst of sandbox creations ratchets the geometric fill target up; hours later the host still holds that many parked Firecracker processes. On a shared host this is pure memory waste and also keeps more processes parked with `oom_score_adj=1000` than the steady state needs. ## Current behavior and workarounds `WarmPool`'s maintenance worker computes its fill target from a geometric high-watermark that only ever ratchets up on acquisitions. There is no time-based decay: once raised, the target stays for the process lifetime. The only workaround is restarting the server, which defeats the purpose of a warm pool. ## Desired behavior - Given a configurable idle TTL (seconds), - When no warm-process acquisition happens within the TTL, - Then the maintenance worker decays the refill target back to the configured low watermark and drains the excess warm processes. - Any acquisition resets the idle clock (busy pools never decay). - `0` disables decay and preserves the current ratchet behavior. ## Proposed approach Add `PoolConfig::idle_ttl` to the `warm-pool` crate: the maintenance loop tracks the last-acquisition instant, wakes on the TTL boundary, and when expired reduces the fill target toward the low watermark and drains the excess entries. Wire it as `[pool.firecracker] idle_ttl_secs` (default 600, `0` disables), documented in `config/default.toml` and the configuration reference. The network-slot and ublk pools keep `idle_ttl: None` (historical ratchet): they hold no guest processes, so idle decay buys nothing there. Implementation ready: https://github.com/ResultadosDigitais/AgentENV (branch `upstream/warm-pool-idle-ttl`), with unit tests for decay, TTL wake-up, and clock reset. ## Compatibility and operational impact API/config changes: new optional key `[pool.firecracker] idle_ttl_secs` (default 600). No breaking change; setting `0` restores exact previous behavior. Snapshot or storage format changes: none. New host/runtime requirements: none. Upgrade and rollback considerations: rolling back simply ignores the new key; behavior with the key absent on old builds is unchanged. ## Alternatives considered - Time-based decay of *parked entries themselves* (per-entry TTL): wastes warm processes that a steady-state workload still needs; decaying the refill target preserves the configured low watermark. - External memory-pressure-driven scaling: couples pool sizing to host-level signals the pool does not currently consume; a TTL is deterministic and testable. - Doing nothing (restart to reset): operationally hostile for long-lived hosts. ## Non-goals - Changing the geometric growth policy itself. - Per-entry idle TTL for parked processes. - Applying decay to the network-slot or ublk pools. ## Contribution I can implement it (implementation ready, PR to follow). ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi referenced this issue

2026-10-29 14:20:36 +00:00

[PR #86] [MERGED] feat: add ARM64 support for AgentENV #169