kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[PR #307] [CLOSED] perf(server): avoid blocking startup on firecracker pool prime #334

New issue

Closed

opened 2026-10-29 14:22:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:20 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/307
Author: @emailcannotbeblank
Created: 9/20/2026
Status: ❌ Closed

Base: main ← Head: fix/async-firecracker-pool-prime

📝 Commits (1)

45c72a1 perf(server): avoid blocking startup on firecracker pool prime

📊 Changes

2 files changed (+7 additions, -3 deletions)

View changed files

📝 scripts/install.sh (+2 -0)
📝 src/bin/server.rs (+5 -3)

📄 Description

What

This PR makes two changes:

Run FirecrackerPool::prime() through tokio::spawn during server startup, so Firecracker pool warm-up does not block the subsequent API startup.
Add an ExecStartPost command to the systemd service generated by scripts/install.sh. It polls /health every 100 ms and waits until the AgentENV service is actually ready.

Why

Test environment

low_watermark = 400
high_watermark = 400

Problem

After running systemctl restart aenv and waiting for eight seconds, the AgentENV API was still not ready:

$ systemctl restart aenv
$ sleep 8
$ aenv ls
Error: transport error

Caused by:
    0: http://127.0.0.1:8000/v2/sandboxes: Connection Failed: Connect error: Connection refused (os error 111)
    1: Connection refused (os error 111)

Cause

In the main function in src/bin/server.rs, server startup synchronously waits for the Firecracker pool to be primed:

if let Err(err) = FirecrackerPool::prime(std::time::Duration::from_secs(10)).await {
    warn!(target: "agentenv", error = %err, "firecracker pool prime failed; continuing startup");
}

This step takes a significant amount of time with the test configuration above. The AgentENV API does not become available until execution reaches the following code and binds the listening address:

let listener = tokio::net::TcpListener::bind(&addr).await?.tap_io(|stream| {
    if let Err(err) = stream.set_nodelay(true) {
        warn!(target: "agentenv", error = %err, "failed to set TCP_NODELAY on incoming connection");
    }
});

At the same time, the original systemd service does not wait for the HTTP API to become ready. Therefore, the systemd command returning does not mean that aenv can already access the API.

N/A. No existing issue was found that directly tracks this startup-blocking behavior.

Scope and non-goals

This PR includes:

Running Firecracker pool priming asynchronously.
Polling /health from the systemd service generated by the installer.
Polling every 100 ms, with at most 900 attempts.

This PR does not include:

Changing the Firecracker pool low_watermark or high_watermark.
Changing pool scaling or refill policy.
Changing the AgentENV API or configuration format.

Design and behavior changes

1. Wait for service readiness in systemd

Add the following command to the service file generated by scripts/install.sh:

ExecStartPost=/bin/sh -c 'for i in $(seq 1 900); do curl -fsS --max-time 0.1 "http://${API_ADDR}/health" >/dev/null 2>&1 && exit 0; sleep 0.1; done; exit 1'

The command checks /health every 100 ms. systemctl start aenv reports success only after the health check succeeds. It reports failure if all 900 attempts fail.

With the health check added, but before making pool priming asynchronous, service startup took:

$ time systemctl start aenv

real    0m10.741s
user    0m0.002s
sys     0m0.007s

2. Prime the Firecracker pool asynchronously

Replace the synchronous wait with:

tokio::spawn(async move {
    if let Err(err) = FirecrackerPool::prime(std::time::Duration::from_secs(10)).await {
        warn!(target: "agentenv", error = %err, "firecracker pool prime failed; continuing startup");
    }
});

The Firecracker pool continues warming in the background while the main flow proceeds with API startup.

After this change, the time for systemctl start aenv to observe a successful /health response was:

$ time systemctl start aenv

real    0m1.745s
user    0m0.002s
sys     0m0.006s

Compatibility and operations

Public API or generated protocol: no change.
Configuration or defaults: no change.
Snapshot manifest, artifact layout, or storage format: no change.
Upgrade and rollback: revert the changes to src/bin/server.rs and scripts/install.sh to restore the previous behavior.
Host requirements, permissions, ports, or dependencies: the health check uses curl and the existing API_ADDR from the service environment file.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests (run, but one external-network case failed in the full suite and passed when retried alone)
make -C services test (passed; this PR does not change services/)
Generated clients/server regenerated (not applicable)
Documentation updated (not applicable)
Startup-time comparison completed

Commands and results:

cargo check -p agentenv --bin server
# passed

make fmt
# passed

make clippy
# passed

make test-unit
# passed

bash scripts/tests/verify-install-service.sh
# passed

make -C services test
# gateway and scheduler tests passed

AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR='' make test-agent-integration
# integration: 20 passed, 1 failed
# failed test: fc::microvm_network_policy_controls_egress
# reason: http://www.baidu.com/ returned an empty reply (curl exit 52)
# the failed test passed when retried alone
# orchestrator_integration: 2 passed (run separately)
# snapshot OSS integration: 5 passed (run separately)

systemctl restart aenv
sleep 8
aenv ls
# Before the change: connection to 127.0.0.1:8000 was refused

time systemctl start aenv
# With health polling but before asynchronous priming: real 10.741s
# After asynchronous priming: real 1.745s

Skipped checks and reasons:

The full make test-e2e target was not run.

Integration-test notes:

make test-agent-integration was run, but the full command failed because fc::microvm_network_policy_controls_egress received curl: (52) Empty reply from server from an external site. The integration-test checkbox therefore remains unchecked.
The failed test passed when retried by itself.
Because the full command stopped after the first test binary failed, orchestrator_integration and the snapshot OSS integration tests were run separately and all passed.
An existing AgentENV service was already using ublk metrics port 9103, so AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR='' was set only for the test process. The existing service was not changed or stopped.

Risks and reviewer notes

Firecracker pool priming no longer blocks API startup, so the pool may still be warming in the background when the API first becomes ready.

Checklist

This PR contains only changes related to service startup readiness, with no unrelated refactoring or formatting.
The new behavior was validated through actual service startup and timing comparisons.
Logs and examples contain no credentials, tokens, or private image references.
No generated code was edited manually.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/307 **Author:** [@emailcannotbeblank](https://github.com/emailcannotbeblank) **Created:** 9/20/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/async-firecracker-pool-prime` --- ### 📝 Commits (1) - [`45c72a1`](https://github.com/kvcache-ai/AgentENV/commit/45c72a14c8a9516bcad603f07c9ccc0e454f7d68) perf(server): avoid blocking startup on firecracker pool prime ### 📊 Changes **2 files changed** (+7 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `scripts/install.sh` (+2 -0) 📝 `src/bin/server.rs` (+5 -3) </details> ### 📄 Description ## What This PR makes two changes: 1. Run `FirecrackerPool::prime()` through `tokio::spawn` during server startup, so Firecracker pool warm-up does not block the subsequent API startup. 2. Add an `ExecStartPost` command to the systemd service generated by `scripts/install.sh`. It polls `/health` every 100 ms and waits until the AgentENV service is actually ready. ## Why ### Test environment ```toml low_watermark = 400 high_watermark = 400 ``` ### Problem After running `systemctl restart aenv` and waiting for eight seconds, the AgentENV API was still not ready: ```console $ systemctl restart aenv $ sleep 8 $ aenv ls Error: transport error Caused by: 0: http://127.0.0.1:8000/v2/sandboxes: Connection Failed: Connect error: Connection refused (os error 111) 1: Connection refused (os error 111) ``` ### Cause In the `main` function in `src/bin/server.rs`, server startup synchronously waits for the Firecracker pool to be primed: ```rust if let Err(err) = FirecrackerPool::prime(std::time::Duration::from_secs(10)).await { warn!(target: "agentenv", error = %err, "firecracker pool prime failed; continuing startup"); } ``` This step takes a significant amount of time with the test configuration above. The AgentENV API does not become available until execution reaches the following code and binds the listening address: ```rust let listener = tokio::net::TcpListener::bind(&addr).await?.tap_io(|stream| { if let Err(err) = stream.set_nodelay(true) { warn!(target: "agentenv", error = %err, "failed to set TCP_NODELAY on incoming connection"); } }); ``` At the same time, the original systemd service does not wait for the HTTP API to become ready. Therefore, the systemd command returning does not mean that `aenv` can already access the API. ## Related issue N/A. No existing issue was found that directly tracks this startup-blocking behavior. ## Scope and non-goals This PR includes: - Running Firecracker pool priming asynchronously. - Polling `/health` from the systemd service generated by the installer. - Polling every 100 ms, with at most 900 attempts. This PR does not include: - Changing the Firecracker pool `low_watermark` or `high_watermark`. - Changing pool scaling or refill policy. - Changing the AgentENV API or configuration format. ## Design and behavior changes ### 1. Wait for service readiness in systemd Add the following command to the service file generated by `scripts/install.sh`: ```ini ExecStartPost=/bin/sh -c 'for i in $(seq 1 900); do curl -fsS --max-time 0.1 "http://${API_ADDR}/health" >/dev/null 2>&1 && exit 0; sleep 0.1; done; exit 1' ``` The command checks `/health` every 100 ms. `systemctl start aenv` reports success only after the health check succeeds. It reports failure if all 900 attempts fail. With the health check added, but before making pool priming asynchronous, service startup took: ```console $ time systemctl start aenv real 0m10.741s user 0m0.002s sys 0m0.007s ``` ### 2. Prime the Firecracker pool asynchronously Replace the synchronous wait with: ```rust tokio::spawn(async move { if let Err(err) = FirecrackerPool::prime(std::time::Duration::from_secs(10)).await { warn!(target: "agentenv", error = %err, "firecracker pool prime failed; continuing startup"); } }); ``` The Firecracker pool continues warming in the background while the main flow proceeds with API startup. After this change, the time for `systemctl start aenv` to observe a successful `/health` response was: ```console $ time systemctl start aenv real 0m1.745s user 0m0.002s sys 0m0.006s ``` ## Compatibility and operations - Public API or generated protocol: no change. - Configuration or defaults: no change. - Snapshot manifest, artifact layout, or storage format: no change. - Upgrade and rollback: revert the changes to `src/bin/server.rs` and `scripts/install.sh` to restore the previous behavior. - Host requirements, permissions, ports, or dependencies: the health check uses `curl` and the existing `API_ADDR` from the service environment file. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [ ] Relevant Rust integration tests (run, but one external-network case failed in the full suite and passed when retried alone) - [x] `make -C services test` (passed; this PR does not change `services/`) - [ ] Generated clients/server regenerated (not applicable) - [ ] Documentation updated (not applicable) - [x] Startup-time comparison completed Commands and results: ```text cargo check -p agentenv --bin server # passed make fmt # passed make clippy # passed make test-unit # passed bash scripts/tests/verify-install-service.sh # passed make -C services test # gateway and scheduler tests passed AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR='' make test-agent-integration # integration: 20 passed, 1 failed # failed test: fc::microvm_network_policy_controls_egress # reason: http://www.baidu.com/ returned an empty reply (curl exit 52) # the failed test passed when retried alone # orchestrator_integration: 2 passed (run separately) # snapshot OSS integration: 5 passed (run separately) systemctl restart aenv sleep 8 aenv ls # Before the change: connection to 127.0.0.1:8000 was refused time systemctl start aenv # With health polling but before asynchronous priming: real 10.741s # After asynchronous priming: real 1.745s ``` Skipped checks and reasons: - The full `make test-e2e` target was not run. Integration-test notes: - `make test-agent-integration` was run, but the full command failed because `fc::microvm_network_policy_controls_egress` received `curl: (52) Empty reply from server` from an external site. The integration-test checkbox therefore remains unchecked. - The failed test passed when retried by itself. - Because the full command stopped after the first test binary failed, `orchestrator_integration` and the snapshot OSS integration tests were run separately and all passed. - An existing AgentENV service was already using ublk metrics port `9103`, so `AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR=''` was set only for the test process. The existing service was not changed or stopped. ## Risks and reviewer notes - Firecracker pool priming no longer blocks API startup, so the pool may still be warming in the background when the API first becomes ready. ## Checklist - [x] This PR contains only changes related to service startup readiness, with no unrelated refactoring or formatting. - [x] The new behavior was validated through actual service startup and timing comparisons. - [x] Logs and examples contain no credentials, tokens, or private image references. - [x] No generated code was edited manually. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:20 +00:00