kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[PR #238] feat(sandbox): bind running Firecracker threads to host CPUs #279

New issue

Open

opened 2026-10-29 14:21:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:46 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/238
Author: @emailcannotbeblank
Created: 9/1/2026
Status: 🔄 Open

Base: main ← Head: feature/bind-cpu-v2

📝 Commits (1)

9d25c73 feat(sandbox): support live CPU affinity binding

📊 Changes

28 files changed (+1871 additions, -5 deletions)

View changed files

📝 Cargo.lock (+18 -0)
📝 Cargo.toml (+1 -0)
📝 crates/aenv/src/client/sandboxes.rs (+76 -2)
➕ crates/aenv/src/commands/cpu_bind.rs (+49 -0)
📝 crates/aenv/src/commands/mod.rs (+1 -0)
📝 crates/aenv/src/main.rs (+3 -0)
📝 docs/src/concepts/sandboxes/working.md (+45 -0)
📝 docs/src/getting-started/aenv-cli/sandboxes.md (+22 -0)
📝 services/gateway/internal/routes.go (+1 -0)
📝 services/gateway/internal/routes_test.go (+3 -0)
📝 services/gateway/internal/server_test.go (+59 -0)
📝 src/api/generated/src/apis/admin.rs (+34 -0)
📝 src/api/generated/src/models.rs (+353 -0)
📝 src/api/generated/src/server/mod.rs (+224 -0)
📝 src/api/impls/admin.rs (+64 -0)
📝 src/api/impls/sandbox.rs (+33 -1)
📝 src/api/openapi.yml (+93 -0)
📝 src/orchestrator/mod.rs (+6 -0)
📝 src/orchestrator/service.rs (+106 -0)
📝 src/orchestrator/tests.rs (+107 -0)

...and 8 more files

📄 Description

What

Add runtime CPU-affinity control for running Firecracker sandboxes.

Add the admin-only POST /sandboxes/{sandboxID}/cpu-affinity API.
Add aenv cpu-bind <sandbox-id> --vcpu <list|*> --core <list>.
Support CPU lists with ranges, duplicates, and optional strides such as
0-10:2.
Bind selected vCPU threads, or every current Firecracker thread with *, to
online host logical CPUs.
Verify every affinity update and roll back earlier updates when a later one
fails.

Why

Hardware architects use AgentENV sandboxes for processor benchmarking and
workload characterization. Runtime CPU affinity supports three main use cases:

Topology-aware performance optimization. Select logical CPUs according
to LLC, NUMA, and SMT topology to improve long-running workload performance.
Stable measurements. Prevent vCPU threads from migrating across many
host CPUs, reducing run-to-run noise in PMU counters, IPC, cache-miss rates,
and latency measurements.
Controlled oversubscription. Place multiple vCPU threads on one logical
CPU, or on SMT siblings of one physical core, to evaluate oversubscription
behavior.

Closes #228

Scope and non-goals

Included:

Runtime affinity changes for running Firecracker sandboxes.
Internal Firecracker PID resolution without exposing the PID through the
public API.
CLI/client, gateway routing, admin API, orchestrator, backend, tests, and
user documentation.

Non-goals:

Exclusive CPU reservation or automatic topology selection.
Persisting affinity across pause/resume or runtime replacement.
Querying current per-thread affinity; this can reuse the process and thread
abstractions introduced here in a follow-up.
Supporting host logical CPU IDs above 1023.

Design and behavior changes

aenv -> gateway -> admin API -> orchestrator -> Firecracker PID
                                               -> /proc thread scan
                                               -> sched_get/setaffinity

The orchestrator accepts only running sandboxes, locks the backend while the
blocking affinity operation runs, and keeps the operation cancellation-safe so
pause, snapshot, or delete cannot replace the Firecracker process midway.

The sandbox layer scans /proc/<pid>/task, identifies vCPU threads by their
fc_vcpu N names, and checks thread start times around numeric-TID syscalls to
narrow the TID-reuse window. Input length, expansion, value count, and CPU IDs
are bounded before allocation. Requested offline CPUs are ignored; an empty
online intersection is rejected before any affinity change.

Before applying changes, the implementation records every target's original
affinity. It then updates and reads back each thread one at a time. A failure or
kernel mismatch triggers best-effort rollback of all threads already changed.

Compatibility and operations

Public API or generated protocol: Adds one admin-only POST endpoint and its
OpenAPI-generated server types. Existing endpoints are unchanged.
Configuration or defaults: No new configuration. The endpoint is registered
by default and protected by the deployment API key; there is no separate
feature flag.
Snapshot manifest, artifact layout, or storage format: N/A; affinity is live
kernel state and is not persisted.
Upgrade and rollback: No data migration is required. Rolling back removes the
API, but an already-applied affinity remains until changed or the Firecracker
process exits.
Host requirements, permissions, ports, or dependencies: Linux /proc and
permission to call sched_setaffinity on Firecracker threads. No new port or
runtime dependency is introduced. Supported logical CPU IDs are 0-1023.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

make fmt
  passed

make clippy
  passed: workspace, all targets, all features, warnings denied

make test-unit
  passed

cargo test -p agentenv cpu_affinity --lib
  passed: 14 passed, 0 failed, 1 ignored helper process

AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR="" make test-agent-integration
  passed: AgentENV integration suites and snapshot OSS tests
  snapshot OSS result: 4 passed, 0 failed

make -C services test
  passed: gateway and scheduler tests

Skipped checks and reasons:

make agentenv-server: generated server changes and their OpenAPI source are
committed, but final regeneration could not be rerun because Java is not
installed in the validation environment.
Benchmarks: not run; this adds an explicit administrative control path rather
than changing a request-processing hot path.
End-to-end tests: not run because they require a privileged KVM/ublk runtime
environment.

Risks and reviewer notes

Incorrect placement can reduce performance for other workloads on the node;
the endpoint is therefore admin-only.
Affinity limits scheduler eligibility but does not reserve CPUs or provide
exclusive access. All selected threads receive the same host CPU set.
* selects threads present when the request is handled; it is not a policy
for threads created later.
Affinity is not restored after pause/resume, because resume creates a new
Firecracker process.
Numeric PID/TID syscalls cannot eliminate reuse races completely. Start-time
checks before and after affinity reads narrow that window.
Rollback is best-effort and reports any thread that could not be restored.

Suggested review order:

src/sandbox/cpu_affinity.rs for parsing, thread selection, verification,
and rollback.
src/orchestrator/service.rs for lifecycle locking and PID handling.
src/api/openapi.yml and src/api/impls/admin.rs for the admin API.
services/gateway/internal/server.go and the aenv client/command for
routing and user-facing behavior.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/238 **Author:** [@emailcannotbeblank](https://github.com/emailcannotbeblank) **Created:** 9/1/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feature/bind-cpu-v2` --- ### 📝 Commits (1) - [`9d25c73`](https://github.com/kvcache-ai/AgentENV/commit/9d25c73b278f1b23f576d8b8395aa7c10b35b875) feat(sandbox): support live CPU affinity binding ### 📊 Changes **28 files changed** (+1871 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+18 -0) 📝 `Cargo.toml` (+1 -0) 📝 `crates/aenv/src/client/sandboxes.rs` (+76 -2) ➕ `crates/aenv/src/commands/cpu_bind.rs` (+49 -0) 📝 `crates/aenv/src/commands/mod.rs` (+1 -0) 📝 `crates/aenv/src/main.rs` (+3 -0) 📝 `docs/src/concepts/sandboxes/working.md` (+45 -0) 📝 `docs/src/getting-started/aenv-cli/sandboxes.md` (+22 -0) 📝 `services/gateway/internal/routes.go` (+1 -0) 📝 `services/gateway/internal/routes_test.go` (+3 -0) 📝 `services/gateway/internal/server_test.go` (+59 -0) 📝 `src/api/generated/src/apis/admin.rs` (+34 -0) 📝 `src/api/generated/src/models.rs` (+353 -0) 📝 `src/api/generated/src/server/mod.rs` (+224 -0) 📝 `src/api/impls/admin.rs` (+64 -0) 📝 `src/api/impls/sandbox.rs` (+33 -1) 📝 `src/api/openapi.yml` (+93 -0) 📝 `src/orchestrator/mod.rs` (+6 -0) 📝 `src/orchestrator/service.rs` (+106 -0) 📝 `src/orchestrator/tests.rs` (+107 -0) _...and 8 more files_ </details> ### 📄 Description  ## What Add runtime CPU-affinity control for running Firecracker sandboxes. - Add the admin-only `POST /sandboxes/{sandboxID}/cpu-affinity` API. - Add `aenv cpu-bind <sandbox-id> --vcpu <list|*> --core <list>`. - Support CPU lists with ranges, duplicates, and optional strides such as `0-10:2`. - Bind selected vCPU threads, or every current Firecracker thread with `*`, to online host logical CPUs. - Verify every affinity update and roll back earlier updates when a later one fails. ## Why Hardware architects use AgentENV sandboxes for processor benchmarking and workload characterization. Runtime CPU affinity supports three main use cases: 1. **Topology-aware performance optimization.** Select logical CPUs according to LLC, NUMA, and SMT topology to improve long-running workload performance. 2. **Stable measurements.** Prevent vCPU threads from migrating across many host CPUs, reducing run-to-run noise in PMU counters, IPC, cache-miss rates, and latency measurements. 3. **Controlled oversubscription.** Place multiple vCPU threads on one logical CPU, or on SMT siblings of one physical core, to evaluate oversubscription behavior. ## Related issue Closes #228 ## Scope and non-goals Included: - Runtime affinity changes for running Firecracker sandboxes. - Internal Firecracker PID resolution without exposing the PID through the public API. - CLI/client, gateway routing, admin API, orchestrator, backend, tests, and user documentation. Non-goals: - Exclusive CPU reservation or automatic topology selection. - Persisting affinity across pause/resume or runtime replacement. - Querying current per-thread affinity; this can reuse the process and thread abstractions introduced here in a follow-up. - Supporting host logical CPU IDs above 1023. ## Design and behavior changes ```text aenv -> gateway -> admin API -> orchestrator -> Firecracker PID -> /proc thread scan -> sched_get/setaffinity ``` The orchestrator accepts only running sandboxes, locks the backend while the blocking affinity operation runs, and keeps the operation cancellation-safe so pause, snapshot, or delete cannot replace the Firecracker process midway. The sandbox layer scans `/proc/<pid>/task`, identifies vCPU threads by their `fc_vcpu N` names, and checks thread start times around numeric-TID syscalls to narrow the TID-reuse window. Input length, expansion, value count, and CPU IDs are bounded before allocation. Requested offline CPUs are ignored; an empty online intersection is rejected before any affinity change. Before applying changes, the implementation records every target's original affinity. It then updates and reads back each thread one at a time. A failure or kernel mismatch triggers best-effort rollback of all threads already changed. ## Compatibility and operations - Public API or generated protocol: Adds one admin-only POST endpoint and its OpenAPI-generated server types. Existing endpoints are unchanged. - Configuration or defaults: No new configuration. The endpoint is registered by default and protected by the deployment API key; there is no separate feature flag. - Snapshot manifest, artifact layout, or storage format: N/A; affinity is live kernel state and is not persisted. - Upgrade and rollback: No data migration is required. Rolling back removes the API, but an already-applied affinity remains until changed or the Firecracker process exits. - Host requirements, permissions, ports, or dependencies: Linux `/proc` and permission to call `sched_setaffinity` on Firecracker threads. No new port or runtime dependency is introduced. Supported logical CPU IDs are 0-1023. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [x] Relevant Rust integration tests - [x] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [x] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results: ```text make fmt passed make clippy passed: workspace, all targets, all features, warnings denied make test-unit passed cargo test -p agentenv cpu_affinity --lib passed: 14 passed, 0 failed, 1 ignored helper process AENV_UBLK_DAEMON_METRICS_LISTEN_ADDR="" make test-agent-integration passed: AgentENV integration suites and snapshot OSS tests snapshot OSS result: 4 passed, 0 failed make -C services test passed: gateway and scheduler tests ``` Skipped checks and reasons: - `make agentenv-server`: generated server changes and their OpenAPI source are committed, but final regeneration could not be rerun because Java is not installed in the validation environment. - Benchmarks: not run; this adds an explicit administrative control path rather than changing a request-processing hot path. - End-to-end tests: not run because they require a privileged KVM/ublk runtime environment. ## Risks and reviewer notes - Incorrect placement can reduce performance for other workloads on the node; the endpoint is therefore admin-only. - Affinity limits scheduler eligibility but does not reserve CPUs or provide exclusive access. All selected threads receive the same host CPU set. - `*` selects threads present when the request is handled; it is not a policy for threads created later. - Affinity is not restored after pause/resume, because resume creates a new Firecracker process. - Numeric PID/TID syscalls cannot eliminate reuse races completely. Start-time checks before and after affinity reads narrow that window. - Rollback is best-effort and reports any thread that could not be restored. Suggested review order: 1. `src/sandbox/cpu_affinity.rs` for parsing, thread selection, verification, and rollback. 2. `src/orchestrator/service.rs` for lifecycle locking and PID handling. 3. `src/api/openapi.yml` and `src/api/impls/admin.rs` for the admin API. 4. `services/gateway/internal/server.go` and the `aenv` client/command for routing and user-facing behavior. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 14:21:46 +00:00

bsakaguchi referenced this issue

2026-10-29 14:22:06 +00:00

[PR #279] fix(overlaybd): derive direct I/O alignment from each file #314