[GH-ISSUE #228] cpu-bind wanted #68

New issue

Open

opened 2026-10-29 14:19:06 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 14:19:06 +00:00

Owner

Copy link

Originally created by @emailcannotbeblank on GitHub (Aug 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/228

Affected component

Select:

Orchestrator / sandbox lifecycle

Problem statement

AgentENV does not provide a supported way to change the host CPU affinity of a running sandbox. This is needed for performance analysis and hardware characterization, where reproducible placement of Firecracker vCPU threads is important.

Today, an operator has to identify the compute node that owns the sandbox, correlate the sandbox with an internal Firecracker process, enumerate its threads through host process information or /proc, and invoke taskset or sched_setaffinity manually. This is unreliable and depends on internal implementation details. It also requires shell access to the exact compute node, so it cannot be used reliably through the AgentENV gateway in a multi-node deployment.

The requested capability is a sandbox-level API and CLI operation that binds selected Firecracker threads to specified host logical CPU IDs while keeping the Firecracker PID and TIDs internal to the compute node.

Use case

I am a hardware architect evaluating processor designs for agent workloads. We use AgentENV sandboxes for microbenchmarking and workload characterization on hosts with more than 100 logical CPUs. A typical session runs dozens of sandboxes and may change affinity before every benchmark run, often dozens of times per day.

The main use cases are:

1. Topology-aware performance optimization. Select logical CPUs according to LLC, NUMA, and SMT topology to improve the performance of long-running workloads.
2. Stable measurements. Prevent vCPU threads from migrating across many host CPUs so that PMU counters, IPC, cache-miss rates, and latency measurements have less run-to-run noise.
3. Controlled oversubscription. Place multiple vCPU threads on the same logical CPU, or on sibling logical CPUs of one physical core, to evaluate oversubscription behavior.

Current behavior and workarounds

There is currently no AgentENV API or CLI operation for changing CPU affinity after a sandbox has started.

The manual workaround is to:

1. Determine which compute node owns the sandbox.
2. Correlate the sandbox with its internal Firecracker process.
3. Enumerate the Firecracker TIDs, for example with ps or /proc/<pid>/task.
4. Apply taskset or sched_setaffinity to individual TIDs.

For example, a Firecracker process can contain threads such as:

TID     COMMAND
831624  firecracker
831628  fc_api
973593  fc_vcpu 0
973594  fc_vcpu 1
973595  kvm-nx-lpage-re

This workaround is insufficient because:

• process and thread IDs are internal, short-lived identifiers and may be reused;
• locating the process through internal log paths is unreliable;
• the operation requires host shell access and usually elevated permissions;
• it cannot be routed through the gateway to a remote compute node;
• malformed or extremely large CPU ranges can cause excessive resource use in a naive implementation;
• offline or unsupported CPU IDs are not handled consistently; and
• applying affinity thread by thread can leave a partially updated sandbox when an operation fails.

Desired behavior

AgentENV should provide an authenticated sandbox operation that changes CPU affinity on the compute node currently running the sandbox.

Acceptance criteria:

• A user can select one or more Firecracker vCPU indexes, such as 0, 0-3, or 0-10:2.
• A wildcard selector, *, targets every thread currently belonging to the Firecracker process, including non-vCPU helper threads.
• A user can specify a taskset-like list of host logical CPU IDs, including lists, ranges, and range strides.
• The compute node resolves the Firecracker PID and TIDs internally; the public API does not expose them.
• The operation is accepted only for a running sandbox and is routed to the compute node that owns it.
• User-controlled CPU lists are length- and range-bounded before expansion.
• Requested host CPUs are checked against online CPUs. Offline CPUs may be reported as ignored, but the operation fails when the online intersection is empty.
• The operation verifies the applied affinity and does not silently report success when only part of the request was applied.
• If a thread update fails, earlier changes are rolled back where possible and rollback failures are reported. The operation is best-effort transactional because Linux does not provide an atomic multi-thread affinity syscall.
• The response reports the normalized CPUs that were applied, ignored offline CPUs, and the number of bound threads.
• When the feature is disabled by node configuration, the compute node returns a clear error without modifying affinity.

Proposed approach

Add a sandbox API operation such as:

POST /sandboxes/{sandboxID}/cpu-affinity
Content-Type: application/json

{
  "vcpu": "0-1",
  "core": "4-7"
}

Add a corresponding CLI command:

aenv cpu-bind <sandbox-id> --vcpu 0-1 --core 4-7
aenv cpu-bind <sandbox-id> --vcpu '*' --core 4-7

The orchestrator should perform lifecycle validation and serialize the operation with other sandbox lifecycle operations. The compute-node sandbox backend should obtain the PID from the Firecracker child process it already owns. A low-level affinity module should enumerate threads, select targets, validate CPU lists, apply and verify affinity, and perform best-effort rollback on failure.

The PID is an internal implementation detail and should not be added to the public API, CLI, metadata, or persistent sandbox records.

I have implemented a working prototype in the following branch and can adapt it based on maintainer feedback:

https://github.com/emailcannotbeblank/AgentENV/tree/feature/bind-cpu

Compatibility and operational impact

API/config changes:

• Adds one sandbox CPU-affinity endpoint and corresponding request/response models.
• Adds an aenv cpu-bind CLI command and SDK client method.
• The prototype adds sandbox.cpu_affinity_enabled as a node-level switch: 1 enables CPU binding, while 0 disables it and causes the compute node to reject the operation immediately. The current default is 1.
• The switch is node-wide rather than per-user. Operators can enable the feature only on nodes or deployments intended for trusted advanced users. Disabling it reduces exposure of host CPU-topology information and prevents unauthorized or accidental affinity changes from degrading the performance of other users' workloads. Per-user authorization, if required, must be enforced by the deployment's API authorization layer.
• Existing sandbox APIs and clients remain unchanged unless the new operation is called.

Snapshot or storage format changes:

• None. No snapshot manifest, sandbox metadata, database, artifact layout, or storage format is changed.

New host/runtime requirements:

• Linux CPU-affinity syscalls must be available.
• The AgentENV runtime user must be allowed to change the affinity of the Firecracker process it owns.
• The current prototype supports host logical CPU IDs 0-1023, matching the fixed-size cpu_set_t used by the implementation.
• No public PID access and no additional host package are required.

Upgrade and rollback considerations:

• The gateway, compute-node server, generated API code, SDK, and CLI should be upgraded together when the new operation is required through a cluster deployment.
• Older nodes or gateways will not recognize the new endpoint and may return 404.
• Applied affinity is Linux process state. It remains in effect until changed or until the Firecracker process exits.
• Runtime affinity is not persisted across pause/resume, sandbox restart, migration, or process recreation.
• Rolling back to an older AgentENV binary removes the API but does not reset affinity on an already-running Firecracker process.

Documentation status:

• The prototype does not yet include comprehensive user and operator documentation.
• If the maintainers agree to add this feature, the pull request will include detailed documentation covering the configuration switch, API and CLI usage, permissions and security considerations, CPU-list syntax and limits, thread-selection semantics, error handling, and lifecycle behavior.

Alternatives considered

1. Expose the Firecracker PID and let users run taskset. Rejected because it leaks an unstable implementation detail, requires access to the exact compute node, is difficult to route through the gateway, and creates PID/TID reuse races.
2. Manage the whole sandbox through a cpuset cgroup. This may be valuable for stronger group-level isolation, but it is a larger lifecycle and host-resource-management change. It also does not by itself provide per-vCPU placement.
3. Configure affinity only when the sandbox starts. This does not support iterative benchmark workflows that need to change placement repeatedly without recreating the sandbox.
4. Keep using host scripts. This is not suitable for a multi-node API-driven platform and cannot provide consistent validation, error reporting, or rollback.

Non-goals

• Exposing Firecracker PIDs or TIDs through the public API or CLI.
• Providing exclusive CPU reservation or preventing unrelated host processes from using the selected CPUs.
• Automatically choosing CPUs based on NUMA, LLC, SMT, or other topology information.
• Changing scheduler capacity accounting or placement decisions.
• Persisting affinity across pause/resume, restart, migration, or Firecracker process recreation.
• Binding arbitrary guest processes or arbitrary host processes.
• Providing hard real-time scheduling guarantees.

Contribution

Select:

I can implement it

I have a working implementation in my fork and can update the code, tests, generated API files, and documentation after the API and scope are agreed upon.

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @emailcannotbeblank on GitHub (Aug 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/228 ### Affected component Select: ```text Orchestrator / sandbox lifecycle ``` ### Problem statement AgentENV does not provide a supported way to change the host CPU affinity of a running sandbox. This is needed for performance analysis and hardware characterization, where reproducible placement of Firecracker vCPU threads is important. Today, an operator has to identify the compute node that owns the sandbox, correlate the sandbox with an internal Firecracker process, enumerate its threads through host process information or `/proc`, and invoke `taskset` or `sched_setaffinity` manually. This is unreliable and depends on internal implementation details. It also requires shell access to the exact compute node, so it cannot be used reliably through the AgentENV gateway in a multi-node deployment. The requested capability is a sandbox-level API and CLI operation that binds selected Firecracker threads to specified host logical CPU IDs while keeping the Firecracker PID and TIDs internal to the compute node. ### Use case I am a hardware architect evaluating processor designs for agent workloads. We use AgentENV sandboxes for microbenchmarking and workload characterization on hosts with more than 100 logical CPUs. A typical session runs dozens of sandboxes and may change affinity before every benchmark run, often dozens of times per day. The main use cases are: 1. **Topology-aware performance optimization.** Select logical CPUs according to LLC, NUMA, and SMT topology to improve the performance of long-running workloads. 2. **Stable measurements.** Prevent vCPU threads from migrating across many host CPUs so that PMU counters, IPC, cache-miss rates, and latency measurements have less run-to-run noise. 3. **Controlled oversubscription.** Place multiple vCPU threads on the same logical CPU, or on sibling logical CPUs of one physical core, to evaluate oversubscription behavior. ### Current behavior and workarounds There is currently no AgentENV API or CLI operation for changing CPU affinity after a sandbox has started. The manual workaround is to: 1. Determine which compute node owns the sandbox. 2. Correlate the sandbox with its internal Firecracker process. 3. Enumerate the Firecracker TIDs, for example with `ps` or `/proc/<pid>/task`. 4. Apply `taskset` or `sched_setaffinity` to individual TIDs. For example, a Firecracker process can contain threads such as: ```text TID COMMAND 831624 firecracker 831628 fc_api 973593 fc_vcpu 0 973594 fc_vcpu 1 973595 kvm-nx-lpage-re ``` This workaround is insufficient because: - process and thread IDs are internal, short-lived identifiers and may be reused; - locating the process through internal log paths is unreliable; - the operation requires host shell access and usually elevated permissions; - it cannot be routed through the gateway to a remote compute node; - malformed or extremely large CPU ranges can cause excessive resource use in a naive implementation; - offline or unsupported CPU IDs are not handled consistently; and - applying affinity thread by thread can leave a partially updated sandbox when an operation fails. ### Desired behavior AgentENV should provide an authenticated sandbox operation that changes CPU affinity on the compute node currently running the sandbox. Acceptance criteria: - A user can select one or more Firecracker vCPU indexes, such as `0`, `0-3`, or `0-10:2`. - A wildcard selector, `*`, targets every thread currently belonging to the Firecracker process, including non-vCPU helper threads. - A user can specify a taskset-like list of host logical CPU IDs, including lists, ranges, and range strides. - The compute node resolves the Firecracker PID and TIDs internally; the public API does not expose them. - The operation is accepted only for a running sandbox and is routed to the compute node that owns it. - User-controlled CPU lists are length- and range-bounded before expansion. - Requested host CPUs are checked against online CPUs. Offline CPUs may be reported as ignored, but the operation fails when the online intersection is empty. - The operation verifies the applied affinity and does not silently report success when only part of the request was applied. - If a thread update fails, earlier changes are rolled back where possible and rollback failures are reported. The operation is best-effort transactional because Linux does not provide an atomic multi-thread affinity syscall. - The response reports the normalized CPUs that were applied, ignored offline CPUs, and the number of bound threads. - When the feature is disabled by node configuration, the compute node returns a clear error without modifying affinity. ### Proposed approach Add a sandbox API operation such as: ```http POST /sandboxes/{sandboxID}/cpu-affinity Content-Type: application/json { "vcpu": "0-1", "core": "4-7" } ``` Add a corresponding CLI command: ```bash aenv cpu-bind <sandbox-id> --vcpu 0-1 --core 4-7 aenv cpu-bind <sandbox-id> --vcpu '*' --core 4-7 ``` The orchestrator should perform lifecycle validation and serialize the operation with other sandbox lifecycle operations. The compute-node sandbox backend should obtain the PID from the Firecracker child process it already owns. A low-level affinity module should enumerate threads, select targets, validate CPU lists, apply and verify affinity, and perform best-effort rollback on failure. The PID is an internal implementation detail and should not be added to the public API, CLI, metadata, or persistent sandbox records. I have implemented a working prototype in the following branch and can adapt it based on maintainer feedback: <https://github.com/emailcannotbeblank/AgentENV/tree/feature/bind-cpu> ### Compatibility and operational impact **API/config changes:** - Adds one sandbox CPU-affinity endpoint and corresponding request/response models. - Adds an `aenv cpu-bind` CLI command and SDK client method. - The prototype adds `sandbox.cpu_affinity_enabled` as a node-level switch: `1` enables CPU binding, while `0` disables it and causes the compute node to reject the operation immediately. The current default is `1`. - The switch is node-wide rather than per-user. Operators can enable the feature only on nodes or deployments intended for trusted advanced users. Disabling it reduces exposure of host CPU-topology information and prevents unauthorized or accidental affinity changes from degrading the performance of other users' workloads. Per-user authorization, if required, must be enforced by the deployment's API authorization layer. - Existing sandbox APIs and clients remain unchanged unless the new operation is called. **Snapshot or storage format changes:** - None. No snapshot manifest, sandbox metadata, database, artifact layout, or storage format is changed. **New host/runtime requirements:** - Linux CPU-affinity syscalls must be available. - The AgentENV runtime user must be allowed to change the affinity of the Firecracker process it owns. - The current prototype supports host logical CPU IDs `0-1023`, matching the fixed-size `cpu_set_t` used by the implementation. - No public PID access and no additional host package are required. **Upgrade and rollback considerations:** - The gateway, compute-node server, generated API code, SDK, and CLI should be upgraded together when the new operation is required through a cluster deployment. - Older nodes or gateways will not recognize the new endpoint and may return `404`. - Applied affinity is Linux process state. It remains in effect until changed or until the Firecracker process exits. - Runtime affinity is not persisted across pause/resume, sandbox restart, migration, or process recreation. - Rolling back to an older AgentENV binary removes the API but does not reset affinity on an already-running Firecracker process. **Documentation status:** - The prototype does not yet include comprehensive user and operator documentation. - If the maintainers agree to add this feature, the pull request will include detailed documentation covering the configuration switch, API and CLI usage, permissions and security considerations, CPU-list syntax and limits, thread-selection semantics, error handling, and lifecycle behavior. ### Alternatives considered 1. **Expose the Firecracker PID and let users run `taskset`.** Rejected because it leaks an unstable implementation detail, requires access to the exact compute node, is difficult to route through the gateway, and creates PID/TID reuse races. 2. **Manage the whole sandbox through a cpuset cgroup.** This may be valuable for stronger group-level isolation, but it is a larger lifecycle and host-resource-management change. It also does not by itself provide per-vCPU placement. 3. **Configure affinity only when the sandbox starts.** This does not support iterative benchmark workflows that need to change placement repeatedly without recreating the sandbox. 4. **Keep using host scripts.** This is not suitable for a multi-node API-driven platform and cannot provide consistent validation, error reporting, or rollback. ### Non-goals - Exposing Firecracker PIDs or TIDs through the public API or CLI. - Providing exclusive CPU reservation or preventing unrelated host processes from using the selected CPUs. - Automatically choosing CPUs based on NUMA, LLC, SMT, or other topology information. - Changing scheduler capacity accounting or placement decisions. - Persisting affinity across pause/resume, restart, migration, or Firecracker process recreation. - Binding arbitrary guest processes or arbitrary host processes. - Providing hard real-time scheduling guarantees. ### Contribution Select: ```text I can implement it ``` I have a working implementation in my fork and can update the code, tests, generated API files, and documentation after the API and scope are agreed upon. ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:06 +00:00

bsakaguchi commented 2026-10-29 14:19:07 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Sep 1, 2026):

Thanks! CPU-binding is valuable, but it can interfere with load balancing and other sandboxes, so it should be used with caution. Since we currently only support single-tenant deployments, this is acceptable. That said, because this feature requires knowledge of the host CPU, I'd suggest moving the API into the admin interface.

<!-- gh-comment-id:5491456723 --> @yingdi-shan commented on GitHub (Sep 1, 2026): Thanks! CPU-binding is valuable, but it can interfere with load balancing and other sandboxes, so it should be used with caution. Since we currently only support single-tenant deployments, this is acceptable. That said, because this feature requires knowledge of the host CPU, I'd suggest moving the API into the admin interface.

bsakaguchi commented 2026-10-29 14:19:07 +00:00

Author

Owner

Copy link

@emailcannotbeblank commented on GitHub (Sep 6, 2026):

Thanks! CPU-binding is valuable, but it can interfere with load balancing and other sandboxes, so it should be used with caution. Since we currently only support single-tenant deployments, this is acceptable. That said, because this feature requires knowledge of the host CPU, I'd suggest moving the API into the admin interface.

The operation has already been moved into the admin interface (src/api/impls/admin.rs) to reflect its privileged nature. AgentENV’s admin authentication flow is not yet fully implemented end-to-end across the CLI, gateway, and runtime server. Completing that authorization design is orthogonal to CPU affinity, so I propose addressing it in a separate PR while keeping this PR focused on the binding functionality.

<!-- gh-comment-id:5556833382 --> @emailcannotbeblank commented on GitHub (Sep 6, 2026): > Thanks! CPU-binding is valuable, but it can interfere with load balancing and other sandboxes, so it should be used with caution. Since we currently only support single-tenant deployments, this is acceptable. That said, because this feature requires knowledge of the host CPU, I'd suggest moving the API into the admin interface. The operation has already been moved into the admin interface (`src/api/impls/admin.rs`) to reflect its privileged nature. AgentENV’s admin authentication flow is not yet fully implemented end-to-end across the CLI, gateway, and runtime server. Completing that authorization design is orthogonal to CPU affinity, so I propose addressing it in a separate PR while keeping this PR focused on the binding functionality.

bsakaguchi referenced this issue 2026-10-29 14:19:13 +00:00

[GH-ISSUE #262] Add offline workload replay for scheduler strategy evaluation #73

bsakaguchi referenced this issue 2026-10-29 14:19:14 +00:00

[GH-ISSUE #262] Add offline workload replay for scheduler strategy evaluation #73

bsakaguchi referenced this issue 2026-10-29 14:19:16 +00:00

[GH-ISSUE #262] Add offline workload replay for scheduler strategy evaluation #73

bsakaguchi referenced this issue 2026-10-29 14:20:21 +00:00

[PR #68] feat(scheduler): add resource-bounded grouped round-robin scheduling #154

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#68

No description provided.