kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #137] Single-node CPU intersection bypasses KVM read-only leaf filtering and breaks sandbox creation #47

New issue

Closed

opened 2026-10-29 14:18:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:47 +00:00

Owner

Originally created by @feimingxliu on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/137

Affected component

Gateway / scheduler, with impact on the Firecracker runtime and observability CPU-template reporting.

AgentENV version

Version or commit: main at 00ba6cb74f309a0cbd65f6ec415640e9e674a371 (v0.1.1-22-g00ba6cb according to git describe)
Container image tag used by the deployment: a site-local convenience tag, not an AgentENV release version
Local source modifications: none

Host environment

OS: Ubuntu 22.04
Kernel: Linux 6.12.33 with PVM/KVM support
Architecture: x86_64
Bare metal, VM, or cloud: VM with PVM enabled
ublk support: ublk_drv loaded and /dev/ublk-control available
CPU: Intel Skylake-family virtual CPU, 16 vCPUs
Memory: 30 GiB
Storage/filesystem: ext4
Kubernetes: single-node k3s cluster
Firecracker: v1.17.0-next.1 from the PVM dependency manifest

Operation and configuration

# Relevant runtime configuration:
AENV_VIRTUALIZATION_MODE=pvm

# Observability and scheduler reporting are enabled. The Firecracker package
# contains an executable cpu-template-helper, so the node runs:
cpu-template-helper template dump -o /dev/stdout

# The scheduler discovers exactly one AgentENV node.

# After the node reports its CPU configuration and receives the scheduler result:
aenv start --cold -d <OCI_IMAGE>

Steps to reproduce

Deploy Scheduler, Gateway, and one AgentENV node with Kubernetes discovery.
Leave observability scheduler reporting enabled and ensure the selected Firecracker dependency contains an executable cpu-template-helper.
Wait for the node log to report received cluster cpu config intersection from scheduler.
Create any cold-start sandbox.
Observe that Firecracker rejects the CPU configuration before boot.
As a control, make cpu-template-helper unavailable/non-executable, restart the scheduler and node, and repeat the same sandbox creation. It succeeds.

The failure reproduced consistently before disabling the helper. With the helper disabled, cold start, command execution, pause, resume, and memory-state preservation all succeeded.

Expected behavior

A CPU configuration returned by the scheduler should be safe to apply to Firecracker. A single-node cluster should still run the same sanitization that removes KVM read-only/dynamic CPUID leaves.

Actual behavior

Sandbox creation fails with HTTP 500. Firecracker returns:

Start microvm error: System configuration error: Failed to create guest config: Template changes a CPUID entry not supported by KVM: Leaf: b, Subleaf: 1

Root cause

IntersectCpuConfigs returns the only input unchanged when the cluster has one CPU report:

github.com/kvcache-ai/AgentENV@00ba6cb74f/services/scheduler/internal/cpu_template.go (L46-L52)

That bypasses JSON validation and the existing KVM read-only leaf filter. The same file already classifies leaf 0xB as read-only and removes it in the normal intersection path:

github.com/kvcache-ai/AgentENV@00ba6cb74f/services/scheduler/internal/cpu_template.go (L83-L85)

github.com/kvcache-ai/AgentENV@00ba6cb74f/services/scheduler/internal/cpu_template.go (L120-L135)

The current single-input unit test explicitly requires byte-for-byte passthrough and does not include a read-only CPUID leaf:

github.com/kvcache-ai/AgentENV@00ba6cb74f/services/scheduler/internal/cpu_template_test.go (L54-L63)

Why leaf 0xB, subleaf 1 appears in the dump:

Newer KVM versions intentionally omit CPUID.(EAX=0BH, ECX=1) from KVM_GET_SUPPORTED_CPUID.
Firecracker inserts that topology subleaf during its later CPUID normalization step.
cpu-template-helper template dump reads the already-configured vCPU and serializes every CPUID entry into custom-template JSON.
AgentENV sends the raw single-node dump back to Firecracker as a pre-boot custom CPU template. Template application happens against the KVM-supported CPUID set, before Firecracker's normalization step, so the injected topology entry is not present and is rejected.

Firecracker normalization source:

https://github.com/kvcache-ai/firecracker-next/blob/v1.17.0-next.1/src/vmm/src/cpu_config/x86_64/cpuid/normalize.rs#L265-L286

Helper dump conversion source:

https://github.com/kvcache-ai/firecracker-next/blob/v1.17.0-next.1/src/cpu-template-helper/src/template/dump/x86_64.rs#L18-L45

This was observed on PVM, but the control flow is not inherently PVM-specific. A recent standard KVM environment can be affected whenever its helper dump contains leaf 0xB.

Suggested fix

Remove the len(jsons) == 1 raw-return fast path, or pass a single input through a dedicated sanitizer before returning it.
Ensure singleton input is parsed and removes kvmReadOnlyLeafIDs, while preserving all safe fields unchanged.
Add a regression test where a single input contains leaf 0xB and verify the output omits it.
Consider validating singleton JSON as well; currently invalid JSON also bypasses parsing in the single-input path.
Optionally run or expose cpu-template-helper template verify before publishing a generated cluster template.

Frequency

Always, with the described single-node configuration.

Regression information

Last known good: unknown
First known bad: present in the initial open-source implementation (8f028b1369a4ce9a66d140174d0a646e72398517)

Additional context

The Firecracker documentation describes template dump as an entry point for creating a template and recommends stripping, reviewing, and verifying the result rather than replaying the raw dump directly:

https://github.com/firecracker-microvm/firecracker/blob/main/docs/cpu_templates/cpu-template-helper.md

AgentENV automates the dump/intersection/replay flow, so it needs to perform the required sanitization itself for every cluster size.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @feimingxliu on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/137 ## Affected component Gateway / scheduler, with impact on the Firecracker runtime and observability CPU-template reporting. ## AgentENV version - Version or commit: `main` at `00ba6cb74f309a0cbd65f6ec415640e9e674a371` (`v0.1.1-22-g00ba6cb` according to `git describe`) - Container image tag used by the deployment: a site-local convenience tag, not an AgentENV release version - Local source modifications: none ## Host environment - OS: Ubuntu 22.04 - Kernel: Linux 6.12.33 with PVM/KVM support - Architecture: x86_64 - Bare metal, VM, or cloud: VM with PVM enabled - ublk support: `ublk_drv` loaded and `/dev/ublk-control` available - CPU: Intel Skylake-family virtual CPU, 16 vCPUs - Memory: 30 GiB - Storage/filesystem: ext4 - Kubernetes: single-node k3s cluster - Firecracker: `v1.17.0-next.1` from the PVM dependency manifest ## Operation and configuration ```shell # Relevant runtime configuration: AENV_VIRTUALIZATION_MODE=pvm # Observability and scheduler reporting are enabled. The Firecracker package # contains an executable cpu-template-helper, so the node runs: cpu-template-helper template dump -o /dev/stdout # The scheduler discovers exactly one AgentENV node. # After the node reports its CPU configuration and receives the scheduler result: aenv start --cold -d <OCI_IMAGE> ``` ## Steps to reproduce 1. Deploy Scheduler, Gateway, and one AgentENV node with Kubernetes discovery. 2. Leave observability scheduler reporting enabled and ensure the selected Firecracker dependency contains an executable `cpu-template-helper`. 3. Wait for the node log to report `received cluster cpu config intersection from scheduler`. 4. Create any cold-start sandbox. 5. Observe that Firecracker rejects the CPU configuration before boot. 6. As a control, make `cpu-template-helper` unavailable/non-executable, restart the scheduler and node, and repeat the same sandbox creation. It succeeds. The failure reproduced consistently before disabling the helper. With the helper disabled, cold start, command execution, pause, resume, and memory-state preservation all succeeded. ## Expected behavior A CPU configuration returned by the scheduler should be safe to apply to Firecracker. A single-node cluster should still run the same sanitization that removes KVM read-only/dynamic CPUID leaves. ## Actual behavior Sandbox creation fails with HTTP 500. Firecracker returns: ```text Start microvm error: System configuration error: Failed to create guest config: Template changes a CPUID entry not supported by KVM: Leaf: b, Subleaf: 1 ``` ## Root cause `IntersectCpuConfigs` returns the only input unchanged when the cluster has one CPU report: https://github.com/kvcache-ai/AgentENV/blob/00ba6cb74f309a0cbd65f6ec415640e9e674a371/services/scheduler/internal/cpu_template.go#L46-L52 That bypasses JSON validation and the existing KVM read-only leaf filter. The same file already classifies leaf `0xB` as read-only and removes it in the normal intersection path: https://github.com/kvcache-ai/AgentENV/blob/00ba6cb74f309a0cbd65f6ec415640e9e674a371/services/scheduler/internal/cpu_template.go#L83-L85 https://github.com/kvcache-ai/AgentENV/blob/00ba6cb74f309a0cbd65f6ec415640e9e674a371/services/scheduler/internal/cpu_template.go#L120-L135 The current single-input unit test explicitly requires byte-for-byte passthrough and does not include a read-only CPUID leaf: https://github.com/kvcache-ai/AgentENV/blob/00ba6cb74f309a0cbd65f6ec415640e9e674a371/services/scheduler/internal/cpu_template_test.go#L54-L63 Why leaf `0xB`, subleaf `1` appears in the dump: 1. Newer KVM versions intentionally omit `CPUID.(EAX=0BH, ECX=1)` from `KVM_GET_SUPPORTED_CPUID`. 2. Firecracker inserts that topology subleaf during its later CPUID normalization step. 3. `cpu-template-helper template dump` reads the already-configured vCPU and serializes every CPUID entry into custom-template JSON. 4. AgentENV sends the raw single-node dump back to Firecracker as a pre-boot custom CPU template. Template application happens against the KVM-supported CPUID set, before Firecracker's normalization step, so the injected topology entry is not present and is rejected. Firecracker normalization source: https://github.com/kvcache-ai/firecracker-next/blob/v1.17.0-next.1/src/vmm/src/cpu_config/x86_64/cpuid/normalize.rs#L265-L286 Helper dump conversion source: https://github.com/kvcache-ai/firecracker-next/blob/v1.17.0-next.1/src/cpu-template-helper/src/template/dump/x86_64.rs#L18-L45 This was observed on PVM, but the control flow is not inherently PVM-specific. A recent standard KVM environment can be affected whenever its helper dump contains leaf `0xB`. ## Suggested fix - Remove the `len(jsons) == 1` raw-return fast path, or pass a single input through a dedicated sanitizer before returning it. - Ensure singleton input is parsed and removes `kvmReadOnlyLeafIDs`, while preserving all safe fields unchanged. - Add a regression test where a single input contains leaf `0xB` and verify the output omits it. - Consider validating singleton JSON as well; currently invalid JSON also bypasses parsing in the single-input path. - Optionally run or expose `cpu-template-helper template verify` before publishing a generated cluster template. ## Frequency Always, with the described single-node configuration. ## Regression information - Last known good: unknown - First known bad: present in the initial open-source implementation (`8f028b1369a4ce9a66d140174d0a646e72398517`) ## Additional context The Firecracker documentation describes `template dump` as an entry point for creating a template and recommends stripping, reviewing, and verifying the result rather than replaying the raw dump directly: https://github.com/firecracker-microvm/firecracker/blob/main/docs/cpu_templates/cpu-template-helper.md AgentENV automates the dump/intersection/replay flow, so it needs to perform the required sanitization itself for every cluster size. ## Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi closed this issue

2026-10-29 14:18:47 +00:00

bsakaguchi referenced this issue

2026-10-29 14:19:17 +00:00

[GH-ISSUE #294] Add lifecycle-aware per-sandbox resource metrics collection #78

bsakaguchi referenced this issue

2026-10-29 14:19:23 +00:00

[GH-ISSUE #333] perf(ublk): default ublk block devices to the none I/O scheduler #85

bsakaguchi referenced this issue

2026-10-29 14:20:07 +00:00

[PR #49] [CLOSED] feat(ublk): add weight-based I/O scheduler for fair sandbox bandwidth sharing #140