[GH-ISSUE #99] Pause allocation failures leave the sandbox detached in Pausing #34

New issue

Closed

opened 2026-10-29 14:18:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:38 +00:00

Owner

Originally created by @morluto on GitHub (Jul 31, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/99

What happened?

If paused-sandbox artifact-root allocation fails, pause_sandbox returns an error after it has already removed the sandbox's runtime handle and proxy route. The metadata remains in Pausing, so the still-running sandbox is no longer reachable and a later pause cannot retry the operation.

This was reproduced at f41abb21324f6b0520abf34b7720aa260ddd10eb with the in-memory orchestrator and a persister that deterministically fails its next allocate_artifact_root call. No KVM or privileged host setup is required.

Where does it happen?

pause_sandbox_inner protects the paused runtime artifacts and then calls detach_sandbox_handle_and_route before allocating the artifact root:

src/orchestrator/service.rs: the handle and route are detached immediately before SandboxPersister::allocate_artifact_root.
On the baseline, allocation uses await?, so its error bypasses the rollback implemented for later sandbox.pause(...) and persistence failures.

The resulting control flow is:

Running
   |
   v
Pausing -- protect PausedSandbox refs
   |
   +-- detach runtime handle
   +-- remove proxy route
   |
   v
allocate_artifact_root() fails
   |
   +-- returns error without rollback
   v
metadata: Pausing   runtime: still alive but detached   route: absent

How can we reproduce it?

Construct an orchestrator with RecordingPersister.
Configure RecordingPersister::fail_next(RecordingCall::AllocateArtifactRoot).
Create a sandbox and call pause_sandbox.
Observe that the call returns SandboxPersistenceFailed.
On the baseline, inspect the postcondition: the metadata is not restored to Running, the runtime handle and proxy route have already been removed, and the temporary PausedSandbox image ownership is not released by this path.

A regression test can exercise the behavior entirely through the existing mock backend and persister seam. The important assertion is not only that the first call fails: after rollback, the sandbox must be routable and a second pause_sandbox call must succeed.

What did you expect to happen?

Artifact-root allocation happens before the backend mutates the running VM, so this failure should be recoverable. The operation should:

restore the runtime handle;
restore the removed proxy route;
transition metadata from Pausing back to Running;
release the temporary PausedSandbox image references; and
return the allocation error while leaving a subsequent pause retryable.

If restoring metadata itself fails, the returned error should preserve both the allocation failure and rollback failure rather than reporting a clean recovery.

Acceptance criteria

A deterministic unit test forces artifact-root allocation to fail after handle/route detachment.
After the failure, metadata is Running with no paused state, the runtime handle is present, and the proxy route is ready.
The temporary RuntimeImageOwner::PausedSandbox reference is released.
No paused state is persisted for the failed attempt.
Retrying pause succeeds and reaches Paused.
Existing rollback behavior for backend pause failures and persistence failures is unchanged.

Anything else we need to know?

This report is scoped to the recoverable allocation failure before SandboxBackend::pause is invoked. It does not propose changing terminal backend pause failures, stop-failure handling, or restart-time persistence cleanup.

Environment for the deterministic reproduction: Linux x86_64, Rust stable toolchain. The proof is unit-level and does not require /dev/kvm because the failure occurs before backend pause execution.

Originally created by @morluto on GitHub (Jul 31, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/99 ## What happened? If paused-sandbox artifact-root allocation fails, `pause_sandbox` returns an error after it has already removed the sandbox's runtime handle and proxy route. The metadata remains in `Pausing`, so the still-running sandbox is no longer reachable and a later pause cannot retry the operation. This was reproduced at `f41abb21324f6b0520abf34b7720aa260ddd10eb` with the in-memory orchestrator and a persister that deterministically fails its next `allocate_artifact_root` call. No KVM or privileged host setup is required. ## Where does it happen? `pause_sandbox_inner` protects the paused runtime artifacts and then calls `detach_sandbox_handle_and_route` before allocating the artifact root: - `src/orchestrator/service.rs`: the handle and route are detached immediately before `SandboxPersister::allocate_artifact_root`. - On the baseline, allocation uses `await?`, so its error bypasses the rollback implemented for later `sandbox.pause(...)` and persistence failures. The resulting control flow is: ```text Running | v Pausing -- protect PausedSandbox refs | +-- detach runtime handle +-- remove proxy route | v allocate_artifact_root() fails | +-- returns error without rollback v metadata: Pausing runtime: still alive but detached route: absent ``` ## How can we reproduce it? 1. Construct an orchestrator with `RecordingPersister`. 2. Configure `RecordingPersister::fail_next(RecordingCall::AllocateArtifactRoot)`. 3. Create a sandbox and call `pause_sandbox`. 4. Observe that the call returns `SandboxPersistenceFailed`. 5. On the baseline, inspect the postcondition: the metadata is not restored to `Running`, the runtime handle and proxy route have already been removed, and the temporary `PausedSandbox` image ownership is not released by this path. A regression test can exercise the behavior entirely through the existing mock backend and persister seam. The important assertion is not only that the first call fails: after rollback, the sandbox must be routable and a second `pause_sandbox` call must succeed. ## What did you expect to happen? Artifact-root allocation happens before the backend mutates the running VM, so this failure should be recoverable. The operation should: 1. restore the runtime handle; 2. restore the removed proxy route; 3. transition metadata from `Pausing` back to `Running`; 4. release the temporary `PausedSandbox` image references; and 5. return the allocation error while leaving a subsequent pause retryable. If restoring metadata itself fails, the returned error should preserve both the allocation failure and rollback failure rather than reporting a clean recovery. ## Acceptance criteria - A deterministic unit test forces artifact-root allocation to fail after handle/route detachment. - After the failure, metadata is `Running` with no paused state, the runtime handle is present, and the proxy route is ready. - The temporary `RuntimeImageOwner::PausedSandbox` reference is released. - No paused state is persisted for the failed attempt. - Retrying pause succeeds and reaches `Paused`. - Existing rollback behavior for backend pause failures and persistence failures is unchanged. ## Anything else we need to know? This report is scoped to the recoverable allocation failure before `SandboxBackend::pause` is invoked. It does not propose changing terminal backend pause failures, stop-failure handling, or restart-time persistence cleanup. Environment for the deterministic reproduction: Linux x86_64, Rust stable toolchain. The proof is unit-level and does not require `/dev/kvm` because the failure occurs before backend pause execution.

bsakaguchi closed this issue

2026-10-29 14:18:38 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:02 +00:00

[PR #34] [CLOSED] fix(bench): handle empty samples in benchmark #131