[GH-ISSUE #41] Recoverable pause failures can leave sandboxes frozen while reported as Running #20

New issue

Closed

opened 2026-10-29 14:18:28 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:18:28 +00:00

Owner

Originally created by @morluto on GitHub (Jul 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/41

Problem

A recoverable backend pause failure can occur after Firecracker has already been paused. The orchestrator currently restores the sandbox handle, proxy route, and Running state without first calling resume().

The sandbox can therefore be discoverable and routable as Running while its VM remains frozen.

pause() returns recoverable error
                |
                v
             resume()
                |
                +-- success
                |      |
                |      v
                |   restore handle + route
                |      |
                |      v
                |   state: Running
                |      |
                |      v
                |   return pause error
                |
                `-- failure
                       |
                       v
                    stop + remove
                       |
                       v
                    state: Absent
                       |
                       v
                    return combined error

Current behavior

On a non-terminal SandboxPauseError, pause_sandbox_inner reinserts the detached handle, restores the route, and transitions Pausing -> Running.

The backend contract only says that a recoverable failure leaves the runtime safe to resume; it does not guarantee that the VM is already running.

Expected behavior

Call the backend's resume() before restoring the handle, route, and Running state.
Restore the sandbox only after resume succeeds.
If resume also fails, stop and remove the sandbox instead of publishing a false healthy state.
Preserve useful context from both the pause and recovery failures.

Impact

Pause and resume are central to AgentENV's low-cost idle environment model. Returning a frozen VM to the running pool can leave agent workloads apparently healthy but unable to make progress, while routing and resource metrics continue to treat the sandbox as active.

Originally created by @morluto on GitHub (Jul 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/41 ## Problem A recoverable backend pause failure can occur after Firecracker has already been paused. The orchestrator currently restores the sandbox handle, proxy route, and `Running` state without first calling `resume()`. The sandbox can therefore be discoverable and routable as `Running` while its VM remains frozen. ```text pause() returns recoverable error | v resume() | +-- success | | | v | restore handle + route | | | v | state: Running | | | v | return pause error | `-- failure | v stop + remove | v state: Absent | v return combined error ``` ## Current behavior On a non-terminal `SandboxPauseError`, `pause_sandbox_inner` reinserts the detached handle, restores the route, and transitions `Pausing -> Running`. The backend contract only says that a recoverable failure leaves the runtime safe to resume; it does not guarantee that the VM is already running. ## Expected behavior - Call the backend's `resume()` before restoring the handle, route, and `Running` state. - Restore the sandbox only after resume succeeds. - If resume also fails, stop and remove the sandbox instead of publishing a false healthy state. - Preserve useful context from both the pause and recovery failures. ## Impact Pause and resume are central to AgentENV's low-cost idle environment model. Returning a frozen VM to the running pool can leave agent workloads apparently healthy but unable to make progress, while routing and resource metrics continue to treat the sandbox as active.

bsakaguchi closed this issue

2026-10-29 14:18:28 +00:00

bsakaguchi commented

2026-10-29 14:18:29 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Jul 29, 2026):

Could you please share the steps to reproduce this issue?

@yingdi-shan commented on GitHub (Jul 29, 2026): Could you please share the steps to reproduce this issue?

bsakaguchi commented

2026-10-29 14:18:29 +00:00

Author

Owner

@morluto commented on GitHub (Jul 29, 2026):

Yes. This can be reproduced directly through the orchestrator's fault-injection test; KVM is not required.

gh pr checkout 44
cargo test -p agentenv --lib \
  pause_failure_rolls_back_to_running_and_preserves_handle -- --nocapture

The test injects a recoverable backend pause failure and checks that the orchestrator calls resume() before restoring the sandbox as Running.

I also ran this regression test against the parent implementation while retaining the test. It failed with resume() called 0 times instead of 1. With the PR implementation, it passes.

The equivalent production condition is a failure during snapshot processing after Firecracker has already been paused.

@morluto commented on GitHub (Jul 29, 2026): Yes. This can be reproduced directly through the orchestrator's fault-injection test; KVM is not required. ```bash gh pr checkout 44 cargo test -p agentenv --lib \ pause_failure_rolls_back_to_running_and_preserves_handle -- --nocapture ``` The test injects a recoverable backend pause failure and checks that the orchestrator calls `resume()` before restoring the sandbox as `Running`. I also ran this regression test against the parent implementation while retaining the test. It failed with `resume()` called 0 times instead of 1. With the PR implementation, it passes. The equivalent production condition is a failure during snapshot processing after Firecracker has already been paused.

bsakaguchi referenced this issue

2026-10-29 14:20:39 +00:00

[PR #93] fix(install): warn for known affected ublk kernels #174

bsakaguchi referenced this issue

2026-10-29 14:21:34 +00:00

[PR #218] [CLOSED] feat: add mincore snapshot profiling and KVM prefault #259