[PR #44] [CLOSED] fix(orchestrator): resume sandboxes after recoverable pause failures #136

New issue

Closed

opened 2026-10-29 14:20:05 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:05 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/44
Author: @morluto
Created: 7/28/2026
Status: ❌ Closed

Base: main ← Head: fix/pause-failure-recovery

📝 Commits (10+)

447a4bb fix(orchestrator): recover running state after pause errors
5b5556c fix(orchestrator): retain retryable pause cleanup ownership
234895a fix(orchestrator): serialize pause and delete recovery
9a936b2 fix(orchestrator): make lifecycle ownership cancellation-safe
9a3bd8d fix(orchestrator): revalidate proxy routes during transitions
3bb7754 test(orchestrator): bound pause recovery synchronization
e5e7458 fix(orchestrator): retain handles across cleanup state errors
50737e2 fix(orchestrator): validate proxy route generations
9ea4ca0 fix(orchestrator): preserve joined pause outcomes
e02408a test(orchestrator): isolate pause restore failure script

📊 Changes

4 files changed (+2135 additions, -170 deletions)

View changed files

📝 src/orchestrator/proxy.rs (+61 -0)
📝 src/orchestrator/service.rs (+585 -65)
📝 src/orchestrator/tests.rs (+1487 -104)
📝 src/sandbox/backend.rs (+2 -1)

📄 Description

Problem

A recoverable backend pause failure can happen after Firecracker has already been paused. The orchestrator restored the sandbox handle, proxy route, and Running state without first resuming the backend, so a frozen VM could be published as healthy.

pause() returns recoverable error
                |
                v
             resume()
                |
                +-- success
                |      |
                |      v
                |   restore handle + route
                |      |
                |      v
                |   state: Running
                |      |
                |      v
                |   return pause error
                |
                `-- failure
                       |
                       v
                    stop + remove
                       |
                       v
                    state: Absent
                       |
                       v
                    return combined error

Changes

Call backend resume() before restoring the handle, route, and Running state after a recoverable pause failure.
If resume fails, stop the backend, remove sandbox metadata, keep the route detached, and release protected image references.
Include both the pause and recovery failures in the returned error.
Document that callers must resume after a recoverable backend pause failure before treating the sandbox as running.

Validation

The regression test failed against the previous implementation because resume() was called 0 times instead of 1.
pause_failure_rolls_back_to_running_and_preserves_handle passes after the fix.
pause_recovery_failure_stops_and_removes_sandbox verifies one stop call, metadata and route removal, and zero running resource metrics.
cargo clippy -p agentenv --all-targets -- -D warnings
cargo fmt --all -- --check

Closes #41.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/44 **Author:** [@morluto](https://github.com/morluto) **Created:** 7/28/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/pause-failure-recovery` --- ### 📝 Commits (10+) - [`447a4bb`](https://github.com/kvcache-ai/AgentENV/commit/447a4bb07e4a86a05effe0104d83f8a8c23781e1) fix(orchestrator): recover running state after pause errors - [`5b5556c`](https://github.com/kvcache-ai/AgentENV/commit/5b5556c60ef5e221cd9558452afde631eec1bb52) fix(orchestrator): retain retryable pause cleanup ownership - [`234895a`](https://github.com/kvcache-ai/AgentENV/commit/234895adfb671d31345a665b304de526571fa0cc) fix(orchestrator): serialize pause and delete recovery - [`9a936b2`](https://github.com/kvcache-ai/AgentENV/commit/9a936b24a678c8f34ad05ec4bff4c62728e113cf) fix(orchestrator): make lifecycle ownership cancellation-safe - [`9a3bd8d`](https://github.com/kvcache-ai/AgentENV/commit/9a3bd8da5ed0c30858d3d6a6499ff1b2eb5c1b42) fix(orchestrator): revalidate proxy routes during transitions - [`3bb7754`](https://github.com/kvcache-ai/AgentENV/commit/3bb77546b0580ac4ab83f5e03f48264be8d2d7bb) test(orchestrator): bound pause recovery synchronization - [`e5e7458`](https://github.com/kvcache-ai/AgentENV/commit/e5e745850a96f3f4736c8358facdd863d2b08fba) fix(orchestrator): retain handles across cleanup state errors - [`50737e2`](https://github.com/kvcache-ai/AgentENV/commit/50737e28008da0eba86db60e3cb05faab8e08e92) fix(orchestrator): validate proxy route generations - [`9ea4ca0`](https://github.com/kvcache-ai/AgentENV/commit/9ea4ca00583aeabddbb864e8587c0b9ad88406e6) fix(orchestrator): preserve joined pause outcomes - [`e02408a`](https://github.com/kvcache-ai/AgentENV/commit/e02408a1053e7ecdca4413618e3f902dea631f20) test(orchestrator): isolate pause restore failure script ### 📊 Changes **4 files changed** (+2135 additions, -170 deletions) <details> <summary>View changed files</summary> 📝 `src/orchestrator/proxy.rs` (+61 -0) 📝 `src/orchestrator/service.rs` (+585 -65) 📝 `src/orchestrator/tests.rs` (+1487 -104) 📝 `src/sandbox/backend.rs` (+2 -1) </details> ### 📄 Description ## Problem A recoverable backend pause failure can happen after Firecracker has already been paused. The orchestrator restored the sandbox handle, proxy route, and `Running` state without first resuming the backend, so a frozen VM could be published as healthy. ```text pause() returns recoverable error | v resume() | +-- success | | | v | restore handle + route | | | v | state: Running | | | v | return pause error | `-- failure | v stop + remove | v state: Absent | v return combined error ``` ## Changes - Call backend `resume()` before restoring the handle, route, and `Running` state after a recoverable pause failure. - If resume fails, stop the backend, remove sandbox metadata, keep the route detached, and release protected image references. - Include both the pause and recovery failures in the returned error. - Document that callers must resume after a recoverable backend pause failure before treating the sandbox as running. ## Validation - The regression test failed against the previous implementation because `resume()` was called 0 times instead of 1. - `pause_failure_rolls_back_to_running_and_preserves_handle` passes after the fix. - `pause_recovery_failure_stops_and_removes_sandbox` verifies one stop call, metadata and route removal, and zero running resource metrics. - `cargo clippy -p agentenv --all-targets -- -D warnings` - `cargo fmt --all -- --check` Closes #41. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:05 +00:00