[GH-ISSUE #372] bug(orchestrator): resume racing shutdown strands the sandbox in Resuming and its paused artifacts are destroyed on next startup #96

New issue

Open

opened 2026-10-29 14:19:31 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:19:31 +00:00

Owner

Copy link

Originally created by @tm-worker on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/372

Affected component

Orchestrator sandbox lifecycle (src/orchestrator/service.rs) + paused-sandbox persistence (src/orchestrator/persistence/file_backed.rs)

AgentENV version

main @ 4968423 (2026-10-12) — re-verified on this revision; first analyzed on 34cdc80 (2026-10-09). The affected files (src/orchestrator/service.rs, src/api/impls/sandbox.rs, src/api/impls/volumes.rs, src/volume.rs) are byte-identical between the two revisions.

Host environment

• Any Linux host is affected. Verified on Debian 12 (bookworm), linux/aarch64, inside a rust:bookworm container on Docker Desktop for macOS (Apple Silicon VM); plain userspace cargo test -p agentenv --lib — no KVM/ublk/network-namespace needed by the reproduction.
• Source modifications: none to product code. All verification ran against an unmodified git-archive copy of the named revision; the only local additions were the reproduction test appended to the test module — exactly the code embedded below — plus a persister test double (GatedMarkResumingPersister) that wraps the repo's own RecordingPersister and gates the first mark_resuming on a oneshot. The test is reproduced verbatim below ("Reproduction test"). It runs inside the orchestrator test module (src/orchestrator/tests.rs, same crate, so the helpers it references are that module's existing test infrastructure) with an in-memory metadata store.

Operation and configuration

Default configuration. The window is between a resume's persisted Resuming mark and the launch_sandbox entry gate — realistically hit when a node drains (rolling redeploy) while an auto-resume (proxy lookup on a paused sandbox) or a client resume is in flight.

Steps to reproduce

Deterministic interleaving (unit test, full code below):

1. Sandbox is Paused (store state Paused, persisted record lifecycle=Paused, artifacts under artifacts/<id>/).
2. Task A starts resume_sandbox(id); it passes the resume entry gate (shutdown not yet requested), CASes the store Paused→Resuming, and is about to / just did mark_resuming (a persisted, fsync'd write).
3. Shutdown begins (is_shutting_down = true) — before Task A reaches launch_sandbox.
4. Task A reaches launch_sandbox, whose FIRST statement is ensure_accepting_lifecycle_operations() → Err(ShuttingDown). It returns without any rollback.
5. Observe: store state stays Resuming. Nothing ever advances it.

The test parks the resume task inside a gated mark_resuming so step 3 lands inside the window deterministically, and asserts the outcome.

Expected behavior

The failed resume must roll back to Paused — exactly like every other failure path in launch_sandbox does via rollback_failed_launch_metadata (which performs a Resuming→Paused CAS plus persister.rollback_resuming). Either the entry-gate rejection for a Resume plan should run that rollback, or the Resuming mark/CAS should happen after the gate.

Actual behavior

• The ShuttingDown rejection at the launch_sandbox entry gate is the ONLY failure path that skips the rollback: build failure, protect failure, start failure, the three shutdown checks inside launch, add failure, wait-ready failure and the terminal-CAS failure all restore Paused (service.rs:2632-2662) — the entry gate at service.rs:2400-2401 does not.
• Consequence A (live process): the sandbox is stranded in Resuming. resume, delete, pause and keep-alive all wait on wait_for_transition(Resuming) for 60s (WAIT_TRANSITION_TIMEOUT) and fail. The drain itself also stalls: shutdown cleanup waits up to 3 rounds × 60s on the stranded sandbox and reports "failed to preserve all sandboxes".
• Consequence B (after restart): the persistence loader treats a record with lifecycle=Resuming as invalid and calls cleanup_invalid_record — it deletes the record AND remove_dir_alls the whole artifacts/<id>/ directory (file_backed.rs:278-282, "discarding paused sandbox record left in resuming state"). The paused memory snapshot and rootfs writable-layer copy are destroyed — even though this VM never began resuming (the launch was rejected before build started).

Logs and diagnostics

Timestamped tracing excerpt from the first run (34cdc80, Linux container):

2026-10-12T10:27:52.915674Z INFO agentenv::orchestrator::persistence::file_backed: loading paused sandbox records store=/tmp/.tmpbXXK4X/sandboxes
2026-10-12T10:27:52.925673Z INFO resume_sandbox: agentenv::orchestrator::service: resuming sandbox sandbox_id=01a12fa7-8fc4-7503-a5fb-8658552fb80b timeout=UseExisting
2026-10-12T10:27:52.929140Z INFO resume_sandbox:launch_sandbox: agentenv::orchestrator::service: rejecting lifecycle operation because orchestrator is shutting down sandbox_id=01a12fa7-8fc4-7503-a5fb-8658552fb80b timeout=UseExisting
2026-10-12T10:27:53 test orchestrator::service::tests::resume_racing_shutdown_strands_sandbox_in_resuming ... ok
2026-10-12T10:27:53 test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 959 filtered out; finished in 0.12s

Re-run on 4968423 (same behavior, same assertions):

test orchestrator::service::tests::resume_racing_shutdown_strands_sandbox_in_resuming ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 966 filtered out; finished in 0.13s

The test asserts store.state == Resuming after the ShuttingDown error — i.e. the bug behavior. With a correct rollback it would assert Paused.

Reproduction test

Append the following to src/orchestrator/tests.rs (its helpers — setup,
paused_resume_metadata, make_orchestrator_without_background_with_factory_and_persister,
MockBackendFactory, InMemoryMetadataStore — are that module's existing test
infrastructure) and run:

cargo test -p agentenv --lib resume_racing_shutdown_strands

/// Test double: gates the first `mark_resuming` call on an external signal
/// so a test can flip `is_shutting_down` inside the resume window between
/// the persisted `Resuming` mark and the `launch_sandbox` entry gate.
struct GatedMarkResumingPersister {
    inner: RecordingPersister,
    gate: tokio::sync::Mutex<Option<tokio::sync::oneshot::Receiver<()>>>,
}

impl GatedMarkResumingPersister {
    fn gated() -> (Self, tokio::sync::oneshot::Sender<()>) {
        let (tx, rx) = tokio::sync::oneshot::channel();
        (
            Self {
                inner: RecordingPersister::default(),
                gate: tokio::sync::Mutex::new(Some(rx)),
            },
            tx,
        )
    }
}

#[async_trait::async_trait]
impl SandboxPersister for GatedMarkResumingPersister {
    async fn load_all<F>(
        &self,
        factory: &F,
    ) -> crate::orchestrator::persistence::PersistenceResult<Vec<SandboxMetadata>>
    where
        F: crate::sandbox::SandboxBackendFactory,
    {
        self.inner.load_all(factory).await
    }

    async fn allocate_artifact_root(
        &self,
        sandbox_id: &SandboxId,
    ) -> crate::orchestrator::persistence::PersistenceResult<Option<std::path::PathBuf>> {
        self.inner.allocate_artifact_root(sandbox_id).await
    }

    async fn persist_paused(
        &self,
        metadata: &SandboxMetadata,
        artifact_root: Option<&std::path::Path>,
        paused_state: &dyn crate::sandbox::PausedSandboxState,
    ) -> crate::orchestrator::persistence::PersistenceResult<()> {
        self.inner
            .persist_paused(metadata, artifact_root, paused_state)
            .await
    }

    async fn mark_resuming(
        &self,
        sandbox_id: &SandboxId,
    ) -> crate::orchestrator::persistence::PersistenceResult<()> {
        if let Some(rx) = self.gate.lock().await.take() {
            let _ = rx.await; // park here until the test flips the flag
        }
        self.inner.mark_resuming(sandbox_id).await
    }

    async fn rollback_resuming(
        &self,
        sandbox_id: &SandboxId,
    ) -> crate::orchestrator::persistence::PersistenceResult<()> {
        self.inner.rollback_resuming(sandbox_id).await
    }

    async fn delete_record(
        &self,
        sandbox_id: &SandboxId,
    ) -> crate::orchestrator::persistence::PersistenceResult<()> {
        self.inner.delete_record(sandbox_id).await
    }

    async fn delete_record_and_artifacts(
        &self,
        sandbox_id: &SandboxId,
    ) -> crate::orchestrator::persistence::PersistenceResult<()> {
        self.inner.delete_record_and_artifacts(sandbox_id).await
    }
}

/// Regression probe for issue "resume racing shutdown strands sandbox in
/// Resuming": the shutdown flag flips while the resume task is parked
/// between its `Resuming` CAS + persisted mark and the `launch_sandbox`
/// entry gate. The entry gate rejects with `ShuttingDown` and — because
/// that path skips `rollback_failed_launch_metadata` — the sandbox is
/// stranded in `Resuming` (pause/delete/keep-alive time out against it, and
/// the next startup discards the paused artifacts entirely).
#[tokio::test]
async fn resume_racing_shutdown_strands_sandbox_in_resuming() -> Result<()> {
    use std::time::Duration;

    setup();
    let sandbox_id = SandboxId::new();
    let (persister, release_mark) = GatedMarkResumingPersister::gated();
    let orchestrator = make_orchestrator_without_background_with_factory_and_persister(
        InMemoryMetadataStore::new(),
        MockBackendFactory::new(),
        persister,
    );
    orchestrator
        .store
        .add(paused_resume_metadata(sandbox_id))
        .await?;

    let task = tokio::spawn({
        let orchestrator = Arc::clone(&orchestrator);
        async move { orchestrator.resume_sandbox(sandbox_id, NewTimeout::UseExisting).await }
    });

    // Wait until the resume CAS flips the store state to `Resuming`; the
    // spawned task is now parked inside the gated `mark_resuming`.
    let deadline = std::time::Instant::now() + Duration::from_secs(10);
    loop {
        let state = orchestrator
            .store
            .get(&sandbox_id)
            .await?
            .expect("paused metadata should exist")
            .state;
        if state == SandboxState::Resuming {
            break;
        }
        assert!(
            std::time::Instant::now() < deadline,
            "resume never reached the Resuming state"
        );
        tokio::time::sleep(Duration::from_millis(5)).await;
    }

    orchestrator.is_shutting_down.store(true, Ordering::Release);
    release_mark.send(()).expect("gate receiver alive");

    let err = task
        .await
        .expect("resume task should not panic")
        .expect_err("resume should fail once shutdown has begun");
    assert!(
        matches!(err, OrchestratorError::ShuttingDown),
        "expected ShuttingDown, got: {err:?}"
    );

    // BUG: nothing rolls the CAS back — the sandbox is stranded.
    let stranded = orchestrator
        .store
        .get(&sandbox_id)
        .await?
        .expect("stranded metadata should exist");
    assert_eq!(
        stranded.state,
        SandboxState::Resuming,
        "resume racing shutdown must strand the sandbox in Resuming (expected rollback to Paused)"
    );
    Ok(())
}

Reproduction frequency

Deterministic in the unit test (gated persister double). In production it requires the shutdown flag to land inside the CAS/mark→launch window (the window contains a fsync'd RocksDB write — milliseconds; rolling redeploys with traffic make this plausible).

Regression information

Not a regression we can trace — the resume rollback machinery exists for every other failure stage; only the entry-gate path lacks it. Existing tests cover shutdown rejection for CREATE plans only (tests.rs:4761 asserts the store is empty after a rejected create); no test covered the resume plan until now.

Additional context

Fix sketch (happy to open a PR):

• in launch_sandbox, when ensure_accepting_lifecycle_operations() fails for a Resume plan, run the same Resuming→Paused CAS + persister.rollback_resuming used by rollback_failed_launch_metadata; or
• move the gate check into resume_sandbox_inner BEFORE the CAS (there is already an entry gate at the top of resume_sandbox_inner — the race window is between that gate and the launch gate), keeping rollback for the launch path.

The secondary unrolled early-exit in the same function (missing paused_state, service.rs:1741-1744, after CAS+mark) should get the same treatment.

Pre-submission checklist

• I searched existing open and closed issues and GitHub Discussions for duplicates — 0 related results.
• I provided a minimal reproduction (deterministic unit test, reproduced in full above).
• I removed credentials, tokens, private image references, and other sensitive information.
• This is not a security vulnerability.

Originally created by @tm-worker on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/372 ### Affected component Orchestrator sandbox lifecycle (`src/orchestrator/service.rs`) + paused-sandbox persistence (`src/orchestrator/persistence/file_backed.rs`) ### AgentENV version `main` @ `4968423` (2026-10-12) — re-verified on this revision; first analyzed on `34cdc80` (2026-10-09). The affected files (`src/orchestrator/service.rs`, `src/api/impls/sandbox.rs`, `src/api/impls/volumes.rs`, `src/volume.rs`) are byte-identical between the two revisions. ### Host environment - Any Linux host is affected. Verified on Debian 12 (bookworm), linux/aarch64, inside a `rust:bookworm` container on Docker Desktop for macOS (Apple Silicon VM); plain userspace `cargo test -p agentenv --lib` — no KVM/ublk/network-namespace needed by the reproduction. - **Source modifications: none to product code.** All verification ran against an **unmodified** git-archive copy of the named revision; the only local additions were the reproduction test appended to the test module — exactly the code embedded below — plus a persister test double (`GatedMarkResumingPersister`) that wraps the repo's own `RecordingPersister` and gates the first `mark_resuming` on a oneshot. The test is reproduced verbatim below ("Reproduction test"). It runs inside the orchestrator test module (`src/orchestrator/tests.rs`, same crate, so the helpers it references are that module's existing test infrastructure) with an in-memory metadata store. ### Operation and configuration Default configuration. The window is between a resume's persisted `Resuming` mark and the `launch_sandbox` entry gate — realistically hit when a node drains (rolling redeploy) while an auto-resume (proxy lookup on a paused sandbox) or a client resume is in flight. ### Steps to reproduce Deterministic interleaving (unit test, full code below): 1. Sandbox is `Paused` (store state Paused, persisted record `lifecycle=Paused`, artifacts under `artifacts/<id>/`). 2. Task A starts `resume_sandbox(id)`; it passes the resume entry gate (shutdown not yet requested), CASes the store `Paused→Resuming`, and is about to / just did `mark_resuming` (a persisted, fsync'd write). 3. Shutdown begins (`is_shutting_down = true`) — before Task A reaches `launch_sandbox`. 4. Task A reaches `launch_sandbox`, whose FIRST statement is `ensure_accepting_lifecycle_operations()` → `Err(ShuttingDown)`. It returns **without any rollback**. 5. Observe: store state stays `Resuming`. Nothing ever advances it. The test parks the resume task inside a gated `mark_resuming` so step 3 lands inside the window deterministically, and asserts the outcome. ### Expected behavior The failed resume must roll back to `Paused` — exactly like every other failure path in `launch_sandbox` does via `rollback_failed_launch_metadata` (which performs a `Resuming→Paused` CAS plus `persister.rollback_resuming`). Either the entry-gate rejection for a Resume plan should run that rollback, or the `Resuming` mark/CAS should happen after the gate. ### Actual behavior - The `ShuttingDown` rejection at the `launch_sandbox` entry gate is the ONLY failure path that skips the rollback: build failure, protect failure, start failure, the three shutdown checks inside launch, add failure, wait-ready failure and the terminal-CAS failure all restore `Paused` (`service.rs:2632-2662`) — the entry gate at `service.rs:2400-2401` does not. - **Consequence A (live process)**: the sandbox is stranded in `Resuming`. resume, delete, pause and keep-alive all wait on `wait_for_transition(Resuming)` for 60s (`WAIT_TRANSITION_TIMEOUT`) and fail. The drain itself also stalls: shutdown cleanup waits up to 3 rounds × 60s on the stranded sandbox and reports "failed to preserve all sandboxes". - **Consequence B (after restart)**: the persistence loader treats a record with `lifecycle=Resuming` as invalid and calls `cleanup_invalid_record` — it deletes the record AND `remove_dir_all`s the whole `artifacts/<id>/` directory (`file_backed.rs:278-282`, "discarding paused sandbox record left in resuming state"). The paused memory snapshot and rootfs writable-layer copy are destroyed — even though this VM never began resuming (the launch was rejected before build started). ### Logs and diagnostics Timestamped tracing excerpt from the first run (`34cdc80`, Linux container): ```text 2026-10-12T10:27:52.915674Z INFO agentenv::orchestrator::persistence::file_backed: loading paused sandbox records store=/tmp/.tmpbXXK4X/sandboxes 2026-10-12T10:27:52.925673Z INFO resume_sandbox: agentenv::orchestrator::service: resuming sandbox sandbox_id=01a12fa7-8fc4-7503-a5fb-8658552fb80b timeout=UseExisting 2026-10-12T10:27:52.929140Z INFO resume_sandbox:launch_sandbox: agentenv::orchestrator::service: rejecting lifecycle operation because orchestrator is shutting down sandbox_id=01a12fa7-8fc4-7503-a5fb-8658552fb80b timeout=UseExisting 2026-10-12T10:27:53 test orchestrator::service::tests::resume_racing_shutdown_strands_sandbox_in_resuming ... ok 2026-10-12T10:27:53 test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 959 filtered out; finished in 0.12s ``` Re-run on `4968423` (same behavior, same assertions): ```text test orchestrator::service::tests::resume_racing_shutdown_strands_sandbox_in_resuming ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 966 filtered out; finished in 0.13s ``` The test asserts `store.state == Resuming` after the ShuttingDown error — i.e. the bug behavior. With a correct rollback it would assert Paused. ### Reproduction test Append the following to `src/orchestrator/tests.rs` (its helpers — `setup`, `paused_resume_metadata`, `make_orchestrator_without_background_with_factory_and_persister`, `MockBackendFactory`, `InMemoryMetadataStore` — are that module's existing test infrastructure) and run: ```bash cargo test -p agentenv --lib resume_racing_shutdown_strands ``` ```rust /// Test double: gates the first `mark_resuming` call on an external signal /// so a test can flip `is_shutting_down` inside the resume window between /// the persisted `Resuming` mark and the `launch_sandbox` entry gate. struct GatedMarkResumingPersister { inner: RecordingPersister, gate: tokio::sync::Mutex<Option<tokio::sync::oneshot::Receiver<()>>>, } impl GatedMarkResumingPersister { fn gated() -> (Self, tokio::sync::oneshot::Sender<()>) { let (tx, rx) = tokio::sync::oneshot::channel(); ( Self { inner: RecordingPersister::default(), gate: tokio::sync::Mutex::new(Some(rx)), }, tx, ) } } #[async_trait::async_trait] impl SandboxPersister for GatedMarkResumingPersister { async fn load_all<F>( &self, factory: &F, ) -> crate::orchestrator::persistence::PersistenceResult<Vec<SandboxMetadata>> where F: crate::sandbox::SandboxBackendFactory, { self.inner.load_all(factory).await } async fn allocate_artifact_root( &self, sandbox_id: &SandboxId, ) -> crate::orchestrator::persistence::PersistenceResult<Option<std::path::PathBuf>> { self.inner.allocate_artifact_root(sandbox_id).await } async fn persist_paused( &self, metadata: &SandboxMetadata, artifact_root: Option<&std::path::Path>, paused_state: &dyn crate::sandbox::PausedSandboxState, ) -> crate::orchestrator::persistence::PersistenceResult<()> { self.inner .persist_paused(metadata, artifact_root, paused_state) .await } async fn mark_resuming( &self, sandbox_id: &SandboxId, ) -> crate::orchestrator::persistence::PersistenceResult<()> { if let Some(rx) = self.gate.lock().await.take() { let _ = rx.await; // park here until the test flips the flag } self.inner.mark_resuming(sandbox_id).await } async fn rollback_resuming( &self, sandbox_id: &SandboxId, ) -> crate::orchestrator::persistence::PersistenceResult<()> { self.inner.rollback_resuming(sandbox_id).await } async fn delete_record( &self, sandbox_id: &SandboxId, ) -> crate::orchestrator::persistence::PersistenceResult<()> { self.inner.delete_record(sandbox_id).await } async fn delete_record_and_artifacts( &self, sandbox_id: &SandboxId, ) -> crate::orchestrator::persistence::PersistenceResult<()> { self.inner.delete_record_and_artifacts(sandbox_id).await } } /// Regression probe for issue "resume racing shutdown strands sandbox in /// Resuming": the shutdown flag flips while the resume task is parked /// between its `Resuming` CAS + persisted mark and the `launch_sandbox` /// entry gate. The entry gate rejects with `ShuttingDown` and — because /// that path skips `rollback_failed_launch_metadata` — the sandbox is /// stranded in `Resuming` (pause/delete/keep-alive time out against it, and /// the next startup discards the paused artifacts entirely). #[tokio::test] async fn resume_racing_shutdown_strands_sandbox_in_resuming() -> Result<()> { use std::time::Duration; setup(); let sandbox_id = SandboxId::new(); let (persister, release_mark) = GatedMarkResumingPersister::gated(); let orchestrator = make_orchestrator_without_background_with_factory_and_persister( InMemoryMetadataStore::new(), MockBackendFactory::new(), persister, ); orchestrator .store .add(paused_resume_metadata(sandbox_id)) .await?; let task = tokio::spawn({ let orchestrator = Arc::clone(&orchestrator); async move { orchestrator.resume_sandbox(sandbox_id, NewTimeout::UseExisting).await } }); // Wait until the resume CAS flips the store state to `Resuming`; the // spawned task is now parked inside the gated `mark_resuming`. let deadline = std::time::Instant::now() + Duration::from_secs(10); loop { let state = orchestrator .store .get(&sandbox_id) .await? .expect("paused metadata should exist") .state; if state == SandboxState::Resuming { break; } assert!( std::time::Instant::now() < deadline, "resume never reached the Resuming state" ); tokio::time::sleep(Duration::from_millis(5)).await; } orchestrator.is_shutting_down.store(true, Ordering::Release); release_mark.send(()).expect("gate receiver alive"); let err = task .await .expect("resume task should not panic") .expect_err("resume should fail once shutdown has begun"); assert!( matches!(err, OrchestratorError::ShuttingDown), "expected ShuttingDown, got: {err:?}" ); // BUG: nothing rolls the CAS back — the sandbox is stranded. let stranded = orchestrator .store .get(&sandbox_id) .await? .expect("stranded metadata should exist"); assert_eq!( stranded.state, SandboxState::Resuming, "resume racing shutdown must strand the sandbox in Resuming (expected rollback to Paused)" ); Ok(()) } ``` ### Reproduction frequency Deterministic in the unit test (gated persister double). In production it requires the shutdown flag to land inside the CAS/mark→launch window (the window contains a fsync'd RocksDB write — milliseconds; rolling redeploys with traffic make this plausible). ### Regression information Not a regression we can trace — the resume rollback machinery exists for every other failure stage; only the entry-gate path lacks it. Existing tests cover shutdown rejection for CREATE plans only (`tests.rs:4761` asserts the store is empty after a rejected create); no test covered the resume plan until now. ### Additional context Fix sketch (happy to open a PR): - in `launch_sandbox`, when `ensure_accepting_lifecycle_operations()` fails for a Resume plan, run the same `Resuming→Paused` CAS + `persister.rollback_resuming` used by `rollback_failed_launch_metadata`; or - move the gate check into `resume_sandbox_inner` BEFORE the CAS (there is already an entry gate at the top of `resume_sandbox_inner` — the race window is between that gate and the launch gate), keeping rollback for the launch path. The secondary unrolled early-exit in the same function (missing `paused_state`, `service.rs:1741-1744`, after CAS+mark) should get the same treatment. ### Pre-submission checklist - [x] I searched existing open and closed issues **and GitHub Discussions** for duplicates — 0 related results. - [x] I provided a minimal reproduction (deterministic unit test, reproduced in full above). - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi commented 2026-10-29 14:19:32 +00:00

Author

Owner

Copy link

@mvanhorn commented on GitHub (Oct 20, 2026):

Resume commits the store and the persisted record from paused to resuming before launch checks whether the process is still accepting work. The shutdown rejection at the start of launch returns immediately and is the only failure there that does not restore the paused record.

Paused runtime state is now required before the paused-to-resuming update, so a missing snapshot returns an internal error and does not mark the record resuming.

https://github.com/kvcache-ai/AgentENV/pull/400

<!-- gh-comment-id:6294283543 --> @mvanhorn commented on GitHub (Oct 20, 2026): Resume commits the store and the persisted record from paused to resuming before launch checks whether the process is still accepting work. The shutdown rejection at the start of launch returns immediately and is the only failure there that does not restore the paused record. Paused runtime state is now required before the paused-to-resuming update, so a missing snapshot returns an internal error and does not mark the record resuming. https://github.com/kvcache-ai/AgentENV/pull/400

bsakaguchi referenced this issue 2026-10-29 14:20:43 +00:00

[PR #104] [MERGED] fix(snapshot): return error instead of panicking in write_commit_marker #179

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#96

No description provided.