kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #326] POSIX startup pack is not persisted because the template workspace is deleted before recording completes #86

New issue

Closed

opened 2026-10-29 14:19:24 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:24 +00:00

Owner

Originally created by @MisaKokoro on GitHub (Sep 29, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/326

Affected component

Snapshot / template builder

AgentENV version

Version or commit: 875faeecfb
Local modifications: none

Host environment

OS: openEuler 24.03 (LTS-SP3)
Kernel: 6.6.0-132.0.0.111.oe2403sp3.aarch64
Architecture: aarch64
Bare metal, VM, or cloud: bare metal
ublk support: available; ublk-backed snapshot builds are working
Storage/filesystem: ext4

Operation and configuration

Configure the POSIX snapshot repository and enable startup-pack recording:

  [snapshot]
  repository_backend = "posix_fs"

  [snapshot.memory_startup_pack]
  enabled = true
  consume_enabled = true

  [backend.posix_fs]
  snapshot_store = "<redacted-snapshot-store>"

  Then rebuild a template:

  aenv pull ubuntu:22.04 --name ubuntu-startup-pack-repro

Steps to reproduce

Build and run AgentENV from upstream commit
875faeecfb.
Configure snapshot.repository_backend = "posix_fs" and enable
[snapshot.memory_startup_pack] as shown above.

Run:

aenv pull ubuntu:22.04 --name ubuntu-startup-pack-repro

Wait for the template build to finish successfully and then wait longer
than record_budget_secs, so the detached recording continuation has had
time to finish.

Inspect the committed snapshot directory:

<snapshot_store>/snapshots/<snapshot_id>/

Inspect the catalog record:

<snapshot_store>/catalog/records/<snapshot_id>.json

Observe that memory-startup.pack is absent and
committed.memory_startup is not populated.
Start a sandbox from the template and observe that POSIX memory-ublk
startup prefetch is not performed.

Expected behavior

The detached startup-pack recorder should retain the template build workspace
until recording and repository attachment have completed.

After a successful recording:

<snapshot_store>/snapshots/<snapshot_id>/memory-startup.pack should exist;
the catalog record should contain committed.memory_startup; and
subsequent sandbox starts should be able to consume the POSIX startup pack.

Snapshot publication should remain asynchronous and should not wait for the
recording to finish.

Actual behavior

The template snapshot is published successfully, but the detached recorder
loses its temporary output directory before it writes
memory-startup.trace.tmp.

As a result:

memory-startup.pack is not created;
committed.memory_startup is not attached to the catalog record; and
subsequent sandbox starts do not perform POSIX memory-ublk prefetch.

The server reports:

  startup pack: daemon recording failed
  reason=write trace tmp file /tmp/snapshot-.../memory-startup.trace.tmp:
  No such file or directory (os error 2)

Logs and diagnostics

startup pack: daemon recording failed
  reason=write trace tmp file /tmp/snapshot-.../memory-startup.trace.tmp:
  No such file or directory (os error 2)

Reproduction frequency

Always

Regression information

Last known good:
First known bad:

Additional context

The failure is caused by a lifetime mismatch between the template build
workspace and the detached startup-pack recording.

TemplateBuilder::execute_and_publish() stores the build artifacts under
TemplateBuildContext.workspace, which owns a TempDir. It starts recording
with:

  StartupRecording {
      trace: tokio::spawn(record_startup_pack(...)),
      keep_alive: Box::new(()),
  }

The POSIX repository publishes the snapshot synchronously, spawns
finish_posix_startup_manifest() as a detached continuation, and returns the
committed record.

After execute_and_publish() returns, TemplateBuildContext is dropped and
its TempDir is removed. The recorder may still be trying to create:

  memory-startup.trace.tmp

The unit keep_alive does not retain the workspace, so the write fails with
ENOENT and the continuation exits without attaching a startup manifest.

A possible fix is to wrap the template workspace in Arc<TempDir> and use the
existing StartupRecording::spawn_with_capture_lease() mechanism. This gives
both the recorder task and repository continuation a workspace lease without
making publication wait for recording.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction or explained why one cannot be provided.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @MisaKokoro on GitHub (Sep 29, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/326 ### Affected component Snapshot / template builder ### AgentENV version - Version or commit: 875faeecfbc2c69eed709de06764003b86ac5c28 - Local modifications: none ### Host environment - OS: openEuler 24.03 (LTS-SP3) - Kernel: 6.6.0-132.0.0.111.oe2403sp3.aarch64 - Architecture: aarch64 - Bare metal, VM, or cloud: bare metal - ublk support: available; ublk-backed snapshot builds are working - Storage/filesystem: ext4 ### Operation and configuration ```shell Configure the POSIX snapshot repository and enable startup-pack recording: [snapshot] repository_backend = "posix_fs" [snapshot.memory_startup_pack] enabled = true consume_enabled = true [backend.posix_fs] snapshot_store = "<redacted-snapshot-store>" Then rebuild a template: aenv pull ubuntu:22.04 --name ubuntu-startup-pack-repro ``` ### Steps to reproduce 1. Build and run AgentENV from upstream commit 875faeecfbc2c69eed709de06764003b86ac5c28. 2. Configure `snapshot.repository_backend = "posix_fs"` and enable `[snapshot.memory_startup_pack]` as shown above. 3. Run: aenv pull ubuntu:22.04 --name ubuntu-startup-pack-repro 4. Wait for the template build to finish successfully and then wait longer than `record_budget_secs`, so the detached recording continuation has had time to finish. 5. Inspect the committed snapshot directory: <snapshot_store>/snapshots/<snapshot_id>/ 6. Inspect the catalog record: <snapshot_store>/catalog/records/<snapshot_id>.json 7. Observe that `memory-startup.pack` is absent and `committed.memory_startup` is not populated. 8. Start a sandbox from the template and observe that POSIX memory-ublk startup prefetch is not performed. ### Expected behavior The detached startup-pack recorder should retain the template build workspace until recording and repository attachment have completed. After a successful recording: - `<snapshot_store>/snapshots/<snapshot_id>/memory-startup.pack` should exist; - the catalog record should contain `committed.memory_startup`; and - subsequent sandbox starts should be able to consume the POSIX startup pack. Snapshot publication should remain asynchronous and should not wait for the recording to finish. ### Actual behavior The template snapshot is published successfully, but the detached recorder loses its temporary output directory before it writes `memory-startup.trace.tmp`. As a result: - `memory-startup.pack` is not created; - `committed.memory_startup` is not attached to the catalog record; and - subsequent sandbox starts do not perform POSIX memory-ublk prefetch. The server reports: ``` startup pack: daemon recording failed reason=write trace tmp file /tmp/snapshot-.../memory-startup.trace.tmp: No such file or directory (os error 2) ``` ### Logs and diagnostics ```text startup pack: daemon recording failed reason=write trace tmp file /tmp/snapshot-.../memory-startup.trace.tmp: No such file or directory (os error 2) ``` ### Reproduction frequency Always ### Regression information - Last known good: - First known bad: ### Additional context The failure is caused by a lifetime mismatch between the template build workspace and the detached startup-pack recording. `TemplateBuilder::execute_and_publish()` stores the build artifacts under `TemplateBuildContext.workspace`, which owns a `TempDir`. It starts recording with: StartupRecording { trace: tokio::spawn(record_startup_pack(...)), keep_alive: Box::new(()), } The POSIX repository publishes the snapshot synchronously, spawns `finish_posix_startup_manifest()` as a detached continuation, and returns the committed record. After `execute_and_publish()` returns, `TemplateBuildContext` is dropped and its `TempDir` is removed. The recorder may still be trying to create: memory-startup.trace.tmp The unit `keep_alive` does not retain the workspace, so the write fails with ENOENT and the continuation exits without attaching a startup manifest. A possible fix is to wrap the template workspace in `Arc<TempDir>` and use the existing `StartupRecording::spawn_with_capture_lease()` mechanism. This gives both the recorder task and repository continuation a workspace lease without making publication wait for recording. ### Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction or explained why one cannot be provided. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:19:24 +00:00