[GH-ISSUE #120] Advance memory snapshot epochs for linear incremental OverlayBD chains #42

New issue

Open

opened 2026-10-29 14:18:43 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:18:43 +00:00

Owner

Copy link

Originally created by @xeonliu on GitHub (Aug 4, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/120

Affected component

Snapshot / template builder

Problem statement

AgentENV can capture a running sandbox without terminating it. The Firecracker
backend pauses the VM, creates snapshot artifacts, and then resumes the same
Firecracker process in place. However, the sandbox's launch configuration
continues to refer to the memory image from which that process originally
started.

This is compatible with the current mincore behavior because each capture
can include a broad resident-page set and remain a sibling over the original
launch parent. It is not sufficient for true per-checkpoint dirty-page
snapshots.

For example, suppose a VM starts from S0:

S0
  -> write page set A
  -> capture S1
  -> resume the same process
  -> write page set B
  -> capture S2

If the dirty epoch is reset after S1, the newest layer for S2 naturally
contains B, not A. If AgentENV still builds S2 over S0, the result is
incomplete:

incorrect: S2 = S0 + delta(B)
required:  S2 = S0 + delta(A) + delta(B)

Simply enabling dirty tracking and clearing its bitmap after every capture can
therefore create snapshots that are missing memory changes. AgentENV needs an
explicit live memory parent, a transactional dirty epoch, and artifact
lifetime rules before it can provide deltas relative to the latest successful
capture.

Companion issue: #119

Use case

Agent and RL workloads frequently capture or fork the same running sandbox at
multiple decision points. A long-running VM may modify only a small page set
between checkpoints, while each checkpoint becomes a base for later rollback,
fork, template publication, or migration.

These workloads need a linear memory checkpoint chain in which each new
OverlayBD layer is relative to the immediately preceding successful local
capture. Without that chain, dirty tracking must either retain an ever-growing
write set relative to the original launch parent or risk producing an
incomplete snapshot.

Current behavior and workarounds

During capture, snapshot_to_dir chooses inherited memory layers from
LaunchMode::Resume and appends the new overlaybd.commit. A non-terminal
capture_snapshot then resumes the same process, but it does not advance the
launch mode or another explicit current-memory-parent field to the newly
captured mem_image.json.

Current options are insufficient:

• Keep dirty pages accumulated relative to the original launch parent. This is
  correct but later captures repeat every page written during the VM's
  lifetime.
• Use mincore so captures contain a broad resident set. This avoids an epoch
  dependency but is not a true write-based incremental mechanism.
• Stop the VM and create a new Firecracker process from every captured
  snapshot. This establishes a new parent but adds avoidable lifecycle and
  restore latency.
• Compact layers after they accumulate. This reduces chain depth but does not
  define which parent a new delta is relative to.

The direct dirty-range path also needs a clear transaction boundary. Dirty
state must not be discarded until AgentENV has sealed the OverlayBD layer,
atomically written its memory image config, and retained every referenced
parent artifact. Firecracker issue #4545 and PR #4580 document the same class
of failure for native Diff snapshots:

• https://github.com/firecracker-microvm/firecracker/issues/4545
• https://github.com/firecracker-microvm/firecracker/pull/4580

Desired behavior

• A successful non-terminal capture advances the running sandbox's current
  memory parent to the captured memory image.
• The next capture emits a layer relative to that current parent rather than
  the original launch snapshot.
• The dirty epoch advances only after a locally complete capture is committed:
  the new layer is sealed, mem_image.json is durable, and inherited artifacts
  remain owned or pinned.
• If capture fails before that commit point, the previous parent and dirty
  epoch remain active so retry cannot omit pages.
• If capture succeeds locally but later repository publication fails, the
  running sandbox still has a valid local parent chain and can be captured or
  paused again.
• Pause/resume, repeated capture, fork, shutdown persistence, and template
  publication all preserve the same layer ordering and ownership rules.
• Existing snapshots and the OverlayBD on-disk format remain compatible.

Proposed approach

Introduce explicit current memory lineage in FirecrackerSandbox, separate
from immutable launch provenance. Conceptually:

launch_memory_parent: where this Firecracker process originally started
current_memory_parent: latest successfully committed local capture
dirty_epoch: changes since current_memory_parent

On capture:

1. Pause Firecracker and freeze/read the current dirty epoch.
2. Write the new dirty ranges through the existing
   process_vm_readv -> compact_to -> overlaybd.commit path.
3. Build a candidate mem_image.json from current_memory_parent + new layer.
4. Seal/sync the layer, atomically persist the image config, and retain or pin
   all referenced local artifacts.
5. Commit/acknowledge the Firecracker dirty epoch.
6. Atomically replace current_memory_parent with the candidate config.
7. Resume the VM.

If any step before the local commit point fails, abort the candidate, preserve
the previous parent, and keep or merge the dirty set for retry.

The custom /vm/dirty-memory-ranges API may need begin/read and commit/abort
semantics. If extending the direct API is not desirable, an initial
implementation could use Firecracker's native transactional Diff snapshot and
retain the intermediate mem.bin until OverlayBD conversion commits. A
conversion failure must not delete the only copy of a successfully dumped
epoch and then resume with cleared dirty state.

I would like maintainer feedback on:

1. Should non-terminal captures form a linear chain, or should the API promise
   independent sibling snapshots over the launch parent?
2. Is the local sealed layer plus atomically written image config the correct
   dirty-epoch commit point, or should repository publication participate?
3. Should current parent state live directly in FirecrackerSandbox, or be
   represented by a separate lineage/lease object shared with snapshot
   artifact ownership?
4. Is a transactional extension to /vm/dirty-memory-ranges acceptable, or is
   the native mem.bin transaction preferred for the first implementation?

Compatibility and operational impact

Introduce explicit current memory lineage in FirecrackerSandbox, separate
from immutable launch provenance. Conceptually:

launch_memory_parent: where this Firecracker process originally started
current_memory_parent: latest successfully committed local capture
dirty_epoch: changes since current_memory_parent

On capture:

1. Pause Firecracker and freeze/read the current dirty epoch.
2. Write the new dirty ranges through the existing
   process_vm_readv -> compact_to -> overlaybd.commit path.
3. Build a candidate mem_image.json from current_memory_parent + new layer.
4. Seal/sync the layer, atomically persist the image config, and retain or pin
   all referenced local artifacts.
5. Commit/acknowledge the Firecracker dirty epoch.
6. Atomically replace current_memory_parent with the candidate config.
7. Resume the VM.

If any step before the local commit point fails, abort the candidate, preserve
the previous parent, and keep or merge the dirty set for retry.

The custom /vm/dirty-memory-ranges API may need begin/read and commit/abort
semantics. If extending the direct API is not desirable, an initial
implementation could use Firecracker's native transactional Diff snapshot and
retain the intermediate mem.bin until OverlayBD conversion commits. A
conversion failure must not delete the only copy of a successfully dumped
epoch and then resume with cleared dirty state.

I would like maintainer feedback on:

1. Should non-terminal captures form a linear chain, or should the API promise
   independent sibling snapshots over the launch parent?
2. Is the local sealed layer plus atomically written image config the correct
   dirty-epoch commit point, or should repository publication participate?
3. Should current parent state live directly in FirecrackerSandbox, or be
   represented by a separate lineage/lease object shared with snapshot
   artifact ownership?
4. Is a transactional extension to /vm/dirty-memory-ranges acceptable, or is
   the native mem.bin transaction preferred for the first implementation?

Compatibility and operational impact

API/config changes:

• No AgentENV public HTTP API change is expected.
• The custom Firecracker dirty-range API may need transaction identifiers and
  commit/abort operations.
• No separate user-facing config is required beyond the tracking mode proposed
  by the companion issue.

Snapshot or storage format changes:

• None expected. mem_image.json and committed memory_layers remain ordered
  bottom-up using existing immutable OverlayBD layers.
• More precise deltas may increase chain length, so existing compaction limits
  remain important.

New host/runtime requirements:

• This feature depends on a reliable dirty-page source, expected to be the
  opt-in mode from the companion issue.
• Artifact ownership may retain additional local layers for the lifetime of a
  running sandbox; memory/disk impact should be measured and bounded.

Upgrade and rollback considerations:

• Existing snapshots remain readable.
• Persisted paused sandboxes must retain enough current-parent information to
  resume across a server restart.
• Mixed-version rollout must not clear a dirty epoch using code that cannot
  reconstruct the corresponding parent chain.
• Rollback can retain launch-parent cumulative semantics until linear epochs
  are enabled; it must not silently reinterpret an already advanced epoch.

Alternatives considered

• Cumulative sibling snapshots: never advance the parent or clear the
  write set. Correct and simpler, but every later capture repeats all writes
  since launch.
• Restart Firecracker after every capture: naturally establishes the new
  backing parent but adds restore latency and disrupts the non-terminal capture
  workflow.
• Keep using mincore: broad sparse captures can remain siblings, but
  read-only resident pages continue to inflate layers.
• Publish before advancing the epoch: gives a repository-backed parent but
  holds the VM paused across potentially slow remote I/O and couples local
  correctness to repository availability.
• Combine this with dirty-page selection in one PR: possible, but makes it
  harder to benchmark page tracking independently and mixes Firecracker page
  selection with AgentENV lineage/artifact ownership.

Non-goals

• Choosing between mincore and KVM dirty tracking; that belongs to the
  companion issue.
• Changing the OverlayBD on-disk format.
• General snapshot DAG branching or cross-sandbox layer garbage collection.
• Changing rootfs or attached-drive snapshot behavior.
• Replacing compression, background download, or repository backends.

Contribution

I can implement it

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @xeonliu on GitHub (Aug 4, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/120 ### Affected component Snapshot / template builder ### Problem statement AgentENV can capture a running sandbox without terminating it. The Firecracker backend pauses the VM, creates snapshot artifacts, and then resumes the same Firecracker process in place. However, the sandbox's launch configuration continues to refer to the memory image from which that process originally started. This is compatible with the current `mincore` behavior because each capture can include a broad resident-page set and remain a sibling over the original launch parent. It is not sufficient for true per-checkpoint dirty-page snapshots. For example, suppose a VM starts from `S0`: ```text S0 -> write page set A -> capture S1 -> resume the same process -> write page set B -> capture S2 ``` If the dirty epoch is reset after `S1`, the newest layer for `S2` naturally contains B, not A. If AgentENV still builds `S2` over `S0`, the result is incomplete: ```text incorrect: S2 = S0 + delta(B) required: S2 = S0 + delta(A) + delta(B) ``` Simply enabling dirty tracking and clearing its bitmap after every capture can therefore create snapshots that are missing memory changes. AgentENV needs an explicit live memory parent, a transactional dirty epoch, and artifact lifetime rules before it can provide deltas relative to the latest successful capture. Companion issue: #119 ### Use case Agent and RL workloads frequently capture or fork the same running sandbox at multiple decision points. A long-running VM may modify only a small page set between checkpoints, while each checkpoint becomes a base for later rollback, fork, template publication, or migration. These workloads need a linear memory checkpoint chain in which each new OverlayBD layer is relative to the immediately preceding successful local capture. Without that chain, dirty tracking must either retain an ever-growing write set relative to the original launch parent or risk producing an incomplete snapshot. ### Current behavior and workarounds During capture, `snapshot_to_dir` chooses inherited memory layers from `LaunchMode::Resume` and appends the new `overlaybd.commit`. A non-terminal `capture_snapshot` then resumes the same process, but it does not advance the launch mode or another explicit current-memory-parent field to the newly captured `mem_image.json`. Current options are insufficient: - Keep dirty pages accumulated relative to the original launch parent. This is correct but later captures repeat every page written during the VM's lifetime. - Use `mincore` so captures contain a broad resident set. This avoids an epoch dependency but is not a true write-based incremental mechanism. - Stop the VM and create a new Firecracker process from every captured snapshot. This establishes a new parent but adds avoidable lifecycle and restore latency. - Compact layers after they accumulate. This reduces chain depth but does not define which parent a new delta is relative to. The direct dirty-range path also needs a clear transaction boundary. Dirty state must not be discarded until AgentENV has sealed the OverlayBD layer, atomically written its memory image config, and retained every referenced parent artifact. Firecracker issue #4545 and PR #4580 document the same class of failure for native Diff snapshots: - https://github.com/firecracker-microvm/firecracker/issues/4545 - https://github.com/firecracker-microvm/firecracker/pull/4580 ### Desired behavior - A successful non-terminal capture advances the running sandbox's current memory parent to the captured memory image. - The next capture emits a layer relative to that current parent rather than the original launch snapshot. - The dirty epoch advances only after a locally complete capture is committed: the new layer is sealed, `mem_image.json` is durable, and inherited artifacts remain owned or pinned. - If capture fails before that commit point, the previous parent and dirty epoch remain active so retry cannot omit pages. - If capture succeeds locally but later repository publication fails, the running sandbox still has a valid local parent chain and can be captured or paused again. - Pause/resume, repeated capture, fork, shutdown persistence, and template publication all preserve the same layer ordering and ownership rules. - Existing snapshots and the OverlayBD on-disk format remain compatible. ### Proposed approach Introduce explicit current memory lineage in `FirecrackerSandbox`, separate from immutable launch provenance. Conceptually: ```text launch_memory_parent: where this Firecracker process originally started current_memory_parent: latest successfully committed local capture dirty_epoch: changes since current_memory_parent ``` On capture: 1. Pause Firecracker and freeze/read the current dirty epoch. 2. Write the new dirty ranges through the existing `process_vm_readv -> compact_to -> overlaybd.commit` path. 3. Build a candidate `mem_image.json` from `current_memory_parent + new layer`. 4. Seal/sync the layer, atomically persist the image config, and retain or pin all referenced local artifacts. 5. Commit/acknowledge the Firecracker dirty epoch. 6. Atomically replace `current_memory_parent` with the candidate config. 7. Resume the VM. If any step before the local commit point fails, abort the candidate, preserve the previous parent, and keep or merge the dirty set for retry. The custom `/vm/dirty-memory-ranges` API may need begin/read and commit/abort semantics. If extending the direct API is not desirable, an initial implementation could use Firecracker's native transactional Diff snapshot and retain the intermediate `mem.bin` until OverlayBD conversion commits. A conversion failure must not delete the only copy of a successfully dumped epoch and then resume with cleared dirty state. I would like maintainer feedback on: 1. Should non-terminal captures form a linear chain, or should the API promise independent sibling snapshots over the launch parent? 2. Is the local sealed layer plus atomically written image config the correct dirty-epoch commit point, or should repository publication participate? 3. Should current parent state live directly in `FirecrackerSandbox`, or be represented by a separate lineage/lease object shared with snapshot artifact ownership? 4. Is a transactional extension to `/vm/dirty-memory-ranges` acceptable, or is the native `mem.bin` transaction preferred for the first implementation? ### Compatibility and operational impact Introduce explicit current memory lineage in `FirecrackerSandbox`, separate from immutable launch provenance. Conceptually: ```text launch_memory_parent: where this Firecracker process originally started current_memory_parent: latest successfully committed local capture dirty_epoch: changes since current_memory_parent ``` On capture: 1. Pause Firecracker and freeze/read the current dirty epoch. 2. Write the new dirty ranges through the existing `process_vm_readv -> compact_to -> overlaybd.commit` path. 3. Build a candidate `mem_image.json` from `current_memory_parent + new layer`. 4. Seal/sync the layer, atomically persist the image config, and retain or pin all referenced local artifacts. 5. Commit/acknowledge the Firecracker dirty epoch. 6. Atomically replace `current_memory_parent` with the candidate config. 7. Resume the VM. If any step before the local commit point fails, abort the candidate, preserve the previous parent, and keep or merge the dirty set for retry. The custom `/vm/dirty-memory-ranges` API may need begin/read and commit/abort semantics. If extending the direct API is not desirable, an initial implementation could use Firecracker's native transactional Diff snapshot and retain the intermediate `mem.bin` until OverlayBD conversion commits. A conversion failure must not delete the only copy of a successfully dumped epoch and then resume with cleared dirty state. I would like maintainer feedback on: 1. Should non-terminal captures form a linear chain, or should the API promise independent sibling snapshots over the launch parent? 2. Is the local sealed layer plus atomically written image config the correct dirty-epoch commit point, or should repository publication participate? 3. Should current parent state live directly in `FirecrackerSandbox`, or be represented by a separate lineage/lease object shared with snapshot artifact ownership? 4. Is a transactional extension to `/vm/dirty-memory-ranges` acceptable, or is the native `mem.bin` transaction preferred for the first implementation? # Compatibility and operational impact API/config changes: - No AgentENV public HTTP API change is expected. - The custom Firecracker dirty-range API may need transaction identifiers and commit/abort operations. - No separate user-facing config is required beyond the tracking mode proposed by the companion issue. Snapshot or storage format changes: - None expected. `mem_image.json` and committed `memory_layers` remain ordered bottom-up using existing immutable OverlayBD layers. - More precise deltas may increase chain length, so existing compaction limits remain important. New host/runtime requirements: - This feature depends on a reliable dirty-page source, expected to be the opt-in mode from the companion issue. - Artifact ownership may retain additional local layers for the lifetime of a running sandbox; memory/disk impact should be measured and bounded. Upgrade and rollback considerations: - Existing snapshots remain readable. - Persisted paused sandboxes must retain enough current-parent information to resume across a server restart. - Mixed-version rollout must not clear a dirty epoch using code that cannot reconstruct the corresponding parent chain. - Rollback can retain launch-parent cumulative semantics until linear epochs are enabled; it must not silently reinterpret an already advanced epoch. ### Alternatives considered - **Cumulative sibling snapshots:** never advance the parent or clear the write set. Correct and simpler, but every later capture repeats all writes since launch. - **Restart Firecracker after every capture:** naturally establishes the new backing parent but adds restore latency and disrupts the non-terminal capture workflow. - **Keep using `mincore`:** broad sparse captures can remain siblings, but read-only resident pages continue to inflate layers. - **Publish before advancing the epoch:** gives a repository-backed parent but holds the VM paused across potentially slow remote I/O and couples local correctness to repository availability. - **Combine this with dirty-page selection in one PR:** possible, but makes it harder to benchmark page tracking independently and mixes Firecracker page selection with AgentENV lineage/artifact ownership. ### Non-goals - Choosing between `mincore` and KVM dirty tracking; that belongs to the companion issue. - Changing the OverlayBD on-disk format. - General snapshot DAG branching or cross-sandbox layer garbage collection. - Changing rootfs or attached-drive snapshot behavior. - Replacing compression, background download, or repository backends. ### Contribution I can implement it ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:18:43 +00:00

bsakaguchi commented 2026-10-29 14:18:43 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Aug 4, 2026):

Thanks for raising this. The current behavior is intentional: non-terminal captures remain independent snapshots over the launch-time memory parent rather than advancing the running VM’s memory parent after every capture.

The main reason is storage reclamation. With many capture/resume cycles, workloads often write repeatedly to the same hot memory regions. A linear chain would preserve a new version of those pages in every immutable OverlayBD layer:

S0 -> hot pages v1 -> hot pages v2 -> hot pages v3

Although only the newest version is visible in the latest snapshot, the older versions cannot be reclaimed while descendants still reference those layers.

Linear layering is acceptable for disk snapshots because disk data generally changes much less than memory between captures. Memory has much higher write volume and churn, so repeated snapshots can quickly accumulate many obsolete copies of the same hot pages.

With the current sibling-snapshot model, each capture is independently restorable from its launch parent. When an older snapshot is no longer needed, its complete delta can be deleted without rewriting or invalidating newer snapshots.

The tradeoff is that each capture may rewrite a larger cumulative resident or dirty set, resulting in longer pause time and additional snapshot I/O. We currently prefer simpler ownership and predictable space reclamation over shorter incremental pauses with an ever-growing dependent layer chain.

This is still a valuable direction, but it requires garbage collection that periodically compacts multiple memory layers into a single layer so obsolete page versions can be reclaimed.

<!-- gh-comment-id:5177045717 --> @yingdi-shan commented on GitHub (Aug 4, 2026): Thanks for raising this. The current behavior is intentional: non-terminal captures remain independent snapshots over the launch-time memory parent rather than advancing the running VM’s memory parent after every capture. The main reason is storage reclamation. With many capture/resume cycles, workloads often write repeatedly to the same hot memory regions. A linear chain would preserve a new version of those pages in every immutable OverlayBD layer: ```text S0 -> hot pages v1 -> hot pages v2 -> hot pages v3 ``` Although only the newest version is visible in the latest snapshot, the older versions cannot be reclaimed while descendants still reference those layers. Linear layering is acceptable for disk snapshots because disk data generally changes much less than memory between captures. Memory has much higher write volume and churn, so repeated snapshots can quickly accumulate many obsolete copies of the same hot pages. With the current sibling-snapshot model, each capture is independently restorable from its launch parent. When an older snapshot is no longer needed, its complete delta can be deleted without rewriting or invalidating newer snapshots. The tradeoff is that each capture may rewrite a larger cumulative resident or dirty set, resulting in longer pause time and additional snapshot I/O. We currently prefer simpler ownership and predictable space reclamation over shorter incremental pauses with an ever-growing dependent layer chain. This is still a valuable direction, but it requires garbage collection that periodically compacts multiple memory layers into a single layer so obsolete page versions can be reclaimed.

bsakaguchi referenced this issue 2026-10-29 14:20:05 +00:00

[PR #45] [CLOSED] fix(sandbox): roll back resources after start failures #137

bsakaguchi referenced this issue 2026-10-29 14:20:45 +00:00

[PR #108] [MERGED] Fix sandbox lifecycle bugs and improve network policy handling #183

bsakaguchi referenced this issue 2026-10-29 14:22:54 +00:00

[PR #380] [MERGED] refactor(sandbox): give Firecracker sandbox resources a single owner and split start paths #390

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#42

No description provided.