[GH-ISSUE #439] FC incremental memory export: explicit bitmap reset to simplify pause/snapshot and RPCs #107

New issue

Open

opened 2026-10-29 14:19:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:40 +00:00

Owner

Originally created by @guozy18 on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/439

This issue proposes changing memory capture from “repeatedly exporting changes since this run started” to “saving only new changes since the last successful save.” On that basis, temporal pause and snapshot can share the same memory-save flow, with the complete storage commit consolidated in the daemon.

Discussion: if snapshots also form a layered chain under this proposal, one concern is that after taking many local snapshots, we may only need the final snapshot. With the current approach, only the latest delta needs to be retained over the shared base; with this proposal, the historical snapshot chain cannot simply be deleted. This is a tradeoff. It would be useful to measure the actual difference between sum(snapshot_diff) and final_snapshot for different workloads.

1. FC: advance the export baseline from the start of this run to the last successful save

Currently, the dirty set is measured from the latest start or restore of the FC process. Taking multiple snapshots during that run does not clear the accumulated dirty set, so pages already saved are exported again by later snapshots.

For example, within the same run, first modify page A and take a snapshot, then modify only page B:

Operation	Current export	Export after the change
First snapshot	Page A	Page A
Second snapshot	Pages A and B	Page B

AgentENV's FC patch therefore needs an incremental export interface with explicit bitmap reset: confirm the reset after the local memory save succeeds, so the next export contains only changes made after that confirmation. Continue using the existing direct-capture and OverlayBD storage paths.

The benefit is to avoid repeatedly reading, copying, encoding, and writing previously saved dirty pages. For a long-running sandbox taking successive snapshots, export work would follow the changes made in the latest interval rather than the growing set of pages modified at any point during the run.

2. Memory saving: maintain one continuously advancing state; pause and snapshot only decide whether to freeze a version

On the current v0.3.0-rc.1 branch (5343438), temporal pause already keeps updating a resumable memory state, while snapshot follows different save rules:

Operation	Current save behavior	With unified state advancement
temporal pause	Write the dirty pages accumulated during this run into a private mutable upper and retain resumable state; it can be reused repeatedly when no snapshot intervenes.	Write new changes since the last save into the private mutable upper and keep it mutable, so the original sandbox can resume and continue updating it.
snapshot	For the first snapshot after resuming from pause, write dirty pages into the mutable upper and then seal it. Later snapshots independently export cumulative deltas without attaching each new delta back to the source base.	Write new changes into the current mutable upper, freeze it as a sealed upper, advance the base, and create a new mutable upper; the result is an independently restorable, reusable version.

The mismatch today is that pause updates a resumable state, while successive snapshots mostly produce independent outputs over a fixed base. The two paths advance state differently. Memory capture therefore includes logic that selects a save method based on operation history, while storage updates are spread between the server's capture flow and the daemon's compaction operations.

After the change, both paths first incorporate the latest changes into the saved state and then decide whether to freeze a version: pause retains a private continuation state; snapshot fixes an immutable version. The saved result becomes the starting point for the next interval, so successive snapshots and interleaved pause/resume follow the same rule.

This allows memory saving to become a complete storage commit operation: the server handles capture and policy, while the daemon performs sealing, compaction, and base updates, reducing state-dependent branches in capture and storage orchestration across modules.

3. Architecture and RPCs: the server organizes capture; the daemon owns the complete storage commit

The diagrams below use the current RC architecture as the baseline and keep the same overall layout for comparison. Yellow denotes compaction decisions, green denotes daemon / OverlayBD execution, and dashed lines denote query dependencies.

Before: disk commits already run in the daemon; memory storage operations are still orchestrated by the server

The current RC implementation already shares compaction policy and delegates immutable-layer merging to the daemon. However, memory sealing, base updates, and snapshot-layer assembly remain in the server; for memory compaction, the daemon mainly receives requests to “merge these files.”

flowchart TB
    Sandbox["FirecrackerSandbox"]
    Memory["Memory capture<br/>FC cumulative dirty → server capture"]
    MemoryStorage["server: seal memory / update config<br/>retain or replace snapshot lowers"]
    Disk["Disk capture"]
    Policy["Shared compaction policy"]
    Manager["UblkDeviceManager<br/>RPC forwarding"]
    Merge["daemon: merge selected immutable layers"]
    Device["daemon: compact / seal upper<br/>restack / update runtime config"]
    Readers["ImageService<br/>local and remote readers / cache"]
    Sandbox --> Memory --> MemoryStorage
    Sandbox --> Disk
    MemoryStorage -. query merge range .-> Policy
    Disk -. query merge range .-> Policy
    MemoryStorage -->|when compaction is needed| Manager
    Disk --> Manager
    Manager -->|CompactLayers RPC| Merge
    Manager -->|SnapshotDevice RPC| Device
    Merge -. reads and caching .-> Readers
    Device -. reuse already-open readers .-> Readers
    classDef policy fill:#fff1cc,stroke:#ad7c14;
    classDef storage fill:#e5f4ed,stroke:#3e8064;
    class Policy policy;
    class Merge,Device,Readers storage;

After: both paths commit their current mutable upper, with storage operations performed by the daemon

After the server writes the latest memory changes into the mutable upper, both memory and disk enter a complete snapshot-commit flow.

Compaction policy stays in server / snapshot / repository: determine which layers may be merged, select the range, and pass the plan to the daemon through the unified RPC. The daemon executes compaction, sealing, base updates, and new-upper creation according to that plan, validating the actual upper state and layer count as needed. No daemon → repository dependency is added.

flowchart TB
    Sandbox["FirecrackerSandbox"]
    Memory["FC per-save dirty<br/>server writes the mutable memory upper"]
    Disk["Disk's already-written mutable upper"]
    Manager["UblkDeviceManager<br/>unified snapshot request"]
    Commit["daemon / OverlayBD<br/>compact / seal as a sealed upper<br/>advance base / create mutable upper"]
    Policy["server / snapshot / repository<br/>compaction eligibility and range"]
    Readers["ImageService<br/>local and remote readers / cache"]
    Output["server: assemble a complete snapshot<br/>VM state + memory snapshot + disk snapshot"]
    Sandbox --> Memory --> Manager
    Sandbox --> Disk --> Manager
    Memory -. query merge range .-> Policy
    Disk -. query merge range .-> Policy
    Manager -->|unified snapshot-commit RPC carrying the compaction plan| Commit
    Commit -. reads and caching .-> Readers
    Commit --> Output
    classDef policy fill:#fff1cc,stroke:#ad7c14;
    classDef storage fill:#e5f4ed,stroke:#3e8064;
    class Policy policy;
    class Commit,Readers storage;

The architectural benefit is a clear division: FC manages increments, the server manages capture and policy, and the daemon manages the complete storage commit. The server supplies the sources and compaction plan; the daemon performs sealing, compaction, base rotation, and configuration updates. The RPC consequently becomes “commit the current private writable state.”

The daemon returns read-only memory and disk storage results. The server combines them with VM state into a complete, restorable snapshot, and makes the snapshot independently retain the files it needs: the snapshot remains restorable after the source sandbox exits or its runtime directory is cleaned. Publication to the snapshot repository remains a subsequent existing flow and is omitted from the diagram.

Memory-save state belongs to each sandbox. Sharing a read-only restore device does not mean sharing mutable state.

Tradeoffs

Per-save increments reduce repeated work during export, but successive snapshots create more sealed uppers. Reuse the existing compaction capability to control layer count, and evaluate overall performance including format conversion, compaction, and restore costs. Do not assume that total pause time will necessarily decrease.

These are goals to be implemented: first establish the new incremental-export semantics, then unify the save flow, and finally consolidate storage responsibilities and RPCs.

Originally created by @guozy18 on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/439 This issue proposes changing memory capture from “repeatedly exporting changes since this run started” to “saving only new changes since the last successful save.” On that basis, temporal pause and snapshot can share the same memory-save flow, with the complete storage commit consolidated in the daemon. Discussion: if snapshots also form a layered chain under this proposal, one concern is that after taking many local snapshots, we may only need the final snapshot. With the current approach, only the latest delta needs to be retained over the shared base; with this proposal, the historical snapshot chain cannot simply be deleted. This is a tradeoff. It would be useful to measure the actual difference between `sum(snapshot_diff)` and `final_snapshot` for different workloads. ### 1. FC: advance the export baseline from the start of this run to the last successful save Currently, the dirty set is measured from the latest start or restore of the FC process. Taking multiple snapshots during that run does not clear the accumulated dirty set, so pages already saved are exported again by later snapshots. For example, within the same run, first modify page A and take a snapshot, then modify only page B: | Operation | Current export | Export after the change | | --- | --- | --- | | First snapshot | Page A | Page A | | Second snapshot | Pages A and B | Page B | AgentENV's FC patch therefore needs an **incremental export interface with explicit bitmap reset**: confirm the reset after the local memory save succeeds, so the next export contains only changes made after that confirmation. Continue using the existing direct-capture and OverlayBD storage paths. **The benefit is to avoid repeatedly reading, copying, encoding, and writing previously saved dirty pages.** For a long-running sandbox taking successive snapshots, export work would follow the changes made in the latest interval rather than the growing set of pages modified at any point during the run. ### 2. Memory saving: maintain one continuously advancing state; pause and snapshot only decide whether to freeze a version On the current [`v0.3.0-rc.1` branch (`5343438`)](https://github.com/kvcache-ai/AgentENV/tree/5343438d9766a0583c80a8f565e494a1fb4e6758), temporal pause already keeps updating a resumable memory state, while snapshot follows different save rules: | Operation | Current save behavior | With unified state advancement | | --- | --- | --- | | temporal pause | Write the dirty pages accumulated during this run into a private mutable upper and retain resumable state; it can be reused repeatedly when no snapshot intervenes. | Write new changes since the last save into the private mutable upper and keep it mutable, so the original sandbox can resume and continue updating it. | | snapshot | For the first snapshot after resuming from pause, write dirty pages into the mutable upper and then seal it. Later snapshots independently export cumulative deltas without attaching each new delta back to the source base. | Write new changes into the current mutable upper, freeze it as a sealed upper, advance the base, and create a new mutable upper; the result is an independently restorable, reusable version. | **The mismatch today is that pause updates a resumable state, while successive snapshots mostly produce independent outputs over a fixed base. The two paths advance state differently.** Memory capture therefore includes logic that selects a save method based on operation history, while storage updates are spread between the server's capture flow and the daemon's compaction operations. After the change, both paths first incorporate the latest changes into the saved state and then decide whether to freeze a version: **pause retains a private continuation state; snapshot fixes an immutable version.** The saved result becomes the starting point for the next interval, so successive snapshots and interleaved pause/resume follow the same rule. This allows memory saving to become a complete storage commit operation: the server handles capture and policy, while the daemon performs sealing, compaction, and base updates, reducing state-dependent branches in capture and storage orchestration across modules. ### 3. Architecture and RPCs: the server organizes capture; the daemon owns the complete storage commit The diagrams below use the current RC architecture as the baseline and keep the same overall layout for comparison. Yellow denotes compaction decisions, green denotes daemon / OverlayBD execution, and dashed lines denote query dependencies. #### Before: disk commits already run in the daemon; memory storage operations are still orchestrated by the server The current RC implementation already shares compaction policy and delegates immutable-layer merging to the daemon. However, memory sealing, base updates, and snapshot-layer assembly remain in the server; for memory compaction, the daemon mainly receives requests to “merge these files.” ```mermaid flowchart TB Sandbox["FirecrackerSandbox"] Memory["Memory capture FC cumulative dirty → server capture"] MemoryStorage["server: seal memory / update config retain or replace snapshot lowers"] Disk["Disk capture"] Policy["Shared compaction policy"] Manager["UblkDeviceManager RPC forwarding"] Merge["daemon: merge selected immutable layers"] Device["daemon: compact / seal upper restack / update runtime config"] Readers["ImageService local and remote readers / cache"] Sandbox --> Memory --> MemoryStorage Sandbox --> Disk MemoryStorage -. query merge range .-> Policy Disk -. query merge range .-> Policy MemoryStorage -->|when compaction is needed| Manager Disk --> Manager Manager -->|CompactLayers RPC| Merge Manager -->|SnapshotDevice RPC| Device Merge -. reads and caching .-> Readers Device -. reuse already-open readers .-> Readers classDef policy fill:#fff1cc,stroke:#ad7c14; classDef storage fill:#e5f4ed,stroke:#3e8064; class Policy policy; class Merge,Device,Readers storage; ``` #### After: both paths commit their current mutable upper, with storage operations performed by the daemon After the server writes the latest memory changes into the mutable upper, both memory and disk enter a complete snapshot-commit flow. **Compaction policy stays in server / snapshot / repository: determine which layers may be merged, select the range, and pass the plan to the daemon through the unified RPC.** The daemon executes compaction, sealing, base updates, and new-upper creation according to that plan, validating the actual upper state and layer count as needed. No daemon → repository dependency is added. ```mermaid flowchart TB Sandbox["FirecrackerSandbox"] Memory["FC per-save dirty server writes the mutable memory upper"] Disk["Disk's already-written mutable upper"] Manager["UblkDeviceManager unified snapshot request"] Commit["daemon / OverlayBD compact / seal as a sealed upper advance base / create mutable upper"] Policy["server / snapshot / repository compaction eligibility and range"] Readers["ImageService local and remote readers / cache"] Output["server: assemble a complete snapshot VM state + memory snapshot + disk snapshot"] Sandbox --> Memory --> Manager Sandbox --> Disk --> Manager Memory -. query merge range .-> Policy Disk -. query merge range .-> Policy Manager -->|unified snapshot-commit RPC carrying the compaction plan| Commit Commit -. reads and caching .-> Readers Commit --> Output classDef policy fill:#fff1cc,stroke:#ad7c14; classDef storage fill:#e5f4ed,stroke:#3e8064; class Policy policy; class Commit,Readers storage; ``` **The architectural benefit is a clear division: FC manages increments, the server manages capture and policy, and the daemon manages the complete storage commit.** The server supplies the sources and compaction plan; the daemon performs sealing, compaction, base rotation, and configuration updates. The RPC consequently becomes “commit the current private writable state.” The daemon returns read-only memory and disk storage results. The server combines them with VM state into a complete, restorable snapshot, and makes the snapshot independently retain the files it needs: **the snapshot remains restorable after the source sandbox exits or its runtime directory is cleaned.** Publication to the snapshot repository remains a subsequent existing flow and is omitted from the diagram. Memory-save state belongs to each sandbox. Sharing a read-only restore device does not mean sharing mutable state. ### Tradeoffs Per-save increments reduce repeated work during export, but successive snapshots create more sealed uppers. Reuse the existing compaction capability to control layer count, and evaluate overall performance including format conversion, compaction, and restore costs. Do not assume that total pause time will necessarily decrease. These are goals to be implemented: first establish the new incremental-export semantics, then unify the save flow, and finally consolidate storage responsibilities and RPCs.

bsakaguchi added the

enhancement

label

2026-10-29 14:19:40 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:46 +00:00

[PR #107] [MERGED] perf(p2p): parallelize catalog peer lookups #185