[GH-ISSUE #363] [Design Proposal] Snapshot data migration from OSS/POSIX to SQL #91

New issue

Open

opened 2026-10-29 14:19:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-29 14:19:28 +00:00

Owner

Copy link

Originally created by @guozy18 on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/363

Affected component

Snapshot / template builder

Problem statement

Part of the parent proposal, using the data model, sandbox hold contract, the online GC rules, and upgrade compatibility.

Snapshot metadata and runtime descriptions are split across records, alias files, and a separate runtime manifest. Copying only the record loses facts needed to resume the same VM or protect its complete dependencies. The transition must also preserve publication, deletion, builds, and existing consumers: a static copy and configuration switch cannot prevent a late write from being lost or an old file from reviving a snapshot already deleted in SQL.

Use case

An existing OSS or POSIX repository is moving to SQL while users continue creating, reading, publishing, pausing, and restoring sandboxes. Metadata can move gradually while long-lived consumers keep their VM state and layer bytes at existing locations. S denotes a public snapshot, U an existing layer UUID brought under the unified identity contract, and H a shared usage hold. Examples use shortened identifiers and digests, not an executable dump; no second layer identity is introduced.

Current behavior and workarounds

A Distributed snapshot combines business metadata and layer references in a record, backend and virtual-size information in a runtime manifest, and binary artifacts. Local records also identify the publishing node; complete private inputs stay with that process. Record/alias queries do not establish a database-visible hold. Deletion removes metadata and performs backend cleanup, while shared-layer reclamation is separate.

Desired behavior

• Imported runnable inputs preserve backend, resources, startup, ordering, mounts and bytes; unpublished templates remain non-runnable, and Local still requires the original process's files.
• Foreground requests and a resumable background importer can run concurrently without overwriting SQL state, rebinding aliases, or reviving deleted snapshots.
• Existing consumers retain their inputs and receive continuous protection before any migration-domain reclamation is enabled.
• Real storage/database tests cover cross-cutover writes, unknown results, duplicate imports, UUID/parent conflicts and List/Delete; failures delay affected conversions without a maintenance pause or forced consumer exit.
• Reconstruct layer chains from complete legacy inputs, reuse verified prefixes, and register only their heads for each snapshot. Normal GC uses the database graph rather than rereading all legacy manifests.

Proposed approach

1. Before and after: repository layout

OSS directories below are key prefixes. Other bucket contents and registry-internal layouts are outside these trees.

BEFORE OSS: <bucket>/<prefix>/
  catalog/{records/{S}.json,aliases/{alias}.json}
  artifacts/{S}/{firecracker-manifest.json,vm_state.bin}
  managed-layers/{digest}
BEFORE POSIX: <snapshot_store>/repository/
  catalog/{records/{S}.json,aliases/{alias}} and corresponding .lock files
  snapshots/{S}/{firecracker-manifest.json,vm_state.bin,commit}
  managed-layers/{escaped-digest}.overlaybd.commit

POSIX escapes colons and slashes in a digest to underscores; OSS does not use this filename transformation or the POSIX commit marker. Import the actual path, not a reconstructed path from another backend.

AFTER OSS: <bucket>/<prefix>/
  <immutable complete manifests>               referenced by manifest_ref
  artifacts/{old-S}/vm_state.bin               existing artifacts retained
  managed-layers/{digest}                      existing digest-based addressing
AFTER POSIX: <snapshot_store>/repository/
  <immutable complete manifests>               referenced by exact adapter locator
  snapshots/{old-S}/vm_state.bin               existing artifacts retained
  managed-layers/{escaped-digest}.overlaybd.commit
SQL:
  snapshots: identity, alias, business metadata, manifest_ref and retirement
  layers: UUID, fixed predecessor, published digest and lifecycle state
  snapshot_heads: snapshot-to-chain-head references
  snapshot_pins: H-to-S holds

These are responsibility and layout sketches, not a frozen DDL or manifest pathname convention. The complete immutable manifest remains in the backend; SQL becomes authoritative for publication, names, dependency heads and holds. A successful manifest upload alone does not publish S. The verified manifest reference, newly registered layers and heads commit together.

Old record, alias, runtime-manifest, lock, and commit files remain during verification and compatibility cleanup. Removing a manifest never authorizes deleting its whole artifact directory. New layer identity and physical deletion/reuse rules must be settled under #361/#364 before reclamation is enabled; retaining digest-based paths is not itself proof that a late DELETE cannot affect a later publication.

Generated OverlayBD image configs, Local private files, paused artifacts, writable uppers, and byte/P2P caches remain node-owned. Registry manifests/configs/blobs remain registry artifacts. AENV-owned retention roots, VM state and startup artifacts retain explicit protection under their respective ownership rules; replacing layer mappings does not remove those obligations.

2. Map every source of metadata

Before	After	Required preservation
record.id / alias / created_at_unix_ms	snapshots.snapshot_id / alias / created_at	Keep S and UTC epoch milliseconds; verify record/alias-file agreement; do not reassign names
snapshot_type / committed presence	Publication/type facts and manifest_ref	Incomplete content remains non-runnable with no published manifest, regardless of an old Distributed default
owner_node_id	Local manifest.location.node_id	Routing location, not ownership or permission for another process to recover files
source / updated_at_unix_ms	Business metadata	Preserve source details, build status/errors, and original update time
resources	Immutable manifest runtime resources, or requested resources before publication	Keep requested and actual build resources distinct
committed.context / startup	Immutable manifest runtime context / startup	Preserve both contexts independently, including commands, environment, workdir and related settings
runtime_versions / image_configs / custom_extension_params	Complete immutable runtime description	Preserve versions, source OCI config JSON, extension data and override rules
rootfs_layers / memory_layers / attached_drives	Ordered manifest references; validated layer UUID/parent relations; snapshot heads	Preserve role, layer/drive order, read_only, virtual_size, mount_path and sub_path; verify UUIDs before admitting them to the graph
disk_publications	Registry roots, provenance and readiness metadata	Preserve retention responsibility; verify the exact repository/manifest digest and public tag
runtime manifest version / backend	Target format_version / runtime.backend	Decode the old codec; only a missing backend defaults to firecracker, never an unknown value
memory/rootfs.virtualSize / attachedDrives	Corresponding runtime sizes and drive descriptions	Cross-check against committed metadata; file size is not virtual size
vmState={} / omitted local paths	Verified VM-state locator; paths materialized locally	An empty serialized object does not mean the binary is absent
alias file containing a JSON string S	The alias binding on the same S	The name comes from the filename; no separate SQL alias table is required
Binary VM state and layers	Exact locators and descriptors in the complete manifest; necessary layer identity in SQL	Verify storage domain, key, digest, size and format; bytes do not enter SQL

New lifecycle fields are initialized only for verified, previously absent target rows. Re-import must not reset retirement, released holds or deletion states, overwrite a published manifest reference, or reinstall already-released heads.

A non-null file UUID is not sufficient evidence of the new identity contract. Classify legacy layers with missing/nil UUIDs, duplicate UUIDs, conflicting content descriptors, or absent/inconsistent predecessors. Validated layers may retain their UUIDs and bytes. Others require an explicit compatibility or conversion outcome before entering the authoritative graph; do not guess a predecessor, silently allocate an unrelated database-only layer identity, or force conflicting rows together.

If normalization requires a new artifact, create a new copy and immutable manifest, and account for any successor layers whose embedded parent changes. Never rewrite a file in place while it may be hard-linked, read by an existing VM, or served through P2P. Existing consumers keep their fixed old inputs and protection. The exact normalization and UUID-generation algorithm belongs to #361, not to the importer.

3. A concrete Distributed snapshot

Suppose a verified legacy S0, alias python-base, uses rootfs [A, B], memory [M], and a read-only attached drive that also uses [A, B]. A and M have no predecessor; B names A as its predecessor. Their existing UUIDs satisfy the new contract.

The importer combines the old record and runtime manifest into one immutable manifest, denoted MF0. It preserves resources, runtime and startup contexts, commands, versions, source image configs, extension parameters, virtual sizes, drive IDs/mounts/subpaths, and exact layer/VM-state locators. In abbreviated form:

MF0:
  format_version, runtime.backend, resources, contexts, startup, versions, ...
  rootfs:          ordered [A, B], head B, virtual_size
  memory:          ordered [M],    head M, virtual_size
  attached drive:  ordered [A, B], head B, read_only, mount_path, sub_path, ...
  artifacts:       verified VM-state descriptor
  provenance:      source and any required retention responsibilities

MF0 is an exact immutable backend reference, not a mutable filename that later publications overwrite. After preparing it outside the SQL transaction, the importer commits this logical state atomically:

Record	Imported facts
snapshots S0	alias python-base, original created_at and business metadata, publication/type facts, manifest_ref=MF0, not retired
layers A	UUID=A, digest=dA, parent_uuid=NULL, available
layers B	UUID=B, digest=dB, parent_uuid=A, available
layers M	UUID=M, digest=dM, parent_uuid=NULL, available
snapshot_heads	(S0, B), (S0, M)

The rootfs and attached drive share B as a head. Their separate roles and layer order remain in MF0; SQL does not duplicate the whole [A, B] mapping for either role. A later snapshot [A, B, C] can reuse A/B and register C plus its head when the producer contract permits it.

No active snapshot_pins can be inferred from these files. Real consumers acquire or adopt H. Absence of imported H is not evidence that GC is safe. A legacy digest reused under conflicting predecessors cannot be treated like this verified example merely because its bytes exist; it follows the identity-validation path above.

4. Unpublished templates, Local files, and registry roots

An old template may default to Distributed while committed is null. Import it as non-runnable, without a published manifest reference or heads. Preserve requested resources, build status/error and timestamps rather than inventing completed content. Publication records actual resources and resolves the requested-resource state atomically. A resulting 4 GiB rootfs is 4096 MiB, not the original requested disk size 0.

A Local record may have owner_node_id=N1 and empty committed arrays even though its actual inputs depend on remote layers. Importing requires the original process's complete fixed inputs. Its manifest may describe relative private files and remote layers, but publishing that description does not grant another process ownership of the private files. Shared ancestors still need their verified heads and source protection; private files do not become independently reclaimable OSS artifacts.

A new process cannot recover public Local content from a node name or leftover directory. Preserve the original process for a verified handoff, or promote there before importing Distributed content. Retain the union of old/new protection needed by existing readers; do not assume the promoted inputs cover all earlier inputs. Without the original inputs, keep the snapshot unavailable or explicitly retire it.

For registry publications, preserve exact repository/manifest identities, subjects, provenance and retention responsibilities. Verify the fixed public tag before recording imageRef readiness. Do not rewrite registry bytes or create a digest merely to import metadata. External layers that depend on an AENV-created retention root must retain it; an unrelated source image does not become AENV-owned. The layer forest alone does not define registry or shared non-layer reclamation.

5. Keep operations available while importing

Operation	Before	During Mixed and after SQL takeover
Get / alias lookup	Read alias and record files	SQL first, including retirement facts; import an untouched old S on demand
List	Enumerate records and filter/sort	Merge old and SQL results by S with SQL precedence; then filter, sort and paginate; SqlOnly queries SQL
Create / Publish	Write records, aliases and completed artifacts	New identities go to SQL; an old S is imported first, then updated through its Catalog contract
Build/publication callback	Update the repository record	Recheck mode at commit, use conditional SQL updates, and register added dependencies atomically
Delete	Remove record/alias and perform backend cleanup	Record retirement in SQL; retain names and bytes until hold/reference rules permit reclamation

First create the schema and deploy compatible nodes that still serve the file catalog. Add capacity, stop assigning new work to old nodes, and keep their existing bindings and consumers until their responsibilities end. A compatible node's readiness checks both SQL and object storage.
The compatible Repository uses fixed modes Oss, Mixed and SqlOnly, records source H and dependencies from the OSS phase, and disables direct physical Delete/rollback bypasses. All metadata writers, tools and asynchronous callbacks must use the common entry point before Mixed starts.

For metadata modification/import, acquire sorted alias locks before sorted S locks and recheck names and mode. These migration locks precede Catalog transaction locks; do not hold a long SQL transaction around storage I/O.
Before an old metadata write, independently commit a durable, cross-process-visible intent. Order its completion confirmation against the Mixed switch in a control transaction:

old writer holds S -> commits intent -> issues OSS metadata write
mode changes to Mixed -> another request for S waits; other S continue
old write finishes -> hand off final result or Delete tombstone to SQL
handoff confirmed -> clear intent -> release S -> next request uses SQL

Unknown old writes retain their S/alias marker. A dropped connection, released lock, or one successful HEAD/GET does not prove that a late write cannot arrive.
Foreground requests and the background tool share ensure_imported(S): resolve pending old writes, accept an existing SQL row including a tombstone, otherwise verify the complete legacy inputs, prepare their immutable manifest, and atomically register its reference, validated layers and required heads/protection. Do not overwrite an existing SQL result.
Alias claims check all SQL reservations, then untouched old alias files and their target records under the same arbitration. SQL absence alone does not make a legacy name free.
List precedence includes SQL tombstones and rows excluded by the user's filter, so an old live record cannot reappear. Apply one (created_at DESC, snapshot_id ASC) order and cursor after merging; never concatenate independently paged backend results. Mixed retains scanning costs without promising a fixed snapshot across pages.
The background tool checkpoints batches and retries through the same helper. Resolve pre-switch submissions, scan the resulting stable record/alias scope, and require an explicit outcome for every old record. Two unchanged listings alone do not prove completion.

6. Adopt existing consumers before enabling reclamation

Each compatible executor stops creating new OSS-only holds before auditing its acquiring, held, paused and background responsibilities. New cold preparation uses ordinary SQL Pin; existing consumers keep their fixed files and devices.
A migration-only adopt_existing_hold(H,S,inputs) atomically registers the original active H, the verified heads of its fixed layer inputs and any other required protection. It can protect an already-retired S; if the source record was deleted, create a retired, unreleased identity without republishing it or claiming a historical alias.
Adopt and Release follow the common H-to-S ordering and the layer/reference arbitration defined by #361/#364. Closed H never reopens. Adoption only adds necessary protection; it does not replace the current manifest_ref, reinstall protection after completed unlinking, or bypass a deleting/terminal layer state through ordinary import or adoption. Missing bytes, conflicting descriptors/UUIDs and incomplete provenance remain unresolved and conservatively retained for reconciliation; they are not evidence that import or consumer adoption is complete.
Include disconnected executors and direct tools in the audit. Completion means every legacy responsibility was returned or adopted, not merely that current online nodes reported success.
During Mixed/adoption, do not advance F or enable reference removal, payload compaction, identity pruning or physical GC. After full import and adoption, enter SqlOnly and close fallback/import/adopt; only then enable the applicable online cleanup. Long-lived paused consumers can remain protected without exiting.
Once S is retired, has no active H and has safely released its heads and other obligations, eligible payload/reference cleanup follows #364. Clearing manifest_ref or deleting a manifest is not permission to delete a whole artifact directory. A named S keeps its alias reservation; eligible unnamed S and Closed H may later be pruned under the historical-data rules. Active H does not expire. Layer terminal retention and physical-key reuse are separate decisions and must be settled before enabling that cleanup.

Compatibility and operational impact

• API/config changes: preserve public metadata projections and existing snapshot/hold identities; introduce Catalog configuration and explicit migration modes, without exposing H as a user-managed API.
• Snapshot or storage format changes: import complete semantics into immutable manifests plus SQL references, retain old bytes for existing consumers, and generate local configs from verified inputs. Existing S/H remain unchanged; new S/H and the monotonic first-admission floor F follow #364. Legacy S/H without reliable birth information are not age-pruned. Layer UUID validation/normalization is a different concern; do not apply S/H birth-time rules to layer UUIDs.
• New host/runtime requirements: database connectivity, a compatible Repository, a resumable migration tool, temporary handoff records, and capacity for rolling node replacement.
• Upgrade and rollback considerations: after a SQL handoff, that S cannot fall back to OSS. Recover with a version that understands Mixed/SqlOnly and the current holds. Subsequent SQL schema changes and multi-version coexistence are covered by the upgrades proposal.

Alternatives considered

A one-time export misses mutations; uncoordinated dual writes create conflicting authorities; a maintenance pause violates availability. Waiting for all old consumers avoids adoption but lets long-lived paused consumers indefinitely delay GC, so this proposal adopts their existing responsibility.

Non-goals

Do not move live VMs, mutate existing layer bytes in place, redesign external images or volumes, infer usage from cache directories, or implement a separate CDC platform. Any new normalized artifact must have an explicit identity and compatibility plan; this issue does not select the OCI UUID algorithm, layer tombstone policy, or a common volume schema. Do not treat successful row copying as proof that migration and recovery are complete.

Contribution

I can provide more design details

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @guozy18 on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/363 ### Affected component Snapshot / template builder ### Problem statement Part of the [parent proposal](https://github.com/kvcache-ai/AgentENV/issues/360), using the [data model](https://github.com/kvcache-ai/AgentENV/issues/361), [sandbox hold contract](https://github.com/kvcache-ai/AgentENV/issues/362), the [online GC rules](https://github.com/kvcache-ai/AgentENV/issues/364), and [upgrade compatibility](https://github.com/kvcache-ai/AgentENV/issues/365). Snapshot metadata and runtime descriptions are split across records, alias files, and a separate runtime manifest. Copying only the record loses facts needed to resume the same VM or protect its complete dependencies. The transition must also preserve publication, deletion, builds, and existing consumers: a static copy and configuration switch cannot prevent a late write from being lost or an old file from reviving a snapshot already deleted in SQL. ### Use case An existing OSS or POSIX repository is moving to SQL while users continue creating, reading, publishing, pausing, and restoring sandboxes. Metadata can move gradually while long-lived consumers keep their VM state and layer bytes at existing locations. S denotes a public snapshot, U an existing layer UUID brought under the unified identity contract, and H a shared usage hold. Examples use shortened identifiers and digests, not an executable dump; no second layer identity is introduced. ### Current behavior and workarounds A Distributed snapshot combines business metadata and layer references in a record, backend and virtual-size information in a runtime manifest, and binary artifacts. Local records also identify the publishing node; complete private inputs stay with that process. Record/alias queries do not establish a database-visible hold. Deletion removes metadata and performs backend cleanup, while shared-layer reclamation is separate. ### Desired behavior - [ ] Imported runnable inputs preserve backend, resources, startup, ordering, mounts and bytes; unpublished templates remain non-runnable, and Local still requires the original process's files. - [ ] Foreground requests and a resumable background importer can run concurrently without overwriting SQL state, rebinding aliases, or reviving deleted snapshots. - [ ] Existing consumers retain their inputs and receive continuous protection before any migration-domain reclamation is enabled. - [ ] Real storage/database tests cover cross-cutover writes, unknown results, duplicate imports, UUID/parent conflicts and List/Delete; failures delay affected conversions without a maintenance pause or forced consumer exit. - [ ] Reconstruct layer chains from complete legacy inputs, reuse verified prefixes, and register only their heads for each snapshot. Normal GC uses the database graph rather than rereading all legacy manifests. ### Proposed approach ### 1. Before and after: repository layout OSS directories below are key prefixes. Other bucket contents and registry-internal layouts are outside these trees. ~~~text BEFORE OSS: <bucket>/<prefix>/ catalog/{records/{S}.json,aliases/{alias}.json} artifacts/{S}/{firecracker-manifest.json,vm_state.bin} managed-layers/{digest} BEFORE POSIX: <snapshot_store>/repository/ catalog/{records/{S}.json,aliases/{alias}} and corresponding .lock files snapshots/{S}/{firecracker-manifest.json,vm_state.bin,commit} managed-layers/{escaped-digest}.overlaybd.commit ~~~ POSIX escapes colons and slashes in a digest to underscores; OSS does not use this filename transformation or the POSIX commit marker. Import the actual path, not a reconstructed path from another backend. ~~~text AFTER OSS: <bucket>/<prefix>/ <immutable complete manifests> referenced by manifest_ref artifacts/{old-S}/vm_state.bin existing artifacts retained managed-layers/{digest} existing digest-based addressing AFTER POSIX: <snapshot_store>/repository/ <immutable complete manifests> referenced by exact adapter locator snapshots/{old-S}/vm_state.bin existing artifacts retained managed-layers/{escaped-digest}.overlaybd.commit SQL: snapshots: identity, alias, business metadata, manifest_ref and retirement layers: UUID, fixed predecessor, published digest and lifecycle state snapshot_heads: snapshot-to-chain-head references snapshot_pins: H-to-S holds ~~~ These are responsibility and layout sketches, not a frozen DDL or manifest pathname convention. The complete immutable manifest remains in the backend; SQL becomes authoritative for publication, names, dependency heads and holds. A successful manifest upload alone does not publish S. The verified manifest reference, newly registered layers and heads commit together. Old record, alias, runtime-manifest, lock, and commit files remain during verification and compatibility cleanup. Removing a manifest never authorizes deleting its whole artifact directory. New layer identity and physical deletion/reuse rules must be settled under #361/#364 before reclamation is enabled; retaining digest-based paths is not itself proof that a late DELETE cannot affect a later publication. Generated OverlayBD image configs, Local private files, paused artifacts, writable uppers, and byte/P2P caches remain node-owned. Registry manifests/configs/blobs remain registry artifacts. AENV-owned retention roots, VM state and startup artifacts retain explicit protection under their respective ownership rules; replacing layer mappings does not remove those obligations. ### 2. Map every source of metadata | Before | After | Required preservation | |---|---|---| | record.id / alias / created_at_unix_ms | snapshots.snapshot_id / alias / created_at | Keep S and UTC epoch milliseconds; verify record/alias-file agreement; do not reassign names | | snapshot_type / committed presence | Publication/type facts and manifest_ref | Incomplete content remains non-runnable with no published manifest, regardless of an old Distributed default | | owner_node_id | Local manifest.location.node_id | Routing location, not ownership or permission for another process to recover files | | source / updated_at_unix_ms | Business metadata | Preserve source details, build status/errors, and original update time | | resources | Immutable manifest runtime resources, or requested resources before publication | Keep requested and actual build resources distinct | | committed.context / startup | Immutable manifest runtime context / startup | Preserve both contexts independently, including commands, environment, workdir and related settings | | runtime_versions / image_configs / custom_extension_params | Complete immutable runtime description | Preserve versions, source OCI config JSON, extension data and override rules | | rootfs_layers / memory_layers / attached_drives | Ordered manifest references; validated layer UUID/parent relations; snapshot heads | Preserve role, layer/drive order, read_only, virtual_size, mount_path and sub_path; verify UUIDs before admitting them to the graph | | disk_publications | Registry roots, provenance and readiness metadata | Preserve retention responsibility; verify the exact repository/manifest digest and public tag | | runtime manifest version / backend | Target format_version / runtime.backend | Decode the old codec; only a missing backend defaults to firecracker, never an unknown value | | memory/rootfs.virtualSize / attachedDrives | Corresponding runtime sizes and drive descriptions | Cross-check against committed metadata; file size is not virtual size | | vmState={} / omitted local paths | Verified VM-state locator; paths materialized locally | An empty serialized object does not mean the binary is absent | | alias file containing a JSON string S | The alias binding on the same S | The name comes from the filename; no separate SQL alias table is required | | Binary VM state and layers | Exact locators and descriptors in the complete manifest; necessary layer identity in SQL | Verify storage domain, key, digest, size and format; bytes do not enter SQL | New lifecycle fields are initialized only for verified, previously absent target rows. Re-import must not reset retirement, released holds or deletion states, overwrite a published manifest reference, or reinstall already-released heads. A non-null file UUID is not sufficient evidence of the new identity contract. Classify legacy layers with missing/nil UUIDs, duplicate UUIDs, conflicting content descriptors, or absent/inconsistent predecessors. Validated layers may retain their UUIDs and bytes. Others require an explicit compatibility or conversion outcome before entering the authoritative graph; do not guess a predecessor, silently allocate an unrelated database-only layer identity, or force conflicting rows together. If normalization requires a new artifact, create a new copy and immutable manifest, and account for any successor layers whose embedded parent changes. Never rewrite a file in place while it may be hard-linked, read by an existing VM, or served through P2P. Existing consumers keep their fixed old inputs and protection. The exact normalization and UUID-generation algorithm belongs to #361, not to the importer. ### 3. A concrete Distributed snapshot Suppose a verified legacy S0, alias `python-base`, uses rootfs `[A, B]`, memory `[M]`, and a read-only attached drive that also uses `[A, B]`. A and M have no predecessor; B names A as its predecessor. Their existing UUIDs satisfy the new contract. The importer combines the old record and runtime manifest into one immutable manifest, denoted `MF0`. It preserves resources, runtime and startup contexts, commands, versions, source image configs, extension parameters, virtual sizes, drive IDs/mounts/subpaths, and exact layer/VM-state locators. In abbreviated form: ~~~text MF0: format_version, runtime.backend, resources, contexts, startup, versions, ... rootfs: ordered [A, B], head B, virtual_size memory: ordered [M], head M, virtual_size attached drive: ordered [A, B], head B, read_only, mount_path, sub_path, ... artifacts: verified VM-state descriptor provenance: source and any required retention responsibilities ~~~ `MF0` is an exact immutable backend reference, not a mutable filename that later publications overwrite. After preparing it outside the SQL transaction, the importer commits this logical state atomically: | Record | Imported facts | |---|---| | snapshots S0 | alias `python-base`, original created_at and business metadata, publication/type facts, manifest_ref=MF0, not retired | | layers A | UUID=A, digest=dA, parent_uuid=NULL, available | | layers B | UUID=B, digest=dB, parent_uuid=A, available | | layers M | UUID=M, digest=dM, parent_uuid=NULL, available | | snapshot_heads | (S0, B), (S0, M) | The rootfs and attached drive share B as a head. Their separate roles and layer order remain in MF0; SQL does not duplicate the whole `[A, B]` mapping for either role. A later snapshot `[A, B, C]` can reuse A/B and register C plus its head when the producer contract permits it. No active `snapshot_pins` can be inferred from these files. Real consumers acquire or adopt H. Absence of imported H is not evidence that GC is safe. A legacy digest reused under conflicting predecessors cannot be treated like this verified example merely because its bytes exist; it follows the identity-validation path above. ### 4. Unpublished templates, Local files, and registry roots An old template may default to Distributed while committed is null. Import it as non-runnable, without a published manifest reference or heads. Preserve requested resources, build status/error and timestamps rather than inventing completed content. Publication records actual resources and resolves the requested-resource state atomically. A resulting 4 GiB rootfs is 4096 MiB, not the original requested disk size 0. A Local record may have owner_node_id=N1 and empty committed arrays even though its actual inputs depend on remote layers. Importing requires the original process's complete fixed inputs. Its manifest may describe relative private files and remote layers, but publishing that description does not grant another process ownership of the private files. Shared ancestors still need their verified heads and source protection; private files do not become independently reclaimable OSS artifacts. A new process cannot recover public Local content from a node name or leftover directory. Preserve the original process for a verified handoff, or promote there before importing Distributed content. Retain the union of old/new protection needed by existing readers; do not assume the promoted inputs cover all earlier inputs. Without the original inputs, keep the snapshot unavailable or explicitly retire it. For registry publications, preserve exact repository/manifest identities, subjects, provenance and retention responsibilities. Verify the fixed public tag before recording imageRef readiness. Do not rewrite registry bytes or create a digest merely to import metadata. External layers that depend on an AENV-created retention root must retain it; an unrelated source image does not become AENV-owned. The layer forest alone does not define registry or shared non-layer reclamation. ### 5. Keep operations available while importing | Operation | Before | During Mixed and after SQL takeover | |---|---|---| | Get / alias lookup | Read alias and record files | SQL first, including retirement facts; import an untouched old S on demand | | List | Enumerate records and filter/sort | Merge old and SQL results by S with SQL precedence; then filter, sort and paginate; SqlOnly queries SQL | | Create / Publish | Write records, aliases and completed artifacts | New identities go to SQL; an old S is imported first, then updated through its Catalog contract | | Build/publication callback | Update the repository record | Recheck mode at commit, use conditional SQL updates, and register added dependencies atomically | | Delete | Remove record/alias and perform backend cleanup | Record retirement in SQL; retain names and bytes until hold/reference rules permit reclamation | First create the schema and deploy compatible nodes that still serve the file catalog. Add capacity, stop assigning new work to old nodes, and keep their existing bindings and consumers until their responsibilities end. A compatible node's readiness checks both SQL and object storage. The compatible Repository uses fixed modes Oss, Mixed and SqlOnly, records source H and dependencies from the OSS phase, and disables direct physical Delete/rollback bypasses. All metadata writers, tools and asynchronous callbacks must use the common entry point before Mixed starts. For metadata modification/import, acquire sorted alias locks before sorted S locks and recheck names and mode. These migration locks precede Catalog transaction locks; do not hold a long SQL transaction around storage I/O. Before an old metadata write, independently commit a durable, cross-process-visible intent. Order its completion confirmation against the Mixed switch in a control transaction: ~~~text old writer holds S -> commits intent -> issues OSS metadata write mode changes to Mixed -> another request for S waits; other S continue old write finishes -> hand off final result or Delete tombstone to SQL handoff confirmed -> clear intent -> release S -> next request uses SQL ~~~ Unknown old writes retain their S/alias marker. A dropped connection, released lock, or one successful HEAD/GET does not prove that a late write cannot arrive. Foreground requests and the background tool share `ensure_imported(S)`: resolve pending old writes, accept an existing SQL row including a tombstone, otherwise verify the complete legacy inputs, prepare their immutable manifest, and atomically register its reference, validated layers and required heads/protection. Do not overwrite an existing SQL result. Alias claims check all SQL reservations, then untouched old alias files and their target records under the same arbitration. SQL absence alone does not make a legacy name free. List precedence includes SQL tombstones and rows excluded by the user's filter, so an old live record cannot reappear. Apply one `(created_at DESC, snapshot_id ASC)` order and cursor after merging; never concatenate independently paged backend results. Mixed retains scanning costs without promising a fixed snapshot across pages. The background tool checkpoints batches and retries through the same helper. Resolve pre-switch submissions, scan the resulting stable record/alias scope, and require an explicit outcome for every old record. Two unchanged listings alone do not prove completion. ### 6. Adopt existing consumers before enabling reclamation Each compatible executor stops creating new OSS-only holds before auditing its acquiring, held, paused and background responsibilities. New cold preparation uses ordinary SQL Pin; existing consumers keep their fixed files and devices. A migration-only `adopt_existing_hold(H,S,inputs)` atomically registers the original active H, the verified heads of its fixed layer inputs and any other required protection. It can protect an already-retired S; if the source record was deleted, create a retired, unreleased identity without republishing it or claiming a historical alias. Adopt and Release follow the common H-to-S ordering and the layer/reference arbitration defined by #361/#364. Closed H never reopens. Adoption only adds necessary protection; it does not replace the current manifest_ref, reinstall protection after completed unlinking, or bypass a deleting/terminal layer state through ordinary import or adoption. Missing bytes, conflicting descriptors/UUIDs and incomplete provenance remain unresolved and conservatively retained for reconciliation; they are not evidence that import or consumer adoption is complete. Include disconnected executors and direct tools in the audit. Completion means every legacy responsibility was returned or adopted, not merely that current online nodes reported success. During Mixed/adoption, do not advance F or enable reference removal, payload compaction, identity pruning or physical GC. After full import and adoption, enter SqlOnly and close fallback/import/adopt; only then enable the applicable online cleanup. Long-lived paused consumers can remain protected without exiting. Once S is retired, has no active H and has safely released its heads and other obligations, eligible payload/reference cleanup follows #364. Clearing manifest_ref or deleting a manifest is not permission to delete a whole artifact directory. A named S keeps its alias reservation; eligible unnamed S and Closed H may later be pruned under the historical-data rules. Active H does not expire. Layer terminal retention and physical-key reuse are separate decisions and must be settled before enabling that cleanup. ### Compatibility and operational impact - **API/config changes:** preserve public metadata projections and existing snapshot/hold identities; introduce Catalog configuration and explicit migration modes, without exposing H as a user-managed API. - **Snapshot or storage format changes:** import complete semantics into immutable manifests plus SQL references, retain old bytes for existing consumers, and generate local configs from verified inputs. Existing S/H remain unchanged; new S/H and the monotonic first-admission floor F follow #364. Legacy S/H without reliable birth information are not age-pruned. Layer UUID validation/normalization is a different concern; do not apply S/H birth-time rules to layer UUIDs. - **New host/runtime requirements:** database connectivity, a compatible Repository, a resumable migration tool, temporary handoff records, and capacity for rolling node replacement. - **Upgrade and rollback considerations:** after a SQL handoff, that S cannot fall back to OSS. Recover with a version that understands Mixed/SqlOnly and the current holds. Subsequent SQL schema changes and multi-version coexistence are covered by the upgrades proposal. ### Alternatives considered A one-time export misses mutations; uncoordinated dual writes create conflicting authorities; a maintenance pause violates availability. Waiting for all old consumers avoids adoption but lets long-lived paused consumers indefinitely delay GC, so this proposal adopts their existing responsibility. ### Non-goals Do not move live VMs, mutate existing layer bytes in place, redesign external images or volumes, infer usage from cache directories, or implement a separate CDC platform. Any new normalized artifact must have an explicit identity and compatibility plan; this issue does not select the OCI UUID algorithm, layer tombstone policy, or a common volume schema. Do not treat successful row copying as proof that migration and recovery are complete. ### Contribution I can provide more design details ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:28 +00:00

bsakaguchi referenced this issue 2026-10-29 14:20:39 +00:00

[PR #91] [MERGED] docs: add static multi-node deployment guide #173

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#91

No description provided.