[GH-ISSUE #361] [Design Proposal] Snapshot Catalog data model and API #88

New issue

Open

opened 2026-10-29 14:19:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-29 14:19:26 +00:00

Owner

Copy link

Originally created by @guozy18 on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/361

Affected component

Snapshot Catalog schema, shared layers, and database transaction interfaces.

Related: Overview #360 · Usage holds #362 · Migration and field mapping #363 · Online GC #364 · Upgrades #365.

Problem statement

A full snapshot-to-layer mapping repeats inherited relationships on every publication. Store each sealed layer's existing UUID and fixed predecessor once; let snapshots retain only their chain heads. The tables below make that proposal concrete without introducing a second layer identity.

1. Before and after

Information	Full mapping proposal	Prefix proposal
Snapshot description	Complete runtime manifest in SQL	manifest_ref in SQL; immutable complete manifest in storage
Layer dependencies	Every snapshot records every layer	Each layer records one predecessor; each snapshot records heads
Usage protection	Shared snapshot hold H	Preserve H and its existing lifecycle
Publication of a known prefix plus a suffix	Insert all snapshot-layer relationships	Insert new layer rows and head relationships; reuse existing ancestors

snapshots S0 ── snapshot_heads ──> B ──parent_uuid──> A
snapshots S1 ── snapshot_heads ──> C ──parent_uuid──> B
snapshot_pins H0 ───────────────> S0

S is a snapshot identity, H is a usage-hold identity, and A/B/C are the existing file UUIDs after their contract is unified. They are not new database-only layer IDs.

2. Candidate table definitions

These four tables describe the snapshot-side layer-retention core, not a claim that four tables already cover the entire snapshot/volume system. Types below use PostgreSQL notation. SQLite uses canonical UUID text, validated JSON text, and equivalent transactional constraints.

snapshots: public identity and content entry

Column	Type / constraint	Why it is needed; what removal would break
snapshot_id	UUID PRIMARY KEY; immutable	Existing public identity and the target of heads/pins; retries must address the same snapshot
alias	TEXT NULL; unique when present, nonblank, case-sensitive	Existing name lookup and permanent name reservation; a retired name must not silently resolve to another S
snapshot_type	TEXT NOT NULL; none, local, or distributed	Distinguishes an unpublished template, process-owned Local input, and repository-resolvable Distributed input
manifest_ref	TEXT NULL; exact immutable manifest locator	Locates the complete fixed runtime description without storing it all in SQL; NULL before publication or after permitted terminal cleanup
metadata	JSONB NULL; object when present, validated by the business codec	Preserves source/build/requested-resource information that may exist before a runnable manifest exists; see the field placement table below
retired	BOOLEAN NOT NULL DEFAULT FALSE; cannot revert	Rejects first admission of a new hold while preserving established users; deleting the row immediately would lose this distinction
created_at	BIGINT NOT NULL; immutable UTC epoch milliseconds	Preserves existing ordering, pagination, and original creation time; it is not the identity-admission timestamp used by F

Snapshot shape	Required state
Unpublished template	snapshot_type=none, no runnable manifest_ref, no published heads; requested resources/build state remain in metadata
Published Local	Manifest identifies publishing node and relative private inputs; opening also requires the original process's file guard; no public alias
Published Distributed	Complete fixed manifest plus all required heads and other retention responsibilities are committed
Retired, still in use	Keep the description and protection required by existing H; Get/List hide the retired snapshot
Terminal	Payload/reference cleanup follows #364; retain the business content kind and any required identity/name record

Promotion may change S's manifest_ref, but it keeps old readers' fixed inputs and the union of old/new heads for as long as their protection requires. An immutable manifest is never overwritten in place.

layers: one row per sealed UUID

Column	Type / constraint	Why it is needed; what removal would break
uuid	UUID PRIMARY KEY; non-nil, immutable	Reuses the layer identity already carried by the file and runtime; removing it would require another identity or conflate distinct artifacts
parent_uuid	UUID NULL; immutable reference to the predecessor; no cascading deletion	Encodes the shared prefix; without it every snapshot would have to enumerate the full chain again
digest	TEXT NOT NULL; nonempty, immutable for the canonical published representation	Identifies the exact final bytes for managed storage and P2P; UUID alone is not proof of byte equality
state	TEXT NOT NULL; available, deleting, deleted lifecycle	Arbitrates new references against reclamation and preserves deletion progress across failures; a zero-reference query alone cannot authorize OSS DELETE

Invariant	Enforcement / boundary
One UUID has one predecessor and canonical published descriptor	Producer, file metadata, runtime descriptor, and SQL must agree; reject conflicts instead of rebinding an existing row
The graph is a forest	New layers refer to an existing valid predecessor or have no predecessor; immutable creation rules must prevent cycles, not just a foreign key
Identity reuse means artifact reuse	Fork, relocation, conversion-cache reuse, and publication retry preserve a sealed artifact's UUID; new content or a different predecessor requires a new artifact
Digest includes embedded identity	Different file UUIDs do not generally produce the same full-file digest; there is no (digest,parent_uuid) rule that merges different embedded UUIDs
Private and shared representations differ	A private sparse file must not be advertised as the canonical shared bytes merely because its UUID matches
Deletion targets are scoped	This four-column sketch assumes a configured managed storage namespace where digest resolves the target. Other domains/imported locations require exact locator handling, not deletion by digest in an arbitrary namespace
Terminal edges and rows	Active predecessor references must remain valid. Retaining a terminal child while pruning its parent affects foreign-key layout; finalize that together with the terminal retention/reuse policy in #364

The existing OCI, memory, compaction, and publication paths must first satisfy this UUID/predecessor contract. The exact OCI UUID-generation algorithm remains open. Producers/transforms should return the final digest/size descriptor to avoid redundant digest scans; known-digest P2P reads must validate that exact content.

snapshot_heads: retain chain tops, not every layer

Column	Type / constraint	Why it is needed; what removal would break
snapshot_id	UUID NOT NULL; references snapshots.snapshot_id	Identifies which snapshot retains the chain
layer_uuid	UUID NOT NULL; references layers.uuid	Identifies the top from which all predecessors remain protected

Primary key: (snapshot_id, layer_uuid). Do not cascade-delete these rows on public Delete; retirement must wait for active use. Role, order, and duplicate uses belong in the manifest. If rootfs and an attached drive both use [A,B], one (S,B) head is sufficient for retention.

snapshot_pins: shared usage responsibility

Column	Type / constraint	Why it is needed; what removal would break
pin_id	UUID PRIMARY KEY; immutable H	Identifies one shared use period and its retries; a sandbox ID is not an equivalent identity
snapshot_id	UUID NOT NULL; immutable H-to-S binding	Identifies the snapshot protected by H and prevents a retry from changing its target
released	BOOLEAN NOT NULL DEFAULT FALSE; false to true only	Distinguishes active protection from a Closed H; deleting H immediately would allow a delayed Pin to reopen it

Do not require a parent foreign key from this table to snapshots: Release may arrive before S or before Pin. First active Pin validates S transactionally; a release-first Closed H cannot later become active. Multiple prepared inputs/VMs/children can share H through local ownership; this is not one row per VM.

Required access paths

Table	Key / index	Query it supports
snapshots	PK on snapshot_id; unique non-NULL alias	Identity/name arbitration
snapshots	(created_at DESC, snapshot_id ASC)	Existing ordered List pagination
snapshots	Retirement scan by snapshot ID, optionally a partial index	Find pending retirement work without blocking first-page candidates forever
layers	PK on uuid; index on parent_uuid	Follow parents and test for direct retaining children
layers	Index on digest; deletion-state scan	Check exact managed-file references and recover pending deletion
snapshot_heads	PK (snapshot_id,layer_uuid); reverse (layer_uuid,snapshot_id)	Enumerate a snapshot's heads and test whether a layer is a head
snapshot_pins	Active index on snapshot_id WHERE NOT released	Decide whether retirement can release a snapshot's heads

No persisted ref_count is required: indexed head/valid-child existence checks are sufficient for logical layer retention. No separate heads_released flag is needed just to make head deletion idempotent. Zero heads does not prove that non-layer artifacts or external deletion work have finished. Any separate all-obligations-complete fact must cover those responsibilities before payload cleanup.

3. Field placement: SQL versus immutable manifest

Existing information	Placement	Preservation rule
Identity, alias, type, creation time, retirement	Dedicated snapshots columns	Needed for lookup, listing, admission, and retirement
Source/provenance business fields, build status/error, original update time	snapshots.metadata	Preserve the existing business projection; a late build error cannot overwrite a published or retired result
Requested resources before publication	snapshots.metadata	An unpublished template has no completed runtime manifest
Actual resources after publication	Immutable manifest	Preserve actual runtime values separately from the build request
Backend, runtime/startup contexts, versions, source image configs, extensions	Immutable manifest	Complete codec validation; missing backend compatibility must not accept an unknown backend
Ordered rootfs/memory/drive/volume-snapshot layers	Immutable manifest; their unique chain heads in SQL	Preserve roles, drive IDs, read-only flags, virtual sizes, mounts and subpaths without duplicating every layer mapping
Exact artifact descriptors and locations	Immutable manifest and required deletion/retention responsibility	Preserve digest/size/format/storage domain; a locator alone does not supply a GC protocol
Local publishing node and private-file relative locations	Local description plus original process guard	A SQL row cannot recover a lost Local snapshot in another process
Node-local absolute paths, byte-cache state, Ready entries	Node-owned state	Do not make process-local ownership shared SQL truth

The detailed old-to-new field mapping remains in #363. Build logs retain their existing ownership.

4. Concrete records: shared prefix, fork, and in-use deletion

All IDs and digests below are shortened examples. mf0/mf1/mf2 are exact references to different immutable manifests, not mutable aliases.

Publish S0

mf0: rootfs=[A,B], memory=[M], runtime configuration and artifact descriptors
B -> A
M has no predecessor

snapshots

snapshot_id	alias	snapshot_type	manifest_ref	metadata	retired	created_at
S0	python-base	distributed	mf0	existing business metadata	false	t0

layers

uuid	parent_uuid	digest	state
A	NULL	dA	available
B	A	dB	available
M	NULL	dM	available

snapshot_heads

snapshot_id	layer_uuid
S0	B
S0	M

snapshot_pins: empty. Publishing S0 does not itself create a usage pin.

Subsequent operations: exact row changes

Suppose S1 adds C to the same rootfs prefix, and a fork from S0 later publishes S2 with D:

mf1: rootfs=[A,B,C], memory=[M]
mf2: rootfs=[A,B,D], memory=[M]

Operation	snapshots	layers	snapshot_heads	snapshot_pins
Cold-open S0	Read/fix mf0	No change	No change	Insert (H0,S0,false) before fetching mf0 or artifacts
Publish S1	Insert S1 with mf1, not retired	Insert (C,B,dC,available) only	Insert (S1,C) and (S1,M)	Keep source H0 while still needed
Fork a child from S0	No change	No shared row merely for fork	No change	Child inherits the required local share of H0; no automatic new H
Publish child as S2	Insert S2 with mf2, not retired	Insert (D,B,dD,available) only	Insert (S2,D) and (S2,M)	Retain H0 while source reads continue
Delete S0 while H0 is active	Set S0.retired=true	No change	Keep (S0,B) and (S0,M)	H0 stays active; valid established use continues
Last H0 responsibility ends	No change	No change	No change yet	Set H0.released=true
GC releases S0 heads	Keep retired S0 and its alias reservation	Keep A/B/C/D/M	Remove (S0,B) and (S0,M)	Keep Closed H0 until eligible history cleanup
S1 later retires, has no H, and its heads are released	Mark S1 retired	C becomes a candidate; B/A/M remain retained by S2	Remove (S1,C) and (S1,M)	No active S1 hold

After releasing S0, the surviving references are:

S1 -> C -> B -> A
S2 -> D -> B -> A
S1/S2 -> M

C can enter deletion only after current checks and durable authorization succeed; deleting its OSS bytes is a separate step. This example does not choose permanent layer tombstones or permission to reuse deleted UUIDs/keys; see #364.

Relationship-write comparison

Case	Full snapshot-layer mapping	Prefix tables
Known k-layer input plus delta new layers	Repeat relationships for the resulting full layer set	Register delta new layers and h unique heads; existing ancestors stay unchanged
First publication of an unfamiliar chain	Register its layer set and mapping	Register the unfamiliar chain and its heads; no prefix saving is assumed
Many snapshots inherit A/B	Repeat A/B mappings for each snapshot	Keep A/B once in layers; each snapshot retains its own heads
Cost moved elsewhere	Direct full-set enumeration	Parent indexes, graph traversal, and branch reclamation

The intended database relationship-write bound is O(delta + h) for known prefixes. Full manifest serialization, uploads, digest work, and initial imports are measured separately; this is not a claim that the entire publish operation is O(delta).

5. Transaction and API boundaries

Entry point	Rows / result	Required atomicity or check
Get / List	Snapshot metadata and manifest reference	Grants no right to consume artifacts; hide retired rows
Publish	S, new layers, heads, and manifest_ref	Prepare bytes/manifest outside SQL; validate reused prefix identities and atomically expose the completed result
Pin(S,H)	Active H plus fixed protected metadata/manifest_ref	Arbitrate against retirement and release-first H; open then reads the manifest under H, outside SQL
Release(S,H)	Closed H, bound to the original S	Idempotent; never reopen H, including when Release precedes Pin
Inspect H	Existing H state	Read-only recovery of the original responsibility; do not manufacture a replacement Pin
Delete(S)	Retired S	Not a direct artifact delete; existing H retries and valid warm reuse remain protected
Release retired heads	Delete S's heads	Lock/arbitrate S, confirm retirement and no active H, then commit; repeating the operation is safe
GC candidate check	Current heads, valid children, layer/physical-target state	A mark is only a candidate; arbitrate with new references, then authorize exact-target deletion before storage I/O

Integration boundary	Responsibility
Manager / business entry points	Admission, build/publication behavior, and local reuse
Publication / open orchestration	Carry completed descriptors, reused UUIDs/heads, source provenance and holds; prepare or read manifests outside SQL
Catalog / SQL adapter	Identity/state arbitration and multi-row transactions
Repository / Resolver	Storage I/O and protected materialization; do not move snapshot/volume business rules into the storage backend
Node hold owner	Durable handoff, local sharing, and final Release after actual responsibilities end

An unknown publication commit retains the original source responsibility and artifacts for reconciliation. It does not authorize compensating physical deletion. Async startup attachment commits its new description reference and protection together; old consumers keep their fixed inputs. Detailed hold/cache rules are in #362, reclamation in #364, and rollout gates in #365.

6. Coverage boundaries to finish before enabling GC

Area	What these tables already express	What still needs a concrete mapping
Embedded volume snapshots	Their layer chains belong in the containing snapshot's heads	Preserve their roles/mounts in the manifest
Mutable standalone volume	May reuse layers and the same retention protocol	How its current backing and still-used old backing map to roots/heads/pins; do not just substitute mutable volume_id for fixed content
Volume reservations	Existing exclusive/read-only admission remains	Reservation is not a substitute for content protection; no new version ID is mandated here
VM state and startup artifacts	Complete descriptors remain in the manifest	Ownership, exact deletion targets and durable cleanup progress; heads alone do not cover them
AENV-owned registry publications	Provenance and required retention roots remain explicit	Their retention/deletion coordination; do not invent a layer parent for them
Local files	Original process guard and protected remote ancestors	No cross-process recovery from SQL metadata alone
Terminal layers and exact storage targets	Lifecycle state and current-reference checks	UUID/key reuse, late DELETE isolation, terminal retention and compatible FK handling, under #364

This table is a coverage boundary, not permission to omit these dependencies or to call the four-table sketch a complete universal resource schema. Existing consumers and legacy files remain protected during the migration described in #363; schema readiness alone does not enable GC.

Acceptance criteria

• Each table, column and index above has an exercised purpose; no full snapshot-to-all-layer mapping is written for a reused prefix.
• Real PostgreSQL and SQLite transactions reproduce the example, plus publication/retirement, Pin/Release and reference/GC races.
• Codecs preserve existing business fields and complete runtime semantics; no in-place rewrite of legacy shared files or silent UUID/predecessor rebinding.
• Validate both low-reuse independent chains and shared-prefix workloads, including SQL writes, lock waits, table/index size and reclamation progress.
• Resolve volume/non-layer coverage and deletion-reuse rules before claiming complete protection or enabling reclamation of existing storage.

Contribution

I can provide more design details. Implementation and validation ownership remain to be discussed.

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @guozy18 on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/361 ## Affected component Snapshot Catalog schema, shared layers, and database transaction interfaces. Related: [Overview #360](https://github.com/kvcache-ai/AgentENV/issues/360) · [Usage holds #362](https://github.com/kvcache-ai/AgentENV/issues/362) · [Migration and field mapping #363](https://github.com/kvcache-ai/AgentENV/issues/363) · [Online GC #364](https://github.com/kvcache-ai/AgentENV/issues/364) · [Upgrades #365](https://github.com/kvcache-ai/AgentENV/issues/365). ## Problem statement A full snapshot-to-layer mapping repeats inherited relationships on every publication. Store each sealed layer's existing UUID and fixed predecessor once; let snapshots retain only their chain heads. The tables below make that proposal concrete without introducing a second layer identity. ## 1. Before and after | Information | Full mapping proposal | Prefix proposal | |---|---|---| | Snapshot description | Complete runtime manifest in SQL | `manifest_ref` in SQL; immutable complete manifest in storage | | Layer dependencies | Every snapshot records every layer | Each layer records one predecessor; each snapshot records heads | | Usage protection | Shared snapshot hold H | Preserve H and its existing lifecycle | | Publication of a known prefix plus a suffix | Insert all snapshot-layer relationships | Insert new layer rows and head relationships; reuse existing ancestors | ```text snapshots S0 ── snapshot_heads ──> B ──parent_uuid──> A snapshots S1 ── snapshot_heads ──> C ──parent_uuid──> B snapshot_pins H0 ───────────────> S0 ``` S is a snapshot identity, H is a usage-hold identity, and A/B/C are the existing file UUIDs after their contract is unified. They are not new database-only layer IDs. ## 2. Candidate table definitions These four tables describe the **snapshot-side layer-retention core**, not a claim that four tables already cover the entire snapshot/volume system. Types below use PostgreSQL notation. SQLite uses canonical UUID text, validated JSON text, and equivalent transactional constraints. ### `snapshots`: public identity and content entry | Column | Type / constraint | Why it is needed; what removal would break | |---|---|---| | `snapshot_id` | `UUID PRIMARY KEY`; immutable | Existing public identity and the target of heads/pins; retries must address the same snapshot | | `alias` | `TEXT NULL`; unique when present, nonblank, case-sensitive | Existing name lookup and permanent name reservation; a retired name must not silently resolve to another S | | `snapshot_type` | `TEXT NOT NULL`; `none`, `local`, or `distributed` | Distinguishes an unpublished template, process-owned Local input, and repository-resolvable Distributed input | | `manifest_ref` | `TEXT NULL`; exact immutable manifest locator | Locates the complete fixed runtime description without storing it all in SQL; NULL before publication or after permitted terminal cleanup | | `metadata` | `JSONB NULL`; object when present, validated by the business codec | Preserves source/build/requested-resource information that may exist before a runnable manifest exists; see the field placement table below | | `retired` | `BOOLEAN NOT NULL DEFAULT FALSE`; cannot revert | Rejects first admission of a new hold while preserving established users; deleting the row immediately would lose this distinction | | `created_at` | `BIGINT NOT NULL`; immutable UTC epoch milliseconds | Preserves existing ordering, pagination, and original creation time; it is not the identity-admission timestamp used by F | | Snapshot shape | Required state | |---|---| | Unpublished template | `snapshot_type=none`, no runnable `manifest_ref`, no published heads; requested resources/build state remain in metadata | | Published Local | Manifest identifies publishing node and relative private inputs; opening also requires the original process's file guard; no public alias | | Published Distributed | Complete fixed manifest plus all required heads and other retention responsibilities are committed | | Retired, still in use | Keep the description and protection required by existing H; Get/List hide the retired snapshot | | Terminal | Payload/reference cleanup follows #364; retain the business content kind and any required identity/name record | Promotion may change S's `manifest_ref`, but it keeps old readers' fixed inputs and the union of old/new heads for as long as their protection requires. An immutable manifest is never overwritten in place. ### `layers`: one row per sealed UUID | Column | Type / constraint | Why it is needed; what removal would break | |---|---|---| | `uuid` | `UUID PRIMARY KEY`; non-nil, immutable | Reuses the layer identity already carried by the file and runtime; removing it would require another identity or conflate distinct artifacts | | `parent_uuid` | `UUID NULL`; immutable reference to the predecessor; no cascading deletion | Encodes the shared prefix; without it every snapshot would have to enumerate the full chain again | | `digest` | `TEXT NOT NULL`; nonempty, immutable for the canonical published representation | Identifies the exact final bytes for managed storage and P2P; UUID alone is not proof of byte equality | | `state` | `TEXT NOT NULL`; `available`, `deleting`, `deleted` lifecycle | Arbitrates new references against reclamation and preserves deletion progress across failures; a zero-reference query alone cannot authorize OSS DELETE | | Invariant | Enforcement / boundary | |---|---| | One UUID has one predecessor and canonical published descriptor | Producer, file metadata, runtime descriptor, and SQL must agree; reject conflicts instead of rebinding an existing row | | The graph is a forest | New layers refer to an existing valid predecessor or have no predecessor; immutable creation rules must prevent cycles, not just a foreign key | | Identity reuse means artifact reuse | Fork, relocation, conversion-cache reuse, and publication retry preserve a sealed artifact's UUID; new content or a different predecessor requires a new artifact | | Digest includes embedded identity | Different file UUIDs do not generally produce the same full-file digest; there is no `(digest,parent_uuid)` rule that merges different embedded UUIDs | | Private and shared representations differ | A private sparse file must not be advertised as the canonical shared bytes merely because its UUID matches | | Deletion targets are scoped | This four-column sketch assumes a configured managed storage namespace where digest resolves the target. Other domains/imported locations require exact locator handling, not deletion by digest in an arbitrary namespace | | Terminal edges and rows | Active predecessor references must remain valid. Retaining a terminal child while pruning its parent affects foreign-key layout; finalize that together with the terminal retention/reuse policy in #364 | The existing OCI, memory, compaction, and publication paths must first satisfy this UUID/predecessor contract. The exact OCI UUID-generation algorithm remains open. Producers/transforms should return the final digest/size descriptor to avoid redundant digest scans; known-digest P2P reads must validate that exact content. ### `snapshot_heads`: retain chain tops, not every layer | Column | Type / constraint | Why it is needed; what removal would break | |---|---|---| | `snapshot_id` | `UUID NOT NULL`; references `snapshots.snapshot_id` | Identifies which snapshot retains the chain | | `layer_uuid` | `UUID NOT NULL`; references `layers.uuid` | Identifies the top from which all predecessors remain protected | Primary key: **`(snapshot_id, layer_uuid)`**. Do not cascade-delete these rows on public Delete; retirement must wait for active use. Role, order, and duplicate uses belong in the manifest. If rootfs and an attached drive both use `[A,B]`, one `(S,B)` head is sufficient for retention. ### `snapshot_pins`: shared usage responsibility | Column | Type / constraint | Why it is needed; what removal would break | |---|---|---| | `pin_id` | `UUID PRIMARY KEY`; immutable H | Identifies one shared use period and its retries; a sandbox ID is not an equivalent identity | | `snapshot_id` | `UUID NOT NULL`; immutable H-to-S binding | Identifies the snapshot protected by H and prevents a retry from changing its target | | `released` | `BOOLEAN NOT NULL DEFAULT FALSE`; false to true only | Distinguishes active protection from a Closed H; deleting H immediately would allow a delayed Pin to reopen it | Do **not** require a parent foreign key from this table to `snapshots`: Release may arrive before S or before Pin. First active Pin validates S transactionally; a release-first Closed H cannot later become active. Multiple prepared inputs/VMs/children can share H through local ownership; this is not one row per VM. ### Required access paths | Table | Key / index | Query it supports | |---|---|---| | `snapshots` | PK on `snapshot_id`; unique non-NULL `alias` | Identity/name arbitration | | `snapshots` | `(created_at DESC, snapshot_id ASC)` | Existing ordered List pagination | | `snapshots` | Retirement scan by snapshot ID, optionally a partial index | Find pending retirement work without blocking first-page candidates forever | | `layers` | PK on `uuid`; index on `parent_uuid` | Follow parents and test for direct retaining children | | `layers` | Index on `digest`; deletion-state scan | Check exact managed-file references and recover pending deletion | | `snapshot_heads` | PK `(snapshot_id,layer_uuid)`; reverse `(layer_uuid,snapshot_id)` | Enumerate a snapshot's heads and test whether a layer is a head | | `snapshot_pins` | Active index on `snapshot_id WHERE NOT released` | Decide whether retirement can release a snapshot's heads | No persisted `ref_count` is required: indexed head/valid-child existence checks are sufficient for logical layer retention. No separate `heads_released` flag is needed just to make head deletion idempotent. **Zero heads does not prove that non-layer artifacts or external deletion work have finished.** Any separate all-obligations-complete fact must cover those responsibilities before payload cleanup. ## 3. Field placement: SQL versus immutable manifest | Existing information | Placement | Preservation rule | |---|---|---| | Identity, alias, type, creation time, retirement | Dedicated `snapshots` columns | Needed for lookup, listing, admission, and retirement | | Source/provenance business fields, build status/error, original update time | `snapshots.metadata` | Preserve the existing business projection; a late build error cannot overwrite a published or retired result | | Requested resources before publication | `snapshots.metadata` | An unpublished template has no completed runtime manifest | | Actual resources after publication | Immutable manifest | Preserve actual runtime values separately from the build request | | Backend, runtime/startup contexts, versions, source image configs, extensions | Immutable manifest | Complete codec validation; missing backend compatibility must not accept an unknown backend | | Ordered rootfs/memory/drive/volume-snapshot layers | Immutable manifest; their unique chain heads in SQL | Preserve roles, drive IDs, read-only flags, virtual sizes, mounts and subpaths without duplicating every layer mapping | | Exact artifact descriptors and locations | Immutable manifest and required deletion/retention responsibility | Preserve digest/size/format/storage domain; a locator alone does not supply a GC protocol | | Local publishing node and private-file relative locations | Local description plus original process guard | A SQL row cannot recover a lost Local snapshot in another process | | Node-local absolute paths, byte-cache state, Ready entries | Node-owned state | Do not make process-local ownership shared SQL truth | The detailed old-to-new field mapping remains in [#363](https://github.com/kvcache-ai/AgentENV/issues/363). Build logs retain their existing ownership. ## 4. Concrete records: shared prefix, fork, and in-use deletion All IDs and digests below are shortened examples. `mf0`/`mf1`/`mf2` are exact references to different immutable manifests, not mutable aliases. ### Publish S0 ```text mf0: rootfs=[A,B], memory=[M], runtime configuration and artifact descriptors B -> A M has no predecessor ``` **`snapshots`** | snapshot_id | alias | snapshot_type | manifest_ref | metadata | retired | created_at | |---|---|---|---|---|---|---| | S0 | python-base | distributed | mf0 | existing business metadata | false | t0 | **`layers`** | uuid | parent_uuid | digest | state | |---|---|---|---| | A | NULL | dA | available | | B | A | dB | available | | M | NULL | dM | available | **`snapshot_heads`** | snapshot_id | layer_uuid | |---|---| | S0 | B | | S0 | M | **`snapshot_pins`**: empty. Publishing S0 does not itself create a usage pin. ### Subsequent operations: exact row changes Suppose S1 adds C to the same rootfs prefix, and a fork from S0 later publishes S2 with D: ```text mf1: rootfs=[A,B,C], memory=[M] mf2: rootfs=[A,B,D], memory=[M] ``` | Operation | `snapshots` | `layers` | `snapshot_heads` | `snapshot_pins` | |---|---|---|---|---| | Cold-open S0 | Read/fix mf0 | No change | No change | Insert `(H0,S0,false)` before fetching mf0 or artifacts | | Publish S1 | Insert S1 with mf1, not retired | Insert `(C,B,dC,available)` only | Insert `(S1,C)` and `(S1,M)` | Keep source H0 while still needed | | Fork a child from S0 | No change | No shared row merely for fork | No change | Child inherits the required local share of H0; no automatic new H | | Publish child as S2 | Insert S2 with mf2, not retired | Insert `(D,B,dD,available)` only | Insert `(S2,D)` and `(S2,M)` | Retain H0 while source reads continue | | Delete S0 while H0 is active | Set `S0.retired=true` | No change | **Keep `(S0,B)` and `(S0,M)`** | H0 stays active; valid established use continues | | Last H0 responsibility ends | No change | No change | No change yet | Set `H0.released=true` | | GC releases S0 heads | Keep retired S0 and its alias reservation | **Keep A/B/C/D/M** | Remove `(S0,B)` and `(S0,M)` | Keep Closed H0 until eligible history cleanup | | S1 later retires, has no H, and its heads are released | Mark S1 retired | C becomes a candidate; B/A/M remain retained by S2 | Remove `(S1,C)` and `(S1,M)` | No active S1 hold | After releasing S0, the surviving references are: ```text S1 -> C -> B -> A S2 -> D -> B -> A S1/S2 -> M ``` C can enter deletion only after current checks and durable authorization succeed; deleting its OSS bytes is a separate step. This example does not choose permanent layer tombstones or permission to reuse deleted UUIDs/keys; see #364. ### Relationship-write comparison | Case | Full snapshot-layer mapping | Prefix tables | |---|---|---| | Known k-layer input plus delta new layers | Repeat relationships for the resulting full layer set | Register delta new layers and h unique heads; existing ancestors stay unchanged | | First publication of an unfamiliar chain | Register its layer set and mapping | Register the unfamiliar chain and its heads; no prefix saving is assumed | | Many snapshots inherit A/B | Repeat A/B mappings for each snapshot | Keep A/B once in `layers`; each snapshot retains its own heads | | Cost moved elsewhere | Direct full-set enumeration | Parent indexes, graph traversal, and branch reclamation | The intended database relationship-write bound is **O(delta + h)** for known prefixes. Full manifest serialization, uploads, digest work, and initial imports are measured separately; this is not a claim that the entire publish operation is O(delta). ## 5. Transaction and API boundaries | Entry point | Rows / result | Required atomicity or check | |---|---|---| | Get / List | Snapshot metadata and manifest reference | Grants no right to consume artifacts; hide retired rows | | Publish | S, new layers, heads, and manifest_ref | Prepare bytes/manifest outside SQL; validate reused prefix identities and atomically expose the completed result | | Pin(S,H) | Active H plus fixed protected metadata/manifest_ref | Arbitrate against retirement and release-first H; open then reads the manifest under H, outside SQL | | Release(S,H) | Closed H, bound to the original S | Idempotent; never reopen H, including when Release precedes Pin | | Inspect H | Existing H state | Read-only recovery of the original responsibility; do not manufacture a replacement Pin | | Delete(S) | Retired S | Not a direct artifact delete; existing H retries and valid warm reuse remain protected | | Release retired heads | Delete S's heads | Lock/arbitrate S, confirm retirement and no active H, then commit; repeating the operation is safe | | GC candidate check | Current heads, valid children, layer/physical-target state | A mark is only a candidate; arbitrate with new references, then authorize exact-target deletion before storage I/O | | Integration boundary | Responsibility | |---|---| | Manager / business entry points | Admission, build/publication behavior, and local reuse | | Publication / open orchestration | Carry completed descriptors, reused UUIDs/heads, source provenance and holds; prepare or read manifests outside SQL | | Catalog / SQL adapter | Identity/state arbitration and multi-row transactions | | Repository / Resolver | Storage I/O and protected materialization; do not move snapshot/volume business rules into the storage backend | | Node hold owner | Durable handoff, local sharing, and final Release after actual responsibilities end | An unknown publication commit retains the original source responsibility and artifacts for reconciliation. It does not authorize compensating physical deletion. Async startup attachment commits its new description reference and protection together; old consumers keep their fixed inputs. Detailed hold/cache rules are in #362, reclamation in #364, and rollout gates in #365. ## 6. Coverage boundaries to finish before enabling GC | Area | What these tables already express | What still needs a concrete mapping | |---|---|---| | Embedded volume snapshots | Their layer chains belong in the containing snapshot's heads | Preserve their roles/mounts in the manifest | | Mutable standalone volume | May reuse `layers` and the same retention protocol | How its current backing and still-used old backing map to roots/heads/pins; do not just substitute mutable volume_id for fixed content | | Volume reservations | Existing exclusive/read-only admission remains | Reservation is not a substitute for content protection; no new version ID is mandated here | | VM state and startup artifacts | Complete descriptors remain in the manifest | Ownership, exact deletion targets and durable cleanup progress; heads alone do not cover them | | AENV-owned registry publications | Provenance and required retention roots remain explicit | Their retention/deletion coordination; do not invent a layer parent for them | | Local files | Original process guard and protected remote ancestors | No cross-process recovery from SQL metadata alone | | Terminal layers and exact storage targets | Lifecycle state and current-reference checks | UUID/key reuse, late DELETE isolation, terminal retention and compatible FK handling, under #364 | This table is a coverage boundary, not permission to omit these dependencies or to call the four-table sketch a complete universal resource schema. Existing consumers and legacy files remain protected during the migration described in #363; schema readiness alone does not enable GC. ## Acceptance criteria - [ ] Each table, column and index above has an exercised purpose; no full snapshot-to-all-layer mapping is written for a reused prefix. - [ ] Real PostgreSQL and SQLite transactions reproduce the example, plus publication/retirement, Pin/Release and reference/GC races. - [ ] Codecs preserve existing business fields and complete runtime semantics; no in-place rewrite of legacy shared files or silent UUID/predecessor rebinding. - [ ] Validate both low-reuse independent chains and shared-prefix workloads, including SQL writes, lock waits, table/index size and reclamation progress. - [ ] Resolve volume/non-layer coverage and deletion-reuse rules before claiming complete protection or enabling reclamation of existing storage. ## Contribution I can provide more design details. Implementation and validation ownership remain to be discussed. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:26 +00:00

bsakaguchi referenced this issue 2026-10-29 14:20:37 +00:00

[PR #88] [MERGED] fix(snapshot): export s3-marked OSS-managed rootfs layers as images #171

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#88

No description provided.