[GH-ISSUE #360] [Design Proposal] SQL-backed snapshot catalog and resource lifecycle #90

New issue

Open

opened 2026-10-29 14:19:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented 2026-10-29 14:19:26 +00:00

Owner

Copy link

Originally created by @guozy18 on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/360

Affected component

Cross-cutting / other: snapshot metadata, shared layer dependencies, snapshot/volume content retention, sandbox resource ownership, and artifact reclamation.

Problem statement

At high snapshot churn, the system needs a consistent answer to four questions: what is required to run a snapshot, which artifacts other snapshots still reference, which consumers are still using them, and when their bytes may be deleted.

Public records, aliases, shared artifact storage, runtime inputs, and local files have different coordination and lifetime boundaries. Moving metadata to SQL should establish a transactional boundary for publication and resource protection. It should not add a database request to every VM operation or retain every completed use and full historical manifest indefinitely.

Use case

Training and evaluation workloads create many independent snapshots with potentially little reuse. A typical flow creates a sandbox, executes work, captures a result, forks or pauses/resumes, and eventually deletes the sandbox and snapshot.

Publication, startup, and reclamation must continue while users are active. Capacity validation must include a workload where successive starts use different snapshots; a high hit rate on a shared template is not an adequate baseline.

Current behavior and workarounds

OSS/POSIX repositories store public records, alias bindings, runtime manifests, and artifact bytes. The resolver materializes runtime inputs, and local guards or leases retain the associated files and work. Local caches reduce byte reads, but do not themselves form a shared ledger of snapshot dependencies and active consumers.

Publication, lookup, local ownership, and physical cleanup therefore have different coordination boundaries. Independent directory scans and local references do not provide a single transaction that commits a public result together with every managed dependency that must remain available.

Desired behavior

• Atomically publish a fixed content description reference, its chain heads, newly registered layers, and business visibility, without expanding every snapshot into a full snapshot-to-layer mapping.
• Retire a snapshot without interrupting consumers that already hold it.
• Reclaim an artifact only after all protecting snapshot, volume, and actual-use relationships have ended.
• Keep valid prepared-input reuse, fork, same-process pause/resume, and ordinary exit on local ownership paths, without an additional Catalog request per operation.
• Clear terminal payloads and trim eligible closed-use history without expiring active consumers.
• Keep existing consumers protected on their original artifact bytes during migration and upgrades. Preserve runtime and volume ownership semantics; incompatible legacy layer identities require an explicit compatibility path rather than in-place file changes.

Proposed approach

Architecture before and after

flowchart LR
  subgraph BEFORE[Before]
    B1["Manager / Repository"] --> B2["OSS / POSIX records and runtime metadata"]
    B1 --> B3["Artifact bytes"]
    B2 --> B4["Resolver and local ownership"]
    B3 --> B4
  end
  subgraph AFTER[After]
    A1["Manager / Repository"] --> A2["In-process Catalog library"]
    A2 --> A3[("SQL content entries, chain heads, layer dependencies, and holds")]
    A1 --> A4["ArtifactStore"]
    A3 --> A5["Protected runtime inputs"]
    A4 --> A5
  end

Catalog is an in-process Rust library, backed by PostgreSQL for a cluster and SQLite for a single-node deployment. It records fixed content, immutable layer dependencies, chain heads, and shared usage holds. These are required facts, not a predetermined table count. Snapshot S and shared usage period H retain their business and lifecycle semantics.

SQL stores each layer UUID's predecessor once and records only the heads retained by each content entry. Walking predecessors reconstructs dependencies without rereading all OSS manifests on every GC round. For snapshots [A,B,C] and [A,B,D] that inherit the same A/B artifacts:

B -> A
C -> B
D -> B
S1 -> C
S2 -> D

An unfamiliar independent chain still requires full initial registration. Prefix reuse comes from inheriting the same sealed artifacts or reusing a valid conversion result; it is not guaranteed for independent OCI cold conversions.

Reuse and unify AENV's existing file UUID rather than adding another prefix UUID. A sealed UUID must have fixed content and predecessor. OCI conversion, memory, compaction, and publication transformations must satisfy that contract. The digest identifies the final published bytes; because the UUID is embedded in the file, distinct UUIDs must not be assumed to share a full-file digest. Producers and transformations should return the final digest/size descriptor for upload and P2P reuse, avoiding redundant digest scans. Known-digest reads must not accept an unrelated UUID match.

The complete immutable runtime manifest may remain in OSS, with manifest_ref and the necessary query, admission, dependency, and lifecycle information in SQL. Snapshot and volume share content-retention mechanisms while keeping their business records and admission rules. A mutable volume ID is not an immutable backing identity: a backing switch must preserve old users' fixed inputs. The precise table representation remains to be specified in #361.

Layer chains do not replace retention rules for VM state, startup artifacts, or registry publication roots. Asynchronous startup attachments need an explicit description/dependency handoff boundary rather than an in-place mutation of an allegedly immutable main manifest.

An H may be shared by a prepared entry, multiple VMs, captured artifacts, and paused state. It is neither one central row per VM nor a time-expiring runtime lease. Actual bytes remain in object storage, shared filesystems, registries, and local storage.

Operation	Before	After
Publish	Repository writes artifacts, records, and aliases	Complete artifacts and manifest first; atomically commit manifest_ref, new layers, heads, and business visibility
Create from a snapshot	Read the record and prepare runtime inputs	Acquire H and fix manifest_ref before reading/materializing artifacts; reuse valid local inputs and H on a hit
Fork / pause / resume	Transfer local artifacts and execution state	Transfer the same ownership together with the required H; inspect existing H when rebuilding durable ownership
Exit	Drain runtime resources and release local ownership	Release H asynchronously after its last actual responsibility ends
Delete a snapshot	Repository handles record and artifact cleanup	Retire S, wait for its H, release heads, then reclaim eligible layers and globally unprotected physical targets

Retirement does not immediately revoke reuse of an already protected prepared entry. Aliases remain bound to their original identity. Whether a deleted UUID or physical key may ever be reused remains a design decision in #364: either account for terminal-record retention, or establish isolation from delayed DELETE requests before enabling reuse-dependent GC.

Design workstreams

Proposal	Responsibility
Catalog schema and API	SQL tables, constraints, transaction ownership, and internal Rust interfaces
Sandbox lifecycle and shared holds	End-to-end ownership transfer and database access boundaries
Data migration before and after	Directory and field mapping, concrete records, and initial OSS/POSIX-to-SQL cutover
Online GC and historical retention	Online marking and reclamation over layer dependencies, physical-delete arbitration, terminal payload cleanup, and eligible S/H history
Database upgrades and mixed-version operation	Schema migrations and V1/V2 coexistence after SQL is already the backend

Initial data conversion and later SQL/application upgrades are separate workflows. A background tool can convert old records gradually, while a compatibility repository coordinates per-snapshot handover and retains existing consumers. Subsequent releases use schema expansion, a period of common-format operation, explicit activation of new data formats, and eventual contraction.

Smallest verifiable implementation sequence

1. Establish the unified layer UUID/predecessor and final-artifact descriptor contracts, including known-digest P2P reads.
2. Implement the complete codec and Catalog transactions against real PostgreSQL and SQLite instances.
3. Connect artifact publication to SQL lookup and remove direct physical-delete compensation from the SQL path.
4. Connect durable local holds through create, fork, capture, pause/recovery, and exit.
5. Validate online GC in an isolated repository, then terminal payload cleanup and identity-history trimming.
6. Validate incremental import and existing-consumer handover before switching stored production data.
7. Exercise a mixed-version upgrade and its rollback boundary before enabling incompatible data shapes.

Each step has an observable result; this does not make partially connected production consumers safe. Implementation tasks should follow these end-to-end boundaries rather than splitting solely by table.

Acceptance criteria

• A published snapshot reconstructs the same logical runtime input, including all managed dependencies and required registry roots.
• Shared layers and non-layer artifacts remain readable until the last protecting relationship ends; final Release occurs after devices and I/O finish. Include cross-snapshot/volume sharing and backing changes.
• Valid hot operations add no Catalog or central metadata request.
• Low-reuse traffic, peak load, and retained history are tested for latency, lock waits, table/index size, and reclamation progress.
• Relationship writes scale with newly registered layers delta plus head count h, rather than total layers k per snapshot. Full manifest serialization and byte processing are measured separately; prefix reuse must not rewrite every ancestor.
• Initial migration preserves continuous service under concurrent publish, delete, build updates, and recovery.
• V1/V2 coexistence, new-format activation, and rollback are verified using actual binaries and database transactions.

Compatibility and operational impact

Database connection configuration, a complete codec, and compatibility checks become part of Catalog setup. New cold preparation, publication, and metadata queries require the database. Existing valid local inputs continue to work during a database interruption; pending releases remain durable.

The complete runtime description must preserve existing backend, virtualization, startup, drive, volume, and extension semantics. Asynchronous publication work must use the same identity arbitration and conditional updates as foreground operations. Additional artifact dependencies become visible atomically with their descriptions.

Initial import preserves existing snapshot/hold identities and exact artifact locations where compatible. Layers with missing, conflicting, or context-inconsistent UUIDs require explicit compatibility handling; existing consumers keep their original files, and normalization must not rewrite them in place. Once SQL owns a snapshot, it cannot fall back to an older file record. Later upgrades keep schema availability separate from the data formats that deployed binaries are allowed to produce and required to read.

Alternatives considered

Alternative	Tradeoff
Store every snapshot-to-layer mapping	Simple direct dependency enumeration, but repeated prefixes cause O(k) relationship writes per publication; compare with O(delta + h) new-layer/head writes and the graph traversal cost
Keep independent file catalogs and scans	Less migration work, but publication and in-use protection still need a shared coordination boundary
Register every VM or renew periodic leases	More central operations; elapsed time still does not prove paused state, devices, or I/O have ended
Retain all terminal history	Preserves complete historical results, but grows with cumulative publication and use
Pause users for migration or GC	Simpler coordination, but fails the continuous-service requirement

Non-goals

This proposal does not add a separate Catalog service, store VM/memory/disk bytes in SQL, redesign scheduling or volume mount/read-write ownership, replace the byte cache, or expand the CLI. Sharing content retention does not require a universal resource table or a second layer identity. It does not make runtime live migration a prerequisite for schema upgrades.

Contribution

I can provide more design details.

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @guozy18 on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/360 ## Affected component Cross-cutting / other: snapshot metadata, shared layer dependencies, snapshot/volume content retention, sandbox resource ownership, and artifact reclamation. ## Problem statement At high snapshot churn, the system needs a consistent answer to four questions: what is required to run a snapshot, which artifacts other snapshots still reference, which consumers are still using them, and when their bytes may be deleted. Public records, aliases, shared artifact storage, runtime inputs, and local files have different coordination and lifetime boundaries. Moving metadata to SQL should establish a transactional boundary for publication and resource protection. It should not add a database request to every VM operation or retain every completed use and full historical manifest indefinitely. ## Use case Training and evaluation workloads create many independent snapshots with potentially little reuse. A typical flow creates a sandbox, executes work, captures a result, forks or pauses/resumes, and eventually deletes the sandbox and snapshot. Publication, startup, and reclamation must continue while users are active. Capacity validation must include a workload where successive starts use different snapshots; a high hit rate on a shared template is not an adequate baseline. ## Current behavior and workarounds OSS/POSIX repositories store public records, alias bindings, runtime manifests, and artifact bytes. The resolver materializes runtime inputs, and local guards or leases retain the associated files and work. Local caches reduce byte reads, but do not themselves form a shared ledger of snapshot dependencies and active consumers. Publication, lookup, local ownership, and physical cleanup therefore have different coordination boundaries. Independent directory scans and local references do not provide a single transaction that commits a public result together with every managed dependency that must remain available. ## Desired behavior - Atomically publish a fixed content description reference, its chain heads, newly registered layers, and business visibility, without expanding every snapshot into a full snapshot-to-layer mapping. - Retire a snapshot without interrupting consumers that already hold it. - Reclaim an artifact only after all protecting snapshot, volume, and actual-use relationships have ended. - Keep valid prepared-input reuse, fork, same-process pause/resume, and ordinary exit on local ownership paths, without an additional Catalog request per operation. - Clear terminal payloads and trim eligible closed-use history without expiring active consumers. - Keep existing consumers protected on their original artifact bytes during migration and upgrades. Preserve runtime and volume ownership semantics; incompatible legacy layer identities require an explicit compatibility path rather than in-place file changes. ## Proposed approach ### Architecture before and after ```mermaid flowchart LR subgraph BEFORE[Before] B1["Manager / Repository"] --> B2["OSS / POSIX records and runtime metadata"] B1 --> B3["Artifact bytes"] B2 --> B4["Resolver and local ownership"] B3 --> B4 end subgraph AFTER[After] A1["Manager / Repository"] --> A2["In-process Catalog library"] A2 --> A3[("SQL content entries, chain heads, layer dependencies, and holds")] A1 --> A4["ArtifactStore"] A3 --> A5["Protected runtime inputs"] A4 --> A5 end ``` Catalog is an in-process Rust library, backed by PostgreSQL for a cluster and SQLite for a single-node deployment. It records fixed content, immutable layer dependencies, chain heads, and shared usage holds. These are required facts, not a predetermined table count. Snapshot **S** and shared usage period **H** retain their business and lifecycle semantics. SQL stores each layer UUID's predecessor once and records only the heads retained by each content entry. Walking predecessors reconstructs dependencies without rereading all OSS manifests on every GC round. For snapshots `[A,B,C]` and `[A,B,D]` that inherit the same A/B artifacts: ```text B -> A C -> B D -> B S1 -> C S2 -> D ``` An unfamiliar independent chain still requires full initial registration. Prefix reuse comes from inheriting the same sealed artifacts or reusing a valid conversion result; it is not guaranteed for independent OCI cold conversions. Reuse and unify AENV's existing file UUID rather than adding another prefix UUID. A sealed UUID must have fixed content and predecessor. OCI conversion, memory, compaction, and publication transformations must satisfy that contract. The digest identifies the final published bytes; because the UUID is embedded in the file, distinct UUIDs must not be assumed to share a full-file digest. Producers and transformations should return the final digest/size descriptor for upload and P2P reuse, avoiding redundant digest scans. Known-digest reads must not accept an unrelated UUID match. The complete immutable runtime manifest may remain in OSS, with `manifest_ref` and the necessary query, admission, dependency, and lifecycle information in SQL. Snapshot and volume share content-retention mechanisms while keeping their business records and admission rules. A mutable volume ID is not an immutable backing identity: a backing switch must preserve old users' fixed inputs. The precise table representation remains to be specified in #361. Layer chains do not replace retention rules for VM state, startup artifacts, or registry publication roots. Asynchronous startup attachments need an explicit description/dependency handoff boundary rather than an in-place mutation of an allegedly immutable main manifest. An H may be shared by a prepared entry, multiple VMs, captured artifacts, and paused state. It is neither one central row per VM nor a time-expiring runtime lease. Actual bytes remain in object storage, shared filesystems, registries, and local storage. | Operation | Before | After | |---|---|---| | Publish | Repository writes artifacts, records, and aliases | Complete artifacts and manifest first; atomically commit manifest_ref, new layers, heads, and business visibility | | Create from a snapshot | Read the record and prepare runtime inputs | Acquire H and fix manifest_ref before reading/materializing artifacts; reuse valid local inputs and H on a hit | | Fork / pause / resume | Transfer local artifacts and execution state | Transfer the same ownership together with the required H; inspect existing H when rebuilding durable ownership | | Exit | Drain runtime resources and release local ownership | Release H asynchronously after its last actual responsibility ends | | Delete a snapshot | Repository handles record and artifact cleanup | Retire S, wait for its H, release heads, then reclaim eligible layers and globally unprotected physical targets | Retirement does not immediately revoke reuse of an already protected prepared entry. Aliases remain bound to their original identity. Whether a deleted UUID or physical key may ever be reused remains a design decision in #364: either account for terminal-record retention, or establish isolation from delayed DELETE requests before enabling reuse-dependent GC. ### Design workstreams | Proposal | Responsibility | |---|---| | [Catalog schema and API](https://github.com/kvcache-ai/AgentENV/issues/361) | SQL tables, constraints, transaction ownership, and internal Rust interfaces | | [Sandbox lifecycle and shared holds](https://github.com/kvcache-ai/AgentENV/issues/362) | End-to-end ownership transfer and database access boundaries | | [Data migration before and after](https://github.com/kvcache-ai/AgentENV/issues/363) | Directory and field mapping, concrete records, and initial OSS/POSIX-to-SQL cutover | | [Online GC and historical retention](https://github.com/kvcache-ai/AgentENV/issues/364) | Online marking and reclamation over layer dependencies, physical-delete arbitration, terminal payload cleanup, and eligible S/H history | | [Database upgrades and mixed-version operation](https://github.com/kvcache-ai/AgentENV/issues/365) | Schema migrations and V1/V2 coexistence after SQL is already the backend | Initial data conversion and later SQL/application upgrades are separate workflows. A background tool can convert old records gradually, while a compatibility repository coordinates per-snapshot handover and retains existing consumers. Subsequent releases use schema expansion, a period of common-format operation, explicit activation of new data formats, and eventual contraction. ### Smallest verifiable implementation sequence 1. Establish the unified layer UUID/predecessor and final-artifact descriptor contracts, including known-digest P2P reads. 2. Implement the complete codec and Catalog transactions against real PostgreSQL and SQLite instances. 3. Connect artifact publication to SQL lookup and remove direct physical-delete compensation from the SQL path. 4. Connect durable local holds through create, fork, capture, pause/recovery, and exit. 5. Validate online GC in an isolated repository, then terminal payload cleanup and identity-history trimming. 6. Validate incremental import and existing-consumer handover before switching stored production data. 7. Exercise a mixed-version upgrade and its rollback boundary before enabling incompatible data shapes. Each step has an observable result; this does not make partially connected production consumers safe. Implementation tasks should follow these end-to-end boundaries rather than splitting solely by table. ### Acceptance criteria - [ ] A published snapshot reconstructs the same logical runtime input, including all managed dependencies and required registry roots. - [ ] Shared layers and non-layer artifacts remain readable until the last protecting relationship ends; final Release occurs after devices and I/O finish. Include cross-snapshot/volume sharing and backing changes. - [ ] Valid hot operations add no Catalog or central metadata request. - [ ] Low-reuse traffic, peak load, and retained history are tested for latency, lock waits, table/index size, and reclamation progress. - [ ] Relationship writes scale with newly registered layers delta plus head count h, rather than total layers k per snapshot. Full manifest serialization and byte processing are measured separately; prefix reuse must not rewrite every ancestor. - [ ] Initial migration preserves continuous service under concurrent publish, delete, build updates, and recovery. - [ ] V1/V2 coexistence, new-format activation, and rollback are verified using actual binaries and database transactions. ## Compatibility and operational impact Database connection configuration, a complete codec, and compatibility checks become part of Catalog setup. New cold preparation, publication, and metadata queries require the database. Existing valid local inputs continue to work during a database interruption; pending releases remain durable. The complete runtime description must preserve existing backend, virtualization, startup, drive, volume, and extension semantics. Asynchronous publication work must use the same identity arbitration and conditional updates as foreground operations. Additional artifact dependencies become visible atomically with their descriptions. Initial import preserves existing snapshot/hold identities and exact artifact locations where compatible. Layers with missing, conflicting, or context-inconsistent UUIDs require explicit compatibility handling; existing consumers keep their original files, and normalization must not rewrite them in place. Once SQL owns a snapshot, it cannot fall back to an older file record. Later upgrades keep schema availability separate from the data formats that deployed binaries are allowed to produce and required to read. ## Alternatives considered | Alternative | Tradeoff | |---|---| | Store every snapshot-to-layer mapping | Simple direct dependency enumeration, but repeated prefixes cause O(k) relationship writes per publication; compare with O(delta + h) new-layer/head writes and the graph traversal cost | | Keep independent file catalogs and scans | Less migration work, but publication and in-use protection still need a shared coordination boundary | | Register every VM or renew periodic leases | More central operations; elapsed time still does not prove paused state, devices, or I/O have ended | | Retain all terminal history | Preserves complete historical results, but grows with cumulative publication and use | | Pause users for migration or GC | Simpler coordination, but fails the continuous-service requirement | ## Non-goals This proposal does not add a separate Catalog service, store VM/memory/disk bytes in SQL, redesign scheduling or volume mount/read-write ownership, replace the byte cache, or expand the CLI. Sharing content retention does not require a universal resource table or a second layer identity. It does not make runtime live migration a prerequisite for schema upgrades. ## Contribution I can provide more design details. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:26 +00:00

bsakaguchi referenced this issue 2026-10-29 14:19:58 +00:00

[PR #27] [CLOSED] feat(scheduler): add projected least-loaded placement strategy #126

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#90

No description provided.