[GH-ISSUE #364] [Design Proposal] Online snapshot garbage collection and terminal history retention #93

New issue

Open

opened 2026-10-29 14:19:29 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented 2026-10-29 14:19:29 +00:00

Owner

Copy link

Originally created by @guozy18 on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/364

Affected component

Snapshot / template builder / shared snapshot-volume content retention

Problem statement

Part of the parent proposal, depending on the data model and API and Sandbox resource holds. This issue covers online reclamation over layer dependencies, physical artifact deletion, and terminal history. Activation during version coexistence is covered by the database upgrade proposal.

Independent snapshots continuously create new identities; a preparation cache cannot eliminate most first registrations in a low-reuse workload. Keeping every terminal snapshot payload/reference and Closed H forever would make SQL history grow with cumulative publications and uses. Deleting rows based only on age would let delayed Publish/Pin requests recreate identities that have already ended.

We need to reclaim unused bytes and safely removable history without letting deletion overtake legitimate consumers.

Use case

Snapshot creation, restore, and publication continue while background cleanup runs. Long-running VMs, paused sandboxes, uploads, and device I/O must retain protection regardless of the cleanup window. Maintenance must not require users to pause these operations.

Capacity planning must include many independent snapshots with little reuse, not assume that a few popular templates dominate reads.

Current behavior and workarounds

The Repository manages snapshot records and artifacts in OSS/POSIX, while node-local files and caches have their own lifecycles. Snapshot deletion removes catalog entries and snapshot-specific artifacts; shared layers require separate cleanup.

Repository metadata, local references, and cache state each describe part of resource lifetime. SQL adds an explicit dependency graph and shared usage records so online reclamation can account for all three. Completed usage periods and retired snapshots then become database history that also needs a safe retention policy.

Desired behavior

• A retired S retains every managed dependency while any actual hold remains active.
• A physical DELETE is issued only after its exact target has no global protecting references and its deletion eligibility has committed.
• A terminal S can discard its manifest reference and unnecessary payload after the content and non-layer dependency handoffs are complete. Eligible Closed H rows and terminal S rows without aliases can later be pruned in batches.
• Active H does not expire by age. Cleanup does not require pausing user operations.
• Retries, lost replies, concurrent new references, and database failures do not let deletion overtake legitimate consumers.

Proposed approach

S identifies a public snapshot and H one shared usage period. The data model defines fixed-content entries, heads, immutable layer UUID/predecessor relationships, and their constraints. Volume content participates in the same retention mechanism without treating its mutable business ID as a fixed backing version.

Mark candidates from SQL; decide deletion from current references

Use mark-and-sweep over the database's protected content entries, heads, and predecessor relationships. A GC round need not fetch every OSS manifest. Marks identify candidates, not permission to delete: publication and acquisition may continue during marking, and sweep must recheck current references under the same arbitration used to establish new references.

Each layer has at most one immutable predecessor. A content head or a valid successor protects that layer, which in turn protects its predecessor and required bytes. Reclaiming an unreachable branch therefore removes eligible leaves before their predecessors. Retired content remains a root until its actual usage responsibilities have ended. Existing H retries and valid warm reuse continue under the established protection. Predecessor protection may end only after new references are atomically fenced, no actual reader still needs the layer, and remaining deletion work does not need to read its predecessor. State predicates and foreign-key handling must distinguish this released dependency from a retained terminal audit record; the latter must not protect predecessors forever.

Keep retirement, head release, layer reclamation, and physical deletion separate

Stage	Data change	Preconditions
Retire content	Reject first admission of a new H/cold acquisition; retain its fixed description, heads, and active holds	Arbitrate retirement against publication and acquisition; preserve already established use
Release heads	Remove the retired content's head relationships and record completion	In one transaction, confirm retirement and no remaining use; preserve required non-layer cleanup responsibilities
Reclaim a logical layer	Change the layer from referenceable to reclamation state; release its predecessor protection as appropriate	Recheck that no content head or valid successor references it, using arbitration shared with concurrent attachment
Authorize physical deletion	Persist deletion eligibility for the exact physical target	Confirm no globally protecting reference remains, including another logical layer or content owner; the physical-target arbitration must cover concurrent publication
Delete bytes	Delete the exact target and persist confirmed completion	The authorization commit is confirmed; storage I/O runs outside SQL transactions

For example:

S1 -> C -> B -> A
S2 -> D -> B -> A

Retiring S1 leaves C/B/A protected while an H remains. After final Release, S1's head can be removed and C reclaimed. S2 continues to protect D/B/A. A snapshot and a volume can protect the same prefix in exactly the same way. When a volume switches backing, old users still protect their old heads; the current volume pointer alone is not a sufficient GC root inventory.

If publication wins reference arbitration, GC must retain the layer. If reclamation has already fenced the layer against new references, publication must follow the defined conflict path. This arbitration concerns the relevant identities and physical targets; it does not pause all new snapshot use or rewrite every ancestor of a reused prefix.

// Illustrative lifecycle boundaries, not a finalized API or table layout.
catalog.release_heads_if_retired_and_unpinned(s1).await?;
catalog.reclaim_layer_if_unreferenced(c).await?;
if let Some(target) = catalog.authorize_physical_delete(c).await? {
    // Return a target only after COMMIT is confirmed, not merely UPDATE RETURNING.
    artifacts.delete_exact(&target).await?;
    catalog.confirm_deleted(&target).await?;
}

A count of zero followed by an uncoordinated OSS DELETE is unsafe. If a commit result is unknown, reconcile the original operation and exact target before issuing deletion. Failed or uncertain physical deletion leaves persistent deleting state and retries the same target. Duplicate workers must be safe; external DELETE need not execute exactly once. An absent target can confirm completion only under a protocol that also accounts for delayed requests and target reuse.

VM state, startup artifacts, and registry publication retention roots remain part of the protected inventory. They must not be omitted or given artificial layer parents simply to fit the forest model.

Distinguish progress, safety state, and candidate discovery

Fact	Responsibility
Retirement versus completed head release	Durable progress for pending-work scans and payload cleanup; head release alone does not establish physical deletion safety
Layer and exact-target deletion state	Fence new references and recover uncertain external deletion without losing the exact target
Mark results or candidate hints	Identify work to revisit; a stale mark or hint is not deletion permission

These responsibilities do not mandate the former refs_released or gc_pending field layout. If a persistent hint is used, clearing it must arbitrate with later reference removal so a concurrent clear cannot lose work. Reference relations remain the source of truth; a persisted ref_count is not a prerequisite for this design.

Resolve identity and key reuse before activating dependent GC

Whether a deleted layer UUID or physical key can be reused is an explicit design choice, not settled by this proposal. A no-resurrection policy must state which terminal records are retained and account for their storage cost. A reuse policy must isolate the new artifact from a delayed DELETE authorized for the old artifact. A database connection ending or a timeout does not prove that an external storage request ended.

Do not enable reclamation that relies on either policy before its admission, retry, and late-request behavior is specified and tested. This issue does not introduce a second layer UUID or silently select a storage-generation scheme.

Clear large payloads before pruning identities

Data	Cleanup rule
Description reference and payload of retired, fully released S	Clear after head release and required artifact handoffs, retaining type and identity state; codecs recognize terminal rows before producing runnable inputs
Active S/H	Retain regardless of age
Terminal S with an alias	Permanently retain a compact identity/name record, without its full payload
Terminal S without an alias, and Closed H	Prune in batches only after the admission boundary below makes recreation impossible
Layer/physical-target identity and deletion state	Preserve the information required by the chosen delayed-request and reuse protocol; S/H age does not prove external I/O has finished or authorize pruning these records

Retained database size includes active resources, permanent aliases, legacy identities, and the layer/deletion records required by the chosen policy, as well as history waiting to be pruned.

A monotonic floor F for first identity admission

New S/H use UUIDv7. First admission uses its immutable generation timestamp, birth(id), rather than business created_at. F is a durable value in a fixed Catalog control record alongside schema metadata and only increases. This control record does not grow per S/H and does not imply a fixed business-table count. F governs snapshot/hold admission and eligible history only; it does not govern layer UUIDs, mutable volume IDs, or external I/O completion.

enum FirstAdmission { Existing, New, Expired }

fn classify(existing: bool, id: Id, committed_floor: Timestamp)
    -> Result<FirstAdmission>
{
    if existing { return Ok(FirstAdmission::Existing); }
    let birth = checked_uuid_v7_birth(id)?;
    // Also validate future clock skew on first admission; deployment parameters are omitted.
    Ok(if birth <= committed_floor { FirstAdmission::Expired }
       else { FirstAdmission::New })
}

Classification must happen inside the same transaction. Every entry point that can first INSERT/upsert S/H, including create, Publish, Pin, Delete, and Release, takes a shared F lock before following the business identity, content-root, and layer lock order specified by the Catalog transactions. It retains the F lock until commit. An existence check outside the transaction cannot justify skipping the barrier.

• Existing row: apply the original target, content, and state checks. Long-lived active S/H can continue to be used and released.
• Missing row with birth > F: perform the normal first-admission checks; this does not guarantee success.
• Missing row with birth <= F: Publish/create/Pin return Expired. Delete/Release confirm that the identity has ended without inserting another tombstone.
• Missing non-UUIDv7 identities cannot enter through normal first registration. Existing legacy rows remain usable; terminal rows without a reliable birth timestamp are conservatively retained.

The background task advances F=max(F, reference_time-W) in a short transaction. PostgreSQL uses FOR UPDATE NOWAIT and skips the attempt if busy; ordinary admission uses FOR SHARE. The advancement transaction scans no business tables and performs no external I/O. SQLite uses the equivalent ordering within its serialized write transactions. An unknown commit reply requires an authoritative read of F.

Separate batches then prune only Closed H with birth <= confirmed F, and S with birth <= confirmed F that is retired, fully unlinked, alias-free, and has no remaining heads, required dependencies, or active H. These batches do not acquire F after business-row locks. F never moves backward; restoring an old backup must not reopen a previously closed admission range.

Ensure background work makes progress

A controlled maintenance job runs online without a separate service or leader election. It scans pending content retirement/head-release work, layer candidates, and persistent physical-deletion work separately. Marking and traversal are bounded database work; they are not a full manifest download pass. Each round captures a primary-key upper bound and uses keyset pagination, proceeds past temporarily blocked candidates, then starts another round from the beginning. S/H history pruning also revisits rows that were previously ineligible.

A layer that still has references is retained; later head or successor removal makes it eligible for another check. A content entry previously blocked by a hold, or a layer marked behind the current cursor, is revisited in the next round rather than being starved by repeatedly processing the first page. Any hint optimization must preserve that progress guarantee.

Clean up unregistered uploads only after their writes/commits have ended, or reliable isolation establishes that cleanup cannot race them without stopping user operations. Otherwise retain them. F is not evidence that an upload, VM, or storage request has finished.

Review and validation

• Set W and acceptable future clock skew from the supported first-registration and retry intervals, and validate the resulting delayed-request behavior.
• Test real transaction interleavings among F advancement, first registration, pruning, and unknown replies. Old identities must not reappear; old active H must remain recoverable and releasable.
• Test shared prefixes, branching, cross-snapshot/volume references, backing changes, in-use retirement, and concurrent publication/acquisition versus GC.
• Test duplicate workers, unknown commits, lost or delayed DELETE replies, and blocked candidates with batch size 1; validate the selected identity/key-reuse contract.
• Measure F lock waits, advancement lag, table/index size, and cleanup capacity under sustained low-reuse workloads.
• Compare full mapping O(k) writes with O(delta + h) new-layer/head writes. Include independent chains, mark traversal, reclamation backlog, and full manifest work separately. Separate active data, prunable history, and retained identities; HTTP average rates are not SQL TPS.
• Verify candidate discovery cannot lose work and that a stale mark cannot override a newly committed reference; measure transaction length before choosing immediate checks or persistent hints.

Compatibility and operational impact

This adds F, codecs for compact terminal rows, and new admission-error semantics. All writers and final Release paths must be compatible, and historical data import must have ended, before pruning is enabled. W limits time from identity generation to first registration, not active H lifetime. A capture that has never been published may eventually return Expired; callers must not silently generate another ID to repeat a side-effecting operation.

F adds shared-lock work on one control record. NOWAIT advancement can lag, so W is not a hard bound on retained history. Measure lock waits, actual F lag, cleanup backlog, and database space-maintenance cost. Low-reuse workloads still pay for Pin and eventual Release on each first preparation.

Aliases remain reserved after retirement, complete historical descriptions are removed after unlinking, and pruned identities no longer reproduce exact historical Conflict results. Asynchronous memory_startup attachment must register its additional dependencies atomically and retain producer responsibility until handoff; include that path in GC validation.

Alternatives considered

• Retain all history: simplest permanent idempotency results, but cumulative storage keeps growing.
• Delete terminal rows using TTL alone: does not prevent delayed requests from recreating them.
• Renew a lease for every consumer: adds access and still cannot prove that device or remote I/O has ended.
• Stop user operations before reclaiming resources: violates the continuous-service requirement.
• Check candidate layers immediately when heads or successor references are released: this can reduce separate candidate work, but increases the transaction critical section. Compare contention and throughput with bounded candidate scanning. In both cases current-reference checks and durable physical-delete authorization remain mandatory.

Non-goals

Do not expire active H, use S/H age to prune layer/deletion records, add a session system for orphan uploads, or change volume mount/read-write ownership or the external registry's own GC contract. This proposal includes shared content protection for volumes and registry retention roots, but does not preselect an identity/key-reuse policy.

Contribution

I can provide more design details

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @guozy18 on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/364 ### Affected component Snapshot / template builder / shared snapshot-volume content retention ### Problem statement Part of the [parent proposal](https://github.com/kvcache-ai/AgentENV/issues/360), depending on [the data model and API](https://github.com/kvcache-ai/AgentENV/issues/361) and [Sandbox resource holds](https://github.com/kvcache-ai/AgentENV/issues/362). This issue covers online reclamation over layer dependencies, physical artifact deletion, and terminal history. Activation during version coexistence is covered by the database upgrade proposal. Independent snapshots continuously create new identities; a preparation cache cannot eliminate most first registrations in a low-reuse workload. Keeping every terminal snapshot payload/reference and Closed H forever would make SQL history grow with cumulative publications and uses. Deleting rows based only on age would let delayed Publish/Pin requests recreate identities that have already ended. We need to reclaim unused bytes and safely removable history without letting deletion overtake legitimate consumers. ### Use case Snapshot creation, restore, and publication continue while background cleanup runs. Long-running VMs, paused sandboxes, uploads, and device I/O must retain protection regardless of the cleanup window. Maintenance must not require users to pause these operations. Capacity planning must include many independent snapshots with little reuse, not assume that a few popular templates dominate reads. ### Current behavior and workarounds The Repository manages snapshot records and artifacts in OSS/POSIX, while node-local files and caches have their own lifecycles. Snapshot deletion removes catalog entries and snapshot-specific artifacts; shared layers require separate cleanup. Repository metadata, local references, and cache state each describe part of resource lifetime. SQL adds an explicit dependency graph and shared usage records so online reclamation can account for all three. Completed usage periods and retired snapshots then become database history that also needs a safe retention policy. ### Desired behavior - [ ] A retired S retains every managed dependency while any actual hold remains active. - [ ] A physical DELETE is issued only after its exact target has no global protecting references and its deletion eligibility has committed. - [ ] A terminal S can discard its manifest reference and unnecessary payload after the content and non-layer dependency handoffs are complete. Eligible Closed H rows and terminal S rows without aliases can later be pruned in batches. - [ ] Active H does not expire by age. Cleanup does not require pausing user operations. - [ ] Retries, lost replies, concurrent new references, and database failures do not let deletion overtake legitimate consumers. ### Proposed approach S identifies a public snapshot and H one shared usage period. The [data model](https://github.com/kvcache-ai/AgentENV/issues/361) defines fixed-content entries, heads, immutable layer UUID/predecessor relationships, and their constraints. Volume content participates in the same retention mechanism without treating its mutable business ID as a fixed backing version. ### Mark candidates from SQL; decide deletion from current references Use mark-and-sweep over the database's protected content entries, heads, and predecessor relationships. A GC round need not fetch every OSS manifest. Marks identify candidates, not permission to delete: publication and acquisition may continue during marking, and sweep must recheck current references under the same arbitration used to establish new references. Each layer has at most one immutable predecessor. A content head or a valid successor protects that layer, which in turn protects its predecessor and required bytes. Reclaiming an unreachable branch therefore removes eligible leaves before their predecessors. Retired content remains a root until its actual usage responsibilities have ended. Existing H retries and valid warm reuse continue under the established protection. Predecessor protection may end only after new references are atomically fenced, no actual reader still needs the layer, and remaining deletion work does not need to read its predecessor. State predicates and foreign-key handling must distinguish this released dependency from a retained terminal audit record; the latter must not protect predecessors forever. ### Keep retirement, head release, layer reclamation, and physical deletion separate | Stage | Data change | Preconditions | |---|---|---| | Retire content | Reject first admission of a new H/cold acquisition; retain its fixed description, heads, and active holds | Arbitrate retirement against publication and acquisition; preserve already established use | | Release heads | Remove the retired content's head relationships and record completion | In one transaction, confirm retirement and no remaining use; preserve required non-layer cleanup responsibilities | | Reclaim a logical layer | Change the layer from referenceable to reclamation state; release its predecessor protection as appropriate | Recheck that no content head or valid successor references it, using arbitration shared with concurrent attachment | | Authorize physical deletion | Persist deletion eligibility for the exact physical target | Confirm no globally protecting reference remains, including another logical layer or content owner; the physical-target arbitration must cover concurrent publication | | Delete bytes | Delete the exact target and persist confirmed completion | The authorization commit is confirmed; storage I/O runs outside SQL transactions | For example: ```text S1 -> C -> B -> A S2 -> D -> B -> A ``` Retiring S1 leaves C/B/A protected while an H remains. After final Release, S1's head can be removed and C reclaimed. S2 continues to protect D/B/A. A snapshot and a volume can protect the same prefix in exactly the same way. When a volume switches backing, old users still protect their old heads; the current volume pointer alone is not a sufficient GC root inventory. If publication wins reference arbitration, GC must retain the layer. If reclamation has already fenced the layer against new references, publication must follow the defined conflict path. This arbitration concerns the relevant identities and physical targets; it does not pause all new snapshot use or rewrite every ancestor of a reused prefix. ```rust // Illustrative lifecycle boundaries, not a finalized API or table layout. catalog.release_heads_if_retired_and_unpinned(s1).await?; catalog.reclaim_layer_if_unreferenced(c).await?; if let Some(target) = catalog.authorize_physical_delete(c).await? { // Return a target only after COMMIT is confirmed, not merely UPDATE RETURNING. artifacts.delete_exact(&target).await?; catalog.confirm_deleted(&target).await?; } ``` A count of zero followed by an uncoordinated OSS DELETE is unsafe. If a commit result is unknown, reconcile the original operation and exact target before issuing deletion. Failed or uncertain physical deletion leaves persistent deleting state and retries the same target. Duplicate workers must be safe; external DELETE need not execute exactly once. An absent target can confirm completion only under a protocol that also accounts for delayed requests and target reuse. VM state, startup artifacts, and registry publication retention roots remain part of the protected inventory. They must not be omitted or given artificial layer parents simply to fit the forest model. ### Distinguish progress, safety state, and candidate discovery | Fact | Responsibility | |---|---| | Retirement versus completed head release | Durable progress for pending-work scans and payload cleanup; head release alone does not establish physical deletion safety | | Layer and exact-target deletion state | Fence new references and recover uncertain external deletion without losing the exact target | | Mark results or candidate hints | Identify work to revisit; a stale mark or hint is not deletion permission | These responsibilities do not mandate the former `refs_released` or `gc_pending` field layout. If a persistent hint is used, clearing it must arbitrate with later reference removal so a concurrent clear cannot lose work. Reference relations remain the source of truth; a persisted ref_count is not a prerequisite for this design. ### Resolve identity and key reuse before activating dependent GC Whether a deleted layer UUID or physical key can be reused is an explicit design choice, not settled by this proposal. A no-resurrection policy must state which terminal records are retained and account for their storage cost. A reuse policy must isolate the new artifact from a delayed DELETE authorized for the old artifact. A database connection ending or a timeout does not prove that an external storage request ended. Do not enable reclamation that relies on either policy before its admission, retry, and late-request behavior is specified and tested. This issue does not introduce a second layer UUID or silently select a storage-generation scheme. ### Clear large payloads before pruning identities | Data | Cleanup rule | |---|---| | Description reference and payload of retired, fully released S | Clear after head release and required artifact handoffs, retaining type and identity state; codecs recognize terminal rows before producing runnable inputs | | Active S/H | Retain regardless of age | | Terminal S with an alias | Permanently retain a compact identity/name record, without its full payload | | Terminal S without an alias, and Closed H | Prune in batches only after the admission boundary below makes recreation impossible | | Layer/physical-target identity and deletion state | Preserve the information required by the chosen delayed-request and reuse protocol; S/H age does not prove external I/O has finished or authorize pruning these records | Retained database size includes active resources, permanent aliases, legacy identities, and the layer/deletion records required by the chosen policy, as well as history waiting to be pruned. ### A monotonic floor F for first identity admission New S/H use UUIDv7. First admission uses its immutable generation timestamp, `birth(id)`, rather than business `created_at`. F is a durable value in a fixed Catalog control record alongside schema metadata and only increases. This control record does not grow per S/H and does not imply a fixed business-table count. F governs snapshot/hold admission and eligible history only; it does not govern layer UUIDs, mutable volume IDs, or external I/O completion. ```rust enum FirstAdmission { Existing, New, Expired } fn classify(existing: bool, id: Id, committed_floor: Timestamp) -> Result<FirstAdmission> { if existing { return Ok(FirstAdmission::Existing); } let birth = checked_uuid_v7_birth(id)?; // Also validate future clock skew on first admission; deployment parameters are omitted. Ok(if birth <= committed_floor { FirstAdmission::Expired } else { FirstAdmission::New }) } ``` Classification must happen inside the same transaction. Every entry point that can first INSERT/upsert S/H, including create, Publish, Pin, Delete, and Release, takes a shared F lock before following the business identity, content-root, and layer lock order specified by the Catalog transactions. It retains the F lock until commit. An existence check outside the transaction cannot justify skipping the barrier. - Existing row: apply the original target, content, and state checks. Long-lived active S/H can continue to be used and released. - Missing row with birth > F: perform the normal first-admission checks; this does not guarantee success. - Missing row with birth <= F: Publish/create/Pin return Expired. Delete/Release confirm that the identity has ended without inserting another tombstone. - Missing non-UUIDv7 identities cannot enter through normal first registration. Existing legacy rows remain usable; terminal rows without a reliable birth timestamp are conservatively retained. The background task advances `F=max(F, reference_time-W)` in a short transaction. PostgreSQL uses `FOR UPDATE NOWAIT` and skips the attempt if busy; ordinary admission uses `FOR SHARE`. The advancement transaction scans no business tables and performs no external I/O. SQLite uses the equivalent ordering within its serialized write transactions. An unknown commit reply requires an authoritative read of F. Separate batches then prune only Closed H with birth <= confirmed F, and S with birth <= confirmed F that is retired, fully unlinked, alias-free, and has no remaining heads, required dependencies, or active H. These batches do not acquire F after business-row locks. F never moves backward; restoring an old backup must not reopen a previously closed admission range. ### Ensure background work makes progress A controlled maintenance job runs online without a separate service or leader election. It scans pending content retirement/head-release work, layer candidates, and persistent physical-deletion work separately. Marking and traversal are bounded database work; they are not a full manifest download pass. Each round captures a primary-key upper bound and uses keyset pagination, proceeds past temporarily blocked candidates, then starts another round from the beginning. S/H history pruning also revisits rows that were previously ineligible. A layer that still has references is retained; later head or successor removal makes it eligible for another check. A content entry previously blocked by a hold, or a layer marked behind the current cursor, is revisited in the next round rather than being starved by repeatedly processing the first page. Any hint optimization must preserve that progress guarantee. Clean up unregistered uploads only after their writes/commits have ended, or reliable isolation establishes that cleanup cannot race them without stopping user operations. Otherwise retain them. F is not evidence that an upload, VM, or storage request has finished. ### Review and validation - [ ] Set W and acceptable future clock skew from the supported first-registration and retry intervals, and validate the resulting delayed-request behavior. - [ ] Test real transaction interleavings among F advancement, first registration, pruning, and unknown replies. Old identities must not reappear; old active H must remain recoverable and releasable. - [ ] Test shared prefixes, branching, cross-snapshot/volume references, backing changes, in-use retirement, and concurrent publication/acquisition versus GC. - [ ] Test duplicate workers, unknown commits, lost or delayed DELETE replies, and blocked candidates with batch size 1; validate the selected identity/key-reuse contract. - [ ] Measure F lock waits, advancement lag, table/index size, and cleanup capacity under sustained low-reuse workloads. - [ ] Compare full mapping O(k) writes with O(delta + h) new-layer/head writes. Include independent chains, mark traversal, reclamation backlog, and full manifest work separately. Separate active data, prunable history, and retained identities; HTTP average rates are not SQL TPS. - [ ] Verify candidate discovery cannot lose work and that a stale mark cannot override a newly committed reference; measure transaction length before choosing immediate checks or persistent hints. ### Compatibility and operational impact This adds F, codecs for compact terminal rows, and new admission-error semantics. All writers and final Release paths must be compatible, and [historical data import](https://github.com/kvcache-ai/AgentENV/issues/363) must have ended, before pruning is enabled. W limits time from identity generation to first registration, not active H lifetime. A capture that has never been published may eventually return Expired; callers must not silently generate another ID to repeat a side-effecting operation. F adds shared-lock work on one control record. NOWAIT advancement can lag, so W is not a hard bound on retained history. Measure lock waits, actual F lag, cleanup backlog, and database space-maintenance cost. Low-reuse workloads still pay for Pin and eventual Release on each first preparation. Aliases remain reserved after retirement, complete historical descriptions are removed after unlinking, and pruned identities no longer reproduce exact historical Conflict results. Asynchronous `memory_startup` attachment must register its additional dependencies atomically and retain producer responsibility until handoff; include that path in GC validation. ### Alternatives considered - Retain all history: simplest permanent idempotency results, but cumulative storage keeps growing. - Delete terminal rows using TTL alone: does not prevent delayed requests from recreating them. - Renew a lease for every consumer: adds access and still cannot prove that device or remote I/O has ended. - Stop user operations before reclaiming resources: violates the continuous-service requirement. - Check candidate layers immediately when heads or successor references are released: this can reduce separate candidate work, but increases the transaction critical section. Compare contention and throughput with bounded candidate scanning. In both cases current-reference checks and durable physical-delete authorization remain mandatory. ### Non-goals Do not expire active H, use S/H age to prune layer/deletion records, add a session system for orphan uploads, or change volume mount/read-write ownership or the external registry's own GC contract. This proposal includes shared content protection for volumes and registry retention roots, but does not preselect an identity/key-reuse policy. ### Contribution I can provide more design details ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label 2026-10-29 14:19:29 +00:00

bsakaguchi commented 2026-10-29 14:19:30 +00:00

Author

Owner

Copy link

@dushulin commented on GitHub (Oct 13, 2026):

@guozy18 hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data.

<!-- gh-comment-id:6140319208 --> @dushulin commented on GitHub (Oct 13, 2026): @guozy18 hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data.

bsakaguchi commented 2026-10-29 14:19:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Oct 20, 2026):

5. @guozy18 hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data.

We are working on it, but this is a large refactor. So this may take some time.

<!-- gh-comment-id:6284634660 --> @yingdi-shan commented on GitHub (Oct 20, 2026): > 5\. [@guozy18](https://github.com/guozy18) hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data. We are working on it, but this is a large refactor. So this may take some time.

bsakaguchi commented 2026-10-29 14:19:30 +00:00

Author

Owner

Copy link

@dushulin commented on GitHub (Oct 20, 2026):

7. > 5. @guozy18 hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data.

We are working on it, but this is a large refactor. So this may take some time.

@yingdi-shan Thank you for your reply. Also, I noticed an earlier P2P issue (#12), which mentioned that snapshots would be stored in an iroh P2P cluster while the data itself stays local. I'd like to ask: before the P2P capability is implemented, what network storage underlies the previously claimed snapshot writes and reads for 1.5 million concurrent sandboxes? What are the QPS and throughput requirements for those operations? And if further scaling is still desired, would it then be constrained by the network storage's performance?

<!-- gh-comment-id:6284886497 --> @dushulin commented on GitHub (Oct 20, 2026): > 7\. > 5. [@guozy18](https://github.com/guozy18) hi, May I ask when this capability will be available? We urgently need it internally. We use JFS as the snapshot storage backend and access it via the POSIX protocol. The number of sandbox snapshots we need to save each day is quite large (at the PB level), and we clean them up on a weekly cycle (so the total capacity is at the 10 PB level). However, we are not sure how to delete the layers data. > > > We are working on it, but this is a large refactor. So this may take some time. @yingdi-shan Thank you for your reply. Also, I noticed an earlier P2P issue (#12), which mentioned that snapshots would be stored in an iroh P2P cluster while the data itself stays local. I'd like to ask: before the P2P capability is implemented, what network storage underlies the previously claimed snapshot writes and reads for 1.5 million concurrent sandboxes? What are the QPS and throughput requirements for those operations? And if further scaling is still desired, would it then be constrained by the network storage's performance?

bsakaguchi commented 2026-10-29 14:19:30 +00:00

Author

Owner

Copy link

@yingdi-shan commented on GitHub (Oct 20, 2026):

Snapshot writes and reads go through cloud object storage (e.g., S3-compatible services). These services scale horizontally and are designed for very high aggregate throughput, so storage performance has not been a bottleneck at our current scale, and we expect it to remain so as we scale further.

<!-- gh-comment-id:6285792080 --> @yingdi-shan commented on GitHub (Oct 20, 2026): Snapshot writes and reads go through cloud object storage (e.g., S3-compatible services). These services scale horizontally and are designed for very high aggregate throughput, so storage performance has not been a bottleneck at our current scale, and we expect it to remain so as we scale further.

bsakaguchi referenced this issue 2026-10-29 14:20:39 +00:00

[PR #93] fix(install): warn for known affected ublk kernels #174

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

coverage-data

gh-pages

v0.3.0-rc.1

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

documentation

  

enhancement

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

No labels

bug

documentation

enhancement

enhancement

pull-request

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/AgentENV#93

No description provided.