[GH-ISSUE #16] Prevent snapshot deletion from removing shared OCI manifests #12

New issue

Closed

opened 2026-10-29 14:18:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:19 +00:00

Owner

Originally created by @huajq on GitHub (Jul 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/16

Originally assigned to: @huajq on GitHub.

Summary

Snapshot image publications can produce identical OCI manifest digests when two publications reference the same OverlayBD layer set.

AgentENV currently records the manifest digest for each publication and deletes the manifest by digest when rolling back or deleting a snapshot. In OCI registries, multiple tags may point to the same content-addressed manifest. Deleting that manifest by digest can therefore invalidate every tag that references it, including tags owned by other snapshots.

Affected code

src/snapshot/repository/backends/common/acr/manifest.rs
src/snapshot/repository/backends/common/acr/publisher.rs
src/snapshot/repository/backends/oss/repository.rs

Current behavior

The generated OCI manifest contains:

a deterministic minimal OCI config descriptor;
the ordered OverlayBD layer descriptors.

It does not contain the snapshot ID, publication tag, or disk subject.

Therefore, two publications with the same layer descriptors generate identical manifest bytes and the same manifest digest.

For example:

agentenv-snapshot-S1 ─┐
                      ├── sha256:M
agentenv-snapshot-S2 ─┘

When snapshot S1 is deleted, AgentENV executes:

DELETE /v2/<repository>/manifests/sha256:M

Because sha256:M is shared, the registry may remove the manifest referenced by both tags. agentenv-snapshot-S2 then becomes unavailable even though snapshot S2 still exists.

This is especially easy to trigger when publishing snapshots whose rootfs consists entirely of remote layers, but it can occur for any two publications that produce the same config and layer descriptors.

Expected behavior

Each snapshot rootfs or attached-drive publication should own a distinct manifest digest, even when its config and OverlayBD layer descriptors are identical to another publication.

Deleting or rolling back one snapshot publication must not invalidate another snapshot's image tag.

Layer and config blobs should remain content-addressed and shared; only the small manifest object needs to be publication-specific.

Proposed fix

Add the exact publication tag to the top-level OCI manifest annotations:

{
  "annotations": {
    "io.agentenv.snapshot.tag": "agentenv-snapshot-<snapshot-id>"
  }
}

Attached-drive manifests should use their complete generated tag:

agentenv-snapshot-<snapshot-id>-drive-<drive-id>-<hash>

Including the publication tag makes the manifest bytes and digest unique per publication while preserving deduplication of:

existing remote OverlayBD layers;
newly uploaded delta layers;
the minimal OCI config blob.

No layer data needs to be duplicated or re-uploaded.

Cleanup reliability

Snapshot deletion should remove external registry publications before deleting the durable snapshot catalog record.

If registry cleanup fails, the catalog record and its persisted publication metadata must remain available so deletion can be retried. Invalid persisted publication metadata must return an error rather than being treated as successful cleanup.

Manifest deletion should remain idempotent: a registry 404 Not Found is considered successful so partially completed multi-publication cleanup can be retried safely.

Originally created by @huajq on GitHub (Jul 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/16 Originally assigned to: @huajq on GitHub. ## Summary Snapshot image publications can produce identical OCI manifest digests when two publications reference the same OverlayBD layer set. AgentENV currently records the manifest digest for each publication and deletes the manifest by digest when rolling back or deleting a snapshot. In OCI registries, multiple tags may point to the same content-addressed manifest. Deleting that manifest by digest can therefore invalidate every tag that references it, including tags owned by other snapshots. ## Affected code - `src/snapshot/repository/backends/common/acr/manifest.rs` - `src/snapshot/repository/backends/common/acr/publisher.rs` - `src/snapshot/repository/backends/oss/repository.rs` ## Current behavior The generated OCI manifest contains: - a deterministic minimal OCI config descriptor; - the ordered OverlayBD layer descriptors. It does not contain the snapshot ID, publication tag, or disk subject. Therefore, two publications with the same layer descriptors generate identical manifest bytes and the same manifest digest. For example: ```text agentenv-snapshot-S1 ─┐ ├── sha256:M agentenv-snapshot-S2 ─┘ ``` When snapshot `S1` is deleted, AgentENV executes: ```text DELETE /v2/<repository>/manifests/sha256:M ``` Because `sha256:M` is shared, the registry may remove the manifest referenced by both tags. `agentenv-snapshot-S2` then becomes unavailable even though snapshot `S2` still exists. This is especially easy to trigger when publishing snapshots whose rootfs consists entirely of remote layers, but it can occur for any two publications that produce the same config and layer descriptors. ## Expected behavior Each snapshot rootfs or attached-drive publication should own a distinct manifest digest, even when its config and OverlayBD layer descriptors are identical to another publication. Deleting or rolling back one snapshot publication must not invalidate another snapshot's image tag. Layer and config blobs should remain content-addressed and shared; only the small manifest object needs to be publication-specific. ## Proposed fix Add the exact publication tag to the top-level OCI manifest annotations: ```json { "annotations": { "io.agentenv.snapshot.tag": "agentenv-snapshot-<snapshot-id>" } } ``` Attached-drive manifests should use their complete generated tag: ```text agentenv-snapshot-<snapshot-id>-drive-<drive-id>-<hash> ``` Including the publication tag makes the manifest bytes and digest unique per publication while preserving deduplication of: - existing remote OverlayBD layers; - newly uploaded delta layers; - the minimal OCI config blob. No layer data needs to be duplicated or re-uploaded. ## Cleanup reliability Snapshot deletion should remove external registry publications before deleting the durable snapshot catalog record. If registry cleanup fails, the catalog record and its persisted publication metadata must remain available so deletion can be retried. Invalid persisted publication metadata must return an error rather than being treated as successful cleanup. Manifest deletion should remain idempotent: a registry `404 Not Found` is considered successful so partially completed multi-publication cleanup can be retried safely.

bsakaguchi

2026-10-29 14:18:19 +00:00