[PR #58] [MERGED] feat(image): discover ACR artifact streaming overlaybd referrers #146

New issue

Closed

opened 2026-10-29 14:20:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:20:12 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/58
Author: @johnsonshi
Created: 7/29/2026
Status: ✅ Merged
Merged: 7/29/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/acr-artifact-streaming-referrer

📝 Commits (1)

01f842a feat(image): discover ACR artifact streaming overlaybd referrers

📊 Changes

2 files changed (+179 additions, -38 deletions)

View changed files

📝 docs/src/configuration/reference.md (+17 -0)
📝 src/image/resolver.rs (+162 -38)

📄 Description

Problem

Azure Container Registry's artifact streaming converts an image to overlaybd and attaches the result as an OCI referrer — but labels it with its own artifactType:

artifactType: application/vnd.azure.artifact.streaming.v1
annotations:  streaming.format=overlaybd, streaming.version=v1,
              streaming.platform.os=linux, streaming.platform.arch=amd64

Referrer discovery matched only application/vnd.containerd.overlaybd.native.v1+json, so ACR-streamed images were never found.

The failure is silent: try_resolve_overlaybd_referrer returns Ok(None), the trace! branch fires, and resolution falls back to downloading and converting the source image locally. Users who enable try_referrers_overlaybd_prefixes for an ACR registry get a working sandbox with none of the benefit — the already-converted overlaybd artifact sitting in the registry is ignored.

What ACR actually publishes

I converted python:3.11-slim in a Premium ACR and read the referrers API directly. The referrer manifest is ordinary overlaybd-native, not turbo-OCI:

{
  "mediaType": "application/vnd.oci.image.layer.v1.tar",
  "digest": "sha256:7c6e3cf3029dc9eb52c8abf0e16c7ea15ca020e6b4f13b4461abce984f80e40d",
  "size": 51212800,
  "annotations": {
    "containerd.io/snapshot/overlaybd/blob-digest": "sha256:7c6e3cf3029dc9eb52c8abf0e16c7ea15ca020e6b4f13b4461abce984f80e40d",
    "containerd.io/snapshot/overlaybd/blob-size": "51212800",
    "containerd.io/snapshot/overlaybd/version": "0.1.0"
  }
}

blob-digest equals the layer's own digest and the blob carries the whole layer. This is exactly the shape classify_manifest already handles — see the existing classify_manifest_recognizes_acr_mediatype_lying_overlaybd_image test. ImageFormat::OverlaybdNative classification and the registryfs_v2 passthrough already work for these images today.

Only the discovery label differed.

Change

Match referrers against OVERLAYBD_REFERRER_ARTIFACT_TYPES, a preference-ordered list of application/vnd.containerd.overlaybd.native.v1+json then application/vnd.azure.artifact.streaming.v1. When both are present the accelerated-container-image type wins, deterministically.
Fetch the referrers index unfiltered and match locally. regctl artifact list takes a single --filter-artifact-type and cannot express "any of these types" in one call.
Thread the matched artifactType back to the caller so the info!/trace! logs and the mismatch error report the type that actually matched, instead of a hardcoded constant that may not be the one that was found.

This widens discovery only. try_resolve_overlaybd_referrer still re-fetches the referrer manifest and rejects anything whose format isn't OverlaybdNative, so a mislabelled referrer still cannot be used. Turbo-OCI referrers remain unselectable, which the new tests assert explicitly.

Tests

Added to src/image/resolver.rs:

parse_overlaybd_referrer_accepts_acr_artifact_streaming — uses the real ACR referrers-index shape, annotations included, so the format is documented in-tree.
parse_overlaybd_referrer_prefers_containerd_native_over_acr_streaming — deterministic preference when both exist.
parse_overlaybd_referrer_ignores_unrelated_artifact_types — SBOM and turbo-OCI referrers are never selected.

Existing referrer tests were updated for the new return type and kept otherwise unchanged.

cargo test -p agentenv --lib image::resolver
   running 15 tests ... test result: ok. 15 passed; 0 failed

Full lib suite goes 630 → 633 passing on my machine. Three pre-existing failures (snapshot::p2p::tests::local_overlaybd_layers_*, snapshot::…::acr::source_image::tests::plans_descriptorless_sparse_overlaybd_delta_for_dense_export) reproduce identically on an unmodified main checkout here and are unrelated to this change — they look environment-specific (aarch64 Linux container).

cargo fmt --all -- --check and cargo clippy -p agentenv --all-targets --all-features -- -D warnings are both clean.

Docs

docs/src/configuration/reference.md now tabulates both accepted artifactTypes under try_referrers_overlaybd_prefixes, notes that turbo-OCI is never selected, and gives the ACR usage example:

try_referrers_overlaybd_prefixes = ["myregistry.azurecr.io/"]

Verification

Verified end to end against a real Premium ACR (az acr artifact-streaming create) and an AKS cluster running Ubuntu 24.04 / kernel 6.8 — where, for what it's worth, ublk_drv and nested KVM are both available on stock nodes.

Happy to adjust naming, split the log/error plumbing into its own commit, or gate the new artifactType behind config if you'd prefer it opt-in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/58 **Author:** [@johnsonshi](https://github.com/johnsonshi) **Created:** 7/29/2026 **Status:** ✅ Merged **Merged:** 7/29/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/acr-artifact-streaming-referrer` --- ### 📝 Commits (1) - [`01f842a`](https://github.com/kvcache-ai/AgentENV/commit/01f842ac9f9618be1876b53d4d1519e7c2a4c458) feat(image): discover ACR artifact streaming overlaybd referrers ### 📊 Changes **2 files changed** (+179 additions, -38 deletions) <details> <summary>View changed files</summary> 📝 `docs/src/configuration/reference.md` (+17 -0) 📝 `src/image/resolver.rs` (+162 -38) </details> ### 📄 Description ## Problem Azure Container Registry's [artifact streaming](https://learn.microsoft.com/azure/container-registry/container-registry-artifact-streaming) converts an image to overlaybd and attaches the result as an OCI referrer — but labels it with its own artifactType: ``` artifactType: application/vnd.azure.artifact.streaming.v1 annotations: streaming.format=overlaybd, streaming.version=v1, streaming.platform.os=linux, streaming.platform.arch=amd64 ``` Referrer discovery matched only `application/vnd.containerd.overlaybd.native.v1+json`, so ACR-streamed images were never found. The failure is **silent**: `try_resolve_overlaybd_referrer` returns `Ok(None)`, the `trace!` branch fires, and resolution falls back to downloading and converting the source image locally. Users who enable `try_referrers_overlaybd_prefixes` for an ACR registry get a working sandbox with none of the benefit — the already-converted overlaybd artifact sitting in the registry is ignored. ## What ACR actually publishes I converted `python:3.11-slim` in a Premium ACR and read the referrers API directly. The referrer manifest is **ordinary overlaybd-native**, not turbo-OCI: ```json { "mediaType": "application/vnd.oci.image.layer.v1.tar", "digest": "sha256:7c6e3cf3029dc9eb52c8abf0e16c7ea15ca020e6b4f13b4461abce984f80e40d", "size": 51212800, "annotations": { "containerd.io/snapshot/overlaybd/blob-digest": "sha256:7c6e3cf3029dc9eb52c8abf0e16c7ea15ca020e6b4f13b4461abce984f80e40d", "containerd.io/snapshot/overlaybd/blob-size": "51212800", "containerd.io/snapshot/overlaybd/version": "0.1.0" } } ``` `blob-digest` equals the layer's own digest and the blob carries the whole layer. This is exactly the shape `classify_manifest` already handles — see the existing `classify_manifest_recognizes_acr_mediatype_lying_overlaybd_image` test. `ImageFormat::OverlaybdNative` classification and the `registryfs_v2` passthrough already work for these images today. **Only the discovery label differed.** ## Change - Match referrers against `OVERLAYBD_REFERRER_ARTIFACT_TYPES`, a preference-ordered list of `application/vnd.containerd.overlaybd.native.v1+json` then `application/vnd.azure.artifact.streaming.v1`. When both are present the accelerated-container-image type wins, deterministically. - Fetch the referrers index unfiltered and match locally. `regctl artifact list` takes a single `--filter-artifact-type` and cannot express "any of these types" in one call. - Thread the matched artifactType back to the caller so the `info!`/`trace!` logs and the mismatch error report the type that actually matched, instead of a hardcoded constant that may not be the one that was found. This widens **discovery only**. `try_resolve_overlaybd_referrer` still re-fetches the referrer manifest and rejects anything whose format isn't `OverlaybdNative`, so a mislabelled referrer still cannot be used. Turbo-OCI referrers remain unselectable, which the new tests assert explicitly. ## Tests Added to `src/image/resolver.rs`: - `parse_overlaybd_referrer_accepts_acr_artifact_streaming` — uses the real ACR referrers-index shape, annotations included, so the format is documented in-tree. - `parse_overlaybd_referrer_prefers_containerd_native_over_acr_streaming` — deterministic preference when both exist. - `parse_overlaybd_referrer_ignores_unrelated_artifact_types` — SBOM and **turbo-OCI** referrers are never selected. Existing referrer tests were updated for the new return type and kept otherwise unchanged. ``` cargo test -p agentenv --lib image::resolver running 15 tests ... test result: ok. 15 passed; 0 failed ``` Full lib suite goes 630 → 633 passing on my machine. Three pre-existing failures (`snapshot::p2p::tests::local_overlaybd_layers_*`, `snapshot::…::acr::source_image::tests::plans_descriptorless_sparse_overlaybd_delta_for_dense_export`) reproduce identically on an unmodified `main` checkout here and are unrelated to this change — they look environment-specific (aarch64 Linux container). `cargo fmt --all -- --check` and `cargo clippy -p agentenv --all-targets --all-features -- -D warnings` are both clean. ## Docs `docs/src/configuration/reference.md` now tabulates both accepted artifactTypes under `try_referrers_overlaybd_prefixes`, notes that turbo-OCI is never selected, and gives the ACR usage example: ```toml try_referrers_overlaybd_prefixes = ["myregistry.azurecr.io/"] ``` ## Verification Verified end to end against a real Premium ACR (`az acr artifact-streaming create`) and an AKS cluster running Ubuntu 24.04 / kernel 6.8 — where, for what it's worth, `ublk_drv` and nested KVM are both available on stock nodes. Happy to adjust naming, split the log/error plumbing into its own commit, or gate the new artifactType behind config if you'd prefer it opt-in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:20:12 +00:00