[PR #226] [MERGED] feat(image): publish converted OCI layers over P2P #270

New issue

Closed

opened 2026-10-29 14:21:40 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:40 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/226
Author: @LSX-s-Software
Created: 8/27/2026
Status: ✅ Merged
Merged: 8/31/2026
Merged by: @LSX-s-Software

Base: main ← Head: feat/p2p

📝 Commits (4)

dce0652 chore(deps): bump iroh to v1.0.3 and iroh-blobs to v0.103.0
49c3217 feat(image-cache): record p2p key and unpublish during eviction
72e2b18 feat(image): publish converted layers to p2p
684532c perf(image): skip p2p publish when disabled and cache p2p lookup results

📊 Changes

11 files changed (+988 additions, -311 deletions)

View changed files

📝 Cargo.lock (+234 -267)
📝 Cargo.toml (+2 -2)
📝 docs/src/internals/p2p-design.md (+3 -0)
📝 docs/src/internals/persistence-artifact-inventory.md (+1 -0)
📝 src/bin/server.rs (+1 -0)
📝 src/image/cache/graph.rs (+124 -1)
📝 src/image/cache/mod.rs (+2 -0)
📝 src/image/cache/service.rs (+549 -39)
📝 src/image/mod.rs (+10 -0)
📝 src/image/oci_image.rs (+58 -2)
📝 src/p2p/mock.rs (+4 -0)

📄 Description

What

Publish converted standard OCI image layers through the existing P2P transport and reuse them across nodes. Track P2P artifact ownership in the image-cache metadata so eviction can remove remote references together with local commit files.

Why

Standard OCI layers currently need to be downloaded and converted independently on each node. Sharing completed OverlayBD conversions avoids repeated registry downloads and conversion work while preserving the existing local-cache and registry fallback paths.

N/A. No issue or prior design discussion is referenced by the commits in this branch.

Scope and non-goals

This PR includes:

A stable oci-layer/v1/{context-hash} P2P key derived from LayerConversionKey.
Reference-mode publication of durable converted OverlayBD commits with protocol, digest, and size metadata.
P2P lookup, download, validation, local persistence, and conversion-index reuse.
Persistent P2P-key ownership on image-cache hard-commit records and unpublish-before-delete during GC.
The iroh 1.0.3 and iroh-blobs 0.103.0 dependency update.

This PR does not change overlaybd-native image handling, snapshot artifact publication, public API routes, or introduce new configuration knobs.

Design and behavior changes

The server initializes the image cache with the same process-wide P2P transport used by the other P2P consumers. For a standard OCI layer, the conversion context is serialized and hashed into the catalog key. Once the converted commit is durable locally, the image cache records ownership and publishes the file by reference.

On a cache-index miss or stale entry, the image cache first attempts the corresponding P2P lookup. Fetched bytes must match the advertised digest and size and contain a sealed OverlayBD layer with the expected UUID before they are persisted and indexed locally. Lookup, fetch, metadata, digest, or structural validation failures are treated as acceleration misses and fall back to registry download and conversion; invalid fetched artifacts are unpublished when possible.

Image-cache GC carries the recorded P2P keys through its collectability checks and unpublishes them before removing the local commit and metadata. Metadata updates are serialized so ownership is preserved across concurrent record/reconcile operations.

Compatibility and operations

Public API or generated protocol: N/A; no public HTTP or generated protocol changes.
Configuration or defaults: N/A; uses the existing P2P transport/configuration and adds no user-facing setting.
Snapshot manifest, artifact layout, or storage format: No snapshot changes. Image-cache hard-commit metadata gains an additive optional p2p_keys field, defaulting to empty for existing records.
Upgrade and rollback: The branch updates iroh to 1.0.3 and iroh-blobs to 0.103.0. Nodes without usable P2P publication or lookup continue through the existing local/registry paths; cache metadata remains readable when no P2P keys are present.
Host requirements, permissions, ports, or dependencies: No new host requirement or port. Existing P2P and registry access prerequisites still apply.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

The author reports that the relevant tests were run successfully before PR creation. Tests were not rerun during PR preparation, per request; exact command output is not available in this shell.

Skipped checks and reasons:

Formatting, clippy, and test commands were not rerun during PR creation because the author had already run the relevant validation. Services tests and code generation are not applicable because this branch does not change services or generated clients/server code. No benchmark was requested or reported.

Risks and reviewer notes

P2P publication is intentionally best-effort for image acceleration; registry conversion remains the correctness fallback.
Remote bytes are validated against metadata, file digest, sealed OverlayBD structure, and the conversion-context UUID before reuse.
Ownership is recorded before publication to avoid leaving a catalog reference without GC metadata after a crash; a failed publication may leave an unserved ownership record that is harmless and will be cleaned with the cache object.
Review the conversion-key stability and the ordering of unpublish versus local deletion in src/image/oci_image.rs and src/image/cache/service.rs.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/226 **Author:** [@LSX-s-Software](https://github.com/LSX-s-Software) **Created:** 8/27/2026 **Status:** ✅ Merged **Merged:** 8/31/2026 **Merged by:** [@LSX-s-Software](https://github.com/LSX-s-Software) **Base:** `main` ← **Head:** `feat/p2p` --- ### 📝 Commits (4) - [`dce0652`](https://github.com/kvcache-ai/AgentENV/commit/dce06522c8383b33673def085c3e89f94dfdaf83) chore(deps): bump iroh to v1.0.3 and iroh-blobs to v0.103.0 - [`49c3217`](https://github.com/kvcache-ai/AgentENV/commit/49c3217ece06f2b4fdb79679362cf5d29413a47d) feat(image-cache): record p2p key and unpublish during eviction - [`72e2b18`](https://github.com/kvcache-ai/AgentENV/commit/72e2b18190d1cc0359e036e3d5b7ceef1e056323) feat(image): publish converted layers to p2p - [`684532c`](https://github.com/kvcache-ai/AgentENV/commit/684532c816225a51d5c99053a2c8ca711efc383b) perf(image): skip p2p publish when disabled and cache p2p lookup results ### 📊 Changes **11 files changed** (+988 additions, -311 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+234 -267) 📝 `Cargo.toml` (+2 -2) 📝 `docs/src/internals/p2p-design.md` (+3 -0) 📝 `docs/src/internals/persistence-artifact-inventory.md` (+1 -0) 📝 `src/bin/server.rs` (+1 -0) 📝 `src/image/cache/graph.rs` (+124 -1) 📝 `src/image/cache/mod.rs` (+2 -0) 📝 `src/image/cache/service.rs` (+549 -39) 📝 `src/image/mod.rs` (+10 -0) 📝 `src/image/oci_image.rs` (+58 -2) 📝 `src/p2p/mock.rs` (+4 -0) </details> ### 📄 Description ## What Publish converted standard OCI image layers through the existing P2P transport and reuse them across nodes. Track P2P artifact ownership in the image-cache metadata so eviction can remove remote references together with local commit files. ## Why Standard OCI layers currently need to be downloaded and converted independently on each node. Sharing completed OverlayBD conversions avoids repeated registry downloads and conversion work while preserving the existing local-cache and registry fallback paths. ## Related issue N/A. No issue or prior design discussion is referenced by the commits in this branch. ## Scope and non-goals This PR includes: - A stable `oci-layer/v1/{context-hash}` P2P key derived from `LayerConversionKey`. - Reference-mode publication of durable converted OverlayBD commits with protocol, digest, and size metadata. - P2P lookup, download, validation, local persistence, and conversion-index reuse. - Persistent P2P-key ownership on image-cache hard-commit records and unpublish-before-delete during GC. - The `iroh` 1.0.3 and `iroh-blobs` 0.103.0 dependency update. This PR does not change overlaybd-native image handling, snapshot artifact publication, public API routes, or introduce new configuration knobs. ## Design and behavior changes The server initializes the image cache with the same process-wide P2P transport used by the other P2P consumers. For a standard OCI layer, the conversion context is serialized and hashed into the catalog key. Once the converted commit is durable locally, the image cache records ownership and publishes the file by reference. On a cache-index miss or stale entry, the image cache first attempts the corresponding P2P lookup. Fetched bytes must match the advertised digest and size and contain a sealed OverlayBD layer with the expected UUID before they are persisted and indexed locally. Lookup, fetch, metadata, digest, or structural validation failures are treated as acceleration misses and fall back to registry download and conversion; invalid fetched artifacts are unpublished when possible. Image-cache GC carries the recorded P2P keys through its collectability checks and unpublishes them before removing the local commit and metadata. Metadata updates are serialized so ownership is preserved across concurrent record/reconcile operations. ## Compatibility and operations - Public API or generated protocol: N/A; no public HTTP or generated protocol changes. - Configuration or defaults: N/A; uses the existing P2P transport/configuration and adds no user-facing setting. - Snapshot manifest, artifact layout, or storage format: No snapshot changes. Image-cache hard-commit metadata gains an additive optional `p2p_keys` field, defaulting to empty for existing records. - Upgrade and rollback: The branch updates `iroh` to 1.0.3 and `iroh-blobs` to 0.103.0. Nodes without usable P2P publication or lookup continue through the existing local/registry paths; cache metadata remains readable when no P2P keys are present. - Host requirements, permissions, ports, or dependencies: No new host requirement or port. Existing P2P and registry access prerequisites still apply. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [x] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results: ```text The author reports that the relevant tests were run successfully before PR creation. Tests were not rerun during PR preparation, per request; exact command output is not available in this shell. ``` Skipped checks and reasons: ```text Formatting, clippy, and test commands were not rerun during PR creation because the author had already run the relevant validation. Services tests and code generation are not applicable because this branch does not change services or generated clients/server code. No benchmark was requested or reported. ``` ## Risks and reviewer notes - P2P publication is intentionally best-effort for image acceleration; registry conversion remains the correctness fallback. - Remote bytes are validated against metadata, file digest, sealed OverlayBD structure, and the conversion-context UUID before reuse. - Ownership is recorded before publication to avoid leaving a catalog reference without GC metadata after a crash; a failed publication may leave an unserved ownership record that is harmless and will be cleaned with the cache object. - Review the conversion-key stability and the ordering of unpublish versus local deletion in `src/image/oci_image.rs` and `src/image/cache/service.rs`. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:40 +00:00