[PR #225] [MERGED] refactor(overlaybd): stream object-store uploads through one implementation #267

New issue

Closed

opened 2026-10-29 14:21:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:38 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/225
Author: @huang-jl
Created: 8/27/2026
Status: ✅ Merged
Merged: 8/28/2026
Merged by: @yingdi-shan

Base: main ← Head: hjl/overlaybd-streaming-upload

📝 Commits (3)

cdbd638 feat(overlaybd): stream file uploads through one shared implementation
1597f21 feat(overlaybd): let a caller track upload progress
bcdb618 feat(overlaybd): expose the object-store backend

📊 Changes

4 files changed (+370 additions, -36 deletions)

View changed files

📝 src/snapshot/repository/backends/oss/client.rs (+19 -30)
📝 storage/overlaybd/src/backend/oss.rs (+234 -5)
📝 storage/overlaybd/src/image/image_service.rs (+40 -1)
📝 storage/overlaybd/tests/oss_backend_minio.rs (+77 -0)

📄 Description

What

Moves AgentENV's streaming multipart upload loop into overlaybd as backend::oss::upload_file_streaming, and routes both callers through it: the snapshot repository's put_file and OssBackend::upload_path. Part size and concurrency are passed in by the caller rather than baked in.

Also adds OssBackend::object_size, an optional progress counter, and an ImageService::oss_backend() accessor so a caller can share the existing backend instead of constructing a second one.

Why

AgentENV's snapshot repository already uploaded files as a streaming multipart. overlaybd's own OssBackend::upload_path did not: it read the entire file into memory, and upload_bytes then cloned that buffer again inside the retry closure. Layers are gigabytes, so a 3 GiB layer needed roughly 6 GiB of resident memory, and the cost grows with the layer, which becomes acceptable.

Rather than add a second implementation, the streaming loop moves into overlaybd and both callers use it, leaving one upload path with one set of retry semantics. Part size and concurrency stay caller-supplied because they set both the memory ceiling and the largest uploadable object (S3 caps a multipart at 10,000 parts), and the two callers do not want the same trade-off.

Design and behavior changes

No changes to existing behaviors, only update interfaces for overlaybd crate users.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/225 **Author:** [@huang-jl](https://github.com/huang-jl) **Created:** 8/27/2026 **Status:** ✅ Merged **Merged:** 8/28/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `hjl/overlaybd-streaming-upload` --- ### 📝 Commits (3) - [`cdbd638`](https://github.com/kvcache-ai/AgentENV/commit/cdbd6382a7573eebd1d0a23fd272b6c2df908eaf) feat(overlaybd): stream file uploads through one shared implementation - [`1597f21`](https://github.com/kvcache-ai/AgentENV/commit/1597f218e19b4e617d3ac984557d30172d7952cd) feat(overlaybd): let a caller track upload progress - [`bcdb618`](https://github.com/kvcache-ai/AgentENV/commit/bcdb618154188292c45087302ae5a8c1823b98f4) feat(overlaybd): expose the object-store backend ### 📊 Changes **4 files changed** (+370 additions, -36 deletions) <details> <summary>View changed files</summary> 📝 `src/snapshot/repository/backends/oss/client.rs` (+19 -30) 📝 `storage/overlaybd/src/backend/oss.rs` (+234 -5) 📝 `storage/overlaybd/src/image/image_service.rs` (+40 -1) 📝 `storage/overlaybd/tests/oss_backend_minio.rs` (+77 -0) </details> ### 📄 Description ## What Moves AgentENV's streaming multipart upload loop into overlaybd as backend::oss::upload_file_streaming, and routes both callers through it: the snapshot repository's put_file and OssBackend::upload_path. Part size and concurrency are passed in by the caller rather than baked in. Also adds OssBackend::object_size, an optional progress counter, and an ImageService::oss_backend() accessor so a caller can share the existing backend instead of constructing a second one. ## Why AgentENV's snapshot repository already uploaded files as a streaming multipart. overlaybd's own OssBackend::upload_path did not: it read the entire file into memory, and upload_bytes then cloned that buffer again inside the retry closure. Layers are gigabytes, so a 3 GiB layer needed roughly 6 GiB of resident memory, and the cost grows with the layer, which becomes acceptable. Rather than add a second implementation, the streaming loop moves into overlaybd and both callers use it, leaving one upload path with one set of retry semantics. Part size and concurrency stay caller-supplied because they set both the memory ceiling and the largest uploadable object (S3 caps a multipart at 10,000 parts), and the two callers do not want the same trade-off. ## Design and behavior changes No changes to existing behaviors, only update interfaces for overlaybd crate users. ## Validation  - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` - [ ] Relevant Rust integration tests - [ ] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [ ] Documentation updated - [ ] Benchmarks or performance comparison completed --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:38 +00:00