[PR #176] [MERGED] Support configurable S3 addressing style #228

New issue

Closed

opened 2026-10-29 14:21:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/176
Author: @guozy18
Created: 8/17/2026
Status: ✅ Merged
Merged: 8/19/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/oss-addressing-style

📝 Commits (1)

b7dcd59 feat(oss): support configurable S3 addressing style

📊 Changes

16 files changed (+289 additions, -11 deletions)

View changed files

📝 config/default.toml (+3 -0)
📝 crates/e2e-tests/tests/snapshot_oss_e2e_test.rs (+1 -0)
📝 crates/object-store-operator/src/operator.rs (+1 -1)
📝 docs/src/configuration/reference.md (+13 -0)
📝 docs/src/getting-started/on-demand-loading.md (+5 -0)
📝 src/cfg.rs (+15 -0)
📝 src/setup/deps.rs (+23 -2)
📝 src/snapshot/image_export/service.rs (+1 -0)
📝 src/snapshot/repository/backends/oss/client.rs (+56 -1)
📝 src/snapshot/repository/backends/oss/config.rs (+16 -3)
📝 src/snapshot/repository/backends/oss/mod.rs (+1 -0)
📝 src/snapshot/repository/backends/oss/repository.rs (+1 -0)
📝 storage/overlaybd/src/backend/oss.rs (+46 -4)
📝 storage/overlaybd/src/config.rs (+12 -0)
➕ storage/overlaybd/tests/oss_addressing_style.rs (+93 -0)
📝 storage/overlaybd/tests/oss_backend_minio.rs (+2 -0)

📄 Description

Summary

Add an optional OSS/S3 bucket addressing-style override and propagate it through both snapshot data paths.

Motivation

The existing OSS backend chooses the addressing style from the endpoint:

Alibaba OSS endpoints and bucket-in-endpoint hosts use virtual-host style.
Other endpoints fall back to path style.

Some S3-compatible providers require or prefer virtual-host addressing but cannot be identified reliably by the endpoint heuristic. Without an explicit override, those providers may generate invalid requests.

This change allows users to explicitly select the required addressing style.

Changes

Add the optional backend.oss.addressing_style configuration:
- virtual
- path
- unset for the existing endpoint-based auto-detection
Propagate the setting to the snapshot OSS client.
Propagate the setting to the generated OverlayBD runtime configuration as ossConfig.defaultAddressingStyle.
Apply the setting to OverlayBD remote managed-layer reads as well as snapshot repository operations and image export.
Reject unsupported addressing-style values.
Document the configuration and preserve the existing default behavior.
Add a wire-level regression test that verifies the actual HTTP method, request path, and Host header.
Keep the implementation focused by removing tests that only duplicated internal enum mappings or parser implementation details.

Configuration example

[backend.oss]
endpoint = "https://t3.storage.dev"
bucket = "agentenv-snapshots"
region = "auto"
addressing_style = "virtual"

When addressing_style is omitted, the existing endpoint-based detection remains unchanged.

Backward compatibility

Existing configurations do not require any migration.

The default behavior is preserved:

recognized Alibaba/bucket-host endpoints continue to use virtual-host addressing;
other endpoints continue to use path-style addressing unless explicitly overridden.

This change does not modify Prometheus metrics or the /metrics endpoints.

Validation

The following checks pass locally:

cargo fmt --all -- --check
git diff --check
cargo test -p object-store-operator
cargo clippy -p object-store-operator --all-targets -- -D warnings

The full AgentENV and OverlayBD test suites require a Linux environment. They could not be run locally on macOS because the workspace's io-uring dependency uses Linux-only libc symbols. Linux CI is required for the full workspace verification.

Relationship to #23

This PR supersedes #23.

The original requirement and initial implementation were proposed by @davidmyriel in #23. Thank you for that work and for the original motivation behind the change.

This PR carries the same behavior forward on top of the current main branch, with a refreshed implementation, focused regression tests, and updated documentation. The current implementation was reorganized to fit the latest codebase while preserving the original user-facing intent.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/176 **Author:** [@guozy18](https://github.com/guozy18) **Created:** 8/17/2026 **Status:** ✅ Merged **Merged:** 8/19/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/oss-addressing-style` --- ### 📝 Commits (1) - [`b7dcd59`](https://github.com/kvcache-ai/AgentENV/commit/b7dcd5945b0bbb93cde11074b8421d00302b9c21) feat(oss): support configurable S3 addressing style ### 📊 Changes **16 files changed** (+289 additions, -11 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+3 -0) 📝 `crates/e2e-tests/tests/snapshot_oss_e2e_test.rs` (+1 -0) 📝 `crates/object-store-operator/src/operator.rs` (+1 -1) 📝 `docs/src/configuration/reference.md` (+13 -0) 📝 `docs/src/getting-started/on-demand-loading.md` (+5 -0) 📝 `src/cfg.rs` (+15 -0) 📝 `src/setup/deps.rs` (+23 -2) 📝 `src/snapshot/image_export/service.rs` (+1 -0) 📝 `src/snapshot/repository/backends/oss/client.rs` (+56 -1) 📝 `src/snapshot/repository/backends/oss/config.rs` (+16 -3) 📝 `src/snapshot/repository/backends/oss/mod.rs` (+1 -0) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+1 -0) 📝 `storage/overlaybd/src/backend/oss.rs` (+46 -4) 📝 `storage/overlaybd/src/config.rs` (+12 -0) ➕ `storage/overlaybd/tests/oss_addressing_style.rs` (+93 -0) 📝 `storage/overlaybd/tests/oss_backend_minio.rs` (+2 -0) </details> ### 📄 Description ## Summary Add an optional OSS/S3 bucket addressing-style override and propagate it through both snapshot data paths. ## Motivation The existing OSS backend chooses the addressing style from the endpoint: - Alibaba OSS endpoints and bucket-in-endpoint hosts use virtual-host style. - Other endpoints fall back to path style. Some S3-compatible providers require or prefer virtual-host addressing but cannot be identified reliably by the endpoint heuristic. Without an explicit override, those providers may generate invalid requests. This change allows users to explicitly select the required addressing style. ## Changes - Add the optional `backend.oss.addressing_style` configuration: - `virtual` - `path` - unset for the existing endpoint-based auto-detection - Propagate the setting to the snapshot OSS client. - Propagate the setting to the generated OverlayBD runtime configuration as `ossConfig.defaultAddressingStyle`. - Apply the setting to OverlayBD remote managed-layer reads as well as snapshot repository operations and image export. - Reject unsupported addressing-style values. - Document the configuration and preserve the existing default behavior. - Add a wire-level regression test that verifies the actual HTTP method, request path, and `Host` header. - Keep the implementation focused by removing tests that only duplicated internal enum mappings or parser implementation details. ## Configuration example ```toml [backend.oss] endpoint = "https://t3.storage.dev" bucket = "agentenv-snapshots" region = "auto" addressing_style = "virtual" ``` When `addressing_style` is omitted, the existing endpoint-based detection remains unchanged. ## Backward compatibility Existing configurations do not require any migration. The default behavior is preserved: - recognized Alibaba/bucket-host endpoints continue to use virtual-host addressing; - other endpoints continue to use path-style addressing unless explicitly overridden. This change does not modify Prometheus metrics or the `/metrics` endpoints. ## Validation The following checks pass locally: - `cargo fmt --all -- --check` - `git diff --check` - `cargo test -p object-store-operator` - `cargo clippy -p object-store-operator --all-targets -- -D warnings` The full AgentENV and OverlayBD test suites require a Linux environment. They could not be run locally on macOS because the workspace's `io-uring` dependency uses Linux-only libc symbols. Linux CI is required for the full workspace verification. ## Relationship to #23 This PR supersedes #23. The original requirement and initial implementation were proposed by @davidmyriel in #23. Thank you for that work and for the original motivation behind the change. This PR carries the same behavior forward on top of the current `main` branch, with a refreshed implementation, focused regression tests, and updated documentation. The current implementation was reorganized to fit the latest codebase while preserving the original user-facing intent. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:15 +00:00