[PR #23] [CLOSED] feat: support virtual-host addressing for S3-compatible snapshot back #125

New issue

Closed

opened 2026-10-29 14:19:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/23
Author: @davidmyriel
Created: 7/27/2026
Status: ❌ Closed

Base: main ← Head: myriel/tigris-data

📝 Commits (6)

a36d945 feat(oss): support configurable S3 addressing style
b54d04a fix(oss): propagate addressing style to overlaybd remote layer reads
70180d0 fix(oss): harden smoke example and prove addressing override on the wire
8221545 test(oss): drop tigris smoke example, harden addressing wire test
472f6eb Update storage/overlaybd/tests/oss_addressing_style.rs
ec6859e Update storage/overlaybd/tests/oss_addressing_style.rs

📊 Changes

14 files changed (+375 additions, -7 deletions)

View changed files

📝 config/default.toml (+6 -0)
📝 docs/src/configuration/reference.md (+13 -0)
📝 docs/src/getting-started/on-demand-loading.md (+5 -0)
📝 src/cfg.rs (+25 -0)
📝 src/setup/deps.rs (+21 -1)
📝 src/snapshot/repository/backends/oss/client.rs (+65 -1)
📝 src/snapshot/repository/backends/oss/config.rs (+37 -3)
📝 src/snapshot/repository/backends/oss/mod.rs (+1 -0)
📝 src/snapshot/repository/backends/oss/repository.rs (+1 -0)
📝 storage/overlaybd/src/backend/oss.rs (+85 -2)
📝 storage/overlaybd/src/config.rs (+13 -0)
➕ storage/overlaybd/tests/oss_addressing_style.rs (+100 -0)
📝 storage/overlaybd/tests/oss_backend_minio.rs (+2 -0)
📝 tests/snapshot_oss_e2e_test.rs (+1 -0)

📄 Description

What problem does this solve?

I tried to point the OSS snapshot backend at an S3-compatible provider that requires virtual-host bucket addressing and found there was no way to make it work.

The backend picks the addressing style automatically in detect_addressing_style: Alibaba OSS endpoints and endpoints that already contain the bucket name get virtual-host style (https://bucket.example.com/key), and everything else falls back to path style (https://example.com/bucket/key). That fallback is the problem — a growing set of providers only reliably support virtual-host addressing (AWS S3 buckets created after the path-style deprecation, Cloudflare R2, Tigris), and for those the auto-detection guesses wrong with no way to override it. The result is confusing request failures at runtime rather than anything that points at addressing style.

What this PR does

Adds an optional addressing_style field to [backend.oss]:

[backend.oss]
endpoint = "https://t3.storage.dev"
bucket = "agentenv-snapshots"
region = "auto"
addressing_style = "virtual"   # or "path"

The behavior is strictly opt-in:

When set, the explicit value wins and is passed straight through to the object-store client.
When unset, the existing detect_addressing_style auto-detection runs exactly as before — zero behavior change for current Alibaba OSS / MinIO setups.

The plumbing is small: a new OssAddressingStyle config enum, a mapping in NormalizedOssConfig, and an override-or-detect resolution in OssClient::new.

Testing

Unit tests covering: the detection fallback for generic endpoints, the existing Aliyun/bucket-in-endpoint detection, the override taking precedence, and the config mapping.
Verified end-to-end against a real Tigris bucket with the included tigris_smoke example, which drives the same build_object_store_operator path the backend uses: write → read-back → verify → delete round-trip succeeds with addressing_style = "virtual".
Also confirmed at the wire level (via a local Host-header echo endpoint) that the two styles genuinely produce different requests — Host: bucket.endpoint vs /bucket/key on the path — so the override reaches the wire rather than being silently dropped.
cargo fmt, cargo clippy -- -D warnings, and cargo test -p object-store-operator are clean.

Docs

docs/src/configuration/reference.md: new row in the [backend.oss] table plus a short worked example.
docs/src/getting-started/on-demand-loading.md: one-line pointer after the OSS example config, since anyone following that guide with a virtual-host-only provider would otherwise hit an opaque failure.
Commented-out example in config/default.toml.

Happy to drop the smoke example if you'd rather keep examples/ lean

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/23 **Author:** [@davidmyriel](https://github.com/davidmyriel) **Created:** 7/27/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `myriel/tigris-data` --- ### 📝 Commits (6) - [`a36d945`](https://github.com/kvcache-ai/AgentENV/commit/a36d94573eb54aba1ab3da30f644f62e574691cb) feat(oss): support configurable S3 addressing style - [`b54d04a`](https://github.com/kvcache-ai/AgentENV/commit/b54d04a710c7fda55410d04faf4321483ac4baa2) fix(oss): propagate addressing style to overlaybd remote layer reads - [`70180d0`](https://github.com/kvcache-ai/AgentENV/commit/70180d035c30e70d59063d202188cc73afce26c5) fix(oss): harden smoke example and prove addressing override on the wire - [`8221545`](https://github.com/kvcache-ai/AgentENV/commit/8221545efb6f23dd73b3e17d2871801ffb443987) test(oss): drop tigris smoke example, harden addressing wire test - [`472f6eb`](https://github.com/kvcache-ai/AgentENV/commit/472f6ebee8450bf1dd33ff7554c84f4b65569eca) Update storage/overlaybd/tests/oss_addressing_style.rs - [`ec6859e`](https://github.com/kvcache-ai/AgentENV/commit/ec6859e6895c77f8d48ec5b1600c60632e4c62d9) Update storage/overlaybd/tests/oss_addressing_style.rs ### 📊 Changes **14 files changed** (+375 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+6 -0) 📝 `docs/src/configuration/reference.md` (+13 -0) 📝 `docs/src/getting-started/on-demand-loading.md` (+5 -0) 📝 `src/cfg.rs` (+25 -0) 📝 `src/setup/deps.rs` (+21 -1) 📝 `src/snapshot/repository/backends/oss/client.rs` (+65 -1) 📝 `src/snapshot/repository/backends/oss/config.rs` (+37 -3) 📝 `src/snapshot/repository/backends/oss/mod.rs` (+1 -0) 📝 `src/snapshot/repository/backends/oss/repository.rs` (+1 -0) 📝 `storage/overlaybd/src/backend/oss.rs` (+85 -2) 📝 `storage/overlaybd/src/config.rs` (+13 -0) ➕ `storage/overlaybd/tests/oss_addressing_style.rs` (+100 -0) 📝 `storage/overlaybd/tests/oss_backend_minio.rs` (+2 -0) 📝 `tests/snapshot_oss_e2e_test.rs` (+1 -0) </details> ### 📄 Description ## What problem does this solve? I tried to point the OSS snapshot backend at an S3-compatible provider that requires virtual-host bucket addressing and found there was no way to make it work. The backend picks the addressing style automatically in `detect_addressing_style`: Alibaba OSS endpoints and endpoints that already contain the bucket name get virtual-host style (`https://bucket.example.com/key`), and everything else falls back to path style (`https://example.com/bucket/key`). That fallback is the problem — a growing set of providers only reliably support virtual-host addressing (AWS S3 buckets created after the path-style deprecation, Cloudflare R2, Tigris), and for those the auto-detection guesses wrong with no way to override it. The result is confusing request failures at runtime rather than anything that points at addressing style. ## What this PR does Adds an optional `addressing_style` field to `[backend.oss]`: ```toml [backend.oss] endpoint = "https://t3.storage.dev" bucket = "agentenv-snapshots" region = "auto" addressing_style = "virtual" # or "path" ``` The behavior is strictly opt-in: - **When set**, the explicit value wins and is passed straight through to the object-store client. - **When unset**, the existing `detect_addressing_style` auto-detection runs exactly as before — zero behavior change for current Alibaba OSS / MinIO setups. The plumbing is small: a new `OssAddressingStyle` config enum, a mapping in `NormalizedOssConfig`, and an override-or-detect resolution in `OssClient::new`. ## Testing - Unit tests covering: the detection fallback for generic endpoints, the existing Aliyun/bucket-in-endpoint detection, the override taking precedence, and the config mapping. - Verified end-to-end against a real Tigris bucket with the included `tigris_smoke` example, which drives the same `build_object_store_operator` path the backend uses: write → read-back → verify → delete round-trip succeeds with `addressing_style = "virtual"`. - Also confirmed at the wire level (via a local Host-header echo endpoint) that the two styles genuinely produce different requests — `Host: bucket.endpoint` vs `/bucket/key` on the path — so the override reaches the wire rather than being silently dropped. - `cargo fmt`, `cargo clippy -- -D warnings`, and `cargo test -p object-store-operator` are clean. ## Docs - `docs/src/configuration/reference.md`: new row in the `[backend.oss]` table plus a short worked example. - `docs/src/getting-started/on-demand-loading.md`: one-line pointer after the OSS example config, since anyone following that guide with a virtual-host-only provider would otherwise hit an opaque failure. - Commented-out example in `config/default.toml`. Happy to drop the smoke example if you'd rather keep `examples/` lean --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:19:58 +00:00