[PR #303] [MERGED] fix(overlaybd): dispatch remote image I/O onto a dedicated remote-io runtime #331

New issue

Closed

opened 2026-10-29 14:22:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/303
Author: @huajq
Created: 9/18/2026
Status: ✅ Merged
Merged: 9/21/2026
Merged by: @yingdi-shan

Base: main ← Head: fix/remote-io-runtime

📝 Commits (1)

574ec18 fix(overlaybd): dispatch remote image I/O onto a dedicated remote-io runtime

📊 Changes

10 files changed (+550 additions, -100 deletions)

View changed files

📝 config/default.toml (+1 -0)
📝 docs/src/internals/architecture.md (+1 -0)
📝 src/cfg.rs (+8 -0)
📝 src/setup/deps.rs (+3 -1)
📝 storage/overlaybd/src/backend/cache/full_file_cache/cache_store.rs (+4 -2)
📝 storage/overlaybd/src/backend/oss.rs (+0 -71)
📝 storage/overlaybd/src/config.rs (+8 -0)
📝 storage/overlaybd/src/image/image_service.rs (+216 -26)
➕ storage/overlaybd/src/io/dispatch_file.rs (+308 -0)
📝 storage/overlaybd/src/io/mod.rs (+1 -0)

📄 Description

What

Fix a runtime-ownership bug in remote image I/O: each ublk device queue runs its own current_thread tokio runtime that is dropped on device teardown, while opendal (GLOBAL_REQWEST_CLIENT) and the cached operators share one process-wide reqwest connection pool. hyper spawns per-connection driver tasks onto whatever runtime polls a request, so pooled connections ended up owned by a random device's runtime — reads from other devices were coupled to that device's scheduling, and device teardown killed connections other devices were actively using.

The fix introduces RuntimeDispatchFile and routes all remote network I/O through a fixed runtime owned by ImageService:

open_backend_source_with_size runs the whole open (including the eager size fetch in RegistryFsV2::open) on the pinned runtime and wraps the opened OSS/registryfs files, so no request or steady-state read is ever polled on a caller's runtime regardless of what the open path does;
the lazy remote-runtime construction and background-download submission also run there, so the cache's internal tokio::spawn calls (download scheduler, eviction/checkpoint workers) bind to it without the cache internals needing to know;
enabled via ublk.overlaybd.remote_io_workers (default 4, validated > 0; emitted as remoteIoWorkers into generated overlaybd global configs). With it unset/zero (e.g. unit tests), remote I/O runs on the runtime that constructed the ImageService;
the custom streaming read_at_into override on OssFile is dropped — the VirtualFile trait default (read_at + copy) now serves those callers.

Why

Pooled HTTP connections were being destroyed by unrelated ublk device teardowns, and cross-device reads were scheduled onto the wrong queue runtime — a correctness/stability bug for concurrent devices with remote layers (OSS snapshots, registry images).

N/A

Scope and non-goals

Included: runtime dispatch for remote I/O, the remote_io_workers config, the read_at_into override removal, docs note in docs/src/internals/architecture.md. Excluded: any behavior change to cache layout, wire protocols, or image formats.

Design and behavior changes

Remote opens and steady-state remote reads are always polled on the pinned per-ImageService runtime; device queue runtimes only see completion of dispatched futures.
The remote runtime's lazy construction and the background-download scheduler submission are also driven on the pinned runtime, binding all cache-internal spawns there.
Peak memory for large buffered source reads changes with the read_at_into removal (whole-range buffer + one copy instead of streamed chunks).

Compatibility and operations

Public API or generated protocol: N/A.
Configuration or defaults: new ublk.overlaybd.remote_io_workers (default 4, must be > 0).
Snapshot manifest, artifact layout, or storage format: N/A.
Upgrade and rollback: no migration; teardown behavior for pooled connections is strictly safer.
Host requirements, permissions, ports, or dependencies: N/A.

Validation

make fmt
make clippy
make test-unit (focused suites)
Relevant Rust integration tests (not run; needs KVM/ublk host)
make -C services test — N/A (services/ untouched)
Generated clients/server regenerated — N/A
Documentation updated

Commands and results:

cargo check --workspace        # clean
cargo test -p overlaybd        # 366 passed
cargo test -p agentenv --lib   # 888 passed
make clippy                    # clean
cargo fmt --all -- --check     # clean

Skipped checks and reasons: integration tests (no KVM on the build host used); the fix is exercised by unit tests.

Risks and reviewer notes

remote_io_workers worker count is per ImageService instance (the daemon holds one shared instance).
The read_at_into removal trades a little peak memory for the simpler default path; the dispatch runtime makes request lifetimes uniform regardless.
Suggested review order: storage/overlaybd/src/io/dispatch_file.rs, then image_service.rs, then backend/oss.rs.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/303 **Author:** [@huajq](https://github.com/huajq) **Created:** 9/18/2026 **Status:** ✅ Merged **Merged:** 9/21/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `fix/remote-io-runtime` --- ### 📝 Commits (1) - [`574ec18`](https://github.com/kvcache-ai/AgentENV/commit/574ec18c44e376a0b0a015c7dc9e072f18192ea9) fix(overlaybd): dispatch remote image I/O onto a dedicated remote-io runtime ### 📊 Changes **10 files changed** (+550 additions, -100 deletions) <details> <summary>View changed files</summary> 📝 `config/default.toml` (+1 -0) 📝 `docs/src/internals/architecture.md` (+1 -0) 📝 `src/cfg.rs` (+8 -0) 📝 `src/setup/deps.rs` (+3 -1) 📝 `storage/overlaybd/src/backend/cache/full_file_cache/cache_store.rs` (+4 -2) 📝 `storage/overlaybd/src/backend/oss.rs` (+0 -71) 📝 `storage/overlaybd/src/config.rs` (+8 -0) 📝 `storage/overlaybd/src/image/image_service.rs` (+216 -26) ➕ `storage/overlaybd/src/io/dispatch_file.rs` (+308 -0) 📝 `storage/overlaybd/src/io/mod.rs` (+1 -0) </details> ### 📄 Description ## What Fix a runtime-ownership bug in remote image I/O: each ublk device queue runs its own `current_thread` tokio runtime that is dropped on device teardown, while opendal (`GLOBAL_REQWEST_CLIENT`) and the cached operators share one process-wide reqwest connection pool. hyper spawns per-connection driver tasks onto whatever runtime polls a request, so pooled connections ended up owned by a random device's runtime — reads from other devices were coupled to that device's scheduling, and device teardown killed connections other devices were actively using. The fix introduces `RuntimeDispatchFile` and routes all remote network I/O through a fixed runtime owned by `ImageService`: - `open_backend_source_with_size` runs the whole open (including the eager size fetch in `RegistryFsV2::open`) on the pinned runtime and wraps the opened OSS/registryfs files, so no request or steady-state read is ever polled on a caller's runtime regardless of what the open path does; - the lazy remote-runtime construction and background-download submission also run there, so the cache's internal `tokio::spawn` calls (download scheduler, eviction/checkpoint workers) bind to it without the cache internals needing to know; - enabled via `ublk.overlaybd.remote_io_workers` (default 4, validated > 0; emitted as `remoteIoWorkers` into generated overlaybd global configs). With it unset/zero (e.g. unit tests), remote I/O runs on the runtime that constructed the `ImageService`; - the custom streaming `read_at_into` override on `OssFile` is dropped — the `VirtualFile` trait default (`read_at` + copy) now serves those callers. ## Why Pooled HTTP connections were being destroyed by unrelated ublk device teardowns, and cross-device reads were scheduled onto the wrong queue runtime — a correctness/stability bug for concurrent devices with remote layers (OSS snapshots, registry images). ## Related issue N/A ## Scope and non-goals Included: runtime dispatch for remote I/O, the `remote_io_workers` config, the `read_at_into` override removal, docs note in `docs/src/internals/architecture.md`. Excluded: any behavior change to cache layout, wire protocols, or image formats. ## Design and behavior changes - Remote opens and steady-state remote reads are always polled on the pinned per-`ImageService` runtime; device queue runtimes only see completion of dispatched futures. - The remote runtime's lazy construction and the background-download scheduler submission are also driven on the pinned runtime, binding all cache-internal spawns there. - Peak memory for large buffered source reads changes with the `read_at_into` removal (whole-range buffer + one copy instead of streamed chunks). ## Compatibility and operations - Public API or generated protocol: N/A. - Configuration or defaults: new `ublk.overlaybd.remote_io_workers` (default 4, must be > 0). - Snapshot manifest, artifact layout, or storage format: N/A. - Upgrade and rollback: no migration; teardown behavior for pooled connections is strictly safer. - Host requirements, permissions, ports, or dependencies: N/A. ## Validation - [x] `make fmt` - [x] `make clippy` - [x] `make test-unit` (focused suites) - [ ] Relevant Rust integration tests (not run; needs KVM/ublk host) - [ ] `make -C services test` — N/A (`services/` untouched) - [ ] Generated clients/server regenerated — N/A - [x] Documentation updated Commands and results: ```text cargo check --workspace # clean cargo test -p overlaybd # 366 passed cargo test -p agentenv --lib # 888 passed make clippy # clean cargo fmt --all -- --check # clean ``` Skipped checks and reasons: integration tests (no KVM on the build host used); the fix is exercised by unit tests. ## Risks and reviewer notes - `remote_io_workers` worker count is per `ImageService` instance (the daemon holds one shared instance). - The `read_at_into` removal trades a little peak memory for the simpler default path; the dispatch runtime makes request lifetimes uniform regardless. - Suggested review order: `storage/overlaybd/src/io/dispatch_file.rs`, then `image_service.rs`, then `backend/oss.rs`. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:22:16 +00:00