[PR #436] [CLOSED] feat(ublk): expose device I/O and stalled request metrics #437

New issue

Closed

opened 2026-10-29 14:23:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:23:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/436
Author: @guozy18
Created: 10/25/2026
Status: ❌ Closed

Base: main ← Head: sync/ublk-device-io-metrics

📝 Commits (1)

939c230 feat: add ublk device-level IO metrics

📊 Changes

8 files changed (+634 additions, -3 deletions)

View changed files

📝 Cargo.lock (+3 -0)
📝 crates/observability/src/lib.rs (+16 -0)
📝 docs/src/configuration/reference.md (+15 -0)
📝 storage/ublk-daemon/src/main.rs (+4 -0)
📝 storage/ublk/Cargo.toml (+3 -0)
📝 storage/ublk/src/impls/overlaybd_target.rs (+32 -3)
📝 storage/ublk/src/lib.rs (+2 -0)
➕ storage/ublk/src/metrics.rs (+559 -0)

📄 Description

What

Expose ublk device I/O requests, completed bytes, read/write latency and in-flight operations on the daemon's existing Prometheus endpoint. Series are aggregated by target and operation, without a device-ID label.

Why

Existing storage metrics do not show whether guest block requests complete, fail or remain stuck. Completion latency alone cannot reveal an I/O that never finishes.

No linked issue.

Scope and non-goals

Record OverlayBD target I/O and publish aggregate daemon metrics. Preserve the current I/O results, error mapping, startup-pack recording, warm-pool image switching and storage ownership.

Design and behavior changes

Register fixed-label metric handles when a target is constructed. Successful requests increment request/byte counters; failures carry their errno. Read/write completions record latency with finite buckets through 30 seconds.
A per-target tracker registers reads/writes at their start. Its RAII guard removes the entry on completion or future cancellation.
One scanner on the daemon's main runtime aggregates all live trackers every five seconds. It separates operations pending for at least 30 seconds, prunes weak references to destroyed targets and resets seen gauges to zero when targets disappear.
The existing Prometheus recorder is installed before targets and the scanner are created.

Compatibility and operations

Public API or generated protocol: additive Prometheus series only; no generated protocol changes.
Configuration or defaults: existing ublk.daemon_metrics_listen_addr remains unchanged. An empty address disables export; tracker/scanner bookkeeping still runs.
Snapshot manifest, artifact layout, or storage format: unchanged.
Upgrade and rollback: no persistent-state migration; rollback removes the additional metrics.
Host requirements, permissions, ports, or dependencies: same daemon listener and host requirements; reuses the workspace's metrics, metrics-util and DashMap dependencies.

Validation

make fmt (equivalent cargo fmt --all -- --check)
make clippy (explicit package/all-target/all-feature command below)
make test-unit
Relevant Rust integration tests (separate KVM/Prometheus end-to-end validation below)
make -C services test (no Go changes)
Generated clients/server regenerated with the documented make target (no generated changes)
Documentation updated
Benchmarks or performance comparison completed

Commands and results on isolated Linux, from this branch alone on 424b852:

cargo fmt --all -- --check: passed
cargo clippy --locked -p uvm-ublk --all-targets --all-features -- -D warnings: passed
cargo test --locked -p uvm-ublk --lib metrics: 4 passed
cargo test --locked -p uvm-ublk --lib impls::overlaybd_target: 5 passed
cargo test --locked -p uvm-ublk-daemon --lib: 49 passed
cargo test --locked -p agentenv-observability: passed (no tests defined)
cargo build --locked -p agentenv --bin server -p uvm-ublk-daemon --bin uvm-ublk-daemon: passed

The original tracker tests cover the 30-second boundary, aggregation, guard removal and zeroing. The existing target tests cover device parameters, error mapping and sparse-image discard.

A non-root KVM node was also started with this branch's separate binaries. Real guest boot I/O appeared on the daemon endpoint with read/write counters, bytes, histograms and no dev_id labels. Snapshot publication and launch, pause/resume and pooled device reuse kept accounting active. After sandbox/snapshot deletion all four in-flight gauges returned to zero. Sandboxes, snapshots and processes were cleaned up.

Skipped checks: unrelated aggregate VM/storage suites and Go checks. No throughput benchmark or deliberately stalled 30-second remote I/O experiment was performed; the slow-I/O boundary and cancellation cleanup are covered by deterministic tracker tests.

Risks and reviewer notes

Every read/write adds an insert/remove in the tracker and an unsampled latency observation; steady-state overhead has not been benchmarked. In-flight values are sampled every five seconds. The standalone ublk binary does not start this daemon scanner or Prometheus endpoint.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/436 **Author:** [@guozy18](https://github.com/guozy18) **Created:** 10/25/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `sync/ublk-device-io-metrics` --- ### 📝 Commits (1) - [`939c230`](https://github.com/kvcache-ai/AgentENV/commit/939c23098569cc7af09cba31726ce27305209379) feat: add ublk device-level IO metrics ### 📊 Changes **8 files changed** (+634 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `Cargo.lock` (+3 -0) 📝 `crates/observability/src/lib.rs` (+16 -0) 📝 `docs/src/configuration/reference.md` (+15 -0) 📝 `storage/ublk-daemon/src/main.rs` (+4 -0) 📝 `storage/ublk/Cargo.toml` (+3 -0) 📝 `storage/ublk/src/impls/overlaybd_target.rs` (+32 -3) 📝 `storage/ublk/src/lib.rs` (+2 -0) ➕ `storage/ublk/src/metrics.rs` (+559 -0) </details> ### 📄 Description ## What Expose ublk device I/O requests, completed bytes, read/write latency and in-flight operations on the daemon's existing Prometheus endpoint. Series are aggregated by target and operation, without a device-ID label. ## Why Existing storage metrics do not show whether guest block requests complete, fail or remain stuck. Completion latency alone cannot reveal an I/O that never finishes. ## Related issue No linked issue. ## Scope and non-goals Record OverlayBD target I/O and publish aggregate daemon metrics. Preserve the current I/O results, error mapping, startup-pack recording, warm-pool image switching and storage ownership. ## Design and behavior changes - Register fixed-label metric handles when a target is constructed. Successful requests increment request/byte counters; failures carry their errno. Read/write completions record latency with finite buckets through 30 seconds. - A per-target tracker registers reads/writes at their start. Its RAII guard removes the entry on completion or future cancellation. - One scanner on the daemon's main runtime aggregates all live trackers every five seconds. It separates operations pending for at least 30 seconds, prunes weak references to destroyed targets and resets seen gauges to zero when targets disappear. - The existing Prometheus recorder is installed before targets and the scanner are created. ## Compatibility and operations - Public API or generated protocol: additive Prometheus series only; no generated protocol changes. - Configuration or defaults: existing `ublk.daemon_metrics_listen_addr` remains unchanged. An empty address disables export; tracker/scanner bookkeeping still runs. - Snapshot manifest, artifact layout, or storage format: unchanged. - Upgrade and rollback: no persistent-state migration; rollback removes the additional metrics. - Host requirements, permissions, ports, or dependencies: same daemon listener and host requirements; reuses the workspace's metrics, metrics-util and DashMap dependencies. ## Validation - [x] `make fmt` (equivalent `cargo fmt --all -- --check`) - [x] `make clippy` (explicit package/all-target/all-feature command below) - [ ] `make test-unit` - [ ] Relevant Rust integration tests (separate KVM/Prometheus end-to-end validation below) - [ ] `make -C services test` (no Go changes) - [ ] Generated clients/server regenerated with the documented `make` target (no generated changes) - [x] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results on isolated Linux, from this branch alone on `424b852`: ```text cargo fmt --all -- --check: passed cargo clippy --locked -p uvm-ublk --all-targets --all-features -- -D warnings: passed cargo test --locked -p uvm-ublk --lib metrics: 4 passed cargo test --locked -p uvm-ublk --lib impls::overlaybd_target: 5 passed cargo test --locked -p uvm-ublk-daemon --lib: 49 passed cargo test --locked -p agentenv-observability: passed (no tests defined) cargo build --locked -p agentenv --bin server -p uvm-ublk-daemon --bin uvm-ublk-daemon: passed ``` The original tracker tests cover the 30-second boundary, aggregation, guard removal and zeroing. The existing target tests cover device parameters, error mapping and sparse-image discard. A non-root KVM node was also started with this branch's separate binaries. Real guest boot I/O appeared on the daemon endpoint with read/write counters, bytes, histograms and no `dev_id` labels. Snapshot publication and launch, pause/resume and pooled device reuse kept accounting active. After sandbox/snapshot deletion all four in-flight gauges returned to zero. Sandboxes, snapshots and processes were cleaned up. Skipped checks: unrelated aggregate VM/storage suites and Go checks. No throughput benchmark or deliberately stalled 30-second remote I/O experiment was performed; the slow-I/O boundary and cancellation cleanup are covered by deterministic tracker tests. ## Risks and reviewer notes Every read/write adds an insert/remove in the tracker and an unsampled latency observation; steady-state overhead has not been benchmarked. In-flight values are sampled every five seconds. The standalone ublk binary does not start this daemon scanner or Prometheus endpoint. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:23:23 +00:00