[PR #178] [MERGED] feat(observability): add overlaybd/ext4 usage gauges on restack snapshot #231

New issue

Closed

opened 2026-10-29 14:21:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:16 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/178
Author: @huajq
Created: 8/17/2026
Status: ✅ Merged
Merged: 8/18/2026
Merged by: @yingdi-shan

Base: main ← Head: feat/overlaybd-usage-metrics

📝 Commits (1)

70fae06 feat(observability): add overlaybd/ext4 usage gauges on restack snapshot

📊 Changes

12 files changed (+325 additions, -24 deletions)

View changed files

📝 src/sandbox/firecracker/overlaybd_snapshot.rs (+5 -1)
📝 src/sandbox/firecracker/sandbox.rs (+1 -0)
📝 src/sandbox/ublk/device.rs (+38 -6)
➕ storage/overlaybd/src/ext4_stat.rs (+121 -0)
📝 storage/overlaybd/src/image/image_file.rs (+60 -1)
📝 storage/overlaybd/src/lib.rs (+1 -0)
📝 storage/overlaybd/src/lsmt/file/types.rs (+1 -1)
📝 storage/ublk-daemon/src/client.rs (+13 -4)
📝 storage/ublk-daemon/src/lib.rs (+3 -1)
📝 storage/ublk-daemon/src/protocol.rs (+63 -4)
📝 storage/ublk-daemon/src/server.rs (+10 -1)
📝 storage/ublk-daemon/tests/ublk_daemon_test.rs (+9 -5)

📄 Description

What

Report per-device usage statistics on the restack-snapshot RPC and export them as Prometheus gauges, so overlaybd layer bloat and TRIM headroom become observable without any guest cooperation:

agentenv_overlaybd_data_bytes{dev_id, kind, stat="valid"} — unique bytes mapped by the device's merged index (≈ what a compact/snapshot of this device would carry).
agentenv_ext4_used_bytes{dev_id, kind} — guest ext4 used bytes probed from the block view (= df "Used").
agentenv_trim_reclaimable_bytes{dev_id, kind} — the difference, i.e. how much a working discard/TRIM path could reclaim.

Why

Block-layer mappings in overlaybd grow monotonically because deletions never reach the backend (Firecracker's virtio-block does not advertise discard). Before investing in discard support we need to quantify the potential savings per sandbox, and afterwards verify them. Today there is no node-local way to see either "how much of a layer is actually referenced" or "how much the guest filesystem actually uses". Sampling piggybacks on the existing restack RPC (once per snapshot per device), so there is no new hot-path work and no polling loop.

N/A (no tracking issue; root-caused from production incident analysis)

Scope and non-goals

Included:

ImageFile::data_stat() passthrough (merged-index mapped bytes; cheap in-memory index scan reusing the existing LSMT data_stat).
New ext4_stat module: best-effort used-bytes probe from the ext4 superblock free-block counter (same quantity as df "Used"; ~130 KB of metadata reads; non-ext4 views yield no sample).
Daemon RestackSnapshotCreated response carries optional data_stat / ext4_used_bytes.
Server records the three gauges on successful restacks, with kind="rootfs"|"drive" labels threaded from the call sites.

Excluded:

Firecracker virtio-block discard support itself (follow-up that these metrics are designed to size and validate).
Layer physical-size ("total") and whole-chain physical metrics (evaluated during review and intentionally dropped as out of scope).

Design and behavior changes

Sampling happens once per restack snapshot per device; failures to compute stats never fail the restack (fields are simply absent).
The ext4 probe reads only the superblock (and counters may lag a mounted filesystem by a checkpoint — acceptable for observability).
New metric series use dev_id labels (a few hundred devices per node; cardinality is bounded by the device count).

Compatibility and operations

Public API or generated protocol: N/A for the public HTTP API. The internal daemon IPC gains optional response fields with #[serde(default)], so new server ↔ old daemon and old server ↔ new daemon both decode fine (rolling-upgrade safe; covered by a protocol test).
Configuration or defaults: N/A.
Snapshot manifest, artifact layout, or storage format: N/A.
Upgrade and rollback: safe; rollback only removes the new gauge series.
Host requirements, permissions, ports, or dependencies: N/A.

Validation

make fmt (cargo fmt --all -- --check)
make clippy (cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets -- -D warnings)
make test-unit equivalent: see commands below

Commands and results:

cargo test -p overlaybd --lib
  → 314 passed; 0 failed (incl. 3 new ext4_stat tests and the
    ImageFile::data_stat passthrough test)
cargo test -p uvm-ublk-daemon
  → 37 passed; 0 failed (incl. old-daemon response compatibility test)
cargo test -p agentenv --lib
  → 723 passed; 2 failed (both pre-existing environment issues:
     privileges capability test, acr docker-credential-helper test;
     unrelated to this change)

Skipped checks and reasons:

make -C services test: no services/ changes.
Generated clients/server: daemon protocol is hand-written serde (no codegen involved).
Documentation: no user-facing docs affected.
Benchmarks: not applicable (observability-only; measurement cost is an in-memory index scan plus ~130 KB of metadata reads, once per snapshot).

Risks and reviewer notes

Low operational risk: all stats collection is best-effort and off the request-critical path; a failure shows up only as a missing sample.
ext4_used_bytes is a heuristic (superblock counters, no journal replay); it is meant for observability, not billing-grade accuracy.
Worth a quick look: the kind label threading through restack_snapshot_overlaybd_device call sites (rootfs vs attached drives).

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/178 **Author:** [@huajq](https://github.com/huajq) **Created:** 8/17/2026 **Status:** ✅ Merged **Merged:** 8/18/2026 **Merged by:** [@yingdi-shan](https://github.com/yingdi-shan) **Base:** `main` ← **Head:** `feat/overlaybd-usage-metrics` --- ### 📝 Commits (1) - [`70fae06`](https://github.com/kvcache-ai/AgentENV/commit/70fae0617afe4425645e6320c96d5d669873f5c6) feat(observability): add overlaybd/ext4 usage gauges on restack snapshot ### 📊 Changes **12 files changed** (+325 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `src/sandbox/firecracker/overlaybd_snapshot.rs` (+5 -1) 📝 `src/sandbox/firecracker/sandbox.rs` (+1 -0) 📝 `src/sandbox/ublk/device.rs` (+38 -6) ➕ `storage/overlaybd/src/ext4_stat.rs` (+121 -0) 📝 `storage/overlaybd/src/image/image_file.rs` (+60 -1) 📝 `storage/overlaybd/src/lib.rs` (+1 -0) 📝 `storage/overlaybd/src/lsmt/file/types.rs` (+1 -1) 📝 `storage/ublk-daemon/src/client.rs` (+13 -4) 📝 `storage/ublk-daemon/src/lib.rs` (+3 -1) 📝 `storage/ublk-daemon/src/protocol.rs` (+63 -4) 📝 `storage/ublk-daemon/src/server.rs` (+10 -1) 📝 `storage/ublk-daemon/tests/ublk_daemon_test.rs` (+9 -5) </details> ### 📄 Description ## What Report per-device usage statistics on the restack-snapshot RPC and export them as Prometheus gauges, so overlaybd layer bloat and TRIM headroom become observable without any guest cooperation: - `agentenv_overlaybd_data_bytes{dev_id, kind, stat="valid"}` — unique bytes mapped by the device's merged index (≈ what a compact/snapshot of this device would carry). - `agentenv_ext4_used_bytes{dev_id, kind}` — guest ext4 used bytes probed from the block view (= `df` "Used"). - `agentenv_trim_reclaimable_bytes{dev_id, kind}` — the difference, i.e. how much a working discard/TRIM path could reclaim. ## Why Block-layer mappings in overlaybd grow monotonically because deletions never reach the backend (Firecracker's virtio-block does not advertise discard). Before investing in discard support we need to quantify the potential savings per sandbox, and afterwards verify them. Today there is no node-local way to see either "how much of a layer is actually referenced" or "how much the guest filesystem actually uses". Sampling piggybacks on the existing restack RPC (once per snapshot per device), so there is no new hot-path work and no polling loop. ## Related issue N/A (no tracking issue; root-caused from production incident analysis) ## Scope and non-goals Included: - `ImageFile::data_stat()` passthrough (merged-index mapped bytes; cheap in-memory index scan reusing the existing LSMT `data_stat`). - New `ext4_stat` module: best-effort used-bytes probe from the ext4 superblock free-block counter (same quantity as `df` "Used"; ~130 KB of metadata reads; non-ext4 views yield no sample). - Daemon `RestackSnapshotCreated` response carries optional `data_stat` / `ext4_used_bytes`. - Server records the three gauges on successful restacks, with `kind="rootfs"|"drive"` labels threaded from the call sites. Excluded: - Firecracker virtio-block discard support itself (follow-up that these metrics are designed to size and validate). - Layer physical-size ("total") and whole-chain physical metrics (evaluated during review and intentionally dropped as out of scope). ## Design and behavior changes - Sampling happens once per restack snapshot per device; failures to compute stats never fail the restack (fields are simply absent). - The ext4 probe reads only the superblock (and counters may lag a mounted filesystem by a checkpoint — acceptable for observability). - New metric series use `dev_id` labels (a few hundred devices per node; cardinality is bounded by the device count). ## Compatibility and operations - Public API or generated protocol: N/A for the public HTTP API. The internal daemon IPC gains optional response fields with `#[serde(default)]`, so new server ↔ old daemon and old server ↔ new daemon both decode fine (rolling-upgrade safe; covered by a protocol test). - Configuration or defaults: N/A. - Snapshot manifest, artifact layout, or storage format: N/A. - Upgrade and rollback: safe; rollback only removes the new gauge series. - Host requirements, permissions, ports, or dependencies: N/A. ## Validation - [x] `make fmt` (cargo fmt --all -- --check) - [x] `make clippy` (cargo clippy -p overlaybd -p uvm-ublk-daemon -p agentenv --all-targets -- -D warnings) - [x] `make test-unit` equivalent: see commands below Commands and results: ```text cargo test -p overlaybd --lib → 314 passed; 0 failed (incl. 3 new ext4_stat tests and the ImageFile::data_stat passthrough test) cargo test -p uvm-ublk-daemon → 37 passed; 0 failed (incl. old-daemon response compatibility test) cargo test -p agentenv --lib → 723 passed; 2 failed (both pre-existing environment issues: privileges capability test, acr docker-credential-helper test; unrelated to this change) ``` Skipped checks and reasons: - `make -C services test`: no `services/` changes. - Generated clients/server: daemon protocol is hand-written serde (no codegen involved). - Documentation: no user-facing docs affected. - Benchmarks: not applicable (observability-only; measurement cost is an in-memory index scan plus ~130 KB of metadata reads, once per snapshot). ## Risks and reviewer notes - Low operational risk: all stats collection is best-effort and off the request-critical path; a failure shows up only as a missing sample. - `ext4_used_bytes` is a heuristic (superblock counters, no journal replay); it is meant for observability, not billing-grade accuracy. - Worth a quick look: the `kind` label threading through `restack_snapshot_overlaybd_device` call sites (rootfs vs attached drives). ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:16 +00:00