[GH-ISSUE #294] Add lifecycle-aware per-sandbox resource metrics collection #78

New issue

Closed

opened 2026-10-29 14:19:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:17 +00:00

Owner

Originally created by @dushulin on GitHub (Sep 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/294

Affected component

Observability

Problem statement

Operators need to identify which running sandbox is CPU-bound, short of guest memory, or running out of filesystem space, and distinguish those measurements from its resource allocation and host-side footprint. Node aggregates and API latency histograms cannot answer these questions for a particular sandbox.

#5 already records that the sandbox metrics collector is not implemented, while raw envd metrics are accessible. This request narrows that compatibility gap into a resource-collection feature with lifecycle and sampling requirements. It does not request the whole E2B API surface or a new monitoring database.

Use case

A multi-node deployment runs many sandboxes on each host. Sandboxes can pause, resume into a different Firecracker process, and share snapshot-backed memory or storage. Operators need periodic resource measurements keyed by sandbox ID without manually discovering each guest endpoint or treating a node's aggregate usage as the sandbox's usage.

Current behavior and workarounds

Reviewed upstream ff148136aebb09705bd2a6a3a40b79368831a465:

Raw envd /metrics provides guest CPU, memory and root filesystem measurements. #159 fixed gateway access to that endpoint; it explicitly did not add a collector or the E2B sandbox metrics APIs.
Node host metrics and sandbox resource allocations are exposed by the existing observability subsystem. They are different accounting scopes from per-sandbox actual usage.
#178 added per-device OverlayBD/ext4 gauges sampled during restack. These are useful storage diagnostics, but are not continuous per-sandbox resource samples.
#47 already discusses per-device logical/local/remote I/O accounting, page-cache effects and a possible cgroup-based redesign. That work should be coordinated separately rather than duplicated here.

Current workarounds are direct envd requests and manual host PID/device inspection. They leave users responsible for sampling, runtime identity, stale data, and counter continuity across pause/resume.

Desired behavior

For a running sandbox, export documented guest CPU, memory and root filesystem measurements associated with its sandbox and node IDs. Clearly distinguish allocation, guest usage, and any host-process CPU/RSS/PSS measurements.
Include sample time or age and collection success. An unreachable guest produces an unavailable/stale result, not a synthetic zero.
Sampling never resumes a paused sandbox or changes its TTL. Paused and deleted sandboxes stop guest collection; old runtime samples cannot be attributed to a new process or reused device.
Handle pause/resume, fork, process exit, and PID/device reuse explicitly. Document counter resets and remove obsolete active series.
Bound collection concurrency and frequency, and serve cached results to scrapers so each scrape does not fan out to all guests. Multiple readers should not unintentionally change the effective guest CPU sampling window.
Preserve existing authorization boundaries. Byte fields must support common guest sizes above 2 GiB without integer overflow.
Provide example queries, accounting definitions, and measured collection overhead for idle and loaded sandboxes before choosing the default interval. History can be retained by an existing Prometheus-compatible monitoring system.

Proposed approach

A node-local background collector can use the orchestrator's authoritative sandbox/runtime associations and reuse the existing metrics endpoint. Start with envd for guest CPU/memory/root filesystem data, and separately named host-process metrics where supported. Host PID statistics are distinct from Firecracker's native JSON metrics.

Keep only technical identity dimensions such as node, sandbox, runtime generation and drive where needed. A configurable, staggered interval in the 15–30 second range is a starting point for measurement, not an established overhead guarantee. More expensive PSS reads can have a separate cadence.

Firecracker-native block/network/error/latency metrics and per-drive I/O accounting can follow independently. They do not replace guest filesystem or memory measurements; physical storage attribution should follow the discussion in #47.

Compatibility and operational impact

API/config changes: additive, optional collector configuration and documented metrics; keep existing endpoints and authentication behavior. Full E2B metrics API compatibility can be a separate follow-up to #5.

Snapshot or storage format changes: none required for an initial collector. Runtime associations can remain node-local and be rebuilt with each runtime.

New host/runtime requirements: reuse existing envd connectivity and permitted read-only process statistics. Do not make a cgroup I/O worker redesign or Firecracker metrics configuration a prerequisite. Unavailable sources must be reported explicitly.

Upgrade and rollback considerations: support disabling collection without changing sandbox lifecycle behavior; ensure stale series and runtime resets are handled during restart/upgrade.

Alternatives considered

Direct envd polling from every consumer: useful for diagnosis, but duplicates sampling and lifecycle handling and can alter CPU sample windows.
Firecracker native metrics alone: useful for VMM/device internals; does not supply the same guest filesystem/memory view and requires handling its flush/counter semantics.
Pod/node monitoring alone: combines the AgentENV process, multiple VMs and shared services, so it does not identify an individual sandbox's usage.
Restack gauges alone: event-triggered device statistics do not provide continuous resource sampling.

Non-goals

Business metadata such as created-by, authors, projects or tenants; these are not generic collector fields or labels.
Billing, quota enforcement, a new time-series database, a Web UI, or a durable lifecycle audit store.
Reimplementing the I/O scheduling/physical-accounting work discussed in #47.

Contribution

Design context and acceptance criteria are included here; implementation ownership is not committed. If a dedicated collector effort already exists beyond #5, please link it so this request can be aligned with that work.

Pre-submission checklist

I searched open and closed issues, PRs, and Discussions and found no dedicated duplicate; related tracking in #5 and #47 is described above.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @dushulin on GitHub (Sep 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/294 ### Affected component Observability ### Problem statement Operators need to identify which running sandbox is CPU-bound, short of guest memory, or running out of filesystem space, and distinguish those measurements from its resource allocation and host-side footprint. Node aggregates and API latency histograms cannot answer these questions for a particular sandbox. #5 already records that the sandbox metrics collector is not implemented, while raw envd metrics are accessible. This request narrows that compatibility gap into a resource-collection feature with lifecycle and sampling requirements. It does not request the whole E2B API surface or a new monitoring database. ### Use case A multi-node deployment runs many sandboxes on each host. Sandboxes can pause, resume into a different Firecracker process, and share snapshot-backed memory or storage. Operators need periodic resource measurements keyed by sandbox ID without manually discovering each guest endpoint or treating a node's aggregate usage as the sandbox's usage. ### Current behavior and workarounds Reviewed upstream `ff148136aebb09705bd2a6a3a40b79368831a465`: - Raw envd `/metrics` provides guest CPU, memory and root filesystem measurements. #159 fixed gateway access to that endpoint; it explicitly did not add a collector or the E2B sandbox metrics APIs. - Node host metrics and sandbox resource allocations are exposed by the existing observability subsystem. They are different accounting scopes from per-sandbox actual usage. - #178 added per-device OverlayBD/ext4 gauges sampled during restack. These are useful storage diagnostics, but are not continuous per-sandbox resource samples. - #47 already discusses per-device logical/local/remote I/O accounting, page-cache effects and a possible cgroup-based redesign. That work should be coordinated separately rather than duplicated here. Current workarounds are direct envd requests and manual host PID/device inspection. They leave users responsible for sampling, runtime identity, stale data, and counter continuity across pause/resume. ### Desired behavior - For a running sandbox, export documented guest CPU, memory and root filesystem measurements associated with its sandbox and node IDs. Clearly distinguish allocation, guest usage, and any host-process CPU/RSS/PSS measurements. - Include sample time or age and collection success. An unreachable guest produces an unavailable/stale result, not a synthetic zero. - Sampling never resumes a paused sandbox or changes its TTL. Paused and deleted sandboxes stop guest collection; old runtime samples cannot be attributed to a new process or reused device. - Handle pause/resume, fork, process exit, and PID/device reuse explicitly. Document counter resets and remove obsolete active series. - Bound collection concurrency and frequency, and serve cached results to scrapers so each scrape does not fan out to all guests. Multiple readers should not unintentionally change the effective guest CPU sampling window. - Preserve existing authorization boundaries. Byte fields must support common guest sizes above 2 GiB without integer overflow. - Provide example queries, accounting definitions, and measured collection overhead for idle and loaded sandboxes before choosing the default interval. History can be retained by an existing Prometheus-compatible monitoring system. ### Proposed approach A node-local background collector can use the orchestrator's authoritative sandbox/runtime associations and reuse the existing metrics endpoint. Start with envd for guest CPU/memory/root filesystem data, and separately named host-process metrics where supported. Host PID statistics are distinct from Firecracker's native JSON metrics. Keep only technical identity dimensions such as node, sandbox, runtime generation and drive where needed. A configurable, staggered interval in the 15–30 second range is a starting point for measurement, not an established overhead guarantee. More expensive PSS reads can have a separate cadence. Firecracker-native block/network/error/latency metrics and per-drive I/O accounting can follow independently. They do not replace guest filesystem or memory measurements; physical storage attribution should follow the discussion in #47. ### Compatibility and operational impact API/config changes: additive, optional collector configuration and documented metrics; keep existing endpoints and authentication behavior. Full E2B metrics API compatibility can be a separate follow-up to #5. Snapshot or storage format changes: none required for an initial collector. Runtime associations can remain node-local and be rebuilt with each runtime. New host/runtime requirements: reuse existing envd connectivity and permitted read-only process statistics. Do not make a cgroup I/O worker redesign or Firecracker metrics configuration a prerequisite. Unavailable sources must be reported explicitly. Upgrade and rollback considerations: support disabling collection without changing sandbox lifecycle behavior; ensure stale series and runtime resets are handled during restart/upgrade. ### Alternatives considered - Direct envd polling from every consumer: useful for diagnosis, but duplicates sampling and lifecycle handling and can alter CPU sample windows. - Firecracker native metrics alone: useful for VMM/device internals; does not supply the same guest filesystem/memory view and requires handling its flush/counter semantics. - Pod/node monitoring alone: combines the AgentENV process, multiple VMs and shared services, so it does not identify an individual sandbox's usage. - Restack gauges alone: event-triggered device statistics do not provide continuous resource sampling. ### Non-goals - Business metadata such as `created-by`, authors, projects or tenants; these are not generic collector fields or labels. - Billing, quota enforcement, a new time-series database, a Web UI, or a durable lifecycle audit store. - Reimplementing the I/O scheduling/physical-accounting work discussed in #47. ### Contribution Design context and acceptance criteria are included here; implementation ownership is not committed. If a dedicated collector effort already exists beyond #5, please link it so this request can be aligned with that work. ### Pre-submission checklist - [x] I searched open and closed issues, PRs, and Discussions and found no dedicated duplicate; related tracking in #5 and #47 is described above. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi closed this issue

2026-10-29 14:19:17 +00:00

bsakaguchi referenced this issue

2026-10-29 14:20:26 +00:00

[PR #76] [1/4] feat(web): scaffold control-plane console and deploy wiring #160

bsakaguchi referenced this issue

2026-10-29 14:20:27 +00:00

[PR #77] [CLOSED] [2/4] feat(web): Gateway connection settings and session cookies #161

bsakaguchi referenced this issue

2026-10-29 14:20:28 +00:00

[PR #78] [CLOSED] [3/4] feat(web): sandbox and snapshot management views #162

bsakaguchi referenced this issue