[GH-ISSUE #151] Gateway blocks sandbox-routed envd /metrics requests #50

New issue

Closed

opened 2026-10-29 14:18:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:18:49 +00:00

Owner

Originally created by @feimingxliu on GitHub (Aug 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/151

Affected component

Gateway / scheduler

AgentENV version

Version or commit: v0.1.2 / db1492b7915a408b37f863c9e3a34b2ccb2fb1b0
Local modifications: none
The relevant unconditional route guard is also present in the initial v0.1.0 release.

Host environment

Not host-specific. The request is rejected by the gateway HTTP handler before scheduler lookup or node proxying, so no Firecracker, kernel, architecture, ublk, or storage condition is required to reproduce it.

Operation and configuration

curl -i "${GATEWAY_URL}/metrics" \
  -H "x-agentenv-sandbox-id: ${SANDBOX_ID}" \
  -H "x-agentenv-target-port: 49983"

The same problem applies to host-based sandbox routing when the requested path is exactly /metrics.

Steps to reproduce

Start a multi-node AgentENV deployment with gateway, scheduler, and at least one runtime node.
Create a running sandbox and record its sandbox ID.
Verify another envd endpoint such as /health can be reached through the gateway with the sandbox routing headers and target port 49983.
Send the request above to the exact /metrics path.
Observe that the gateway returns 404 without issuing LookupNode or forwarding the request to the runtime node.

Expected behavior

When /metrics carries valid sandbox routing headers, or is addressed through a configured sandbox proxy host, the gateway should treat it as sandbox data-plane traffic:

gateway /metrics
  -> runtime node /proxy/metrics
  -> sandbox <interaction-ip>:49983/metrics

The envd response should be returned to the client. An unrouted request to the gateway HTTP listener at /metrics should continue to return 404. Gateway Prometheus metrics should remain available only from the separate metrics listener, which defaults to :9102.

Actual behavior

The gateway handler checks the exact path before any routing decision:

if r.URL.Path == "/metrics" {
    http.NotFound(w, r)
    return
}

This unconditional guard runs before host-route parsing, sandbox-header parsing, scheduler lookup, and upstream proxying. Consequently, a valid sandbox-routed envd metrics request always receives:

HTTP/1.1 404 Not Found
404 page not found

The later data-plane path rewriting already maps header-based and host-based requests to the runtime node /proxy subtree, and the node proxy strips that prefix before forwarding to the requested sandbox port. The early gateway guard is therefore the blocking point.

Logs and diagnostics

No runtime-node or envd request log is produced because the gateway terminates the request locally. The behavior follows deterministically from services/gateway/internal/server.go::Server.Handler.

Reproduction frequency

Always

Regression information

Last known good: none known
First known bad: v0.1.0 initial open-source release

Suggested acceptance criteria

Header-routed GET /metrics reaches envd on the requested sandbox port.
Host-routed GET /metrics reaches envd on the host-derived sandbox port.
Unrouted gateway GET /metrics remains unavailable on the public HTTP listener.
Missing or malformed sandbox routing information returns the same stable client errors used by other data-plane requests.
Sandbox-routed /metrics is included in gateway proxy instrumentation; only a truly local/unrouted /metrics request bypasses instrumentation.
Tests cover header routing, host routing, unrouted access, and malformed routing headers.

Additional context

Related to #5, which tracks the broader E2B API compatibility matrix and marks GET /sandboxes/{sandboxID}/metrics and GET /sandboxes/metrics as unsupported. This report is narrower and does not propose implementing those E2B endpoints; it concerns routing the existing raw envd GET /metrics endpoint through the generic sandbox data-plane proxy.

Pre-submission checklist

I searched existing open and closed issues and did not find a duplicate.
I provided a minimal reproduction.
I removed credentials, tokens, private image references, and other sensitive information.
This is not a security vulnerability.

Originally created by @feimingxliu on GitHub (Aug 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/151 ## Affected component Gateway / scheduler ## AgentENV version - Version or commit: v0.1.2 / `db1492b7915a408b37f863c9e3a34b2ccb2fb1b0` - Local modifications: none - The relevant unconditional route guard is also present in the initial v0.1.0 release. ## Host environment Not host-specific. The request is rejected by the gateway HTTP handler before scheduler lookup or node proxying, so no Firecracker, kernel, architecture, ublk, or storage condition is required to reproduce it. ## Operation and configuration ```shell curl -i "${GATEWAY_URL}/metrics" \ -H "x-agentenv-sandbox-id: ${SANDBOX_ID}" \ -H "x-agentenv-target-port: 49983" ``` The same problem applies to host-based sandbox routing when the requested path is exactly `/metrics`. ## Steps to reproduce 1. Start a multi-node AgentENV deployment with gateway, scheduler, and at least one runtime node. 2. Create a running sandbox and record its sandbox ID. 3. Verify another envd endpoint such as `/health` can be reached through the gateway with the sandbox routing headers and target port `49983`. 4. Send the request above to the exact `/metrics` path. 5. Observe that the gateway returns 404 without issuing `LookupNode` or forwarding the request to the runtime node. ## Expected behavior When `/metrics` carries valid sandbox routing headers, or is addressed through a configured sandbox proxy host, the gateway should treat it as sandbox data-plane traffic: ```text gateway /metrics -> runtime node /proxy/metrics -> sandbox <interaction-ip>:49983/metrics ``` The envd response should be returned to the client. An unrouted request to the gateway HTTP listener at `/metrics` should continue to return 404. Gateway Prometheus metrics should remain available only from the separate metrics listener, which defaults to `:9102`. ## Actual behavior The gateway handler checks the exact path before any routing decision: ```go if r.URL.Path == "/metrics" { http.NotFound(w, r) return } ``` This unconditional guard runs before host-route parsing, sandbox-header parsing, scheduler lookup, and upstream proxying. Consequently, a valid sandbox-routed envd metrics request always receives: ```text HTTP/1.1 404 Not Found 404 page not found ``` The later data-plane path rewriting already maps header-based and host-based requests to the runtime node `/proxy` subtree, and the node proxy strips that prefix before forwarding to the requested sandbox port. The early gateway guard is therefore the blocking point. ## Logs and diagnostics No runtime-node or envd request log is produced because the gateway terminates the request locally. The behavior follows deterministically from `services/gateway/internal/server.go::Server.Handler`. ## Reproduction frequency Always ## Regression information - Last known good: none known - First known bad: v0.1.0 initial open-source release ## Suggested acceptance criteria - Header-routed `GET /metrics` reaches envd on the requested sandbox port. - Host-routed `GET /metrics` reaches envd on the host-derived sandbox port. - Unrouted gateway `GET /metrics` remains unavailable on the public HTTP listener. - Missing or malformed sandbox routing information returns the same stable client errors used by other data-plane requests. - Sandbox-routed `/metrics` is included in gateway proxy instrumentation; only a truly local/unrouted `/metrics` request bypasses instrumentation. - Tests cover header routing, host routing, unrouted access, and malformed routing headers. ## Additional context Related to #5, which tracks the broader E2B API compatibility matrix and marks `GET /sandboxes/{sandboxID}/metrics` and `GET /sandboxes/metrics` as unsupported. This report is narrower and does not propose implementing those E2B endpoints; it concerns routing the existing raw envd `GET /metrics` endpoint through the generic sandbox data-plane proxy. ## Pre-submission checklist - [x] I searched existing open and closed issues and did not find a duplicate. - [x] I provided a minimal reproduction. - [x] I removed credentials, tokens, private image references, and other sensitive information. - [x] This is not a security vulnerability.

bsakaguchi

2026-10-29 14:18:49 +00:00