[GH-ISSUE #157] Domain-based allowOut enforcement: offering our TCP egress proxy implementation — design check first #52

New issue

Closed

opened 2026-10-29 14:18:52 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:18:52 +00:00

Owner

Originally created by @sunkencity999 on GitHub (Aug 11, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/157

Context

src/sandbox/network/policy.rs already accepts and validates domain entries in allowOut, then refuses them at runtime:

domain entries in allowOut require the TCP egress proxy, which is not enabled yet: {:?}

So the API surface, the validation, and the error path exist — the enforcement does not. This issue is to ask whether you would welcome that implementation upstream, and to describe the approach before writing a PR against it.

We needed domain-level egress for a deployment where sandboxes must reach a small set of package registries and nothing else, so we built it against v0.1.1 and have been running it in production. Rather than carry the fork indefinitely, we would rather it live here — but only if the design matches where you want the networking layer to go.

Approach

A TCP egress proxy on the node, off by default behind config:

[network.egress.proxy]
enabled = true
http_port = 18080
tls_port = 18443

Each deny-by-default sandbox's outbound tcp/80 and tcp/443 are DNAT-redirected to the proxy, per network slot.
The proxy matches the HTTP Host header (:80) or the TLS SNI (:443) against that sandbox's allowOut domains, supporting exact and *.wildcard entries.
The name is resolved host-side, not by the sandbox. A forged SNI cannot select the upstream address, which closes the obvious DNS-rebinding path.
Matching connections are spliced; non-matching get a 403 on :80 and a closed connection on :443.
Slot teardown flushes rules, including warm slots, so policy does not leak between sandbox generations. This overlaps with the concerns in #56 and #103.

Domain rules only arm under a Deny base policy, which is consistent with how allowOut already behaves for CIDRs.

Honest limitations

Worth stating up front, because they bound what this feature claims:

Non-web ports are not proxied. tcp/80 and tcp/443 only; everything else remains governed by CIDR rules and the base policy.
UDP/QUIC is not proxied. A client that negotiates QUIC bypasses the allowlist unless UDP is otherwise denied.
DNS remains reachable, so DNS tunnelling is an available exfiltration channel. This is the same residual that Unit 42 demonstrated against a comparable "isolated" sandbox mode earlier this year, so it should be documented rather than implied away.
The host firewall must admit the redirected traffic. A default-deny INPUT chain silently blackholes it, and the symptom is confusing: every proxied destination times out, allowed or not, which looks like a broken allowlist rather than a firewall problem. With ufw the rule is scoped to the internal ranges:
```
ufw allow proto tcp from 10.11.0.0/16 to 10.12.0.0/16 port 18080,18443
```
(matching host_interaction_cidr and veth_cidr defaults). We would document this next to the config rather than leave people to discover it.

Size

Roughly 880 lines against v0.1.1: a new src/sandbox/network/egress_proxy.rs (~540) plus wiring in policy.rs, slot.rs, cfg/network.rs, bin/server.rs, the sandbox impl, and docs. Unit tests cover SNI and Host parsing, wildcard matching, and the per-slot registry. There is a companion CLI change (aenv start --allow-out, plus an aenv network subcommand for live updates) which could land separately or not at all if you would rather shape the CLI yourself.

It is written against v0.1.1 and would need rebasing onto current main — the networking layer has moved since (#108, #110, #145), so some of the slot handling will want reworking rather than replaying.

Questions before we write anything

Do you want domain egress in AgentENV at all, or is the intent that operators put a proxy in front and the placeholder error stays?
If yes: proxy-in-process as described, or would you prefer it as a separate helper the node supervises?
Any constraint we should respect from the RL-training workloads this is primarily built for — throughput on the proxy path, or per-slot memory?

Happy to split it into reviewable pieces (config plumbing, proxy core, slot wiring, docs) rather than one large PR. Also happy to drop it entirely if it cuts across something you have planned — no attachment to our implementation, only to not maintaining a fork of it.

Originally created by @sunkencity999 on GitHub (Aug 11, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/157 ### Context `src/sandbox/network/policy.rs` already accepts and validates domain entries in `allowOut`, then refuses them at runtime: ``` domain entries in allowOut require the TCP egress proxy, which is not enabled yet: {:?} ``` So the API surface, the validation, and the error path exist — the enforcement does not. This issue is to ask whether you would welcome that implementation upstream, and to describe the approach before writing a PR against it. We needed domain-level egress for a deployment where sandboxes must reach a small set of package registries and nothing else, so we built it against v0.1.1 and have been running it in production. Rather than carry the fork indefinitely, we would rather it live here — but only if the design matches where you want the networking layer to go. ### Approach A TCP egress proxy on the node, off by default behind config: ```toml [network.egress.proxy] enabled = true http_port = 18080 tls_port = 18443 ``` - Each deny-by-default sandbox's outbound tcp/80 and tcp/443 are DNAT-redirected to the proxy, per network slot. - The proxy matches the HTTP `Host` header (:80) or the TLS SNI (:443) against that sandbox's `allowOut` domains, supporting exact and `*.wildcard` entries. - **The name is resolved host-side, not by the sandbox.** A forged SNI cannot select the upstream address, which closes the obvious DNS-rebinding path. - Matching connections are spliced; non-matching get a 403 on :80 and a closed connection on :443. - Slot teardown flushes rules, including warm slots, so policy does not leak between sandbox generations. This overlaps with the concerns in #56 and #103. Domain rules only arm under a `Deny` base policy, which is consistent with how `allowOut` already behaves for CIDRs. ### Honest limitations Worth stating up front, because they bound what this feature claims: - **Non-web ports are not proxied.** tcp/80 and tcp/443 only; everything else remains governed by CIDR rules and the base policy. - **UDP/QUIC is not proxied.** A client that negotiates QUIC bypasses the allowlist unless UDP is otherwise denied. - **DNS remains reachable**, so DNS tunnelling is an available exfiltration channel. This is the same residual that Unit 42 demonstrated against a comparable "isolated" sandbox mode earlier this year, so it should be documented rather than implied away. - The host firewall must admit the redirected traffic. A default-deny INPUT chain silently blackholes it, and the symptom is confusing: *every* proxied destination times out, allowed or not, which looks like a broken allowlist rather than a firewall problem. With ufw the rule is scoped to the internal ranges: ``` ufw allow proto tcp from 10.11.0.0/16 to 10.12.0.0/16 port 18080,18443 ``` (matching `host_interaction_cidr` and `veth_cidr` defaults). We would document this next to the config rather than leave people to discover it. ### Size Roughly 880 lines against v0.1.1: a new `src/sandbox/network/egress_proxy.rs` (~540) plus wiring in `policy.rs`, `slot.rs`, `cfg/network.rs`, `bin/server.rs`, the sandbox impl, and docs. Unit tests cover SNI and Host parsing, wildcard matching, and the per-slot registry. There is a companion CLI change (`aenv start --allow-out`, plus an `aenv network` subcommand for live updates) which could land separately or not at all if you would rather shape the CLI yourself. It is written against v0.1.1 and would need rebasing onto current `main` — the networking layer has moved since (#108, #110, #145), so some of the slot handling will want reworking rather than replaying. ### Questions before we write anything 1. Do you want domain egress in AgentENV at all, or is the intent that operators put a proxy in front and the placeholder error stays? 2. If yes: proxy-in-process as described, or would you prefer it as a separate helper the node supervises? 3. Any constraint we should respect from the RL-training workloads this is primarily built for — throughput on the proxy path, or per-slot memory? Happy to split it into reviewable pieces (config plumbing, proxy core, slot wiring, docs) rather than one large PR. Also happy to drop it entirely if it cuts across something you have planned — no attachment to our implementation, only to not maintaining a fork of it.

bsakaguchi closed this issue

2026-10-29 14:18:53 +00:00

bsakaguchi commented

2026-10-29 14:18:54 +00:00

Author

Owner

@yingdi-shan commented on GitHub (Aug 24, 2026):

Thanks, closes as PR #207 adds support for domain-based egress control.

@yingdi-shan commented on GitHub (Aug 24, 2026): Thanks, closes as PR #207 adds support for domain-based egress control.

bsakaguchi referenced this issue

2026-10-29 14:20:08 +00:00

[PR #53] [CLOSED] feat(api): render server OpenAPI from tag-based templates #142