[GH-ISSUE #204] Cache sandbox-to-node routing in the AENV gateway #66

New issue

Open

opened 2026-10-29 14:19:04 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:19:04 +00:00

Owner

Originally created by @yingdi-shan on GitHub (Aug 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/204

Affected component

Gateway / scheduler

Problem statement

The gateway calls the scheduler LookupNode RPC for every request containing a sandbox ID. This adds scheduler load and latency.

Use case

Large scale deployments handling frequent data-plane and control-plane requests for long-lived sandboxes. The same sandbox may receive many requests through different gateway replicas, while deletes and node changes may be handled by any replica.

Current behavior and workarounds

Every sandbox-routed request performs a scheduler lookup. There is no gateway cache. The scheduler has no explicit binding-delete RPC, and ReportSandboxEvent currently logs delete events without mutating the binding store. A local-only delete eviction would therefore leave other gateways with stale entries until their process restarts or a scheduler binding expires.

Desired behavior

Repeated requests for an active sandbox should avoid scheduler lookups while a valid route-cache entry exists.
Each gateway may keep a bounded, process-local positive TTL cache keyed by sandbox ID and storing the node ID and endpoint.
The scheduler remains the source of truth. Successful lookups may be cached; NotFound and scheduler errors must not be negative-cached.
After a successful delete, the handling gateway evicts its local entry and the shared scheduler binding is removed, so other gateways converge after cache expiry or stale-route detection.
If a cached route receives a reliable sandbox-not-found response, the gateway evicts the entry and performs one uncached scheduler lookup. If the scheduler also returns NotFound, return 404. If it returns a new node, retry once only for a replay-safe request.
Streaming, WebSocket, non-replayable, and ambiguous data-plane 404 requests must not be retried.
Cache hits, misses, evictions, and refetches should be observable, and concurrent misses for one sandbox should be coalesced.

Proposed approach

Add a bounded process-local cache in the gateway. Query the configured scheduler on cache misses, and populate or refresh entries after successful LookupNode calls and successful create or fork assignment recording.

Compatibility and operational impact

API/config changes:

No client-facing API change is required.

Snapshot or storage format changes:

None.

New host/runtime requirements:

None. The cache is in process memory.

Alternatives considered

Keep doing a scheduler lookup for every request: correct but adds avoidable scheduler traffic and latency.
Use a distributed cache or pub/sub invalidation for every gateway: provides faster invalidation but adds infrastructure and another consistency path.
Evict only the gateway that handled delete: insufficient because other gateways retain stale mappings.

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @yingdi-shan on GitHub (Aug 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/204 ## Affected component Gateway / scheduler ## Problem statement The gateway calls the scheduler `LookupNode` RPC for every request containing a sandbox ID. This adds scheduler load and latency. ## Use case Large scale deployments handling frequent data-plane and control-plane requests for long-lived sandboxes. The same sandbox may receive many requests through different gateway replicas, while deletes and node changes may be handled by any replica. ## Current behavior and workarounds Every sandbox-routed request performs a scheduler lookup. There is no gateway cache. The scheduler has no explicit binding-delete RPC, and `ReportSandboxEvent` currently logs delete events without mutating the binding store. A local-only delete eviction would therefore leave other gateways with stale entries until their process restarts or a scheduler binding expires. ## Desired behavior - Repeated requests for an active sandbox should avoid scheduler lookups while a valid route-cache entry exists. - Each gateway may keep a bounded, process-local positive TTL cache keyed by sandbox ID and storing the node ID and endpoint. - The scheduler remains the source of truth. Successful lookups may be cached; `NotFound` and scheduler errors must not be negative-cached. - After a successful delete, the handling gateway evicts its local entry and the shared scheduler binding is removed, so other gateways converge after cache expiry or stale-route detection. - If a cached route receives a reliable sandbox-not-found response, the gateway evicts the entry and performs one uncached scheduler lookup. If the scheduler also returns `NotFound`, return 404. If it returns a new node, retry once only for a replay-safe request. - Streaming, WebSocket, non-replayable, and ambiguous data-plane 404 requests must not be retried. - Cache hits, misses, evictions, and refetches should be observable, and concurrent misses for one sandbox should be coalesced. ## Proposed approach Add a bounded process-local cache in the gateway. Query the configured scheduler on cache misses, and populate or refresh entries after successful `LookupNode` calls and successful create or fork assignment recording. ## Compatibility and operational impact API/config changes: No client-facing API change is required. Snapshot or storage format changes: None. New host/runtime requirements: None. The cache is in process memory. ## Alternatives considered - Keep doing a scheduler lookup for every request: correct but adds avoidable scheduler traffic and latency. - Use a distributed cache or pub/sub invalidation for every gateway: provides faster invalidation but adds infrastructure and another consistency path. - Evict only the gateway that handled delete: insufficient because other gateways retain stale mappings. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi added the

enhancement

label

2026-10-29 14:19:04 +00:00

bsakaguchi commented

2026-10-29 14:19:05 +00:00

Author

Owner

@DAOZHENREN commented on GitHub (Aug 22, 2026):

Hi, I'd like to work on this issue.

My proposed implementation plan is:

Add a bounded, process-local positive TTL route cache in the gateway,
keyed by sandbox ID and storing the resolved scheduler node.
Coalesce concurrent cache misses for the same sandbox so that only one
LookupNode RPC is issued.
Populate or refresh the cache after successful LookupNode calls and
successful create/fork assignment recording. Scheduler errors and
NotFound responses will not be cached.
Evict the local entry after a successful sandbox deletion.
Add conditional binding removal to the scheduler binding store and process
delete lifecycle events in ReportSandboxEvent. The removal would only
delete the binding if it still points to the reporting node, preventing a
delayed delete event from removing a newer assignment.
On a reliable control-plane sandbox-not-found response from a cached route,
evict and perform one uncached scheduler lookup. Retry at most once and only
for explicitly replay-safe control-plane requests. WebSocket, streaming,
non-replayable, and data-plane 404 responses will not be retried.
Add bounded-cardinality Prometheus metrics for cache hits, misses,
evictions, and refetches, together with unit tests for expiry, capacity,
concurrency, invalidation, and retry behavior.

Before I start the implementation, could you confirm:

Is extending ReportSandboxEvent with conditional delete handling preferred,
or would you rather introduce an explicit scheduler binding-delete RPC?
Should the initial replay-safe whitelist be limited to control-plane
GET, HEAD, and DELETE requests?
Do you have preferred defaults or names for the cache capacity and TTL
configuration?

If this direction looks good, I can prepare the implementation and tests.

@DAOZHENREN commented on GitHub (Aug 22, 2026): Hi, I'd like to work on this issue. My proposed implementation plan is: 1. Add a bounded, process-local positive TTL route cache in the gateway, keyed by sandbox ID and storing the resolved scheduler node. 2. Coalesce concurrent cache misses for the same sandbox so that only one `LookupNode` RPC is issued. 3. Populate or refresh the cache after successful `LookupNode` calls and successful create/fork assignment recording. Scheduler errors and `NotFound` responses will not be cached. 4. Evict the local entry after a successful sandbox deletion. 5. Add conditional binding removal to the scheduler binding store and process delete lifecycle events in `ReportSandboxEvent`. The removal would only delete the binding if it still points to the reporting node, preventing a delayed delete event from removing a newer assignment. 6. On a reliable control-plane sandbox-not-found response from a cached route, evict and perform one uncached scheduler lookup. Retry at most once and only for explicitly replay-safe control-plane requests. WebSocket, streaming, non-replayable, and data-plane 404 responses will not be retried. 7. Add bounded-cardinality Prometheus metrics for cache hits, misses, evictions, and refetches, together with unit tests for expiry, capacity, concurrency, invalidation, and retry behavior. Before I start the implementation, could you confirm: - Is extending `ReportSandboxEvent` with conditional delete handling preferred, or would you rather introduce an explicit scheduler binding-delete RPC? - Should the initial replay-safe whitelist be limited to control-plane `GET`, `HEAD`, and `DELETE` requests? - Do you have preferred defaults or names for the cache capacity and TTL configuration? If this direction looks good, I can prepare the implementation and tests.

bsakaguchi referenced this issue

2026-10-29 14:20:17 +00:00

[PR #66] [MERGED] fix(template): create WORKDIR directories during template builds #151

bsakaguchi referenced this issue

2026-10-29 14:20:18 +00:00

[PR #67] [MERGED] fix(template): provision a missing default user at the end of a build #152