kvcache-ai/AgentENV

Fork 0

mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00

[GH-ISSUE #90] RFC: explicit placement reservation lifecycle and shared ownership #29

New issue

Closed

opened 2026-10-29 14:18:34 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:18:34 +00:00

Owner

Originally created by @catyans on GitHub (Jul 31, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/90

Affected component

Gateway / scheduler (cross-cutting with sandbox lifecycle and node heartbeat reporting)

Problem statement

Resource-aware placement needs to account for creates that have been scheduled but are not yet visible in node heartbeats. PR #27 currently does this with scheduler-local, TTL-bounded reservations and aggregate heartbeat deltas.

That closes the immediate concurrent-selection race, but it cannot reliably determine the outcome or ownership of an individual placement:

A malformed endpoint, proxy failure, client cancellation, or node rejection before heartbeat accounting leaves a phantom reservation until TTL.
create_successes, create_fails, and allocated-resource gauges are node-wide aggregates. They cannot identify which reservation completed, especially when requests finish out of order.
Concurrent deletes and node restarts can offset or reset gauge/counter deltas.
Multiple scheduler replicas do not share reservation ownership, so traffic observed by one replica can be attributed to another replica's reservation.
Template-based creates do not currently carry the resolved CPU/memory requirements through the scheduling API.

The result is a choice between under-accounting (possible overcommit) and conservative stale accounting (possible false ErrNoNodes). A fixed TTL only bounds the duration; it does not provide causal correctness.

Related review findings on #27:

Use case

This is needed when AgentENV uses resource-aware placement for concurrent sandbox creation, particularly:

bursts of creates whose node heartbeats lag behind scheduling;
scheduler and Gateway replicas operating concurrently;
dispatch retries, cancellations, rejected creates, and partial service failures;
heterogeneous sandbox CPU/memory requests, including template-based creates.

At low concurrency the current TTL fallback is usually sufficient. At higher concurrency or during downstream failure, request-level identity is needed to avoid both overcommit and prolonged false capacity exhaustion.

Current behavior and workarounds

PR #27 records an in-memory reservation immediately after selecting a node. Heartbeat generations and aggregate success/failure/allocation deltas are used as best-effort reconciliation, with a configurable TTL as the final cleanup mechanism.

This is intentionally process-local and heuristic. Making the TTL shorter reduces stale capacity but reopens the concurrent-placement race for slow creates; making it longer protects against overcommit but increases the duration of false exhaustion. Aggregate counters cannot resolve this trade-off because they do not establish causality.

Desired behavior

Every resource-bearing create placement has an opaque reservation identity.
A dispatch failure or cancellation releases exactly that reservation.
A successful node acceptance confirms exactly that reservation and binds it to the resulting sandbox/assignment identity.
Confirm, cancel, retry, and expiry operations are idempotent.
Scheduler replicas observe shared reservation ownership and cannot consume or release another request's capacity accidentally.
A process crash does not leave capacity reserved indefinitely.
Node restarts and unrelated creates/deletes do not corrupt reconciliation.
Template-based creates provide resolved resource requirements before resource-aware admission, or are conservatively rejected when a known-full dimension cannot be evaluated.
Existing clients and non-resource-aware strategies continue to work during a staged rollout.

Proposed approach

1. Return an opaque reservation token

Extend the scheduling response for sandbox creation with an optional reservation_token. The token is absent for operations and strategies that do not reserve capacity.

The reservation record contains at least:

token
node_id
requested_cpu
requested_memory
state
created_at
lease_expires_at
sandbox_or_assignment_id (optional)
version

Suggested states:

reserved -> dispatched -> bound
reserved/dispatched -> cancelled
reserved/dispatched -> expired

State transitions use compare-and-set semantics and are idempotent.

2. Add explicit lifecycle operations

Add internal RPCs or equivalent control-plane operations:

ConfirmReservation(token, sandbox_or_assignment_id) after the selected node accepts the create;
CancelReservation(token, reason) on endpoint validation failure, proxy/transport failure, client cancellation, node rejection, or exhausted retry;
optionally RenewReservation(token) if dispatch can legitimately exceed the initial lease.

The Gateway owns these calls because it observes the downstream request outcome. A retry must either reuse the same token for the same node or explicitly cancel it before acquiring another reservation.

3. Use shared ownership for replicated schedulers

For multi-replica deployments, store reservations in the existing shared control-plane store (for example Redis) with atomic state transitions and lease expiry. A single-process in-memory implementation can remain available for development, but it must implement the same interface and semantics.

The store should support:

create-if-absent by token;
idempotent confirm/cancel;
compare-and-set state transitions;
expiration indexes or lease-based garbage collection;
aggregation of active reservations by node for projected-capacity checks.

4. Bind outcomes to sandbox identity

The confirmed reservation should be bound to the resulting sandbox/assignment ID. Reconciliation then operates on identities rather than gauge deltas.

The preferred end state is for node heartbeat/inventory data to include the sandbox/assignment identity already known to the control plane. Once that identity is observed as active, the pending reservation can leave projected capacity without ambiguity. If node protocol changes must be staged later, the shared binding still provides authoritative confirm/cancel behavior, with lease expiry as crash recovery.

5. Carry resolved resources for template creates

Resource-aware scheduling must receive CPU and memory after template resolution. If that cannot be added in the first phase, an unknown resource dimension must not be admitted when observed allocation plus active reservations is already at its known capacity.

6. Add observability

Expose at least:

active reservations by node and state;
reservation age and expiry count;
confirm/cancel latency and result;
idempotent replay/conflict counts;
capacity rejection counts;
orphaned or unbound confirmation counts.

Logs should include the reservation token in a safe, traceable form and correlate it with the sandbox/assignment ID.

Compatibility and operational impact

API/config changes:

Add an optional reservation token to the scheduling response.
Add confirm/cancel operations and, if needed, resolved resource hints for template creates.
Retain a configurable lease/TTL as crash recovery, not primary reconciliation.

Snapshot or storage format changes:

Add ephemeral reservation records to the shared control-plane store. No sandbox snapshot format change is required.

New host/runtime requirements:

No new host requirement. Replicated schedulers require the configured shared store to provide atomic operations and expiry.

Upgrade and rollback considerations:

Gate the lifecycle behind a capability/version check or feature flag.
During rolling upgrade, an old Gateway may ignore the optional token; those reservations fall back to lease expiry.
New Gateways must tolerate a scheduler that does not return a token.
Keep current TTL reconciliation temporarily as a compatibility fallback, then remove aggregate outcome attribution after all components support the protocol.

Rollout plan

Introduce the optional token, reservation-store interface, and idempotent confirm/cancel operations.
Wire Gateway dispatch success/failure/cancellation paths and add failure-injection tests.
Add shared-store ownership for scheduler replicas.
Bind reservations to sandbox/assignment IDs and reconcile from identity-bearing inventory/heartbeat data.
Resolve template resources before placement.
Remove aggregate-counter/gauge attribution once the compatibility window closes.

Acceptance criteria

Two out-of-order creates with different resource sizes can complete or fail in either order without releasing the wrong capacity.
A pre-dispatch validation error and a transport failure release their exact reservations promptly.
Duplicate confirm/cancel calls are safe and return stable results.
A scheduler or Gateway crash leaves reservations recoverable through shared state and bounded lease expiry.
Two scheduler replicas produce the same projected reservation total and cannot double-release.
Concurrent sandbox deletion does not affect create-reservation reconciliation.
Node restart/counter reset does not acknowledge unrelated reservations.
Template creates cannot bypass a known CPU or memory capacity limit.
Unit, integration, race, and failure-injection tests cover all state transitions and replica races.

Alternatives considered

Only use a shorter fixed TTL: reduces stale duration but allows slow in-flight creates to be forgotten and capacity to be overcommitted.
Only use a longer fixed TTL: protects slow creates but makes downstream failures cause prolonged false exhaustion.
Infer outcomes from aggregate counters/gauges: cannot correlate out-of-order requests, concurrent deletes, node restarts, or multiple scheduler replicas.
Keep reservations process-local: cannot provide shared ownership or crash recovery in a replicated deployment.
Reserve only after dispatch succeeds: leaves the concurrent selection window open and can overcommit before dispatch.

Non-goals

A distributed transaction spanning Gateway, scheduler, and node.
Replacing all scheduler strategies.
Persisting reservations as durable user-visible sandbox state.
Changing runtime resource enforcement on the node.

Contribution

I can implement it and validate the multi-node behavior.

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @catyans on GitHub (Jul 31, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/90 ## Affected component Gateway / scheduler (cross-cutting with sandbox lifecycle and node heartbeat reporting) ## Problem statement Resource-aware placement needs to account for creates that have been scheduled but are not yet visible in node heartbeats. PR #27 currently does this with scheduler-local, TTL-bounded reservations and aggregate heartbeat deltas. That closes the immediate concurrent-selection race, but it cannot reliably determine the outcome or ownership of an individual placement: - A malformed endpoint, proxy failure, client cancellation, or node rejection before heartbeat accounting leaves a phantom reservation until TTL. - `create_successes`, `create_fails`, and allocated-resource gauges are node-wide aggregates. They cannot identify which reservation completed, especially when requests finish out of order. - Concurrent deletes and node restarts can offset or reset gauge/counter deltas. - Multiple scheduler replicas do not share reservation ownership, so traffic observed by one replica can be attributed to another replica's reservation. - Template-based creates do not currently carry the resolved CPU/memory requirements through the scheduling API. The result is a choice between under-accounting (possible overcommit) and conservative stale accounting (possible false `ErrNoNodes`). A fixed TTL only bounds the duration; it does not provide causal correctness. Related review findings on #27: - https://github.com/kvcache-ai/AgentENV/pull/27#discussion_r3679899069 - https://github.com/kvcache-ai/AgentENV/pull/27#discussion_r3679899070 - https://github.com/kvcache-ai/AgentENV/pull/27#discussion_r3679959316 - https://github.com/kvcache-ai/AgentENV/pull/27#discussion_r3679959318 - https://github.com/kvcache-ai/AgentENV/pull/27#discussion_r3679959319 ## Use case This is needed when AgentENV uses resource-aware placement for concurrent sandbox creation, particularly: - bursts of creates whose node heartbeats lag behind scheduling; - scheduler and Gateway replicas operating concurrently; - dispatch retries, cancellations, rejected creates, and partial service failures; - heterogeneous sandbox CPU/memory requests, including template-based creates. At low concurrency the current TTL fallback is usually sufficient. At higher concurrency or during downstream failure, request-level identity is needed to avoid both overcommit and prolonged false capacity exhaustion. ## Current behavior and workarounds PR #27 records an in-memory reservation immediately after selecting a node. Heartbeat generations and aggregate success/failure/allocation deltas are used as best-effort reconciliation, with a configurable TTL as the final cleanup mechanism. This is intentionally process-local and heuristic. Making the TTL shorter reduces stale capacity but reopens the concurrent-placement race for slow creates; making it longer protects against overcommit but increases the duration of false exhaustion. Aggregate counters cannot resolve this trade-off because they do not establish causality. ## Desired behavior - Every resource-bearing create placement has an opaque reservation identity. - A dispatch failure or cancellation releases exactly that reservation. - A successful node acceptance confirms exactly that reservation and binds it to the resulting sandbox/assignment identity. - Confirm, cancel, retry, and expiry operations are idempotent. - Scheduler replicas observe shared reservation ownership and cannot consume or release another request's capacity accidentally. - A process crash does not leave capacity reserved indefinitely. - Node restarts and unrelated creates/deletes do not corrupt reconciliation. - Template-based creates provide resolved resource requirements before resource-aware admission, or are conservatively rejected when a known-full dimension cannot be evaluated. - Existing clients and non-resource-aware strategies continue to work during a staged rollout. ## Proposed approach ### 1. Return an opaque reservation token Extend the scheduling response for sandbox creation with an optional `reservation_token`. The token is absent for operations and strategies that do not reserve capacity. The reservation record contains at least: ```text token node_id requested_cpu requested_memory state created_at lease_expires_at sandbox_or_assignment_id (optional) version ``` Suggested states: ```text reserved -> dispatched -> bound reserved/dispatched -> cancelled reserved/dispatched -> expired ``` State transitions use compare-and-set semantics and are idempotent. ### 2. Add explicit lifecycle operations Add internal RPCs or equivalent control-plane operations: - `ConfirmReservation(token, sandbox_or_assignment_id)` after the selected node accepts the create; - `CancelReservation(token, reason)` on endpoint validation failure, proxy/transport failure, client cancellation, node rejection, or exhausted retry; - optionally `RenewReservation(token)` if dispatch can legitimately exceed the initial lease. The Gateway owns these calls because it observes the downstream request outcome. A retry must either reuse the same token for the same node or explicitly cancel it before acquiring another reservation. ### 3. Use shared ownership for replicated schedulers For multi-replica deployments, store reservations in the existing shared control-plane store (for example Redis) with atomic state transitions and lease expiry. A single-process in-memory implementation can remain available for development, but it must implement the same interface and semantics. The store should support: - create-if-absent by token; - idempotent confirm/cancel; - compare-and-set state transitions; - expiration indexes or lease-based garbage collection; - aggregation of active reservations by node for projected-capacity checks. ### 4. Bind outcomes to sandbox identity The confirmed reservation should be bound to the resulting sandbox/assignment ID. Reconciliation then operates on identities rather than gauge deltas. The preferred end state is for node heartbeat/inventory data to include the sandbox/assignment identity already known to the control plane. Once that identity is observed as active, the pending reservation can leave projected capacity without ambiguity. If node protocol changes must be staged later, the shared binding still provides authoritative confirm/cancel behavior, with lease expiry as crash recovery. ### 5. Carry resolved resources for template creates Resource-aware scheduling must receive CPU and memory after template resolution. If that cannot be added in the first phase, an unknown resource dimension must not be admitted when observed allocation plus active reservations is already at its known capacity. ### 6. Add observability Expose at least: - active reservations by node and state; - reservation age and expiry count; - confirm/cancel latency and result; - idempotent replay/conflict counts; - capacity rejection counts; - orphaned or unbound confirmation counts. Logs should include the reservation token in a safe, traceable form and correlate it with the sandbox/assignment ID. ## Compatibility and operational impact API/config changes: - Add an optional reservation token to the scheduling response. - Add confirm/cancel operations and, if needed, resolved resource hints for template creates. - Retain a configurable lease/TTL as crash recovery, not primary reconciliation. Snapshot or storage format changes: - Add ephemeral reservation records to the shared control-plane store. No sandbox snapshot format change is required. New host/runtime requirements: - No new host requirement. Replicated schedulers require the configured shared store to provide atomic operations and expiry. Upgrade and rollback considerations: - Gate the lifecycle behind a capability/version check or feature flag. - During rolling upgrade, an old Gateway may ignore the optional token; those reservations fall back to lease expiry. - New Gateways must tolerate a scheduler that does not return a token. - Keep current TTL reconciliation temporarily as a compatibility fallback, then remove aggregate outcome attribution after all components support the protocol. ## Rollout plan 1. Introduce the optional token, reservation-store interface, and idempotent confirm/cancel operations. 2. Wire Gateway dispatch success/failure/cancellation paths and add failure-injection tests. 3. Add shared-store ownership for scheduler replicas. 4. Bind reservations to sandbox/assignment IDs and reconcile from identity-bearing inventory/heartbeat data. 5. Resolve template resources before placement. 6. Remove aggregate-counter/gauge attribution once the compatibility window closes. ## Acceptance criteria - Two out-of-order creates with different resource sizes can complete or fail in either order without releasing the wrong capacity. - A pre-dispatch validation error and a transport failure release their exact reservations promptly. - Duplicate confirm/cancel calls are safe and return stable results. - A scheduler or Gateway crash leaves reservations recoverable through shared state and bounded lease expiry. - Two scheduler replicas produce the same projected reservation total and cannot double-release. - Concurrent sandbox deletion does not affect create-reservation reconciliation. - Node restart/counter reset does not acknowledge unrelated reservations. - Template creates cannot bypass a known CPU or memory capacity limit. - Unit, integration, race, and failure-injection tests cover all state transitions and replica races. ## Alternatives considered - **Only use a shorter fixed TTL:** reduces stale duration but allows slow in-flight creates to be forgotten and capacity to be overcommitted. - **Only use a longer fixed TTL:** protects slow creates but makes downstream failures cause prolonged false exhaustion. - **Infer outcomes from aggregate counters/gauges:** cannot correlate out-of-order requests, concurrent deletes, node restarts, or multiple scheduler replicas. - **Keep reservations process-local:** cannot provide shared ownership or crash recovery in a replicated deployment. - **Reserve only after dispatch succeeds:** leaves the concurrent selection window open and can overcommit before dispatch. ## Non-goals - A distributed transaction spanning Gateway, scheduler, and node. - Replacing all scheduler strategies. - Persisting reservations as durable user-visible sandbox state. - Changing runtime resource enforcement on the node. ## Contribution I can implement it and validate the multi-node behavior. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi closed this issue

2026-10-29 14:18:34 +00:00

bsakaguchi commented

2026-10-29 14:18:35 +00:00

Author

Owner

@catyans commented on GitHub (Aug 3, 2026):

@LSX-s-Software Following up because PR #27 remains blocked by the four unresolved reservation lifecycle findings linked above. Before expanding the patch across Gateway, scheduler, shared state, and node identity, could maintainers confirm the preferred sequencing: (1) narrow #27 to heartbeat-snapshot least-loaded scoring and implement this RFC separately, or (2) land the reservation-token lifecycle first and keep #27 blocked on it? I can proceed with either path and validate the replica/failure cases on the Lingjun cluster.

@catyans commented on GitHub (Aug 3, 2026): @LSX-s-Software Following up because PR #27 remains blocked by the four unresolved reservation lifecycle findings linked above. Before expanding the patch across Gateway, scheduler, shared state, and node identity, could maintainers confirm the preferred sequencing: (1) narrow #27 to heartbeat-snapshot least-loaded scoring and implement this RFC separately, or (2) land the reservation-token lifecycle first and keep #27 blocked on it? I can proceed with either path and validate the replica/failure cases on the Lingjun cluster.

bsakaguchi commented

2026-10-29 14:18:35 +00:00

Author

Owner

@LSX-s-Software commented on GitHub (Aug 4, 2026):

@catyans Thank you for the proposal and for the thoughtful work on improving the scheduler.

The proposed strategy is interesting, and we agree it could bring benefits in certain scenarios. However, implementing it would require substantial changes to the scheduler’s core logic, along with significant validation effort and ongoing maintenance cost.

Given our current priorities, we don’t have the bandwidth to take on this level of work right now. As a result, we’d prefer to put this proposal on hold and revisit it when we have more resources or when the need becomes more pressing.

That said, if you’re able to further refine the design, provide performance data, or experiment with a prototype, we’d be very interested to see those updates — they would be valuable for future evaluation.

Thanks again for your contribution and for engaging with the project!

@LSX-s-Software commented on GitHub (Aug 4, 2026): @catyans Thank you for the proposal and for the thoughtful work on improving the scheduler. The proposed strategy is interesting, and we agree it could bring benefits in certain scenarios. However, implementing it would require substantial changes to the scheduler’s core logic, along with significant validation effort and ongoing maintenance cost. Given our current priorities, we don’t have the bandwidth to take on this level of work right now. As a result, we’d prefer to put this proposal on hold and revisit it when we have more resources or when the need becomes more pressing. That said, if you’re able to further refine the design, provide performance data, or experiment with a prototype, we’d be very interested to see those updates — they would be valuable for future evaluation. Thanks again for your contribution and for engaging with the project!

bsakaguchi commented

2026-10-29 14:18:35 +00:00

Author

Owner

@catyans commented on GitHub (Aug 5, 2026):

Understood. I will put this design on hold and close the RFC for now, along with the dependent PR #27. The failure model, rollout plan, and acceptance criteria will remain here for future evaluation; if the need becomes pressing, I can refresh the design and rerun the multi-node validation against the then-current control plane.

@catyans commented on GitHub (Aug 5, 2026): Understood. I will put this design on hold and close the RFC for now, along with the dependent PR #27. The failure model, rollout plan, and acceptance criteria will remain here for future evaluation; if the need becomes pressing, I can refresh the design and rerun the multi-node validation against the then-current control plane.

bsakaguchi referenced this issue

2026-10-29 14:20:01 +00:00

[PR #29] [MERGED] doc: link overlaybd docs in README #128