[GH-ISSUE #402] scheduler: store-level fencing token for in-flight Redis writes during failover #101

New issue

Open

opened 2026-10-29 14:19:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:35 +00:00

Owner

Originally created by @NickNYU on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/402

Affected component

Gateway / scheduler

Problem statement

The scheduler's leader gate fences request entry, not writes already in flight. When the old leader loses the Lease it force-stops, but a handler that already passed the gate (RecordAssignment, Heartbeat reconcile) can still commit its Redis write after the new leader takes over — there is no store-level fencing token. Surfaced in the #341 review and accepted there as a documented residual; this issue tracks the proper fix.

Use case

HA scheduler failover under write load: a slow heartbeat reconcile (roster diff computation) or a RecordAssignment retry races the old leader's process exit.

Current behavior and workarounds

Accepted residual, documented in services/README.md ("In-flight writes during failover"):

The window is the few hundred milliseconds between the old leader's force-stop and its process exit; the lease_duration > renew_deadline invariant already guarantees no new request is served concurrently by two replicas.
Every binding write is idempotent and TTL-bound.
Heartbeats carry an authoritative roster: a binding wrongly deleted by a stale in-flight reconcile is restored by the node's next heartbeat (within one report interval); a stale RecordAssignment is overwritten by the gateway's retry onto the new leader.

The workaround bounds the damage but cannot prevent it; worst case is a routing NotFound for an affected sandbox until the next heartbeat.

Desired behavior

A stale leader's write is rejected at the store, not merely unlikely: once a new leader is serving, writes carrying an older epoch become no-ops.
Given an old leader partitioned after its last renew, when its in-flight handler commits after the standby's takeover, then the binding store is unchanged by that commit.

Proposed approach

Leader epoch sourced from the Lease (e.g. LeaderTransitions observed at acquire), held in the leadership snapshot.
Every binding write (Record, reconcile delete/upsert) carries the epoch; the Redis path becomes a conditional write (Lua) that applies only if the epoch is not older than the stored one.
The in-memory store applies the same check under its lock for parity.

Compatibility and operational impact

API/config changes: none.

Snapshot or storage format changes: Redis binding values gain an epoch field (or a companion epoch key); old entries treated as epoch 0.

New host/runtime requirements: none.

Upgrade and rollback considerations: mixed old/new schedulers during rollout must tolerate missing epochs (treat absent as oldest).

Alternatives considered

Bounded drain on leadership loss (wait for in-flight handlers with a timeout): shrinks but does not eliminate the window, and extends the old leader's lifetime past takeover.
Accept the residual permanently: chosen for #341; this issue exists so the decision can be revisited with production evidence.

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @NickNYU on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/402 ## Affected component Gateway / scheduler ## Problem statement The scheduler's leader gate fences request *entry*, not writes already in flight. When the old leader loses the Lease it force-stops, but a handler that already passed the gate (`RecordAssignment`, `Heartbeat` reconcile) can still commit its Redis write after the new leader takes over — there is no store-level fencing token. Surfaced in the #341 review and accepted there as a documented residual; this issue tracks the proper fix. ## Use case HA scheduler failover under write load: a slow heartbeat reconcile (roster diff computation) or a `RecordAssignment` retry races the old leader's process exit. ## Current behavior and workarounds Accepted residual, documented in `services/README.md` ("In-flight writes during failover"): - The window is the few hundred milliseconds between the old leader's force-stop and its process exit; the `lease_duration > renew_deadline` invariant already guarantees no *new* request is served concurrently by two replicas. - Every binding write is idempotent and TTL-bound. - Heartbeats carry an authoritative roster: a binding wrongly deleted by a stale in-flight reconcile is restored by the node's next heartbeat (within one report interval); a stale `RecordAssignment` is overwritten by the gateway's retry onto the new leader. The workaround bounds the damage but cannot prevent it; worst case is a routing `NotFound` for an affected sandbox until the next heartbeat. ## Desired behavior - A stale leader's write is rejected at the store, not merely unlikely: once a new leader is serving, writes carrying an older epoch become no-ops. - Given an old leader partitioned after its last renew, when its in-flight handler commits after the standby's takeover, then the binding store is unchanged by that commit. ## Proposed approach - Leader epoch sourced from the Lease (e.g. `LeaderTransitions` observed at acquire), held in the leadership snapshot. - Every binding write (`Record`, reconcile delete/upsert) carries the epoch; the Redis path becomes a conditional write (Lua) that applies only if the epoch is not older than the stored one. - The in-memory store applies the same check under its lock for parity. ## Compatibility and operational impact API/config changes: none. Snapshot or storage format changes: Redis binding values gain an epoch field (or a companion epoch key); old entries treated as epoch 0. New host/runtime requirements: none. Upgrade and rollback considerations: mixed old/new schedulers during rollout must tolerate missing epochs (treat absent as oldest). ## Alternatives considered - Bounded drain on leadership loss (wait for in-flight handlers with a timeout): shrinks but does not eliminate the window, and extends the old leader's lifetime past takeover. - Accept the residual permanently: chosen for #341; this issue exists so the decision can be revisited with production evidence. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi referenced this issue

2026-10-29 14:20:43 +00:00

[PR #103] [MERGED] fix(sandbox): isolate envd bootstrap connections across generations #180