[GH-ISSUE #401] scheduler: self-invalidating leader gate (single-goroutine elect loop) #102

New issue

Open

opened 2026-10-29 14:19:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:36 +00:00

Owner

Originally created by @NickNYU on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/401

Affected component

Gateway / scheduler

Problem statement

The scheduler's leader gate reads only the local acquisition flag, which lags the real Lease holder by up to renew_deadline. In #341 this is made safe by the validated timing inequality (lease_duration > renew_deadline): client-go stops the old leader within renew_deadline of its last successful renew, while a standby can only acquire after lease_duration — so no two replicas serve concurrently.

But the safety argument rests on client-go's callback timing rather than on anything self-evident in the scheduler's own code, and a partitioned old leader keeps serving for up to renew_deadline — legal single-writer, but longer than necessary.

Use case

HA scheduler deployment (see deploy/k8s/overlays/ha): the leader pod is partitioned from the API server but stays alive and keeps receiving traffic on its (soon-to-be-removed) endpoint.

Current behavior and workarounds

The gate trusts the local acquisition flag; a holder watch that cross-checked the Lease against the API server was removed in #341 because it had a lost-update race (two writers of the leadership state snapshot) and the timing inequality makes its window unreachable.
Fail-closed timing is therefore client-go's renew_deadline, not anything the scheduler controls directly.

Desired behavior

The gate fails closed at last-successful-renew + lease_duration/2, read directly from the leadership snapshot — self-invalidating reads, no watcher goroutine.
Exactly one writer of the leadership state: a single goroutine per election cycle reads the Lease, decides, CAS-writes via the LeaseLock, and applies the local state — zero async hops between the API round trip and the state store.
At-most-once acquisition per process: the first definitive loss or local expiry ends the loop and the process exits.

Proposed approach

Already implemented on NickNYU/AgentENV branch feat/scheduler-elect-loop (commit 59d9ce6, stacked on #341):

leadershipSnapshot.expiresAt = last successful acquire/renew + lease_duration/2 (sofa-registry-style duration/2 safety margin); the gate is leader && now < expiresAt.
electLoop replaces client-go's leaderelection.Run callbacks; standby acquire keeps client-go semantics (observe the unexpired record, then resourceVersion CAS; AcquireTime/LeaderTransitions bookkeeping preserved).
renew_deadline config field removed; validation becomes lease_duration >= 4 * retry_period so the serving margin spans at least two election ticks.
e2e T4 re-scoped to the timing invariant.

Compatibility and operational impact

API/config changes: removes scheduler.leader_election.renew_deadline (leader election is new in #341 and unreleased, so no migration).

Snapshot or storage format changes: none.

New host/runtime requirements: none.

Upgrade and rollback considerations: behavior change on partition — the old leader stops serving at lease_duration/2 instead of renew_deadline; strictly faster fail-closed.

Alternatives considered

Keep the 1s holder watch — rejected: two-writer lost update on the state snapshot, plus an extra API read per second per replica.
Keep the #341 state (local flag + timing inequality) — accepted for #341; this issue tracks the stricter follow-up.

Pre-submission checklist

I searched existing issues and discussions and did not find a duplicate.
I described a concrete problem and use case, not only a proposed technology.
I understand that acceptance of the problem does not imply approval of a particular implementation.

Originally created by @NickNYU on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/401 ## Affected component Gateway / scheduler ## Problem statement The scheduler's leader gate reads only the local acquisition flag, which lags the real Lease holder by up to `renew_deadline`. In #341 this is made safe by the validated timing inequality (`lease_duration > renew_deadline`): client-go stops the old leader within `renew_deadline` of its last successful renew, while a standby can only acquire after `lease_duration` — so no two replicas serve concurrently. But the safety argument rests on client-go's callback timing rather than on anything self-evident in the scheduler's own code, and a partitioned old leader keeps serving for up to `renew_deadline` — legal single-writer, but longer than necessary. ## Use case HA scheduler deployment (see `deploy/k8s/overlays/ha`): the leader pod is partitioned from the API server but stays alive and keeps receiving traffic on its (soon-to-be-removed) endpoint. ## Current behavior and workarounds - The gate trusts the local acquisition flag; a holder watch that cross-checked the Lease against the API server was removed in #341 because it had a lost-update race (two writers of the leadership state snapshot) and the timing inequality makes its window unreachable. - Fail-closed timing is therefore client-go's `renew_deadline`, not anything the scheduler controls directly. ## Desired behavior - The gate fails closed at last-successful-renew + `lease_duration/2`, read directly from the leadership snapshot — self-invalidating reads, no watcher goroutine. - Exactly one writer of the leadership state: a single goroutine per election cycle reads the Lease, decides, CAS-writes via the LeaseLock, and applies the local state — zero async hops between the API round trip and the state store. - At-most-once acquisition per process: the first definitive loss or local expiry ends the loop and the process exits. ## Proposed approach Already implemented on `NickNYU/AgentENV` branch `feat/scheduler-elect-loop` (commit 59d9ce6, stacked on #341): - `leadershipSnapshot.expiresAt` = last successful acquire/renew + `lease_duration/2` (sofa-registry-style duration/2 safety margin); the gate is `leader && now < expiresAt`. - `electLoop` replaces client-go's `leaderelection.Run` callbacks; standby acquire keeps client-go semantics (observe the unexpired record, then resourceVersion CAS; `AcquireTime`/`LeaderTransitions` bookkeeping preserved). - `renew_deadline` config field removed; validation becomes `lease_duration >= 4 * retry_period` so the serving margin spans at least two election ticks. - e2e T4 re-scoped to the timing invariant. ## Compatibility and operational impact API/config changes: removes `scheduler.leader_election.renew_deadline` (leader election is new in #341 and unreleased, so no migration). Snapshot or storage format changes: none. New host/runtime requirements: none. Upgrade and rollback considerations: behavior change on partition — the old leader stops serving at `lease_duration/2` instead of `renew_deadline`; strictly faster fail-closed. ## Alternatives considered - Keep the 1s holder watch — rejected: two-writer lost update on the state snapshot, plus an extra API read per second per replica. - Keep the #341 state (local flag + timing inequality) — accepted for #341; this issue tracks the stricter follow-up. ## Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

bsakaguchi referenced this issue

2026-10-29 14:20:43 +00:00

[PR #103] [MERGED] fix(sandbox): isolate envd bootstrap connections across generations #180