kvcache-ai/AgentENV
1
0
Fork 0
mirror of https://github.com/kvcache-ai/AgentENV.git synced 2026-11-05 01:17:36 +00:00
Code Issues 91 Projects Packages Wiki Activity

[GH-ISSUE #259] Add scheduler HA using Kubernetes Lease leader election #71

New issue
Open
opened 2026-10-29 14:19:09 +00:00 by bsakaguchi · 10 comments
bsakaguchi commented 2026-10-29 14:19:09 +00:00
Owner
Copy link

Originally created by @yingdi-shan on GitHub (Sep 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/259

Affected component

Gateway / scheduler

Problem statement

AgentENV currently requires one read/write scheduler. Redis-backed query-only replicas preserve existing sandbox routing, but scheduling, node inspection, assignment writes, and P2P APIs remain unavailable if the primary fails.

Running multiple read/write schedulers is unsafe because heartbeat state is replica-local. This causes resource limits to be applied inconsistently, as reported in #191.

Use case

Production Kubernetes deployments need scheduler redundancy for runtime failures, upgrades, and node maintenance without operating multiple active schedulers or introducing a custom replication protocol.

Current behavior and workarounds

The Kubernetes deployment runs one scheduler replica. Operators can add query-only replicas backed by Redis, but these provide data-plane lookup availability only. The practical workaround is restarting the single primary scheduler and waiting for heartbeats to rebuild its state.

Desired behavior

  • Multiple scheduler pods may be deployed.
  • Exactly one scheduler performs scheduling and processes node reports.
  • Standbys participate only in leader election and health reporting.
  • A standby automatically takes leadership after leader failure.
  • The promoted leader rebuilds mappings and observations from node heartbeats.
  • Missing observations during recovery are not treated as unlimited capacity.
  • Leadership loss immediately stops the old leader from serving.
  • Liveness remains healthy on standbys; readiness exposes only the leader.
  • Failover completes within the configured lease duration plus heartbeat recovery time.
  • Three scheduler replicas pass the resource-limit scenario from #191.

Proposed approach

Use client-go leader election with a coordination.k8s.io/Lease.

Add scheduler configuration for:

{
  "leader_election": {
    "enabled": true,
    "lease_name": "agentenv-scheduler",
    "lease_namespace": "agentenv-system",
    "lease_duration": "15s",
    "renew_deadline": "10s",
    "retry_period": "2s"
  }
}

On leadership acquisition:

  • Start discovery and the scheduler service.
  • Accept heartbeats immediately.
  • Schedule only onto nodes with fresh observations.
  • Return Unavailable rather than false NotFound results while in-memory bindings are rebuilding.

On leadership loss:

  • Mark the scheduler service non-serving.
  • Stop accepting RPCs and exit so existing gRPC connections reconnect.

Update Kubernetes manifests to use three replicas, grant Lease RBAC permissions, and use separate liveness and leader-readiness checks. Leader election and --query-only should be mutually exclusive.

Compatibility and operational impact

API/config changes: Add optional scheduler leader-election configuration. No protobuf API change is expected.

Snapshot or storage format changes: None.

New host/runtime requirements: Kubernetes Lease API access and corresponding RBAC permissions.

Upgrade and rollback considerations: Leader election should remain disabled by default. Existing single-replica and query-only deployments continue to work. Rollback requires scaling the scheduler back to one replica.

Alternatives considered

  • Active-active schedulers with all observations stored in Redis: larger consistency and operational scope.
  • Chain replication between schedulers: requires custom membership, failure detection, reconfiguration, and fencing.
  • Heartbeat fanout to every scheduler: duplicates state and still permits concurrent scheduling races.
  • Current primary plus query-only replicas: does not provide control-plane failover.

Pre-submission checklist

  • I searched existing issues and discussions and did not find a duplicate. #191 is related but covers the existing correctness bug.
  • I described a concrete problem and use case, not only a proposed technology.
  • I understand that acceptance of the problem does not imply approval of a particular implementation.
Originally created by @yingdi-shan on GitHub (Sep 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/259 ### Affected component Gateway / scheduler ### Problem statement AgentENV currently requires one read/write scheduler. Redis-backed query-only replicas preserve existing sandbox routing, but scheduling, node inspection, assignment writes, and P2P APIs remain unavailable if the primary fails. Running multiple read/write schedulers is unsafe because heartbeat state is replica-local. This causes resource limits to be applied inconsistently, as reported in #191. ### Use case Production Kubernetes deployments need scheduler redundancy for runtime failures, upgrades, and node maintenance without operating multiple active schedulers or introducing a custom replication protocol. ### Current behavior and workarounds The Kubernetes deployment runs one scheduler replica. Operators can add query-only replicas backed by Redis, but these provide data-plane lookup availability only. The practical workaround is restarting the single primary scheduler and waiting for heartbeats to rebuild its state. ### Desired behavior - Multiple scheduler pods may be deployed. - Exactly one scheduler performs scheduling and processes node reports. - Standbys participate only in leader election and health reporting. - A standby automatically takes leadership after leader failure. - The promoted leader rebuilds mappings and observations from node heartbeats. - Missing observations during recovery are not treated as unlimited capacity. - Leadership loss immediately stops the old leader from serving. - Liveness remains healthy on standbys; readiness exposes only the leader. - Failover completes within the configured lease duration plus heartbeat recovery time. - Three scheduler replicas pass the resource-limit scenario from #191. ### Proposed approach Use `client-go` leader election with a `coordination.k8s.io/Lease`. Add scheduler configuration for: ```json { "leader_election": { "enabled": true, "lease_name": "agentenv-scheduler", "lease_namespace": "agentenv-system", "lease_duration": "15s", "renew_deadline": "10s", "retry_period": "2s" } } ``` On leadership acquisition: - Start discovery and the scheduler service. - Accept heartbeats immediately. - Schedule only onto nodes with fresh observations. - Return `Unavailable` rather than false `NotFound` results while in-memory bindings are rebuilding. On leadership loss: - Mark the scheduler service non-serving. - Stop accepting RPCs and exit so existing gRPC connections reconnect. Update Kubernetes manifests to use three replicas, grant Lease RBAC permissions, and use separate liveness and leader-readiness checks. Leader election and `--query-only` should be mutually exclusive. ### Compatibility and operational impact API/config changes: Add optional scheduler leader-election configuration. No protobuf API change is expected. Snapshot or storage format changes: None. New host/runtime requirements: Kubernetes Lease API access and corresponding RBAC permissions. Upgrade and rollback considerations: Leader election should remain disabled by default. Existing single-replica and query-only deployments continue to work. Rollback requires scaling the scheduler back to one replica. ### Alternatives considered - Active-active schedulers with all observations stored in Redis: larger consistency and operational scope. - Chain replication between schedulers: requires custom membership, failure detection, reconfiguration, and fencing. - Heartbeat fanout to every scheduler: duplicates state and still permits concurrent scheduling races. - Current primary plus query-only replicas: does not provide control-plane failover. ### Pre-submission checklist - [x] I searched existing issues and discussions and did not find a duplicate. #191 is related but covers the existing correctness bug. - [x] I described a concrete problem and use case, not only a proposed technology. - [x] I understand that acceptance of the problem does not imply approval of a particular implementation.
bsakaguchi commented 2026-10-29 14:19:10 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 22, 2026):

Hi @yingdi-shan,

I'd love to take this one if nobody has started yet.

The Lease-based plan looks solid to me — standbys staying out of the serving path, the promoted leader rebuilding from heartbeats, and returning Unavailable instead of false NotFound during recovery are exactly the details that make or break a failover.

A bit of background on me: I used to work on Redis itself (source-level changes), and before that on SOFA-Registry's meta server, where I spent a lot of time on leader election and slot-table balancing. So leader election over a Redis-backed binding store is familiar territory.

I noticed #232 implemented a very similar design but was closed because it was meant for the tonbo-io fork. Is the intent here to bring that work upstream, or would you prefer a fresh implementation? Either works for me.

If it helps, while doing this I'd also be happy to look at the Redis side of the failover story — e.g. how bindings behave while the old leader is gone (TTL windows, what query-only replicas serve during the gap), and any small wins in the current store usage.

If the scope sounds right, I can start with a short design note here (config surface, failover sequence, RBAC/manifest changes) before writing code — that's the rhythm that worked well for me on #37.

<!-- gh-comment-id:5779670179 --> @NickNYU commented on GitHub (Sep 22, 2026): Hi @yingdi-shan, I'd love to take this one if nobody has started yet. The Lease-based plan looks solid to me — standbys staying out of the serving path, the promoted leader rebuilding from heartbeats, and returning Unavailable instead of false NotFound during recovery are exactly the details that make or break a failover. A bit of background on me: I used to work on Redis itself (source-level changes), and before that on SOFA-Registry's meta server, where I spent a lot of time on leader election and slot-table balancing. So leader election over a Redis-backed binding store is familiar territory. I noticed #232 implemented a very similar design but was closed because it was meant for the tonbo-io fork. Is the intent here to bring that work upstream, or would you prefer a fresh implementation? Either works for me. If it helps, while doing this I'd also be happy to look at the Redis side of the failover story — e.g. how bindings behave while the old leader is gone (TTL windows, what query-only replicas serve during the gap), and any small wins in the current store usage. If the scope sounds right, I can start with a short design note here (config surface, failover sequence, RBAC/manifest changes) before writing code — that's the rhythm that worked well for me on #37.
bsakaguchi commented 2026-10-29 14:19:11 +00:00
Author
Owner
Copy link

@yingdi-shan commented on GitHub (Sep 23, 2026):

Hi! Thanks a lot for the detailed write-up, and we'd be happy for you to take this one.

One change: we'd like leader election to use Kubernetes Leases (e.g. client-go's leaderelection) instead of Redis, so election stays without relying on Redis.

A short design note first sounds great. Looking forward to it!

<!-- gh-comment-id:5790482367 --> @yingdi-shan commented on GitHub (Sep 23, 2026): Hi! Thanks a lot for the detailed write-up, and we'd be happy for you to take this one. One change: we'd like leader election to use Kubernetes Leases (e.g. client-go's leaderelection) instead of Redis, so election stays without relying on Redis. A short design note first sounds great. Looking forward to it!
bsakaguchi commented 2026-10-29 14:19:11 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 23, 2026):

Hi @yingdi-shan,

Great, thank you!

Understood on the election mechanism: client-go leaderelection over a coordination.k8s.io/Lease, with no Redis in the election path. Redis stays where it is today — bindings storage for the query-only path.

I'll put together a short design note covering the config surface, the failover sequence, the liveness/readiness split, and the RBAC/manifest changes, and post it here before writing any code.

<!-- gh-comment-id:5791740885 --> @NickNYU commented on GitHub (Sep 23, 2026): Hi @yingdi-shan, Great, thank you! Understood on the election mechanism: client-go leaderelection over a coordination.k8s.io/Lease, with no Redis in the election path. Redis stays where it is today — bindings storage for the query-only path. I'll put together a short design note covering the config surface, the failover sequence, the liveness/readiness split, and the RBAC/manifest changes, and post it here before writing any code.
bsakaguchi commented 2026-10-29 14:19:11 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 27, 2026):

Hi @yingdi-shan, I did another pass over the design and reworked it. The approach is the same Lease-based one, with two additions from reviewing the code more carefully: an explicit query-only topology for HA deployments (so gateway reads stay off the leader lifecycle), and a spelled-out compatibility story (disabled = today's behavior). Here's the updated design note:

<!-- gh-comment-id:5856147924 --> @NickNYU commented on GitHub (Sep 27, 2026): Hi @yingdi-shan, I did another pass over the design and reworked it. The approach is the same Lease-based one, with two additions from reviewing the code more carefully: an explicit query-only topology for HA deployments (so gateway reads stay off the leader lifecycle), and a spelled-out compatibility story (disabled = today's behavior). Here's the updated design note:
bsakaguchi commented 2026-10-29 14:19:11 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 27, 2026):

Design note: scheduler HA via Kubernetes Lease leader election (#259)

Goal

  1. Multiple scheduler pods
  2. exactly one primary scheduler
  3. standbys elect and wait
  4. failover preserves existing-sandbox routing and never treats missing observations as free capacity
  5. No protobuf changes
  6. leader election off by default

Non-Goal

  1. No change to the heartbeat protocol. Nodes keep pushing; nothing changes in the runtime reporter.
  2. No active-active or sharded scheduling. Standbys collect nothing.
  3. No change to binding TTL. The failover window around binding expiry is a known follow-up and is out of scope here.
  4. No gateway or node-side code changes. Traffic shifts come from Service endpoints, not client logic.
  5. No change to query-only serving logic. It already reads bindings from Redis and stays as-is.

High Level Design

  1. Run 3 scheduler replicas behind one K8S Service. They compete for a single coordination.k8s.io/Lease via client-go leaderelection. One winner; the rest stand by.
  2. Only the leader is ready. The readiness probe checks a leader-specific health service, so Service endpoints contain only the leader. Write traffic (Schedule, RecordAssignment) and heartbeats land on the leader automatically.
  3. A standby still runs its gRPC server, but a leader-gate interceptor answers every Scheduler RPC with Unavailable. Standbys stay liveness-healthy.
  4. Losing the lease kills the process. Clients reconnect through the Service and land on the new leader. A partitioned ex-leader stops serving within renew_deadline.
  5. HA also requires an explicit query-only deployment. Gateway reads (LookupNode) are pinned to it; if a deployment leaves reads on the fallback path, they stay coupled to the leader lifecycle.
  6. A new leader accepts heartbeats immediately. Bindings come from Redis, so existing sandboxes keep their routing. Scheduling is limited to freshly observed nodes until the view is rebuilt.
flowchart LR
    subgraph sched["Scheduler Deployment · 3 replicas, leader-elected"]
        L["Leader<br/>(ready, serving)"]
        S1["Standby"]
        S2["Standby"]
    end
    subgraph qo["Query-only Deployment · always ready"]
        Q1["Query-only"]
        Q2["Query-only"]
    end
    GW["Gateway"]
    RD[("Redis<br/>bindings")]
    NODE["agentenv server nodes"]
    LEASE[["K8S Lease"]]

    NODE -- "④ heartbeat" --> L
    GW -- "① Schedule / ③ RecordAssignment (write)" --> L
    GW -- "② LookupNode (read)" --> Q1
    L <-- "binding write" --> RD
    Q1 <-- "binding read" --> RD
    Q2 <-- "binding read" --> RD
    L <-. "acquire / renew" .-> LEASE
    S1 <-. "elect" .-> LEASE
    S2 <-. "elect" .-> LEASE

Failover budget: lease_duration (detect) + informer sync + one reconnect backoff (rebuild observations). Within this window, reads keep working through the query-only path; new scheduling waits for fresh observations instead of guessing.

Compatibility

With election disabled (default), behavior is identical to today for every existing deployment shape. All changes sit behind the switch.

Low Level Design

1. Config surface

"scheduler": {
  "leader_election": {
    "enabled": false,
    "lease_name": "agentenv-scheduler",
    "lease_namespace": "agentenv-system",
    "lease_duration": "15s",
    "renew_deadline": "10s",
    "retry_period": "2s"
  }
}

Startup validation (fail fast):

  1. leader_election.enabled requires kubernetes discovery and a Redis binding store. In-memory bindings do not survive failover, so existing-sandbox routing would break. Election itself never touches Redis.
  2. Mutually exclusive with --query-only.
  3. Timing follows client-go rules: lease_duration > renew_deadline > retry_period * 2.

2. Leader gate and health probes

  1. Every pod starts the gRPC server and the metrics server at process start, before election.
  2. gRPC registration happens before Serve, so the gate is an interceptor, not late registration. A unary interceptor chained with the existing metrics interceptor returns Unavailable("not the leader") for every Scheduler RPC until the pod holds the lease.
  3. Liveness reports the overall health status: SERVING from process start. Unchanged.
  4. Readiness reports a new health service agentenv.scheduler.Scheduler/leader: SERVING only while holding the lease. The readiness probe switches to grpc_health_probe -service=agentenv.scheduler.Scheduler/leader, so Service endpoints contain only the leader.
  5. Election disabled (default): both probes SERVING from start, identical to today.

3. Leader lifecycle

  1. OnStartedLeading: start (or resume) kubernetes discovery; accept heartbeats; readiness SERVING; enter the recovery window.
  2. OnStoppedLeading: readiness NOT_SERVING, then exit. Exit drops all connections; clients reconnect through the Service to the new leader; the pod restarts as a standby. This is the fence.
  3. OnNewLeader: log the new leader id.

4. Recovery window

Right after winning, the heartbeat-rebuilt in-memory state is empty. Redis bindings are intact. The window lasts until the leader has heard from every live node once (bounded by lease acquisition + one report_ttl).

  1. Heartbeat: accept immediately. Heartbeats are how the window ends.
  2. Schedule: only nodes observed after acquisition are candidates. None available: return Unavailable. Never fall back to unobserved nodes. This is the #191 fix: no snapshot means "unknown", not "unlimited".
  3. LookupNode / GetNode: serve from Redis as usual. A missing binding returns Unavailable while the window is open, NotFound after it closes, so "not yet rebuilt" is not reported as "does not exist".
  4. P2P artifact APIs: rebuilt from heartbeats like observations; same fresh-only rule.

5. HA topology: query-only as an explicit component

  1. Gateway has two scheduler clients today: writes go to the elected Service; reads go to a query-only address, falling back to the primary when unconfigured.
  2. Under HA, that fallback is a liability: read availability would track the leader lifecycle. Deployments enable HA by pointing gateway reads at a dedicated query-only Deployment and Service.
  3. Query-only pods stay always-ready, never join election, and keep serving from Redis during failover.
  4. Traffic rules: node heartbeats to the elected Service; gateway writes to the elected Service; gateway reads to the query-only Service.

6. Upgrade and rollback ordering

  1. Upgrade the scheduler binary first. New binaries behave exactly like today while election is off.
  2. Then update manifests: RBAC, readiness probe, replicas, config.
  3. Reversing the config and scaling back to one replica is the rollback.

7. Manifests and RBAC

  1. role.yaml: grant coordination.k8s.io leases get/create/update.
  2. scheduler-deployment.yaml: replicas: 3; readiness probe adds -service=.../leader; liveness unchanged.
  3. New query-only Deployment and Service manifests for HA deployments.
  4. deploy/k8s/base/config/scheduler.json: enable leader_election. Code default stays disabled, so static and docker-compose deployments are unaffected.

8. Testing

  1. Unit: leader gate allow/deny; config validation (bad timing triple, query-only conflict, missing redis/discovery); recovery-window fresh-observation filtering; LookupNode Unavailable vs NotFound.
  2. Integration (Kind, 3 replicas): kill the leader, assert failover within the lease + reconnect budget; run the #191 resource-limit scenario and assert it passes; assert standbys reject Schedule with Unavailable while staying liveness-healthy; assert existing-sandbox routing survives via the query-only path.

9. Future work: warm standbys via sharded collection

This design keeps standby state cold, which is exactly why the recovery window exists. A later evolution: shard node observations across replicas (consistent hashing, not plain modulo), guard each shard with a Redis lease + fencing token, and write observations to a shared store readable on the scheduling path. Standbys become warm, failover needs no rebuild, and the recovery window goes away.

That change moves scheduling's correctness argument from "one decider" to "fresh enough shared state", so it needs its own design discussion and stays out of scope here.

Open Questions

  1. Require Redis when leader election is enabled? Recommended yes, same as #232. Alternative: allow in-memory bindings and document that routing for pre-failover sandboxes is lost.
  2. The readiness health service name agentenv.scheduler.Scheduler/leader — any preference?
  3. Query-only topology: a dedicated Deployment plus Service, or fold it into the same Deployment with a role label? A separate Deployment keeps the two roles independently scalable.
<!-- gh-comment-id:5856150402 --> @NickNYU commented on GitHub (Sep 27, 2026): # Design note: scheduler HA via Kubernetes Lease leader election (#259) ## Goal 1. Multiple scheduler pods 2. exactly one primary scheduler 3. standbys elect and wait 4. failover preserves existing-sandbox routing and never treats missing observations as free capacity 5. No protobuf changes 6. leader election off by default ## Non-Goal 1. No change to the heartbeat protocol. Nodes keep pushing; nothing changes in the runtime reporter. 2. No active-active or sharded scheduling. Standbys collect nothing. 3. No change to binding TTL. The failover window around binding expiry is a known follow-up and is out of scope here. 4. No gateway or node-side code changes. Traffic shifts come from Service endpoints, not client logic. 5. No change to query-only serving logic. It already reads bindings from Redis and stays as-is. ## High Level Design 1. Run 3 scheduler replicas behind one K8S Service. They compete for a single `coordination.k8s.io/Lease` via client-go leaderelection. One winner; the rest stand by. 2. Only the leader is ready. The readiness probe checks a leader-specific health service, so Service endpoints contain only the leader. Write traffic (Schedule, RecordAssignment) and heartbeats land on the leader automatically. 3. A standby still runs its gRPC server, but a leader-gate interceptor answers every Scheduler RPC with `Unavailable`. Standbys stay liveness-healthy. 4. Losing the lease kills the process. Clients reconnect through the Service and land on the new leader. A partitioned ex-leader stops serving within `renew_deadline`. 5. HA also requires an explicit query-only deployment. Gateway reads (LookupNode) are pinned to it; if a deployment leaves reads on the fallback path, they stay coupled to the leader lifecycle. 6. A new leader accepts heartbeats immediately. Bindings come from Redis, so existing sandboxes keep their routing. Scheduling is limited to freshly observed nodes until the view is rebuilt. ```mermaid flowchart LR subgraph sched["Scheduler Deployment · 3 replicas, leader-elected"] L["Leader<br/>(ready, serving)"] S1["Standby"] S2["Standby"] end subgraph qo["Query-only Deployment · always ready"] Q1["Query-only"] Q2["Query-only"] end GW["Gateway"] RD[("Redis<br/>bindings")] NODE["agentenv server nodes"] LEASE[["K8S Lease"]] NODE -- "④ heartbeat" --> L GW -- "① Schedule / ③ RecordAssignment (write)" --> L GW -- "② LookupNode (read)" --> Q1 L <-- "binding write" --> RD Q1 <-- "binding read" --> RD Q2 <-- "binding read" --> RD L <-. "acquire / renew" .-> LEASE S1 <-. "elect" .-> LEASE S2 <-. "elect" .-> LEASE ``` Failover budget: `lease_duration` (detect) + informer sync + one reconnect backoff (rebuild observations). Within this window, reads keep working through the query-only path; new scheduling waits for fresh observations instead of guessing. ### Compatibility With election disabled (default), behavior is identical to today for every existing deployment shape. All changes sit behind the switch. ## Low Level Design ### 1. Config surface ```json "scheduler": { "leader_election": { "enabled": false, "lease_name": "agentenv-scheduler", "lease_namespace": "agentenv-system", "lease_duration": "15s", "renew_deadline": "10s", "retry_period": "2s" } } ``` Startup validation (fail fast): 1. `leader_election.enabled` requires kubernetes discovery and a Redis binding store. In-memory bindings do not survive failover, so existing-sandbox routing would break. Election itself never touches Redis. 2. Mutually exclusive with `--query-only`. 3. Timing follows client-go rules: `lease_duration > renew_deadline > retry_period * 2`. ### 2. Leader gate and health probes 1. Every pod starts the gRPC server and the metrics server at process start, before election. 2. gRPC registration happens before `Serve`, so the gate is an interceptor, not late registration. A unary interceptor chained with the existing metrics interceptor returns `Unavailable("not the leader")` for every Scheduler RPC until the pod holds the lease. 3. Liveness reports the overall health status: `SERVING` from process start. Unchanged. 4. Readiness reports a new health service `agentenv.scheduler.Scheduler/leader`: `SERVING` only while holding the lease. The readiness probe switches to `grpc_health_probe -service=agentenv.scheduler.Scheduler/leader`, so Service endpoints contain only the leader. 5. Election disabled (default): both probes `SERVING` from start, identical to today. ### 3. Leader lifecycle 1. `OnStartedLeading`: start (or resume) kubernetes discovery; accept heartbeats; readiness `SERVING`; enter the recovery window. 2. `OnStoppedLeading`: readiness `NOT_SERVING`, then exit. Exit drops all connections; clients reconnect through the Service to the new leader; the pod restarts as a standby. This is the fence. 3. `OnNewLeader`: log the new leader id. ### 4. Recovery window Right after winning, the heartbeat-rebuilt in-memory state is empty. Redis bindings are intact. The window lasts until the leader has heard from every live node once (bounded by lease acquisition + one `report_ttl`). 1. `Heartbeat`: accept immediately. Heartbeats are how the window ends. 2. `Schedule`: only nodes observed after acquisition are candidates. None available: return `Unavailable`. Never fall back to unobserved nodes. This is the #191 fix: no snapshot means "unknown", not "unlimited". 3. `LookupNode` / `GetNode`: serve from Redis as usual. A missing binding returns `Unavailable` while the window is open, `NotFound` after it closes, so "not yet rebuilt" is not reported as "does not exist". 4. P2P artifact APIs: rebuilt from heartbeats like observations; same fresh-only rule. ### 5. HA topology: query-only as an explicit component 1. Gateway has two scheduler clients today: writes go to the elected Service; reads go to a query-only address, falling back to the primary when unconfigured. 2. Under HA, that fallback is a liability: read availability would track the leader lifecycle. Deployments enable HA by pointing gateway reads at a dedicated query-only Deployment and Service. 3. Query-only pods stay always-ready, never join election, and keep serving from Redis during failover. 4. Traffic rules: node heartbeats to the elected Service; gateway writes to the elected Service; gateway reads to the query-only Service. ### 6. Upgrade and rollback ordering 1. Upgrade the scheduler binary first. New binaries behave exactly like today while election is off. 2. Then update manifests: RBAC, readiness probe, replicas, config. 3. Reversing the config and scaling back to one replica is the rollback. ### 7. Manifests and RBAC 1. `role.yaml`: grant `coordination.k8s.io` `leases` get/create/update. 2. `scheduler-deployment.yaml`: `replicas: 3`; readiness probe adds `-service=.../leader`; liveness unchanged. 3. New `query-only` Deployment and Service manifests for HA deployments. 4. `deploy/k8s/base/config/scheduler.json`: enable `leader_election`. Code default stays disabled, so static and docker-compose deployments are unaffected. ### 8. Testing 1. Unit: leader gate allow/deny; config validation (bad timing triple, query-only conflict, missing redis/discovery); recovery-window fresh-observation filtering; LookupNode `Unavailable` vs `NotFound`. 2. Integration (Kind, 3 replicas): kill the leader, assert failover within the lease + reconnect budget; run the #191 resource-limit scenario and assert it passes; assert standbys reject `Schedule` with `Unavailable` while staying liveness-healthy; assert existing-sandbox routing survives via the query-only path. ### 9. Future work: warm standbys via sharded collection This design keeps standby state cold, which is exactly why the recovery window exists. A later evolution: shard node observations across replicas (consistent hashing, not plain modulo), guard each shard with a Redis lease + fencing token, and write observations to a shared store readable on the scheduling path. Standbys become warm, failover needs no rebuild, and the recovery window goes away. That change moves scheduling's correctness argument from "one decider" to "fresh enough shared state", so it needs its own design discussion and stays out of scope here. ## Open Questions 1. Require Redis when leader election is enabled? Recommended yes, same as #232. Alternative: allow in-memory bindings and document that routing for pre-failover sandboxes is lost. 2. The readiness health service name `agentenv.scheduler.Scheduler/leader` — any preference? 3. Query-only topology: a dedicated Deployment plus Service, or fold it into the same Deployment with a role label? A separate Deployment keeps the two roles independently scalable.
bsakaguchi commented 2026-10-29 14:19:11 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 27, 2026):

Test plan for the design above

Unit cases are the CI floor; Kind integration cases assert end-to-end invariants. F1 is the signature case: it turns "failover preserves existing-sandbox routing" into an automated assertion.

A. Config validation (startup fail-fast, unit)

# Case Input Expected
A1 LE on, no redis_addr leader_election.enabled=true, no redis_addr startup fails with "LE requires Redis binding store"
A2 LE on, non-k8s discovery discovery.mode=static, LE on startup fails, requires kubernetes discovery
A3 LE on with --query-only both flags set startup fails, flags are mutually exclusive
A4 invalid timing triple lease_duration=10s, renew_deadline=15s startup fails, timing constraint reported
A5 election off (regression) default config starts fine, behavior identical to today

B. Leader gate and probes (unit + Kind)

# Case Input Expected
B1 standby rejects writes Schedule against a standby Unavailable("not the leader"), no side effects
B2 standby rejects heartbeats Heartbeat against a standby same (gate covers every Scheduler RPC)
B3 health check bypass grpc health check against a standby not gated, normal response (else probes deadlock)
B4 standby probes liveness / readiness on a standby liveness=SERVING, readiness=NOT_SERVING, absent from Service endpoints
B5 leader probes both probes on the leader both SERVING, present in endpoints

C. Election lifecycle (Kind)

# Case Input Expected
C1 acquire 3 replicas start, Lease empty exactly one leader; OnNewLeader logged
C2 failover kill -9 the leader pod new leader ready within lease_duration (15s), endpoints converge
C3 fencing (partition) block leader ↔ API server traffic old leader exits within renew_deadline (10s); new leader takes over; no dual-primary window
C4 graceful handoff rolling-upgrade (delete leader pod) same as C2, clients reconnect transparently

D. Recovery window (unit + Kind)

# Case Input Expected
D1 fresh-only scheduling new leader, some nodes not yet heartbeating Schedule only picks nodes observed after acquisition
D2 no fresh observations zero heartbeats since acquisition Schedule returns Unavailable; never schedules onto unobserved nodes (the #191 assertion)
D3 per-node convergence first heartbeat from a node arrives node immediately becomes schedulable (window closes per node)
D4 LookupNode during window binding exists in Redis returns the node (reads are window-independent)
D5 missing binding during window binding absent from Redis Unavailable, not NotFound ("not yet rebuilt" ≠ "does not exist")
D6 after window same lookup after the window closes NotFound

E. #191 scenario (Kind)

# Case Input Expected
E1 three-replica ceiling 3 replicas, max_sandbox_count=10, create 200 sandboxes at full speed every node stays ≤10; Schedule returns Unavailable once at ceiling; zero overshoot (vs. the measured 3.0x in #191)
E2 observation coverage leader's observed node set at any time equals all nodes (heartbeats converge on the leader)

F. Failover end-to-end (Kind)

# Case Input Expected
F1 routing continuity create N sandboxes, kill the leader, keep calling LookupNode for old sandboxes via query-only correct node every time; zero NotFound, zero gap (the issue's core assertion, automated)
F2 binding TTL floor (phase 3) F1 scenario with bindings silent for 80s bindings survive; F1's result still holds
F3 new-sandbox recovery create sandboxes after failover Schedule works again once observations rebuild, lands on healthy nodes
F4 rollback disable election, scale back to 1 replica behavior returns to the single-replica shape (upgrade path is reversible)

Layering. Unit (A/B/D) is the CI floor. Kind (C/E/F) asserts one end-to-end invariant per case. F1 is deliberately designed to pass without the TTL floor (nodes re-heartbeat quickly in a normal failover); F2 is the floor's dedicated assertion, so a floor regression is pinpointed. If only three cases survive, keep F1, E1, C3.

<!-- gh-comment-id:5861014705 --> @NickNYU commented on GitHub (Sep 27, 2026): ### Test plan for the design above Unit cases are the CI floor; Kind integration cases assert end-to-end invariants. F1 is the signature case: it turns "failover preserves existing-sandbox routing" into an automated assertion. **A. Config validation (startup fail-fast, unit)** | # | Case | Input | Expected | |---|---|---|---| | A1 | LE on, no redis_addr | `leader_election.enabled=true`, no `redis_addr` | startup fails with "LE requires Redis binding store" | | A2 | LE on, non-k8s discovery | `discovery.mode=static`, LE on | startup fails, requires kubernetes discovery | | A3 | LE on with `--query-only` | both flags set | startup fails, flags are mutually exclusive | | A4 | invalid timing triple | `lease_duration=10s, renew_deadline=15s` | startup fails, timing constraint reported | | A5 | election off (regression) | default config | starts fine, behavior identical to today | **B. Leader gate and probes (unit + Kind)** | # | Case | Input | Expected | |---|---|---|---| | B1 | standby rejects writes | `Schedule` against a standby | `Unavailable("not the leader")`, no side effects | | B2 | standby rejects heartbeats | `Heartbeat` against a standby | same (gate covers every Scheduler RPC) | | B3 | health check bypass | grpc health check against a standby | not gated, normal response (else probes deadlock) | | B4 | standby probes | liveness / readiness on a standby | liveness=`SERVING`, readiness=`NOT_SERVING`, absent from Service endpoints | | B5 | leader probes | both probes on the leader | both `SERVING`, present in endpoints | **C. Election lifecycle (Kind)** | # | Case | Input | Expected | |---|---|---|---| | C1 | acquire | 3 replicas start, Lease empty | exactly one leader; `OnNewLeader` logged | | C2 | failover | `kill -9` the leader pod | new leader ready within `lease_duration` (15s), endpoints converge | | C3 | fencing (partition) | block leader ↔ API server traffic | old leader exits within `renew_deadline` (10s); new leader takes over; no dual-primary window | | C4 | graceful handoff | rolling-upgrade (delete leader pod) | same as C2, clients reconnect transparently | **D. Recovery window (unit + Kind)** | # | Case | Input | Expected | |---|---|---|---| | D1 | fresh-only scheduling | new leader, some nodes not yet heartbeating | `Schedule` only picks nodes observed after acquisition | | D2 | no fresh observations | zero heartbeats since acquisition | `Schedule` returns `Unavailable`; never schedules onto unobserved nodes (the #191 assertion) | | D3 | per-node convergence | first heartbeat from a node arrives | node immediately becomes schedulable (window closes per node) | | D4 | LookupNode during window | binding exists in Redis | returns the node (reads are window-independent) | | D5 | missing binding during window | binding absent from Redis | `Unavailable`, not `NotFound` ("not yet rebuilt" ≠ "does not exist") | | D6 | after window | same lookup after the window closes | `NotFound` | **E. #191 scenario (Kind)** | # | Case | Input | Expected | |---|---|---|---| | E1 | three-replica ceiling | 3 replicas, `max_sandbox_count=10`, create 200 sandboxes at full speed | every node stays ≤10; `Schedule` returns `Unavailable` once at ceiling; zero overshoot (vs. the measured 3.0x in #191) | | E2 | observation coverage | leader's observed node set at any time | equals all nodes (heartbeats converge on the leader) | **F. Failover end-to-end (Kind)** | # | Case | Input | Expected | |---|---|---|---| | F1 | **routing continuity** | create N sandboxes, kill the leader, keep calling `LookupNode` for old sandboxes via query-only | correct node every time; zero NotFound, zero gap (the issue's core assertion, automated) | | F2 | binding TTL floor (phase 3) | F1 scenario with bindings silent for 80s | bindings survive; F1's result still holds | | F3 | new-sandbox recovery | create sandboxes after failover | `Schedule` works again once observations rebuild, lands on healthy nodes | | F4 | rollback | disable election, scale back to 1 replica | behavior returns to the single-replica shape (upgrade path is reversible) | **Layering.** Unit (A/B/D) is the CI floor. Kind (C/E/F) asserts one end-to-end invariant per case. F1 is deliberately designed to pass without the TTL floor (nodes re-heartbeat quickly in a normal failover); F2 is the floor's dedicated assertion, so a floor regression is pinpointed. If only three cases survive, keep F1, E1, C3.
bsakaguchi commented 2026-10-29 14:19:12 +00:00
Author
Owner
Copy link

@yingdi-shan commented on GitHub (Sep 28, 2026):

Design note: scheduler HA via Kubernetes Lease leader election (#259)

Goal

  1. Multiple scheduler pods
  2. exactly one primary scheduler
  3. standbys elect and wait
  4. failover preserves existing-sandbox routing and never treats missing observations as free capacity
  5. No protobuf changes
  6. leader election off by default

Non-Goal

  1. No change to the heartbeat protocol. Nodes keep pushing; nothing changes in the runtime reporter.
  2. No active-active or sharded scheduling. Standbys collect nothing.
  3. No change to binding TTL. The failover window around binding expiry is a known follow-up and is out of scope here.
  4. No gateway or node-side code changes. Traffic shifts come from Service endpoints, not client logic.
  5. No change to query-only serving logic. It already reads bindings from Redis and stays as-is.

High Level Design

  1. Run 3 scheduler replicas behind one K8S Service. They compete for a single coordination.k8s.io/Lease via client-go leaderelection. One winner; the rest stand by.
  2. Only the leader is ready. The readiness probe checks a leader-specific health service, so Service endpoints contain only the leader. Write traffic (Schedule, RecordAssignment) and heartbeats land on the leader automatically.
  3. A standby still runs its gRPC server, but a leader-gate interceptor answers every Scheduler RPC with Unavailable. Standbys stay liveness-healthy.
  4. Losing the lease kills the process. Clients reconnect through the Service and land on the new leader. A partitioned ex-leader stops serving within renew_deadline.
  5. HA also requires an explicit query-only deployment. Gateway reads (LookupNode) are pinned to it; if a deployment leaves reads on the fallback path, they stay coupled to the leader lifecycle.
  6. A new leader accepts heartbeats immediately. Bindings come from Redis, so existing sandboxes keep their routing. Scheduling is limited to freshly observed nodes until the view is rebuilt.

Failover budget: lease_duration (detect) + informer sync + one reconnect backoff (rebuild observations). Within this window, reads keep working through the query-only path; new scheduling waits for fresh observations instead of guessing.

Compatibility

With election disabled (default), behavior is identical to today for every existing deployment shape. All changes sit behind the switch.

Low Level Design

1. Config surface

"scheduler": {
"leader_election": {
"enabled": false,
"lease_name": "agentenv-scheduler",
"lease_namespace": "agentenv-system",
"lease_duration": "15s",
"renew_deadline": "10s",
"retry_period": "2s"
}
}
Startup validation (fail fast):

  1. leader_election.enabled requires kubernetes discovery and a Redis binding store. In-memory bindings do not survive failover, so existing-sandbox routing would break. Election itself never touches Redis.
  2. Mutually exclusive with --query-only.
  3. Timing follows client-go rules: lease_duration > renew_deadline > retry_period * 2.

2. Leader gate and health probes

  1. Every pod starts the gRPC server and the metrics server at process start, before election.
  2. gRPC registration happens before Serve, so the gate is an interceptor, not late registration. A unary interceptor chained with the existing metrics interceptor returns Unavailable("not the leader") for every Scheduler RPC until the pod holds the lease.
  3. Liveness reports the overall health status: SERVING from process start. Unchanged.
  4. Readiness reports a new health service agentenv.scheduler.Scheduler/leader: SERVING only while holding the lease. The readiness probe switches to grpc_health_probe -service=agentenv.scheduler.Scheduler/leader, so Service endpoints contain only the leader.
  5. Election disabled (default): both probes SERVING from start, identical to today.

3. Leader lifecycle

  1. OnStartedLeading: start (or resume) kubernetes discovery; accept heartbeats; readiness SERVING; enter the recovery window.
  2. OnStoppedLeading: readiness NOT_SERVING, then exit. Exit drops all connections; clients reconnect through the Service to the new leader; the pod restarts as a standby. This is the fence.
  3. OnNewLeader: log the new leader id.

4. Recovery window

Right after winning, the heartbeat-rebuilt in-memory state is empty. Redis bindings are intact. The window lasts until the leader has heard from every live node once (bounded by lease acquisition + one report_ttl).

  1. Heartbeat: accept immediately. Heartbeats are how the window ends.
  2. Schedule: only nodes observed after acquisition are candidates. None available: return Unavailable. Never fall back to unobserved nodes. This is the scheduler.node_resource_limit is silently unenforced for most nodes when the scheduler runs more than one replica #191 fix: no snapshot means "unknown", not "unlimited".
  3. LookupNode / GetNode: serve from Redis as usual. A missing binding returns Unavailable while the window is open, NotFound after it closes, so "not yet rebuilt" is not reported as "does not exist".
  4. P2P artifact APIs: rebuilt from heartbeats like observations; same fresh-only rule.

5. HA topology: query-only as an explicit component

  1. Gateway has two scheduler clients today: writes go to the elected Service; reads go to a query-only address, falling back to the primary when unconfigured.
  2. Under HA, that fallback is a liability: read availability would track the leader lifecycle. Deployments enable HA by pointing gateway reads at a dedicated query-only Deployment and Service.
  3. Query-only pods stay always-ready, never join election, and keep serving from Redis during failover.
  4. Traffic rules: node heartbeats to the elected Service; gateway writes to the elected Service; gateway reads to the query-only Service.

6. Upgrade and rollback ordering

  1. Upgrade the scheduler binary first. New binaries behave exactly like today while election is off.
  2. Then update manifests: RBAC, readiness probe, replicas, config.
  3. Reversing the config and scaling back to one replica is the rollback.

7. Manifests and RBAC

  1. role.yaml: grant coordination.k8s.io leases get/create/update.
  2. scheduler-deployment.yaml: replicas: 3; readiness probe adds -service=.../leader; liveness unchanged.
  3. New query-only Deployment and Service manifests for HA deployments.
  4. deploy/k8s/base/config/scheduler.json: enable leader_election. Code default stays disabled, so static and docker-compose deployments are unaffected.

8. Testing

  1. Unit: leader gate allow/deny; config validation (bad timing triple, query-only conflict, missing redis/discovery); recovery-window fresh-observation filtering; LookupNode Unavailable vs NotFound.
  2. Integration (Kind, 3 replicas): kill the leader, assert failover within the lease + reconnect budget; run the scheduler.node_resource_limit is silently unenforced for most nodes when the scheduler runs more than one replica #191 resource-limit scenario and assert it passes; assert standbys reject Schedule with Unavailable while staying liveness-healthy; assert existing-sandbox routing survives via the query-only path.

9. Future work: warm standbys via sharded collection

This design keeps standby state cold, which is exactly why the recovery window exists. A later evolution: shard node observations across replicas (consistent hashing, not plain modulo), guard each shard with a Redis lease + fencing token, and write observations to a shared store readable on the scheduling path. Standbys become warm, failover needs no rebuild, and the recovery window goes away.

That change moves scheduling's correctness argument from "one decider" to "fresh enough shared state", so it needs its own design discussion and stays out of scope here.

Open Questions

  1. Require Redis when leader election is enabled? Recommended yes, same as feat: add single-active scheduler HA #232. Alternative: allow in-memory bindings and document that routing for pre-failover sandboxes is lost.
  2. The readiness health service name agentenv.scheduler.Scheduler/leader — any preference?
  3. Query-only topology: a dedicated Deployment plus Service, or fold it into the same Deployment with a role label? A separate Deployment keeps the two roles independently scalable.

Thanks! I'd suggest keeping Redis optional and running the scheduler as a single Deployment with multiple replicas, rather than requiring a separate query-only Deployment.

Only the leader handles scheduling, assignment writes, and heartbeats. When Redis is configured, every standby replica acts as a query-only node and can serve LookupNode from the shared bindings. Without Redis, lookups are routed to the leader. After a failover, the new leader rebuilds the sandbox-to-node mappings by proactively sending Heartbeat RPCs to every node.

The readiness health service name agentenv.scheduler.Scheduler/leader works for me.

<!-- gh-comment-id:5864059821 --> @yingdi-shan commented on GitHub (Sep 28, 2026): > # Design note: scheduler HA via Kubernetes Lease leader election ([#259](https://github.com/kvcache-ai/AgentENV/issues/259)) > ## Goal > 1. Multiple scheduler pods > 2. exactly one primary scheduler > 3. standbys elect and wait > 4. failover preserves existing-sandbox routing and never treats missing observations as free capacity > 5. No protobuf changes > 6. leader election off by default > > ## Non-Goal > 1. No change to the heartbeat protocol. Nodes keep pushing; nothing changes in the runtime reporter. > 2. No active-active or sharded scheduling. Standbys collect nothing. > 3. No change to binding TTL. The failover window around binding expiry is a known follow-up and is out of scope here. > 4. No gateway or node-side code changes. Traffic shifts come from Service endpoints, not client logic. > 5. No change to query-only serving logic. It already reads bindings from Redis and stays as-is. > > ## High Level Design > 1. Run 3 scheduler replicas behind one K8S Service. They compete for a single `coordination.k8s.io/Lease` via client-go leaderelection. One winner; the rest stand by. > 2. Only the leader is ready. The readiness probe checks a leader-specific health service, so Service endpoints contain only the leader. Write traffic (Schedule, RecordAssignment) and heartbeats land on the leader automatically. > 3. A standby still runs its gRPC server, but a leader-gate interceptor answers every Scheduler RPC with `Unavailable`. Standbys stay liveness-healthy. > 4. Losing the lease kills the process. Clients reconnect through the Service and land on the new leader. A partitioned ex-leader stops serving within `renew_deadline`. > 5. HA also requires an explicit query-only deployment. Gateway reads (LookupNode) are pinned to it; if a deployment leaves reads on the fallback path, they stay coupled to the leader lifecycle. > 6. A new leader accepts heartbeats immediately. Bindings come from Redis, so existing sandboxes keep their routing. Scheduling is limited to freshly observed nodes until the view is rebuilt. > > > Failover budget: `lease_duration` (detect) + informer sync + one reconnect backoff (rebuild observations). Within this window, reads keep working through the query-only path; new scheduling waits for fresh observations instead of guessing. > > ### Compatibility > With election disabled (default), behavior is identical to today for every existing deployment shape. All changes sit behind the switch. > > ## Low Level Design > ### 1. Config surface > "scheduler": { > "leader_election": { > "enabled": false, > "lease_name": "agentenv-scheduler", > "lease_namespace": "agentenv-system", > "lease_duration": "15s", > "renew_deadline": "10s", > "retry_period": "2s" > } > } > Startup validation (fail fast): > > 1. `leader_election.enabled` requires kubernetes discovery and a Redis binding store. In-memory bindings do not survive failover, so existing-sandbox routing would break. Election itself never touches Redis. > 2. Mutually exclusive with `--query-only`. > 3. Timing follows client-go rules: `lease_duration > renew_deadline > retry_period * 2`. > > ### 2. Leader gate and health probes > 1. Every pod starts the gRPC server and the metrics server at process start, before election. > 2. gRPC registration happens before `Serve`, so the gate is an interceptor, not late registration. A unary interceptor chained with the existing metrics interceptor returns `Unavailable("not the leader")` for every Scheduler RPC until the pod holds the lease. > 3. Liveness reports the overall health status: `SERVING` from process start. Unchanged. > 4. Readiness reports a new health service `agentenv.scheduler.Scheduler/leader`: `SERVING` only while holding the lease. The readiness probe switches to `grpc_health_probe -service=agentenv.scheduler.Scheduler/leader`, so Service endpoints contain only the leader. > 5. Election disabled (default): both probes `SERVING` from start, identical to today. > > ### 3. Leader lifecycle > 1. `OnStartedLeading`: start (or resume) kubernetes discovery; accept heartbeats; readiness `SERVING`; enter the recovery window. > 2. `OnStoppedLeading`: readiness `NOT_SERVING`, then exit. Exit drops all connections; clients reconnect through the Service to the new leader; the pod restarts as a standby. This is the fence. > 3. `OnNewLeader`: log the new leader id. > > ### 4. Recovery window > Right after winning, the heartbeat-rebuilt in-memory state is empty. Redis bindings are intact. The window lasts until the leader has heard from every live node once (bounded by lease acquisition + one `report_ttl`). > > 1. `Heartbeat`: accept immediately. Heartbeats are how the window ends. > 2. `Schedule`: only nodes observed after acquisition are candidates. None available: return `Unavailable`. Never fall back to unobserved nodes. This is the [scheduler.node_resource_limit is silently unenforced for most nodes when the scheduler runs more than one replica #191](https://github.com/kvcache-ai/AgentENV/issues/191) fix: no snapshot means "unknown", not "unlimited". > 3. `LookupNode` / `GetNode`: serve from Redis as usual. A missing binding returns `Unavailable` while the window is open, `NotFound` after it closes, so "not yet rebuilt" is not reported as "does not exist". > 4. P2P artifact APIs: rebuilt from heartbeats like observations; same fresh-only rule. > > ### 5. HA topology: query-only as an explicit component > 1. Gateway has two scheduler clients today: writes go to the elected Service; reads go to a query-only address, falling back to the primary when unconfigured. > 2. Under HA, that fallback is a liability: read availability would track the leader lifecycle. Deployments enable HA by pointing gateway reads at a dedicated query-only Deployment and Service. > 3. Query-only pods stay always-ready, never join election, and keep serving from Redis during failover. > 4. Traffic rules: node heartbeats to the elected Service; gateway writes to the elected Service; gateway reads to the query-only Service. > > ### 6. Upgrade and rollback ordering > 1. Upgrade the scheduler binary first. New binaries behave exactly like today while election is off. > 2. Then update manifests: RBAC, readiness probe, replicas, config. > 3. Reversing the config and scaling back to one replica is the rollback. > > ### 7. Manifests and RBAC > 1. `role.yaml`: grant `coordination.k8s.io` `leases` get/create/update. > 2. `scheduler-deployment.yaml`: `replicas: 3`; readiness probe adds `-service=.../leader`; liveness unchanged. > 3. New `query-only` Deployment and Service manifests for HA deployments. > 4. `deploy/k8s/base/config/scheduler.json`: enable `leader_election`. Code default stays disabled, so static and docker-compose deployments are unaffected. > > ### 8. Testing > 1. Unit: leader gate allow/deny; config validation (bad timing triple, query-only conflict, missing redis/discovery); recovery-window fresh-observation filtering; LookupNode `Unavailable` vs `NotFound`. > 2. Integration (Kind, 3 replicas): kill the leader, assert failover within the lease + reconnect budget; run the [scheduler.node_resource_limit is silently unenforced for most nodes when the scheduler runs more than one replica #191](https://github.com/kvcache-ai/AgentENV/issues/191) resource-limit scenario and assert it passes; assert standbys reject `Schedule` with `Unavailable` while staying liveness-healthy; assert existing-sandbox routing survives via the query-only path. > > ### 9. Future work: warm standbys via sharded collection > This design keeps standby state cold, which is exactly why the recovery window exists. A later evolution: shard node observations across replicas (consistent hashing, not plain modulo), guard each shard with a Redis lease + fencing token, and write observations to a shared store readable on the scheduling path. Standbys become warm, failover needs no rebuild, and the recovery window goes away. > > That change moves scheduling's correctness argument from "one decider" to "fresh enough shared state", so it needs its own design discussion and stays out of scope here. > > ## Open Questions > 1. Require Redis when leader election is enabled? Recommended yes, same as [feat: add single-active scheduler HA #232](https://github.com/kvcache-ai/AgentENV/pull/232). Alternative: allow in-memory bindings and document that routing for pre-failover sandboxes is lost. > 2. The readiness health service name `agentenv.scheduler.Scheduler/leader` — any preference? > 3. Query-only topology: a dedicated Deployment plus Service, or fold it into the same Deployment with a role label? A separate Deployment keeps the two roles independently scalable. Thanks! I'd suggest keeping Redis optional and running the scheduler as a single Deployment with multiple replicas, rather than requiring a separate query-only Deployment. Only the leader handles scheduling, assignment writes, and heartbeats. When Redis is configured, every standby replica acts as a query-only node and can serve LookupNode from the shared bindings. Without Redis, lookups are routed to the leader. After a failover, the new leader rebuilds the sandbox-to-node mappings by proactively sending Heartbeat RPCs to every node. The readiness health service name agentenv.scheduler.Scheduler/leader works for me.
bsakaguchi commented 2026-10-29 14:19:12 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Sep 28, 2026):

Hi @yingdi-shan
Thanks for reply, this works for me. Adjusted design, in short:

Single Deployment, standbys double as readers.

  1. One scheduler Deployment with N replicas behind one Service.

  2. The leader gate becomes a read/write matrix instead of a blanket reject:

    RPC category Leader Standby (with Redis) Standby (without Redis)
    Writes: Schedule, RecordAssignment, Heartbeat, sandbox events, P2P writes serves Unavailable Unavailable
    Reads: LookupNode, GetNode serves serves (from Redis) Unavailable (retries land on leader)
    health check — never gated same

    No separate query-only Deployment, no gateway changes.

Redis stays optional.

  1. With Redis, standbys serve reads as above.
  2. Without Redis, a standby rejects reads too, and lookups reach the leader via client retry — degraded mode, documented as "routing for pre-failover sandboxes is lost on failover".

Startup validation keeps only the --query-only flag conflict and the client-go timing check.

Active rebuild on leadership acquisition.

  1. On OnStartedLeading, the new leader walks its node registry
    1. the EndpointSlice informer already runs on every replica from process start, so the list is warm
    2. then pulls each node's admin /nodes snapshot, which the agentenv server already exposes with the same fields as a heartbeat.

As the responses go through the exact same ingest path as Heartbeat (update observations, ReconcileNode to refresh bindings), so observations and bindings rebuild in one RTT per node instead of one reporter backoff.

  1. The fresh-only scheduling rule stays as a guard until the pull completes. The informer itself keeps running on standbys — it is read-only watch state, not an observation, so this does not reopen #191.

One confirmation:

in the no-Redis degraded mode, is "standby rejects reads with Unavailable, client retries land on the leader" the behavior you have in mind for lookups?
@yingdi-shan
Thanks

<!-- gh-comment-id:5872298475 --> @NickNYU commented on GitHub (Sep 28, 2026): Hi @yingdi-shan Thanks for reply, this works for me. Adjusted design, in short: **Single Deployment, standbys double as readers.** 1. One scheduler Deployment with N replicas behind one Service. 2. The leader gate becomes a read/write matrix instead of a blanket reject: | RPC category | Leader | Standby (with Redis) | Standby (without Redis) | |---|---|---|---| | Writes: Schedule, RecordAssignment, Heartbeat, sandbox events, P2P writes | serves | `Unavailable` | `Unavailable` | | Reads: LookupNode, GetNode | serves | serves (from Redis) | `Unavailable` (retries land on leader) | | health check | — | never gated | same | No separate query-only Deployment, no gateway changes. **Redis stays optional.** 1. With Redis, standbys serve reads as above. 2. Without Redis, a standby rejects reads too, and lookups reach the leader via client retry — degraded mode, documented as "routing for pre-failover sandboxes is lost on failover". Startup validation keeps only the `--query-only` flag conflict and the client-go timing check. **Active rebuild on leadership acquisition.** 1. On `OnStartedLeading`, the new leader walks its node registry 1. the EndpointSlice informer already runs on every replica from process start, so the list is warm 2. then pulls each node's admin `/nodes` snapshot, which the agentenv server already exposes with the same fields as a heartbeat. > As the responses go through the exact same ingest path as `Heartbeat` (update observations, `ReconcileNode` to refresh bindings), so observations and bindings rebuild in one RTT per node instead of one reporter backoff. 2. The fresh-only scheduling rule stays as a guard until the pull completes. The informer itself keeps running on standbys — it is read-only watch state, not an observation, so this does not reopen #191. ### One confirmation: in the no-Redis degraded mode, is "standby rejects reads with `Unavailable`, client retries land on the leader" the behavior you have in mind for lookups? @yingdi-shan Thanks
bsakaguchi commented 2026-10-29 14:19:12 +00:00
Author
Owner
Copy link

@yingdi-shan commented on GitHub (Sep 29, 2026):

Thanks! Your design looks good to me!

<!-- gh-comment-id:5881913595 --> @yingdi-shan commented on GitHub (Sep 29, 2026): Thanks! Your design looks good to me!
bsakaguchi commented 2026-10-29 14:19:12 +00:00
Author
Owner
Copy link

@NickNYU commented on GitHub (Oct 5, 2026):

Hi @yingdi-shan,

The consolidated design after the discussion above — this supersedes my earlier note. Implemented in #341.

  1. Single Deployment, standbys double as readers. N replicas compete for one coordination.k8s.io/Lease via client-go leaderelection. The leader gate is a read/write matrix: writes (Schedule, RecordAssignment, Heartbeat, sandbox events, P2P writes) return Unavailable on any non-leader; reads (LookupNode / GetNode) are served by standbys from the shared Redis bindings. The RPC table classifies every method explicitly and panics on unclassified ones, so proto upgrades fail loudly instead of silently gating.
  2. Redis stays optional. With Redis, standbys serve reads as above. Without Redis, standbys reject reads too and lookups reach the leader via client retry — degraded mode, documented as "routing for pre-failover sandboxes is lost on failover". Startup validation keeps only the --query-only conflict and the client-go timing check.
  3. Refresh on leadership acquisition, not waiting. On OnStartedLeading the new leader walks its warm node registry (the EndpointSlice informer runs on every replica from process start) and pulls each node's admin /nodes + /sandboxes (HTTP, x-api-key from SCHEDULER_NODE_ADMIN_API_KEY, mounted from the shared agentenv-auth Secret). Responses go through the exact same ingest path as Heartbeat, so observations and bindings rebuild in one RTT per node instead of one reporter backoff. Fan-out is bounded by leader_election.snapshot_pull_concurrency (default 4).
  4. Recovery window. Bounded by one report_ttl: Schedule only considers nodes observed at or after acquisition — zero fresh observations returns Unavailable, never a fallback to unobserved nodes (the #191 fix: no snapshot means "unknown", not "unlimited"). LookupNode/GetNode return Unavailable (not NotFound) for missing entries while the window is open.
  5. Fencing and probes. Liveness reports the overall health, unchanged; readiness reports scheduler.v1.Scheduler/leader, SERVING only while holding the lease, so Service endpoints contain only the leader and gateway needs no changes. Losing the lease flips readiness, then exits the process — a partitioned ex-leader stops serving within renew_deadline.
  6. Binding TTL floor. Under election the effective TTL is max(binding_ttl, lease_duration + 90s) so bindings outlive the failover budget; authoritative cleanup stays with ReconcileNode, so raising it is harmless.
  7. Compatibility. leader_election is off by default; disabled behavior is identical to today for every deployment shape (unit-tested). No protobuf, gateway, or node-side changes. Upgrade: binary first, then manifests (RBAC, probe, replicas, config). Rollback: disable and scale back to 1.

Tests: unit + fake-clientset suites cover the config surface, gate matrix, election lifecycle (exactly one leader, failover, fencing with callback ordering), recovery-window semantics, and the admin pull path. End-to-end scenarios (including the #191 resource-limit reproduction on 3 replicas) are spec'd in services/scheduler/kind for CI.

On #191: this PR intentionally does not auto-close it. The failure mode is fixed when election is enabled; with it off (the default), behavior is unchanged. Whether to close #191 or keep it for the doc note is your call.

Thanks!

<!-- gh-comment-id:5988686052 --> @NickNYU commented on GitHub (Oct 5, 2026): Hi @yingdi-shan, The consolidated design after the discussion above — this supersedes my earlier note. Implemented in #341. 1. **Single Deployment, standbys double as readers.** N replicas compete for one `coordination.k8s.io/Lease` via client-go leaderelection. The leader gate is a read/write matrix: writes (Schedule, RecordAssignment, Heartbeat, sandbox events, P2P writes) return `Unavailable` on any non-leader; reads (LookupNode / GetNode) are served by standbys from the shared Redis bindings. The RPC table classifies every method explicitly and panics on unclassified ones, so proto upgrades fail loudly instead of silently gating. 2. **Redis stays optional.** With Redis, standbys serve reads as above. Without Redis, standbys reject reads too and lookups reach the leader via client retry — degraded mode, documented as "routing for pre-failover sandboxes is lost on failover". Startup validation keeps only the `--query-only` conflict and the client-go timing check. 3. **Refresh on leadership acquisition, not waiting.** On `OnStartedLeading` the new leader walks its warm node registry (the EndpointSlice informer runs on every replica from process start) and pulls each node's admin `/nodes` + `/sandboxes` (HTTP, `x-api-key` from `SCHEDULER_NODE_ADMIN_API_KEY`, mounted from the shared `agentenv-auth` Secret). Responses go through the exact same ingest path as `Heartbeat`, so observations and bindings rebuild in one RTT per node instead of one reporter backoff. Fan-out is bounded by `leader_election.snapshot_pull_concurrency` (default 4). 4. **Recovery window.** Bounded by one `report_ttl`: Schedule only considers nodes observed at or after acquisition — zero fresh observations returns `Unavailable`, never a fallback to unobserved nodes (the #191 fix: no snapshot means "unknown", not "unlimited"). LookupNode/GetNode return `Unavailable` (not `NotFound`) for missing entries while the window is open. 5. **Fencing and probes.** Liveness reports the overall health, unchanged; readiness reports `scheduler.v1.Scheduler/leader`, SERVING only while holding the lease, so Service endpoints contain only the leader and gateway needs no changes. Losing the lease flips readiness, then exits the process — a partitioned ex-leader stops serving within `renew_deadline`. 6. **Binding TTL floor.** Under election the effective TTL is `max(binding_ttl, lease_duration + 90s)` so bindings outlive the failover budget; authoritative cleanup stays with `ReconcileNode`, so raising it is harmless. 7. **Compatibility.** `leader_election` is off by default; disabled behavior is identical to today for every deployment shape (unit-tested). No protobuf, gateway, or node-side changes. Upgrade: binary first, then manifests (RBAC, probe, replicas, config). Rollback: disable and scale back to 1. Tests: unit + fake-clientset suites cover the config surface, gate matrix, election lifecycle (exactly one leader, failover, fencing with callback ordering), recovery-window semantics, and the admin pull path. End-to-end scenarios (including the #191 resource-limit reproduction on 3 replicas) are spec'd in `services/scheduler/kind` for CI. On #191: this PR intentionally does not auto-close it. The failure mode is fixed when election is enabled; with it off (the default), behavior is unchanged. Whether to close #191 or keep it for the doc note is your call. Thanks!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
coverage-data
gh-pages
v0.3.0-rc.1
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
documentation

   
enhancement

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

No labels
bug
documentation
enhancement
enhancement
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/AgentENV#71
No description provided.