[PR #232] [CLOSED] feat: add single-active scheduler HA #275

New issue

Closed

opened 2026-10-29 14:21:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:21:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/232
Author: @ethe
Created: 8/30/2026
Status: ❌ Closed

Base: main ← Head: feat/scheduler-ha-upstream

📝 Commits (1)

370476f feat: add scheduler leader election

📊 Changes

11 files changed (+463 additions, -44 deletions)

View changed files

📝 CLAUDE.md (+1 -1)
📝 docs/src/configuration/env-vars.md (+1 -0)
📝 docs/src/internals/architecture.md (+1 -1)
📝 services/README.md (+3 -1)
➕ services/scheduler/cmd/leadership.go (+81 -0)
➕ services/scheduler/cmd/leadership_test.go (+43 -0)
📝 services/scheduler/cmd/main.go (+87 -22)
📝 services/scheduler/internal/service.go (+26 -7)
📝 services/scheduler/internal/service_test.go (+25 -0)
📝 services/shared/config/config.go (+91 -12)
📝 services/shared/config/config_test.go (+104 -0)

📄 Description

Summary

add Kubernetes Lease leader election for full Scheduler replicas
require Redis-backed sandbox bindings and Kubernetes discovery when leader election is enabled
keep followers out of the serving gRPC endpoints and reject non-health RPCs
require fresh READY heartbeats before a newly elected Scheduler places new work
preserve existing-sandbox routing through independent query-only replicas

Why

Running multiple full Scheduler replicas behind one Service currently creates independent in-memory node heartbeat/resource views, round-robin cursors, and artifact indexes. Even with Redis-backed bindings, active-active scheduling can use stale or incomplete capacity state. This change makes the full Scheduler single-active while retaining query-only data-plane availability.

Validation

make -C services test
go test ./... from services/
go vet ./... from services/

Deployment contract

Leader-election deployments need a Service whose readiness probe uses gRPC health, Lease RBAC for the full Scheduler service account, a unique Pod identity through SCHEDULER_LEADER_ELECTION_IDENTITY, Redis-backed bindings, and separate query-only replicas for uninterrupted LookupNode. Observed node/resource state and the P2P artifact index remain intentionally ephemeral and are rebuilt after failover.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/232 **Author:** [@ethe](https://github.com/ethe) **Created:** 8/30/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/scheduler-ha-upstream` --- ### 📝 Commits (1) - [`370476f`](https://github.com/kvcache-ai/AgentENV/commit/370476f175f5a12a455646c933a19fa1a9d7232e) feat: add scheduler leader election ### 📊 Changes **11 files changed** (+463 additions, -44 deletions) <details> <summary>View changed files</summary> 📝 `CLAUDE.md` (+1 -1) 📝 `docs/src/configuration/env-vars.md` (+1 -0) 📝 `docs/src/internals/architecture.md` (+1 -1) 📝 `services/README.md` (+3 -1) ➕ `services/scheduler/cmd/leadership.go` (+81 -0) ➕ `services/scheduler/cmd/leadership_test.go` (+43 -0) 📝 `services/scheduler/cmd/main.go` (+87 -22) 📝 `services/scheduler/internal/service.go` (+26 -7) 📝 `services/scheduler/internal/service_test.go` (+25 -0) 📝 `services/shared/config/config.go` (+91 -12) 📝 `services/shared/config/config_test.go` (+104 -0) </details> ### 📄 Description ## Summary - add Kubernetes Lease leader election for full Scheduler replicas - require Redis-backed sandbox bindings and Kubernetes discovery when leader election is enabled - keep followers out of the serving gRPC endpoints and reject non-health RPCs - require fresh READY heartbeats before a newly elected Scheduler places new work - preserve existing-sandbox routing through independent query-only replicas ## Why Running multiple full Scheduler replicas behind one Service currently creates independent in-memory node heartbeat/resource views, round-robin cursors, and artifact indexes. Even with Redis-backed bindings, active-active scheduling can use stale or incomplete capacity state. This change makes the full Scheduler single-active while retaining query-only data-plane availability. ## Validation - `make -C services test` - `go test ./...` from `services/` - `go vet ./...` from `services/` ## Deployment contract Leader-election deployments need a Service whose readiness probe uses gRPC health, Lease RBAC for the full Scheduler service account, a unique Pod identity through `SCHEDULER_LEADER_ELECTION_IDENTITY`, Redis-backed bindings, and separate query-only replicas for uninterrupted `LookupNode`. Observed node/resource state and the P2P artifact index remain intentionally ephemeral and are rebuilt after failover. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 14:21:43 +00:00