[PR #341] feat(scheduler): add HA via Kubernetes Lease leader election #364

New issue

Open

opened 2026-10-29 14:22:39 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:22:39 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/AgentENV/pull/341
Author: @NickNYU
Created: 10/5/2026
Status: 🔄 Open

Base: main ← Head: feat/scheduler-ha-leader-election

📝 Commits (10+)

d83ec8b feat(scheduler): add leader-election config surface and HA design
fa7ebea feat(api,scheduler): sync-node-snapshots rebuild and admin heartbeat parity
99ffe29 feat(scheduler): leadership domain with Lease election and recovery window
a9df2ee fix(scheduler): takeover correctness in main wiring, discovery, and binding TTL
9f89d54 test(scheduler): Kind HA failover e2e harness with four scenarios
6274c90 docs(scheduler): HA deployment overlay and failover-gap notes
0f9aab9 fix(scheduler): hand discovery a ready channel only when startup waits
4b4a0d8 fix(scheduler): atomic freshness check for pulled node snapshots
112008f test(scheduler-e2e): probe T4's live demoted leader via API isolation
1dff931 test(scheduler-e2e): compare restartCount against a per-test baseline

📊 Changes

37 files changed (+4300 additions, -88 deletions)

View changed files

➕ .github/workflows/scheduler-ha-e2e.yml (+96 -0)
📝 deploy/k8s/base/role.yaml (+10 -0)
➕ deploy/k8s/overlays/ha/README.md (+65 -0)
➕ deploy/k8s/overlays/ha/config/scheduler.json (+25 -0)
➕ deploy/k8s/overlays/ha/kustomization.yaml (+65 -0)
📝 services/README.md (+29 -1)
📝 services/go.mod (+5 -1)
📝 services/go.sum (+12 -6)
➕ services/scheduler/cmd/leadership_loss.go (+21 -0)
➕ services/scheduler/cmd/leadership_loss_test.go (+46 -0)
📝 services/scheduler/cmd/main.go (+123 -30)
➕ services/scheduler/e2e/Dockerfile.stubnode (+12 -0)
➕ services/scheduler/e2e/harness_test.go (+542 -0)
➕ services/scheduler/e2e/k8s/redis.yaml (+30 -0)
➕ services/scheduler/e2e/k8s/scheduler.yaml (+140 -0)
➕ services/scheduler/e2e/k8s/stubnodes.yaml (+50 -0)
➕ services/scheduler/e2e/stubnode/main.go (+309 -0)
➕ services/scheduler/internal/discovery_ready_test.go (+89 -0)
📝 services/scheduler/internal/kubernetes_discovery.go (+35 -9)
➕ services/scheduler/internal/leadership.go (+438 -0)

...and 17 more files

📄 Description

What

Add optional high availability to the scheduler via Kubernetes Lease leader election. N scheduler replicas compete for a single coordination.k8s.io/Lease; exactly one leader schedules and processes node reports, standbys stay liveness-healthy and serve LookupNode/GetNode from the shared Redis bindings. On failover, the new leader rebuilds state by pulling every node's admin snapshot (sync-node-snapshots) instead of waiting out a reporter backoff, and applies recovery-window semantics until the rebuild completes. Off by default; with it off, behavior is identical to today.

Why

The scheduler is a single point of failure today (replicas: 1): scheduling, assignment writes, node inspection, and P2P APIs all go down with it, and query-only replicas only preserve data-plane lookups. Running multiple read/write replicas directly is unsafe — heartbeat state is replica-local, which silently breaks node_resource_limit (#191, measured 3.0x overshoot on 8 nodes).

Closes #259

Relates to #191 — the #191 failure mode is fixed when leader election is enabled: exactly one scheduler processes heartbeats and schedules, and the recovery window never treats missing observations as free capacity. With election disabled (the default), behavior is unchanged, so this PR intentionally does not auto-close #191; whether to close it (or keep it for the doc note, direction 1 in the issue) is a maintainer call. The full #191 reproduction scenario is spec'd for Kind/CI (services/scheduler/kind), not run locally.

Scope and non-goals

In scope:

scheduler.leader_election config surface + startup validation (fail fast; mutually exclusive with --query-only; client-go timing rules). Redis stays optional; without it the documented degraded mode applies.
Single exported Leadership contract + factory + null object: gate interceptor (read/write matrix), leader readiness health service scheduler.v1.Scheduler/leader, client-go Lease elector with fencing-by-exit.
Recovery window: fresh-observations-only scheduling (#191 fix semantics), Unavailable-not-NotFound lookups during rebuild.
Sync-node-snapshots: post-acquisition pull of every node's admin /nodes through the shared heartbeat ingest path (observations only, no binding reconcile — see Design); concurrency via leader_election.snapshot_pull_concurrency.
Binding TTL floor under election (max(binding_ttl, lease_duration + 90s)) so bindings outlive the failover budget.
Manifests: new deploy/k8s/overlays/ha (3 replicas, readiness probe on the leader health service, election config, secret-mounted admin key); base manifests untouched except additive leases RBAC (inert when off).

Non-goals:

No protobuf changes; no gateway or node-side code changes; no heartbeat protocol changes.
No active-active or sharded scheduling; standbys collect no observations (possible future work, deliberately out of scope).
No separate query-only Deployment changes; existing query-only mode is untouched and mutually exclusive with election.
Kind integration cases are spec'd (build-tagged) but not wired into CI yet.

Design and behavior changes

flowchart LR
    subgraph sched["Scheduler Deployment · N replicas, leader-elected"]
        L["Leader<br/>(ready, serving)"]
        S1["Standby<br/>(reads from Redis)"]
        S2["Standby<br/>(reads from Redis)"]
    end
    GW["Gateway"]
    RD[("Redis<br/>bindings")]
    NODE["agentenv server nodes"]
    LEASE[["K8S Lease"]]

    NODE -- "④ heartbeat" --> L
    GW -- "① Schedule / ③ RecordAssignment (write)" --> L
    GW -- "② LookupNode (read)" --> sched
    L <-- "binding write" --> RD
    S1 <-. "binding read" .-> RD
    S2 <-. "binding read" .-> RD
    L <-. "acquire / renew" .-> LEASE
    S1 <-. "elect" .-> LEASE
    S2 <-. "elect" .-> LEASE
    L <-. "⑤ pull /nodes (refresh)" .-> NODE

Key behaviors:

Read/write gate: health checks never gated; leader serves everything; standbys with Redis serve LookupNode/GetNode, reject the rest with Unavailable("not the leader"); without Redis reads retry onto the leader. The RPC table classifies every method explicitly and panics on unclassified ones, so proto upgrades fail loudly.
Readiness split: liveness reports the overall health (SERVING from start, unchanged); readiness reports scheduler.v1.Scheduler/leader (SERVING only while holding the lease), so Service endpoints contain only the leader and gateway needs no changes.
Fencing: OnStoppedLeading flips readiness, then cancels the process root context — the ex-leader exits through the same graceful shutdown path as SIGTERM and restarts as a standby. A partitioned ex-leader stops serving within renew_deadline.
Recovery window (bounded by one report_ttl): heartbeats accepted immediately; Schedule considers only nodes observed at or after acquisition (via NodeRegistry.LastReportAt), zero fresh observations returns Unavailable — never a fallback to unobserved nodes; LookupNode/GetNode return Unavailable (not NotFound) for missing entries while the window is open.
Sync-node-snapshots: on acquisition the leader pulls every node's admin /nodes snapshot (HTTP, x-api-key from SCHEDULER_NODE_ADMIN_API_KEY mounted from the shared agentenv-auth Secret) and feeds it through the exact same ingest path as Heartbeat, so observations rebuild in seconds rather than a reporter backoff (~60s worst case). Bindings are deliberately not reconciled from a pull: a pull cannot see paused sandboxes or in-flight template builds, so reconciling would delete live bindings on failover (#341 review). Binding refresh stays with heartbeats, and the TTL floor covers the gap.
Object model: one exported interface (Leadership) + one factory + a null object for the disabled path; implementations (manager, snapshot, elector) are package-private. State is an immutable snapshot swapped atomically by the manager; the Service sees leadership only through a 2-method read-only seam.

Compatibility and operations

Public API or generated protocol: none. No protobuf changes; gateway and runtime nodes need no changes.
Configuration or defaults: leader_election is off by default; disabled behavior is identical to today for every deployment shape (validated in unit tests). New optional keys: leader_election.* (durations as strings, e.g. "15s"), scheduler.node_admin_api_key (credential — prefer SCHEDULER_NODE_ADMIN_API_KEY env/Secret).
Snapshot manifest, artifact layout, or storage format: N/A (no changes).
Upgrade and rollback: upgrade the scheduler binary first (it serves today's behavior while election is off and registers the leader health service), then apply manifests (RBAC, readiness probe, replicas, config). Rollback: disable the config and scale back to 1 replica.
Host requirements, permissions, ports, or dependencies: Kubernetes only — scheduler Role needs coordination.k8s.io/leases get/create/update (added to deploy/k8s/base/role.yaml). Redis optional but recommended; without it failover loses routing for pre-failover sandboxes (documented). No new Go module dependencies beyond client-go's existing testing packages.

Validation

make fmt
make clippy
make test-unit
Relevant Rust integration tests
make -C services test (required when services/ changes)
Generated clients/server regenerated with the documented make target
Documentation updated
Benchmarks or performance comparison completed

Commands and results:

# Note: system Go is 1.20.6 and cannot parse go.mod (go 1.25.0);
# all Go commands below used a go1.25.1 toolchain.

go1.25.1 test ./...            # all packages pass (gateway, scheduler, config, logging)
go1.25.1 vet ./...             # clean
gofmt -l scheduler/ shared/    # clean
kubectl kustomize deploy/k8s/overlays/ha
  # renders: replicas=3, readiness -service=scheduler.v1.Scheduler/leader,
  # leader_election config, leases RBAC, secret-mounted admin key

New tests cover: config validation (A), gate matrix (B), election lifecycle on a fake clientset (C1 exactly-one-leader, C2 failover, C3 fencing on renew failure + callback ordering), recovery window semantics (D: fresh-only, zero-fresh Unavailable, ingest path sharing, pull failure, lookup window semantics), binding TTL floor, and admin snapshot fetcher mapping/auth/ingest compatibility.

Skipped checks and reasons:

Rust checks (make fmt, clippy, test-unit, integration): no Rust changes in this PR.
Generated code: no protobuf changes.
Kind integration (groups C/E/F end-to-end): spec'd under services/scheduler/kind with a build tag; the repo has no Kind harness and the dev machine is macOS. Left for CI; the fake-clientset suite covers election semantics offline.
Benchmarks: no performance-sensitive changes; the pull path is a bounded one-off fan-out per failover.

Risks and reviewer notes

Biggest surface: services/scheduler/internal/leadership.go (facade + gate + elector) and service.go (recovery window). Suggested read order: 92b2089 (config) → f92133c (leadership domain) → f1a67c3 (sync-node-snapshots) → aac26dd (main wiring + TTL floor) → 8a36b6c (Kind specs) → 2ac0f4c (manifests/docs).
With election disabled, the gate interceptor is a pass-through and every leadership method is a no-op; the "identical to today" claim is covered by unit tests.
The recovery window intentionally prefers Unavailable over optimistic scheduling right after failover — a deliberate availability-vs-correctness trade, matching the issue's "missing observations are not unlimited capacity".
ListNodes is currently gated on standbys (reads the informer-backed registry, semantically standby-serveable); kept gated for now, flagged for a follow-up decision.

Checklist

The PR contains one coherent change and no unrelated formatting or refactoring.
New behavior is covered by tests, or I explained why testing is impractical.
Logs and examples contain no credentials, tokens, or private registry information.
I did not manually edit generated code without updating its source and regenerating it.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/AgentENV/pull/341 **Author:** [@NickNYU](https://github.com/NickNYU) **Created:** 10/5/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/scheduler-ha-leader-election` --- ### 📝 Commits (10+) - [`d83ec8b`](https://github.com/kvcache-ai/AgentENV/commit/d83ec8baa2a9093c6a5bf140e85159d8ccb2fe60) feat(scheduler): add leader-election config surface and HA design - [`fa7ebea`](https://github.com/kvcache-ai/AgentENV/commit/fa7ebeaa5ea964a60f111aff7749c079bc0c6524) feat(api,scheduler): sync-node-snapshots rebuild and admin heartbeat parity - [`99ffe29`](https://github.com/kvcache-ai/AgentENV/commit/99ffe29c5e7a3e77fe9c7a7563e2d63701f4ad99) feat(scheduler): leadership domain with Lease election and recovery window - [`a9df2ee`](https://github.com/kvcache-ai/AgentENV/commit/a9df2eec22287c7276c56d69d183acf74ff46b4d) fix(scheduler): takeover correctness in main wiring, discovery, and binding TTL - [`9f89d54`](https://github.com/kvcache-ai/AgentENV/commit/9f89d54712f347db548f175c411d821cb5f4c09c) test(scheduler): Kind HA failover e2e harness with four scenarios - [`6274c90`](https://github.com/kvcache-ai/AgentENV/commit/6274c90636b91c291266f80e18eea45bc383b408) docs(scheduler): HA deployment overlay and failover-gap notes - [`0f9aab9`](https://github.com/kvcache-ai/AgentENV/commit/0f9aab9b341668d6854c92796a7130f889e48248) fix(scheduler): hand discovery a ready channel only when startup waits - [`4b4a0d8`](https://github.com/kvcache-ai/AgentENV/commit/4b4a0d82ed13755677dac7a570b56017dece98ef) fix(scheduler): atomic freshness check for pulled node snapshots - [`112008f`](https://github.com/kvcache-ai/AgentENV/commit/112008f38b9eb4e5a06ae907d70a7c528250c023) test(scheduler-e2e): probe T4's live demoted leader via API isolation - [`1dff931`](https://github.com/kvcache-ai/AgentENV/commit/1dff931c91830e619c7830ac74f49f9e3aee3c30) test(scheduler-e2e): compare restartCount against a per-test baseline ### 📊 Changes **37 files changed** (+4300 additions, -88 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/scheduler-ha-e2e.yml` (+96 -0) 📝 `deploy/k8s/base/role.yaml` (+10 -0) ➕ `deploy/k8s/overlays/ha/README.md` (+65 -0) ➕ `deploy/k8s/overlays/ha/config/scheduler.json` (+25 -0) ➕ `deploy/k8s/overlays/ha/kustomization.yaml` (+65 -0) 📝 `services/README.md` (+29 -1) 📝 `services/go.mod` (+5 -1) 📝 `services/go.sum` (+12 -6) ➕ `services/scheduler/cmd/leadership_loss.go` (+21 -0) ➕ `services/scheduler/cmd/leadership_loss_test.go` (+46 -0) 📝 `services/scheduler/cmd/main.go` (+123 -30) ➕ `services/scheduler/e2e/Dockerfile.stubnode` (+12 -0) ➕ `services/scheduler/e2e/harness_test.go` (+542 -0) ➕ `services/scheduler/e2e/k8s/redis.yaml` (+30 -0) ➕ `services/scheduler/e2e/k8s/scheduler.yaml` (+140 -0) ➕ `services/scheduler/e2e/k8s/stubnodes.yaml` (+50 -0) ➕ `services/scheduler/e2e/stubnode/main.go` (+309 -0) ➕ `services/scheduler/internal/discovery_ready_test.go` (+89 -0) 📝 `services/scheduler/internal/kubernetes_discovery.go` (+35 -9) ➕ `services/scheduler/internal/leadership.go` (+438 -0) _...and 17 more files_ </details> ### 📄 Description ## What Add optional high availability to the scheduler via Kubernetes Lease leader election. N scheduler replicas compete for a single `coordination.k8s.io/Lease`; exactly one leader schedules and processes node reports, standbys stay liveness-healthy and serve `LookupNode`/`GetNode` from the shared Redis bindings. On failover, the new leader rebuilds state by pulling every node's admin snapshot (sync-node-snapshots) instead of waiting out a reporter backoff, and applies recovery-window semantics until the rebuild completes. Off by default; with it off, behavior is identical to today. ## Why The scheduler is a single point of failure today (`replicas: 1`): scheduling, assignment writes, node inspection, and P2P APIs all go down with it, and query-only replicas only preserve data-plane lookups. Running multiple read/write replicas directly is unsafe — heartbeat state is replica-local, which silently breaks `node_resource_limit` (#191, measured 3.0x overshoot on 8 nodes). ## Related issue Closes #259 Relates to #191 — the #191 failure mode is fixed **when leader election is enabled**: exactly one scheduler processes heartbeats and schedules, and the recovery window never treats missing observations as free capacity. With election disabled (the default), behavior is unchanged, so this PR intentionally does not auto-close #191; whether to close it (or keep it for the doc note, direction 1 in the issue) is a maintainer call. The full #191 reproduction scenario is spec'd for Kind/CI (`services/scheduler/kind`), not run locally. ## Scope and non-goals In scope: - `scheduler.leader_election` config surface + startup validation (fail fast; mutually exclusive with `--query-only`; client-go timing rules). Redis stays optional; without it the documented degraded mode applies. - Single exported `Leadership` contract + factory + null object: gate interceptor (read/write matrix), leader readiness health service `scheduler.v1.Scheduler/leader`, client-go Lease elector with fencing-by-exit. - Recovery window: fresh-observations-only scheduling (#191 fix semantics), Unavailable-not-NotFound lookups during rebuild. - Sync-node-snapshots: post-acquisition pull of every node's admin `/nodes` through the shared heartbeat ingest path (observations only, no binding reconcile — see Design); concurrency via `leader_election.snapshot_pull_concurrency`. - Binding TTL floor under election (`max(binding_ttl, lease_duration + 90s)`) so bindings outlive the failover budget. - Manifests: new `deploy/k8s/overlays/ha` (3 replicas, readiness probe on the leader health service, election config, secret-mounted admin key); base manifests untouched except additive `leases` RBAC (inert when off). Non-goals: - No protobuf changes; no gateway or node-side code changes; no heartbeat protocol changes. - No active-active or sharded scheduling; standbys collect no observations (possible future work, deliberately out of scope). - No separate query-only Deployment changes; existing query-only mode is untouched and mutually exclusive with election. - Kind integration cases are spec'd (build-tagged) but not wired into CI yet. ## Design and behavior changes ```mermaid flowchart LR subgraph sched["Scheduler Deployment · N replicas, leader-elected"] L["Leader (ready, serving)"] S1["Standby (reads from Redis)"] S2["Standby (reads from Redis)"] end GW["Gateway"] RD[("Redis bindings")] NODE["agentenv server nodes"] LEASE[["K8S Lease"]] NODE -- "④ heartbeat" --> L GW -- "① Schedule / ③ RecordAssignment (write)" --> L GW -- "② LookupNode (read)" --> sched L <-- "binding write" --> RD S1 <-. "binding read" .-> RD S2 <-. "binding read" .-> RD L <-. "acquire / renew" .-> LEASE S1 <-. "elect" .-> LEASE S2 <-. "elect" .-> LEASE L <-. "⑤ pull /nodes (refresh)" .-> NODE ``` Key behaviors: - **Read/write gate**: health checks never gated; leader serves everything; standbys with Redis serve `LookupNode`/`GetNode`, reject the rest with `Unavailable("not the leader")`; without Redis reads retry onto the leader. The RPC table classifies every method explicitly and panics on unclassified ones, so proto upgrades fail loudly. - **Readiness split**: liveness reports the overall health (SERVING from start, unchanged); readiness reports `scheduler.v1.Scheduler/leader` (SERVING only while holding the lease), so Service endpoints contain only the leader and gateway needs no changes. - **Fencing**: `OnStoppedLeading` flips readiness, then cancels the process root context — the ex-leader exits through the same graceful shutdown path as SIGTERM and restarts as a standby. A partitioned ex-leader stops serving within `renew_deadline`. - **Recovery window** (bounded by one `report_ttl`): heartbeats accepted immediately; `Schedule` considers only nodes observed at or after acquisition (via `NodeRegistry.LastReportAt`), zero fresh observations returns `Unavailable` — never a fallback to unobserved nodes; `LookupNode`/`GetNode` return `Unavailable` (not `NotFound`) for missing entries while the window is open. - **Sync-node-snapshots**: on acquisition the leader pulls every node's admin `/nodes` snapshot (HTTP, `x-api-key` from `SCHEDULER_NODE_ADMIN_API_KEY` mounted from the shared `agentenv-auth` Secret) and feeds it through the exact same ingest path as `Heartbeat`, so observations rebuild in seconds rather than a reporter backoff (~60s worst case). Bindings are deliberately **not** reconciled from a pull: a pull cannot see paused sandboxes or in-flight template builds, so reconciling would delete live bindings on failover (#341 review). Binding refresh stays with heartbeats, and the TTL floor covers the gap. - **Object model**: one exported interface (`Leadership`) + one factory + a null object for the disabled path; implementations (manager, snapshot, elector) are package-private. State is an immutable snapshot swapped atomically by the manager; the Service sees leadership only through a 2-method read-only seam. ## Compatibility and operations - Public API or generated protocol: none. No protobuf changes; gateway and runtime nodes need no changes. - Configuration or defaults: `leader_election` is off by default; disabled behavior is identical to today for every deployment shape (validated in unit tests). New optional keys: `leader_election.*` (durations as strings, e.g. `"15s"`), `scheduler.node_admin_api_key` (credential — prefer `SCHEDULER_NODE_ADMIN_API_KEY` env/Secret). - Snapshot manifest, artifact layout, or storage format: N/A (no changes). - Upgrade and rollback: upgrade the scheduler binary first (it serves today's behavior while election is off and registers the leader health service), then apply manifests (RBAC, readiness probe, replicas, config). Rollback: disable the config and scale back to 1 replica. - Host requirements, permissions, ports, or dependencies: Kubernetes only — scheduler Role needs `coordination.k8s.io/leases` get/create/update (added to `deploy/k8s/base/role.yaml`). Redis optional but recommended; without it failover loses routing for pre-failover sandboxes (documented). No new Go module dependencies beyond client-go's existing testing packages. ## Validation - [ ] `make fmt` - [ ] `make clippy` - [ ] `make test-unit` - [ ] Relevant Rust integration tests - [x] `make -C services test` (required when `services/` changes) - [ ] Generated clients/server regenerated with the documented `make` target - [x] Documentation updated - [ ] Benchmarks or performance comparison completed Commands and results: ```text # Note: system Go is 1.20.6 and cannot parse go.mod (go 1.25.0); # all Go commands below used a go1.25.1 toolchain. go1.25.1 test ./... # all packages pass (gateway, scheduler, config, logging) go1.25.1 vet ./... # clean gofmt -l scheduler/ shared/ # clean kubectl kustomize deploy/k8s/overlays/ha # renders: replicas=3, readiness -service=scheduler.v1.Scheduler/leader, # leader_election config, leases RBAC, secret-mounted admin key ``` New tests cover: config validation (A), gate matrix (B), election lifecycle on a fake clientset (C1 exactly-one-leader, C2 failover, C3 fencing on renew failure + callback ordering), recovery window semantics (D: fresh-only, zero-fresh Unavailable, ingest path sharing, pull failure, lookup window semantics), binding TTL floor, and admin snapshot fetcher mapping/auth/ingest compatibility. Skipped checks and reasons: - Rust checks (`make fmt`, `clippy`, `test-unit`, integration): no Rust changes in this PR. - Generated code: no protobuf changes. - Kind integration (groups C/E/F end-to-end): spec'd under `services/scheduler/kind` with a build tag; the repo has no Kind harness and the dev machine is macOS. Left for CI; the fake-clientset suite covers election semantics offline. - Benchmarks: no performance-sensitive changes; the pull path is a bounded one-off fan-out per failover. ## Risks and reviewer notes - Biggest surface: `services/scheduler/internal/leadership.go` (facade + gate + elector) and `service.go` (recovery window). Suggested read order: `92b2089` (config) → `f92133c` (leadership domain) → `f1a67c3` (sync-node-snapshots) → `aac26dd` (main wiring + TTL floor) → `8a36b6c` (Kind specs) → `2ac0f4c` (manifests/docs). - With election disabled, the gate interceptor is a pass-through and every leadership method is a no-op; the "identical to today" claim is covered by unit tests. - The recovery window intentionally prefers `Unavailable` over optimistic scheduling right after failover — a deliberate availability-vs-correctness trade, matching the issue's "missing observations are not unlimited capacity". - `ListNodes` is currently gated on standbys (reads the informer-backed registry, semantically standby-serveable); kept gated for now, flagged for a follow-up decision. ## Checklist - [x] The PR contains one coherent change and no unrelated formatting or refactoring. - [x] New behavior is covered by tests, or I explained why testing is impractical. - [x] Logs and examples contain no credentials, tokens, or private registry information. - [x] I did not manually edit generated code without updating its source and regenerating it. --- 🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

bsakaguchi added the

pull-request

label

2026-10-29 14:22:39 +00:00