[GH-ISSUE #447] Scheduler gRPC health keeps passing while node discovery is dead — silent multi-day wedge (production incident 2026-10-28) #109

New issue

Open

opened 2026-10-29 14:19:41 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:19:41 +00:00

Owner

Originally created by @mingfang on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/447

Summary

In production (v0.2.4) the scheduler pod sat Running 1/1 with zero restarts for 39 hours while being functionally dead: every template/build operation through it failed. Kubernetes never restarted it because its liveness and readiness probes (exec /grpc_health_probe -addr=127.0.0.1:9090) kept passing the whole time.

Evidence

The pod's last log lines (39h before detection) are the EndpointSlice watch dying during a kube-apiserver flap:
- failed to list *v1.EndpointSlice: ... dial tcp 172.27.0.1:443: connect: connection refused
- watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost")
After that: no log lines at all for 39 hours (client-go's reflector never recovered and never logged again), no restarts, grpc_health_probe SERVING throughout.
Client-visible symptom: template alias updates refused / "gateway stuck" (sandbox data traffic bypasses the scheduler, so VMs kept working — which is why it stayed unnoticed).

Root cause of the detection gap

The gRPC health service only proves the process is alive, not that node discovery works. A dead EndpointSlice watch is invisible to grpc_health_probe.

Proposed fix

Make the scheduler's gRPC health reflect registry freshness, so the existing k8s probes auto-heal (restart) within ~1 minute:

Track the last successful watch event / last recorded node heartbeat (s.nodes.ListObserved already knows heartbeat freshness — services/scheduler/internal/service.go:263 refreshObservedNodesMetrics).
Register a custom health service (or flip the default one) to report NOT_SERVING when no node heartbeat has been recorded for N minutes while the deployment expects nodes.
Optional/extra: export a agentenv_scheduler_watch_last_event_age_seconds metric on :9101 so the dead-watch state is observable from the outside (today only agentenv_scheduler_observed_nodes{status=...} exists, which is heartbeat-based and can stay 1 on a stale cache).

Relevant code: services/scheduler/internal/kubernetes_discovery.go, services/scheduler/internal/service.go, services/scheduler/internal/metrics.go, services/scheduler/cmd/main.go:93.

Mitigation deployed meanwhile

An external watchdog CronJob polls agentenv_scheduler_observed_nodes{status="ready"} every 5 min, alerts, and restarts the pod once the zero persists past the startup window (heal latency ≤ ~10 min). The in-process health fix would make all of that redundant and cut the window to the probe period.

Originally created by @mingfang on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/AgentENV/issues/447 ## Summary In production (v0.2.4) the scheduler pod sat `Running 1/1` with **zero restarts for 39 hours** while being functionally dead: every template/build operation through it failed. Kubernetes never restarted it because its liveness and readiness probes (`exec /grpc_health_probe -addr=127.0.0.1:9090`) kept passing the whole time. ## Evidence - The pod's last log lines (39h before detection) are the EndpointSlice watch dying during a kube-apiserver flap: - `failed to list *v1.EndpointSlice: ... dial tcp 172.27.0.1:443: connect: connection refused` - `watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost")` - After that: **no log lines at all** for 39 hours (client-go's reflector never recovered and never logged again), no restarts, `grpc_health_probe` SERVING throughout. - Client-visible symptom: template alias updates refused / "gateway stuck" (sandbox data traffic bypasses the scheduler, so VMs kept working — which is why it stayed unnoticed). ## Root cause of the detection gap The gRPC health service only proves the **process** is alive, not that **node discovery** works. A dead EndpointSlice watch is invisible to `grpc_health_probe`. ## Proposed fix Make the scheduler's gRPC health reflect registry freshness, so the existing k8s probes auto-heal (restart) within ~1 minute: - Track the last successful watch event / last recorded node heartbeat (`s.nodes.ListObserved` already knows heartbeat freshness — `services/scheduler/internal/service.go:263` `refreshObservedNodesMetrics`). - Register a custom health service (or flip the default one) to report `NOT_SERVING` when no node heartbeat has been recorded for N minutes while the deployment expects nodes. - Optional/extra: export a `agentenv_scheduler_watch_last_event_age_seconds` metric on :9101 so the dead-watch state is observable from the outside (today only `agentenv_scheduler_observed_nodes{status=...}` exists, which is heartbeat-based and can stay 1 on a stale cache). Relevant code: `services/scheduler/internal/kubernetes_discovery.go`, `services/scheduler/internal/service.go`, `services/scheduler/internal/metrics.go`, `services/scheduler/cmd/main.go:93`. ## Mitigation deployed meanwhile An external watchdog CronJob polls `agentenv_scheduler_observed_nodes{status="ready"}` every 5 min, alerts, and restarts the pod once the zero persists past the startup window (heal latency ≤ ~10 min). The in-process health fix would make all of that redundant and cut the window to the probe period.

bsakaguchi referenced this issue

2026-10-29 14:20:46 +00:00

[PR #109] [MERGED] perf(net): disable Nagle on the API server, proxy and envd clients #184

bsakaguchi referenced this issue

2026-10-29 14:20:49 +00:00

[PR #114] fix(proxy): probe upstream liveness with TCP keepalive #190