[PR #61] [CLOSED] Re-create terminal pods that disappear, and stop serving dead ones #62

New issue

Closed

opened 2026-10-18 21:13:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/terminals/pull/61
Author: @joshhuyn
Created: 9/16/2026
Status: ❌ Closed

Base: main ← Head: recreate-resources

📝 Commits (1)

0b03591 Re-create terminal pods that disappear, and stop serving dead ones

📊 Changes

6 files changed (+963 additions, -208 deletions)

View changed files

📝 CHANGELOG.md (+13 -0)
📝 README.md (+11 -0)
📝 operator/handler.py (+395 -119)
📝 terminals/backends/kubernetes_operator.py (+105 -89)
📝 terminals/config.py (+3 -0)
➕ tests/test_operator_reconcile.py (+436 -0)

📄 Description

#60

The operator only built child resources from on.create, so a pod that was deleted, evicted, or lost with its node stayed gone while the Terminal still reported phase: Running — the orchestrator kept proxying to a Service with no endpoints until the session was abandoned. Child creation moved into a level-triggered _ensure_children, now shared by on.create, a kopf.timer reconcile loop (TERMINALS_RECONCILE_INTERVAL_SECONDS, default 15), on.resume so restarts re-adopt existing CRs, and the pod-DELETED watcher. Succeeded/Failed pods are deleted first so the next pass can replace them.
Pods removed by the idle timeout must stay down: idle_check now marks the CR Idle before deleting the pod, reconcile skips Idle, and _ensure_children re-reads the live phase through a guard right before touching the pod so a sweep running off a stale cache cannot undo a cull.
_ensure_secret reads the api-key back out of an existing Secret, so a re-created pod keeps the credential the orchestrator already handed out instead of coming up with a key nobody holds.
Status updates go through _patch_terminal_status, which sends application/merge-patch+json. The client otherwise defaults to application/json-patch+json and rejects a dict body, so pod state could silently fail to reach the resource.
A pod with a deletionTimestamp reports Running and ready for its whole termination grace period; it is no longer advertised as usable.
TERMINALS_KUBERNETES_NODE_SELECTOR and TERMINALS_KUBERNETES_TOLERATIONS are parsed once from configure() and cached, rather than on every _build_pod_manifest call. Both raise on malformed input, and the reconcile loop rebuilds manifests every sweep for every Terminal, so a single typo would otherwise mean no pod is ever created and the same traceback repeats forever. They come from the operator's own environment and cannot change while it runs, so a bad value now stops the operator at startup with an error naming the variable and the expected shape instead of degrading into a silent never-schedules state.
The kubernetes-operator backend judges readiness by the Ready condition rather than phase alone (_is_serving), and re-creates a Terminal that never becomes ready instead of failing every request from then on. The wait is bounded by the new TERMINALS_TERMINAL_READY_TIMEOUT (default 120) and returns early if the CR vanishes mid-wait. Duplicated connect/cache bookkeeping in ensure_terminal collapsed into _connect_info / _remember / _reprovision.
New tests/test_operator_reconcile.py covers the reconcile, pod-event and scheduling-config paths against stubbed kopf/kubernetes clients.

Description

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/terminals/pull/61 **Author:** [@joshhuyn](https://github.com/joshhuyn) **Created:** 9/16/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `recreate-resources` --- ### 📝 Commits (1) - [`0b03591`](https://github.com/open-webui/terminals/commit/0b03591a126355a613078686bff2080cc497b032) Re-create terminal pods that disappear, and stop serving dead ones ### 📊 Changes **6 files changed** (+963 additions, -208 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+13 -0) 📝 `README.md` (+11 -0) 📝 `operator/handler.py` (+395 -119) 📝 `terminals/backends/kubernetes_operator.py` (+105 -89) 📝 `terminals/config.py` (+3 -0) ➕ `tests/test_operator_reconcile.py` (+436 -0) </details> ### 📄 Description #60 - The operator only built child resources from `on.create`, so a pod that was deleted, evicted, or lost with its node stayed gone while the `Terminal` still reported `phase: Running` — the orchestrator kept proxying to a Service with no endpoints until the session was abandoned. Child creation moved into a level-triggered `_ensure_children`, now shared by `on.create`, a `kopf.timer` reconcile loop (`TERMINALS_RECONCILE_INTERVAL_SECONDS`, default 15), `on.resume` so restarts re-adopt existing CRs, and the pod-DELETED watcher. Succeeded/Failed pods are deleted first so the next pass can replace them. - Pods removed by the idle timeout must stay down: `idle_check` now marks the CR `Idle` *before* deleting the pod, reconcile skips `Idle`, and `_ensure_children` re-reads the live phase through a guard right before touching the pod so a sweep running off a stale cache cannot undo a cull. - `_ensure_secret` reads the api-key back out of an existing Secret, so a re-created pod keeps the credential the orchestrator already handed out instead of coming up with a key nobody holds. - Status updates go through `_patch_terminal_status`, which sends `application/merge-patch+json`. The client otherwise defaults to `application/json-patch+json` and rejects a dict body, so pod state could silently fail to reach the resource. - A pod with a `deletionTimestamp` reports Running and ready for its whole termination grace period; it is no longer advertised as usable. - `TERMINALS_KUBERNETES_NODE_SELECTOR` and `TERMINALS_KUBERNETES_TOLERATIONS` are parsed once from `configure()` and cached, rather than on every `_build_pod_manifest` call. Both raise on malformed input, and the reconcile loop rebuilds manifests every sweep for every Terminal, so a single typo would otherwise mean no pod is ever created and the same traceback repeats forever. They come from the operator's own environment and cannot change while it runs, so a bad value now stops the operator at startup with an error naming the variable and the expected shape instead of degrading into a silent never-schedules state. - The `kubernetes-operator` backend judges readiness by the `Ready` condition rather than `phase` alone (`_is_serving`), and re-creates a `Terminal` that never becomes ready instead of failing every request from then on. The wait is bounded by the new `TERMINALS_TERMINAL_READY_TIMEOUT` (default 120) and returns early if the CR vanishes mid-wait. Duplicated connect/cache bookkeeping in `ensure_terminal` collapsed into `_connect_info` / `_remember` / `_reprovision`. - New `tests/test_operator_reconcile.py` covers the reconcile, pod-event and scheduling-config paths against stubbed kopf/kubernetes clients. ## Description  ## Related Issues  --- ## Contributor License Agreement  - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](./CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:13:54 +00:00