open-webui/terminals

Fork 0

mirror of https://github.com/open-webui/terminals.git synced 2026-11-05 01:37:21 +00:00

[PR #16] [MERGED] fix: auto-recover errored terminal CRs in operator backend #49

New issue

Closed

opened 2026-10-18 21:13:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:46 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/terminals/pull/16
Author: @james-westbrook
Created: 3/31/2026
Status: ✅ Merged
Merged: 4/1/2026
Merged by: @tjbck

Base: main ← Head: fix-k8s-errored-terminals

📝 Commits (1)

22bddc0 fix: auto-recover errored terminal CRs in operator backend

📊 Changes

1 file changed (+6 additions, -1 deletions)

View changed files

📝 terminals/backends/kubernetes_operator.py (+6 -1)

📄 Description

Description

Problem

When the operator backend encounters an existing Terminal CR stuck in phase: Error, it falls through to the "still provisioning" wait path, which polls for up to 120 seconds waiting for the CR to reach Running. Since the operator will never recover an Error-phase CR on its own, this always times out and returns a 502 to the user.

This was discovered during EKS E2E testing. A transient pod failure (e.g. EBS volume attachment timeout, readiness probe deadline exceeded) caused a Terminal CR to enter phase: Error with reason: PodNotReady. On subsequent requests for the same user, the orchestrator found the existing Error CR via a 409 conflict on create, returned it, and then waited indefinitely for it to become Running — which never happened.

The only recovery path was manual deletion of the stale CR (kubectl delete terminals.openwebui.com <name>).

Fix

In ensure_terminal(), the Idle phase already triggers a delete-and-reprovision cycle. This PR extends that same handling to the Error phase:

# Before
if phase == "Idle":
    await self._delete_terminal_cr(user_id, policy_id)
    return await self.provision(...)

# After
if phase in ("Idle", "Error"):
    log.info("Terminal CR %s in phase %s — deleting and re-provisioning", ...)
    await self._delete_terminal_cr(user_id, policy_id)
    return await self.provision(...)

When a user requests a terminal and the existing CR is in Error, the orchestrator now:

Logs the stale CR name and phase
Deletes the failed CR (and its owned resources via the operator's finalizer)
Creates a fresh CR and waits for it to become Running

Testing

Verified on both Kind (local) and EKS (us-west-2) clusters, with both the Kubernetes backend and Kubernetes Operator backend performing as expected after this change.

The bug was originally caught when a first test run left Alice's Terminal CR in Error phase. A subsequent test run failed (all Alice-related: provision timeout, execute timeout, file ops timeout) because the stale CR was never cleaned up. After applying this fix, the orchestrator auto-recovers and all manual tests pass.

Changed files

terminals/backends/kubernetes_operator.py — extend Idle-phase delete-and-reprovision to also cover Error phase

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/terminals/pull/16 **Author:** [@james-westbrook](https://github.com/james-westbrook) **Created:** 3/31/2026 **Status:** ✅ Merged **Merged:** 4/1/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `main` ← **Head:** `fix-k8s-errored-terminals` --- ### 📝 Commits (1) - [`22bddc0`](https://github.com/open-webui/terminals/commit/22bddc010fd9e4a0db1b014c1154b0a395721bf5) fix: auto-recover errored terminal CRs in operator backend ### 📊 Changes **1 file changed** (+6 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `terminals/backends/kubernetes_operator.py` (+6 -1) </details> ### 📄 Description ## Description ### Problem When the operator backend encounters an existing Terminal CR stuck in `phase: Error`, it falls through to the "still provisioning" wait path, which polls for up to 120 seconds waiting for the CR to reach `Running`. Since the operator will never recover an Error-phase CR on its own, this always times out and returns a 502 to the user. This was discovered during EKS E2E testing. A transient pod failure (e.g. EBS volume attachment timeout, readiness probe deadline exceeded) caused a Terminal CR to enter `phase: Error` with `reason: PodNotReady`. On subsequent requests for the same user, the orchestrator found the existing Error CR via a 409 conflict on create, returned it, and then waited indefinitely for it to become `Running` — which never happened. The only recovery path was manual deletion of the stale CR (`kubectl delete terminals.openwebui.com <name>`). ### Fix In `ensure_terminal()`, the `Idle` phase already triggers a delete-and-reprovision cycle. This PR extends that same handling to the `Error` phase: ```python # Before if phase == "Idle": await self._delete_terminal_cr(user_id, policy_id) return await self.provision(...) # After if phase in ("Idle", "Error"): log.info("Terminal CR %s in phase %s — deleting and re-provisioning", ...) await self._delete_terminal_cr(user_id, policy_id) return await self.provision(...) ``` When a user requests a terminal and the existing CR is in `Error`, the orchestrator now: 1. Logs the stale CR name and phase 2. Deletes the failed CR (and its owned resources via the operator's finalizer) 3. Creates a fresh CR and waits for it to become `Running` ### Testing Verified on both Kind (local) and EKS (us-west-2) clusters, with both the Kubernetes backend and Kubernetes Operator backend performing as expected after this change. The bug was originally caught when a first test run left Alice's Terminal CR in `Error` phase. A subsequent test run failed (all Alice-related: provision timeout, execute timeout, file ops timeout) because the stale CR was never cleaned up. After applying this fix, the orchestrator auto-recovers and all manual tests pass. ### Changed files - `terminals/backends/kubernetes_operator.py` — extend `Idle`-phase delete-and-reprovision to also cover `Error` phase ## Contributor License Agreement  - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](./CLA.md), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:13:46 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/terminals#49

No description provided.

Rows
Columns