[PR #48] [CLOSED] fix: bound reaper sweep concurrency and per-operation time so one hung teardown can't stall it #63

New issue

Closed

opened 2026-10-18 21:13:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/terminals/pull/48
Author: @Classic298
Created: 7/20/2026
Status: ❌ Closed

Base: main ← Head: claude/reaper-sweep-stall-6tlabg

📝 Commits (1)

a1f2fd0 fix: bound reaper sweep concurrency and per-operation time so one hung teardown can't stall it

📊 Changes

3 files changed (+136 additions, -72 deletions)

View changed files

📝 README.md (+2 -0)
📝 terminals/backends/base.py (+132 -72)
📝 terminals/config.py (+2 -0)

📄 Description

_reap_idle processed every tracked instance in a single sequential, timeout-less pass, and _reaper_loop awaits the full sweep before the next cycle. On large fleets, when a batch of scheduled resets came due at once, one slow or hung Docker/K8s call blocked the await and froze the entire loop with nothing logged — idle reaping and scheduled resets both silently stopped, and containers leaked until the Docker bridge hit its 1024-port cap. Only the provision-path reset kept working, so resets were applied only for reconnecting users.

Split the sweep into a cheap scan that collects due instances, then reap them concurrently under a semaphore (TERMINALS_REAPER_CONCURRENCY, default 8) with a timeout on each teardown/reset call (TERMINALS_REAPER_OP_TIMEOUT_SECONDS, default 120). A timed-out teardown is logged and the instance stays tracked so the next sweep retries it, rather than orphaning a possibly-still-running container. Worst-case sweep duration is now bounded instead of unbounded.

Fixes #46

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/terminals/pull/48 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 7/20/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `claude/reaper-sweep-stall-6tlabg` --- ### 📝 Commits (1) - [`a1f2fd0`](https://github.com/open-webui/terminals/commit/a1f2fd0ff2dc3eb4bbd538eb104db8f4595f82cc) fix: bound reaper sweep concurrency and per-operation time so one hung teardown can't stall it ### 📊 Changes **3 files changed** (+136 additions, -72 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+2 -0) 📝 `terminals/backends/base.py` (+132 -72) 📝 `terminals/config.py` (+2 -0) </details> ### 📄 Description _reap_idle processed every tracked instance in a single sequential, timeout-less pass, and _reaper_loop awaits the full sweep before the next cycle. On large fleets, when a batch of scheduled resets came due at once, one slow or hung Docker/K8s call blocked the await and froze the entire loop with nothing logged — idle reaping and scheduled resets both silently stopped, and containers leaked until the Docker bridge hit its 1024-port cap. Only the provision-path reset kept working, so resets were applied only for reconnecting users. Split the sweep into a cheap scan that collects due instances, then reap them concurrently under a semaphore (TERMINALS_REAPER_CONCURRENCY, default 8) with a timeout on each teardown/reset call (TERMINALS_REAPER_OP_TIMEOUT_SECONDS, default 120). A timed-out teardown is logged and the instance stays tracked so the next sweep retries it, rather than orphaning a possibly-still-running container. Worst-case sweep duration is now bounded instead of unbounded. Fixes #46 ## Contributor License Agreement  - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](./CLA.md), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:13:54 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/terminals#63

No description provided.

Rows
Columns