[PR #43] [CLOSED] PERF/FIX: Reduce orchestrator CPU at high session counts #59

New issue

Closed

opened 2026-10-18 21:13:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-18 21:13:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/terminals/pull/43
Author: @Classic298
Created: 7/15/2026
Status: ❌ Closed

Base: main ← Head: perf-hotpath-upstream

📝 Commits (2)

d3967d5 Reduce orchestrator CPU at high session counts
5da4999 Merge branch 'main' into perf-hotpath-upstream

📊 Changes

9 files changed (+239 additions, -42 deletions)

View changed files

📝 README.md (+4 -0)
📝 dev.sh (+1 -1)
📝 pyproject.toml (+6 -1)
📝 terminals/backends/base.py (+36 -0)
📝 terminals/cli.py (+4 -0)
📝 terminals/config.py (+10 -0)
📝 terminals/middleware.py (+30 -13)
📝 terminals/routers/auth.py (+29 -0)
📝 terminals/routers/proxy.py (+119 -27)

📄 Description

Description

At ~500-600 concurrent terminals, the orchestrator process is permanently pinned
at 100% of one CPU core (it's a single asyncio process). We profiled a production
deployment with py-spy and eliminated the hot spots; the same load now runs at
~32% of one core (GIL contention dropped from 92% to 24%).

Performance changes:

One small HTTP client per upstream (host, port) instead of a single shared
connection pool. httpcore scans the entire pool on every request event, with a
readability syscall per idle keepalive connection — with hundreds of distinct
container origins in one pool this alone consumed ~69% of the event loop.
Stale origins are LRU-evicted and their sockets closed.
WebSocket permessage-deflate disabled by default on both proxy legs
(TERMINALS_WS_COMPRESSION re-enables). Compressing every terminal frame in
pure Python is a large per-frame cost for traffic that rarely leaves the host.
Confirmed-running container status is cached for TERMINALS_STATUS_CACHE_TTL
seconds (default 30, 0 restores check-every-request), removing two Docker
inspects + JSON parsing per proxied request. On connection failure the proxy
invalidates the cache and re-resolves mid-request, so a dead container is
re-provisioned on retry instead of returning 502.
Successful Open WebUI token validations cached for TERMINALS_TOKEN_CACHE_TTL
seconds (default 60, 0 disables) in JWT mode — previously one Open WebUI
round trip per proxied request. Note: a revoked token stays usable up to TTL.
Access logging is now opt-in (TERMINALS_ACCESS_LOG); each record walks the
stack in the loguru intercept handler. Maintainers may prefer default true —
happy to flip.
RequestIdMiddleware rewritten as pure ASGI (BaseHTTPMiddleware adds
per-request task overhead).

Bug fixes found along the way:

active_ws_connections leaked on failed connect attempts (early returns
skipped the decrement).
WS upgrade failures (InvalidHandshake, e.g. container still initializing)
escaped the retry loop as unhandled exceptions.
Wheels shipped without the admin UI: .gitignore contains build/ and
hatchling excludes VCS-ignored files even when they match include patterns,
so terminals/frontend/build was missing from pip install . and GET /
returned 401 in Docker images. Fixed by declaring it via artifacts.

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/terminals/pull/43 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 7/15/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `perf-hotpath-upstream` --- ### 📝 Commits (2) - [`d3967d5`](https://github.com/open-webui/terminals/commit/d3967d52eac8f43a97c4e03fcfadf3723d8a21a0) Reduce orchestrator CPU at high session counts - [`5da4999`](https://github.com/open-webui/terminals/commit/5da4999b5a804315f171d5cef621783eb21b438b) Merge branch 'main' into perf-hotpath-upstream ### 📊 Changes **9 files changed** (+239 additions, -42 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+4 -0) 📝 `dev.sh` (+1 -1) 📝 `pyproject.toml` (+6 -1) 📝 `terminals/backends/base.py` (+36 -0) 📝 `terminals/cli.py` (+4 -0) 📝 `terminals/config.py` (+10 -0) 📝 `terminals/middleware.py` (+30 -13) 📝 `terminals/routers/auth.py` (+29 -0) 📝 `terminals/routers/proxy.py` (+119 -27) </details> ### 📄 Description ## Description At ~500-600 concurrent terminals, the orchestrator process is permanently pinned at 100% of one CPU core (it's a single asyncio process). We profiled a production deployment with py-spy and eliminated the hot spots; the same load now runs at ~32% of one core (GIL contention dropped from 92% to 24%). **Performance changes:** - One small HTTP client per upstream `(host, port)` instead of a single shared connection pool. httpcore scans the entire pool on every request event, with a readability syscall per idle keepalive connection — with hundreds of distinct container origins in one pool this alone consumed ~69% of the event loop. Stale origins are LRU-evicted and their sockets closed. - WebSocket permessage-deflate disabled by default on both proxy legs (`TERMINALS_WS_COMPRESSION` re-enables). Compressing every terminal frame in pure Python is a large per-frame cost for traffic that rarely leaves the host. - Confirmed-running container status is cached for `TERMINALS_STATUS_CACHE_TTL` seconds (default 30, `0` restores check-every-request), removing two Docker inspects + JSON parsing per proxied request. On connection failure the proxy invalidates the cache and re-resolves mid-request, so a dead container is re-provisioned on retry instead of returning 502. - Successful Open WebUI token validations cached for `TERMINALS_TOKEN_CACHE_TTL` seconds (default 60, `0` disables) in JWT mode — previously one Open WebUI round trip per proxied request. Note: a revoked token stays usable up to TTL. - Access logging is now opt-in (`TERMINALS_ACCESS_LOG`); each record walks the stack in the loguru intercept handler. Maintainers may prefer default `true` — happy to flip. - `RequestIdMiddleware` rewritten as pure ASGI (`BaseHTTPMiddleware` adds per-request task overhead). **Bug fixes found along the way:** - `active_ws_connections` leaked on failed connect attempts (early returns skipped the decrement). - WS upgrade failures (`InvalidHandshake`, e.g. container still initializing) escaped the retry loop as unhandled exceptions. - Wheels shipped without the admin UI: `.gitignore` contains `build/` and hatchling excludes VCS-ignored files even when they match `include` patterns, so `terminals/frontend/build` was missing from `pip install .` and `GET /` returned 401 in Docker images. Fixed by declaring it via `artifacts`. ## Contributor License Agreement  - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](./CLA.md), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-18 21:13:52 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

open-webui/terminals#59

No description provided.

Rows
Columns