mirror of
https://github.com/open-webui/terminals.git
synced 2026-11-05 01:37:21 +00:00
[GH-ISSUE #41] ISSUE: Terminal Orchestrator EXTREMELY SLOW and CPU bound at growing user bases - unusable speeds, CPU bound, permanently at 100% CPU usage (one full core) #18
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
open-webui/terminals#18
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Classic298 on GitHub (Jul 15, 2026).
Original GitHub issue: https://github.com/open-webui/terminals/issues/41
at many many hundreds of containers, it's CPU usage is permanently bound to 100% and never less. Completely CPU bottlenecked.
Something about the code is very inefficient.
What makes it worse, you cant scale the orchestrator itself.
The user containers themselves dont use that much CPU, but the orchestrator is permanently bound to 100% and the Terminal User interface in Open WebUI is noticably VERY slow and the Orchestrator, as per my debugging and log analysis, also takes multiple seconds to answer simple list files requests.
@owui-terminator[bot] commented on GitHub (Jul 15, 2026):
🔍 Related Issues Found
I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:
🟢 #39 issue: policy doesnt trigger for constantly running containers
This issue is about the orchestrator never timing out constantly running containers, which points to scaling/runtime behavior in the orchestrator under many active containers. It is not the same symptom, but it touches the same orchestrator load and lifecycle behavior that could contribute to heavy CPU use.
by Classic298
🟢 #36 issue: Terminal system prompt is never injected for orchestrator connections (
/systemfetch omitsX-User-Id)This issue involves the orchestrator proxy path and repeated failed fetches through
/p/{policy_id}/*, which is adjacent to the slow orchestrator requests described here. While the main bug is missing headers, it is still relevant because it exercises the same request path and backend logic that could impact latency under load.by Classic298
💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.
This comment was generated automatically. React with 👍 if helpful, 👎 if not.
@Classic298 commented on GitHub (Jul 15, 2026):
https://github.com/user-attachments/assets/1541dc05-772e-484e-ac09-d166133a24e0
Unacceptably slow - and the process is CONSTANTLY bound at 100%
@Classic298 commented on GitHub (Jul 15, 2026):
Orchestrator CPU bottleneck at ~500 concurrent terminals: single-core ceiling, per-frame WS compression, per-request Docker inspects (+ related bugs found while investigating)
Environment
main(v0.0.5), Docker backendterminals serveprocess is permanently pinned at exactly ~100% CPU (one full core) in htop whiledockerdsits at ~2%. Terminal I/O latency degrades for all users; the process cannot use a second core."Exactly 100%" is the tell: the orchestrator is one Python process, so the GIL caps it at ~1 core, and the entire proxy data plane runs on that one asyncio event loop. Everything below was found by reading the code and verified against library sources; line references are against current
main.A. Hot-path performance issues
A1. WebSocket permessage-deflate is enabled by default on BOTH proxy legs (dominant CPU cost)
cli.py:60callsuvicorn.run(...)withoutws_per_message_deflate, and uvicorn's default isTrue.routers/proxy.py:497callswebsockets.connect(upstream_url), and thewebsocketsdefault iscompression="deflate".Every terminal frame — every keystroke echo, every chunk of command output, for every session — is zlib-compressed and decompressed in pure Python, twice (once per leg). Terminal traffic is high-frequency small frames, the worst case for per-frame deflate (per-frame CPU overhead, negligible bandwidth win on a LAN/localhost hop). At hundreds of concurrent sessions this dominates the event loop.
Suggested fix: disable compression by default on both legs (
ws_per_message_deflate=False,compression=None), with an opt-in setting for deployments where clients are on slow links.A2. Every proxied HTTP request performs two Docker
inspectcallsBackend.ensure_terminal()'s fast path callsstatus()on every request (backends/base.py:117), andDockerBackend.status()doescontainers.get(instance_id)+container.show()(backends/docker.py:318-319) — two identicalGET /containers/{id}/jsonround trips, each returning a ~20 KB JSON document parsed in Python. The proxy hot path (_resolve_instance) therefore costs 2 Docker API calls + 2 large JSON parses per request, per user, before a single byte is proxied.Suggested fix: trust a confirmed-running status for a short TTL (and/or subscribe to the Docker events stream to invalidate on
die/stop— see C3), invalidating on proxy connection failure so a dead container is still re-detected promptly.A3. JWT mode: one HTTP round trip to Open WebUI per proxied request
When
TERMINALS_OPEN_WEBUI_URLis set,verify_api_keycallsvalidate_token()on every request (routers/auth.py:74), which issuesGET {open_webui}/api/v1/auths/(routers/auth.py:38) with no caching. Every proxied request pays a full network round trip plus JSON parse before touching the terminal. Same on WS connects (routers/proxy.py:451).Suggested fix: short-TTL token-validation cache (keyed by token hash), or accept locally-verifiable signed tokens.
A4. Request bodies are fully buffered in memory
_proxy_requestreads the entire request body up front for retry purposes (routers/proxy.py:150). Large file uploads through/files/*are buffered fully in orchestrator RAM instead of streamed, costing memory and latency at concurrency.Suggested fix: stream bodies for requests above a size threshold and restrict retries to requests with replayable bodies.
A5.
RequestIdMiddlewareuses Starlette'sBaseHTTPMiddlewaremiddleware.py:10—BaseHTTPMiddlewareis a known per-request overhead source (it wraps every request in an additional task/anyio machinery) and it sits on the hottest path in the system. A pure-ASGI middleware does the same job for near zero cost.A6. Minor:
_wait_until_readybuilds a newhttpx.AsyncClientper poll iterationbackends/docker.py:172constructs and tears down a client every 0.5 s while waiting for a container to become healthy. Provisioning-path only, but trivially avoidable.A7. Minor: duplicate inspects in discovery paths
_extract_instance_infoalways callscontainer.show()(backends/docker.py:189) even when the caller already holds the inspect payload — e.g.reconcile()inspects atdocker.py:239and then_extract_instance_infore-inspects atdocker.py:263, doubling Docker API calls for every container on startup.B. Correctness bugs found while investigating
B1.
active_ws_connectionscounter leaks on failed connection attemptsThe counter is incremented on entry to
_ws_proxy_handler(routers/proxy.py:479), but the early-return failure paths (instance-resolve failure, connect-retry exhaustion) return before thetry/finallythat decrements it. Every WS attempt against an unreachable container permanently inflates the stats counter.B2. WS connect retry loop misses handshake failures
routers/proxy.py:499catches only(ConnectionRefusedError, OSError).websockets.connect()also raisesInvalidHandshakesubclasses (e.g.InvalidStatus) when the container's HTTP port answers but the upgrade returns non-101 — typical for a container that is still initializing. Those escape as unhandled ASGI exceptions instead of hitting the retry loop.B3.
provision()retry loop can die withUnboundLocalError, and the StorageOpt fallback fires on any errorIn
backends/docker.py, if the final retry attempt raises 409 whileStorageOptis set, the 409 branch is skipped (attempt < max_conflict_retriesis false), the StorageOpt fallback pops the quota andcontinues, the loop exhausts, andcontaineris referenced unbound atdocker.py:161→UnboundLocalErrorsurfaces as a 500. Separately, that fallback catches everyDockerError(not just quota-unsupported ones), so unrelated provisioning errors silently drop the user's storage limit and retry.B4. Wheel packaging silently drops the admin UI (
.gitignorevs hatchling).gitignorecontainsbuild/, which matchesterminals/frontend/build/. Hatchling excludes VCS-ignored files even when they matchincludepatterns — ignored build outputs must be declared via theartifactsoption. As a result,pip install .produces a wheel withoutterminals/frontend/build, so in any image built from the repo Dockerfile (which pip-installs the package; the console script imports from site-packages, not/app),FRONTEND_BUILD_DIR.exists()isFalse, the UI routes are never registered, andGET /falls through to the auth-guarded catch-all proxy and returns401 {"detail":"Missing Authorization header"}instead of the admin UI.Fix: add to
pyproject.toml:B5.
create_or_replacedestroys running containers instead of adopting thembackends/docker.py:126provisions withcontainers.create_or_replace. In aiodocker,create_or_replacestops and deletes an existing container whenever the desired config differs — and it always differs, because each provision generates a freshOPEN_TERMINAL_API_KEY. Any situation where a running container exists but isn't in the in-memory_instancesmap (state loss, crash-restart race, or any attempt to run more than one orchestrator process) replaces the user's live container and kills their sessions, where adopting the existing container (its name is deterministic and the API key is recoverable from its env) would be transparent.C. Architectural ceiling
C1. The orchestrator is a single Python process with no worker support
cli.py:60runs uvicorn with noworkersoption. The GIL caps the process at ~1 core regardless of host size — on a 24-core machine, 23+ cores are unreachable to the data plane. All terminal bytes for every user (WS pump inrouters/proxy.py) traverse this one event loop.C2. All orchestration state is process-local, so you can't just add workers
_instances,_activity,_specs, and per-keyasyncio.Locks live in process memory (backends/base.py). Naively runninguvicorn --workers Ntoday is unsafe:/api/v1/terminals,stop,refresh) would only see the receiving worker's subset.Multi-worker (or multi-replica) support needs: container adoption by deterministic name instead of replace, shared last-active state (the DB is already a dependency), and conflict-safe provisioning (
create+ adopt-on-409).C3. Status checking is poll-based; the Docker events API is unused
Rather than inspecting containers per request (A2), the backend could subscribe to the Docker events stream (label-filtered
die/stop/destroy) and invalidate tracked instances event-driven, making the request fast path a pure dictionary lookup with no staleness window.Reference implementation
We run this in production and have working patches for most of the above (happy to send PRs — split however you prefer):
Classic298/terminals@claude/terminals-perf-only-5jz2w3TERMINALS_WORKERS, container adoption via deterministic names,create+ adopt-on-conflict provisioning (fixes B3/B5), DB-shared last-active timestamps with a fail-closed reaper, SQLite WAL + file-locked startup migrations — branch:Classic298/terminals@claude/terminals-cpu-bottleneck-5jz2w3How to reproduce / measure
On any deployment with a few hundred active sessions:
Before the fixes, the profile is dominated by zlib/permessage-deflate frames and JSON parsing of Docker inspect payloads;
dockerdstaying at ~2% CPU whileterminals servepins a core confirms the cost is Python-side. After A1+A2 alone, our orchestrator CPU dropped from a saturated core to a small fraction of one at the same load.@Classic298 commented on GitHub (Jul 15, 2026):
Follow-up: profiling data from production (~600 concurrent terminals) — two more CPU sinks
We ran
py-spy topagainst the live orchestrator after applying the compression + status-cache fixes from the original report. Two additional hot spots showed up that aren't in the list above:1. httpx/httpcore shared connection pool scanning — ~69% of the event loop.
The proxy uses a single shared
httpx.AsyncClientfor all upstream containers (routers/proxy.py). httpcore's_assign_requests_to_connectionsruns on every request start/finish and iterates the entire pool —is_idle()/has_expired()checks per connection, includingis_socket_readable(), which is a syscall per idle keepalive connection. With each user container being a distincthost:portorigin in one pool, this is effectively O(pool size) work per request event: at ~600 containers the profile showed ~69% of main-thread time in this tree (_assign_requests_to_connections69% total,is_idle28%,has_expired18%,is_socket_readable8%). The 4skeepalive_expiryin_get_proxy_clientamplifies it via constant connection churn.Fix that worked for us: one small client per upstream
(host, port)(a few keepalive connections each, LRU-evicted for torn-down containers) instead of one shared pool — each scan then touches a handful of connections for exactly one origin. Implemented in the branches referenced above.2. Per-request access logging through the loguru intercept — measurable at high request rates.
Every request generates a uvicorn access-log record that passes through
_InterceptHandler.emit(logging.py), which walks the Python stack frame-by-frame (logging.currentframe()loop) per record before loguru formats and writes it. Making access logs opt-out recovered a few percent of CPU on top of everything else.Also worth noting for anyone profiling this themselves: py-spy reports aiosqlite's
_connection_worker_threadas ~100% "active" per DB connection — that's an artifact (the thread blocks inSimpleQueue.get()); htop shows the truth.Profiling confirmed the middleware (
BaseHTTPMiddleware) and JWT per-request validation costs already listed above are real but secondary to the pool scanning.@Classic298 commented on GitHub (Jul 15, 2026):
https://github.com/open-webui/terminals/pull/43/