open-webui/terminals
1
0
Fork 0
mirror of https://github.com/open-webui/terminals.git synced 2026-11-05 01:37:21 +00:00
Code Issues 14 Projects Packages Wiki Activity

[GH-ISSUE #52] Bug: Proxy httpx.ReadTimeout when /execute?wait=300 reaches its allowed wait limit #24

New issue
Closed
opened 2026-10-18 21:13:30 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-18 21:13:30 +00:00
Owner
Copy link

Originally created by @Classic298 on GitHub (Aug 5, 2026).
Original GitHub issue: https://github.com/open-webui/terminals/issues/52

Summary

Terminals v0.1.0 uses a hard-coded 300 second httpx timeout for the internal proxy connection to a per-user Open Terminal container. Open Terminal's /execute endpoint also permits wait=300.

When a caller sends POST /p/{policy_id}/execute?wait=300, Terminals can time out while waiting for the upstream response headers, even though the per-user terminal is still handling a valid request. There is no margin for normal scheduling, process cleanup, log reading, or response serialization after the command wait reaches 300 seconds.

The resulting httpx.ReadTimeout is unhandled by the proxy route and produces an ASGI exception / 500 response.

Environment

  • Terminals: ghcr.io/open-webui/terminals:v0.1.0
  • Backend: Docker
  • Per-user terminal image: ghcr.io/open-webui/open-terminal
  • Request route: named-policy proxy (/p/{policy_id}/execute)
  • Reverse proxy: nginx with proxy_read_timeout 1800s and proxy_send_timeout 1800s

The nginx timeout is not involved: the failure occurs on the direct Docker-network connection from Terminals to the per-user container.

Reproduction

  1. Start Terminals v0.1.0 with the Docker backend and a policy.

  2. Send an execute request through Terminals with a long wait, for example:

    curl -X POST 'https://terminals.example.com/p/ai-computer/execute?wait=300&tail=10000' \
  -H 'Authorization: Bearer <terminals-api-key>' \
  -H 'X-User-Id: <user-id>' \
  -H 'Content-Type: application/json' \
  --data '{"command":"sleep 300"}'

  3. Repeat under normal service load and/or with a command that finishes close to the five-minute boundary.

Actual result

Terminals logs an unhandled exception and the caller receives a failed request. Relevant log excerpt:

httpcore.ReadTimeout
...
httpx.ReadTimeout
...
Request('POST', 'http://terminals-<user>-ai-computer:8000/execute?wait=300&tail=10000')
...
AsyncHTTP11Connection._receive_event(timeout=300.0)

The error is raised at the internal proxy request:

terminals/routers/proxy.py:180 in _proxy_request
    upstream_resp = await client.send(...)

Root cause

In terminals/routers/proxy.py, Terminals v0.1.0 creates its upstream client with a fixed overall timeout of 300 seconds:

client = httpx.AsyncClient(
    timeout=httpx.Timeout(300.0, connect=10.0),
    limits=httpx.Limits(...),
)

The proxied Open Terminal endpoint accepts wait values up to 300 seconds. Therefore wait=300 and the proxy's upstream read timeout have the same deadline.

Even when the command itself completes within the allowed wait, the upstream still has to finish process/log handling and return response headers. Any overhead at the boundary makes the outer proxy time out first.

Expected result

A valid /execute?wait=300 request should not fail solely because the Terminals proxy has the same 300-second deadline as the endpoint's maximum permitted wait.

If the upstream does not respond within the configured proxy deadline, Terminals should return a deliberate, structured gateway-timeout response rather than an unhandled ASGI stack trace.

Proposed fix

Preferred: configurable upstream proxy timeouts

Add settings such as:

TERMINALS_PROXY_CONNECT_TIMEOUT_SECONDS=10
TERMINALS_PROXY_READ_TIMEOUT_SECONDS=360
TERMINALS_PROXY_WRITE_TIMEOUT_SECONDS=300
TERMINALS_PROXY_POOL_TIMEOUT_SECONDS=300

Then apply them in _get_proxy_client():

httpx.AsyncClient(
    timeout=httpx.Timeout(
        connect=settings.proxy_connect_timeout_seconds,
        read=settings.proxy_read_timeout_seconds,
        write=settings.proxy_write_timeout_seconds,
        pool=settings.proxy_pool_timeout_seconds,
    ),
    limits=...,
)

The default read timeout should be greater than the maximum supported Open Terminal /execute?wait duration. For example, 360 seconds leaves a 60-second response-processing margin.

Alternative / additional improvement: derive the deadline for /execute

For requests to /execute, parse the validated wait query value and set the per-request upstream read timeout to wait + safety_margin. This avoids an arbitrary global ceiling for an endpoint that intentionally supports long-polling.

Error handling

Handle httpx.ReadTimeout explicitly in _proxy_request() and return a meaningful 504 Gateway Timeout response. Do not blindly retry POST /execute after a read timeout, because the child command may already have started and a retry could run it twice.

Workaround

A local wrapper image that changes the internal read timeout from 300 to 360 seconds prevents the boundary timeout, but it is not maintainable because the timeout is currently hard-coded and has no environment-variable configuration.

Originally created by @Classic298 on GitHub (Aug 5, 2026). Original GitHub issue: https://github.com/open-webui/terminals/issues/52 ## Summary Terminals v0.1.0 uses a hard-coded **300 second** `httpx` timeout for the internal proxy connection to a per-user Open Terminal container. Open Terminal's `/execute` endpoint also permits `wait=300`. When a caller sends `POST /p/{policy_id}/execute?wait=300`, Terminals can time out while waiting for the upstream response headers, even though the per-user terminal is still handling a valid request. There is no margin for normal scheduling, process cleanup, log reading, or response serialization after the command wait reaches 300 seconds. The resulting `httpx.ReadTimeout` is unhandled by the proxy route and produces an ASGI exception / 500 response. ## Environment - Terminals: `ghcr.io/open-webui/terminals:v0.1.0` - Backend: Docker - Per-user terminal image: `ghcr.io/open-webui/open-terminal` - Request route: named-policy proxy (`/p/{policy_id}/execute`) - Reverse proxy: nginx with `proxy_read_timeout 1800s` and `proxy_send_timeout 1800s` The nginx timeout is not involved: the failure occurs on the direct Docker-network connection from Terminals to the per-user container. ## Reproduction 1. Start Terminals v0.1.0 with the Docker backend and a policy. 2. Send an execute request through Terminals with a long wait, for example: ```bash curl -X POST 'https://terminals.example.com/p/ai-computer/execute?wait=300&tail=10000' \ -H 'Authorization: Bearer <terminals-api-key>' \ -H 'X-User-Id: <user-id>' \ -H 'Content-Type: application/json' \ --data '{"command":"sleep 300"}' ``` 3. Repeat under normal service load and/or with a command that finishes close to the five-minute boundary. ## Actual result Terminals logs an unhandled exception and the caller receives a failed request. Relevant log excerpt: ```text httpcore.ReadTimeout ... httpx.ReadTimeout ... Request('POST', 'http://terminals-<user>-ai-computer:8000/execute?wait=300&tail=10000') ... AsyncHTTP11Connection._receive_event(timeout=300.0) ``` The error is raised at the internal proxy request: ```text terminals/routers/proxy.py:180 in _proxy_request upstream_resp = await client.send(...) ``` ## Root cause In `terminals/routers/proxy.py`, Terminals v0.1.0 creates its upstream client with a fixed overall timeout of 300 seconds: ```python client = httpx.AsyncClient( timeout=httpx.Timeout(300.0, connect=10.0), limits=httpx.Limits(...), ) ``` The proxied Open Terminal endpoint accepts `wait` values up to 300 seconds. Therefore `wait=300` and the proxy's upstream read timeout have the same deadline. Even when the command itself completes within the allowed wait, the upstream still has to finish process/log handling and return response headers. Any overhead at the boundary makes the outer proxy time out first. ## Expected result A valid `/execute?wait=300` request should not fail solely because the Terminals proxy has the same 300-second deadline as the endpoint's maximum permitted wait. If the upstream does not respond within the configured proxy deadline, Terminals should return a deliberate, structured gateway-timeout response rather than an unhandled ASGI stack trace. ## Proposed fix ### Preferred: configurable upstream proxy timeouts Add settings such as: ```text TERMINALS_PROXY_CONNECT_TIMEOUT_SECONDS=10 TERMINALS_PROXY_READ_TIMEOUT_SECONDS=360 TERMINALS_PROXY_WRITE_TIMEOUT_SECONDS=300 TERMINALS_PROXY_POOL_TIMEOUT_SECONDS=300 ``` Then apply them in `_get_proxy_client()`: ```python httpx.AsyncClient( timeout=httpx.Timeout( connect=settings.proxy_connect_timeout_seconds, read=settings.proxy_read_timeout_seconds, write=settings.proxy_write_timeout_seconds, pool=settings.proxy_pool_timeout_seconds, ), limits=..., ) ``` The default read timeout should be greater than the maximum supported Open Terminal `/execute?wait` duration. For example, 360 seconds leaves a 60-second response-processing margin. ### Alternative / additional improvement: derive the deadline for `/execute` For requests to `/execute`, parse the validated `wait` query value and set the per-request upstream read timeout to `wait + safety_margin`. This avoids an arbitrary global ceiling for an endpoint that intentionally supports long-polling. ### Error handling Handle `httpx.ReadTimeout` explicitly in `_proxy_request()` and return a meaningful `504 Gateway Timeout` response. Do **not** blindly retry `POST /execute` after a read timeout, because the child command may already have started and a retry could run it twice. ## Workaround A local wrapper image that changes the internal read timeout from 300 to 360 seconds prevents the boundary timeout, but it is not maintainable because the timeout is currently hard-coded and has no environment-variable configuration.
bsakaguchi commented 2026-10-18 21:13:31 +00:00
Author
Owner
Copy link

@owui-terminator[bot] commented on GitHub (Aug 5, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

  1. 🟣 #24 issue: _proxy_request returns 500 on stale keep-alive connections (RemoteProtocolError not retried)
    Related at the proxy layer: both issues are about _proxy_request in terminals/routers/proxy.py surfacing upstream httpx failures as 500s instead of handling them gracefully. This one is specifically a stale-connection RemoteProtocolError, while the new issue is a boundary ReadTimeout, but the shared missing error handling and proxy behavior make it the closest match.
    by Classic298

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:5193774868 --> @owui-terminator[bot] commented on GitHub (Aug 5, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟣 [#24](https://github.com/open-webui/terminals/issues/24) **issue: _proxy_request returns 500 on stale keep-alive connections (RemoteProtocolError not retried)** *Related at the proxy layer: both issues are about `_proxy_request` in `terminals/routers/proxy.py` surfacing upstream `httpx` failures as 500s instead of handling them gracefully. This one is specifically a stale-connection `RemoteProtocolError`, while the new issue is a boundary `ReadTimeout`, but the shared missing error handling and proxy behavior make it the closest match.* *by Classic298* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.
bsakaguchi commented 2026-10-18 21:13:31 +00:00
Author
Owner
Copy link

@Classic298 commented on GitHub (Aug 8, 2026):

@tjbck

<!-- gh-comment-id:5228336158 --> @Classic298 commented on GitHub (Aug 8, 2026): @tjbck
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v0.2.8
v0.2.7
v0.2.6
v0.2.5
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.0
v0.0.8
v0.0.7
v0.0.6
v0.0.5
v0.0.4
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
open-webui/terminals#24
No description provided.