[GH-ISSUE #24] issue: _proxy_request returns 500 on stale keep-alive connections (RemoteProtocolError not retried) #7

New issue

Closed

opened 2026-10-18 21:13:19 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-18 21:13:19 +00:00

Owner

Originally created by @Classic298 on GitHub (May 13, 2026).
Original GitHub issue: https://github.com/open-webui/terminals/issues/24

Summary

When the orchestrator's pooled HTTP connection to an upstream user-container instance is closed server-side after a long-lived keep-alive session, the next request the orchestrator tries to make on that connection fails with httpx.RemoteProtocolError: Server disconnected without sending a response. and is surfaced as a 500 to the caller (Open WebUI). The next request opens a fresh connection and succeeds, so the failure is transient — but it produces visible 500s in the Open WebUI UI any time a connection ages out mid-poll.

Environment

terminals: ghcr.io/open-webui/terminals:latest (Docker backend)
User image: ghcr.io/open-webui/open-terminal:latest
Open WebUI polls e.g. GET /p/<policy>/ports every 5s for the PortList component, which means a long-lived connection accumulates many requests.

Observed

A connection that had handled ~50 requests on the same TCP socket gets closed server-side by uvicorn (presumably idle timeout or max-keepalive-requests). httpx writes the next request to the now-dead connection:

... <AsyncHTTP11Connection [...:8000, CLOSED, Request Count: 53]>
... receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
... 172.30.0.1:55506 - "GET /p/computer/ports HTTP/1.1" 500
...
httpx.RemoteProtocolError: Server disconnected without sending a response.

5 seconds later, on the next poll, a fresh TCP connection is opened (connect_tcp.started) and the same request returns 200. Steady state resumes.

Root cause

In terminals/routers/proxy.py:153-175, _proxy_request retries only on httpx.ConnectError / httpx.ConnectTimeout:

for attempt in range(max_retries):
    try:
        upstream_resp = await client.send(...)
        break
    except (httpx.ConnectError, httpx.ConnectTimeout) as e:
        ...

It does not retry on httpx.RemoteProtocolError (or httpx.ReadError), which is the textbook signature of a stale-keepalive race in HTTP/1.1 connection pools. The body is already read upfront (line 147), so an idempotent retry would be safe for the GET/HEAD/OPTIONS verbs and acceptable for POST/PUT/PATCH/DELETE that have already serialised their body bytes.

Suggested fix

Extend the retry except clause to include the keep-alive-race exceptions, retrying at least once without sleep (a fresh connection is essentially free):

except (httpx.ConnectError, httpx.ConnectTimeout) as e:
    # existing logic: container still starting, slow backoff
    ...
except (httpx.RemoteProtocolError, httpx.ReadError) as e:
    # stale keep-alive connection — retry once on a fresh connection
    if attempt < max_retries - 1:
        logger.debug("Stale connection to {} ({}), retrying on fresh connection", target_url, e)
        # no sleep — fresh connection should succeed immediately
    else:
        logger.error("Proxy request to {} failed after {} retries: {}", target_url, max_retries, e)
        return Response(
            content='{"error": "Upstream connection failed"}',
            status_code=502,
            media_type="application/json",
        )

Alternatively, set a short keepalive_expiry on the shared httpx.AsyncClient (in _get_proxy_client) so the pool recycles connections before uvicorn's idle timeout can race with the next request. Both fixes together would be belt-and-braces.

Why this matters

Open WebUI polls /ports, /files/list, etc. on intervals from the terminal panel. Each transient 500 surfaces in the browser console and (depending on the component) can briefly blank UI state until the next polling tick succeeds. Users perceive intermittent flakiness even though the system is healthy.

Reproduction

Deploy terminals against the Docker backend with a configured policy.
Open a terminal session in Open WebUI.
Leave the chat open with the PortList visible (or just hit any orchestrator route on a loop) for ~5 minutes.
Watch the orchestrator logs — Request Count on the pooled connection climbs. At some point the upstream closes it; the next request fails with RemoteProtocolError → 500.

Originally created by @Classic298 on GitHub (May 13, 2026). Original GitHub issue: https://github.com/open-webui/terminals/issues/24 ## Summary When the orchestrator's pooled HTTP connection to an upstream user-container instance is closed server-side after a long-lived keep-alive session, the next request the orchestrator tries to make on that connection fails with `httpx.RemoteProtocolError: Server disconnected without sending a response.` and is surfaced as a **500** to the caller (Open WebUI). The next request opens a fresh connection and succeeds, so the failure is transient — but it produces visible 500s in the Open WebUI UI any time a connection ages out mid-poll. ## Environment - terminals: `ghcr.io/open-webui/terminals:latest` (Docker backend) - User image: `ghcr.io/open-webui/open-terminal:latest` - Open WebUI polls e.g. `GET /p/<policy>/ports` every 5s for the PortList component, which means a long-lived connection accumulates many requests. ## Observed A connection that had handled ~50 requests on the same TCP socket gets closed server-side by uvicorn (presumably idle timeout or max-keepalive-requests). httpx writes the next request to the now-dead connection: ``` ... <AsyncHTTP11Connection [...:8000, CLOSED, Request Count: 53]> ... receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.') ... 172.30.0.1:55506 - "GET /p/computer/ports HTTP/1.1" 500 ... httpx.RemoteProtocolError: Server disconnected without sending a response. ``` 5 seconds later, on the next poll, a fresh TCP connection is opened (`connect_tcp.started`) and the same request returns 200. Steady state resumes. ## Root cause In [`terminals/routers/proxy.py:153-175`](https://github.com/open-webui/terminals/blob/main/terminals/routers/proxy.py#L153), `_proxy_request` retries only on `httpx.ConnectError` / `httpx.ConnectTimeout`: ```python for attempt in range(max_retries): try: upstream_resp = await client.send(...) break except (httpx.ConnectError, httpx.ConnectTimeout) as e: ... ``` It does not retry on `httpx.RemoteProtocolError` (or `httpx.ReadError`), which is the textbook signature of a stale-keepalive race in HTTP/1.1 connection pools. The body is already read upfront (line 147), so an idempotent retry would be safe for the GET/HEAD/OPTIONS verbs and acceptable for POST/PUT/PATCH/DELETE that have already serialised their body bytes. ## Suggested fix Extend the retry except clause to include the keep-alive-race exceptions, retrying **at least once** without sleep (a fresh connection is essentially free): ```python except (httpx.ConnectError, httpx.ConnectTimeout) as e: # existing logic: container still starting, slow backoff ... except (httpx.RemoteProtocolError, httpx.ReadError) as e: # stale keep-alive connection — retry once on a fresh connection if attempt < max_retries - 1: logger.debug("Stale connection to {} ({}), retrying on fresh connection", target_url, e) # no sleep — fresh connection should succeed immediately else: logger.error("Proxy request to {} failed after {} retries: {}", target_url, max_retries, e) return Response( content='{"error": "Upstream connection failed"}', status_code=502, media_type="application/json", ) ``` Alternatively, set a short `keepalive_expiry` on the shared `httpx.AsyncClient` (in `_get_proxy_client`) so the pool recycles connections before uvicorn's idle timeout can race with the next request. Both fixes together would be belt-and-braces. ## Why this matters Open WebUI polls `/ports`, `/files/list`, etc. on intervals from the terminal panel. Each transient 500 surfaces in the browser console and (depending on the component) can briefly blank UI state until the next polling tick succeeds. Users perceive intermittent flakiness even though the system is healthy. ## Reproduction 1. Deploy `terminals` against the Docker backend with a configured policy. 2. Open a terminal session in Open WebUI. 3. Leave the chat open with the PortList visible (or just hit any orchestrator route on a loop) for ~5 minutes. 4. Watch the orchestrator logs — `Request Count` on the pooled connection climbs. At some point the upstream closes it; the next request fails with `RemoteProtocolError` → 500.

bsakaguchi closed this issue

2026-10-18 21:13:20 +00:00

bsakaguchi commented

2026-10-18 21:13:20 +00:00

Author

Owner

@Classic298 commented on GitHub (Jun 22, 2026):

@james-westbrook

@Classic298 commented on GitHub (Jun 22, 2026): @james-westbrook

bsakaguchi commented

2026-10-18 21:13:21 +00:00

Author