[GH-ISSUE #302] mcpo pins one CPU core at 100% indefinitely when an upstream streamable-HTTP MCP server returns a transient HTTP error (e.g. 502) mid-connection #135

New issue

Open

opened 2026-10-18 21:12:04 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-18 21:12:04 +00:00

Owner

Originally created by @jlacour-git on GitHub (Jul 15, 2026).
Original GitHub issue: https://github.com/open-webui/mcpo/issues/302

Summary

A single streamable-HTTP MCP upstream that returns a transient HTTP error (a 502 Bad Gateway in our
case) during an established connection causes mcpo to leave a broken async-generator teardown behind.
anyio raises RuntimeError: Attempted to exit cancel scope in a different task than it was entered in,
and from that point one CPU core is pinned at ~100% indefinitely — the process never recovers on
its own. Only podman restart mcpo (killing the process) clears it. We have observed it run ~14
hours (== full host uptime) before being noticed via fan/thermal.

This is distinct from the existing reports:

open-webui/open-webui #18316 (2+ streamable servers) and #24042 (100% CPU after cancel) — both CLOSED,
neither cures this path.
modelcontextprotocol/python-sdk #915 — cancel-scope error when the server is not available at
startup. Ours is a runtime error on an already-established connection, not startup.
modelcontextprotocol/python-sdk #2150 — server-side shutdown not terminating sessions.

The trigger here is specifically a transient upstream HTTP error on a live connection.

Environment

mcpo 0.0.20 (image ghcr.io/open-webui/mcpo:main, digest sha256:05ccaa3bb0de51eaa9d00a96c4583dde93897c1b45812b34ed3de3b033f06fbf) — this is the current :main amd64 build (verified against the live registry manifest 2026-07-15; not a stale local image)
mcp 1.26.0, anyio 4.12.1, httpx 0.28.1, httpx-sse 0.4.3, uvicorn 0.41.0, starlette 0.52.1
Python 3.12.12, Linux (Podman container, rootless)
config: one streamable-http upstream + one stdio upstream (mcp-server-fetch)

Config (redacted)

{
  "mcpServers": {
    "paperless": { "type": "streamable-http", "url": "https://<UPSTREAM-MCP-HOST>/mcp" },
    "fetch":     { "command": "uvx", "args": ["mcp-server-fetch"] }
  }
}

Symptom signature

One process pinned ~100% on a single core, thread state R, wchan=0 — a busy loop, not workload.
Container still reports healthy; logs show no tool traffic while spinning.
Never self-recovers; only a process restart clears it.

Actual traceback (from our logs)

The connection was live; the upstream returned 502 Bad Gateway mid-request. raise_for_status()
raised inside the streamable-HTTP client's TaskGroup; during teardown GeneratorExit propagated
through the client's async generator and anyio failed the cancel-scope exit:

future: <Task finished name='Task-9' coro=<<async_generator_athrow without __name__>()>
  exception=RuntimeError('Attempted to exit cancel scope in a different task than it was entered in')>
  + Exception Group Traceback (most recent call last):
  |   File ".../anyio/_backends/_asyncio.py", line 783, in __aexit__
  |     raise BaseExceptionGroup(
  | BaseExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions)
  +-+---------------- 1 ----------------
    |   File ".../mcp/client/streamable_http.py", line 565, in handle_request_async
    |     await self._handle_post_request(ctx)
    |   File ".../mcp/client/streamable_http.py", line 358, in _handle_post_request
    |     response.raise_for_status()
    | httpx.HTTPStatusError: Server error '502 Bad Gateway' for url 'https://<UPSTREAM-MCP-HOST>/mcp'
    +---------------- 2 ----------------
    |   File ".../mcp/client/streamable_http.py", line 670, in streamable_http_client
    |     yield (
    |   File ".../mcp/client/streamable_http.py", line 722, in streamablehttp_client
    |     yield streams
    | GeneratorExit

Steps to reproduce

Configure mcpo with one streamable-http upstream.
Keep the connection established, then make the upstream return a transient error (e.g. put a proxy
in front that returns 502 for one request, or bounce the upstream so an in-flight request 502s).
Observe one core pin at ~100% and stay there; mcpo never recovers without a process restart.

Expected
A transient upstream HTTP error should surface as a normal error for that request; the client should
tear down the failed connection cleanly (child task signals; the owning task closes its own cancel
scope) and mcpo should keep serving. It should never enter an unrecoverable busy-loop.

Root-cause note (for maintainers)
The failure is the anyio cancel-scope ownership rule being violated during streamable-HTTP teardown:
a cancel scope entered by the supervisor task is exited from the child (reader) task's context via a
shared stream/generator aclose(). anyio can only detect this; the fix has to be in the client's
connection lifecycle (child signals, parent closes its own scope), and it needs to be robust to the
teardown being triggered by an in-flight HTTPStatusError, not just clean shutdown.

Workaround we deployed
A host watchdog that detects the sustained single-core pin (Δ CPU jiffies ≈ one full core sustained
over several minutes) and restarts mcpo + alerts. Effective but obviously a band-aid.

Originally created by @jlacour-git on GitHub (Jul 15, 2026). Original GitHub issue: https://github.com/open-webui/mcpo/issues/302 **Summary** A single streamable-HTTP MCP upstream that returns a transient HTTP error (a `502 Bad Gateway` in our case) during an established connection causes mcpo to leave a broken async-generator teardown behind. anyio raises `RuntimeError: Attempted to exit cancel scope in a different task than it was entered in`, and from that point **one CPU core is pinned at ~100% indefinitely** — the process never recovers on its own. Only `podman restart mcpo` (killing the process) clears it. We have observed it run **~14 hours** (== full host uptime) before being noticed via fan/thermal. This is distinct from the existing reports: - open-webui/open-webui #18316 (2+ streamable servers) and #24042 (100% CPU after cancel) — both CLOSED, neither cures this path. - modelcontextprotocol/python-sdk #915 — cancel-scope error when the server is **not available at startup**. Ours is a **runtime** error on an **already-established** connection, not startup. - modelcontextprotocol/python-sdk #2150 — server-side shutdown not terminating sessions. The trigger here is specifically a **transient upstream HTTP error on a live connection**. **Environment** - mcpo `0.0.20` (image `ghcr.io/open-webui/mcpo:main`, digest `sha256:05ccaa3bb0de51eaa9d00a96c4583dde93897c1b45812b34ed3de3b033f06fbf`) — this is the **current `:main` amd64 build** (verified against the live registry manifest 2026-07-15; not a stale local image) - mcp `1.26.0`, anyio `4.12.1`, httpx `0.28.1`, httpx-sse `0.4.3`, uvicorn `0.41.0`, starlette `0.52.1` - Python `3.12.12`, Linux (Podman container, rootless) - config: one `streamable-http` upstream + one stdio upstream (`mcp-server-fetch`) **Config (redacted)** ```json { "mcpServers": { "paperless": { "type": "streamable-http", "url": "https://<UPSTREAM-MCP-HOST>/mcp" }, "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] } } } ``` **Symptom signature** - One process pinned ~100% on a single core, thread state `R`, `wchan=0` — a busy loop, not workload. - Container still reports `healthy`; logs show no tool traffic while spinning. - Never self-recovers; only a process restart clears it. **Actual traceback (from our logs)** The connection was live; the upstream returned `502 Bad Gateway` mid-request. `raise_for_status()` raised inside the streamable-HTTP client's TaskGroup; during teardown `GeneratorExit` propagated through the client's async generator and anyio failed the cancel-scope exit: ``` future: <Task finished name='Task-9' coro=<<async_generator_athrow without __name__>()> exception=RuntimeError('Attempted to exit cancel scope in a different task than it was entered in')> + Exception Group Traceback (most recent call last): | File ".../anyio/_backends/_asyncio.py", line 783, in __aexit__ | raise BaseExceptionGroup( | BaseExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions) +-+---------------- 1 ---------------- | File ".../mcp/client/streamable_http.py", line 565, in handle_request_async | await self._handle_post_request(ctx) | File ".../mcp/client/streamable_http.py", line 358, in _handle_post_request | response.raise_for_status() | httpx.HTTPStatusError: Server error '502 Bad Gateway' for url 'https://<UPSTREAM-MCP-HOST>/mcp' +---------------- 2 ---------------- | File ".../mcp/client/streamable_http.py", line 670, in streamable_http_client | yield ( | File ".../mcp/client/streamable_http.py", line 722, in streamablehttp_client | yield streams | GeneratorExit ``` **Steps to reproduce** 1. Configure mcpo with one `streamable-http` upstream. 2. Keep the connection established, then make the upstream return a transient error (e.g. put a proxy in front that returns `502` for one request, or bounce the upstream so an in-flight request 502s). 3. Observe one core pin at ~100% and stay there; mcpo never recovers without a process restart. **Expected** A transient upstream HTTP error should surface as a normal error for that request; the client should tear down the failed connection cleanly (child task signals; the owning task closes its own cancel scope) and mcpo should keep serving. It should never enter an unrecoverable busy-loop. **Root-cause note (for maintainers)** The failure is the anyio cancel-scope ownership rule being violated during streamable-HTTP teardown: a cancel scope entered by the supervisor task is exited from the child (reader) task's context via a shared stream/generator `aclose()`. anyio can only *detect* this; the fix has to be in the client's connection lifecycle (child signals, parent closes its own scope), and it needs to be robust to the teardown being triggered by an in-flight `HTTPStatusError`, not just clean shutdown. **Workaround we deployed** A host watchdog that detects the sustained single-core pin (Δ CPU jiffies ≈ one full core sustained over several minutes) and restarts mcpo + alerts. Effective but obviously a band-aid.

bsakaguchi commented

2026-10-18 21:12:05 +00:00

Author

Owner

@giuliocapecchi-tuboolar commented on GitHub (Sep 11, 2026):

Same symptom here (mcpo 0.0.20, mcp 1.26.0, anyio 4.12.1), different trigger: a streamable-http backend whose port is closed when mcpo starts. One core at 100% until the container is restarted; we have seen it run 11 hours. Reproduces in about 60 seconds, with and without --hot-reload.

The RuntimeError is the MCP SDK's problem, but the CPU spin itself is anyio, not mcpo: agronholm/anyio#1111, fixed in anyio 4.14.0. CancelScope._deliver_cancellation sets should_retry = True for a task that is already done, task.cancel() is a no-op on it, so the callback reschedules itself via call_soon on every loop iteration. py-spy dump --locals on the spinning process shows exactly that:

_deliver_cancellation (anyio/_backends/_asyncio.py:578)
    should_retry: True   current: None
    task: <_asyncio.Task ...>   waiter: None

Verified fix, same image and same trigger:

FROM ghcr.io/open-webui/mcpo:main
RUN uv pip install --python /app/.venv/bin/python 'anyio>=4.14'

CPU goes from 100% to 0.1% over 5 minutes. The RuntimeError still appears in the logs, and a backend that was down at startup still needs a restart to be picked up, but the process no longer burns a core. An anyio>=4.14 pin in mcpo's dependencies would fix this for everyone.

@giuliocapecchi-tuboolar commented on GitHub (Sep 11, 2026): Same symptom here (mcpo 0.0.20, mcp 1.26.0, anyio 4.12.1), different trigger: a `streamable-http` backend whose port is closed when mcpo starts. One core at 100% until the container is restarted; we have seen it run 11 hours. Reproduces in about 60 seconds, with and without `--hot-reload`. The RuntimeError is the MCP SDK's problem, but the CPU spin itself is **anyio, not mcpo**: [agronholm/anyio#1111](https://github.com/agronholm/anyio/issues/1111), fixed in anyio 4.14.0. `CancelScope._deliver_cancellation` sets `should_retry = True` for a task that is already done, `task.cancel()` is a no-op on it, so the callback reschedules itself via `call_soon` on every loop iteration. `py-spy dump --locals` on the spinning process shows exactly that: ``` _deliver_cancellation (anyio/_backends/_asyncio.py:578) should_retry: True current: None task: <_asyncio.Task ...> waiter: None ``` Verified fix, same image and same trigger: ```dockerfile FROM ghcr.io/open-webui/mcpo:main RUN uv pip install --python /app/.venv/bin/python 'anyio>=4.14' ``` CPU goes from 100% to 0.1% over 5 minutes. The RuntimeError still appears in the logs, and a backend that was down at startup still needs a restart to be picked up, but the process no longer burns a core. An `anyio>=4.14` pin in mcpo's dependencies would fix this for everyone.

bsakaguchi referenced this issue

2026-10-18 21:12:31 +00:00

[PR #135] [MERGED] add sse_read_timeout=None back #184

bsakaguchi referenced this issue

2026-10-18 21:12:33 +00:00

[PR #155] [CLOSED] feat: implement mcp resource #189

bsakaguchi referenced this issue

2026-10-18 21:12:33 +00:00

[PR #161] [MERGED] fix: Enable environment variable inheritance for Smithery cloud services authentication #190