[GH-ISSUE #2145] [Bug]: TCP connection pool causes SIGSEGV in consumer process (unfixed root cause of #1267) #589

New issue

Closed

opened 2026-10-29 21:16:54 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:16:54 +00:00

Owner

Originally created by @mustafayildirim on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2145

Originally assigned to: @alogfans on GitHub.

Bug Report

Hi @alogfans — filing a fresh report to add reproduction + production-impact data on top of #1267 (CLOSED 2026-03-03). #1267 was closed by the cosmetic static_cast<const void*>(addr) logging change, but its underlying use-after-free in ClientSession::writeBody is unfixed and reproduces deterministically on 0.3.10.post2.

Environment

Mooncake version: mooncake-transfer-engine-non-cuda==0.3.10.post2 (pip wheel)
Caller: vllm==0.21.0 with the built-in MooncakeStoreConnector (vLLM PR #40900) — kv_role: kv_both, kv_connector: MooncakeStoreConnector
Transport: protocol: tcp
Pool: MC_TCP_ENABLE_CONNECTION_POOL=1 (the env added in #1508)
Cluster: Kubernetes, 2 vLLM client pods + 3 Mooncake Store replicas, all on a private CNI overlay (10.151.0.0/16). Mooncake master + etcd at cluster DNS names.

Crash signature (matches #1267 byte-for-byte)

E tcp_transport.cpp:406] ClientSession::writeBody failed.
  Attempt to write data 0x7f2390cbc000 using buffer 0x7f2390cbc000. Error: Bad address (value: 14)
E tcp_transport.cpp:406] ClientSession::writeBody failed.
  Attempt to write data 0x7f2437f68000 using buffer 0x7f2437f68000. Error: Bad address (value: 14)
... (many more, increasing in rate)
E transfer_task.cpp:281] Batch ... completed with task failures: task_ids=[74, 75, 76, 77]
!!!!!!! Segfault encountered !!!!!!!
  File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd

Note data pointer == buffer pointer in every log line — consistent with the upper layer freeing the buffer before asio's async_write completes (matches @ykwd's diagnosis in #1267).

What's new vs #1267

1. Production impact is worse than #1267 documented

Because the Mooncake client is loaded as a Python extension inside the vLLM worker process, the SIGSEGV doesn't just lose KV cache data — it kills the engine core entirely. vLLM's output_handler propagates the death as EngineDeadError:

ERROR [serving.py:997]   File ".../vllm/v1/engine/async_llm.py", line 660, in output_handler
ERROR [serving.py:997]     outputs = await engine_core.get_output_async()
ERROR [serving.py:997]   File ".../vllm/v1/engine/core_client.py", line 998, in get_output_async
ERROR [serving.py:997]     raise self._format_exception(outputs) from None
ERROR [serving.py:997] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.
(APIServer pid=1) INFO:     ... - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error

Every concurrent HTTP streaming request fails with 500, not just the in-flight Mooncake put. The pod recycles (~2-3 minutes) before the API server is reachable again.

2. We can reproduce on demand

Sustained synthetic chat traffic (Mistral 24B, fp8 KV cache, ~250-token inputs, max_tokens=20) reliably triggers the segfault within minutes. Both client pods in our test crashed; the second crashed within the next 5-minute monitoring window after the first.

What we'd like

Reopen #1267 (or this) as a tracked bug. The current CLOSED state on #1267 makes operators believe the issue is resolved.
A doc warning on MC_TCP_ENABLE_CONNECTION_POOL=1 in troubleshooting.md explicitly noting it can cause SIGSEGV in the consumer process. The current text discusses fd leaks + a circular-wait hang hypothesis but does not mention the actual UAF that crashes the process.
Confirmation on the roadmap: is the fix expected to land via #1633 (tcp_hp transport) or as a targeted patch to the existing transport? Knowing the plan helps operators pin to a version once it's available.

Happy to collect more data (gdb dump, asan build) if helpful — let me know what would be most useful.

Cross-referenced upstream issues / PRs:

#1267 — original writeBody panic report (CLOSED, root cause unfixed)
#519 — same Bad-address symptom (CLOSED)
#1071 — related receiver-side race condition
#1508 — the connection-pool PR ("default disabled for stability")
#1633 — tcp_hp transport replacement (OPEN)
#1954 — troubleshooting docs for TCP exhaustion

Originally created by @mustafayildirim on GitHub (May 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2145 Originally assigned to: @alogfans on GitHub. ### Bug Report Hi @alogfans — filing a fresh report to add reproduction + production-impact data on top of [#1267](https://github.com/kvcache-ai/Mooncake/issues/1267) (CLOSED 2026-03-03). #1267 was closed by the cosmetic `static_cast<const void*>(addr)` logging change, but its underlying use-after-free in `ClientSession::writeBody` is **unfixed** and reproduces deterministically on `0.3.10.post2`. #### Environment - **Mooncake version**: `mooncake-transfer-engine-non-cuda==0.3.10.post2` (pip wheel) - **Caller**: `vllm==0.21.0` with the built-in `MooncakeStoreConnector` (vLLM PR #40900) — `kv_role: kv_both`, `kv_connector: MooncakeStoreConnector` - **Transport**: `protocol: tcp` - **Pool**: `MC_TCP_ENABLE_CONNECTION_POOL=1` (the env added in #1508) - **Cluster**: Kubernetes, 2 vLLM client pods + 3 Mooncake Store replicas, all on a private CNI overlay (`10.151.0.0/16`). Mooncake master + etcd at cluster DNS names. #### Crash signature (matches #1267 byte-for-byte) ``` E tcp_transport.cpp:406] ClientSession::writeBody failed. Attempt to write data 0x7f2390cbc000 using buffer 0x7f2390cbc000. Error: Bad address (value: 14) E tcp_transport.cpp:406] ClientSession::writeBody failed. Attempt to write data 0x7f2437f68000 using buffer 0x7f2437f68000. Error: Bad address (value: 14) ... (many more, increasing in rate) E transfer_task.cpp:281] Batch ... completed with task failures: task_ids=[74, 75, 76, 77] !!!!!!! Segfault encountered !!!!!!! File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd ``` Note `data` pointer == `buffer` pointer in every log line — consistent with the upper layer freeing the buffer before asio's `async_write` completes (matches @ykwd's diagnosis in #1267). #### What's new vs #1267 ##### 1. Production impact is worse than #1267 documented Because the Mooncake client is loaded as a Python extension inside the vLLM worker process, the SIGSEGV doesn't just lose KV cache data — it kills the engine core entirely. vLLM's `output_handler` propagates the death as `EngineDeadError`: ``` ERROR [serving.py:997] File ".../vllm/v1/engine/async_llm.py", line 660, in output_handler ERROR [serving.py:997] outputs = await engine_core.get_output_async() ERROR [serving.py:997] File ".../vllm/v1/engine/core_client.py", line 998, in get_output_async ERROR [serving.py:997] raise self._format_exception(outputs) from None ERROR [serving.py:997] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. (APIServer pid=1) INFO: ... - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error ``` Every concurrent HTTP streaming request fails with **500**, not just the in-flight Mooncake put. The pod recycles (~2-3 minutes) before the API server is reachable again. ##### 2. We can reproduce on demand Sustained synthetic chat traffic (Mistral 24B, fp8 KV cache, ~250-token inputs, max_tokens=20) reliably triggers the segfault within minutes. Both client pods in our test crashed; the second crashed within the next 5-minute monitoring window after the first. #### What we'd like 1. **Reopen #1267 (or this) as a tracked bug.** The current CLOSED state on #1267 makes operators believe the issue is resolved. 2. **A doc warning on `MC_TCP_ENABLE_CONNECTION_POOL=1`** in [troubleshooting.md](https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md) explicitly noting it can cause SIGSEGV in the consumer process. The current text discusses fd leaks + a circular-wait hang hypothesis but does not mention the actual UAF that crashes the process. 3. **Confirmation on the roadmap**: is the fix expected to land via #1633 (`tcp_hp` transport) or as a targeted patch to the existing transport? Knowing the plan helps operators pin to a version once it's available. Happy to collect more data (gdb dump, asan build) if helpful — let me know what would be most useful. Cross-referenced upstream issues / PRs: - [#1267](https://github.com/kvcache-ai/Mooncake/issues/1267) — original writeBody panic report (CLOSED, root cause unfixed) - [#519](https://github.com/kvcache-ai/Mooncake/issues/519) — same Bad-address symptom (CLOSED) - [#1071](https://github.com/kvcache-ai/Mooncake/issues/1071) — related receiver-side race condition - [#1508](https://github.com/kvcache-ai/Mooncake/pull/1508) — the connection-pool PR ("default disabled for stability") - [#1633](https://github.com/kvcache-ai/Mooncake/pull/1633) — `tcp_hp` transport replacement (OPEN) - [#1954](https://github.com/kvcache-ai/Mooncake/pull/1954) — troubleshooting docs for TCP exhaustion

bsakaguchi closed this issue

2026-10-29 21:16:54 +00:00

bsakaguchi commented

2026-10-29 21:16:54 +00:00

Author

Owner

@alogfans commented on GitHub (May 21, 2026):

Thanks for the error report! We will fix & validate TCP transport to be pointer-safe.

@alogfans commented on GitHub (May 21, 2026): Thanks for the error report! We will fix & validate TCP transport to be pointer-safe.

bsakaguchi commented

2026-10-29 21:16:55 +00:00

Author

Owner

@alogfans commented on GitHub (May 22, 2026):

We develop a PR that postpone socket reuse and buffer free until the last async_write is fully done via boost::post. I think it will be helpful for your environment.

@alogfans commented on GitHub (May 22, 2026): We develop a PR that postpone socket reuse and buffer free until the last async_write is fully done via boost::post. I think it will be helpful for your environment.