[GH-ISSUE #3637] [tent] BoundedMPSCQueue push() livelock: full submit queue spins both producers and RDMA worker threads forever, engine never recovers #928

New issue

Closed

opened 2026-10-29 21:21:00 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:21:00 +00:00

Owner

Originally created by @ai2open on GitHub (Aug 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3637

Summary

In the TENT RDMA transport, BoundedMPSCQueue::push() spins with std::this_thread::yield() when the queue is full — it has no failure or blocking exit. Once a per-worker submit queue (capacity 8192) fills up during a sustained transfer-failure period, the RDMA worker thread itself blocks in push() while re-enqueueing retried slices, stops popping, and every producer thread then spins in push() as well. The whole transfer engine livelocks: no WRs are posted, no logs are emitted, and the engine cannot recover even after the underlying RDMA problem clears — only a process restart unwedges it.

This is very likely the mechanism behind the submit blocked ... pending in the thousands that never drains (wr_count=0) symptom observed in #3596.

Environment

mooncake-transfer-engine (TENT, MC_USE_TENT=1), vLLM P/D disaggregation, KV blocks striped across 8 RoCE rails via batch_transfer_sync_write
Observed on v0.3.11.post1 + backports of #3188 / #3598 / #3604; the relevant code is identical on current main (3b5a5941)

Observed behavior

After an RDMA link flap (handled correctly by the #3604 evictAll path), a later stall began without any link/verbs event. From that point:

KV pushes from prefill to decode never completed; every request failed with a 90s TTFT timeout.
TENT logging went completely silent (no fallback, no retry, no connect messages) — the worker threads never reach any log point again.
The engine never recovered by itself.

gdb evidence

Attaching to a vLLM prefill worker process shows 86 threads spinning in sched_yield inside BoundedMPSCQueue<RdmaSliceList, 8192>::push, including all 6 TENT worker threads:

#0  __GI_sched_yield
#3  mooncake::tent::BoundedMPSCQueue<mooncake::tent::RdmaSliceList, 8192ul>::push(...)
#4  mooncake::tent::Workers::submit(mooncake::tent::RdmaSliceList&, int)
#5  mooncake::tent::Workers::submit(mooncake::tent::RdmaSlice*)
#6  mooncake::tent::Workers::asyncPostSend()
#7  mooncake::tent::Workers::workerThread(int)

The remaining ~80 threads are KV-connector producer threads also stuck in Workers::submit() → push(). The CQ polling thread stays alive but has nothing to reap (no WRs are ever posted).

Root cause

push() is spin-only when the queue is full (bounded_mpsc_queue.h): there is no try_push, no spin budget, no error return.
The consumer re-enqueues into the same bounded queue it drains. On current main the worker thread calls submit(slice) from at least three retry paths:
- asyncPostSend() when getEndpoint() fails (~workers.cpp:464),
- partially posted slices in the submit path (~workers.cpp:532),
- WC-error retry in the poll/completion path (~workers.cpp:709).
Under any sustained failure (dead QPs, unreachable peer), the retry rate exceeds the completion rate, the queue fills to 8192, and then:
- the worker thread spins in push() while re-queueing a retried slice → it never pops again,
- producers spin in push() → no new slice is ever accepted,
- no slice is failed, nothing is logged → permanent silent livelock.

A bounded queue whose consumer can block on enqueue is a work-conservation violation: one full queue wedges the entire engine.

Suggested fixes

Give push() a failure exit (e.g. try_push() returning bool, or a bounded spin budget) and propagate it: Workers::submit() failure → updateSliceStatus(slice, FAILED) so transfers fail fast and surface errors instead of wedging.
Never let the worker thread block on enqueue: if a retry cannot be re-queued, fail the slice immediately (or move retries to an unbounded overflow list drained with priority).
Add queue-depth / push-failure metrics and at least a rate-limited warning when the queue is full — today a full queue is completely silent.

#3596 — the submit blocked, pending never drains symptom reported there matches this mechanism
#3173 / #3188, #3595 / #3598, #3604 — with those fixes deployed, this livelock is the remaining wedge mechanism we hit

Happy to provide full gdb thread dumps, core stacks from both peers, or help validate a patch.

Originally created by @ai2open on GitHub (Aug 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3637 ## Summary In the TENT RDMA transport, `BoundedMPSCQueue::push()` spins with `std::this_thread::yield()` when the queue is full — it has no failure or blocking exit. Once a per-worker submit queue (capacity 8192) fills up during a sustained transfer-failure period, the RDMA **worker thread itself blocks in `push()` while re-enqueueing retried slices**, stops popping, and every producer thread then spins in `push()` as well. The whole transfer engine livelocks: no WRs are posted, no logs are emitted, and the engine **cannot recover even after the underlying RDMA problem clears** — only a process restart unwedges it. This is very likely the mechanism behind the `submit blocked ... pending in the thousands that never drains (wr_count=0)` symptom observed in #3596. ## Environment - `mooncake-transfer-engine` (TENT, `MC_USE_TENT=1`), vLLM P/D disaggregation, KV blocks striped across 8 RoCE rails via `batch_transfer_sync_write` - Observed on v0.3.11.post1 + backports of #3188 / #3598 / #3604; the relevant code is identical on current `main` (3b5a5941) ## Observed behavior After an RDMA link flap (handled correctly by the #3604 `evictAll` path), a later stall began **without any link/verbs event**. From that point: - KV pushes from prefill to decode never completed; every request failed with a 90s TTFT timeout. - TENT logging went completely silent (no fallback, no retry, no connect messages) — the worker threads never reach any log point again. - The engine never recovered by itself. ## gdb evidence Attaching to a vLLM prefill worker process shows **86 threads spinning in `sched_yield` inside `BoundedMPSCQueue<RdmaSliceList, 8192>::push`**, including **all 6 TENT worker threads**: ``` #0 __GI_sched_yield #3 mooncake::tent::BoundedMPSCQueue<mooncake::tent::RdmaSliceList, 8192ul>::push(...) #4 mooncake::tent::Workers::submit(mooncake::tent::RdmaSliceList&, int) #5 mooncake::tent::Workers::submit(mooncake::tent::RdmaSlice*) #6 mooncake::tent::Workers::asyncPostSend() #7 mooncake::tent::Workers::workerThread(int) ``` The remaining ~80 threads are KV-connector producer threads also stuck in `Workers::submit()` → `push()`. The CQ polling thread stays alive but has nothing to reap (no WRs are ever posted). ## Root cause 1. `push()` is spin-only when the queue is full (`bounded_mpsc_queue.h`): there is no `try_push`, no spin budget, no error return. 2. The consumer re-enqueues into the same bounded queue it drains. On current `main` the worker thread calls `submit(slice)` from at least three retry paths: - `asyncPostSend()` when `getEndpoint()` fails (~workers.cpp:464), - partially posted slices in the submit path (~workers.cpp:532), - WC-error retry in the poll/completion path (~workers.cpp:709). 3. Under any sustained failure (dead QPs, unreachable peer), the retry rate exceeds the completion rate, the queue fills to 8192, and then: - the worker thread spins in `push()` while re-queueing a retried slice → it never pops again, - producers spin in `push()` → no new slice is ever accepted, - no slice is failed, nothing is logged → **permanent silent livelock**. A bounded queue whose consumer can block on enqueue is a work-conservation violation: one full queue wedges the entire engine. ## Suggested fixes 1. Give `push()` a failure exit (e.g. `try_push()` returning `bool`, or a bounded spin budget) and propagate it: `Workers::submit()` failure → `updateSliceStatus(slice, FAILED)` so transfers fail fast and surface errors instead of wedging. 2. Never let the worker thread block on enqueue: if a retry cannot be re-queued, fail the slice immediately (or move retries to an unbounded overflow list drained with priority). 3. Add queue-depth / push-failure metrics and at least a rate-limited warning when the queue is full — today a full queue is completely silent. ## Related - #3596 — the `submit blocked, pending never drains` symptom reported there matches this mechanism - #3173 / #3188, #3595 / #3598, #3604 — with those fixes deployed, this livelock is the remaining wedge mechanism we hit Happy to provide full gdb thread dumps, core stacks from both peers, or help validate a patch.

bsakaguchi closed this issue

2026-10-29 21:21:00 +00:00

bsakaguchi commented

2026-10-29 21:21:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 24, 2026):

Thanks for opening this issue, @ai2open!

Field	Value
Issue	#3637
GitHub user ID	`226425123`
Reporter	@ai2open

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 24, 2026): Thanks for opening this issue, @ai2open! | Field | Value | |-------|-------| | **Issue** | #3637 | | **GitHub user ID** | `226425123` | | **Reporter** | @ai2open | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.