[GH-ISSUE #3523] [TENT] RDMA link-flap recovery gaps: hung WRs wedge QPs silently, swallowed bootstrap rejection, no PORT_ACTIVE fallback #902

New issue

Open

opened 2026-10-29 21:20:43 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:43 +00:00

Owner

Originally created by @ai2open on GitHub (Aug 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3523

[TENT] RDMA link-flap recovery gaps: hung WRs wedge QPs silently (no error WC, no software timeout, no reconnect), plus swallowed bootstrap rejection and missing PORT_ACTIVE fallback

Scenario

Prefill/Decode disaggregation (KV cache transfer) over RoCE with the TENT backend
(MC_USE_TENT=1), RDMA NICs are bonded (mlx5_bond_*).

Observed behavior: after PD transfer timeouts, or an RDMA NIC link down/up,
subsequent transfers never recover on their own. The classic (non-TENT) engine does
not show this behavior in the same environment.

Controlled experiment (single-host repro)

ip link set <netdev of mlx5_bond_4> down, wait ~4 min, ip link set up.

Timeline (all from logs):

08:18:06 — link down; all 8 engine processes correctly log Action: mlx5_bond_4 down.
08:19:03-08:19:54 — fallback device selection floods (~780k reroute attempts);
throughput degrades but does not drop to zero (other NICs still serve).
08:19:35+ — application-level 20 s timeouts (batch_transfer_sync_write TIMEOUT,
ret=-1).
08:21:54 — link up; all 8 processes correctly log Action: mlx5_bond_4 up
(so PORT_ACTIVE handling worked in this run).
After link up: partial recovery only. All 4 pp_ranks submitted RDMA writes
(25 each), but pp_rank 1/2 succeeded 25/25 while pp_rank 0/3 failed 25/25 —
and stayed broken until the end of the log (16+ min later).

The crucial part: for the failing ranks after link-up, the TENT layer logged
absolutely nothing:

0 × Detected error WQE (no error completions)
0 × transfer timeout (software) (slice-level software timeout never fired)
0 × Unable to connect endpoint (no reconnect attempts/failures)
0 × Endpoint marked for destruction

Slices were submitted at the app level and then vanished: never completed, never
errored, never reported by any TENT diagnostic path.

Root cause analysis

Defect A (primary, confirmed by experiment): hung WRs during link-down wedge the QP/endpoint forever

WRs posted to QPs on the downed NIC hang silently: while the port is down the
hardware neither transmits nor completes/errors them. After the link comes back,
those WRs still never complete and never produce an error CQE, so:

The QP's send-depth accounting (wr_depth / cq_outstanding) stays consumed
forever;
New slices posted to that endpoint either cannot be posted (QP full) or queue
behind dead WRs and are never sent;
Because these slices never reach the posted/completed/error paths, none of
TENT's detection hooks fire: no error WC (so no disableEndpoint), and the
software slice timeout never triggers for them, so the endpoint is never
destroyed and rebuilt;
The endpoint becomes a permanent blackhole. In our experiment the two ranks
whose paths crossed the downed NIC never recovered (16+ min, until shutdown),
while ranks on other NICs recovered immediately.

There is currently no mechanism in TENT that reclaims an endpoint with
outstanding WRs after a link recovery (e.g. destroying endpoints with stale
outstanding WRs on PORT_ACTIVE), and no visibility into stuck posted WRs
(the classic engine gained exactly this diagnostic in #2872 via
MC_TRACK_RDMA_POSTED_SLICES; TENT has nothing equivalent).

Defect B: bootstrap RPC swallows the peer's rejection reason → misleading "Missing peer GID in bootstrap"

ControlService::onBootstrapRdma (tent/src/runtime/control_plane.cpp) ignores the
return value of bootstrap_callback_. When the peer rejects the bootstrap (stale
endpoint retired, device down, ...), the real reason goes into
local_desc.reply_msg, but the RPC still returns success with an empty local_gid,
and the initiator reports the misleading Missing peer GID in bootstrap
(tent/src/transport/rdma/endpoint.cpp). The classic engine fixed the equivalent
issue in #2959 (structured handshake rejection replies); TENT still has it at
v0.3.12.

Defect C (latent, not triggered in this run): `PORT_ACTIVE` can be lost → context stuck in DEVICE_PAUSED forever

The async fd is registered with EPOLLET (tent/src/transport/rdma/context.cpp),
but Workers::monitorThread does epoll_wait(..., maxevents=1, 100) and
handleContextEvents reads exactly one ibv_async_event per wakeup
(tent/src/transport/rdma/workers.cpp). Under edge-triggered semantics, when
multiple events queue (e.g. a fast flap), PORT_ACTIVE may never be processed.
resume() has no other caller, and there is no periodic ibv_query_port
validation, so a context can stay DEVICE_PAUSED forever (making
Workers::getEndpoint fail all transfers on that local NIC permanently). The
classic engine is saved from this by its monitor worker unconditionally
re-activating contexts every second; TENT has no such fallback. In our controlled
experiment (down ~4 min, single events far apart) this race did not trigger, but
it remains a latent risk for fast flaps.

Defect D: GID/LID cached at init and never refreshed

RdmaContext reads gid_/lid_ once in openDevice() and always advertises these
cached values in bootstrap. If the GID changes after a flap while the process keeps
running, peers build AH entries from stale address vectors. The classic engine fixed
this in #2878 (refresh on HCA/GID change events); TENT still caches forever at
v0.3.12.

#2872 — RDMA rail-failure handling + CQ timeout diagnostics (MC_TRACK_RDMA_POSTED_SLICES)
#2959 — RDMA NIC failover recovery: local-RNIC handoff, MC_RDMA_RAIL_PAUSE_SECONDS,
structured handshake rejection replies
#2878 — refresh RDMA metadata on HCA and GID change events
#2941 — pause active reconnects to failed peers

Verified at tag v0.3.12: tent/src/runtime/control_plane.cpp and
tent/src/transport/rdma/workers.cpp still contain defects A–D.

Suggested fixes for TENT

On PORT_ACTIVE (context resume), proactively destroy endpoints that still have
outstanding WRs instead of waiting forever for CQEs that will never arrive;
more generally, give an endpoint with stuck outstanding WRs a deadline after
which it is torn down and rebuilt.
Port the #2872-style posted-slice tracking / CQ-timeout diagnostics to TENT so
"slice posted but never completed" is visible.
Extend the software slice timeout to cover slices sitting in the per-worker
request queues waiting to be posted, not only posted inflight slices.
Propagate bootstrap failures through the control RPC (carry reply_msg or an
error status back to the initiator).
Drain the async event fd until ibv_get_async_event returns empty on each epoll
wakeup, and add a periodic ibv_query_port-based resume fallback.
Refresh the cached GID/LID on port/GID change events, like #2878 does for the
classic engine.

Happy to provide full logs or help test patches.

Originally created by @ai2open on GitHub (Aug 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3523 # [TENT] RDMA link-flap recovery gaps: hung WRs wedge QPs silently (no error WC, no software timeout, no reconnect), plus swallowed bootstrap rejection and missing PORT_ACTIVE fallback ## Scenario Prefill/Decode disaggregation (KV cache transfer) over RoCE with the TENT backend (`MC_USE_TENT=1`), RDMA NICs are bonded (`mlx5_bond_*`). **Observed behavior:** after PD transfer timeouts, or an RDMA NIC link down/up, subsequent transfers never recover on their own. The classic (non-TENT) engine does not show this behavior in the same environment. ## Controlled experiment (single-host repro) `ip link set <netdev of mlx5_bond_4> down`, wait ~4 min, `ip link set up`. Timeline (all from logs): - `08:18:06` — link down; all 8 engine processes correctly log `Action: mlx5_bond_4 down`. - `08:19:03-08:19:54` — `fallback device selection` floods (~780k reroute attempts); throughput degrades but does not drop to zero (other NICs still serve). - `08:19:35+` — application-level 20 s timeouts (`batch_transfer_sync_write TIMEOUT`, ret=-1). - `08:21:54` — link up; all 8 processes correctly log `Action: mlx5_bond_4 up` (so PORT_ACTIVE handling worked in this run). - After link up: **partial recovery only**. All 4 pp_ranks submitted RDMA writes (25 each), but pp_rank 1/2 succeeded 25/25 while **pp_rank 0/3 failed 25/25** — and stayed broken until the end of the log (16+ min later). The crucial part: for the failing ranks after link-up, the TENT layer logged **absolutely nothing**: - 0 × `Detected error WQE` (no error completions) - 0 × `transfer timeout (software)` (slice-level software timeout never fired) - 0 × `Unable to connect endpoint` (no reconnect attempts/failures) - 0 × `Endpoint marked for destruction` Slices were submitted at the app level and then vanished: never completed, never errored, never reported by any TENT diagnostic path. ## Root cause analysis ### Defect A (primary, confirmed by experiment): hung WRs during link-down wedge the QP/endpoint forever WRs posted to QPs on the downed NIC hang silently: while the port is down the hardware neither transmits nor completes/errors them. After the link comes back, those WRs **still never complete and never produce an error CQE**, so: 1. The QP's send-depth accounting (`wr_depth` / `cq_outstanding`) stays consumed forever; 2. New slices posted to that endpoint either cannot be posted (QP full) or queue behind dead WRs and are never sent; 3. Because these slices never reach the posted/completed/error paths, **none of TENT's detection hooks fire**: no error WC (so no `disableEndpoint`), and the software slice timeout never triggers for them, so the endpoint is never destroyed and rebuilt; 4. The endpoint becomes a permanent blackhole. In our experiment the two ranks whose paths crossed the downed NIC never recovered (16+ min, until shutdown), while ranks on other NICs recovered immediately. There is currently **no mechanism** in TENT that reclaims an endpoint with outstanding WRs after a link recovery (e.g. destroying endpoints with stale outstanding WRs on `PORT_ACTIVE`), and no visibility into stuck posted WRs (the classic engine gained exactly this diagnostic in #2872 via `MC_TRACK_RDMA_POSTED_SLICES`; TENT has nothing equivalent). ### Defect B: bootstrap RPC swallows the peer's rejection reason → misleading "Missing peer GID in bootstrap" `ControlService::onBootstrapRdma` (`tent/src/runtime/control_plane.cpp`) ignores the return value of `bootstrap_callback_`. When the peer rejects the bootstrap (stale endpoint retired, device down, ...), the real reason goes into `local_desc.reply_msg`, but the RPC still returns success with an empty `local_gid`, and the initiator reports the misleading `Missing peer GID in bootstrap` (`tent/src/transport/rdma/endpoint.cpp`). The classic engine fixed the equivalent issue in #2959 (structured handshake rejection replies); TENT still has it at v0.3.12. ### Defect C (latent, not triggered in this run): `PORT_ACTIVE` can be lost → context stuck in DEVICE_PAUSED forever The async fd is registered with `EPOLLET` (`tent/src/transport/rdma/context.cpp`), but `Workers::monitorThread` does `epoll_wait(..., maxevents=1, 100)` and `handleContextEvents` reads exactly one `ibv_async_event` per wakeup (`tent/src/transport/rdma/workers.cpp`). Under edge-triggered semantics, when multiple events queue (e.g. a fast flap), `PORT_ACTIVE` may never be processed. `resume()` has no other caller, and there is no periodic `ibv_query_port` validation, so a context can stay `DEVICE_PAUSED` forever (making `Workers::getEndpoint` fail all transfers on that local NIC permanently). The classic engine is saved from this by its monitor worker unconditionally re-activating contexts every second; TENT has no such fallback. In our controlled experiment (down ~4 min, single events far apart) this race did not trigger, but it remains a latent risk for fast flaps. ### Defect D: GID/LID cached at init and never refreshed `RdmaContext` reads `gid_`/`lid_` once in `openDevice()` and always advertises these cached values in bootstrap. If the GID changes after a flap while the process keeps running, peers build AH entries from stale address vectors. The classic engine fixed this in #2878 (refresh on HCA/GID change events); TENT still caches forever at v0.3.12. ## Related classic-engine fixes (none of them touch TENT) - #2872 — RDMA rail-failure handling + CQ timeout diagnostics (`MC_TRACK_RDMA_POSTED_SLICES`) - #2959 — RDMA NIC failover recovery: local-RNIC handoff, `MC_RDMA_RAIL_PAUSE_SECONDS`, structured handshake rejection replies - #2878 — refresh RDMA metadata on HCA and GID change events - #2941 — pause active reconnects to failed peers Verified at tag v0.3.12: `tent/src/runtime/control_plane.cpp` and `tent/src/transport/rdma/workers.cpp` still contain defects A–D. ## Suggested fixes for TENT 1. On `PORT_ACTIVE` (context resume), proactively destroy endpoints that still have outstanding WRs instead of waiting forever for CQEs that will never arrive; more generally, give an endpoint with stuck outstanding WRs a deadline after which it is torn down and rebuilt. 2. Port the #2872-style posted-slice tracking / CQ-timeout diagnostics to TENT so "slice posted but never completed" is visible. 3. Extend the software slice timeout to cover slices sitting in the per-worker request queues waiting to be posted, not only posted inflight slices. 4. Propagate bootstrap failures through the control RPC (carry `reply_msg` or an error status back to the initiator). 5. Drain the async event fd until `ibv_get_async_event` returns empty on each epoll wakeup, and add a periodic `ibv_query_port`-based resume fallback. 6. Refresh the cached GID/LID on port/GID change events, like #2878 does for the classic engine. Happy to provide full logs or help test patches.

bsakaguchi commented

2026-10-29 21:20:44 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 19, 2026):

Thanks for opening this issue, @ai2open!

Field	Value
Issue	#3523
GitHub user ID	`226425123`
Reporter	@ai2open

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 19, 2026): Thanks for opening this issue, @ai2open! | Field | Value | |-------|-------| | **Issue** | #3523 | | **GitHub user ID** | `226425123` | | **Reporter** | @ai2open | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:22:23 +00:00

[GH-ISSUE #4207] [RFC]: Export process and allocator memory metrics, and complete the jemalloc build option #1056

bsakaguchi referenced this issue

2026-10-29 21:31:10 +00:00

[PR #902] [MERGED] [Store]: Add option to use jemalloc in mooncake store master #1893