[GH-ISSUE #4662] [Bug]: TENT remote NIC selection can pair a rail NIC with a peer NIC it cannot reach; each such pair costs an RC retry timeout before the rail is paused #1188

New issue

Open

opened 2026-10-29 21:23:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:44 +00:00

Owner

Originally created by @xiaodouzi666 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4662

Bug Report

What happens

Workers::selectOptimalDevice picks the remote device from the rail mapping (findBestRemoteDevice, by position/NUMA) or from the target's device list at a random start, preferring the same NUMA node (src/transport/rdma/workers.cpp:1540-1590, as of 342c8aee). Nothing in the selection asks whether the local and the remote NIC are on networks that reach each other. On hosts with a front-end NIC (mlx5_0, overlay network) and four rail NICs (separate rail subnets), with all five NICs enabled on both sides, the selector pairs a rail NIC with the peer's mlx5_0: the QP's RC retries exhaust after ~3.7 s (IBV_WC_RETRY_EXC_ERR), the rail monitor pauses that pair, and the slice is retried on another pair. Requests that cannot complete within the 10 s budget fail. The rail monitor state is per worker, so each worker pays the timeout once for the same unreachable pair.

Measured (two hosts, five NICs each: mlx5_0 on the management network, mlx5_1-4 on four rail subnets with policy routing between them): a 5×5 NIC-pair matrix, each cell an engine restricted to that one pair, agrees with ib_write_bw in both directions on all 25 pairs — the 16 rail↔rail pairs work (cross pairs included), mlx5_0↔mlx5_0 works over the management network, and the 8 pairs between mlx5_0 and a rail NIC do not (transport retry counter exceeded, then Rail paused). With automatic selection and all five NICs enabled on both sides, the same test-run session spent an extra ~18 s (A→B) and ~33 s (B→A) on those pairs before its transfers completed, and a transfer with a 10 s deadline fails outright. On a single host the position-based mapping likewise pairs a rail NIC with the local mlx5_0.

Why it matters

Default deployments usually leave all NICs enabled; the cost shows up as multi-second stalls and timeouts right after start, and the only trace is a Rail paused: local_nic=<id> remote_nic=<id> line after the retry timeout, with NIC indices rather than names. Users work around it with MC_TE_FILTERS or a rail file, but only after finding out.

Possible directions

When choosing the remote NIC, prefer candidates whose published GID (IPv4-mapped RoCE v2 GIDs carry the subnet) is in the same subnet as, or routable from, the local NIC's address — the descriptor already carries every device's GID; or let the rail file express "no path" pairs; and share an unreachable-pair verdict across workers so the timeout is paid once per pair rather than once per worker. We can prepare a patch if one of these is acceptable.

Reproduction

Two hosts with a front-end NIC on a different network than the rail NICs; all NICs enabled; start transfers and watch "Rail paused: local_nic=… remote_nic=…" after ~3.7 s. Our #2864 diagnostics tool shows the matrix with tent diagnostics test-run --rail-matrix.

Environment: commit 342c8aee; Linux x86_64 hosts with mlx5 RoCE v2 NICs: a front-end NIC on the management network and four rail NICs on separate subnets with policy routing.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @xiaodouzi666 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4662 ### Bug Report **What happens** `Workers::selectOptimalDevice` picks the remote device from the rail mapping (`findBestRemoteDevice`, by position/NUMA) or from the target's device list at a random start, preferring the same NUMA node (`src/transport/rdma/workers.cpp:1540-1590`, as of `342c8aee`). Nothing in the selection asks whether the local and the remote NIC are on networks that reach each other. On hosts with a front-end NIC (`mlx5_0`, overlay network) and four rail NICs (separate rail subnets), with all five NICs enabled on both sides, the selector pairs a rail NIC with the peer's `mlx5_0`: the QP's RC retries exhaust after ~3.7 s (`IBV_WC_RETRY_EXC_ERR`), the rail monitor pauses that pair, and the slice is retried on another pair. Requests that cannot complete within the 10 s budget fail. The rail monitor state is per worker, so each worker pays the timeout once for the same unreachable pair. **Measured** (two hosts, five NICs each: `mlx5_0` on the management network, `mlx5_1-4` on four rail subnets with policy routing between them): a 5×5 NIC-pair matrix, each cell an engine restricted to that one pair, agrees with `ib_write_bw` in both directions on all 25 pairs — the 16 rail↔rail pairs work (cross pairs included), `mlx5_0`↔`mlx5_0` works over the management network, and the 8 pairs between `mlx5_0` and a rail NIC do not (`transport retry counter exceeded`, then `Rail paused`). With automatic selection and all five NICs enabled on both sides, the same `test-run` session spent an extra ~18 s (A→B) and ~33 s (B→A) on those pairs before its transfers completed, and a transfer with a 10 s deadline fails outright. On a single host the position-based mapping likewise pairs a rail NIC with the local `mlx5_0`. **Why it matters** Default deployments usually leave all NICs enabled; the cost shows up as multi-second stalls and timeouts right after start, and the only trace is a `Rail paused: local_nic=<id> remote_nic=<id>` line after the retry timeout, with NIC indices rather than names. Users work around it with `MC_TE_FILTERS` or a rail file, but only after finding out. **Possible directions** When choosing the remote NIC, prefer candidates whose published GID (IPv4-mapped RoCE v2 GIDs carry the subnet) is in the same subnet as, or routable from, the local NIC's address — the descriptor already carries every device's GID; or let the rail file express "no path" pairs; and share an unreachable-pair verdict across workers so the timeout is paid once per pair rather than once per worker. We can prepare a patch if one of these is acceptable. **Reproduction** Two hosts with a front-end NIC on a different network than the rail NICs; all NICs enabled; start transfers and watch "Rail paused: local_nic=… remote_nic=…" after ~3.7 s. Our #2864 diagnostics tool shows the matrix with `tent diagnostics test-run --rail-matrix`. **Environment**: commit `342c8aee`; Linux x86_64 hosts with mlx5 RoCE v2 NICs: a front-end NIC on the management network and four rail NICs on separate subnets with policy routing. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:44 +00:00

bsakaguchi commented

2026-10-29 21:23:46 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 14, 2026):

Thanks for opening this issue, @xiaodouzi666!

Field	Value
Issue	#4662
GitHub user ID	`77219630`
Reporter	@xiaodouzi666

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 14, 2026): Thanks for opening this issue, @xiaodouzi666! | Field | Value | |-------|-------| | **Issue** | #4662 | | **GitHub user ID** | `77219630` | | **Reporter** | @xiaodouzi666 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:29 +00:00

[PR #1188] [MERGED] [Doc] add docs of Mooncake EP integration with SGLang #2060