[GH-ISSUE #4661] [Bug]: TENT RDMA path MTU is one value per engine, clamped to the smallest active MTU of any opened NIC, and not matched with the peer #1187

New issue

Open

opened 2026-10-29 21:23:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:44 +00:00

Owner

Originally created by @xiaodouzi666 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4661

Bug Report

What happens

RdmaTransport builds one RdmaParams and shares it with every RdmaContext (src/transport/rdma/rdma_transport.cpp:343, context.cpp:319). When a context is constructed it clamps the shared value to its own port: params_->endpoint.path_mtu = MIN(params_->endpoint.path_mtu, port_attr.active_mtu) (context.cpp:837). So one NIC with a small active MTU lowers the path MTU of every QP on every NIC of that engine; every endpoint later takes params_->path_mtu into RTR (endpoint.cpp:1002, :1197). The value is not exchanged with the peer. (Line numbers as of 342c8aee.)

Two consequences we hit on hosts with a front-end NIC (mlx5_0, overlay netdev, MTU 1500 → active MTU 1024) and four rail NICs (MTU 4096):

With the default device list (all five NICs), every rail QP runs at path MTU 1024 instead of 4096. An engine restricted to the rails (whitelist) runs at 4096.
When the two sides end up with different path MTUs — one side whitelists the rails, the other does not; or MC_MTU=1024 on one side — RDMA READs larger than one MTU fail with IBV_WC_LOC_LEN_ERR ("local length error"), the slice is retried, the rail pair gets paused, and the request either completes over TCP after the transport failover (when TCP is loaded) or fails at the transfer timeout. WRITEs and one-packet READs are unaffected, which makes it look like a size-dependent flake.

Measured (hosts with five NICs each):

Engine A started with MC_MTU=1024, engine B with the default 4096 (two processes on one host, then again across two hosts): the 1 KiB READ passes, a 64 MiB READ fails on RDMA (local length error, rail paused) and the request completes over TCP after the failover; the check took ~20 s instead of well under a second.
One engine with all five NICs (path MTU clamped to 1024 by mlx5_0) against single-NIC engines (4096 each) on the same host: 24 of 25 NIC pairs failed the same way ("only a READ of one KiB works; path MTU 4096 here, 1024 on the target") until the single-NIC side was pinned to 1024 as well, after which every rail pair passed. Same NICs — only the per-engine clamp differed.
Across two hosts, the rail NICs run at 4096 when mlx5_0 is left out of both device lists and at 1024 when it is in.

Why it matters

The clamp silently costs throughput on every rail whenever a low-MTU NIC is in the list (common: the front-end NIC is on an overlay), and a path-MTU mismatch between peers is reported as transfer errors far from the cause.

Possible directions

Keep the path MTU per context (per NIC) instead of one shared value, so the rails are not clamped by the front-end NIC; and either carry each device's path MTU in the segment descriptor (DeviceDesc) so the QP pair takes the minimum of the two, or at least log a WARNING at bootstrap when the peer's published MTU differs. We can prepare a patch if either shape is acceptable.

Reproduction

Two engines on hosts with mixed-MTU NICs; whitelist the rails on one side only; READ 64 MiB. Our #2864 diagnostics tool reports this as rdma.mtu.uniform (RDMA-013) locally and as PEER-004 fail.mtu across hosts.

Environment: commit 342c8aee; Linux x86_64 hosts with mlx5 RoCE v2 NICs: a front-end NIC on an overlay netdev (MTU 1500, active MTU 1024) and four rails at MTU 4096.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @xiaodouzi666 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4661 ### Bug Report **What happens** `RdmaTransport` builds one `RdmaParams` and shares it with every `RdmaContext` (`src/transport/rdma/rdma_transport.cpp:343`, `context.cpp:319`). When a context is constructed it clamps the shared value to its own port: `params_->endpoint.path_mtu = MIN(params_->endpoint.path_mtu, port_attr.active_mtu)` (`context.cpp:837`). So one NIC with a small active MTU lowers the path MTU of every QP on every NIC of that engine; every endpoint later takes `params_->path_mtu` into RTR (`endpoint.cpp:1002`, `:1197`). The value is not exchanged with the peer. (Line numbers as of `342c8aee`.) Two consequences we hit on hosts with a front-end NIC (`mlx5_0`, overlay netdev, MTU 1500 → active MTU 1024) and four rail NICs (MTU 4096): 1. With the default device list (all five NICs), every rail QP runs at path MTU 1024 instead of 4096. An engine restricted to the rails (whitelist) runs at 4096. 2. When the two sides end up with different path MTUs — one side whitelists the rails, the other does not; or `MC_MTU=1024` on one side — RDMA READs larger than one MTU fail with `IBV_WC_LOC_LEN_ERR` ("local length error"), the slice is retried, the rail pair gets paused, and the request either completes over TCP after the transport failover (when TCP is loaded) or fails at the transfer timeout. WRITEs and one-packet READs are unaffected, which makes it look like a size-dependent flake. **Measured** (hosts with five NICs each): - Engine A started with `MC_MTU=1024`, engine B with the default 4096 (two processes on one host, then again across two hosts): the 1 KiB READ passes, a 64 MiB READ fails on RDMA (`local length error`, rail paused) and the request completes over TCP after the failover; the check took ~20 s instead of well under a second. - One engine with all five NICs (path MTU clamped to 1024 by `mlx5_0`) against single-NIC engines (4096 each) on the same host: 24 of 25 NIC pairs failed the same way ("only a READ of one KiB works; path MTU 4096 here, 1024 on the target") until the single-NIC side was pinned to 1024 as well, after which every rail pair passed. Same NICs — only the per-engine clamp differed. - Across two hosts, the rail NICs run at 4096 when `mlx5_0` is left out of both device lists and at 1024 when it is in. **Why it matters** The clamp silently costs throughput on every rail whenever a low-MTU NIC is in the list (common: the front-end NIC is on an overlay), and a path-MTU mismatch between peers is reported as transfer errors far from the cause. **Possible directions** Keep the path MTU per context (per NIC) instead of one shared value, so the rails are not clamped by the front-end NIC; and either carry each device's path MTU in the segment descriptor (`DeviceDesc`) so the QP pair takes the minimum of the two, or at least log a WARNING at bootstrap when the peer's published MTU differs. We can prepare a patch if either shape is acceptable. **Reproduction** Two engines on hosts with mixed-MTU NICs; whitelist the rails on one side only; READ 64 MiB. Our #2864 diagnostics tool reports this as `rdma.mtu.uniform` (RDMA-013) locally and as PEER-004 `fail.mtu` across hosts. **Environment**: commit `342c8aee`; Linux x86_64 hosts with mlx5 RoCE v2 NICs: a front-end NIC on an overlay netdev (MTU 1500, active MTU 1024) and four rails at MTU 4096. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:23:44 +00:00

bsakaguchi commented

2026-10-29 21:23:46 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 14, 2026):

Thanks for opening this issue, @xiaodouzi666!

Field	Value
Issue	#4661
GitHub user ID	`77219630`
Reporter	@xiaodouzi666

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 14, 2026): Thanks for opening this issue, @xiaodouzi666! | Field | Value | |-------|-------| | **Issue** | #4661 | | **GitHub user ID** | `77219630` | | **Reporter** | @xiaodouzi666 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:27 +00:00

[PR #1187] [MERGED] [TE] Support rdma traffic class by environmental variable #2059