kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2960] [RFC]: Enhanced RDMA RNIC Failover and Recovery #764

New issue

Closed

opened 2026-10-29 21:19:15 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:19:15 +00:00

Owner

Originally created by @alogfans on GitHub (Jul 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2960

Changes proposed

Background

Mooncake Transfer Engine supports selecting local and remote RNICs in multi-RDMA-NIC environments through topology and NIC priority matrices. In production, however, a single RNIC may experience link down events, port errors, GID changes, QP errors, or other transient failures.

The current implementation already includes partial recovery mechanisms, such as endpoint reconstruction, GID reprobe, ready ACK, and metadata refresh. However, real RNIC down/up testing still exposed several gaps:

When a preferred local RNIC becomes unavailable, queued or newly submitted slices may not smoothly fail over to another local RNIC.
Peer RNIC failure and local RNIC failure are not clearly separated, so a local hardware issue may be misclassified as a remote rail issue.
After an RNIC transitions from PORT_ERR to PORT_ACTIVE, immediately publishing it as available may be too early. In real RoCE/mlx5 environments, the data path may still reject connections shortly after link-up.
Passive handshake failures do not always return structured error information to the active side, making diagnosis difficult.
Reusing old QPs after a pre-connected endpoint handshake failure may conflict with QP information already cached by the peer.

This RFC proposes improving RDMA RNIC failover recovery so that transfers can continue through fallback RNICs when a single RNIC fails, and recovered RNICs are safely republished.

Goals

Support handing off unfinished slices to another active local RNIC after a local RNIC failure.
Distinguish local RNIC failures from peer rail failures.
Update and publish local topology when a context becomes inactive or recovers.
Delay reactivating a recovered RNIC after IBV_EVENT_PORT_ACTIVE.
Improve handshake failure diagnostics.
Preserve compatibility with existing RDMA ready ACK, GID reprobe, and endpoint lifecycle mechanisms.
Provide a real link down/up validation script.

Non-Goals

Do not introduce a new transport protocol.
Do not change the upper-level Transfer Engine API.
Do not guarantee lossless recovery from every hardware fatal error.
Do not refactor Store or TENT failover mechanisms in this RFC.
Do not change the consistency model of metadata backends.

Design Overview

1. Local RNIC Handoff

When a local RDMA context is marked inactive, or when the worker pool detects a local completion failure, the current worker no longer fails the slice immediately. Instead, it tries to hand the slice off to another active local context.

During handoff, the following remote-side information is preserved:

target_id
peer_nic_path
dest_addr
dest_rkey

Only local send-side information is updated:

source_lkey
rdma.endpoint = nullptr

The slice is then re-enqueued into the fallback context's worker pool.

This avoids treating a local RNIC failure as a remote rail failure and reduces the need to re-resolve remote metadata.

2. Peer Rail Failure

For non-local completion failures or handshake failures, if an alternative peer rail is available, the current peer rail is paused and the slice is redispatched to another peer RNIC.

Peer rail pause state is managed with rail state tracking:

Track error count.
Set pause_until_ns once the threshold is reached.
Automatically make the rail available again after the pause expires.

For explicit handshake or ready ACK timeouts, the peer rail can be paused immediately.

3. Context Inactive State and Topology Publishing

When any of the following async events are received, the RDMA context is marked inactive:

IBV_EVENT_DEVICE_FATAL
IBV_EVENT_CQ_ERR
IBV_EVENT_WQ_FATAL
IBV_EVENT_PORT_ERR
IBV_EVENT_LID_CHANGE

After the context becomes inactive, Transfer Engine refreshes the local topology, disables that RNIC in the topology, and republishes the local segment descriptor.

Once peers refresh metadata, they can avoid selecting the failed RNIC.

4. Delayed Recovery

Receiving IBV_EVENT_PORT_ACTIVE does not necessarily mean the data path is immediately usable. In real RoCE/mlx5 environments, RTR/RTS transitions or path resolution may still fail for a short period after link-up.

The recovery flow therefore does not immediately republish the RNIC as active. Instead:

Receive PORT_ACTIVE.
Set recovery_activate_after_ns.
The monitor worker waits until the recovery deadline.
Refresh the current GID.
If GID refresh succeeds, mark the context active again.
Republish local topology.

The default recovery delay is 30 seconds.

5. Endpoint and QP Lifecycle Improvements

For endpoints that have not yet completed connection setup, handshake failure no longer causes the old QPs to be reset and reused.

The reason is that when the active side sees a failure, the passive side may already have received and cached the old QP numbers. Reusing those QPs could make the next handshake ambiguous with the previous partially processed handshake.

The new behavior is:

For pre-connected endpoint disconnects, rebuild fresh QPs.
For connected endpoint destruction, transition QPs to ERR so outstanding WRs can be flushed.

6. Passive Handshake Error Replies

When the passive handshake callback fails, the server still returns a structured handshake response and fills reply_msg.

This lets the active side observe a concrete rejection reason instead of seeing an empty response or decode failure.

Example errors include:

Invalid peer_nic_path.
Local RDMA context not found.
Endpoint unavailable.
Fresh endpoint unavailable after retiring a stale endpoint.

7. Submit-Path Fallback for Inactive Devices

The submit path may preselect a local device at request granularity. If that selected device later becomes inactive, the request-level selection must not be reused.

The new behavior is:

Reuse the preselected context only if it is still active.
If the preselected context is inactive, fall back to slice-level retry selection.
Slice-level selection skips inactive contexts and chooses a fallback RNIC.

Validation

Unit and Integration Tests

The following scenarios were validated with mlx5_1 through mlx5_4:

Sender primary RNIC inactive, transfer succeeds through fallback local RNIC.
Sender RNIC receives an injected PORT_ERR, while transfers continue for 20 seconds.
Receiver RNIC receives an injected PORT_ERR, while transfers continue for 20 seconds.
Topology selection does not choose empty/disabled devices.
RDMA context GID reprobe behavior remains correct.

Real Link Test

A real RDMA link failover validation script is added:

mooncake-transfer-engine/scripts/real_rdma_link_failover.sh

Default topology:

Sender primary: mlx5_2
Sender fallback: mlx5_1
Receiver primary: mlx5_3
Receiver fallback: mlx5_4

The script starts a target and an initiator, then runs:

ip link set <fault_netdev> down
sleep <down_for_sec>
ip link set <fault_netdev> up

Pass criteria:

The initiator completes successfully.
The primary RNIC is observed as inactive.
The fallback RNIC is not marked inactive.
No data integrity error is detected.

Compatibility

No user-facing API changes.
No required metadata format changes.
Existing ready_ack, GID reprobe, and metadata refresh behavior remains backward compatible.
The new failover behavior only applies to RDMA multi-RNIC/fallback scenarios.
In single-RNIC scenarios, if no handoff target exists, transfers still fail according to the existing retry/failure semantics.

Risks

1. Local Failure Misclassification

Some WC statuses may be caused by either local or remote issues. The implementation only classifies clearly local WC statuses as local failures to avoid disabling a context too aggressively.

2. Slow Recovery

The 30-second recovery delay may cause a recovered RNIC to rejoin later than strictly necessary. This is intentionally conservative and prioritizes stability.

3. Metadata Refresh Delay

Whether peers quickly avoid an inactive RNIC depends on metadata refresh or remote segment descriptor reload timing.

4. Multi-Worker Concurrency

Handoff enqueues slices across worker pools, so slice lifecycle and completion counters must remain correct under concurrency.

Future Work

Make the recovery delay configurable.
Integrate the real link failover test into a hardware CI environment.
Add more WC status classification and vendor-specific handling.
Expose RNIC inactive/recovered metrics.
Extend show-link diagnostics to display inactive topology state.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @alogfans on GitHub (Jul 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2960 ### Changes proposed ## Background Mooncake Transfer Engine supports selecting local and remote RNICs in multi-RDMA-NIC environments through topology and NIC priority matrices. In production, however, a single RNIC may experience link down events, port errors, GID changes, QP errors, or other transient failures. The current implementation already includes partial recovery mechanisms, such as endpoint reconstruction, GID reprobe, ready ACK, and metadata refresh. However, real RNIC down/up testing still exposed several gaps: 1. When a preferred local RNIC becomes unavailable, queued or newly submitted slices may not smoothly fail over to another local RNIC. 2. Peer RNIC failure and local RNIC failure are not clearly separated, so a local hardware issue may be misclassified as a remote rail issue. 3. After an RNIC transitions from `PORT_ERR` to `PORT_ACTIVE`, immediately publishing it as available may be too early. In real RoCE/mlx5 environments, the data path may still reject connections shortly after link-up. 4. Passive handshake failures do not always return structured error information to the active side, making diagnosis difficult. 5. Reusing old QPs after a pre-connected endpoint handshake failure may conflict with QP information already cached by the peer. This RFC proposes improving RDMA RNIC failover recovery so that transfers can continue through fallback RNICs when a single RNIC fails, and recovered RNICs are safely republished. ## Goals - Support handing off unfinished slices to another active local RNIC after a local RNIC failure. - Distinguish local RNIC failures from peer rail failures. - Update and publish local topology when a context becomes inactive or recovers. - Delay reactivating a recovered RNIC after `IBV_EVENT_PORT_ACTIVE`. - Improve handshake failure diagnostics. - Preserve compatibility with existing RDMA ready ACK, GID reprobe, and endpoint lifecycle mechanisms. - Provide a real link down/up validation script. ## Non-Goals - Do not introduce a new transport protocol. - Do not change the upper-level Transfer Engine API. - Do not guarantee lossless recovery from every hardware fatal error. - Do not refactor Store or TENT failover mechanisms in this RFC. - Do not change the consistency model of metadata backends. ## Design Overview ### 1. Local RNIC Handoff When a local RDMA context is marked inactive, or when the worker pool detects a local completion failure, the current worker no longer fails the slice immediately. Instead, it tries to hand the slice off to another active local context. During handoff, the following remote-side information is preserved: - `target_id` - `peer_nic_path` - `dest_addr` - `dest_rkey` Only local send-side information is updated: - `source_lkey` - `rdma.endpoint = nullptr` The slice is then re-enqueued into the fallback context's worker pool. This avoids treating a local RNIC failure as a remote rail failure and reduces the need to re-resolve remote metadata. ### 2. Peer Rail Failure For non-local completion failures or handshake failures, if an alternative peer rail is available, the current peer rail is paused and the slice is redispatched to another peer RNIC. Peer rail pause state is managed with rail state tracking: - Track error count. - Set `pause_until_ns` once the threshold is reached. - Automatically make the rail available again after the pause expires. For explicit handshake or ready ACK timeouts, the peer rail can be paused immediately. ### 3. Context Inactive State and Topology Publishing When any of the following async events are received, the RDMA context is marked inactive: - `IBV_EVENT_DEVICE_FATAL` - `IBV_EVENT_CQ_ERR` - `IBV_EVENT_WQ_FATAL` - `IBV_EVENT_PORT_ERR` - `IBV_EVENT_LID_CHANGE` After the context becomes inactive, Transfer Engine refreshes the local topology, disables that RNIC in the topology, and republishes the local segment descriptor. Once peers refresh metadata, they can avoid selecting the failed RNIC. ### 4. Delayed Recovery Receiving `IBV_EVENT_PORT_ACTIVE` does not necessarily mean the data path is immediately usable. In real RoCE/mlx5 environments, RTR/RTS transitions or path resolution may still fail for a short period after link-up. The recovery flow therefore does not immediately republish the RNIC as active. Instead: 1. Receive `PORT_ACTIVE`. 2. Set `recovery_activate_after_ns`. 3. The monitor worker waits until the recovery deadline. 4. Refresh the current GID. 5. If GID refresh succeeds, mark the context active again. 6. Republish local topology. The default recovery delay is 30 seconds. ### 5. Endpoint and QP Lifecycle Improvements For endpoints that have not yet completed connection setup, handshake failure no longer causes the old QPs to be reset and reused. The reason is that when the active side sees a failure, the passive side may already have received and cached the old QP numbers. Reusing those QPs could make the next handshake ambiguous with the previous partially processed handshake. The new behavior is: - For pre-connected endpoint disconnects, rebuild fresh QPs. - For connected endpoint destruction, transition QPs to `ERR` so outstanding WRs can be flushed. ### 6. Passive Handshake Error Replies When the passive handshake callback fails, the server still returns a structured handshake response and fills `reply_msg`. This lets the active side observe a concrete rejection reason instead of seeing an empty response or decode failure. Example errors include: - Invalid `peer_nic_path`. - Local RDMA context not found. - Endpoint unavailable. - Fresh endpoint unavailable after retiring a stale endpoint. ### 7. Submit-Path Fallback for Inactive Devices The submit path may preselect a local device at request granularity. If that selected device later becomes inactive, the request-level selection must not be reused. The new behavior is: - Reuse the preselected context only if it is still active. - If the preselected context is inactive, fall back to slice-level retry selection. - Slice-level selection skips inactive contexts and chooses a fallback RNIC. ## Validation ### Unit and Integration Tests The following scenarios were validated with `mlx5_1` through `mlx5_4`: - Sender primary RNIC inactive, transfer succeeds through fallback local RNIC. - Sender RNIC receives an injected `PORT_ERR`, while transfers continue for 20 seconds. - Receiver RNIC receives an injected `PORT_ERR`, while transfers continue for 20 seconds. - Topology selection does not choose empty/disabled devices. - RDMA context GID reprobe behavior remains correct. ### Real Link Test A real RDMA link failover validation script is added: ```bash mooncake-transfer-engine/scripts/real_rdma_link_failover.sh ``` Default topology: - Sender primary: `mlx5_2` - Sender fallback: `mlx5_1` - Receiver primary: `mlx5_3` - Receiver fallback: `mlx5_4` The script starts a target and an initiator, then runs: ```bash ip link set <fault_netdev> down sleep <down_for_sec> ip link set <fault_netdev> up ``` Pass criteria: - The initiator completes successfully. - The primary RNIC is observed as inactive. - The fallback RNIC is not marked inactive. - No data integrity error is detected. ## Compatibility - No user-facing API changes. - No required metadata format changes. - Existing `ready_ack`, GID reprobe, and metadata refresh behavior remains backward compatible. - The new failover behavior only applies to RDMA multi-RNIC/fallback scenarios. - In single-RNIC scenarios, if no handoff target exists, transfers still fail according to the existing retry/failure semantics. ## Risks ### 1. Local Failure Misclassification Some WC statuses may be caused by either local or remote issues. The implementation only classifies clearly local WC statuses as local failures to avoid disabling a context too aggressively. ### 2. Slow Recovery The 30-second recovery delay may cause a recovered RNIC to rejoin later than strictly necessary. This is intentionally conservative and prioritizes stability. ### 3. Metadata Refresh Delay Whether peers quickly avoid an inactive RNIC depends on metadata refresh or remote segment descriptor reload timing. ### 4. Multi-Worker Concurrency Handoff enqueues slices across worker pools, so slice lifecycle and completion counters must remain correct under concurrency. ## Future Work - Make the recovery delay configurable. - Integrate the real link failover test into a hardware CI environment. - Add more WC status classification and vendor-specific handling. - Expose RNIC inactive/recovered metrics. - Extend show-link diagnostics to display inactive topology state. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:19:15 +00:00

closed this issue
added the
Transfer Engine

RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Thanks for opening this issue, @alogfans!

Field	Value
Issue	#2960
GitHub user ID	`6733552`
Reporter	@alogfans

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Thanks for opening this issue, @alogfans! | Field | Value | |-------|-------| | **Issue** | #2960 | | **GitHub user ID** | `6733552` | | **Reporter** | @alogfans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 15, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 15, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:16 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 22, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 22, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:30 +00:00

[PR #764] [MERGED] [TE] Fix adxl error code in ascend-direct-transport #1807