[PR #1077] [MERGED] [RDMA] Fix RDMA device selection to prioritize GIDs with network devices #1996

New issue

Closed

opened 2026-10-29 21:31:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:31:59 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1077
Author: @uniqueni
Created: 11/18/2025
Status: ✅ Merged
Merged: 11/21/2025
Merged by: @stmatengss

Base: main ← Head: bugfix_device_gid

📝 Commits (2)

ca566d0 add gid state
ac7506d [RDMA] Add hasNetworkDevice() helper and improve naming

📊 Changes

3 files changed (+75 additions, -28 deletions)

View changed files

📝 mooncake-transfer-engine/include/config.h (+1 -1)
📝 mooncake-transfer-engine/include/transport/rdma_transport/rdma_context.h (+12 -4)
📝 mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp (+62 -23)

📄 Description

Problem Statement

Current Behavior

The existing RDMA device initialization logic selects the first available device with any valid GID (Global Identifier), regardless of whether that GID has an associated network device (ndev). This can result in poor device selection where:

Devices with fallback GIDs (no network device) are chosen
Better devices with production-ready GIDs (with proper network interfaces) are ignored
RDMA performance is degraded due to non-optimal device selection

Example Issue

Real-world scenario from logs:

❌ System selected: mlx5_2, GID index 3 (no ndev - fallback GID)
✅ Better choice:    mlx5_1, GID index 7 (associated with eth1 - production ready)

like issue https://github.com/kvcache-ai/Mooncake/issues/1052

Root Cause

ibv_get_device_list() returns devices in arbitrary system order
Original code accepts the first device with any valid GID type (RoCE v2 or InfiniBand)
No quality assessment performed on GID's network device association
Devices without proper network configuration are not filtered out

Solution

Design Overview

Introduced a GID quality classification system to enable intelligent RDMA device selection based on network device associations.

Changes Made

1. Added GID Quality Enum (`rdma_context.h`)

enum class GidNetworkState {
    GID_WITH_NETWORK = 0,      // Best: GID with network device (e.g., eth1)
    GID_WITHOUT_NETWORK = 1,   // Fallback: GID without ndev
    GID_NOT_FOUND = 2          // Unusable: No suitable GID found
};

2. Enhanced `getBestGidIndex()` Function

Before:

int getBestGidIndex(const std::string &device_name,
                    struct ibv_context *context,
                    ibv_port_attr &port_attr,
                    uint8_t port);

After:

GidNetworkState getBestGidIndex(const std::string &device_name,
                                struct ibv_context *context,
                                ibv_port_attr &port_attr,
                                uint8_t port,
                                int &gid_index);  // Output parameter

Key improvements:

Returns GidNetworkState quality level instead of just index
Uses output parameter for GID index value
Enables caller to make informed device selection decisions

3. Changed GID Auto-Selection Sentinel Value

Before: Used gid_index == 0 to indicate auto-selection
After: Uses gid_index < 0 (i.e., -1) as sentinel value

Why this matters:

Allows users to explicitly specify GID index 0 via MC_GID_INDEX=0
-1 is now the default value for auto-selection
More robust and unambiguous API design

Changes:

config.h: Changed default gid_index = -1 (was 0)
rdma_context.h: Changed construct() default parameter to -1
rdma_context.cpp: Changed condition from == 0 to < 0

4. Modified `openRdmaDevice()` Logic

// Automatic GID selection with quality check
if (gid_index < 0) {  // Changed from == 0
    int found_gid_index = -1;
    gid_state = getBestGidIndex(device_name, context, port_attr, port,
                                found_gid_index);

    if (gid_state != GidNetworkState::GID_NOT_FOUND) {
        LOG(INFO) << "Find best gid index: " << found_gid_index
                  << " (quality: "
                  << (gid_state == GidNetworkState::GID_WITH_NETWORK
                      ? "with network device" : "without network device")
                  << ")";
        gid_index = found_gid_index;
    } else {
        // No suitable GID found, return error
        return ERR_CONTEXT;
    }
}

// 🔑 KEY CHANGE: Reject devices with fallback GIDs
if (gid_state == GidNetworkState::GID_WITHOUT_NETWORK) {
    LOG(WARNING) << "Device " << device_name
                 << " has GID without network device, "
                 << "may not be optimal for RDMA operations";
    // Return error to force transport layer to try next device
    return ERR_CONTEXT;
}

Behavior:

Devices with GID_WITHOUT_NETWORK are rejected (return ERR_CONTEXT)
Forces the transport layer to continue iterating through device topology
Only accepts devices with GID_WITH_NETWORK quality
Results in optimal device selection

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1077 **Author:** [@uniqueni](https://github.com/uniqueni) **Created:** 11/18/2025 **Status:** ✅ Merged **Merged:** 11/21/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `bugfix_device_gid` --- ### 📝 Commits (2) - [`ca566d0`](https://github.com/kvcache-ai/Mooncake/commit/ca566d052acf2c55b2c0c5578e766e8de5b98317) add gid state - [`ac7506d`](https://github.com/kvcache-ai/Mooncake/commit/ac7506dd9945cdf1ce4365a8bd27c7a7a0d63fd9) [RDMA] Add hasNetworkDevice() helper and improve naming ### 📊 Changes **3 files changed** (+75 additions, -28 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/include/config.h` (+1 -1) 📝 `mooncake-transfer-engine/include/transport/rdma_transport/rdma_context.h` (+12 -4) 📝 `mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp` (+62 -23) </details> ### 📄 Description ### Problem Statement #### Current Behavior The existing RDMA device initialization logic selects the first available device with any valid GID (Global Identifier), regardless of whether that GID has an associated network device (ndev). This can result in poor device selection where: - Devices with **fallback GIDs** (no network device) are chosen - Better devices with **production-ready GIDs** (with proper network interfaces) are ignored - RDMA performance is degraded due to non-optimal device selection #### Example Issue Real-world scenario from logs: ``` ❌ System selected: mlx5_2, GID index 3 (no ndev - fallback GID) ✅ Better choice: mlx5_1, GID index 7 (associated with eth1 - production ready) like issue https://github.com/kvcache-ai/Mooncake/issues/1052 ``` #### Root Cause 1. `ibv_get_device_list()` returns devices in arbitrary system order 2. Original code accepts the first device with any valid GID type (RoCE v2 or InfiniBand) 3. No quality assessment performed on GID's network device association 4. Devices without proper network configuration are not filtered out ### Solution #### Design Overview Introduced a **GID quality classification system** to enable intelligent RDMA device selection based on network device associations. #### Changes Made ##### 1. Added GID Quality Enum (`rdma_context.h`) ```cpp enum class GidNetworkState { GID_WITH_NETWORK = 0, // Best: GID with network device (e.g., eth1) GID_WITHOUT_NETWORK = 1, // Fallback: GID without ndev GID_NOT_FOUND = 2 // Unusable: No suitable GID found }; ``` ##### 2. Enhanced `getBestGidIndex()` Function **Before:** ```cpp int getBestGidIndex(const std::string &device_name, struct ibv_context *context, ibv_port_attr &port_attr, uint8_t port); ``` **After:** ```cpp GidNetworkState getBestGidIndex(const std::string &device_name, struct ibv_context *context, ibv_port_attr &port_attr, uint8_t port, int &gid_index); // Output parameter ``` **Key improvements:** - Returns `GidNetworkState` quality level instead of just index - Uses output parameter for GID index value - Enables caller to make informed device selection decisions ##### 3. Changed GID Auto-Selection Sentinel Value **Before:** Used `gid_index == 0` to indicate auto-selection **After:** Uses `gid_index < 0` (i.e., `-1`) as sentinel value **Why this matters:** - Allows users to explicitly specify GID index 0 via `MC_GID_INDEX=0` - `-1` is now the default value for auto-selection - More robust and unambiguous API design **Changes:** - `config.h`: Changed default `gid_index = -1` (was `0`) - `rdma_context.h`: Changed `construct()` default parameter to `-1` - `rdma_context.cpp`: Changed condition from `== 0` to `< 0` ##### 4. Modified `openRdmaDevice()` Logic ```cpp // Automatic GID selection with quality check if (gid_index < 0) { // Changed from == 0 int found_gid_index = -1; gid_state = getBestGidIndex(device_name, context, port_attr, port, found_gid_index); if (gid_state != GidNetworkState::GID_NOT_FOUND) { LOG(INFO) << "Find best gid index: " << found_gid_index << " (quality: " << (gid_state == GidNetworkState::GID_WITH_NETWORK ? "with network device" : "without network device") << ")"; gid_index = found_gid_index; } else { // No suitable GID found, return error return ERR_CONTEXT; } } // 🔑 KEY CHANGE: Reject devices with fallback GIDs if (gid_state == GidNetworkState::GID_WITHOUT_NETWORK) { LOG(WARNING) << "Device " << device_name << " has GID without network device, " << "may not be optimal for RDMA operations"; // Return error to force transport layer to try next device return ERR_CONTEXT; } ``` **Behavior:** - Devices with `GID_WITHOUT_NETWORK` are rejected (return `ERR_CONTEXT`) - Forces the transport layer to continue iterating through device topology - Only accepts devices with `GID_WITH_NETWORK` quality - Results in optimal device selection --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:31:59 +00:00