kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1754] [Bug]: Transfer Engine eRDMA register memory failed #509

New issue

Closed

opened 2026-10-29 21:15:40 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented

2026-10-29 21:15:40 +00:00

Owner

Originally created by @easonyang1010 on GitHub (Mar 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1754

Bug Report

when I deployed a 3p1d service, 2p1d is fine, 1p had some error
I0326 14:21:37.463763 14940 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.53.4.57:16155 address 0x7f46080e8000--0x7f46080f0000...
E0326 14:23:05.675921 18944 rdma_context.cpp:265] Failed to register memory 0x7f105035d000: Input/output error [5]...
W0326 14:23:05.734648 1307 rdma_transport.cpp:400] RdmaTransport: Failed to register memory: addr 0x7f4cfa000000 length 4885774336...
E0326 14:23:06.526178 18039 rdma_transport.cpp:529] Memory region not registered by any active device(s): 0x7f3e26088000...

I notice this, but when I run dmesg, no "Out of mr size" appear

ibv_devinfo -v got this

I tried ib_write_bw everything is fine.
Do you have any idea? Thanks a lot
If you need extra info, just tell me.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @easonyang1010 on GitHub (Mar 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1754 ### Bug Report when I deployed a 3p1d service, 2p1d is fine, 1p had some error `I0326 14:21:37.463763 14940 transfer_metadata_dump.cpp:51] Failed to get segment descriptor for segment 10.53.4.57:16155 address 0x7f46080e8000--0x7f46080f0000`... `E0326 14:23:05.675921 18944 rdma_context.cpp:265] Failed to register memory 0x7f105035d000: Input/output error [5]`... `W0326 14:23:05.734648 1307 rdma_transport.cpp:400] RdmaTransport: Failed to register memory: addr 0x7f4cfa000000 length 4885774336`... `E0326 14:23:06.526178 18039 rdma_transport.cpp:529] Memory region not registered by any active device(s): 0x7f3e26088000`... I notice this, but when I run dmesg, **no "Out of mr size" appear** <img width="801" height="426" alt="Image" src="https://github.com/user-attachments/assets/47cfed11-f559-404f-8ecd-b44ae498d718" /> <img width="1576" height="1282" alt="Image" src="https://github.com/user-attachments/assets/60e3c0d4-658c-42c3-8fa9-bb64cc6ea9cf" /> `ibv_devinfo -v` got this <img width="560" height="905" alt="Image" src="https://github.com/user-attachments/assets/a8333e55-3588-44e0-96e6-1ed19dd11746" /> I tried ib_write_bw everything is fine. Do you have any idea? Thanks a lot If you need extra info, just tell me. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:15:40 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:15:41 +00:00

Author

Owner

@alogfans commented on GitHub (Mar 27, 2026):

In some environments, we need root privilege to allow driver registering memory. @stmatengss confirm this?

@alogfans commented on GitHub (Mar 27, 2026): In some environments, we need root privilege to allow driver registering memory. @stmatengss confirm this?

bsakaguchi commented

2026-10-29 21:15:41 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 27, 2026):

I run this in k8s pod, indeed it is root
run id in pod
uid=0(root) gid=0(root) groups=0(root)

@easonyang1010 commented on GitHub (Mar 27, 2026): I run this in k8s pod, indeed it is root run `id` in pod `uid=0(root) gid=0(root) groups=0(root)`

bsakaguchi commented

2026-10-29 21:15:41 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 27, 2026):

I found it is the same issue for https://github.com/kvcache-ai/Mooncake/issues/1407
I notice a PR for mooncake-store has been merged. So for mooncake-transfer-engine, this is also work?

@easonyang1010 commented on GitHub (Mar 27, 2026): I found it is the same issue for https://github.com/kvcache-ai/Mooncake/issues/1407 I notice a PR for mooncake-store has been merged. So for mooncake-transfer-engine, this is also work?

bsakaguchi commented

2026-10-29 21:15:42 +00:00

Author

Owner

@stmatengss commented on GitHub (Mar 27, 2026):

It doesn't work for transfer engine. Mooncake store supports using hugepages for RDMA registration. For Transfer Engine, the user must choose which memory to register. BTW, we can offer a huge page allocator, allowing users to directly acquire registered huge pages. CC. @YiXR

@stmatengss commented on GitHub (Mar 27, 2026): It doesn't work for transfer engine. Mooncake store supports using hugepages for RDMA registration. For Transfer Engine, the user must choose which memory to register. BTW, we can offer a huge page allocator, allowing users to directly acquire registered huge pages. CC. @YiXR

bsakaguchi commented

2026-10-29 21:15:42 +00:00

Author

Owner

@stmatengss commented on GitHub (Mar 27, 2026):

@easonyang1010 Let me try to open a PR to fix it.

@stmatengss commented on GitHub (Mar 27, 2026): @easonyang1010 Let me try to open a PR to fix it.

bsakaguchi commented

2026-10-29 21:15:42 +00:00

Author

Owner

@stmatengss commented on GitHub (Mar 27, 2026):

However, the root cause seems to be poor GDR support. Please check it according to https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions. Alternatively, use project skills for troubleshooting.

@stmatengss commented on GitHub (Mar 27, 2026): However, the root cause seems to be poor GDR support. Please check it according to https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions. Alternatively, use project skills for troubleshooting.

bsakaguchi commented

2026-10-29 21:15:42 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 28, 2026):

@stmatengss hi, I can confirm it's not GDR problem.

server side
client side
driver
I can use this code to reproduce the problem

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <infiniband/verbs.h>

#define TEST_ALIGN       4096        // 内存页对齐
#define BATCH_SIZE_GB    8           // 每批注册 8GB
#define TOTAL_SIZE_GB    200         // 累计注册 200GB
#define GB_TO_BYTES(x)   ((x) * 1024ULL * 1024ULL * 1024ULL)

// 用于保存每一批的内存指针和MR指针
typedef struct {
    void *buf;           // 内存指针
    struct ibv_mr *mr;   // 注册的MR
} mr_buf_t;

int main() {
    struct ibv_device **dev_list;
    struct ibv_device *ib_dev;
    struct ibv_context *ctx;
    struct ibv_pd *pd;
    struct ibv_device_attr dev_attr = {};
    int ret = 0;
    int dev_idx = 0;

    // 计算配置
    const size_t batch_size = GB_TO_BYTES(BATCH_SIZE_GB);
    const int batch_num = TOTAL_SIZE_GB / BATCH_SIZE_GB;

    // ===================== 关键：创建数组保存所有 buf + mr =====================
    mr_buf_t *mr_list = calloc(batch_num, sizeof(mr_buf_t));
    if (!mr_list) {
        perror("calloc failed");
        return -1;
    }

    // 1. 获取RDMA设备列表
    dev_list = ibv_get_device_list(NULL);
    if (!dev_list) {
        perror("ibv_get_device_list failed");
        ret = -1;
        goto free_mr_list;
    }

    // 2. 打开第一个RDMA网卡
    ib_dev = dev_list[dev_idx];
    ctx = ibv_open_device(ib_dev);
    if (!ctx) {
        perror("ibv_open_device failed");
        ret = -1;
        goto free_dev_list;
    }

    // 查询设备属性
    if (ibv_query_device(ctx, &dev_attr)) {
        perror("ibv_query_device failed");
        ret = -1;
        goto close_ctx;
    }

    // 打印网卡硬件信息
    printf("===== RDMA 网卡硬件信息 =====\n");
    printf("网卡名称: %s\n", ibv_get_device_name(ib_dev));
    printf("单个MR最大支持大小: %lu 字节 (%.2f GB)\n",
           dev_attr.max_mr_size,
           (double)dev_attr.max_mr_size / 1024 / 1024 / 1024);
    printf("=============================\n\n");

    // 安全检查
    if (batch_size > dev_attr.max_mr_size) {
        printf("错误：单批 %dGB 超过网卡最大MR限制！\n", BATCH_SIZE_GB);
        ret = -1;
        goto close_ctx;
    }

    // 3. 创建保护域PD
    pd = ibv_alloc_pd(ctx);
    if (!pd) {
        perror("ibv_alloc_pd failed");
        ret = -1;
        goto close_ctx;
    }

    printf("===== 开始分批注册内存 =====\n");
    printf("总目标：%dGB | 每批：%dGB | 总批数：%d\n\n",
           TOTAL_SIZE_GB, BATCH_SIZE_GB, batch_num);

    // ===================== 循环注册，不释放 =====================
    size_t registered_total = 0;
    for (int i = 0; i < batch_num; i++) {
        printf("[%d/%d] 注册第 %d 版 %dGB 内存...\n",
               i+1, batch_num, i+1, BATCH_SIZE_GB);

        // 分配对齐内存
        if (posix_memalign(&mr_list[i].buf, TEST_ALIGN, batch_size) != 0) {
            perror("posix_memalign failed");
            ret = -1;
            goto cleanup_all;
        }

        // 注册MR
        mr_list[i].mr = ibv_reg_mr(
            pd,
            mr_list[i].buf,
            batch_size,
            IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE
        );

        if (!mr_list[i].mr) {
            perror("ibv_reg_mr failed");
            free(mr_list[i].buf);
            mr_list[i].buf = NULL;
            ret = -1;
            goto cleanup_all;
        }

        registered_total += batch_size;
        printf("✅ 成功 | Lkey: 0x%04x | Rkey: 0x%04x\n",
               mr_list[i].mr->lkey, mr_list[i].mr->rkey);
    }

    printf("\n===== ✅ 全部注册完成 =====\n");
    printf("累计已注册：%.2f GB\n", (double)registered_total / GB_TO_BYTES(1));
    printf("所有内存 & MR 已保留，可正常使用！\n\n");

    // ===================== 在这里可以写你的业务逻辑 =====================
    // 例如：收发数据、测试、业务处理
    // 所有 mr_list[i].buf / mr_list[i].mr 都可以直接使用
    // ==================================================================

cleanup_all:
    // ===================== 最后统一释放所有资源 =====================
    printf("\n===== 开始统一释放所有内存 & MR =====\n");
    for (int i = 0; i < batch_num; i++) {
        if (mr_list[i].mr) {
            ibv_dereg_mr(mr_list[i].mr);
            printf("[%d] 已注销MR\n", i+1);
        }
        if (mr_list[i].buf) {
            free(mr_list[i].buf);
            printf("[%d] 已释放内存\n", i+1);
        }
    }

    // 清理RDMA资源
    ibv_dealloc_pd(pd);
close_ctx:
    ibv_close_device(ctx);
free_dev_list:
    ibv_free_device_list(dev_list);
free_mr_list:
    free(mr_list);

    return ret;
}

gcc test_mr.c -o test_mr -libverbs
sudo ./test_mr

result
you can see this page, for the same problem
https://developer.aliyun.com/article/1661045

hope for your reply, tks

@easonyang1010 commented on GitHub (Mar 28, 2026): @stmatengss hi, I can confirm it's not GDR problem. - server side <img width="780" height="603" alt="Image" src="https://github.com/user-attachments/assets/36638fcd-f82a-4fb9-a5b6-610fc74067d0" /> - client side <img width="793" height="476" alt="Image" src="https://github.com/user-attachments/assets/a93905aa-460c-4f4f-a790-af2c7922cd1d" /> - driver <img width="1560" height="260" alt="Image" src="https://github.com/user-attachments/assets/0d31e51d-46ec-4c1d-ba9d-ac2780a2cb87" /> - I can use this code to reproduce the problem ```c #include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <infiniband/verbs.h> #define TEST_ALIGN 4096 // 内存页对齐 #define BATCH_SIZE_GB 8 // 每批注册 8GB #define TOTAL_SIZE_GB 200 // 累计注册 200GB #define GB_TO_BYTES(x) ((x) * 1024ULL * 1024ULL * 1024ULL) // 用于保存每一批的内存指针和MR指针 typedef struct { void *buf; // 内存指针 struct ibv_mr *mr; // 注册的MR } mr_buf_t; int main() { struct ibv_device **dev_list; struct ibv_device *ib_dev; struct ibv_context *ctx; struct ibv_pd *pd; struct ibv_device_attr dev_attr = {}; int ret = 0; int dev_idx = 0; // 计算配置 const size_t batch_size = GB_TO_BYTES(BATCH_SIZE_GB); const int batch_num = TOTAL_SIZE_GB / BATCH_SIZE_GB; // ===================== 关键：创建数组保存所有 buf + mr ===================== mr_buf_t *mr_list = calloc(batch_num, sizeof(mr_buf_t)); if (!mr_list) { perror("calloc failed"); return -1; } // 1. 获取RDMA设备列表 dev_list = ibv_get_device_list(NULL); if (!dev_list) { perror("ibv_get_device_list failed"); ret = -1; goto free_mr_list; } // 2. 打开第一个RDMA网卡 ib_dev = dev_list[dev_idx]; ctx = ibv_open_device(ib_dev); if (!ctx) { perror("ibv_open_device failed"); ret = -1; goto free_dev_list; } // 查询设备属性 if (ibv_query_device(ctx, &dev_attr)) { perror("ibv_query_device failed"); ret = -1; goto close_ctx; } // 打印网卡硬件信息 printf("===== RDMA 网卡硬件信息 =====\n"); printf("网卡名称: %s\n", ibv_get_device_name(ib_dev)); printf("单个MR最大支持大小: %lu 字节 (%.2f GB)\n", dev_attr.max_mr_size, (double)dev_attr.max_mr_size / 1024 / 1024 / 1024); printf("=============================\n\n"); // 安全检查 if (batch_size > dev_attr.max_mr_size) { printf("错误：单批 %dGB 超过网卡最大MR限制！\n", BATCH_SIZE_GB); ret = -1; goto close_ctx; } // 3. 创建保护域PD pd = ibv_alloc_pd(ctx); if (!pd) { perror("ibv_alloc_pd failed"); ret = -1; goto close_ctx; } printf("===== 开始分批注册内存 =====\n"); printf("总目标：%dGB | 每批：%dGB | 总批数：%d\n\n", TOTAL_SIZE_GB, BATCH_SIZE_GB, batch_num); // ===================== 循环注册，不释放 ===================== size_t registered_total = 0; for (int i = 0; i < batch_num; i++) { printf("[%d/%d] 注册第 %d 版 %dGB 内存...\n", i+1, batch_num, i+1, BATCH_SIZE_GB); // 分配对齐内存 if (posix_memalign(&mr_list[i].buf, TEST_ALIGN, batch_size) != 0) { perror("posix_memalign failed"); ret = -1; goto cleanup_all; } // 注册MR mr_list[i].mr = ibv_reg_mr( pd, mr_list[i].buf, batch_size, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE ); if (!mr_list[i].mr) { perror("ibv_reg_mr failed"); free(mr_list[i].buf); mr_list[i].buf = NULL; ret = -1; goto cleanup_all; } registered_total += batch_size; printf("✅ 成功 | Lkey: 0x%04x | Rkey: 0x%04x\n", mr_list[i].mr->lkey, mr_list[i].mr->rkey); } printf("\n===== ✅ 全部注册完成 =====\n"); printf("累计已注册：%.2f GB\n", (double)registered_total / GB_TO_BYTES(1)); printf("所有内存 & MR 已保留，可正常使用！\n\n"); // ===================== 在这里可以写你的业务逻辑 ===================== // 例如：收发数据、测试、业务处理 // 所有 mr_list[i].buf / mr_list[i].mr 都可以直接使用 // ================================================================== cleanup_all: // ===================== 最后统一释放所有资源 ===================== printf("\n===== 开始统一释放所有内存 & MR =====\n"); for (int i = 0; i < batch_num; i++) { if (mr_list[i].mr) { ibv_dereg_mr(mr_list[i].mr); printf("[%d] 已注销MR\n", i+1); } if (mr_list[i].buf) { free(mr_list[i].buf); printf("[%d] 已释放内存\n", i+1); } } // 清理RDMA资源 ibv_dealloc_pd(pd); close_ctx: ibv_close_device(ctx); free_dev_list: ibv_free_device_list(dev_list); free_mr_list: free(mr_list); return ret; } ``` ```shell gcc test_mr.c -o test_mr -libverbs sudo ./test_mr ``` - result ![Image](https://github.com/user-attachments/assets/1d072693-b29d-4aec-8ac0-b8dd46340219) - you can see this page, for the same problem https://developer.aliyun.com/article/1661045 hope for your reply, tks

bsakaguchi commented

2026-10-29 21:15:42 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 31, 2026):

@stmatengss hi, do you have any updates

@easonyang1010 commented on GitHub (Mar 31, 2026): @stmatengss hi, do you have any updates