kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2017] [Bug]: `ibv_reg_mr` is silently truncated to `max_mr_size`, but `BufferDesc.length` advertises the original size → `IBV_WC_REM_ACCESS_ERR` #559

New issue

Closed

opened 2026-10-29 21:16:26 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:16:26 +00:00

Owner

Originally created by @Jasen2201 on GitHub (Apr 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2017

Bug Report

Summary

When registerLocalMemory() is called with length > device.max_mr_size,
Mooncake registers only max_mr_size bytes but tells peers the buffer is the
full original length. Any RDMA WRITE to an address past the truncation
boundary hits IBV_WC_REM_ACCESS_ERR (CQE error 10).

The bug is address-driven, not size-driven: even a tiny 64 KiB write fails
if its target address falls past the boundary.

Root cause

length is shrunk locally inside registerMemoryRegionInternal, but the
caller still uses the original value when building the segment descriptor:

rdma_context.cpp:212-218

int RdmaContext::registerMemoryRegionInternal(void *addr, size_t length, ...) {
    if (length > (size_t)globalConfig().max_mr_size) {
        PLOG(WARNING) << "shrink it to " << globalConfig().max_mr_size;
        length = (size_t)globalConfig().max_mr_size;   // local-only, value-passed
    }
    mrMeta.mr = ibv_reg_mr(pd_, addr, length, access);  // protects only first 2 GiB

rdma_transport.cpp:234, 295-297

context_list_[i]->registerMemoryRegion(addr, length, ar);  // original length
...
buffer_desc.length = length;                                // advertises full size!

config.cpp:387 also clamps max_mr_size to the device cap, so raising
MC_MAX_MR_SIZE cannot work around it on hardware-limited NICs.

Why this is hit on AMD MI355X (288 GB HBM)

NIC: AMD Pensando ionic, max_mr_size = 2 GiB
Workload: vLLM PD-disaggregated DeepSeek-R1 (MLA, 61 layers, 576 B/token)
Per-layer KV buffer ≈ 3.25 GiB on 288 GB HBM at gpu-memory-utilization=0.9
(room for ~6 M tokens; the 2 GiB MR cutoff sits at ~3.73 M tokens)

ionic_comp_msn:1277: cqe with error 10
IBV_EVENT_QP_ACCESS_ERR

This will hit any deployment where per-layer KV exceeds max_mr_size —
large-HBM AMD MI300/355 today, larger GPUs in the future.

Minimal reproducer

Two nodes, one RDMA NIC each. Register a 3 GiB buffer, write to offsets
straddling the 2 GiB boundary.

target.py

import argparse, json, os, time, torch
from mooncake.engine import TransferEngine

p = argparse.ArgumentParser()
p.add_argument("--ip", required=True)
p.add_argument("--buffer-gb", type=float, default=3.0)
args = p.parse_args()

torch.cuda.set_device(0)
eng = TransferEngine()
assert eng.initialize(args.ip, "P2PHANDSHAKE", "rdma", "") == 0

n = int(args.buffer_gb * (1 << 30))
t = torch.zeros(n, dtype=torch.uint8, device="cuda:0")
assert eng.register_memory(t.data_ptr(), n) == 0

json.dump({"server_name": f"{args.ip}:{eng.get_rpc_port()}",
           "base_addr": t.data_ptr()}, open("handshake.json", "w"))
while True: time.sleep(10)

initiator.py

import argparse, json, torch
from mooncake.engine import TransferEngine

GiB, MiB, SLICE = 1 << 30, 1 << 20, 64 * 1024
p = argparse.ArgumentParser()
p.add_argument("--ip", required=True)
args = p.parse_args()

torch.cuda.set_device(0)
hs = json.load(open("handshake.json"))
eng = TransferEngine()
assert eng.initialize(args.ip, "P2PHANDSHAKE", "rdma", "") == 0
src = torch.ones(SLICE, dtype=torch.uint8, device="cuda:0")
assert eng.register_memory(src.data_ptr(), SLICE) == 0

for name, off in [("0", 0), ("1.9G", int(1.9*GiB)),
                  ("2G+1M", 2*GiB + MiB), ("2.5G", int(2.5*GiB))]:
    ret = eng.batch_transfer_sync_write(
        hs["server_name"], [src.data_ptr()], [hs["base_addr"] + off], [SLICE])
    print(f"  off={name:6s}  within_2G={(off+SLICE)<=2*GiB!s:5}  "
          f"ret={ret}  {'PASS' if ret==0 else 'FAIL'}")

Result on ionic (`max_mr_size = 2 GiB`)

offset	within MR	result
`0`	yes	PASS
`1.9 GiB`	yes	PASS
`2G + 1M`	no	FAIL
`2.5 GiB`	no	FAIL

Suggested fixes

Auto-chunk in registerLocalMemoryInternal — split buffers larger than
max_mr_size into multiple MRs, publishing one BufferDesc per chunk.
BufferDesc schema does not need to change; the existing address-based
lookup naturally routes each WRITE to the right rkey.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Jasen2201 on GitHub (Apr 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2017 ### Bug Report ## Summary When `registerLocalMemory()` is called with `length > device.max_mr_size`, Mooncake registers only `max_mr_size` bytes but tells peers the buffer is the full original length. Any RDMA WRITE to an address past the truncation boundary hits `IBV_WC_REM_ACCESS_ERR` (CQE error 10). The bug is **address-driven, not size-driven**: even a tiny 64 KiB write fails if its target address falls past the boundary. ## Root cause `length` is shrunk locally inside `registerMemoryRegionInternal`, but the caller still uses the original value when building the segment descriptor: **`rdma_context.cpp:212-218`** ```cpp int RdmaContext::registerMemoryRegionInternal(void *addr, size_t length, ...) { if (length > (size_t)globalConfig().max_mr_size) { PLOG(WARNING) << "shrink it to " << globalConfig().max_mr_size; length = (size_t)globalConfig().max_mr_size; // local-only, value-passed } mrMeta.mr = ibv_reg_mr(pd_, addr, length, access); // protects only first 2 GiB ``` **`rdma_transport.cpp:234, 295-297`** ```cpp context_list_[i]->registerMemoryRegion(addr, length, ar); // original length ... buffer_desc.length = length; // advertises full size! ``` `config.cpp:387` also clamps `max_mr_size` to the device cap, so raising `MC_MAX_MR_SIZE` cannot work around it on hardware-limited NICs. ## Why this is hit on AMD MI355X (288 GB HBM) - NIC: AMD Pensando ionic, `max_mr_size = 2 GiB` - Workload: vLLM PD-disaggregated DeepSeek-R1 (MLA, 61 layers, 576 B/token) - Per-layer KV buffer ≈ 3.25 GiB on 288 GB HBM at `gpu-memory-utilization=0.9` (room for ~6 M tokens; the 2 GiB MR cutoff sits at ~3.73 M tokens) ``` ionic_comp_msn:1277: cqe with error 10 IBV_EVENT_QP_ACCESS_ERR ``` This will hit any deployment where per-layer KV exceeds `max_mr_size` — large-HBM AMD MI300/355 today, larger GPUs in the future. ## Minimal reproducer Two nodes, one RDMA NIC each. Register a 3 GiB buffer, write to offsets straddling the 2 GiB boundary. **`target.py`** ```python import argparse, json, os, time, torch from mooncake.engine import TransferEngine p = argparse.ArgumentParser() p.add_argument("--ip", required=True) p.add_argument("--buffer-gb", type=float, default=3.0) args = p.parse_args() torch.cuda.set_device(0) eng = TransferEngine() assert eng.initialize(args.ip, "P2PHANDSHAKE", "rdma", "") == 0 n = int(args.buffer_gb * (1 << 30)) t = torch.zeros(n, dtype=torch.uint8, device="cuda:0") assert eng.register_memory(t.data_ptr(), n) == 0 json.dump({"server_name": f"{args.ip}:{eng.get_rpc_port()}", "base_addr": t.data_ptr()}, open("handshake.json", "w")) while True: time.sleep(10) ``` **`initiator.py`** ```python import argparse, json, torch from mooncake.engine import TransferEngine GiB, MiB, SLICE = 1 << 30, 1 << 20, 64 * 1024 p = argparse.ArgumentParser() p.add_argument("--ip", required=True) args = p.parse_args() torch.cuda.set_device(0) hs = json.load(open("handshake.json")) eng = TransferEngine() assert eng.initialize(args.ip, "P2PHANDSHAKE", "rdma", "") == 0 src = torch.ones(SLICE, dtype=torch.uint8, device="cuda:0") assert eng.register_memory(src.data_ptr(), SLICE) == 0 for name, off in [("0", 0), ("1.9G", int(1.9*GiB)), ("2G+1M", 2*GiB + MiB), ("2.5G", int(2.5*GiB))]: ret = eng.batch_transfer_sync_write( hs["server_name"], [src.data_ptr()], [hs["base_addr"] + off], [SLICE]) print(f" off={name:6s} within_2G={(off+SLICE)<=2*GiB!s:5} " f"ret={ret} {'PASS' if ret==0 else 'FAIL'}") ``` ### Result on ionic (`max_mr_size = 2 GiB`) | offset | within MR | result | |-----------|-----------|--------| | `0` | yes | PASS | | `1.9 GiB` | yes | PASS | | `2G + 1M` | no | **FAIL** | | `2.5 GiB` | no | **FAIL** | ## Suggested fixes 1. **Auto-chunk in `registerLocalMemoryInternal`** — split buffers larger than `max_mr_size` into multiple MRs, publishing one `BufferDesc` per chunk. `BufferDesc` schema does not need to change; the existing address-based lookup naturally routes each WRITE to the right `rkey`. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:16:26 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Author

Owner

@he-yufeng commented on GitHub (May 4, 2026):

Happy to take this. The root cause and fix-1 (auto-chunk in registerLocalMemoryInternal) read clean, and #1912 already established the auto-split pattern in the EFA twin (efa_context.cpp / efa_transport.cpp), so there is a reference implementation to mirror.

Quick sketch of the plan, want to confirm direction before coding so I do not duplicate work:

In RdmaContext::registerMemoryRegionInternal (rdma_context.cpp:222-228), drop the silent length = max_mr_size shrink. Instead, have registerMemoryRegion (the public entry) split [addr, length) into ceil(length / max_mr_size) chunks, call registerMemoryRegionInternal once per chunk, and store one MemoryRegionMeta per chunk in memory_region_metas_ keyed by the chunk start address.
In RdmaTransport::registerLocalMemory (rdma_transport.cpp:234, 295-297), publish one BufferDesc per chunk (each with its own lkey/rkey and chunk-local length), so the address-based lookup at the WRITE side naturally routes each slice to the right MR. BufferDesc schema does not need to change.
The deregister / close paths walk the per-chunk MemoryRegionMeta list.
Acceptance: a regression test mirroring efa_single_nic_large_mr_test for the ibverbs path — register a 2 * max_mr_size + 1MiB buffer, write at offsets 0, max_mr_size - 1MiB, max_mr_size + 1MiB, 2 * max_mr_size + 1MiB, all return 0.

A few open questions before I code:

Should the chunk size match max_mr_size exactly, or expose an MC_MR_CHUNK_SIZE knob (analogous to EFA-side tuning)?
Page-size alignment: EFA needed PTE-aware splitting because of the 24M-PTE ceiling. Generic RDMA should be fine with byte-level splits at max_mr_size, since max_num_pte is not exposed by ibverbs the same way. Confirm this is correct?
Should the chunked registerMemoryRegion keep silent-shrink semantics behind a config flag for backward compat, or is the new auto-split the only path?

If this all looks reasonable to you, I will start on the patch this week. Thanks!

@he-yufeng commented on GitHub (May 4, 2026): Happy to take this. The root cause and fix-1 (auto-chunk in `registerLocalMemoryInternal`) read clean, and #1912 already established the auto-split pattern in the EFA twin (`efa_context.cpp` / `efa_transport.cpp`), so there is a reference implementation to mirror. Quick sketch of the plan, want to confirm direction before coding so I do not duplicate work: 1. In `RdmaContext::registerMemoryRegionInternal` (`rdma_context.cpp:222-228`), drop the silent `length = max_mr_size` shrink. Instead, have `registerMemoryRegion` (the public entry) split `[addr, length)` into `ceil(length / max_mr_size)` chunks, call `registerMemoryRegionInternal` once per chunk, and store one `MemoryRegionMeta` per chunk in `memory_region_metas_` keyed by the chunk start address. 2. In `RdmaTransport::registerLocalMemory` (`rdma_transport.cpp:234, 295-297`), publish one `BufferDesc` per chunk (each with its own `lkey/rkey` and chunk-local `length`), so the address-based lookup at the WRITE side naturally routes each slice to the right MR. `BufferDesc` schema does not need to change. 3. The deregister / close paths walk the per-chunk `MemoryRegionMeta` list. 4. Acceptance: a regression test mirroring `efa_single_nic_large_mr_test` for the ibverbs path — register a `2 * max_mr_size + 1MiB` buffer, write at offsets `0`, `max_mr_size - 1MiB`, `max_mr_size + 1MiB`, `2 * max_mr_size + 1MiB`, all return 0. A few open questions before I code: - Should the chunk size match `max_mr_size` exactly, or expose an `MC_MR_CHUNK_SIZE` knob (analogous to EFA-side tuning)? - Page-size alignment: EFA needed PTE-aware splitting because of the 24M-PTE ceiling. Generic RDMA should be fine with byte-level splits at `max_mr_size`, since `max_num_pte` is not exposed by ibverbs the same way. Confirm this is correct? - Should the chunked `registerMemoryRegion` keep silent-shrink semantics behind a config flag for backward compat, or is the new auto-split the only path? If this all looks reasonable to you, I will start on the patch this week. Thanks!

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Author

Owner

@alogfans commented on GitHub (May 8, 2026):

@he-yufeng Two issues to be carefully considered:

1' the buffers in the metadata reflect the actual presentation of memory regions (rather than user invocations)
2' when a single transfer request across multiple real memory regions, the slice should be splited according to the memory region boundary.

@alogfans commented on GitHub (May 8, 2026): @he-yufeng Two issues to be carefully considered: 1' the buffers in the metadata reflect the actual presentation of memory regions (rather than user invocations) 2' when a single transfer request across multiple real memory regions, the slice should be splited according to the memory region boundary.

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Author

Owner

@he-yufeng commented on GitHub (May 8, 2026):

Thanks, that helps. I will treat the split as part of the visible MR model, not just as a registration fallback.

Concretely:

memory_region_metas_ / published BufferDescs should be per actual ibverbs MR chunk, keyed by the chunk start, with chunk-local length/lkey/rkey. The original user call is only the input range.
read/write slicing cannot assume one descriptor covers the full request. If a transfer range crosses a real MR boundary, it should be split at that boundary and each sub-slice should use the matching descriptor.

I will check the lookup and transfer path first so the patch handles both the registration-side metadata and the transfer-side boundary split in the same change.

@he-yufeng commented on GitHub (May 8, 2026): Thanks, that helps. I will treat the split as part of the visible MR model, not just as a registration fallback. Concretely: - `memory_region_metas_` / published `BufferDesc`s should be per actual ibverbs MR chunk, keyed by the chunk start, with chunk-local length/lkey/rkey. The original user call is only the input range. - read/write slicing cannot assume one descriptor covers the full request. If a transfer range crosses a real MR boundary, it should be split at that boundary and each sub-slice should use the matching descriptor. I will check the lookup and transfer path first so the patch handles both the registration-side metadata and the transfer-side boundary split in the same change.

bsakaguchi commented

2026-10-29 21:16:27 +00:00

Author

Owner

@wangshaochuang commented on GitHub (May 26, 2026):

Thanks, that helps. I will treat the split as part of the visible MR model, not just as a registration fallback.

Concretely:

memory_region_metas_ / published BufferDescs should be per actual ibverbs MR chunk, keyed by the chunk start, with chunk-local length/lkey/rkey. The original user call is only the input range.

read/write slicing cannot assume one descriptor covers the full request. If a transfer range crosses a real MR boundary, it should be split at that boundary and each sub-slice should use the matching descriptor.

I will check the lookup and transfer path first so the patch handles both the registration-side metadata and the transfer-side boundary split in the same change.

Hi, 我在其他的网卡上也遇到了这个问题，请问这个问题当前有patch可以验证了吗？🙏

@wangshaochuang commented on GitHub (May 26, 2026): > Thanks, that helps. I will treat the split as part of the visible MR model, not just as a registration fallback. > > Concretely: > > * `memory_region_metas_` / published `BufferDesc`s should be per actual ibverbs MR chunk, keyed by the chunk start, with chunk-local length/lkey/rkey. The original user call is only the input range. > * read/write slicing cannot assume one descriptor covers the full request. If a transfer range crosses a real MR boundary, it should be split at that boundary and each sub-slice should use the matching descriptor. > > I will check the lookup and transfer path first so the patch handles both the registration-side metadata and the transfer-side boundary split in the same change. Hi, 我在其他的网卡上也遇到了这个问题，请问这个问题当前有patch可以验证了吗？🙏

bsakaguchi referenced this issue

2026-10-29 21:29:12 +00:00

[PR #559] [CLOSED] 【test】add remount test and batchevict test for mooncakestore #1650