[GH-ISSUE #2511] [Bug]: When enable expandable_segments, mooncake register memory failed #664

New issue

Closed

opened 2026-10-29 21:17:49 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented 2026-10-29 21:17:49 +00:00

Owner

Copy link

Originally created by @MARD1NO on GitHub (Jun 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2511

Originally assigned to: @MARD1NO, @Icedcoco on GitHub.

Bug Report

When I export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

it will cause registration failed, like this issue mentioned: https://github.com/kvcache-ai/Mooncake/issues/351#issuecomment-2888725354

Will Mooncake support it?

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

Originally created by @MARD1NO on GitHub (Jun 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2511 Originally assigned to: @MARD1NO, @Icedcoco on GitHub. ### Bug Report When I export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True it will cause registration failed, like this issue mentioned: https://github.com/kvcache-ai/Mooncake/issues/351#issuecomment-2888725354 Will Mooncake support it? ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi 2026-10-29 21:17:49 +00:00

• closed this issue
• added the
  bug
  label

bsakaguchi commented 2026-10-29 21:17:50 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 17, 2026):

Thanks for opening this issue, @MARD1NO!

Field	Value
Issue	#2511
GitHub user ID	42901638
Reporter	@MARD1NO

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4726463928 --> @github-actions[bot] commented on GitHub (Jun 17, 2026): Thanks for opening this issue, @MARD1NO! | Field | Value | |-------|-------| | **Issue** | #2511 | | **GitHub user ID** | `42901638` | | **Reporter** | @MARD1NO | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:17:50 +00:00

Author

Owner

Copy link

@Icedcoco commented on GitHub (Jun 18, 2026):

Thanks for reporting this. expandable_segments:True changes how PyTorch allocates CUDA memory, and Mooncake's RDMA registration depends on whether that GPU memory can be registered by the underlying GDR/RDMA stack. The same register memory failed error can also be caused by host RDMA/GDR configuration, so we need a bit more information to distinguish the cases.

Could you share:

1. Mooncake version or commit SHA
2. PyTorch version, CUDA runtime/driver version, and GPU model
3. The integration you are using, e.g. SGLang/vLLM version and the startup command
4. The full Mooncake error log around registerLocalMemory, especially lines containing Failed to retrieve dmabuf, ibv_reg_mr, ibv_reg_dmabuf_mr, or Failed to register memory
5. Whether the issue disappears after completely unsetting PYTORCH_CUDA_ALLOC_CONF instead of setting expandable_segments:False
6. Whether nvidia_peermem is loaded on the host:

   lsmod | grep -E 'nvidia_peermem|nv_peer_mem'
   

7. Whether basic GDR/RDMA checks pass, for example ibv_devinfo, nvidia-smi topo -m, and GDRCopy tests if available

This will help us tell whether the failure is caused by PyTorch expandable CUDA VMM allocation compatibility, missing nvidia_peermem/dmabuf support, or a more general RDMA environment issue.

<!-- gh-comment-id:4739297702 --> @Icedcoco commented on GitHub (Jun 18, 2026): Thanks for reporting this. `expandable_segments:True` changes how PyTorch allocates CUDA memory, and Mooncake's RDMA registration depends on whether that GPU memory can be registered by the underlying GDR/RDMA stack. The same `register memory failed` error can also be caused by host RDMA/GDR configuration, so we need a bit more information to distinguish the cases. Could you share: 1. Mooncake version or commit SHA 2. PyTorch version, CUDA runtime/driver version, and GPU model 3. The integration you are using, e.g. SGLang/vLLM version and the startup command 4. The full Mooncake error log around `registerLocalMemory`, especially lines containing `Failed to retrieve dmabuf`, `ibv_reg_mr`, `ibv_reg_dmabuf_mr`, or `Failed to register memory` 5. Whether the issue disappears after completely unsetting `PYTORCH_CUDA_ALLOC_CONF` instead of setting `expandable_segments:False` 6. Whether `nvidia_peermem` is loaded on the host: ```bash lsmod | grep -E 'nvidia_peermem|nv_peer_mem' ``` 7. Whether basic GDR/RDMA checks pass, for example `ibv_devinfo`, `nvidia-smi topo -m`, and GDRCopy tests if available This will help us tell whether the failure is caused by PyTorch expandable CUDA VMM allocation compatibility, missing `nvidia_peermem`/dmabuf support, or a more general RDMA environment issue.

bsakaguchi commented 2026-10-29 21:17:50 +00:00

Author

Owner

Copy link

@MARD1NO commented on GitHub (Jun 18, 2026):

Thanks for reporting this. expandable_segments:True changes how PyTorch allocates CUDA memory, and Mooncake's RDMA registration depends on whether that GPU memory can be registered by the underlying GDR/RDMA stack. The same register memory failed error can also be caused by host RDMA/GDR configuration, so we need a bit more information to distinguish the cases.

Could you share:

1. Mooncake version or commit SHA
2. PyTorch version, CUDA runtime/driver version, and GPU model
3. The integration you are using, e.g. SGLang/vLLM version and the startup command
4. The full Mooncake error log around registerLocalMemory, especially lines containing Failed to retrieve dmabuf, ibv_reg_mr, ibv_reg_dmabuf_mr, or Failed to register memory
5. Whether the issue disappears after completely unsetting PYTORCH_CUDA_ALLOC_CONF instead of setting expandable_segments:False
6. Whether nvidia_peermem is loaded on the host:
   lsmod | grep -E 'nvidia_peermem|nv_peer_mem'
7. Whether basic GDR/RDMA checks pass, for example ibv_devinfo, nvidia-smi topo -m, and GDRCopy tests if available

This will help us tell whether the failure is caused by PyTorch expandable CUDA VMM allocation compatibility, missing nvidia_peermem/dmabuf support, or a more general RDMA environment issue.

Thanks for your reply :D

1. Mooncake version is:

mooncake-transfer-engine                 0.3.9

2. Torch version is torch2.8 + cuda12.9, I check this commit, this contain the modification: https://github.com/pytorch/pytorch/pull/153261

3. I used in our Inference engine

4. When I unset the PYTORCH_CUDA_ALLOC_CONF, it disappear issue.

5. I'm sure nvidia_peermem has been loaded.

when I disable expandable_segements before create paged kv cache, the issue also disappear, and I notice that vllm also process in this way

<!-- gh-comment-id:4739363395 --> @MARD1NO commented on GitHub (Jun 18, 2026): > Thanks for reporting this. `expandable_segments:True` changes how PyTorch allocates CUDA memory, and Mooncake's RDMA registration depends on whether that GPU memory can be registered by the underlying GDR/RDMA stack. The same `register memory failed` error can also be caused by host RDMA/GDR configuration, so we need a bit more information to distinguish the cases. > > Could you share: > > 1. Mooncake version or commit SHA > 2. PyTorch version, CUDA runtime/driver version, and GPU model > 3. The integration you are using, e.g. SGLang/vLLM version and the startup command > 4. The full Mooncake error log around `registerLocalMemory`, especially lines containing `Failed to retrieve dmabuf`, `ibv_reg_mr`, `ibv_reg_dmabuf_mr`, or `Failed to register memory` > 5. Whether the issue disappears after completely unsetting `PYTORCH_CUDA_ALLOC_CONF` instead of setting `expandable_segments:False` > 6. Whether `nvidia_peermem` is loaded on the host: > lsmod | grep -E 'nvidia_peermem|nv_peer_mem' > 7. Whether basic GDR/RDMA checks pass, for example `ibv_devinfo`, `nvidia-smi topo -m`, and GDRCopy tests if available > > This will help us tell whether the failure is caused by PyTorch expandable CUDA VMM allocation compatibility, missing `nvidia_peermem`/dmabuf support, or a more general RDMA environment issue. Thanks for your reply :D 1. Mooncake version is: ``` mooncake-transfer-engine 0.3.9 ``` 2. Torch version is torch2.8 + cuda12.9, I check this commit, this contain the modification: https://github.com/pytorch/pytorch/pull/153261 3. I used in our Inference engine 5. When I unset the PYTORCH_CUDA_ALLOC_CONF, it disappear issue. 6. I'm sure nvidia_peermem has been loaded. when I disable expandable_segements before create paged kv cache, the issue also disappear, and I notice that vllm also process in this way

bsakaguchi commented 2026-10-29 21:17:50 +00:00

Author

Owner

Copy link

@Icedcoco commented on GitHub (Jun 18, 2026):

Thanks for the details. This narrows the issue down quite a bit.

Since the failure disappears when PYTORCH_CUDA_ALLOC_CONF is unset, and also disappears when expandable_segments is disabled before creating the paged KV cache, this does not look like a general RDMA/GDR environment issue. It is more likely related to how mooncake-transfer-engine 0.3.9 registers PyTorch expandable CUDA VMM allocations.

One relevant detail is that 0.3.9 still has the older dmabuf registration path. After that release, we merged several related fixes, especially:

• #2034
• #2035
• #2066

#2035 is particularly relevant here because framework allocators can return a pointer inside a larger CUDA allocation, while cuMemGetHandleForAddressRange requires the real allocation boundary. PyTorch expandable segments are more likely to expose this.

Could you try a Mooncake build from current main or any release that includes those fixes? If it still fails, please share the first Mooncake error lines around registration, especially whether the failure is from Failed to retrieve dmabuf, ibv_reg_dmabuf_mr, ibv_reg_mr, or Failed to register memory ... Bad address.

<!-- gh-comment-id:4739584760 --> @Icedcoco commented on GitHub (Jun 18, 2026): Thanks for the details. This narrows the issue down quite a bit. Since the failure disappears when `PYTORCH_CUDA_ALLOC_CONF` is unset, and also disappears when `expandable_segments` is disabled before creating the paged KV cache, this does not look like a general RDMA/GDR environment issue. It is more likely related to how `mooncake-transfer-engine 0.3.9` registers PyTorch expandable CUDA VMM allocations. One relevant detail is that `0.3.9` still has the older dmabuf registration path. After that release, we merged several related fixes, especially: - #2034 - #2035 - #2066 #2035 is particularly relevant here because framework allocators can return a pointer inside a larger CUDA allocation, while `cuMemGetHandleForAddressRange` requires the real allocation boundary. PyTorch expandable segments are more likely to expose this. Could you try a Mooncake build from current `main` or any release that includes those fixes? If it still fails, please share the first Mooncake error lines around registration, especially whether the failure is from `Failed to retrieve dmabuf`, `ibv_reg_dmabuf_mr`, `ibv_reg_mr`, or `Failed to register memory ... Bad address`.

bsakaguchi commented 2026-10-29 21:17:50 +00:00

Author

Owner

Copy link

@MARD1NO commented on GitHub (Jun 18, 2026):

Thanks for the details. This narrows the issue down quite a bit.

Since the failure disappears when PYTORCH_CUDA_ALLOC_CONF is unset, and also disappears when expandable_segments is disabled before creating the paged KV cache, this does not look like a general RDMA/GDR environment issue. It is more likely related to how mooncake-transfer-engine 0.3.9 registers PyTorch expandable CUDA VMM allocations.

One relevant detail is that 0.3.9 still has the older dmabuf registration path. After that release, we merged several related fixes, especially:

• [TransferEngine] Init CUDA primary context before dmabuf-based mem registration #2034
• [TransferEngine] Use allocation base addr for dmabuf-based mem registration #2035
• [CI/Build] Switch WITH_NVIDIA_PEERMEM to env variable #2066

#2035 is particularly relevant here because framework allocators can return a pointer inside a larger CUDA allocation, while cuMemGetHandleForAddressRange requires the real allocation boundary. PyTorch expandable segments are more likely to expose this.

Could you try a Mooncake build from current main or any release that includes those fixes? If it still fails, please share the first Mooncake error lines around registration, especially whether the failure is from Failed to retrieve dmabuf, ibv_reg_dmabuf_mr, ibv_reg_mr, or Failed to register memory ... Bad address.

Thanks ! I will try a newer version of mooncake

<!-- gh-comment-id:4739631918 --> @MARD1NO commented on GitHub (Jun 18, 2026): > Thanks for the details. This narrows the issue down quite a bit. > > Since the failure disappears when `PYTORCH_CUDA_ALLOC_CONF` is unset, and also disappears when `expandable_segments` is disabled before creating the paged KV cache, this does not look like a general RDMA/GDR environment issue. It is more likely related to how `mooncake-transfer-engine 0.3.9` registers PyTorch expandable CUDA VMM allocations. > > One relevant detail is that `0.3.9` still has the older dmabuf registration path. After that release, we merged several related fixes, especially: > > * [[TransferEngine] Init CUDA primary context before dmabuf-based mem registration #2034](https://github.com/kvcache-ai/Mooncake/pull/2034) > * [[TransferEngine] Use allocation base addr for dmabuf-based mem registration #2035](https://github.com/kvcache-ai/Mooncake/pull/2035) > * [[CI/Build] Switch WITH_NVIDIA_PEERMEM to env variable #2066](https://github.com/kvcache-ai/Mooncake/pull/2066) > > [#2035](https://github.com/kvcache-ai/Mooncake/pull/2035) is particularly relevant here because framework allocators can return a pointer inside a larger CUDA allocation, while `cuMemGetHandleForAddressRange` requires the real allocation boundary. PyTorch expandable segments are more likely to expose this. > > Could you try a Mooncake build from current `main` or any release that includes those fixes? If it still fails, please share the first Mooncake error lines around registration, especially whether the failure is from `Failed to retrieve dmabuf`, `ibv_reg_dmabuf_mr`, `ibv_reg_mr`, or `Failed to register memory ... Bad address`. Thanks ! I will try a newer version of mooncake

bsakaguchi commented 2026-10-29 21:17:51 +00:00

Author

Owner

Copy link

@hope1262946533 commented on GitHub (Aug 19, 2026):

Retested on the latest release — still fails. Below are the exact error lines you asked for, and the root cause, which turns out to be #2035 itself rather than its absence.

Versions tested

version	includes	result
mooncake-transfer-engine-cuda13 0.3.11.post1	#2034, #2035, #2066	fails — 1944 registrations per side, all EINVAL
mooncake-transfer-engine-cuda13 0.3.12.post1 (latest on PyPI)	additionally #2416, #2523	fails — same EINVAL on the same buffers

So the environment already had all three fixes you listed before this report; 0.3.12.post1 adds #2523 and does not change the outcome. Confirmed the new build was actually loaded (mooncake.__file__ pointed at the 0.3.12.post1 install, and the error line numbers moved from rdma_context.cpp:346 to rdma_context.cpp:588).

First error lines on 0.3.12.post1

Prefill side:

E0819 11:21:45.193161 147554 rdma_context.cpp:588] Failed to register memory 0x12397fd5800: Invalid argument [22]
E0819 11:21:45.193186 147554 rdma_transport.cpp:302] Failed to register memory region with context 0
W0819 11:21:45.193463 147554 rdma_transport.cpp:481] RdmaTransport: Failed to register memory: addr 0x12397fd5800 length 1916853120
W0819 11:21:45.199296 147554 rdma_transport.cpp:481] RdmaTransport: Failed to register memory: addr 0x1009700b000 length 191692800

Decode side is identical in shape (addr 0x12421e3d000 length 2013598080 and addr 0x1009700b000 length 201427200).

To answer the specific question: the failure is ibv_reg_dmabuf_mr() returning EINVAL — Failed to register memory ...: Invalid argument [22]. It is not Failed to retrieve dmabuf, not ibv_reg_mr, and not Bad address. Bad address [14] is what the other branch produces (WITH_NVIDIA_PEERMEM=1, plain ibv_reg_mr on a VMM pointer); this report is about the dma-buf branch with WITH_NVIDIA_PEERMEM=0.

Per side, 16 error lines are logged on 0.3.12.post1 against 1944 on 0.3.11.post1, but those two counts are not comparable — registration aborts and rolls back on the first failing chunk, so a differing number of attempts is reached. In both versions the failures are on the large buffers (the KV buffer plus a ~190 MB buffer); buffers smaller than one expandable-segment chunk succeed. The first request then returns HTTP 500 with Failed to get kvcache from prefill instance.

Root cause

RdmaContext::exportDmabuf() derives the dma-buf export range from cuMemGetAddressRange(). For memory allocated through the CUDA virtual memory management API (cuMemCreate + cuMemMap) — which is exactly what PyTorch uses with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — cuMemGetAddressRange() returns only the physical mapping that contains addr, not the surrounding contiguous VA reservation.

PyTorch grows expandable segments in large_segment_size steps, 20 MiB by default (c10/core/AllocatorConfig.h: std::atomic<size_t> large_segment_size_{20971520}; // 20 MB by default). So the exported dma-buf is 20 MiB while the registered buffer is often larger than 1 GiB. The kernel requires offset + length <= dma-buf size, which is violated, and ibv_reg_dmabuf_mr() returns EINVAL.

Current main (and 0.3.12.post1), mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp:

// exportDmabuf(), ~line 433
cuMemGetAddressRange(&allocBase, &allocSize, (CUdeviceptr)addr);
cuMemGetHandleForAddressRange(&dmabuf_fd, allocBase, allocSize,
                              CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0);
out.offset = (uintptr_t)addr - (uintptr_t)allocBase;

// registerMemoryRegionInternal(), ~line 592
mrMeta.mr = ibv_reg_dmabuf_mr(pd_, exp.offset, length, (uintptr_t)addr,
                              exp.fd, access);

Two things make this hard to diagnose:

1. Nothing validates exp.offset + length <= allocSize.
2. The failure log is PLOG(ERROR) << "Failed to register memory " << addr; — it prints neither length, nor exp.offset, nor the exported size, so the log cannot point at the cause.

Captured arguments from a real run

Interposed the actual ibv_reg_dmabuf_mr() call with an LD_PRELOAD library that forwards to the real function unchanged and additionally reads the fd size with lseek(fd, 0, SEEK_END):

offset=8214528  length=1916853120  iova=0x12397fd5800  fd=240
dmabuf_size=20971520   offset+length=1925067648   -> FAIL errno=22 (Invalid argument)

iova % 4096 == offset % 4096 holds, so this is not the page-offset rule; it is purely the size.

The hardware and driver do support the wider export

Standalone probe using only the CUDA driver API and libibverbs (no Mooncake): reserve one VA range with cuMemAddressReserve, map 8 separate cuMemCreate handles into it contiguously, then

cuMemGetHandleForAddressRange(&fd, base, total,
                              CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0);
ibv_reg_dmabuf_mr(pd, 0, total, 0, fd, access);   /* -> succeeds */

Exporting a range that spans several mappings is therefore legal. Only the choice of range is wrong.

The same probe shows the other two combinations behave as expected on this setup: cuMemAlloc + ibv_reg_mr succeeds; VMM memory + ibv_reg_mr (the WITH_NVIDIA_PEERMEM=1 path) fails with EFAULT, which is the other half of what users report.

Enlarging the PyTorch granularity is not a viable workaround

large_segment_size_mb enlarges allocSize, but a buffer can start anywhere inside a segment, so offset_within_segment + length <= segment_size cannot hold for all buffers at any finite segment size. Measured on an 8-GPU prefill instance, KV buffers up to 1.57 GiB, 1944 registrations per side:

large_segment_size_mb	dma-buf size	failed registrations
20 (default)	20 MiB	1944 / 1944
4096	4 GiB	208 / 1944

Observed extremes with 4 GiB segments: max offset 4,269,312,512 (right at the segment end), max offset + length 5,956,546,112. In addition, 4 GiB segments leave about 13.4 GiB reserved but unallocated per GPU, which makes decode-side CUDA graph capture OOM (Tried to allocate 4.00 GiB) at mem_fraction_static 0.979 and 0.95 alike.

Verified fix

Exporting [addr, addr + length) with a page-aligned base — instead of the reported allocation — makes registration succeed with the default 20 MiB granularity:

configuration	failed registrations (prefill / decode)	end-to-end
expandable_segments:True, current main	1944 / —	HTTP 500, Failed to send kv chunk
expandable_segments:True + corrected export range	0 / 0	correct output

Environment: SGLang PD disaggregation with prefill and decode on separate 8-GPU hosts, DeepSeek-V4-Pro, ConnectX-8 RoCE (mlx5_bond_0..7), mooncake-transfer-engine-cuda13 0.3.11.post1, torch 2.11.0+cu130, WITH_NVIDIA_PEERMEM=0, MC_GID_INDEX=3.

Note on #2035 and #2523

#2035 introduced the cuMemGetAddressRange() normalisation to handle caching-allocator sub-allocations, and #2523 restated the assumption as "cuMemGetAddressRange normalised every export to the true allocation base". That assumption holds for cudaMalloc-backed memory but not for VMM-backed memory, which is where this bug lives. A fix should keep the existing behaviour for the cudaMalloc case and only change the range when the reported allocation demonstrably cannot cover the registration.

A pull request follows.

<!-- gh-comment-id:5341823061 --> @hope1262946533 commented on GitHub (Aug 19, 2026): Retested on the latest release — **still fails**. Below are the exact error lines you asked for, and the root cause, which turns out to be #2035 itself rather than its absence. ## Versions tested | version | includes | result | | --- | --- | --- | | `mooncake-transfer-engine-cuda13` **0.3.11.post1** | #2034, #2035, #2066 | fails — 1944 registrations per side, all `EINVAL` | | `mooncake-transfer-engine-cuda13` **0.3.12.post1** (latest on PyPI) | additionally #2416, #2523 | **fails** — same `EINVAL` on the same buffers | So the environment already had all three fixes you listed before this report; 0.3.12.post1 adds #2523 and does not change the outcome. Confirmed the new build was actually loaded (`mooncake.__file__` pointed at the 0.3.12.post1 install, and the error line numbers moved from `rdma_context.cpp:346` to `rdma_context.cpp:588`). ## First error lines on 0.3.12.post1 Prefill side: ``` E0819 11:21:45.193161 147554 rdma_context.cpp:588] Failed to register memory 0x12397fd5800: Invalid argument [22] E0819 11:21:45.193186 147554 rdma_transport.cpp:302] Failed to register memory region with context 0 W0819 11:21:45.193463 147554 rdma_transport.cpp:481] RdmaTransport: Failed to register memory: addr 0x12397fd5800 length 1916853120 W0819 11:21:45.199296 147554 rdma_transport.cpp:481] RdmaTransport: Failed to register memory: addr 0x1009700b000 length 191692800 ``` Decode side is identical in shape (`addr 0x12421e3d000 length 2013598080` and `addr 0x1009700b000 length 201427200`). To answer the specific question: the failure is **`ibv_reg_dmabuf_mr()` returning `EINVAL`** — `Failed to register memory ...: Invalid argument [22]`. It is **not** `Failed to retrieve dmabuf`, **not** `ibv_reg_mr`, and **not** `Bad address`. `Bad address [14]` is what the other branch produces (`WITH_NVIDIA_PEERMEM=1`, plain `ibv_reg_mr` on a VMM pointer); this report is about the dma-buf branch with `WITH_NVIDIA_PEERMEM=0`. Per side, 16 error lines are logged on 0.3.12.post1 against 1944 on 0.3.11.post1, but those two counts are not comparable — registration aborts and rolls back on the first failing chunk, so a differing number of attempts is reached. In both versions the failures are on the large buffers (the KV buffer plus a ~190 MB buffer); buffers smaller than one expandable-segment chunk succeed. The first request then returns HTTP 500 with `Failed to get kvcache from prefill instance`. ## Root cause `RdmaContext::exportDmabuf()` derives the dma-buf export range from `cuMemGetAddressRange()`. For memory allocated through the CUDA virtual memory management API (`cuMemCreate` + `cuMemMap`) — which is exactly what PyTorch uses with `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — `cuMemGetAddressRange()` returns **only the physical mapping that contains `addr`**, not the surrounding contiguous VA reservation. PyTorch grows expandable segments in `large_segment_size` steps, **20 MiB by default** (`c10/core/AllocatorConfig.h`: `std::atomic<size_t> large_segment_size_{20971520}; // 20 MB by default`). So the exported dma-buf is 20 MiB while the registered buffer is often larger than 1 GiB. The kernel requires `offset + length <= dma-buf size`, which is violated, and `ibv_reg_dmabuf_mr()` returns `EINVAL`. Current `main` (and 0.3.12.post1), `mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp`: ```cpp // exportDmabuf(), ~line 433 cuMemGetAddressRange(&allocBase, &allocSize, (CUdeviceptr)addr); cuMemGetHandleForAddressRange(&dmabuf_fd, allocBase, allocSize, CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0); out.offset = (uintptr_t)addr - (uintptr_t)allocBase; // registerMemoryRegionInternal(), ~line 592 mrMeta.mr = ibv_reg_dmabuf_mr(pd_, exp.offset, length, (uintptr_t)addr, exp.fd, access); ``` Two things make this hard to diagnose: 1. Nothing validates `exp.offset + length <= allocSize`. 2. The failure log is `PLOG(ERROR) << "Failed to register memory " << addr;` — it prints neither `length`, nor `exp.offset`, nor the exported size, so the log cannot point at the cause. ## Captured arguments from a real run Interposed the actual `ibv_reg_dmabuf_mr()` call with an `LD_PRELOAD` library that forwards to the real function unchanged and additionally reads the fd size with `lseek(fd, 0, SEEK_END)`: ``` offset=8214528 length=1916853120 iova=0x12397fd5800 fd=240 dmabuf_size=20971520 offset+length=1925067648 -> FAIL errno=22 (Invalid argument) ``` `iova % 4096 == offset % 4096` holds, so this is not the page-offset rule; it is purely the size. ## The hardware and driver do support the wider export Standalone probe using only the CUDA driver API and libibverbs (no Mooncake): reserve one VA range with `cuMemAddressReserve`, map 8 separate `cuMemCreate` handles into it contiguously, then ```c cuMemGetHandleForAddressRange(&fd, base, total, CU_MEM_RANGE_HANDLE_TYPE_DMA_BUF_FD, 0); ibv_reg_dmabuf_mr(pd, 0, total, 0, fd, access); /* -> succeeds */ ``` Exporting a range that spans several mappings is therefore legal. Only the choice of range is wrong. The same probe shows the other two combinations behave as expected on this setup: `cuMemAlloc` + `ibv_reg_mr` succeeds; VMM memory + `ibv_reg_mr` (the `WITH_NVIDIA_PEERMEM=1` path) fails with `EFAULT`, which is the other half of what users report. ## Enlarging the PyTorch granularity is not a viable workaround `large_segment_size_mb` enlarges `allocSize`, but a buffer can start anywhere inside a segment, so `offset_within_segment + length <= segment_size` cannot hold for all buffers at any finite segment size. Measured on an 8-GPU prefill instance, KV buffers up to 1.57 GiB, 1944 registrations per side: | `large_segment_size_mb` | dma-buf size | failed registrations | | --- | --- | --- | | 20 (default) | 20 MiB | 1944 / 1944 | | 4096 | 4 GiB | 208 / 1944 | Observed extremes with 4 GiB segments: max offset 4,269,312,512 (right at the segment end), max `offset + length` 5,956,546,112. In addition, 4 GiB segments leave about 13.4 GiB `reserved but unallocated` per GPU, which makes decode-side CUDA graph capture OOM (`Tried to allocate 4.00 GiB`) at `mem_fraction_static` 0.979 and 0.95 alike. ## Verified fix Exporting `[addr, addr + length)` with a page-aligned base — instead of the reported allocation — makes registration succeed with the **default** 20 MiB granularity: | configuration | failed registrations (prefill / decode) | end-to-end | | --- | --- | --- | | `expandable_segments:True`, current `main` | 1944 / — | HTTP 500, `Failed to send kv chunk` | | `expandable_segments:True` + corrected export range | **0 / 0** | correct output | Environment: SGLang PD disaggregation with prefill and decode on separate 8-GPU hosts, DeepSeek-V4-Pro, ConnectX-8 RoCE (`mlx5_bond_0..7`), `mooncake-transfer-engine-cuda13 0.3.11.post1`, torch 2.11.0+cu130, `WITH_NVIDIA_PEERMEM=0`, `MC_GID_INDEX=3`. ## Note on #2035 and #2523 #2035 introduced the `cuMemGetAddressRange()` normalisation to handle caching-allocator sub-allocations, and #2523 restated the assumption as "`cuMemGetAddressRange` normalised every export to the true allocation base". That assumption holds for `cudaMalloc`-backed memory but not for VMM-backed memory, which is where this bug lives. A fix should keep the existing behaviour for the `cudaMalloc` case and only change the range when the reported allocation demonstrably cannot cover the registration. A pull request follows.

bsakaguchi commented 2026-10-29 21:17:51 +00:00

Author

Owner

Copy link

@enternal111 commented on GitHub (Aug 20, 2026):

same error

<!-- gh-comment-id:5353730515 --> @enternal111 commented on GitHub (Aug 20, 2026): same error

bsakaguchi referenced this issue 2026-10-29 21:29:51 +00:00

[PR #664] [CLOSED] [TransferEngine] Add definition to treat MNNVL batch transfer seperately #1729

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#664

No description provided.