[GH-ISSUE #3639] [TE] Unconditional ibv_fork_init() makes every register/unregister cycle permanently split VMAs (MADV_DONTFORK never reverted) → vm.max_map_count exhaustion #929

New issue

Closed

opened 2026-10-29 21:21:00 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:21:00 +00:00

Owner

Originally created by @thincal on GitHub (Aug 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3639

Summary

RdmaContext unconditionally calls ibv_fork_init() at engine startup (and the new TENT transport does the same). Once verbs fork protection is active, every ibv_reg_mr performed through the transfer engine issues madvise(range, MADV_DONTFORK) on the registered range, and the corresponding unregister never issues MADV_DOFORK. The kernel therefore tags those pages VM_DONTCOPY, which splits the containing VMA and makes it permanently unmergeable with its neighbors.

For workloads that register/unregister buffers at high frequency (per-request buffer registration rather than a registered-once pool), this is a monotonic leak of VMA boundaries: vm.max_map_count is eventually exhausted, after which ibv_reg_mr and ibv_create_qp fail with ENOMEM — silently, with nothing in dmesg — and the host application typically falls over.

Impact (production incident)

A production multimodal serving deployment (SGLang, EPD disaggregated encoder→prefill transfer of per-request image/video embeddings, i.e. one register/transfer/unregister cycle per embedding) hit exactly this:

Each encoder process registered/unregistered every embedding tensor. VMA count ratcheted up at a sustained 17–70 new split VMAs per minute per process (measured), never shrinking.
Once the per-process map count hit the limit, every RDMA operation failed: Python side mooncake register_memory failed (ret=-202 ERR_CONTEXT), C++ side Failed to register memory 0x...: Cannot allocate memory [12] (rdma_context.cpp) and Failed to create QP: Cannot allocate memory (rdma_endpoint.cpp), followed by a crash loop.
dmesg was completely clean — zero mlx5 messages, no CREATE_MKEY no resources, no page-allocation failures. The ENOMEM originates in userspace: ibv_create_qp needs to mmap doorbell/UAR pages and ibv_reg_mr's backing allocation needs fresh mmaps, both of which fail once vm.max_map_count is reached. This makes the failure very hard to attribute without knowing the mechanism.
Raising vm.max_map_count (65530 default → 1M) only delays the cliff (~days at our traffic rate); it does not stop the ratchet.

Root cause

Unconditional fork protection, no switch:

v0.3.10.post1, mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp (RdmaContext ctor):
```
static std::once_flag g_once_flag;
auto fork_init = []() {
    int ret = ibv_fork_init();
    if (ret) PLOG(ERROR) << "RDMA context setup failed: fork compatibility";
};
std::call_once(g_once_flag, fork_init);
```
Same in current main / v0.3.12.post1 TENT path: mooncake-transfer-engine/tent/src/transport/rdma/context.cpp (~L298): int ret = verbs_.ibv_fork_init();

ibv_fork_init() is process-global: once any engine context exists, every ibv_reg_mr in the process (mooncake's and any other verbs user's) madvises its range MADV_DONTFORK.
Deregistration does not undo it. RdmaContext::unregisterMemoryRegion() destroys the MR but never calls madvise(addr, len, MADV_DOFORK). VM_DONTCOPY persists; even after the memory is freed and recycled by the allocator, the VMA split boundaries stay, and the next registration at a different size/offset splits further — a one-way ratchet. (Sizes vary per request in real workloads; with fully static sizes/addresses it would plateau, which is why this escapes simple benchmarks.)

Evidence collected on the affected host

strace -f -e trace=%memory -k on the encoder process: one madvise(0x..., <registered-size>, MADV_DONTFORK) per engine.register() call, issued from the threads performing the registration.
/proc/<pid>/smaps: the split VMA families (exact sizes = the registered embedding sizes) carry the dc flag — e.g. VmFlags: rd wr mr mw me dc ac sd — on ~91k VMAs in one process; the remaining split pieces (recycled memory) keep their boundaries without the flag.
/proc/<pid>/maps line count grows monotonically with traffic and never drops; the dominant entries are two exact-size families matching the registered buffer sizes (36,864 B and 147,456 B in our case, ~88k each after ~13h).
Clean dmesg as described above (rules out firmware MKEY exhaustion, which was our initial hypothesis from the troubleshooting docs).

Minimal reproduction

# pip install mooncake-transfer-engine torch
# Single NIC is enough: ibv_fork_init() is process-global and the VMA split
# happens inside ibv_reg_mr regardless of context count. No transfer is
# needed -- the split occurs at registration time; unregister simply never
# undoes it.
import random, torch
from mooncake.engine import TransferEngine

eng = TransferEngine()
assert eng.initialize("127.0.0.1:15000", "P2PHANDSHAKE", "rdma", "") == 0  # fork init happens here

base = sum(1 for _ in open("/proc/self/maps"))
for i in range(1, 20001):
    n = random.randrange(1 << 16, 1 << 23)        # continuously varying sizes ...
    t = torch.empty(n, dtype=torch.uint8)          # ... on allocator-churned addresses
    addr = t.data_ptr()
    assert eng.register_memory(addr, n) == 0
    assert eng.unregister_memory(addr) == 0
    if i % 2000 == 0:
        cur = sum(1 for _ in open("/proc/self/maps"))
        print(f"{i:6d} cycles: maps {base} -> {cur}  (+{cur - base})")

Watch the /proc/self/maps line count: it ratchets up with register/unregister cycles and never comes back down. The essential ingredients are register→unregister cycles of varying sizes on varying addresses; with constant sizes/addresses the split count plateaus, which is why simple benchmarks miss it.

Suggested fixes

Restore mergeability on unregister: in unregisterMemoryRegion (rdma_transport and the TENT path), call madvise(addr, length, MADV_DOFORK) after ibv_dereg_mr so the kernel can clear VM_DONTCOPY and re-merge the split pieces. This keeps fork protection semantics for live MRs while making the churn reversible.
Make fork protection opt-in or opt-out (e.g. MC_IB_FORK_PROTECTION=0/1): integrators that never fork (long-lived inference servers are the common case for the transfer engine) currently pay permanent address-space fragmentation with no way to decline.
Optionally, document the precise vm.max_map_count interaction in the troubleshooting guide — today the doc lists raising it as a workaround for MKEY exhaustion, but the mechanism above is a distinct failure mode with the same two error strings (Failed to register memory ... Resource temporarily unavailable, Failed to create QP: Cannot allocate memory) and no firmware involvement.

Workaround we adopted

Register-once buffer pools (never unregister): transfer out of a fixed set of pooled, permanently registered buffers. VMA count plateaus at the pool size instead of growing with traffic. (SGLang side: the EPD encoder sender now copies each embedding into a registered-once pool buffer, mirroring what the receiver already did.)

Originally created by @thincal on GitHub (Aug 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3639 ## Summary `RdmaContext` unconditionally calls `ibv_fork_init()` at engine startup (and the new TENT transport does the same). Once verbs fork protection is active, **every `ibv_reg_mr` performed through the transfer engine issues `madvise(range, MADV_DONTFORK)` on the registered range, and the corresponding `unregister` never issues `MADV_DOFORK`**. The kernel therefore tags those pages `VM_DONTCOPY`, which splits the containing VMA and makes it permanently unmergeable with its neighbors. For workloads that register/unregister buffers at high frequency (per-request buffer registration rather than a registered-once pool), this is a monotonic leak of VMA boundaries: **`vm.max_map_count` is eventually exhausted**, after which `ibv_reg_mr` and `ibv_create_qp` fail with `ENOMEM` — silently, with nothing in dmesg — and the host application typically falls over. ## Impact (production incident) A production multimodal serving deployment (SGLang, EPD disaggregated encoder→prefill transfer of per-request image/video embeddings, i.e. one register/transfer/unregister cycle per embedding) hit exactly this: * Each encoder process registered/unregistered every embedding tensor. VMA count ratcheted up at a sustained **17–70 new split VMAs per minute** per process (measured), never shrinking. * Once the per-process map count hit the limit, every RDMA operation failed: Python side `mooncake register_memory failed (ret=-202 ERR_CONTEXT)`, C++ side `Failed to register memory 0x...: Cannot allocate memory [12]` (`rdma_context.cpp`) and `Failed to create QP: Cannot allocate memory` (`rdma_endpoint.cpp`), followed by a crash loop. * **dmesg was completely clean** — zero `mlx5` messages, no `CREATE_MKEY no resources`, no page-allocation failures. The `ENOMEM` originates in userspace: `ibv_create_qp` needs to mmap doorbell/UAR pages and `ibv_reg_mr`'s backing allocation needs fresh mmaps, both of which fail once `vm.max_map_count` is reached. This makes the failure very hard to attribute without knowing the mechanism. * Raising `vm.max_map_count` (65530 default → 1M) only delays the cliff (~days at our traffic rate); it does not stop the ratchet. ## Root cause 1. Unconditional fork protection, no switch: `v0.3.10.post1`, `mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp` (RdmaContext ctor): ```cpp static std::once_flag g_once_flag; auto fork_init = []() { int ret = ibv_fork_init(); if (ret) PLOG(ERROR) << "RDMA context setup failed: fork compatibility"; }; std::call_once(g_once_flag, fork_init); ``` Same in current main / v0.3.12.post1 TENT path: `mooncake-transfer-engine/tent/src/transport/rdma/context.cpp` (~L298): `int ret = verbs_.ibv_fork_init();` `ibv_fork_init()` is process-global: once any engine context exists, **every** `ibv_reg_mr` in the process (mooncake's and any other verbs user's) madvises its range `MADV_DONTFORK`. 2. Deregistration does not undo it. `RdmaContext::unregisterMemoryRegion()` destroys the MR but never calls `madvise(addr, len, MADV_DOFORK)`. `VM_DONTCOPY` persists; even after the memory is freed and recycled by the allocator, the VMA split boundaries stay, and the next registration at a different size/offset splits further — a one-way ratchet. (Sizes vary per request in real workloads; with fully static sizes/addresses it would plateau, which is why this escapes simple benchmarks.) ## Evidence collected on the affected host * `strace -f -e trace=%memory -k` on the encoder process: one `madvise(0x..., <registered-size>, MADV_DONTFORK)` per `engine.register()` call, issued from the threads performing the registration. * `/proc/<pid>/smaps`: the split VMA families (exact sizes = the registered embedding sizes) carry the `dc` flag — e.g. `VmFlags: rd wr mr mw me dc ac sd` — on ~91k VMAs in one process; the remaining split pieces (recycled memory) keep their boundaries without the flag. * `/proc/<pid>/maps` line count grows monotonically with traffic and never drops; the dominant entries are two exact-size families matching the registered buffer sizes (36,864 B and 147,456 B in our case, ~88k each after ~13h). * Clean dmesg as described above (rules out firmware MKEY exhaustion, which was our initial hypothesis from the troubleshooting docs). ## Minimal reproduction ```python # pip install mooncake-transfer-engine torch # Single NIC is enough: ibv_fork_init() is process-global and the VMA split # happens inside ibv_reg_mr regardless of context count. No transfer is # needed -- the split occurs at registration time; unregister simply never # undoes it. import random, torch from mooncake.engine import TransferEngine eng = TransferEngine() assert eng.initialize("127.0.0.1:15000", "P2PHANDSHAKE", "rdma", "") == 0 # fork init happens here base = sum(1 for _ in open("/proc/self/maps")) for i in range(1, 20001): n = random.randrange(1 << 16, 1 << 23) # continuously varying sizes ... t = torch.empty(n, dtype=torch.uint8) # ... on allocator-churned addresses addr = t.data_ptr() assert eng.register_memory(addr, n) == 0 assert eng.unregister_memory(addr) == 0 if i % 2000 == 0: cur = sum(1 for _ in open("/proc/self/maps")) print(f"{i:6d} cycles: maps {base} -> {cur} (+{cur - base})") ``` Watch the `/proc/self/maps` line count: it ratchets up with register/unregister cycles and never comes back down. The essential ingredients are register→unregister cycles of **varying sizes** on **varying addresses**; with constant sizes/addresses the split count plateaus, which is why simple benchmarks miss it. ## Suggested fixes 1. **Restore mergeability on unregister**: in `unregisterMemoryRegion` (rdma_transport and the TENT path), call `madvise(addr, length, MADV_DOFORK)` after `ibv_dereg_mr` so the kernel can clear `VM_DONTCOPY` and re-merge the split pieces. This keeps fork protection semantics for live MRs while making the churn reversible. 2. **Make fork protection opt-in or opt-out** (e.g. `MC_IB_FORK_PROTECTION=0/1`): integrators that never fork (long-lived inference servers are the common case for the transfer engine) currently pay permanent address-space fragmentation with no way to decline. 3. Optionally, document the precise `vm.max_map_count` interaction in the troubleshooting guide — today the doc lists raising it as a workaround for MKEY exhaustion, but the mechanism above is a distinct failure mode with the same two error strings (`Failed to register memory ... Resource temporarily unavailable`, `Failed to create QP: Cannot allocate memory`) and no firmware involvement. ## Workaround we adopted Register-once buffer pools (never unregister): transfer out of a fixed set of pooled, permanently registered buffers. VMA count plateaus at the pool size instead of growing with traffic. (SGLang side: the EPD encoder sender now copies each embedding into a registered-once pool buffer, mirroring what the receiver already did.)

bsakaguchi closed this issue

2026-10-29 21:21:00 +00:00

bsakaguchi commented

2026-10-29 21:21:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 24, 2026):

Thanks for opening this issue, @thincal!

Field	Value
Issue	#3639
GitHub user ID	`1206509`
Reporter	@thincal

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 24, 2026): Thanks for opening this issue, @thincal! | Field | Value | |-------|-------| | **Issue** | #3639 | | **GitHub user ID** | `1206509` | | **Reporter** | @thincal | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:21:01 +00:00

Author

Owner

@thincal commented on GitHub (Aug 25, 2026):

And some other findings FYI:

Before calling ibv_fork_init(), query ibv_is_fork_initialized() when the symbol is available.

If it returns IBV_FORK_UNNEEDED, do not call ibv_fork_init(). Fork protection is unnecessary on this verbs/kernel stack, and skipping it avoids the MADV_DONTFORK VMA-splitting path entirely.
If it returns others or the symbol is unavailable, call ibv_fork_init() as before.

@thincal commented on GitHub (Aug 25, 2026): And some other findings FYI: Before calling `ibv_fork_init()`, query `ibv_is_fork_initialized()` when the symbol is available. - If it returns `IBV_FORK_UNNEEDED`, do not call `ibv_fork_init()`. Fork protection is unnecessary on this verbs/kernel stack, and skipping it avoids the `MADV_DONTFORK` VMA-splitting path entirely. - If it returns others or the symbol is unavailable, call `ibv_fork_init()` as before.

bsakaguchi referenced this issue

2026-10-29 21:31:17 +00:00

[PR #929] [MERGED] feat: add batch_put_from_multi_buffers #1907