kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1382] [Bug]: start mooncake_store_service failed: Failed to register memory: Cannot allocate memory #446

New issue

Closed

opened 2026-10-29 21:14:38 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:14:38 +00:00

Owner

Originally created by @pigeonsoup on GitHub (Jan 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1382

Originally assigned to: @zhangzuo21 on GitHub.

Bug Report

Description

I am unable to allocate a global_segment_size exceeding approximately 64 GiB when starting the mooncake_store client as a standalone process.

When I set global_segment_size to 80 GiB (~84424509440 bytes), the RDMA memory registration fails with the following error:

E0115 11:22:06.866998 3663955 rdma_context.cpp:257] Failed to register memory 0x7f7186000000: Cannot allocate memory [12]

However, when I reduce global_segment_size to 60 GiB (~64424509440 bytes), everything works correctly.

Note: This issue only occurs on my A100 DGX machine. I have tested the same configuration on a different machine without encountering this problem.

Steps to Reproduce

Step 1: Start mooncake master

$ mooncake_master

Step 2: Start mooncake_store client

$ python -m mooncake.mooncake_store_service --config=~/mooncake_store.json --port=8082 &

Configuration file (mooncake_store.json):

{
    "local_hostname": "localhost",
    "metadata_server": "P2PHANDSHAKE",
    "global_segment_size": 84424509440,
    "local_buffer_size": 1073741824,
    "protocol": "rdma",
    "device_name": "ibp12s0,ibp75s0",
    "master_server_address": "localhost:50051"
}

Failed Log Output (80 GiB)

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0115 11:22:03.135975 3663584 client_metric.cpp:76] Client metrics enabled (default enabled)
I0115 11:22:03.136967 3663584 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I0115 11:22:03.136976 3663584 client.cpp:46] client_id=238537508524411216-13750972002696097672
I0115 11:22:03.136981 3663584 client.cpp:54] Client metrics enabled but reporting disabled (interval=0)
I0115 11:22:03.139430 3663584 client.cpp:373] Storage root directory is not set. persisting data is disabled.
I0115 11:22:03.139565 3663584 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0115 11:22:03.139590 3663584 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12486
I0115 11:22:03.139632 3663584 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:15647
I0115 11:22:03.139782 3663584 client.cpp:268] Transfer engine auto discovery is disabled for protocol: rdma
I0115 11:22:03.139788 3663584 client.cpp:280] Using specified RDMA devices: ibp12s0,ibp75s0
I0115 11:22:03.142277 3663584 client.cpp:290] Topology discovery complete with specified devices. Found 2 HCAs
I0115 11:22:03.142313 3663584 rdma_context.cpp:77] Using SIEVE endpoint store
I0115 11:22:03.172011 3663584 rdma_context.cpp:547] Find best gid index: 0 on ibp75s0/
I0115 11:22:03.173031 3663584 rdma_context.cpp:140] RDMA device: ibp75s0, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:2e
I0115 11:22:03.173046 3663584 rdma_context.cpp:77] Using SIEVE endpoint store
I0115 11:22:03.203779 3663584 rdma_context.cpp:547] Find best gid index: 0 on ibp12s0/
I0115 11:22:03.210212 3663584 rdma_context.cpp:140] RDMA device: ibp12s0, LID: 7, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:f2
I0115 11:22:03.235469 3663584 pybind_client.cpp:196] Registering local memory: 1073741824 bytes
I0115 11:22:03.750491 3663584 pybind_client.cpp:220] Mounting segment: 84424509440 bytes, 84424509440 of 84424509440
E0115 11:22:06.866998 3663955 rdma_context.cpp:257] Failed to register memory 0x7f7186000000: Cannot allocate memory [12]
E0115 11:22:09.215817 3663584 client.cpp:1401] register_local_memory_failed base=0x7f7186000000 size=84424509440, error=-202
E0115 11:22:09.215894 3663584 pybind_client.cpp:235] Failed to mount segment: INVALID_PARAMS
2025-01-15 11:22:09,216 - root - ERROR - Store startup failed: Store initialization failed
2025-01-15 11:22:09,218 - root - ERROR - Service error: Failed to start store service
2025-01-15 11:22:19,840 - root - INFO - Mooncake service stopped

Successful Log Output (60 GiB)

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0115 11:21:21.385442 3662714 client_metric.cpp:76] Client metrics enabled (default enabled)
I0115 11:21:21.386453 3662714 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I0115 11:21:21.386466 3662714 client.cpp:46] client_id=17384060875720772085-9293998721192312501
I0115 11:21:21.386471 3662714 client.cpp:54] Client metrics enabled but reporting disabled (interval=0)
I0115 11:21:21.388849 3662714 client.cpp:373] Storage root directory is not set. persisting data is disabled.
I0115 11:21:21.388901 3662714 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0115 11:21:21.388914 3662714 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 14176
I0115 11:21:21.388936 3662714 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16856
I0115 11:21:21.389061 3662714 client.cpp:268] Transfer engine auto discovery is disabled for protocol: rdma
I0115 11:21:21.389068 3662714 client.cpp:280] Using specified RDMA devices: ibp12s0,ibp75s0
I0115 11:21:21.391410 3662714 client.cpp:290] Topology discovery complete with specified devices. Found 2 HCAs
I0115 11:21:21.391443 3662714 rdma_context.cpp:77] Using SIEVE endpoint store
I0115 11:21:21.420115 3662714 rdma_context.cpp:547] Find best gid index: 0 on ibp75s0/
I0115 11:21:21.422024 3662714 rdma_context.cpp:140] RDMA device: ibp75s0, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:2e
I0115 11:21:21.422053 3662714 rdma_context.cpp:77] Using SIEVE endpoint store
I0115 11:21:21.456758 3662714 rdma_context.cpp:547] Find best gid index: 0 on ibp12s0/
I0115 11:21:21.458775 3662714 rdma_context.cpp:140] RDMA device: ibp12s0, LID: 7, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:f2
I0115 11:21:21.485944 3662714 pybind_client.cpp:196] Registering local memory: 1073741824 bytes
I0115 11:21:21.961967 3662714 pybind_client.cpp:220] Mounting segment: 64424509440 bytes, 64424509440 of 64424509440
2025-01-15 11:21:27,003 - root - INFO - Store service started on localhost
2025-01-15 11:21:27,008 - root - INFO - REST API started on port 8082

Environment

Software

Mooncake version: mooncake_transfer_engine-0.3.8.post1-cp313-cp313-manylinux_2_17_x86_64.manylinux_2_35_x86_64.whl
Kernel version: 5.15.0-161-generic
Python: 3.13

Hardware

Machine: NVIDIA DGX A100
Total memory: 1 TiB (722 GiB available)
InfiniBand devices: 8x Mellanox ConnectX-6 (MT4123)
IB firmware: 20.31.2354
IB link rate: 200 Gb/s (HDR)

System Limits

memlock limit: 132077284 KB (~126 GiB) - sufficient for the requested 80 GiB
vm.max_map_count: 262144
vm.overcommit_memory: 1

InfiniBand Status

$ ibstat
CA 'ibp12s0'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.31.2354
        Hardware version: 0
        Node GUID: 0x043f720300d3acf2
        System image GUID: 0x043f720300d3acf2
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 7
                LMC: 0
                SM lid: 7
                Capability mask: 0x2651e84a
                Port GUID: 0x043f720300d3acf2
                Link layer: InfiniBand
CA 'ibp75s0'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.31.2354
        Hardware version: 0
        Node GUID: 0x0c42a10300ba1b2e
        System image GUID: 0x0c42a10300ba1b2e
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 8
                LMC: 0
                SM lid: 7
                Capability mask: 0x2651e84a
                Port GUID: 0x0c42a10300ba1b2e
                Link layer: InfiniBand

Additional Notes

The memlock ulimit (126 GiB) appears to be sufficient for the requested 80 GiB allocation.
System has ample free memory (722 GiB available).
This issue is specific to this DGX A100 machine - the same configuration works on other machines.
The error occurs during ibv_reg_mr() call in rdma_context.cpp:257.
The failure happens consistently around the 64 GiB boundary.

Possible Causes (Speculation)

This might be related to:

DGX-specific RDMA/GPU memory management constraints
NUMA topology affecting large memory registrations
Some hidden per-device or per-HCA memory registration limit on this hardware

Any guidance on further debugging or known workarounds would be appreciated!

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @pigeonsoup on GitHub (Jan 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1382 Originally assigned to: @zhangzuo21 on GitHub. ### Bug Report ## Description I am unable to allocate a `global_segment_size` exceeding approximately 64 GiB when starting the mooncake_store client as a standalone process. When I set `global_segment_size` to 80 GiB (~84424509440 bytes), the RDMA memory registration fails with the following error: ``` E0115 11:22:06.866998 3663955 rdma_context.cpp:257] Failed to register memory 0x7f7186000000: Cannot allocate memory [12] ``` However, when I reduce `global_segment_size` to 60 GiB (~64424509440 bytes), everything works correctly. **Note:** This issue only occurs on my A100 DGX machine. I have tested the same configuration on a different machine without encountering this problem. ## Steps to Reproduce ### Step 1: Start mooncake master ```bash $ mooncake_master ``` ### Step 2: Start mooncake_store client ```bash $ python -m mooncake.mooncake_store_service --config=~/mooncake_store.json --port=8082 & ``` **Configuration file (`mooncake_store.json`):** ```json { "local_hostname": "localhost", "metadata_server": "P2PHANDSHAKE", "global_segment_size": 84424509440, "local_buffer_size": 1073741824, "protocol": "rdma", "device_name": "ibp12s0,ibp75s0", "master_server_address": "localhost:50051" } ``` ### Failed Log Output (80 GiB) ``` WARNING: Logging before InitGoogleLogging() is written to STDERR I0115 11:22:03.135975 3663584 client_metric.cpp:76] Client metrics enabled (default enabled) I0115 11:22:03.136967 3663584 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I0115 11:22:03.136976 3663584 client.cpp:46] client_id=238537508524411216-13750972002696097672 I0115 11:22:03.136981 3663584 client.cpp:54] Client metrics enabled but reporting disabled (interval=0) I0115 11:22:03.139430 3663584 client.cpp:373] Storage root directory is not set. persisting data is disabled. I0115 11:22:03.139565 3663584 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0115 11:22:03.139590 3663584 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12486 I0115 11:22:03.139632 3663584 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:15647 I0115 11:22:03.139782 3663584 client.cpp:268] Transfer engine auto discovery is disabled for protocol: rdma I0115 11:22:03.139788 3663584 client.cpp:280] Using specified RDMA devices: ibp12s0,ibp75s0 I0115 11:22:03.142277 3663584 client.cpp:290] Topology discovery complete with specified devices. Found 2 HCAs I0115 11:22:03.142313 3663584 rdma_context.cpp:77] Using SIEVE endpoint store I0115 11:22:03.172011 3663584 rdma_context.cpp:547] Find best gid index: 0 on ibp75s0/ I0115 11:22:03.173031 3663584 rdma_context.cpp:140] RDMA device: ibp75s0, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:2e I0115 11:22:03.173046 3663584 rdma_context.cpp:77] Using SIEVE endpoint store I0115 11:22:03.203779 3663584 rdma_context.cpp:547] Find best gid index: 0 on ibp12s0/ I0115 11:22:03.210212 3663584 rdma_context.cpp:140] RDMA device: ibp12s0, LID: 7, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:f2 I0115 11:22:03.235469 3663584 pybind_client.cpp:196] Registering local memory: 1073741824 bytes I0115 11:22:03.750491 3663584 pybind_client.cpp:220] Mounting segment: 84424509440 bytes, 84424509440 of 84424509440 E0115 11:22:06.866998 3663955 rdma_context.cpp:257] Failed to register memory 0x7f7186000000: Cannot allocate memory [12] E0115 11:22:09.215817 3663584 client.cpp:1401] register_local_memory_failed base=0x7f7186000000 size=84424509440, error=-202 E0115 11:22:09.215894 3663584 pybind_client.cpp:235] Failed to mount segment: INVALID_PARAMS 2025-01-15 11:22:09,216 - root - ERROR - Store startup failed: Store initialization failed 2025-01-15 11:22:09,218 - root - ERROR - Service error: Failed to start store service 2025-01-15 11:22:19,840 - root - INFO - Mooncake service stopped ``` ### Successful Log Output (60 GiB) ``` WARNING: Logging before InitGoogleLogging() is written to STDERR I0115 11:21:21.385442 3662714 client_metric.cpp:76] Client metrics enabled (default enabled) I0115 11:21:21.386453 3662714 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I0115 11:21:21.386466 3662714 client.cpp:46] client_id=17384060875720772085-9293998721192312501 I0115 11:21:21.386471 3662714 client.cpp:54] Client metrics enabled but reporting disabled (interval=0) I0115 11:21:21.388849 3662714 client.cpp:373] Storage root directory is not set. persisting data is disabled. I0115 11:21:21.388901 3662714 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0115 11:21:21.388914 3662714 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 14176 I0115 11:21:21.388936 3662714 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16856 I0115 11:21:21.389061 3662714 client.cpp:268] Transfer engine auto discovery is disabled for protocol: rdma I0115 11:21:21.389068 3662714 client.cpp:280] Using specified RDMA devices: ibp12s0,ibp75s0 I0115 11:21:21.391410 3662714 client.cpp:290] Topology discovery complete with specified devices. Found 2 HCAs I0115 11:21:21.391443 3662714 rdma_context.cpp:77] Using SIEVE endpoint store I0115 11:21:21.420115 3662714 rdma_context.cpp:547] Find best gid index: 0 on ibp75s0/ I0115 11:21:21.422024 3662714 rdma_context.cpp:140] RDMA device: ibp75s0, LID: 8, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:0c:42:a1:03:00:ba:1b:2e I0115 11:21:21.422053 3662714 rdma_context.cpp:77] Using SIEVE endpoint store I0115 11:21:21.456758 3662714 rdma_context.cpp:547] Find best gid index: 0 on ibp12s0/ I0115 11:21:21.458775 3662714 rdma_context.cpp:140] RDMA device: ibp12s0, LID: 7, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:04:3f:72:03:00:d3:ac:f2 I0115 11:21:21.485944 3662714 pybind_client.cpp:196] Registering local memory: 1073741824 bytes I0115 11:21:21.961967 3662714 pybind_client.cpp:220] Mounting segment: 64424509440 bytes, 64424509440 of 64424509440 2025-01-15 11:21:27,003 - root - INFO - Store service started on localhost 2025-01-15 11:21:27,008 - root - INFO - REST API started on port 8082 ``` ## Environment ### Software - **Mooncake version:** `mooncake_transfer_engine-0.3.8.post1-cp313-cp313-manylinux_2_17_x86_64.manylinux_2_35_x86_64.whl` - **Kernel version:** 5.15.0-161-generic - **Python:** 3.13 ### Hardware - **Machine:** NVIDIA DGX A100 - **Total memory:** 1 TiB (722 GiB available) - **InfiniBand devices:** 8x Mellanox ConnectX-6 (MT4123) - **IB firmware:** 20.31.2354 - **IB link rate:** 200 Gb/s (HDR) ### System Limits - **memlock limit:** 132077284 KB (~126 GiB) - sufficient for the requested 80 GiB - **vm.max_map_count:** 262144 - **vm.overcommit_memory:** 1 ### InfiniBand Status ``` $ ibstat CA 'ibp12s0' CA type: MT4123 Number of ports: 1 Firmware version: 20.31.2354 Hardware version: 0 Node GUID: 0x043f720300d3acf2 System image GUID: 0x043f720300d3acf2 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 7 LMC: 0 SM lid: 7 Capability mask: 0x2651e84a Port GUID: 0x043f720300d3acf2 Link layer: InfiniBand CA 'ibp75s0' CA type: MT4123 Number of ports: 1 Firmware version: 20.31.2354 Hardware version: 0 Node GUID: 0x0c42a10300ba1b2e System image GUID: 0x0c42a10300ba1b2e Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 8 LMC: 0 SM lid: 7 Capability mask: 0x2651e84a Port GUID: 0x0c42a10300ba1b2e Link layer: InfiniBand ``` ## Additional Notes - The memlock ulimit (126 GiB) appears to be sufficient for the requested 80 GiB allocation. - System has ample free memory (722 GiB available). - This issue is **specific to this DGX A100 machine** - the same configuration works on other machines. - The error occurs during `ibv_reg_mr()` call in `rdma_context.cpp:257`. - The failure happens consistently around the 64 GiB boundary. ## Possible Causes (Speculation) This might be related to: 1. DGX-specific RDMA/GPU memory management constraints 2. NUMA topology affecting large memory registrations 3. Some hidden per-device or per-HCA memory registration limit on this hardware Any guidance on further debugging or known workarounds would be appreciated! ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:14:38 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:14:39 +00:00

Author

Owner

@zhangzuo21 commented on GitHub (Jan 15, 2026):

Thanks for reporting the issue. I've reproduced your problem with the same hardware settings. The issue is caused by ulimit of the system. The default ulimit without sudo is 126G. From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. I've tried to use only one device and it begins to fail when the size exceeds 126G. You can use sudo or change the ulimit to avoid the problem.

@zhangzuo21 commented on GitHub (Jan 15, 2026): Thanks for reporting the issue. I've reproduced your problem with the same hardware settings. The issue is caused by ulimit of the system. The default ulimit without sudo is 126G. From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. I've tried to use only one device and it begins to fail when the size exceeds 126G. You can use sudo or change the ulimit to avoid the problem.

bsakaguchi commented

2026-10-29 21:14:39 +00:00

Author

Owner

@stmatengss commented on GitHub (Jan 18, 2026):

Thanks for reporting the issue. I've reproduced your problem with the same hardware settings. The issue is caused by ulimit of the system. The default ulimit without sudo is 126G. From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. I've tried to use only one device and it begins to fail when the size exceeds 126G. You can use sudo or change the ulimit to avoid the problem.

You can update this case to trouble shooting doc. Thx!

@stmatengss commented on GitHub (Jan 18, 2026): > Thanks for reporting the issue. I've reproduced your problem with the same hardware settings. The issue is caused by ulimit of the system. The default ulimit without sudo is 126G. From your settings we can see that you used two InfiniBand devices and tried to register 80G memory, in mooncake transfer engine, the settings led to register 80G one time for each device, accumulating to 160G registered, which is greater than the limit. I've tried to use only one device and it begins to fail when the size exceeds 126G. You can use sudo or change the ulimit to avoid the problem. You can update this case to trouble shooting doc. Thx!