[GH-ISSUE #848] [Feature Request]: Mooncake Store Speed up large-size segment mounting #256

New issue

Closed

opened 2026-10-29 21:11:37 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented 2026-10-29 21:11:37 +00:00

Owner

Copy link

Originally created by @xiaguan on GitHub (Sep 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/848

Originally assigned to: @staryxchen on GitHub.

Describe your feature request

We've noticed that mounting very large memory segments (e.g., at the TB scale) can take several minutes, which hinders practical deployment.

The overhead mainly comes from two parts:

• Actual physical allocation, where memset (or equivalent) might benefit from parallelization to speed it up.
  Registration with RDMA — this could be accelerated by using pre-allocated large pages (e.g., 2MB or 1GB page sizes), which might also improve runtime performance.

• Using large pages and parallel init could significantly reduce mounting time for such large segments.

Suggestions :

• Consider supporting libhugetlbfs or explicit mmap with hugepage hints.
• Maybe add a config flag to enable parallel zero-initialization for big segments.

Also, if we’re adding more environment variables, it feels like we’re piling up too many already — makes the user experience pretty clunky. Maybe it’s time we support a builder pattern in the pyclient for more readable and maintainable config construction?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @xiaguan on GitHub (Sep 17, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/848 Originally assigned to: @staryxchen on GitHub. ### Describe your feature request We've noticed that mounting very large memory segments (e.g., at the TB scale) can take several minutes, which hinders practical deployment. The overhead mainly comes from two parts: * Actual physical allocation, where memset (or equivalent) might benefit from parallelization to speed it up. Registration with RDMA — this could be accelerated by using pre-allocated large pages (e.g., 2MB or 1GB page sizes), which might also improve runtime performance. * Using large pages and parallel init could significantly reduce mounting time for such large segments. Suggestions : * Consider supporting libhugetlbfs or explicit mmap with hugepage hints. * Maybe add a config flag to enable parallel zero-initialization for big segments. Also, if we’re adding more environment variables, it feels like we’re piling up too many already — makes the user experience pretty clunky. Maybe it’s time we support a builder pattern in the pyclient for more readable and maintainable config construction? ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:11:37 +00:00

• closed this issue
• added the
  enhancement
  label

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@staryxchen commented on GitHub (Sep 17, 2025):

Hi @xiaguan
We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later.

<!-- gh-comment-id:3301848315 --> @staryxchen commented on GitHub (Sep 17, 2025): Hi @xiaguan We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later.

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@staryxchen commented on GitHub (Sep 17, 2025):

Hi @xiaguan We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later.

Hi @xiaguan @stmatengss
I have submitted the PR #855 , PTAL

<!-- gh-comment-id:3302986991 --> @staryxchen commented on GitHub (Sep 17, 2025): > Hi [@xiaguan](https://github.com/xiaguan) We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later. Hi @xiaguan @stmatengss I have submitted the PR #855 , PTAL

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Sep 18, 2025):

• For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating
  first‑touch minor faults for later reads.
• This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages:
  • touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s)
  • touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s)
  • memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s

<!-- gh-comment-id:3305245483 --> @xiaguan commented on GitHub (Sep 18, 2025): - For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating first‑touch minor faults for later reads. - This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages: - touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s) - touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s) - memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@staryxchen commented on GitHub (Sep 18, 2025):

• For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating
  first‑touch minor faults for later reads.

• This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages:

  • touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s)
  • touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s)
  • memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s

Perhaps we could touch-read all memory before calling ibv_reg_mr in the register_memory function? However, I'm concerned that when memory is insufficient, this might cause the process to be killed due to OOM. Since the stack belongs to functions within the Mooncake Transfer Engine, this could lead to some misunderstandings.

<!-- gh-comment-id:3305288400 --> @staryxchen commented on GitHub (Sep 18, 2025): > * For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating > first‑touch minor faults for later reads. > * This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages: > > * touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s) > * touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s) > * memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s Perhaps we could touch-read all memory before calling `ibv_reg_mr` in the `register_memory` function? However, I'm concerned that when memory is insufficient, this might cause the process to be killed due to OOM. Since the stack belongs to functions within the Mooncake Transfer Engine, this could lead to some misunderstandings.

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Sep 18, 2025):

Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway.

<!-- gh-comment-id:3305807322 --> @xiaguan commented on GitHub (Sep 18, 2025): Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway.

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@staryxchen commented on GitHub (Sep 18, 2025):

Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway.

After further consideration, I believe implicit pre-allocated memory within mooncake remains feasible, as ibv_reg_mr also pre-allocate memory. Since not all applications preallocate memory, enabling parallel registration without prior allocation could actually degrade performance.

<!-- gh-comment-id:3306345359 --> @staryxchen commented on GitHub (Sep 18, 2025): > Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway. After further consideration, I believe implicit pre-allocated memory within mooncake remains feasible, as `ibv_reg_mr ` also pre-allocate memory. Since not all applications preallocate memory, enabling parallel registration without prior allocation could actually degrade performance.

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Sep 23, 2025):

I made a series of benchmarks, the results of which may give some insights into how to speed up the mounting.

In all the benchmarks:

• The page size is 4KB
• The mmap, which is used to trigger page fault when allocating the memory, is revoked as the following code:

void* ptr = mmap(nullptr, size, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);

First, register 4 GB segment:

No pre-touch:
register time taken: 2550 ms

Pre-touch with 1 thread:
Touch time taken: 1884 ms
register time taken: 1562 ms

Pre-touch with 4 threads:
Touch time taken: 1869 ms
register time taken: 1373 ms

mmap
Alloc time taken: 1261 ms
register time taken: 1463 ms

Next, register 40 GB segment:

No pre-touch:
register time taken: 21825 ms

Pre-touch with 1 thread:
Touch time taken: 16318 ms
register time taken: 12898 ms

Pre-touch with 4 threads:
Touch time taken: 4996 ms
register time taken: 13565 ms

mmap:
Alloc time taken: 11287 ms
register time taken: 12672 ms

Register 160 GB segment:

No pre-touch:
register time taken: 76314 ms

Pre-touch with 1 thread:
Touch time taken: 64724 ms
register time taken: 35149 ms

Pre-touch with 4 threads:
Touch time taken: 23619 ms
register time taken: 36703 ms

Pre-touch with 16 threads:
Touch time taken: 17830 ms
register time taken: 36892 ms

mmap:
Alloc time taken: 44055 ms
register time taken: 35009 ms

From the above data, we can observe:

• With only a single thread, the mmap method is faster than pre-touching, since pre-touching repeatedly switches between user mode and kernel mode. In fact, in this case, directly registering the memory without either mmap or pre-touching is the fastest approach.
• When the segment is sufficiently large, multi-threaded pre-touching becomes faster, as it can fully utilize the available memory bandwidth.
• However, using too many threads does not improve performance, since the process becomes memory-bandwidth bound once a certain number of threads is reached.

Conclusion:

• Skip pre-touching when the segment size is small or when only a few threads are available.
• Otherwise, use a fixed number of threads for pre-touching.

<!-- gh-comment-id:3323125526 --> @ykwd commented on GitHub (Sep 23, 2025): I made a series of benchmarks, the results of which may give some insights into how to speed up the mounting. In all the benchmarks: - The page size is 4KB - The mmap, which is used to trigger page fault when allocating the memory, is revoked as the following code: ```c++ void* ptr = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0); ``` **First, register 4 GB segment:** No pre-touch: register time taken: 2550 ms Pre-touch with 1 thread: Touch time taken: 1884 ms register time taken: 1562 ms Pre-touch with 4 threads: Touch time taken: 1869 ms register time taken: 1373 ms mmap Alloc time taken: 1261 ms register time taken: 1463 ms **Next, register 40 GB segment:** No pre-touch: register time taken: 21825 ms Pre-touch with 1 thread: Touch time taken: 16318 ms register time taken: 12898 ms Pre-touch with 4 threads: Touch time taken: 4996 ms register time taken: 13565 ms mmap: Alloc time taken: 11287 ms register time taken: 12672 ms **Register 160 GB segment:** No pre-touch: register time taken: 76314 ms Pre-touch with 1 thread: Touch time taken: 64724 ms register time taken: 35149 ms Pre-touch with 4 threads: Touch time taken: 23619 ms register time taken: 36703 ms Pre-touch with 16 threads: Touch time taken: 17830 ms register time taken: 36892 ms mmap: Alloc time taken: 44055 ms register time taken: 35009 ms From the above data, we can observe: * With only a single thread, the `mmap` method is faster than pre-touching, since pre-touching repeatedly switches between user mode and kernel mode. In fact, in this case, directly registering the memory without either `mmap` or pre-touching is the fastest approach. * When the segment is sufficiently large, multi-threaded pre-touching becomes faster, as it can fully utilize the available memory bandwidth. * However, using too many threads does not improve performance, since the process becomes memory-bandwidth bound once a certain number of threads is reached. **Conclusion:** * Skip pre-touching when the segment size is small or when only a few threads are available. * Otherwise, use a fixed number of threads for pre-touching.

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Sep 23, 2025):

The benchmark of RdmaTransport::registerLocalMemory is conducted on a server with 8 mlx5 RDMA NICs.

The register memory region time is recorded for each NIC.

Register 4 GB segment:

Without pre-touch
Register memory region time taken: 1307 ms
Register memory region time taken: 107 ms
Register memory region time taken: 106 ms
Register memory region time taken: 105 ms
Register memory region time taken: 103 ms
Register memory region time taken: 110 ms
Register memory region time taken: 105 ms
Register memory region time taken: 107 ms
Get memory location time taken: 216 ms
Add local memory buffer time taken: 1 ms
Total time taken: 2273 ms

With pre-touch:
Register memory region time taken: 161 ms
Register memory region time taken: 159 ms
Register memory region time taken: 159 ms
Register memory region time taken: 158 ms
Register memory region time taken: 159 ms
Register memory region time taken: 161 ms
Register memory region time taken: 157 ms
Register memory region time taken: 155 ms
Get memory location time taken: 265 ms
Add local memory buffer time taken: 1 ms
Total time taken: 1541 ms

Without pre-touch, parallel register:
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7643 ms
Register memory region time taken: 7647 ms
Register memory region time taken: 7644 ms
Register memory region time taken: 7646 ms
Get memory location time taken: 266 ms
Add local memory buffer time taken: 1 ms
Total time taken: 7925 ms

With pre-touch, parallel register:
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Get memory location time taken: 228 ms
Add local memory buffer time taken: 2 ms
Total time taken: 438 ms

Register 40 GB segment:

Without pre-touch
Register memory region time taken: 12045 ms
Register memory region time taken: 1082 ms
Register memory region time taken: 1084 ms
Register memory region time taken: 1090 ms
Register memory region time taken: 1072 ms
Register memory region time taken: 1105 ms
Register memory region time taken: 1220 ms
Register memory region time taken: 1213 ms
Get memory location time taken: 1636 ms
Add local memory buffer time taken: 1 ms
Total time taken: 21634 ms

With pre-touch:
Register memory region time taken: 1339 ms
Register memory region time taken: 1257 ms
Register memory region time taken: 1245 ms
Register memory region time taken: 1246 ms
Register memory region time taken: 1261 ms
Register memory region time taken: 1281 ms
Register memory region time taken: 1282 ms
Register memory region time taken: 1253 ms
Get memory location time taken: 1851 ms
Add local memory buffer time taken: 1 ms
Total time taken: 12083 ms

Without pre-touch, parallel register:
Register memory region time taken: 85478 ms
Register memory region time taken: 85479 ms
Register memory region time taken: 85484 ms
Register memory region time taken: 85493 ms
Register memory region time taken: 85494 ms
Register memory region time taken: 85499 ms
Register memory region time taken: 85498 ms
Register memory region time taken: 85502 ms
Get memory location time taken: 1825 ms
Add local memory buffer time taken: 1 ms
Total time taken: 87416 ms

With pre-touch, parallel register:
Register memory region time taken: 1665 ms
Register memory region time taken: 1667 ms
Register memory region time taken: 1676 ms
Register memory region time taken: 1680 ms
Register memory region time taken: 1681 ms
Register memory region time taken: 1684 ms
Register memory region time taken: 1689 ms
Register memory region time taken: 1704 ms
Get memory location time taken: 1878 ms
Add local memory buffer time taken: 2 ms
Total time taken: 3651 ms

Register 160 GB segment:

Without pre-touch
Register memory region time taken: 46193 ms
Register memory region time taken: 3310 ms
Register memory region time taken: 3308 ms
Register memory region time taken: 3311 ms
Register memory region time taken: 3278 ms
Register memory region time taken: 3246 ms
Register memory region time taken: 3246 ms
Register memory region time taken: 3269 ms
Get memory location time taken: 5805 ms
Add local memory buffer time taken: 1 ms
Total time taken: 75290 ms

With pre-touch:
Register memory region time taken: 3608 ms
Register memory region time taken: 3517 ms
Register memory region time taken: 3490 ms
Register memory region time taken: 3476 ms
Register memory region time taken: 3488 ms
Register memory region time taken: 3491 ms
Register memory region time taken: 3519 ms
Register memory region time taken: 3481 ms
Get memory location time taken: 5869 ms
Add local memory buffer time taken: 1 ms
Total time taken: 34223 ms

Without pre-touch, parallel register:
Not test

With pre-touch, parallel register:
Register memory region time taken: 4052 ms
Register memory region time taken: 4146 ms
Register memory region time taken: 4200 ms
Register memory region time taken: 4214 ms
Register memory region time taken: 4214 ms
Register memory region time taken: 4269 ms
Register memory region time taken: 4322 ms
Register memory region time taken: 4322 ms
Get memory location time taken: 5674 ms
Add local memory buffer time taken: 1 ms
Total time taken: 10261 ms

From the above data, we can observe:

• Without pre-touching, registering on the first NIC takes a very long time due to page faults. In contrast, with pre-touching enabled, registering across all NICs takes roughly the same amount of time.
• Without pre-touching, concurrent registration slows down significantly—even much slower than sequential registration. In contrast, with pre-touching, concurrent registration can greatly reduce the total time.

Conclusion:

• Perform concurrent registration only when pre-touching is used.

Combining this with the earlier pre-touching tests, we can conclude two practical strategies:

• When the segment size is small or only a few threads are available, skip pre-touching and register sequentially.
• Otherwise, use a fixed number of threads for pre-touching, and then use num_nic + 1 threads to concurrently register the memory region and get memory locations.

cc. @alogfans (Background: mooncake store needs to register hundreds or even 1TB memory; it will take several minutes to register. We want to speed up this process.)

<!-- gh-comment-id:3323274931 --> @ykwd commented on GitHub (Sep 23, 2025): The benchmark of `RdmaTransport::registerLocalMemory` is conducted on a server with 8 mlx5 RDMA NICs. The register memory region time is recorded for each NIC. **Register 4 GB segment:** Without pre-touch Register memory region time taken: 1307 ms Register memory region time taken: 107 ms Register memory region time taken: 106 ms Register memory region time taken: 105 ms Register memory region time taken: 103 ms Register memory region time taken: 110 ms Register memory region time taken: 105 ms Register memory region time taken: 107 ms Get memory location time taken: 216 ms Add local memory buffer time taken: 1 ms Total time taken: 2273 ms With pre-touch: Register memory region time taken: 161 ms Register memory region time taken: 159 ms Register memory region time taken: 159 ms Register memory region time taken: 158 ms Register memory region time taken: 159 ms Register memory region time taken: 161 ms Register memory region time taken: 157 ms Register memory region time taken: 155 ms Get memory location time taken: 265 ms Add local memory buffer time taken: 1 ms Total time taken: 1541 ms Without pre-touch, parallel register: Register memory region time taken: 7645 ms Register memory region time taken: 7645 ms Register memory region time taken: 7645 ms Register memory region time taken: 7645 ms Register memory region time taken: 7643 ms Register memory region time taken: 7647 ms Register memory region time taken: 7644 ms Register memory region time taken: 7646 ms Get memory location time taken: 266 ms Add local memory buffer time taken: 1 ms Total time taken: 7925 ms With pre-touch, parallel register: Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Register memory region time taken: 202 ms Get memory location time taken: 228 ms Add local memory buffer time taken: 2 ms Total time taken: 438 ms **Register 40 GB segment:** Without pre-touch Register memory region time taken: 12045 ms Register memory region time taken: 1082 ms Register memory region time taken: 1084 ms Register memory region time taken: 1090 ms Register memory region time taken: 1072 ms Register memory region time taken: 1105 ms Register memory region time taken: 1220 ms Register memory region time taken: 1213 ms Get memory location time taken: 1636 ms Add local memory buffer time taken: 1 ms Total time taken: 21634 ms With pre-touch: Register memory region time taken: 1339 ms Register memory region time taken: 1257 ms Register memory region time taken: 1245 ms Register memory region time taken: 1246 ms Register memory region time taken: 1261 ms Register memory region time taken: 1281 ms Register memory region time taken: 1282 ms Register memory region time taken: 1253 ms Get memory location time taken: 1851 ms Add local memory buffer time taken: 1 ms Total time taken: 12083 ms Without pre-touch, parallel register: Register memory region time taken: 85478 ms Register memory region time taken: 85479 ms Register memory region time taken: 85484 ms Register memory region time taken: 85493 ms Register memory region time taken: 85494 ms Register memory region time taken: 85499 ms Register memory region time taken: 85498 ms Register memory region time taken: 85502 ms Get memory location time taken: 1825 ms Add local memory buffer time taken: 1 ms Total time taken: 87416 ms With pre-touch, parallel register: Register memory region time taken: 1665 ms Register memory region time taken: 1667 ms Register memory region time taken: 1676 ms Register memory region time taken: 1680 ms Register memory region time taken: 1681 ms Register memory region time taken: 1684 ms Register memory region time taken: 1689 ms Register memory region time taken: 1704 ms Get memory location time taken: 1878 ms Add local memory buffer time taken: 2 ms Total time taken: 3651 ms **Register 160 GB segment:** Without pre-touch Register memory region time taken: 46193 ms Register memory region time taken: 3310 ms Register memory region time taken: 3308 ms Register memory region time taken: 3311 ms Register memory region time taken: 3278 ms Register memory region time taken: 3246 ms Register memory region time taken: 3246 ms Register memory region time taken: 3269 ms Get memory location time taken: 5805 ms Add local memory buffer time taken: 1 ms Total time taken: 75290 ms With pre-touch: Register memory region time taken: 3608 ms Register memory region time taken: 3517 ms Register memory region time taken: 3490 ms Register memory region time taken: 3476 ms Register memory region time taken: 3488 ms Register memory region time taken: 3491 ms Register memory region time taken: 3519 ms Register memory region time taken: 3481 ms Get memory location time taken: 5869 ms Add local memory buffer time taken: 1 ms Total time taken: 34223 ms Without pre-touch, parallel register: Not test With pre-touch, parallel register: Register memory region time taken: 4052 ms Register memory region time taken: 4146 ms Register memory region time taken: 4200 ms Register memory region time taken: 4214 ms Register memory region time taken: 4214 ms Register memory region time taken: 4269 ms Register memory region time taken: 4322 ms Register memory region time taken: 4322 ms Get memory location time taken: 5674 ms Add local memory buffer time taken: 1 ms Total time taken: 10261 ms From the above data, we can observe: * Without pre-touching, registering on the first NIC takes a very long time due to page faults. In contrast, with pre-touching enabled, registering across all NICs takes roughly the same amount of time. * Without pre-touching, concurrent registration slows down significantly—even much slower than sequential registration. In contrast, with pre-touching, concurrent registration can greatly reduce the total time. **Conclusion:** * Perform concurrent registration only when pre-touching is used. **Combining this with the earlier pre-touching tests, we can conclude two practical strategies:** * When the segment size is small or only a few threads are available, skip pre-touching and register sequentially. * Otherwise, use a fixed number of threads for pre-touching, and then use `num_nic + 1` threads to concurrently register the memory region and get memory locations. cc. @alogfans (Background: mooncake store needs to register hundreds or even 1TB memory; it will take several minutes to register. We want to speed up this process.)

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@alogfans commented on GitHub (Sep 24, 2025):

@ykwd Thanks for your experiments. Pre-touching essentially reduce the overhead of page allocation (which may constraits by giant locks). Can allocating 2MB or even 2GB hugepages further reduce latency?

<!-- gh-comment-id:3326778286 --> @alogfans commented on GitHub (Sep 24, 2025): @ykwd Thanks for your experiments. Pre-touching essentially reduce the overhead of page allocation (which may constraits by giant locks). Can allocating 2MB or even 2GB hugepages further reduce latency?

bsakaguchi commented 2026-10-29 21:11:39 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Sep 28, 2025):

Solved in this pr: https://github.com/kvcache-ai/Mooncake/pull/875

<!-- gh-comment-id:3342270535 --> @ykwd commented on GitHub (Sep 28, 2025): Solved in this pr: https://github.com/kvcache-ai/Mooncake/pull/875

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#256

No description provided.