kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2026] [Performance]: Improve erdma register/unregister performance by reducing local memory region bookkeeping overhead #564

New issue

Closed

opened 2026-10-29 21:16:29 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:16:29 +00:00

Owner

Originally created by @stmatengss on GitHub (May 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2026

We observed that erdma-based registerLocalMemory and unregisterLocalMemory performance in Mooncake can be improved by reducing the overhead of local memory region bookkeeping in the TransferEngine.

Background

The current optimization replaces linear scans over registered memory regions with ordered address-based indexing in the local bookkeeping path. The main goal is to reduce the cost of:

overlap checks during registerLocalMemory
address-to-region lookup during key retrieval
region removal during unregisterLocalMemory

This is especially relevant for erdma deployments with a large number of already-registered regions, where bookkeeping overhead becomes more visible.

What changed locally

We added two manual benchmarks:

mooncake-transfer-engine/tests/memory_region_index_bench.cpp: synthetic bookkeeping benchmark comparing linear scan vs ordered index
mooncake-transfer-engine/tests/transfer_engine_register_bench.cpp: end-to-end benchmark that exercises real TransferEngine::init, installTransport, registerLocalMemory, and unregisterLocalMemory

Initial results

Synthetic bookkeeping benchmark showed large improvements for lookup-hit and register/unregister churn workloads:

lookup-hit workload: up to about 27x speedup
register/unregister churn: up to about 22x speedup

End-to-end erdma benchmark showed that the benefit depends on the number of already-registered regions:

with smaller preload sets, the cost of real ibv_reg_mr dominates and the bookkeeping optimization may not translate into immediate end-to-end gains
with larger preload sets, the optimization becomes visible

Measured end-to-end erdma numbers

Environment:

protocol: rdma
device: erdma_1
metadata server: local HTTP metadata service
buffer size: 4096
buffer stride: 8192
update_metadata: false

Case 1

preload_regions = 1024
measure_regions = 128
rounds = 5

Current optimized branch:

register: 280.052 ms total, 437581 ns/op
unregister: 217.292 ms total, 339519 ns/op

Main baseline:

register: 151.719 ms total, 237060 ns/op
unregister: 105.843 ms total, 165380 ns/op

Case 2

preload_regions = 8192
measure_regions = 64
rounds = 5

Current optimized branch:

register: 487.616 ms total, 1523800 ns/op
unregister: 346.793 ms total, 1083730 ns/op

Main baseline:

register: 565.516 ms total, 1767240 ns/op
unregister: 423.441 ms total, 1323250 ns/op

Observed improvement in the high-preload case:

register: about 13.8% faster
unregister: about 18.1% faster

Takeaway

For erdma, this optimization is most helpful when Mooncake already maintains a large registered-region set and additional user-space registrations continue to happen on top of that. In small-region or low-preload cases, the verbs registration cost itself can dominate and hide the bookkeeping improvement.

Suggested next steps

run a preload sweep, for example 256, 1024, 4096, 8192, 16384
measure both update_metadata=false and update_metadata=true
separate pure bookkeeping cost from ibv_reg_mr cost more clearly in follow-up benchmarks
validate whether the same trend holds for other RDMA devices beyond erdma

Originally created by @stmatengss on GitHub (May 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2026 We observed that erdma-based `registerLocalMemory` and `unregisterLocalMemory` performance in Mooncake can be improved by reducing the overhead of local memory region bookkeeping in the TransferEngine. ## Background The current optimization replaces linear scans over registered memory regions with ordered address-based indexing in the local bookkeeping path. The main goal is to reduce the cost of: - overlap checks during `registerLocalMemory` - address-to-region lookup during key retrieval - region removal during `unregisterLocalMemory` This is especially relevant for erdma deployments with a large number of already-registered regions, where bookkeeping overhead becomes more visible. ## What changed locally We added two manual benchmarks: - `mooncake-transfer-engine/tests/memory_region_index_bench.cpp`: synthetic bookkeeping benchmark comparing linear scan vs ordered index - `mooncake-transfer-engine/tests/transfer_engine_register_bench.cpp`: end-to-end benchmark that exercises real `TransferEngine::init`, `installTransport`, `registerLocalMemory`, and `unregisterLocalMemory` ## Initial results Synthetic bookkeeping benchmark showed large improvements for lookup-hit and register/unregister churn workloads: - lookup-hit workload: up to about 27x speedup - register/unregister churn: up to about 22x speedup End-to-end erdma benchmark showed that the benefit depends on the number of already-registered regions: - with smaller preload sets, the cost of real `ibv_reg_mr` dominates and the bookkeeping optimization may not translate into immediate end-to-end gains - with larger preload sets, the optimization becomes visible ## Measured end-to-end erdma numbers Environment: - protocol: `rdma` - device: `erdma_1` - metadata server: local HTTP metadata service - buffer size: 4096 - buffer stride: 8192 - update_metadata: false ### Case 1 - preload_regions = 1024 - measure_regions = 128 - rounds = 5 Current optimized branch: - register: 280.052 ms total, 437581 ns/op - unregister: 217.292 ms total, 339519 ns/op Main baseline: - register: 151.719 ms total, 237060 ns/op - unregister: 105.843 ms total, 165380 ns/op ### Case 2 - preload_regions = 8192 - measure_regions = 64 - rounds = 5 Current optimized branch: - register: 487.616 ms total, 1523800 ns/op - unregister: 346.793 ms total, 1083730 ns/op Main baseline: - register: 565.516 ms total, 1767240 ns/op - unregister: 423.441 ms total, 1323250 ns/op Observed improvement in the high-preload case: - register: about 13.8% faster - unregister: about 18.1% faster ## Takeaway For erdma, this optimization is most helpful when Mooncake already maintains a large registered-region set and additional user-space registrations continue to happen on top of that. In small-region or low-preload cases, the verbs registration cost itself can dominate and hide the bookkeeping improvement. ## Suggested next steps - run a preload sweep, for example 256, 1024, 4096, 8192, 16384 - measure both `update_metadata=false` and `update_metadata=true` - separate pure bookkeeping cost from `ibv_reg_mr` cost more clearly in follow-up benchmarks - validate whether the same trend holds for other RDMA devices beyond erdma

bsakaguchi

2026-10-29 21:16:29 +00:00

closed this issue
added the
Transfer Engine

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:16:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 2, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Aug 2, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:16:31 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 10, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Aug 10, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:29:15 +00:00

[PR #564] [MERGED] add support for asynchronous batch transfer to accelerate transfer operation #1657