[PR #855] [CLOSED] [WIP]feat(rdma): add parallel memory region registration support #1869

New issue

Closed

opened 2026-10-29 21:30:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:30:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/855
Author: @staryxchen
Created: 9/17/2025
Status: ❌ Closed

Base: main ← Head: feat/add-parallel-mr-registration

📝 Commits (3)

ea4cd87 feat(rdma): add parallel memory region registration support
0aa5f89 format and remove some comment
3015f10 fix(config): change parallel memory region registration to default disabled

📊 Changes

3 files changed (+76 additions, -12 deletions)

View changed files

📝 mooncake-transfer-engine/include/config.h (+1 -0)
📝 mooncake-transfer-engine/src/config.cpp (+9 -0)
📝 mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp (+66 -12)

📄 Description

Summary

This PR introduces a configurable parallel memory region registration feature with significant performance improvements for pre-allocated memory scenarios, while maintaining backward compatibility.

I conducted several tests to validate performance (test code is also attached).
perf.tar.gz

Test Configuration

Memory Size: 500GB (DRAM)
Test Scenarios:
- Pre-allocated memory (memory allocated and initialized)
- Non-pre-allocated memory (memory allocated but not initialized)
Configuration Options:
- With Optimization: MC_DISABLE_PARALLEL_REG_MR not set (parallel registration enabled)
- Without Optimization: MC_DISABLE_PARALLEL_REG_MR=1 (sequential registration)

Performance Results

Pre-allocated Memory Scenario

Operation	With Optimization (Parallel)	Without Optimization (Sequential)	Performance Improvement
allocate_memory	3.557 seconds	3.810 seconds	N/A
register_memory	11.662 seconds	49.198 seconds	321.9% faster
unregister_memory	1.459 seconds	11.754 seconds	705.4% faster
Total Time	17.353 seconds	65.399 seconds	276.8% faster

Non-pre-allocated Memory Scenario

Operation	With Optimization (Parallel)	Without Optimization (Sequential)	Performance Improvement
allocate_memory	0.000 seconds	0.001 seconds	N/A
register_memory	461.999 seconds	86.930 seconds	-431.3% slower
unregister_memory	1.822 seconds	11.498 seconds	531.0% faster
Total Time	464.469 seconds	98.945 seconds	-369.4% slower

I had the AI summarize and analyze the test results. Below is the AI's output:

Key Performance Findings

Pre-allocated Memory (500GB)

Memory registration: 4.2x faster (11.7s → 49.2s)
Memory unregistration: 8.1x faster (1.5s → 11.8s)
Total operation time: 2.8x faster (17.4s → 65.4s)

Non-pre-allocated Memory (500GB)

Memory registration: 5.3x slower (462s → 87s)
Memory unregistration: 6.3x faster (1.8s → 11.5s)
Total operation time: 3.7x slower (464s → 99s)

Analysis

Pre-allocated memory benefits from parallel registration because:

Memory is already pinned and in physical memory
Multiple RDMA contexts can be utilized simultaneously
Better CPU core utilization

Non-pre-allocated memory performs better with sequential registration because:

Reduces memory paging overhead and I/O contention
Avoids kernel-level resource conflicts during large allocations

Conclusion

The parallel memory registration optimization provides significant performance benefits for pre-allocated memory scenarios, with up to 8x improvement in unregistration performance. However, for large non-pre-allocated memory allocations, sequential registration performs better due to reduced resource contention and kernel overhead.

The MC_PARALLEL_REG_MR configuration option provides the flexibility to choose the optimal strategy based on the specific use case and memory allocation patterns of the application.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/855 **Author:** [@staryxchen](https://github.com/staryxchen) **Created:** 9/17/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `feat/add-parallel-mr-registration` --- ### 📝 Commits (3) - [`ea4cd87`](https://github.com/kvcache-ai/Mooncake/commit/ea4cd87e22dbc858b1a623a3dcbeee8726f7532f) feat(rdma): add parallel memory region registration support - [`0aa5f89`](https://github.com/kvcache-ai/Mooncake/commit/0aa5f8942baceabeee675da225d00f157c0b5b0e) format and remove some comment - [`3015f10`](https://github.com/kvcache-ai/Mooncake/commit/3015f109550d5386013a692b5778694051151262) fix(config): change parallel memory region registration to default disabled ### 📊 Changes **3 files changed** (+76 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/include/config.h` (+1 -0) 📝 `mooncake-transfer-engine/src/config.cpp` (+9 -0) 📝 `mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp` (+66 -12) </details> ### 📄 Description ## Summary This PR introduces a configurable parallel memory region registration feature with significant performance improvements for pre-allocated memory scenarios, while maintaining backward compatibility. I conducted several tests to validate performance (test code is also attached). [perf.tar.gz](https://github.com/user-attachments/files/22397979/perf.tar.gz) ## Test Configuration - **Memory Size**: 500GB (DRAM) - **Test Scenarios**: - Pre-allocated memory (memory allocated and initialized) - Non-pre-allocated memory (memory allocated but not initialized) - **Configuration Options**: - **With Optimization**: `MC_DISABLE_PARALLEL_REG_MR` not set (parallel registration enabled) - **Without Optimization**: `MC_DISABLE_PARALLEL_REG_MR=1` (sequential registration) ## Performance Results ### Pre-allocated Memory Scenario | Operation | With Optimization (Parallel) | Without Optimization (Sequential) | Performance Improvement | |-----------|------------------------------|-----------------------------------|------------------------| | **allocate_memory** | 3.557 seconds | 3.810 seconds | N/A | | **register_memory** | 11.662 seconds | 49.198 seconds | **321.9% faster** | | **unregister_memory** | 1.459 seconds | 11.754 seconds | **705.4% faster** | | **Total Time** | 17.353 seconds | 65.399 seconds | **276.8% faster** | ### Non-pre-allocated Memory Scenario | Operation | With Optimization (Parallel) | Without Optimization (Sequential) | Performance Improvement | |-----------|------------------------------|-----------------------------------|------------------------| | **allocate_memory** | 0.000 seconds | 0.001 seconds | N/A | | **register_memory** | 461.999 seconds | 86.930 seconds | **-431.3% slower** | | **unregister_memory** | 1.822 seconds | 11.498 seconds | **531.0% faster** | | **Total Time** | 464.469 seconds | 98.945 seconds | **-369.4% slower** | I had the AI summarize and analyze the test results. Below is the AI's output: ## Key Performance Findings ### Pre-allocated Memory (500GB) - **Memory registration**: 4.2x faster (11.7s → 49.2s) - **Memory unregistration**: 8.1x faster (1.5s → 11.8s) - **Total operation time**: 2.8x faster (17.4s → 65.4s) ### Non-pre-allocated Memory (500GB) - **Memory registration**: 5.3x slower (462s → 87s) - **Memory unregistration**: 6.3x faster (1.8s → 11.5s) - **Total operation time**: 3.7x slower (464s → 99s) ## Analysis **Pre-allocated memory benefits from parallel registration** because: - Memory is already pinned and in physical memory - Multiple RDMA contexts can be utilized simultaneously - Better CPU core utilization **Non-pre-allocated memory performs better with sequential registration** because: - Reduces memory paging overhead and I/O contention - Avoids kernel-level resource conflicts during large allocations ## Conclusion The parallel memory registration optimization provides significant performance benefits for pre-allocated memory scenarios, with up to 8x improvement in unregistration performance. However, for large non-pre-allocated memory allocations, sequential registration performs better due to reduced resource contention and kernel overhead. The `MC_PARALLEL_REG_MR` configuration option provides the flexibility to choose the optimal strategy based on the specific use case and memory allocation patterns of the application. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:30:58 +00:00