[PR #1343] [MERGED] [Store]: Use SharedMutex in MetadataShard for better performance #2175

New issue

Closed

opened 2026-10-29 21:33:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:33:21 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1343
Author: @nickyc975
Created: 1/8/2026
Status: ✅ Merged
Merged: 1/12/2026
Merged by: @stmatengss

Base: main ← Head: jinlong/metadata-shard-opt

📝 Commits (6)

a6b0ed3 [Store]: Use 'const std::string&' as param for MasterService::GetReplicaList
ef98ecd [Store]: Use SharedMutex in MetadataShard for better performance
921645d [Store]: Address code review problems
8c1e11b [Store]: Address code review problems
d666a82 Fix typo
8fc7d2e Simplify ObjectMetadata::IsValid

📊 Changes

4 files changed (+343 additions, -134 deletions)

View changed files

📝 mooncake-store/include/master_service.h (+117 -31)
📝 mooncake-store/include/mutex.h (+57 -0)
📝 mooncake-store/src/master_service.cpp (+95 -103)
📝 mooncake-store/tests/mutex_test.cpp (+74 -0)

📄 Description

Description

Currently, we use Mutex to ensure mutually exclusive access to MetadataShard during concurrent operations. However, this mutual exclusion is unnecessary for concurrent read operations and actually leads to performance degradation. Therefore, we propose to replace Mutex with SharedMutex. This will increase the parallelism of read operations, thereby improving Master's read QPS.

We adopt RAII-style lock management for matadata shard. All access to the shards should be proxied through MetadataShardAccessorRO and MetadataShardAccessorRW. MetadataAccessor is also splitted into MetadataAccessorRO and MetadataAccessorRW.

This PR also updated MasterService:: GetReplicaList to accept const std::string& instead of std::string_view as parameter.

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other
  - Performance improvement

How Has This Been Tested?

The performance improvement is validated with master_bench. The test configurations are as follows:

# Start master
mooncake_master --rpc_thread_num=64 --eviction_high_watermark_ratio=0.8 --eviction_ratio=0.2

# Test BatchPut performance
./mooncake-store/benchmarks/master_bench --num_segments=16 --duration=30 --num_clients=8 --num_threads=4 --batch_size=4096 --master_server=127.0.0.1:50051 --operation=BatchPut

# Test BatchGet performance
./mooncake-store/benchmarks/master_bench --num_segments=16 --duration=30 --num_clients=8 --num_threads=4 --batch_size=4096 --master_server=127.0.0.1:50051 --operation=BatchGet --num_keys=1000000

The QPS results of 3 rounds of tests are as follows:

Round	BatchPut Before Opt	BatchGet Before Opt	BatchPut After Opt	BatchGet After Opt
1	4016810.67	11488733.87	4028825.60	12360635.73
2	4024183.47	11458423.47	4013533.87	12356949.33
3	3968887.47	11479176.53	3980492.80	12363639.47

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1343 **Author:** [@nickyc975](https://github.com/nickyc975) **Created:** 1/8/2026 **Status:** ✅ Merged **Merged:** 1/12/2026 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `jinlong/metadata-shard-opt` --- ### 📝 Commits (6) - [`a6b0ed3`](https://github.com/kvcache-ai/Mooncake/commit/a6b0ed37f0a8b8498c0134bf1ab0df638ac0a3c8) [Store]: Use 'const std::string&' as param for MasterService::GetReplicaList - [`ef98ecd`](https://github.com/kvcache-ai/Mooncake/commit/ef98ecd6b595543e29e413cdb71d796778b16887) [Store]: Use SharedMutex in MetadataShard for better performance - [`921645d`](https://github.com/kvcache-ai/Mooncake/commit/921645d0f0cd13bd3f847543d1ad5a6b1d86af2d) [Store]: Address code review problems - [`8c1e11b`](https://github.com/kvcache-ai/Mooncake/commit/8c1e11bc5a8d89cf66c5e19950e106cb22988dd2) [Store]: Address code review problems - [`d666a82`](https://github.com/kvcache-ai/Mooncake/commit/d666a8277414c75340608a7a1f079c53a475bbf6) Fix typo - [`8fc7d2e`](https://github.com/kvcache-ai/Mooncake/commit/8fc7d2e10e988fe10dc0b16dc7d28d5dbf8bdd80) Simplify ObjectMetadata::IsValid ### 📊 Changes **4 files changed** (+343 additions, -134 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/include/master_service.h` (+117 -31) 📝 `mooncake-store/include/mutex.h` (+57 -0) 📝 `mooncake-store/src/master_service.cpp` (+95 -103) 📝 `mooncake-store/tests/mutex_test.cpp` (+74 -0) </details> ### 📄 Description ## Description Currently, we use `Mutex` to ensure mutually exclusive access to `MetadataShard` during concurrent operations. However, this mutual exclusion is unnecessary for concurrent read operations and actually leads to performance degradation. Therefore, we propose to replace `Mutex` with `SharedMutex`. This will increase the parallelism of read operations, thereby improving Master's read QPS. We adopt RAII-style lock management for matadata shard. All access to the shards should be proxied through `MetadataShardAccessorRO` and `MetadataShardAccessorRW`. `MetadataAccessor` is also splitted into `MetadataAccessorRO` and `MetadataAccessorRW`. This PR also updated `MasterService:: GetReplicaList` to accept `const std::string&` instead of `std::string_view` as parameter. ## Type of Change * Types - [ ] Bug fix - [ ] New feature - [ ] Transfer Engine - [ ] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [x] Other - [x] Performance improvement ## How Has This Been Tested? The performance improvement is validated with `master_bench`. The test configurations are as follows: ```bash # Start master mooncake_master --rpc_thread_num=64 --eviction_high_watermark_ratio=0.8 --eviction_ratio=0.2 # Test BatchPut performance ./mooncake-store/benchmarks/master_bench --num_segments=16 --duration=30 --num_clients=8 --num_threads=4 --batch_size=4096 --master_server=127.0.0.1:50051 --operation=BatchPut # Test BatchGet performance ./mooncake-store/benchmarks/master_bench --num_segments=16 --duration=30 --num_clients=8 --num_threads=4 --batch_size=4096 --master_server=127.0.0.1:50051 --operation=BatchGet --num_keys=1000000 ``` The QPS results of 3 rounds of tests are as follows: |Round|BatchPut Before Opt|BatchGet Before Opt|BatchPut After Opt|BatchGet After Opt| |:-:|:-:|:-:|:-:|:-:| |1|4016810.67|11488733.87|4028825.60|12360635.73| |2|4024183.47|11458423.47|4013533.87|12356949.33| |3|3968887.47|11479176.53|3980492.80|12363639.47| ## Checklist - [x] I have performed a self-review of my own code. - [x] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:33:21 +00:00