[PR #902] [MERGED] [Store]: Add option to use jemalloc in mooncake store master #1893

New issue

Closed

opened 2026-10-29 21:31:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:31:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/902
Author: @nickyc975
Created: 9/29/2025
Status: ✅ Merged
Merged: 9/30/2025
Merged by: @ykwd

Base: main ← Head: jinlong/jemalloc

📝 Commits (2)

4375b11 [Store]: Add option to use jemalloc in mooncake store master
1353825 Address code review issues

📊 Changes

4 files changed (+22 additions, -1 deletions)

View changed files

📝 .devcontainer/Dockerfile (+5 -1)
📝 CMakeLists.txt (+2 -0)
📝 dependencies.sh (+1 -0)
📝 mooncake-store/src/CMakeLists.txt (+14 -0)

📄 Description

With more tests and analysis, we found that the root cause of #834 is serious memory fragmentation.

Insertion and deletion of metadata in Mooncake Master are accompanied by allocation and deallocation of small memory objects (typically hundreds of bytes). With large amount of metadata being inserted and removed, the memory space of Master gets seriously fragmented, resulting in unexpected memory consumption.

Using a more modern memory allocator (i.e. jemalloc) can effectively alleviate this problem. Here is a comparision between the default libc allocator and jemalloc:

Test script:

#!/bin/bash

# Specify the file path to store the result.
output_file=$1

top -b -n 1 -w 512 | grep mooncake_master >> $output_file
for ((i=0; i<10; i++)); do
    python3 mooncake-store/tests/stress_cluster_benchmark.py \
        --role=prefill --metadata-server=P2PHANDSHAKE --protocol=rdma \
        --device-name=mlx5_bond_0 --local-hostname=127.0.0.1:8081 \
        --local-buffer-size=1024 --global-segment-size=32768 \
        --max-requests=33554432 --value-length=1024 --batch-size=4096 \
        --wait-time=0
    # Sleep for a while to let master calm down.
    sleep 30
    top -b -n 1 -w 512 | grep mooncake_master >> $output_file
done

Test results:

Round	glibc malloc mem usage	glibc malloc performance	jemalloc mem usage	jemalloc performance
0	37760	-	22468	-
1	9.3g	126.98 MB/s	7.6g	134.65 MB/s
2	18.4g	132.49 MB/s	7.8g	143.53 MB/s
3	18.4g	137.45 MB/s	14.9g	150.32 MB/s
4	18.4g	148.84 MB/s	14.9g	140.36 MB/s
5	18.4g	135.92 MB/s	14.8g	145.69 MB/s
6	18.4g	152.01 MB/s	7.9g	142.08 MB/s
7	18.4g	148.36 MB/s	7.7g	144.45 MB/s
8	18.4g	148.53 MB/s	14.8g	139.66 MB/s
9	18.4g	140.32 MB/s	15.0g	146.60 MB/s
10	18.4g	140.91 MB/s	14.9g	140.92 MB/s

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/902 **Author:** [@nickyc975](https://github.com/nickyc975) **Created:** 9/29/2025 **Status:** ✅ Merged **Merged:** 9/30/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `jinlong/jemalloc` --- ### 📝 Commits (2) - [`4375b11`](https://github.com/kvcache-ai/Mooncake/commit/4375b11d8153c222371a80e3197f272ccd9aef78) [Store]: Add option to use jemalloc in mooncake store master - [`1353825`](https://github.com/kvcache-ai/Mooncake/commit/13538257c17a2ed795dd760cc42ddb956e56130f) Address code review issues ### 📊 Changes **4 files changed** (+22 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `.devcontainer/Dockerfile` (+5 -1) 📝 `CMakeLists.txt` (+2 -0) 📝 `dependencies.sh` (+1 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+14 -0) </details> ### 📄 Description With more tests and analysis, we found that the root cause of #834 is serious memory fragmentation. Insertion and deletion of metadata in Mooncake Master are accompanied by allocation and deallocation of small memory objects (typically hundreds of bytes). With large amount of metadata being inserted and removed, the memory space of Master gets seriously fragmented, resulting in unexpected memory consumption. Using a more modern memory allocator (i.e. [jemalloc](https://jemalloc.net/)) can effectively alleviate this problem. Here is a comparision between the default libc allocator and jemalloc: Test script: ```bash #!/bin/bash # Specify the file path to store the result. output_file=$1 top -b -n 1 -w 512 | grep mooncake_master >> $output_file for ((i=0; i<10; i++)); do python3 mooncake-store/tests/stress_cluster_benchmark.py \ --role=prefill --metadata-server=P2PHANDSHAKE --protocol=rdma \ --device-name=mlx5_bond_0 --local-hostname=127.0.0.1:8081 \ --local-buffer-size=1024 --global-segment-size=32768 \ --max-requests=33554432 --value-length=1024 --batch-size=4096 \ --wait-time=0 # Sleep for a while to let master calm down. sleep 30 top -b -n 1 -w 512 | grep mooncake_master >> $output_file done ``` Test results: | Round | glibc malloc mem usage | glibc malloc performance | jemalloc mem usage | jemalloc performance | |:-:|:-:|:-:|:-:|:-:| |0| 37760 | - | 22468 | - | |1| 9.3g | 126.98 MB/s | 7.6g | 134.65 MB/s | |2| 18.4g | 132.49 MB/s | 7.8g | 143.53 MB/s | |3| 18.4g | 137.45 MB/s | 14.9g | 150.32 MB/s | |4| 18.4g | 148.84 MB/s | 14.9g | 140.36 MB/s | |5| 18.4g | 135.92 MB/s | 14.8g | 145.69 MB/s | |6| 18.4g | 152.01 MB/s | 7.9g | 142.08 MB/s | |7| 18.4g | 148.36 MB/s | 7.7g | 144.45 MB/s | |8| 18.4g | 148.53 MB/s | 14.8g | 139.66 MB/s | |9| 18.4g | 140.32 MB/s | 15.0g | 146.60 MB/s | |10| 18.4g | 140.91 MB/s | 14.9g | 140.92 MB/s | --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:31:10 +00:00