kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[PR #641] [MERGED] [Store] Import Offset Allocator #1712

New issue
Closed
opened 2026-10-29 21:29:43 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-29 21:29:43 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/641
Author: @ykwd
Created: 7/18/2025
Status: ✅ Merged
Merged: 7/18/2025
Merged by: @ykwd

Base: main ← Head: offset_allocator

📝 Commits (10+)

  • 8cf9c00 refactor(store): replace manual slice management with offset_allocator
  • cae6b22 refactor(store): improve buffer management with allocation handle
  • a7e992a chore(ci): temporarily disable ssd test
  • 73b312d test(offsetAllocator): add max contiguous allocation after small allocations test
  • 09736b9 revert modifications of store_py and ci
  • 39c352e Optimize the offset allocator; Add unit tests
  • 3955d9b Add benchmarks for allocator
  • 85784dd Enable allocation size more than 4GB
  • ae3c988 rename class name
  • 40ced6d Enhance the unit tests

📊 Changes

9 files changed (+2287 additions, -2 deletions)

View changed files

📝 mooncake-store/CMakeLists.txt (+2 -1)
➕ mooncake-store/benchmarks/CMakeLists.txt (+3 -0)
➕ mooncake-store/benchmarks/allocator_bench.cpp (+179 -0)
➕ mooncake-store/benchmarks/allocator_bench_result.md (+176 -0)
➕ mooncake-store/include/offset_allocator/offset_allocator.hpp (+184 -0)
📝 mooncake-store/src/CMakeLists.txt (+2 -1)
➕ mooncake-store/src/offset_allocator.cpp (+600 -0)
📝 mooncake-store/tests/CMakeLists.txt (+11 -0)
➕ mooncake-store/tests/offset_allocator_test.cpp (+1130 -0)

📄 Description

Content

Import a new offset allocator to address the following issue:

  1. Support memory allocation with a size larger than 16MB, which will help eliminate redundant memcpy invocations within non–zero-copy get interfaces. https://github.com/kvcache-ai/Mooncake/pull/624
  2. Auto-merge the freed memory to avoid this bug: https://github.com/kvcache-ai/Mooncake/issues/632

Compared with the original implementation, the imported OffsetAllocator is wrapped to support registering or allocating with a size larger than 3.75 GB, and is further optimized for LLM inference scenario. The optimization will:
a) slightly decrease the memory utilization ratio in general cases;
b) makes no difference when the allocated size is equal to a bin size;
c) largely improve the memory utilization ratio when the allocated size is mostly uniform and not equal to any bin size.
The LLM inference tasks, in most cases, will put objects with mostly uniform sizes, which falls into the latter two cases.
See the following benchmark reports for details.

Result

  • alloc size: The size of each object
  • utilization ratio: The total allocated size / total space
  • time: time in nanoseconds for each object allocation
  • OffsetAllocator optimization: whether round up the allocated size to a bin size

Uniform size, size equals power of 2

OffsetAllocator (After Optimization)

Alloc size: 32, min util ratio: 1, avg util ratio: 1, time: 544 ns
Alloc size: 128, min util ratio: 1, avg util ratio: 1, time: 417 ns
Alloc size: 512, min util ratio: 1, avg util ratio: 1, time: 174 ns
Alloc size: 2048, min util ratio: 1, avg util ratio: 1, time: 406 ns
Alloc size: 8192, min util ratio: 1, avg util ratio: 1, time: 180 ns
Alloc size: 32768, min util ratio: 1, avg util ratio: 1, time: 133 ns
Alloc size: 131072, min util ratio: 1, avg util ratio: 1, time: 109 ns
Alloc size: 524288, min util ratio: 1, avg util ratio: 1, time: 100 ns
Alloc size: 2097152, min util ratio: 1, avg util ratio: 1, time: 99 ns
Alloc size: 8388608, min util ratio: 1, avg util ratio: 1, time: 99 ns
Alloc size: 33554432, min util ratio: 1, avg util ratio: 1, time: 98 ns

OffsetAllocator (Before Optimization)

Alloc size: 32, min util ratio: 1, avg util ratio: 1, time: 539 ns
Alloc size: 128, min util ratio: 1, avg util ratio: 1, time: 419 ns
Alloc size: 512, min util ratio: 1, avg util ratio: 1, time: 217 ns
Alloc size: 2048, min util ratio: 1, avg util ratio: 1, time: 408 ns
Alloc size: 8192, min util ratio: 1, avg util ratio: 1, time: 175 ns
Alloc size: 32768, min util ratio: 1, avg util ratio: 1, time: 130 ns
Alloc size: 131072, min util ratio: 1, avg util ratio: 1, time: 107 ns
Alloc size: 524288, min util ratio: 1, avg util ratio: 1, time: 99 ns
Alloc size: 2097152, min util ratio: 1, avg util ratio: 1, time: 100 ns
Alloc size: 8388608, min util ratio: 1, avg util ratio: 1, time: 98 ns
Alloc size: 33554432, min util ratio: 1, avg util ratio: 1, time: 98 ns

Uniform size, size equals power of 2 +/- 17

OffsetAllocator (After Optimization)

Alloc size: 15, min util ratio: 1, avg util ratio: 1, time: 568 ns
Alloc size: 111, min util ratio: 0.991071, avg util ratio: 0.991071, time: 441 ns
Alloc size: 495, min util ratio: 0.966797, avg util ratio: 0.966797, time: 178 ns
Alloc size: 2031, min util ratio: 0.991699, avg util ratio: 0.991699, time: 418 ns
Alloc size: 8175, min util ratio: 0.997925, avg util ratio: 0.997925, time: 170 ns
Alloc size: 32751, min util ratio: 0.999481, avg util ratio: 0.999481, time: 133 ns
Alloc size: 131055, min util ratio: 0.99987, avg util ratio: 0.99987, time: 109 ns
Alloc size: 524271, min util ratio: 0.999968, avg util ratio: 0.999968, time: 100 ns
Alloc size: 2097135, min util ratio: 0.999992, avg util ratio: 0.999992, time: 99 ns
Alloc size: 8388591, min util ratio: 0.999998, avg util ratio: 0.999998, time: 98 ns
Alloc size: 33554415, min util ratio: 0.999999, avg util ratio: 0.999999, time: 99 ns
Alloc size: 49, min util ratio: 0.942308, avg util ratio: 0.942308, time: 508 ns
Alloc size: 145, min util ratio: 0.906249, avg util ratio: 0.906249, time: 372 ns
Alloc size: 529, min util ratio: 0.918399, avg util ratio: 0.918399, time: 172 ns
Alloc size: 2065, min util ratio: 0.896267, avg util ratio: 0.896267, time: 403 ns
Alloc size: 8209, min util ratio: 0.89073, avg util ratio: 0.89073, time: 174 ns
Alloc size: 32785, min util ratio: 0.889347, avg util ratio: 0.889347, time: 131 ns
Alloc size: 131089, min util ratio: 0.88897, avg util ratio: 0.88897, time: 105 ns
Alloc size: 524305, min util ratio: 0.888701, avg util ratio: 0.888701, time: 102 ns
Alloc size: 2097169, min util ratio: 0.888679, avg util ratio: 0.888679, time: 100 ns
Alloc size: 8388625, min util ratio: 0.886721, avg util ratio: 0.886721, time: 100 ns
Alloc size: 33554449, min util ratio: 0.875, avg util ratio: 0.875, time: 100 ns

OffsetAllocator (Before Optimization)

Alloc size: 15, min util ratio: 1, avg util ratio: 1, time: 566 ns
Alloc size: 111, min util ratio: 0.669866, avg util ratio: 0.710845, time: 703 ns
Alloc size: 495, min util ratio: 0.665779, avg util ratio: 0.676874, time: 238 ns
Alloc size: 2031, min util ratio: 0.668333, avg util ratio: 0.705411, time: 637 ns
Alloc size: 8175, min util ratio: 0.666175, avg util ratio: 0.676474, time: 242 ns
Alloc size: 32751, min util ratio: 0.664435, avg util ratio: 0.669078, time: 168 ns
Alloc size: 131055, min util ratio: 0.66062, avg util ratio: 0.667341, time: 124 ns
Alloc size: 524271, min util ratio: 0.653055, avg util ratio: 0.666993, time: 118 ns
Alloc size: 2097135, min util ratio: 0.64062, avg util ratio: 0.666873, time: 116 ns
Alloc size: 8388591, min util ratio: 0.605468, avg util ratio: 0.667812, time: 115 ns
Alloc size: 33554415, min util ratio: 0.5625, avg util ratio: 0.670944, time: 116 ns
Alloc size: 49, min util ratio: 0.692229, avg util ratio: 0.753062, time: 1122 ns
Alloc size: 145, min util ratio: 0.667789, avg util ratio: 0.700907, time: 572 ns
Alloc size: 529, min util ratio: 0.66577, avg util ratio: 0.676238, time: 238 ns
Alloc size: 2065, min util ratio: 0.667926, avg util ratio: 0.704884, time: 632 ns
Alloc size: 8209, min util ratio: 0.665708, avg util ratio: 0.676372, time: 239 ns
Alloc size: 32785, min util ratio: 0.664224, avg util ratio: 0.669058, time: 168 ns
Alloc size: 131089, min util ratio: 0.659631, avg util ratio: 0.667287, time: 129 ns
Alloc size: 524305, min util ratio: 0.652609, avg util ratio: 0.666884, time: 122 ns
Alloc size: 2097169, min util ratio: 0.638677, avg util ratio: 0.666516, time: 120 ns
Alloc size: 8388625, min util ratio: 0.60547, avg util ratio: 0.665131, time: 121 ns
Alloc size: 33554449, min util ratio: 0.546875, avg util ratio: 0.660917, time: 120 ns

Uniform size, size equals power of 2 multiply 0.9 or 1.1

OffsetAllocator (After Optimization)

Alloc size: 28, min util ratio: 1, avg util ratio: 1, time: 543 ns
Alloc size: 115, min util ratio: 0.958333, avg util ratio: 0.958333, time: 418 ns
Alloc size: 460, min util ratio: 0.958332, avg util ratio: 0.958332, time: 189 ns
Alloc size: 1843, min util ratio: 0.959896, avg util ratio: 0.959896, time: 418 ns
Alloc size: 7372, min util ratio: 0.959895, avg util ratio: 0.959895, time: 197 ns
Alloc size: 29491, min util ratio: 0.959993, avg util ratio: 0.959993, time: 135 ns
Alloc size: 117964, min util ratio: 0.959979, avg util ratio: 0.959979, time: 111 ns
Alloc size: 471859, min util ratio: 0.959985, avg util ratio: 0.959985, time: 100 ns
Alloc size: 1887436, min util ratio: 0.959765, avg util ratio: 0.959765, time: 99 ns
Alloc size: 7549747, min util ratio: 0.959766, avg util ratio: 0.959766, time: 99 ns
Alloc size: 30198988, min util ratio: 0.95625, avg util ratio: 0.95625, time: 99 ns
Alloc size: 35, min util ratio: 0.972222, avg util ratio: 0.972222, time: 531 ns
Alloc size: 140, min util ratio: 0.972222, avg util ratio: 0.972222, time: 397 ns
Alloc size: 563, min util ratio: 0.977427, avg util ratio: 0.977427, time: 180 ns
Alloc size: 2252, min util ratio: 0.97743, avg util ratio: 0.97743, time: 389 ns
Alloc size: 9011, min util ratio: 0.977752, avg util ratio: 0.977752, time: 183 ns
Alloc size: 36044, min util ratio: 0.977752, avg util ratio: 0.977752, time: 133 ns
Alloc size: 144179, min util ratio: 0.977739, avg util ratio: 0.977739, time: 106 ns
Alloc size: 576716, min util ratio: 0.977538, avg util ratio: 0.977538, time: 103 ns
Alloc size: 2306867, min util ratio: 0.977539, avg util ratio: 0.977539, time: 99 ns
Alloc size: 9227468, min util ratio: 0.975391, avg util ratio: 0.975391, time: 99 ns
Alloc size: 36909875, min util ratio: 0.9625, avg util ratio: 0.9625, time: 100 ns

OffsetAllocator (Before Optimization)

Alloc size: 28, min util ratio: 1, avg util ratio: 1, time: 539 ns
Alloc size: 115, min util ratio: 0.669299, avg util ratio: 0.709245, time: 701 ns
Alloc size: 460, min util ratio: 0.665825, avg util ratio: 0.677532, time: 255 ns
Alloc size: 1843, min util ratio: 0.669352, avg util ratio: 0.709202, time: 691 ns
Alloc size: 7372, min util ratio: 0.66619, avg util ratio: 0.677401, time: 260 ns
Alloc size: 29491, min util ratio: 0.664311, avg util ratio: 0.669511, time: 172 ns
Alloc size: 117964, min util ratio: 0.661812, avg util ratio: 0.667356, time: 133 ns
Alloc size: 471859, min util ratio: 0.654345, avg util ratio: 0.667048, time: 123 ns
Alloc size: 1887436, min util ratio: 0.640722, avg util ratio: 0.666447, time: 121 ns
Alloc size: 7549747, min util ratio: 0.611719, avg util ratio: 0.666847, time: 119 ns
Alloc size: 30198988, min util ratio: 0.548437, avg util ratio: 0.669799, time: 125 ns
Alloc size: 35, min util ratio: 0.7098, avg util ratio: 0.774162, time: 1306 ns
Alloc size: 140, min util ratio: 0.667934, avg util ratio: 0.702151, time: 599 ns
Alloc size: 563, min util ratio: 0.665599, avg util ratio: 0.675548, time: 239 ns
Alloc size: 2252, min util ratio: 0.667371, avg util ratio: 0.701623, time: 601 ns
Alloc size: 9011, min util ratio: 0.665485, avg util ratio: 0.675528, time: 244 ns
Alloc size: 36044, min util ratio: 0.663248, avg util ratio: 0.668912, time: 170 ns
Alloc size: 144179, min util ratio: 0.660308, avg util ratio: 0.666934, time: 127 ns
Alloc size: 576716, min util ratio: 0.654467, avg util ratio: 0.66679, time: 122 ns
Alloc size: 2306867, min util ratio: 0.633789, avg util ratio: 0.666159, time: 121 ns
Alloc size: 9227468, min util ratio: 0.597266, avg util ratio: 0.666037, time: 118 ns
Alloc size: 36909875, min util ratio: 0.55, avg util ratio: 0.669564, time: 121 ns

Random Size

OffsetAllocator (After Optimization)

util ratio (min / p99 / p90 / p50 / max / avg): 0.544250 / 0.713338 / 0.779739 / 0.847867 / 0.952591 / 0.841576
avg alloc time: 145.575738 ns/op

OffsetAllocator (Before Optimization)

util ratio (min / p99 / p90 / p50 / max / avg): 0.569255 / 0.712076 / 0.781224 / 0.855046 / 0.976057 / 0.848873
avg alloc time: 142.508508 ns/op

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/641 **Author:** [@ykwd](https://github.com/ykwd) **Created:** 7/18/2025 **Status:** ✅ Merged **Merged:** 7/18/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `offset_allocator` --- ### 📝 Commits (10+) - [`8cf9c00`](https://github.com/kvcache-ai/Mooncake/commit/8cf9c00772c61cadf7968162fa5a5e7936b5a00a) refactor(store): replace manual slice management with offset_allocator - [`cae6b22`](https://github.com/kvcache-ai/Mooncake/commit/cae6b22921675fc0eb2b34316a96c3c171c36889) refactor(store): improve buffer management with allocation handle - [`a7e992a`](https://github.com/kvcache-ai/Mooncake/commit/a7e992a570800476183572b9bf739ac221dfea0f) chore(ci): temporarily disable ssd test - [`73b312d`](https://github.com/kvcache-ai/Mooncake/commit/73b312d97f8646a27aca16f7fd61c07df85188ca) test(offsetAllocator): add max contiguous allocation after small allocations test - [`09736b9`](https://github.com/kvcache-ai/Mooncake/commit/09736b9b9401c8786fb5ec542a0c7c8b8939b471) revert modifications of store_py and ci - [`39c352e`](https://github.com/kvcache-ai/Mooncake/commit/39c352e264af849469914a673e2ec74eaa02668f) Optimize the offset allocator; Add unit tests - [`3955d9b`](https://github.com/kvcache-ai/Mooncake/commit/3955d9bdfd2e9fc3947e18cc29c2c7bb527a7109) Add benchmarks for allocator - [`85784dd`](https://github.com/kvcache-ai/Mooncake/commit/85784ddd51073d88f4c2c7f11c05bf4069da03e6) Enable allocation size more than 4GB - [`ae3c988`](https://github.com/kvcache-ai/Mooncake/commit/ae3c988a14f87727411375d67dc615b0555a2165) rename class name - [`40ced6d`](https://github.com/kvcache-ai/Mooncake/commit/40ced6d74f57ed016817471b99a67c8e0af27d98) Enhance the unit tests ### 📊 Changes **9 files changed** (+2287 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/CMakeLists.txt` (+2 -1) ➕ `mooncake-store/benchmarks/CMakeLists.txt` (+3 -0) ➕ `mooncake-store/benchmarks/allocator_bench.cpp` (+179 -0) ➕ `mooncake-store/benchmarks/allocator_bench_result.md` (+176 -0) ➕ `mooncake-store/include/offset_allocator/offset_allocator.hpp` (+184 -0) 📝 `mooncake-store/src/CMakeLists.txt` (+2 -1) ➕ `mooncake-store/src/offset_allocator.cpp` (+600 -0) 📝 `mooncake-store/tests/CMakeLists.txt` (+11 -0) ➕ `mooncake-store/tests/offset_allocator_test.cpp` (+1130 -0) </details> ### 📄 Description ## Content Import a new offset allocator to address the following issue: 1. Support memory allocation with a size larger than 16MB, which will help eliminate redundant memcpy invocations within non–zero-copy get interfaces. https://github.com/kvcache-ai/Mooncake/pull/624 2. Auto-merge the freed memory to avoid this bug: https://github.com/kvcache-ai/Mooncake/issues/632 Compared with the original implementation, the imported OffsetAllocator is wrapped to support registering or allocating with a size larger than 3.75 GB, and is further optimized for LLM inference scenario. The optimization will: a) slightly decrease the memory utilization ratio in general cases; b) makes no difference when the allocated size is equal to a bin size; c) largely improve the memory utilization ratio when the allocated size is mostly uniform and not equal to any bin size. The LLM inference tasks, in most cases, will put objects with mostly uniform sizes, which falls into the latter two cases. See the following benchmark reports for details. ## Result - alloc size: The size of each object - utilization ratio: The total allocated size / total space - time: time in nanoseconds for each object allocation - OffsetAllocator optimization: whether round up the allocated size to a bin size ### Uniform size, size equals power of 2 **OffsetAllocator (After Optimization)** ``` Alloc size: 32, min util ratio: 1, avg util ratio: 1, time: 544 ns Alloc size: 128, min util ratio: 1, avg util ratio: 1, time: 417 ns Alloc size: 512, min util ratio: 1, avg util ratio: 1, time: 174 ns Alloc size: 2048, min util ratio: 1, avg util ratio: 1, time: 406 ns Alloc size: 8192, min util ratio: 1, avg util ratio: 1, time: 180 ns Alloc size: 32768, min util ratio: 1, avg util ratio: 1, time: 133 ns Alloc size: 131072, min util ratio: 1, avg util ratio: 1, time: 109 ns Alloc size: 524288, min util ratio: 1, avg util ratio: 1, time: 100 ns Alloc size: 2097152, min util ratio: 1, avg util ratio: 1, time: 99 ns Alloc size: 8388608, min util ratio: 1, avg util ratio: 1, time: 99 ns Alloc size: 33554432, min util ratio: 1, avg util ratio: 1, time: 98 ns ``` **OffsetAllocator (Before Optimization)** ``` Alloc size: 32, min util ratio: 1, avg util ratio: 1, time: 539 ns Alloc size: 128, min util ratio: 1, avg util ratio: 1, time: 419 ns Alloc size: 512, min util ratio: 1, avg util ratio: 1, time: 217 ns Alloc size: 2048, min util ratio: 1, avg util ratio: 1, time: 408 ns Alloc size: 8192, min util ratio: 1, avg util ratio: 1, time: 175 ns Alloc size: 32768, min util ratio: 1, avg util ratio: 1, time: 130 ns Alloc size: 131072, min util ratio: 1, avg util ratio: 1, time: 107 ns Alloc size: 524288, min util ratio: 1, avg util ratio: 1, time: 99 ns Alloc size: 2097152, min util ratio: 1, avg util ratio: 1, time: 100 ns Alloc size: 8388608, min util ratio: 1, avg util ratio: 1, time: 98 ns Alloc size: 33554432, min util ratio: 1, avg util ratio: 1, time: 98 ns ``` ### Uniform size, size equals power of 2 +/- 17 **OffsetAllocator (After Optimization)** ``` Alloc size: 15, min util ratio: 1, avg util ratio: 1, time: 568 ns Alloc size: 111, min util ratio: 0.991071, avg util ratio: 0.991071, time: 441 ns Alloc size: 495, min util ratio: 0.966797, avg util ratio: 0.966797, time: 178 ns Alloc size: 2031, min util ratio: 0.991699, avg util ratio: 0.991699, time: 418 ns Alloc size: 8175, min util ratio: 0.997925, avg util ratio: 0.997925, time: 170 ns Alloc size: 32751, min util ratio: 0.999481, avg util ratio: 0.999481, time: 133 ns Alloc size: 131055, min util ratio: 0.99987, avg util ratio: 0.99987, time: 109 ns Alloc size: 524271, min util ratio: 0.999968, avg util ratio: 0.999968, time: 100 ns Alloc size: 2097135, min util ratio: 0.999992, avg util ratio: 0.999992, time: 99 ns Alloc size: 8388591, min util ratio: 0.999998, avg util ratio: 0.999998, time: 98 ns Alloc size: 33554415, min util ratio: 0.999999, avg util ratio: 0.999999, time: 99 ns Alloc size: 49, min util ratio: 0.942308, avg util ratio: 0.942308, time: 508 ns Alloc size: 145, min util ratio: 0.906249, avg util ratio: 0.906249, time: 372 ns Alloc size: 529, min util ratio: 0.918399, avg util ratio: 0.918399, time: 172 ns Alloc size: 2065, min util ratio: 0.896267, avg util ratio: 0.896267, time: 403 ns Alloc size: 8209, min util ratio: 0.89073, avg util ratio: 0.89073, time: 174 ns Alloc size: 32785, min util ratio: 0.889347, avg util ratio: 0.889347, time: 131 ns Alloc size: 131089, min util ratio: 0.88897, avg util ratio: 0.88897, time: 105 ns Alloc size: 524305, min util ratio: 0.888701, avg util ratio: 0.888701, time: 102 ns Alloc size: 2097169, min util ratio: 0.888679, avg util ratio: 0.888679, time: 100 ns Alloc size: 8388625, min util ratio: 0.886721, avg util ratio: 0.886721, time: 100 ns Alloc size: 33554449, min util ratio: 0.875, avg util ratio: 0.875, time: 100 ns ``` **OffsetAllocator (Before Optimization)** ``` Alloc size: 15, min util ratio: 1, avg util ratio: 1, time: 566 ns Alloc size: 111, min util ratio: 0.669866, avg util ratio: 0.710845, time: 703 ns Alloc size: 495, min util ratio: 0.665779, avg util ratio: 0.676874, time: 238 ns Alloc size: 2031, min util ratio: 0.668333, avg util ratio: 0.705411, time: 637 ns Alloc size: 8175, min util ratio: 0.666175, avg util ratio: 0.676474, time: 242 ns Alloc size: 32751, min util ratio: 0.664435, avg util ratio: 0.669078, time: 168 ns Alloc size: 131055, min util ratio: 0.66062, avg util ratio: 0.667341, time: 124 ns Alloc size: 524271, min util ratio: 0.653055, avg util ratio: 0.666993, time: 118 ns Alloc size: 2097135, min util ratio: 0.64062, avg util ratio: 0.666873, time: 116 ns Alloc size: 8388591, min util ratio: 0.605468, avg util ratio: 0.667812, time: 115 ns Alloc size: 33554415, min util ratio: 0.5625, avg util ratio: 0.670944, time: 116 ns Alloc size: 49, min util ratio: 0.692229, avg util ratio: 0.753062, time: 1122 ns Alloc size: 145, min util ratio: 0.667789, avg util ratio: 0.700907, time: 572 ns Alloc size: 529, min util ratio: 0.66577, avg util ratio: 0.676238, time: 238 ns Alloc size: 2065, min util ratio: 0.667926, avg util ratio: 0.704884, time: 632 ns Alloc size: 8209, min util ratio: 0.665708, avg util ratio: 0.676372, time: 239 ns Alloc size: 32785, min util ratio: 0.664224, avg util ratio: 0.669058, time: 168 ns Alloc size: 131089, min util ratio: 0.659631, avg util ratio: 0.667287, time: 129 ns Alloc size: 524305, min util ratio: 0.652609, avg util ratio: 0.666884, time: 122 ns Alloc size: 2097169, min util ratio: 0.638677, avg util ratio: 0.666516, time: 120 ns Alloc size: 8388625, min util ratio: 0.60547, avg util ratio: 0.665131, time: 121 ns Alloc size: 33554449, min util ratio: 0.546875, avg util ratio: 0.660917, time: 120 ns ``` ### Uniform size, size equals power of 2 multiply 0.9 or 1.1 **OffsetAllocator (After Optimization)** ``` Alloc size: 28, min util ratio: 1, avg util ratio: 1, time: 543 ns Alloc size: 115, min util ratio: 0.958333, avg util ratio: 0.958333, time: 418 ns Alloc size: 460, min util ratio: 0.958332, avg util ratio: 0.958332, time: 189 ns Alloc size: 1843, min util ratio: 0.959896, avg util ratio: 0.959896, time: 418 ns Alloc size: 7372, min util ratio: 0.959895, avg util ratio: 0.959895, time: 197 ns Alloc size: 29491, min util ratio: 0.959993, avg util ratio: 0.959993, time: 135 ns Alloc size: 117964, min util ratio: 0.959979, avg util ratio: 0.959979, time: 111 ns Alloc size: 471859, min util ratio: 0.959985, avg util ratio: 0.959985, time: 100 ns Alloc size: 1887436, min util ratio: 0.959765, avg util ratio: 0.959765, time: 99 ns Alloc size: 7549747, min util ratio: 0.959766, avg util ratio: 0.959766, time: 99 ns Alloc size: 30198988, min util ratio: 0.95625, avg util ratio: 0.95625, time: 99 ns Alloc size: 35, min util ratio: 0.972222, avg util ratio: 0.972222, time: 531 ns Alloc size: 140, min util ratio: 0.972222, avg util ratio: 0.972222, time: 397 ns Alloc size: 563, min util ratio: 0.977427, avg util ratio: 0.977427, time: 180 ns Alloc size: 2252, min util ratio: 0.97743, avg util ratio: 0.97743, time: 389 ns Alloc size: 9011, min util ratio: 0.977752, avg util ratio: 0.977752, time: 183 ns Alloc size: 36044, min util ratio: 0.977752, avg util ratio: 0.977752, time: 133 ns Alloc size: 144179, min util ratio: 0.977739, avg util ratio: 0.977739, time: 106 ns Alloc size: 576716, min util ratio: 0.977538, avg util ratio: 0.977538, time: 103 ns Alloc size: 2306867, min util ratio: 0.977539, avg util ratio: 0.977539, time: 99 ns Alloc size: 9227468, min util ratio: 0.975391, avg util ratio: 0.975391, time: 99 ns Alloc size: 36909875, min util ratio: 0.9625, avg util ratio: 0.9625, time: 100 ns ``` **OffsetAllocator (Before Optimization)** ``` Alloc size: 28, min util ratio: 1, avg util ratio: 1, time: 539 ns Alloc size: 115, min util ratio: 0.669299, avg util ratio: 0.709245, time: 701 ns Alloc size: 460, min util ratio: 0.665825, avg util ratio: 0.677532, time: 255 ns Alloc size: 1843, min util ratio: 0.669352, avg util ratio: 0.709202, time: 691 ns Alloc size: 7372, min util ratio: 0.66619, avg util ratio: 0.677401, time: 260 ns Alloc size: 29491, min util ratio: 0.664311, avg util ratio: 0.669511, time: 172 ns Alloc size: 117964, min util ratio: 0.661812, avg util ratio: 0.667356, time: 133 ns Alloc size: 471859, min util ratio: 0.654345, avg util ratio: 0.667048, time: 123 ns Alloc size: 1887436, min util ratio: 0.640722, avg util ratio: 0.666447, time: 121 ns Alloc size: 7549747, min util ratio: 0.611719, avg util ratio: 0.666847, time: 119 ns Alloc size: 30198988, min util ratio: 0.548437, avg util ratio: 0.669799, time: 125 ns Alloc size: 35, min util ratio: 0.7098, avg util ratio: 0.774162, time: 1306 ns Alloc size: 140, min util ratio: 0.667934, avg util ratio: 0.702151, time: 599 ns Alloc size: 563, min util ratio: 0.665599, avg util ratio: 0.675548, time: 239 ns Alloc size: 2252, min util ratio: 0.667371, avg util ratio: 0.701623, time: 601 ns Alloc size: 9011, min util ratio: 0.665485, avg util ratio: 0.675528, time: 244 ns Alloc size: 36044, min util ratio: 0.663248, avg util ratio: 0.668912, time: 170 ns Alloc size: 144179, min util ratio: 0.660308, avg util ratio: 0.666934, time: 127 ns Alloc size: 576716, min util ratio: 0.654467, avg util ratio: 0.66679, time: 122 ns Alloc size: 2306867, min util ratio: 0.633789, avg util ratio: 0.666159, time: 121 ns Alloc size: 9227468, min util ratio: 0.597266, avg util ratio: 0.666037, time: 118 ns Alloc size: 36909875, min util ratio: 0.55, avg util ratio: 0.669564, time: 121 ns ``` ### Random Size **OffsetAllocator (After Optimization)** ``` util ratio (min / p99 / p90 / p50 / max / avg): 0.544250 / 0.713338 / 0.779739 / 0.847867 / 0.952591 / 0.841576 avg alloc time: 145.575738 ns/op ``` **OffsetAllocator (Before Optimization)** ``` util ratio (min / p99 / p90 / p50 / max / avg): 0.569255 / 0.712076 / 0.781224 / 0.855046 / 0.976057 / 0.848873 avg alloc time: 142.508508 ns/op ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-29 21:29:43 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#1712
No description provided.