[GH-ISSUE #632] [Feature Request]: does mooncacke-store need a cachelib rebalance strategy #207

New issue

Closed

opened 2026-10-29 21:10:53 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:10:53 +00:00

Owner

Originally created by @wwq2333 on GitHub (Jul 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/632

Describe your feature request

When the cached obj sizes are not strictly consistent, it leads to poor fragmentation in the cache memory footprint;

Cachelib offers several rebalancing strategies to offset this behavior by asking the cache to restructure the underlying memory allocated among objects of different sizes.

This phenomenon occurs when we use mooncacke-store cluster as a large kvcache pool. Even if we remove_all all obj, putting values of different sizes will frequently fail to allocate; So should we set up some rebalance strategy?

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @wwq2333 on GitHub (Jul 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/632 ### Describe your feature request When the cached obj sizes are not strictly consistent, it leads to poor fragmentation in the cache memory footprint; > Cachelib offers several rebalancing strategies to offset this behavior by asking the cache to restructure the underlying memory allocated among objects of different sizes. This phenomenon occurs when we use mooncacke-store cluster as a large kvcache pool. Even if we remove_all all obj, putting values of different sizes will frequently fail to allocate; So should we set up some rebalance strategy? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:10:53 +00:00

bsakaguchi commented

2026-10-29 21:10:54 +00:00

Author

Owner

@ykwd commented on GitHub (Jul 16, 2025):

Thanks for reporting this — the issue you encountered is indeed a bug.

The root cause is that we only integrated a part of the cachelib. In the current implementation, there are multiple AllocationClass, each responsible for a different size range. When a memory allocation of size S is requested, the system selects an AllocationClass whose managed size is the smallest one greater than or equal to S. Once memory has been allocated under a specific AllocationClass, it remains tied to that class even after being freed, and cannot be reused for allocations of other sizes.

We've identified this issue and are actively working on a fix. Sorry for the inconvenience!

@ykwd commented on GitHub (Jul 16, 2025): Thanks for reporting this — the issue you encountered is indeed a bug. The root cause is that we only integrated a part of the `cachelib`. In the current implementation, there are multiple `AllocationClass`, each responsible for a different size range. When a memory allocation of size **S** is requested, the system selects an `AllocationClass` whose managed size is the smallest one greater than or equal to **S**. Once memory has been allocated under a specific `AllocationClass`, it remains tied to that class even after being freed, and cannot be reused for allocations of other sizes. We've identified this issue and are actively working on a fix. Sorry for the inconvenience!

bsakaguchi commented

2026-10-29 21:10:54 +00:00

Author

Owner

@ykwd commented on GitHub (Jul 24, 2025):

In this pr https://github.com/kvcache-ai/Mooncake/pull/657, we’ve introduced a new memory allocator, the OffsetAllocator, which resolves this issue. The previous CacheLib-based allocator is now optional, and fixes for it are currently in progress.

@ykwd commented on GitHub (Jul 24, 2025): In this pr https://github.com/kvcache-ai/Mooncake/pull/657, we’ve introduced a new memory allocator, the OffsetAllocator, which resolves this issue. The previous CacheLib-based allocator is now optional, and fixes for it are currently in progress.