[GH-ISSUE #381] [RFC][Store] Support cache-aware scheduler #134

New issue

Closed

opened 2026-10-29 21:09:57 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:09:57 +00:00

Owner

Originally created by @zhaoyongke on GitHub (May 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/381

In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations.
DeepSeek API's price is lowered by context caching technology. Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate.

With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow:

whether a key is exist in store
which node(s) holds the value corresponding to a key
which device(s) holds this key? VRAM, DRAM or SSD?

For better cache-hit and runtime performance, some optimizations are also needed:

RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy
memcpy optimization for faster loading
Near-Engine cache managing, sync each engine's meta to master(Currently we are working on SGLang HiRadixCache)
Preload/Warmup high-demand kvcaches with num_replicas > 1 in run-time(WIP)

Welcome to discuss!

Originally created by @zhaoyongke on GitHub (May 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/381 In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations. DeepSeek API's price is lowered by [context caching technology](https://api-docs.deepseek.com/news/news0802). Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate. With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow: - whether a key is exist in store - which node(s) holds the value corresponding to a key - which device(s) holds this key? VRAM, DRAM or SSD? For better cache-hit and runtime performance, some optimizations are also needed: - RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy - [memcpy optimization for faster loading](https://github.com/kvcache-ai/Mooncake/pull/340) - Near-Engine cache managing, sync each engine's meta to master(Currently we are working on SGLang HiRadixCache) - Preload/Warmup high-demand kvcaches with num_replicas > 1 in run-time(WIP) Welcome to discuss!

bsakaguchi

2026-10-29 21:09:57 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:09:57 +00:00

Author

Owner

@stmatengss commented on GitHub (May 22, 2025):

In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations. DeepSeek API's price is lowered by context caching technology. Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate.

With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow:

whether a key is exist in store

which node(s) holds the value corresponding to a key

which device(s) holds this key? VRAM, DRAM or SSD?

For better cache-hit and runtime performance, some optimizations are also needed:

RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)

memcpy optimization for faster loading

Near-Engine cache managing, sync each engine's meta to master

Welcome to discuss!

"RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)" => agree with that.
To support cache-aware schedulers, we should provide more attributes for each kv item.
Currently, SSD support is WIP. Any opinion?

@stmatengss commented on GitHub (May 22, 2025): > In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations. DeepSeek API's price is lowered by [context caching technology](https://api-docs.deepseek.com/news/news0802). Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate. > > With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow: > > * whether a key is exist in store > * which node(s) holds the value corresponding to a key > * which device(s) holds this key? VRAM, DRAM or SSD? > > For better cache-hit and runtime performance, some optimizations are also needed: > > * RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP) > * [memcpy optimization for faster loading](https://github.com/kvcache-ai/Mooncake/pull/340) > * Near-Engine cache managing, sync each engine's meta to master > > Welcome to discuss! 1. "RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)" => agree with that. 2. To support cache-aware schedulers, we should provide more attributes for each kv item. 3. Currently, SSD support is WIP. Any opinion?

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@zhaoyongke commented on GitHub (May 22, 2025):

In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations. DeepSeek API's price is lowered by context caching technology. Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate.
With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow:

whether a key is exist in store

which node(s) holds the value corresponding to a key

which device(s) holds this key? VRAM, DRAM or SSD?

For better cache-hit and runtime performance, some optimizations are also needed:

RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)

memcpy optimization for faster loading

Near-Engine cache managing, sync each engine's meta to master

Welcome to discuss!

"RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)" => agree with that.

To support cache-aware schedulers, we should provide more attributes for each kv item.

Currently, SSD support is WIP. Any opinion?

Mooncake Store only hold part kvs, some near-engine kvs are in VRAM or DRAM(with Paged Attention and Radix Cache management), they're important for cache-aware scheduler too.

KVs offloaded to SSD or 3FS are permanent, not suit for cache-aware node scheduler because it takes longer to get()/put() than VRAM/DRAM, and better for lower cost offline inferencing cases but not suit for lower latency online service. We're planning to benchmark all these cases in our H20 cluster.

@zhaoyongke commented on GitHub (May 22, 2025): > > In LLM inference clusters, a global scheduler is often used for load-balancing, choose the optimal nodes for new requests processing. KVCaches can also be reused if a node has processed the same prompt or a common prefix to reduce computations. DeepSeek API's price is lowered by [context caching technology](https://api-docs.deepseek.com/news/news0802). Cache-aware scheduler will allocate nodes with both lower computing load and higher cache-hit rate. > > With Mooncake Store, a master service is launched as a storage allocation indicator. It also provides kvcache put/get metrics and health status. It should provide more information for cache-aware schedulers as follow: > > > > * whether a key is exist in store > > * which node(s) holds the value corresponding to a key > > * which device(s) holds this key? VRAM, DRAM or SSD? > > > > For better cache-hit and runtime performance, some optimizations are also needed: > > > > * RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP) > > * [memcpy optimization for faster loading](https://github.com/kvcache-ai/Mooncake/pull/340) > > * Near-Engine cache managing, sync each engine's meta to master > > > > Welcome to discuss! > > 1. "RandomAllocationStrategy should be changed to LocalFirstAllocationStrategy(WIP)" => agree with that. > 2. To support cache-aware schedulers, we should provide more attributes for each kv item. > 3. Currently, SSD support is WIP. Any opinion? Mooncake Store only hold part kvs, some near-engine kvs are in VRAM or DRAM(with Paged Attention and Radix Cache management), they're important for cache-aware scheduler too. KVs offloaded to SSD or 3FS are permanent, not suit for cache-aware node scheduler because it takes longer to get()/put() than VRAM/DRAM, and better for lower cost offline inferencing cases but not suit for lower latency online service. We're planning to benchmark all these cases in our H20 cluster.

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 11, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 11, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:09:58 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 19, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 19, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:26:37 +00:00

[PR #134] [CLOSED] [TransferEngine] Improve local-to-local memcpy performance #1347

bsakaguchi referenced this issue

2026-10-29 21:26:49 +00:00

[PR #170] [CLOSED] [TransferEngine] Add shared memory transport and refactor metadata format #1369