[GH-ISSUE #1062] [RFC]: Add Local Cache Mechanism for Mooncake Store Client #351

New issue

Closed

opened 2026-10-29 21:13:10 +00:00 by bsakaguchi · 9 comments

bsakaguchi commented 2026-10-29 21:13:10 +00:00

Owner

Copy link

Originally created by @Shichang-Zhang on GitHub (Nov 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1062

Originally assigned to: @Shichang-Zhang on GitHub.

Changes proposed

Introduction

Hello Mooncake community,
We are from the openFuyao community and are actively working on cloud-native LLM inference acceleration. We would like to share our initial proposal, which introduces a hot data caching mechanism based on the LRU (Least Recently Used) eviction strategy for Mooncake Store. This optimization aims to improve memory utilization, reduce cross-node network access overhead, enhance local availability of hot data, and thereby improve the performance of single-key and batch get operations in distributed KV Cache storage.

Motivation

The Master Service randomly assigns the data slices to storage nodes in the cluster. This architectural design results in frequent cross-node network transmission to get each slice when accessing specific KV Cache data. Furthermore, research from the Mooncake paper indicates that over 50% of data is accessed only once, while some hot data can be accessed tens of thousands of times.

This provides clear directions for the optimization:

• Improve local node hit rate for storage slices
• Improve read efficiency for data slices stored on the local node

This proposal introduces a client-side caching mechanism based on the LRU eviction strategy, which accelerates KV Cache reads by caching hot data, thereby improving TTFT.

Goals

• Performance Improvement: After enabling local hot cache optimization and when local slices hit rate reaches 30%+, the TTFT metric is expected to improve by 10% compared to the disabled scenario
• Local Hit Rate Improvement: Under typical workloads, local slices hit rate improves to 30%, get/batchGet interface latency is expected to reduce by 40%+
• Interface Compatibility: Maintain Mooncake's get/batchGet interface unchanged, supporting multiple inference engines (including vLLM, SGLang, etc.) to use KV Cache read acceleration

Non-Goals

• Do not involve optimization at the SSD storage layer
• Do not involve Master Service refactoring or metadata descriptor struct changes

Proposal

Architecture

This section provides an overall architectural view of the Mooncake Store acceleration solution.

Logical Architecture View

This proposal enhances Mooncake's storage strategy:

Logical Architecture Description:

• Inference Engine Layer: Maintain interface compatibility, no modifications required
• Mooncake Client: Add hot cache module (highlighted in green), optimize transfer submission logic
• Transport Layer: Process local and remote transfers in parallel
• Storage Layer: Master Service manages distributed storage. Add local hot cache storage (highlighted in green).

Implementation

Feature Overview

The current Mooncake get/batchGet request processing flow is: query metadata → submit transfer task → wait for transfer completion → return data directly.

The optimization introduces a hot data caching mechanism on the Client side:

• During Client initialization, allocate dedicated memory region for storing hot slices
• When receiving get/batchGet requests, check local cache first
• After Transfer Engine completes remote slices reads, asynchronously put remote slices to local cache

Sequence Diagram

1. Initialization Phase: When creating Client, initialize hot cache based on configuration parameter local_hot_cache_size_ (default value 0, standing for disabled)
2. Metadata Query: When Get/BatchGet requests arrive, first query Master service to obtain storage location information (replica descriptor) for target KV Cache
3. Cache Query: Iterate through all slices and query local cache. If cache hit, update replica descriptor to local store address
4. Transfer Submission: Submit read requests to Transfer Submitter. If target address is local node, automatically select LOCAL_MEMCPY transport strategy for optimization
5. Cache Update: After transfer completion, update remotely transferred slices to local hot cache through asynchronous task handler

Data Structures

• New Client class members:

  • hot_cache_: Hot cache manager instance
  • hot_cache_handler_: Hot cache asynchronous task handler instance

• LocalHotCache class: Hot cache management class for cache lifecycle management, it involves:

  • total_size_bytes_: Hot cache size limit (in bytes)
  • blocks_: Hot cache memory block list, store actual cached slices
  • lru_queue_: Doubly linked list, maintain LRU eviction strategy
  • key_to_lru_it_: Cache key-value mapping table, where key is {request_key}_{slice_index} and value is LRU list node iterator
  • lru_mutex_: Mutex for concurrent access protection

• HotMemBlock struct: Hot cache data block structure

  • address_: Memory address pointer
  • size_: Data block size (in bytes)

• LocalHotCacheHandler class: Hot cache asynchronous task handler

  • Asynchronously update TE transferred slices to hot cache after Get/BatchGet read complete

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Shichang-Zhang on GitHub (Nov 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1062 Originally assigned to: @Shichang-Zhang on GitHub. ### Changes proposed ## Introduction Hello Mooncake community, We are from the [openFuyao community](https://www.openfuyao.cn/zh/) and are actively working on cloud-native LLM inference acceleration. We would like to share our initial proposal, which introduces a hot data caching mechanism based on the LRU (Least Recently Used) eviction strategy for Mooncake Store. This optimization aims to improve memory utilization, reduce cross-node network access overhead, enhance local availability of hot data, and thereby improve the performance of single-key and batch get operations in distributed KV Cache storage. ## Motivation The Master Service randomly assigns the data slices to storage nodes in the cluster. This architectural design results in frequent cross-node network transmission to get each slice when accessing specific KV Cache data. Furthermore, research from the [Mooncake paper](https://arxiv.org/pdf/2407.00079) indicates that over 50% of data is accessed only once, while some hot data can be accessed tens of thousands of times. This provides clear directions for the optimization: - Improve local node hit rate for storage slices - Improve read efficiency for data slices stored on the local node This proposal introduces a client-side caching mechanism based on the LRU eviction strategy, which accelerates KV Cache reads by caching hot data, thereby improving TTFT. ### Goals - **Performance Improvement**: After enabling local hot cache optimization and when local slices hit rate reaches 30%+, the TTFT metric is expected to improve by 10% compared to the disabled scenario - **Local Hit Rate Improvement**: Under typical workloads, local slices hit rate improves to 30%, get/batchGet interface latency is expected to reduce by 40%+ - **Interface Compatibility**: Maintain Mooncake's get/batchGet interface unchanged, supporting multiple inference engines (including vLLM, SGLang, etc.) to use KV Cache read acceleration ### Non-Goals - Do not involve optimization at the SSD storage layer - Do not involve Master Service refactoring or metadata descriptor struct changes ## Proposal ### Architecture This section provides an overall architectural view of the Mooncake Store acceleration solution. #### Logical Architecture View This proposal enhances Mooncake's storage strategy: <img width="1107" height="847" alt="Image" src="https://github.com/user-attachments/assets/7402dce9-45b1-4962-b980-b99cbbbcb925" /> **Logical Architecture Description:** - **Inference Engine Layer**: Maintain interface compatibility, no modifications required - **Mooncake Client**: Add hot cache module (highlighted in green), optimize transfer submission logic - **Transport Layer**: Process local and remote transfers in parallel - **Storage Layer**: Master Service manages distributed storage. Add local hot cache storage (highlighted in green). ### Implementation #### Feature Overview The current Mooncake get/batchGet request processing flow is: query metadata → submit transfer task → wait for transfer completion → return data directly. The optimization introduces a hot data caching mechanism on the Client side: - During Client initialization, allocate dedicated memory region for storing hot slices - When receiving get/batchGet requests, check local cache first - After Transfer Engine completes remote slices reads, asynchronously put remote slices to local cache #### Sequence Diagram <img width="1248" height="576" alt="Image" src="https://github.com/user-attachments/assets/2d1bca0a-811a-4689-8465-d86f3c0ef51f" /> 1. **Initialization Phase**: When creating Client, initialize hot cache based on configuration parameter `local_hot_cache_size_` (default value 0, standing for disabled) 2. **Metadata Query**: When Get/BatchGet requests arrive, first query Master service to obtain storage location information (replica descriptor) for target KV Cache 3. **Cache Query**: Iterate through all slices and query local cache. If cache hit, update replica descriptor to local store address 4. **Transfer Submission**: Submit read requests to Transfer Submitter. If target address is local node, automatically select `LOCAL_MEMCPY` transport strategy for optimization 5. **Cache Update**: After transfer completion, update remotely transferred slices to local hot cache through asynchronous task handler #### Data Structures - **New `Client` class members**: - `hot_cache_`: Hot cache manager instance - `hot_cache_handler_`: Hot cache asynchronous task handler instance - **`LocalHotCache` class**: Hot cache management class for cache lifecycle management, it involves: - `total_size_bytes_`: Hot cache size limit (in bytes) - `blocks_`: Hot cache memory block list, store actual cached slices - `lru_queue_`: Doubly linked list, maintain LRU eviction strategy - `key_to_lru_it_`: Cache key-value mapping table, where key is `{request_key}_{slice_index}` and value is LRU list node iterator - `lru_mutex_`: Mutex for concurrent access protection - **`HotMemBlock` struct**: Hot cache data block structure - `address_`: Memory address pointer - `size_`: Data block size (in bytes) - **`LocalHotCacheHandler` class**: Hot cache asynchronous task handler - Asynchronously update TE transferred slices to hot cache after Get/BatchGet read complete ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:13:10 +00:00

• closed this issue
• added the
  feature request
  stale
  auto-closed
  enhancement
  labels

bsakaguchi commented 2026-10-29 21:13:11 +00:00

Author

Owner

Copy link

@stmatengss commented on GitHub (Nov 17, 2025):

Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client.

<!-- gh-comment-id:3542750086 --> @stmatengss commented on GitHub (Nov 17, 2025): Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client.

bsakaguchi commented 2026-10-29 21:13:11 +00:00

Author

Owner

Copy link

@stmatengss commented on GitHub (Nov 17, 2025):

Another ideal, it seems we should support two kinds of clients. One is a standalone client without master. The other is distributed deployment with many clients and a master.

<!-- gh-comment-id:3542751532 --> @stmatengss commented on GitHub (Nov 17, 2025): Another ideal, it seems we should support two kinds of clients. One is a standalone client without master. The other is distributed deployment with many clients and a master.

bsakaguchi commented 2026-10-29 21:13:11 +00:00

Author

Owner

Copy link

@Keithwwa commented on GitHub (Nov 18, 2025):

Great idea! I have a few more questions:

1. The configuration of the local_hot_cache_size_ value seems to strongly depend on personal experience and application scenarios. Is this configuration value necessary?
2. Is the eviction policy for the local hot cache the same as that for the non-hot cache? @Shichang-Zhang

<!-- gh-comment-id:3544704872 --> @Keithwwa commented on GitHub (Nov 18, 2025): Great idea! I have a few more questions: 1. The configuration of the `local_hot_cache_size_` value seems to strongly depend on personal experience and application scenarios. Is this configuration value necessary? 2. Is the eviction policy for the local hot cache the same as that for the non-hot cache? @Shichang-Zhang

bsakaguchi commented 2026-10-29 21:13:11 +00:00

Author

Owner

Copy link

@Shichang-Zhang commented on GitHub (Nov 18, 2025):

Great idea! I have a few more questions:

1. The configuration of the local_hot_cache_size_ value seems to strongly depend on personal experience and application scenarios. Is this configuration value necessary?
2. Is the eviction policy for the local hot cache the same as that for the non-hot cache? @Shichang-Zhang

Thanks for you r comments!

1. Yes. The configuration of local_hot_cache_size_ is strongly depends on the inference scenario. We expect to achieve an optimal balance, where the benefit from reduced cross-node data transfer outweighs the extra local cache memory cost. local_hot_cache_size_ is configurable, user can simply disable by setting it to 0. Currently we are conducting tests across several scenarios, and we will detail the configuration recommendations with test scenarios in the PR.
2. Yes. Currently, the eviction policy for local hot cache is LRU, but there are more optimizations for non-hot cache LRU eviction such as soft-pin. They have no straightforward relationship.

<!-- gh-comment-id:3546825400 --> @Shichang-Zhang commented on GitHub (Nov 18, 2025): > Great idea! I have a few more questions: > > 1. The configuration of the `local_hot_cache_size_` value seems to strongly depend on personal experience and application scenarios. Is this configuration value necessary? > 2. Is the eviction policy for the local hot cache the same as that for the non-hot cache? [@Shichang-Zhang](https://github.com/Shichang-Zhang) Thanks for you r comments! 1. Yes. The configuration of `local_hot_cache_size_` is strongly depends on the inference scenario. We expect to achieve an optimal balance, where the benefit from reduced cross-node data transfer outweighs the extra local cache memory cost. `local_hot_cache_size_` is configurable, user can simply disable by setting it to 0. Currently we are conducting tests across several scenarios, and we will detail the configuration recommendations with test scenarios in the PR. 2. Yes. Currently, the eviction policy for local hot cache is LRU, but there are more optimizations for non-hot cache LRU eviction such as soft-pin. They have no straightforward relationship.

bsakaguchi commented 2026-10-29 21:13:12 +00:00

Author

Owner

Copy link

@Shichang-Zhang commented on GitHub (Nov 18, 2025):

Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client.

Appreciate the affirmation! I’ve completed parts of implementation and performance verification. The PR will be raised in the next few days, and I’ll link it to this issue once submitted.

<!-- gh-comment-id:3546870615 --> @Shichang-Zhang commented on GitHub (Nov 18, 2025): > Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client. Appreciate the affirmation! I’ve completed parts of implementation and performance verification. The PR will be raised in the next few days, and I’ll link it to this issue once submitted.

bsakaguchi commented 2026-10-29 21:13:12 +00:00

Author

Owner

Copy link

@Mag-FelixFelicis commented on GitHub (Nov 19, 2025):

Interesting idea!

<!-- gh-comment-id:3551020887 --> @Mag-FelixFelicis commented on GitHub (Nov 19, 2025): Interesting idea!

bsakaguchi commented 2026-10-29 21:13:12 +00:00

Author

Owner

Copy link

@NUABO commented on GitHub (Dec 2, 2025):

Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client.

Appreciate the affirmation! I’ve completed parts of implementation and performance verification. The PR will be raised in the next few days, and I’ll link it to this issue once submitted.

What good news!

<!-- gh-comment-id:3600742414 --> @NUABO commented on GitHub (Dec 2, 2025): > > Good idea. LGTM. Looking forward to your PR! Local caching can improve performance, and we should consider making this feature optional for the generic client. > > Appreciate the affirmation! I’ve completed parts of implementation and performance verification. The PR will be raised in the next few days, and I’ll link it to this issue once submitted. What good news!

bsakaguchi commented 2026-10-29 21:13:12 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 19, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4748522258 --> @github-actions[bot] commented on GitHub (Jun 19, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:13:13 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 26, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4806399911 --> @github-actions[bot] commented on GitHub (Jun 26, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#351

No description provided.