[PR #587] [MERGED] [Store] Soft Pin for Important Object #1672

New issue

Closed

opened 2026-10-29 21:29:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:23 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/587
Author: @ykwd
Created: 7/3/2025
Status: ✅ Merged
Merged: 7/8/2025
Merged by: @ykwd

Base: main ← Head: feature/vip_kv

📝 Commits (10+)

0842a25 Basic implementation with unit tests
fdf08fe Bug fix
bad6935 Optimize the eviciton algorithm
851875e Add flag to configure allowing evict oft pinned objects or not
9187cbc Update docs for soft pin
c1c443c Bug fix and Minor updates
e399d7b Merge main
9bfc61c Minor update
ab455a0 merge main
89e47bc Fix ci error: Only enable ssd offload feature in test_ssd_offload_in_evict.py

📊 Changes

18 files changed (+613 additions, -102 deletions)

View changed files

📝 .github/workflows/ci.yml (+1 -30)
📝 doc/en/mooncake-store-preview.md (+15 -1)
📝 doc/zh/mooncake-store-preview.md (+15 -1)
📝 docs/source/design/mooncake-store-preview.md (+15 -1)
📝 mooncake-store/include/ha_helper.h (+6 -2)
📝 mooncake-store/include/master_metric_manager.h (+4 -0)
📝 mooncake-store/include/master_service.h (+56 -13)
📝 mooncake-store/include/rpc_service.h (+6 -1)
📝 mooncake-store/include/types.h (+5 -0)
📝 mooncake-store/src/ha_helper.cpp (+9 -5)
📝 mooncake-store/src/master.cpp (+13 -2)
📝 mooncake-store/src/master_metric_manager.cpp (+12 -1)
📝 mooncake-store/src/master_service.cpp (+174 -25)
📝 mooncake-store/tests/allocation_strategy_test.cpp (+9 -9)
📝 mooncake-store/tests/client_integration_test.cpp (+3 -3)
📝 mooncake-store/tests/master_service_test.cpp (+263 -2)
📝 mooncake-wheel/tests/test_ssd_offload_in_evict.py (+1 -3)
📝 scripts/run_tests.sh (+6 -3)

📄 Description

This PR adds a soft pin mechanism for important and frequently used objects, such as system prompts. When putting an object, it can be configured to enable soft pin. During eviction, objects that are not soft pinned are prioritized for eviction. Soft pinned objects are only evicted when memory is insufficient and no other objects are eligible for eviction.

If a soft pinned object is not accessed for an extended period, its soft pin status will be removed. If it is accessed again later, it will automatically be soft pinned once more.

There are two startup parameters in master_service related to the soft pin mechanism:

default_kv_soft_pin_ttl: The duration (in milliseconds) after which a soft pinned object will have its soft pin status removed if not accessed. The default value is 30 minutes.
allow_evict_soft_pinned_objects: Whether soft pinned objects are allowed to be evicted. The default value is true.

Notably, soft pinned objects can still be removed using APIs such as Remove or RemoveAll.

P.S.

Currently, the soft pin cannot be set via Python APIs, as the Put or BatchPut APIs in Python do not have configuration parameters. Therefore, the soft pin mechanism is marked as "Upcoming" in the docs.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/587 **Author:** [@ykwd](https://github.com/ykwd) **Created:** 7/3/2025 **Status:** ✅ Merged **Merged:** 7/8/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `feature/vip_kv` --- ### 📝 Commits (10+) - [`0842a25`](https://github.com/kvcache-ai/Mooncake/commit/0842a254ce93423c80171cdb0a5e40b222a65751) Basic implementation with unit tests - [`fdf08fe`](https://github.com/kvcache-ai/Mooncake/commit/fdf08feeed3c221c741c3c90458b7949ef6bc4ec) Bug fix - [`bad6935`](https://github.com/kvcache-ai/Mooncake/commit/bad69355f3fb184673aa902fe8028d529a4a03c4) Optimize the eviciton algorithm - [`851875e`](https://github.com/kvcache-ai/Mooncake/commit/851875e25e3f6a23bed0408c680676c2a846ba77) Add flag to configure allowing evict oft pinned objects or not - [`9187cbc`](https://github.com/kvcache-ai/Mooncake/commit/9187cbcce53d92d0c7226973c6323462bf937c89) Update docs for soft pin - [`c1c443c`](https://github.com/kvcache-ai/Mooncake/commit/c1c443c3c88fac64cf6693ae22b1a4e910ae3318) Bug fix and Minor updates - [`e399d7b`](https://github.com/kvcache-ai/Mooncake/commit/e399d7b99a2d4d86f70c6686ffc7dd4c80058947) Merge main - [`9bfc61c`](https://github.com/kvcache-ai/Mooncake/commit/9bfc61c5a58a635f9b50432d88387be3529008c9) Minor update - [`ab455a0`](https://github.com/kvcache-ai/Mooncake/commit/ab455a05ddc816df09fc1a65d7aa98439ab89b4a) merge main - [`89e47bc`](https://github.com/kvcache-ai/Mooncake/commit/89e47bc7899a0245abd64f2f57e8c6d1aa9217f7) Fix ci error: Only enable ssd offload feature in test_ssd_offload_in_evict.py ### 📊 Changes **18 files changed** (+613 additions, -102 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+1 -30) 📝 `doc/en/mooncake-store-preview.md` (+15 -1) 📝 `doc/zh/mooncake-store-preview.md` (+15 -1) 📝 `docs/source/design/mooncake-store-preview.md` (+15 -1) 📝 `mooncake-store/include/ha_helper.h` (+6 -2) 📝 `mooncake-store/include/master_metric_manager.h` (+4 -0) 📝 `mooncake-store/include/master_service.h` (+56 -13) 📝 `mooncake-store/include/rpc_service.h` (+6 -1) 📝 `mooncake-store/include/types.h` (+5 -0) 📝 `mooncake-store/src/ha_helper.cpp` (+9 -5) 📝 `mooncake-store/src/master.cpp` (+13 -2) 📝 `mooncake-store/src/master_metric_manager.cpp` (+12 -1) 📝 `mooncake-store/src/master_service.cpp` (+174 -25) 📝 `mooncake-store/tests/allocation_strategy_test.cpp` (+9 -9) 📝 `mooncake-store/tests/client_integration_test.cpp` (+3 -3) 📝 `mooncake-store/tests/master_service_test.cpp` (+263 -2) 📝 `mooncake-wheel/tests/test_ssd_offload_in_evict.py` (+1 -3) 📝 `scripts/run_tests.sh` (+6 -3) </details> ### 📄 Description ## Contents This PR adds a soft pin mechanism for important and frequently used objects, such as system prompts. When putting an object, it can be configured to enable soft pin. During eviction, objects that are not soft pinned are prioritized for eviction. Soft pinned objects are only evicted when memory is insufficient and no other objects are eligible for eviction. If a soft pinned object is not accessed for an extended period, its soft pin status will be removed. If it is accessed again later, it will automatically be soft pinned once more. There are two startup parameters in `master_service` related to the soft pin mechanism: - `default_kv_soft_pin_ttl`: The duration (in milliseconds) after which a soft pinned object will have its soft pin status removed if not accessed. The default value is `30 minutes`. - `allow_evict_soft_pinned_objects`: Whether soft pinned objects are allowed to be evicted. The default value is `true`. Notably, soft pinned objects can still be removed using APIs such as `Remove` or `RemoveAll`. ## P.S. Currently, the soft pin cannot be set via Python APIs, as the `Put` or `BatchPut` APIs in Python do not have configuration parameters. Therefore, the soft pin mechanism is marked as "Upcoming" in the docs. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:23 +00:00