[GH-ISSUE #4518] [RFC]: Session-aware KV membership and explicit soft-pin controls #1152

New issue

Open

opened 2026-10-29 21:23:22 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:22 +00:00

Owner

Originally created by @magicYang1573 on GitHub (Oct 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4518

Changes proposed

Motivation

A serving system can reuse the same KV prefix across conversations and across data-parallel workers. An insert-only session tag cannot represent all current users of a shared object, and applications need to release retention intent after context compaction without deleting cached data.

This proposal adds runtime session membership and explicit soft-retention controls to Mooncake Store. Applications provide a unique string session ID; multiple clients may contribute keys to it. Store continues to operate on physical object keys and does not interpret prompts, token order, or sliding-window attention.

API

Python methods on an initialized real MooncakeDistributedStore client:

Interface	Purpose
`ReplicateConfig.kv_sessions: list[list[str]]`	Optionally tag each written physical key with one or more sessions. Registration happens at write admission and also covers duplicate Put.
`attach_kv_session(session_id, keys)`	Add associations to existing objects, e.g. after a prefix hit; return a status for each key. Create an unknown session on its first successful association.
`pin_kv_session(session_id)`	Give an existing session's current and subsequent members soft-pin priority, without a TTL.
`unpin_kv_session(session_id)`	Remove this session's retention preference while preserving membership and other sessions' pins.
`update_kv_session(session_id, keep_keys)`	Retain the intersection of current membership and one complete physical-key keep set after compaction/rollback. Do not attach new keys or delete payloads.
`close_kv_session(session_id)`	Remove the session's associations and pin contribution; shared object data remains cached.
`get_kv_session(session_id)`	Inspect session ID, pin state, and member count.
`list_kv_session_keys(session_id, cursor="", limit=1024)`	Enumerate current members using bounded, weakly consistent pages.

C++ clients expose corresponding operations, with SetKvSessionPin for pin/unpin. No create, generation handle, or per-key detach API is introduced. Session IDs use the client's existing tenant context; callers do not supply an additional tenant parameter.

Semantics and implementation

Membership is many-to-many and independent of group_ids, placement, and ranged-write sessions. It does not impose atomic group eviction.
An object is effectively soft pinned if any associated session is pinned or its existing native TTL soft pin is active. Eligible ordinary objects remain preferred eviction victims; native allow_evict_soft_pinned_objects controls pressure fallback. Read leases, hard pins, busy replicas, and group rules retain their existing meaning.
The last member's removal deletes the session and its pin state. A later attach/tagged write starts an unpinned session. Pinning an unknown session fails; empty sessions do not preserve future pin intent.
Pin/unpin/update are best effort policy operations and may overlap inference, writes, and registration. Late registration may restore removed membership. Update is a paged walk, not a whole-session transaction. Exact pruning requires optional caller coordination; final close requires stopping new session work and has no tombstone.
A bounded registry indexes sessions to keys; object metadata holds session references for eviction checks. Short index critical sections and existing metadata locks protect memory safety. Pin changes one atomic flag; eviction reads it without the registry lock.
Runtime session state is not persisted in snapshots or HA logs. Applications rebuild associations and desired pins after restart/promotion.
Untagged writes retain the existing RPC names/config encoding. Tagged writes use separate admission RPCs so old masters cannot silently ignore tags. Session-aware clients and masters must both support the feature.
Configurable session/member/per-object/batch limits bound registry growth. Large sessions and high fan-out still need workload-scale performance evaluation.

Relationship to existing proposals

#2819 / #2830 discuss workload hints with insert-only session/priority metadata. This proposal addresses mutable many-to-many membership, whole-session lifecycle controls, and an explicit binary soft-pin signal consumed by eviction. It does not add numeric priority or persist session metadata. The API overlap should be discussed before either interface is stabilized.
#2948 / #2835 discuss fixed-lifetime retention for existing groups, including intent before objects exist. This proposal instead uses explicit pin/unpin on nonempty sessions and preserves independent group semantics.

Validation and review questions

The implementation includes native registry/master/group regression tests and real TCP Python client tests for shared membership, tagged writes, cleanup, limits, update, and eviction behavior. A local SGLang DPA=8 integration also exercises session controls during eight concurrent decodes; this is functional evidence, not a performance claim.

Feedback requested on the API naming and its relationship to workload hints, the runtime-only lifecycle, and the explicit pin/unpin versus TTL retention model.

Before submitting a new issue...

Searched related issues and PRs and read the Store/session and contribution documentation.

AI assistance: OpenAI Codex assisted with implementation, tests, and this proposal. The associated implementation PR will be submitted as a draft pending human line-by-line review.

Originally created by @magicYang1573 on GitHub (Oct 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4518 ### Changes proposed ## Motivation A serving system can reuse the same KV prefix across conversations and across data-parallel workers. An insert-only session tag cannot represent all current users of a shared object, and applications need to release retention intent after context compaction without deleting cached data. This proposal adds runtime session membership and explicit soft-retention controls to Mooncake Store. Applications provide a unique string session ID; multiple clients may contribute keys to it. Store continues to operate on physical object keys and does not interpret prompts, token order, or sliding-window attention. ## API Python methods on an initialized real `MooncakeDistributedStore` client: | Interface | Purpose | | --- | --- | | `ReplicateConfig.kv_sessions: list[list[str]]` | Optionally tag each written physical key with one or more sessions. Registration happens at write admission and also covers duplicate Put. | | `attach_kv_session(session_id, keys)` | Add associations to existing objects, e.g. after a prefix hit; return a status for each key. Create an unknown session on its first successful association. | | `pin_kv_session(session_id)` | Give an existing session's current and subsequent members soft-pin priority, without a TTL. | | `unpin_kv_session(session_id)` | Remove this session's retention preference while preserving membership and other sessions' pins. | | `update_kv_session(session_id, keep_keys)` | Retain the intersection of current membership and one complete physical-key keep set after compaction/rollback. Do not attach new keys or delete payloads. | | `close_kv_session(session_id)` | Remove the session's associations and pin contribution; shared object data remains cached. | | `get_kv_session(session_id)` | Inspect session ID, pin state, and member count. | | `list_kv_session_keys(session_id, cursor="", limit=1024)` | Enumerate current members using bounded, weakly consistent pages. | C++ clients expose corresponding operations, with `SetKvSessionPin` for pin/unpin. No create, generation handle, or per-key detach API is introduced. Session IDs use the client's existing tenant context; callers do not supply an additional tenant parameter. ## Semantics and implementation - Membership is many-to-many and independent of `group_ids`, placement, and ranged-write sessions. It does not impose atomic group eviction. - An object is effectively soft pinned if any associated session is pinned or its existing native TTL soft pin is active. Eligible ordinary objects remain preferred eviction victims; native `allow_evict_soft_pinned_objects` controls pressure fallback. Read leases, hard pins, busy replicas, and group rules retain their existing meaning. - The last member's removal deletes the session and its pin state. A later attach/tagged write starts an unpinned session. Pinning an unknown session fails; empty sessions do not preserve future pin intent. - Pin/unpin/update are best effort policy operations and may overlap inference, writes, and registration. Late registration may restore removed membership. Update is a paged walk, not a whole-session transaction. Exact pruning requires optional caller coordination; final close requires stopping new session work and has no tombstone. - A bounded registry indexes sessions to keys; object metadata holds session references for eviction checks. Short index critical sections and existing metadata locks protect memory safety. Pin changes one atomic flag; eviction reads it without the registry lock. - Runtime session state is not persisted in snapshots or HA logs. Applications rebuild associations and desired pins after restart/promotion. - Untagged writes retain the existing RPC names/config encoding. Tagged writes use separate admission RPCs so old masters cannot silently ignore tags. Session-aware clients and masters must both support the feature. - Configurable session/member/per-object/batch limits bound registry growth. Large sessions and high fan-out still need workload-scale performance evaluation. ## Relationship to existing proposals - #2819 / #2830 discuss workload hints with insert-only session/priority metadata. This proposal addresses mutable many-to-many membership, whole-session lifecycle controls, and an explicit binary soft-pin signal consumed by eviction. It does not add numeric priority or persist session metadata. The API overlap should be discussed before either interface is stabilized. - #2948 / #2835 discuss fixed-lifetime retention for existing groups, including intent before objects exist. This proposal instead uses explicit pin/unpin on nonempty sessions and preserves independent group semantics. ## Validation and review questions The implementation includes native registry/master/group regression tests and real TCP Python client tests for shared membership, tagged writes, cleanup, limits, update, and eviction behavior. A local SGLang DPA=8 integration also exercises session controls during eight concurrent decodes; this is functional evidence, not a performance claim. Feedback requested on the API naming and its relationship to workload hints, the runtime-only lifecycle, and the explicit pin/unpin versus TTL retention model. ### Before submitting a new issue... - [x] Searched related issues and PRs and read the Store/session and contribution documentation. AI assistance: OpenAI Codex assisted with implementation, tests, and this proposal. The associated implementation PR will be submitted as a draft pending human line-by-line review.

bsakaguchi commented

2026-10-29 21:23:23 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Thanks for opening this issue, @magicYang1573!

Field	Value
Issue	#4518
GitHub user ID	`102229987`
Reporter	@magicYang1573

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 8, 2026): Thanks for opening this issue, @magicYang1573! | Field | Value | |-------|-------| | **Issue** | #4518 | | **GitHub user ID** | `102229987` | | **Reporter** | @magicYang1573 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.