kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1887] [Feature Request & Discussion]: Group Semantics for Objects in Mooncake Store #537

New issue

Closed

opened 2026-10-29 21:16:04 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:16:04 +00:00

Owner

Originally created by @ykwd on GitHub (Apr 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1887

Describe your feature request

Motivation

In many real-world scenarios, multiple objects stored in Mooncake Store actually represent different parts of a single logical entity.

Examples include:

In KV cache: the key tensor and value tensor are stored as separate objects.
In SGLang pipeline parallelism (PP): tensors for the same token's kvcache across different layers are stored as separate objects.

We refer to such related objects as belonging to the same group.

Due to limitations in upper-layer engines:

Objects in the same group may be uploaded (put) by different processes.
They may also be accessed (exists, get) by different processes.
Inside Mooncake Store, these objects may be evicted at different times.

This can lead to inconsistencies and correctness issues.

Desired Behavior

We would like to introduce group-level semantics so that objects within the same group share consistent visibility and lifecycle behavior:

Group-aware existence checks

When calling exists, users should not only know whether a single object exists, but also whether all objects in its group are present.
- Example: If only part of a token’s KV cache exists, the system may choose not to get it.
- In current SGLang PP implementation, if different ranks successfully get different tokens, it may lead to errors or inconsistencies.
Group-level eviction

All objects in the same group should be evicted together, rather than independently.

Relation to Prefix-aware Eviction

This feature is closely related to prefix-aware eviction.

If prefix-aware eviction is implemented, it should also operate at the group level. For example:

The key and value tensors of the same token correspond to the same node in a radix tree, and should therefore be treated as a single unit.

Proposed Design (Discussion)

Introducing a group abstraction into the current interface and architecture is non-trivial. Below are some initial thoughts and open questions for discussion:

1. Forward Compatibility

Existing APIs should continue to work unchanged.
The system should support cases where no group is specified.
- Possible approaches:
  - Treat each object as its own group by default.
  - Or treat objects as having no group unless explicitly specified.

2. API Changes

Since objects in the same group may be uploaded by different processes, we need a way to determine whether a group is complete.

We propose extending the put interface to include group metadata:

group_id: identifier of the group
group_size: total number of objects in the group
group_index: index of the current object within the group

Example (8-way pipeline parallelism):

Rank 0 uploads: (tokenA_0, group=tokenA, index=0/8)
Rank 1 uploads: (tokenA_1, group=tokenA, index=1/8)
...

This allows the system to determine when all objects in a group are present.

3. Design Considerations

a. Sharding Strategy

Should all objects in a group reside in the same shard?
- Pros: simpler metadata management
- Cons: potential contention when multiple RPCs access the same shard (lock contention)
If distributed across shards:
- Cross-shard coordination becomes complex and inefficient
- Group operations may require acquiring multiple shard locks → potential deadlocks

b. Metadata Design

How should group metadata be stored and managed efficiently?
How to track group completeness and lifecycle?

c. SSD Offloading Compatibility (Optional)

How should group semantics interact with SSD offloading mechanisms?

d. Compatibility with Existing remove API

How should remove behave with grouped objects?
- Remove individual object?
- Or remove the entire group?

e. Fault Tolerance

Group semantics introduce additional failure modes:

What if some objects in a group are never uploaded?
What if a put operation starts but never completes?
What if a client goes offline and only part of the group remains?

These cases require careful handling, similar to (but more complex than) current per-object fault tolerance.

How to Contribute

This feature is relatively complex and touches multiple parts of the system.

We would like to first align on the design and approach through discussion before moving to implementation.

Contributions are very welcome! In particular:

Feedback on API design
Suggestions for metadata and sharding strategies
Thoughts on fault tolerance and edge cases
Prior art or similar designs in other systems

Thanks in advance for your thoughts and discussion 🙌

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @ykwd on GitHub (Apr 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1887 ### Describe your feature request ### Motivation In many real-world scenarios, multiple objects stored in Mooncake Store actually represent different parts of a single logical entity. Examples include: * In KV cache: the **key tensor** and **value tensor** are stored as separate objects. * In **SGLang pipeline parallelism (PP)**: tensors for the same token's kvcache across different layers are stored as separate objects. We refer to such related objects as belonging to the same **group**. Due to limitations in upper-layer engines: * Objects in the same group may be uploaded (`put`) by **different processes**. * They may also be accessed (`exists`, `get`) by **different processes**. * Inside Mooncake Store, these objects may be **evicted at different times**. This can lead to inconsistencies and correctness issues. --- ### Desired Behavior We would like to introduce **group-level semantics** so that objects within the same group share consistent visibility and lifecycle behavior: 1. **Group-aware existence checks** When calling `exists`, users should not only know whether a single object exists, but also whether **all objects in its group are present**. * Example: If only part of a token’s KV cache exists, the system may choose not to `get` it. * In current SGLang PP implementation, if different ranks successfully `get` different tokens, it may lead to **errors or inconsistencies**. 2. **Group-level eviction** All objects in the same group should be **evicted together**, rather than independently. --- ### Relation to Prefix-aware Eviction This feature is closely related to **prefix-aware eviction**. If prefix-aware eviction is implemented, it should also operate at the **group level**. For example: * The key and value tensors of the same token correspond to the **same node in a radix tree**, and should therefore be treated as a single unit. --- ### Proposed Design (Discussion) Introducing a group abstraction into the current interface and architecture is non-trivial. Below are some initial thoughts and open questions for discussion: #### 1. Forward Compatibility * Existing APIs should continue to work unchanged. * The system should support cases where no group is specified. * Possible approaches: * Treat each object as its own group by default. * Or treat objects as having no group unless explicitly specified. #### 2. API Changes Since objects in the same group may be uploaded by different processes, we need a way to determine whether a group is complete. We propose extending the `put` interface to include group metadata: * `group_id`: identifier of the group * `group_size`: total number of objects in the group * `group_index`: index of the current object within the group Example (8-way pipeline parallelism): * Rank 0 uploads: `(tokenA_0, group=tokenA, index=0/8)` * Rank 1 uploads: `(tokenA_1, group=tokenA, index=1/8)` * ... This allows the system to determine when **all objects in a group are present**. --- #### 3. Design Considerations **a. Sharding Strategy** * Should all objects in a group reside in the same shard? * **Pros**: simpler metadata management * **Cons**: potential contention when multiple RPCs access the same shard (lock contention) * If distributed across shards: * Cross-shard coordination becomes **complex and inefficient** * Group operations may require acquiring multiple shard locks → potential **deadlocks** --- **b. Metadata Design** * How should group metadata be stored and managed efficiently? * How to track group completeness and lifecycle? --- **c. SSD Offloading Compatibility (Optional)** * How should group semantics interact with SSD offloading mechanisms? --- **d. Compatibility with Existing `remove` API** * How should `remove` behave with grouped objects? * Remove individual object? * Or remove the entire group? --- **e. Fault Tolerance** Group semantics introduce additional failure modes: * What if some objects in a group are **never uploaded**? * What if a `put` operation starts but never completes? * What if a client goes offline and only part of the group remains? These cases require careful handling, similar to (but more complex than) current per-object fault tolerance. --- ### How to Contribute This feature is relatively complex and touches multiple parts of the system. We would like to **first align on the design and approach** through discussion before moving to implementation. Contributions are very welcome! In particular: * Feedback on API design * Suggestions for metadata and sharding strategies * Thoughts on fault tolerance and edge cases * Prior art or similar designs in other systems --- Thanks in advance for your thoughts and discussion 🙌 ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:16:04 +00:00

closed this issue
added the
high priority
label

bsakaguchi commented

2026-10-29 21:16:05 +00:00

Author

Owner

@CAICAIIs commented on GitHub (Apr 24, 2026):

This is a very interesting design problem. I’d be happy to work on this issue.

From the discussion so far, my current understanding is that the main design fork is whether we want

only a group-aware completeness signal for exists, while get remains per-object, or
true all-or-nothing group visibility and lifecycle semantics

I think that decision affects a few follow-up questions

whether per-object metadata is enough, or if explicit group-level state is needed
how partial groups from crashed/offline writers should be handled, for example via timeout or GC
whether removing a single member should affect only that object or invalidate/remove the whole group
what semantics are expected for cross-shard groups

Curious which direction is preferred here, since it seems to define the boundary of the feature.

@CAICAIIs commented on GitHub (Apr 24, 2026): This is a very interesting design problem. I’d be happy to work on this issue. From the discussion so far, my current understanding is that the main design fork is whether we want - only a group-aware completeness signal for `exists`, while `get` remains per-object, or - true all-or-nothing group visibility and lifecycle semantics I think that decision affects a few follow-up questions - whether per-object metadata is enough, or if explicit group-level state is needed - how partial groups from crashed/offline writers should be handled, for example via timeout or GC - whether removing a single member should affect only that object or invalidate/remove the whole group - what semantics are expected for cross-shard groups Curious which direction is preferred here, since it seems to define the boundary of the feature.

bsakaguchi commented

2026-10-29 21:16:05 +00:00

Author

Owner

@huangtingwei9988 commented on GitHub (Apr 29, 2026):

A really nice design! once the functionality is implemented, I'll be using it in sglang right away.

@huangtingwei9988 commented on GitHub (Apr 29, 2026): A really nice design! once the functionality is implemented, I'll be using it in sglang right away.

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@Colors-111 commented on GitHub (Apr 29, 2026):

The group semantics described in this issue seem to naturally correspond to the RadixTreeNode design proposed in the Radix Tree RFC (#1998 ). The key insight is that the prefix hash already provides the group boundary:
Different model configurations use different key formats:

MLA models:  {"<hash>"}
MHA models:  {"<hash>_0_k", "<hash>_0_v", "<hash>_1_k", ...}
MHA+TP:      {"<hash>_0_k_tp_0", "<hash>_0_k_tp_1", ...}

But the first 64 characters are always a blake3 hex digest identifying a RadixAttention node.
This naturally partitions objects into groups (nodes) — each prefix_hash corresponds to one group, and all objects sharing that prefix are its members. The Radix Tree design already treats each node as a unit for both registration and eviction (leaf-level atomicity). The one gap is group_size: the current design doesn't store the expected number of keys per node, so we can't distinguish "partially uploaded" from "partially evicted." Adding an expected_key_count field to RadixTreeNode would close this gap — the inference framework provides the value since it knows the PP/TP topology.

@Colors-111 commented on GitHub (Apr 29, 2026): The group semantics described in this issue seem to naturally correspond to the RadixTreeNode design proposed in the Radix Tree RFC (#1998 ). The key insight is that the prefix hash already provides the group boundary: Different model configurations use different key formats: ``` MLA models: {"<hash>"} MHA models: {"<hash>_0_k", "<hash>_0_v", "<hash>_1_k", ...} MHA+TP: {"<hash>_0_k_tp_0", "<hash>_0_k_tp_1", ...} ``` But the first 64 characters are always a blake3 hex digest identifying a RadixAttention node. This naturally partitions objects into groups (nodes) — each prefix_hash corresponds to one group, and all objects sharing that prefix are its members. The Radix Tree design already treats each node as a unit for both registration and eviction (leaf-level atomicity). The one gap is group_size: the current design doesn't store the expected number of keys per node, so we can't distinguish "partially uploaded" from "partially evicted." Adding an expected_key_count field to RadixTreeNode would close this gap — the inference framework provides the value since it knows the PP/TP topology.

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@Srinivasoo7 commented on GitHub (May 1, 2026):

@Colors-111 that makes sense. After reading RFC #1732 and the newer Radix Tree RFC #1998, I think the group boundary for this issue may already be naturally defined by the RadixTreeNode.

My understanding is:

group_id can map to prefix_hash
group members can map to registered_keys under the RadixTreeNode
the first 64 characters of the key already identify the RadixAttention node across MLA, MHA, and MHA+TP key formats
leaf-level atomicity in the Radix Tree RFC already moves eviction toward group/node-level lifecycle semantics

So I believe the remaining unresolved piece is not “how do we define a group?” but “how do we know whether the group/node is complete?”

Currently RadixTreeNode has registered_keys, but it does not appear to store the expected number of keys for that node. Because of that, we cannot distinguish:

a partially uploaded group
a partially evicted group
a complete group with fewer keys because of model topology

Curious to understand would adding something like expected_key_count to RadixTreeNode be a reasonable minimal extension?

For example:

prefix_hash identifies the group/node
registered_keys tracks currently present keys
expected_key_count is provided by the inference framework because it knows the PP/TP/topology layout
node completeness becomes registered_keys.size() == expected_key_count
group-aware exists can return true only when the node is complete
eviction can continue to operate at leaf/node granularity

This seems to avoid introducing a separate generic group_id/group_size/group_index abstraction for the RadixAttention use case, while still solving the visibility/completeness issue described here

The remaining design question would be failure handling: if a writer crashes and the node never reaches expected_key_count, should the incomplete node stay invisible until TTL/GC removes it, or should it be explicitly marked incomplete/in-flight?

@Srinivasoo7 commented on GitHub (May 1, 2026): @Colors-111 that makes sense. After reading RFC #1732 and the newer Radix Tree RFC #1998, I think the group boundary for this issue may already be naturally defined by the RadixTreeNode. My understanding is: - group_id can map to prefix_hash - group members can map to registered_keys under the RadixTreeNode - the first 64 characters of the key already identify the RadixAttention node across MLA, MHA, and MHA+TP key formats - leaf-level atomicity in the Radix Tree RFC already moves eviction toward group/node-level lifecycle semantics So I believe the remaining unresolved piece is not “how do we define a group?” but “how do we know whether the group/node is complete?” Currently RadixTreeNode has registered_keys, but it does not appear to store the expected number of keys for that node. Because of that, we cannot distinguish: 1. a partially uploaded group 2. a partially evicted group 3. a complete group with fewer keys because of model topology Curious to understand would adding something like expected_key_count to RadixTreeNode be a reasonable minimal extension? For example: - prefix_hash identifies the group/node - registered_keys tracks currently present keys - expected_key_count is provided by the inference framework because it knows the PP/TP/topology layout - node completeness becomes registered_keys.size() == expected_key_count - group-aware exists can return true only when the node is complete - eviction can continue to operate at leaf/node granularity This seems to avoid introducing a separate generic group_id/group_size/group_index abstraction for the RadixAttention use case, while still solving the visibility/completeness issue described here The remaining design question would be failure handling: if a writer crashes and the node never reaches expected_key_count, should the incomplete node stay invisible until TTL/GC removes it, or should it be explicitly marked incomplete/in-flight?

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@CAICAIIs commented on GitHub (May 9, 2026):

I’d like to propose a phased implementation plan for this issue.

My current understanding is that the core problem is group completeness and group-level lifecycle consistency, rather than a full prefix/radix-tree implementation in the first PR.

Proposed PR breakdown

PR 1: Group metadata and group-aware exists

Add optional group metadata for objects:
- group_id
- expected_member_count
- optional group_index
Keep existing per-object APIs backward compatible.
Track group completeness in the master.
Add a group-aware exists API that can report:
- whether the queried object exists
- whether the whole group is complete
- expected/current member counts
- group state
Handle partial uploads and expired PutStart cleanup so incomplete groups do not become visible as complete.

PR 2: Group-level eviction

Update automatic eviction so grouped objects are treated as all-or-nothing candidates.
If any member is leased, hard-pinned, referenced, or otherwise not evictable, skip the whole group.
Preserve existing eviction behavior for ungrouped objects.

PR 3: Lifecycle and integration follow-ups

Define and implement explicit group removal semantics if needed.
Decide how group semantics interact with SSD offloading and HA snapshot/restore.
Document how SGLang/RadixAttention can map prefix_hash to group_id, with the inference framework providing expected_member_count.

This keeps the first PR focused on the minimal correctness primitive: group completeness. It also avoids depending on a full Radix Tree design before that direction is finalized.

Would it be okay for me to work on this issue?

@CAICAIIs commented on GitHub (May 9, 2026): I’d like to propose a phased implementation plan for this issue. My current understanding is that the core problem is group completeness and group-level lifecycle consistency, rather than a full prefix/radix-tree implementation in the first PR. ### Proposed PR breakdown #### PR 1: Group metadata and group-aware exists - Add optional group metadata for objects: - `group_id` - `expected_member_count` - optional `group_index` - Keep existing per-object APIs backward compatible. - Track group completeness in the master. - Add a group-aware exists API that can report: - whether the queried object exists - whether the whole group is complete - expected/current member counts - group state - Handle partial uploads and expired `PutStart` cleanup so incomplete groups do not become visible as complete. #### PR 2: Group-level eviction - Update automatic eviction so grouped objects are treated as all-or-nothing candidates. - If any member is leased, hard-pinned, referenced, or otherwise not evictable, skip the whole group. - Preserve existing eviction behavior for ungrouped objects. #### PR 3: Lifecycle and integration follow-ups - Define and implement explicit group removal semantics if needed. - Decide how group semantics interact with SSD offloading and HA snapshot/restore. - Document how SGLang/RadixAttention can map `prefix_hash` to `group_id`, with the inference framework providing `expected_member_count`. This keeps the first PR focused on the minimal correctness primitive: group completeness. It also avoids depending on a full Radix Tree design before that direction is finalized. Would it be okay for me to work on this issue?

bsakaguchi commented

2026-10-29 21:16:06 +00:00

Author

Owner

@stmatengss commented on GitHub (May 9, 2026):

@zxpdemonio PTAL.

@stmatengss commented on GitHub (May 9, 2026): @zxpdemonio PTAL.

bsakaguchi commented

2026-10-29 21:16:07 +00:00

Author

Owner

@ykwd commented on GitHub (May 19, 2026):

@CAICAIIs Sorry for the late reply. I think the solution you proposed is very detailed, and the various issues were considered quite thoroughly.

However, @yokinoshitayoki later ran some benchmarks and found that, when the global memory pool becomes large enough, the probability that only part of the objects in a group are evicted while others remain is actually quite low. At the same time, considering that network communication in distributed systems can always fail, guaranteeing that all objects in a group can always be fetched together would require substantial changes — including major modifications on the SGLang/vLLM side as well.

Taking everything into account, we proposed a best-effort strategy instead, aiming to support group semantics with minimal changes: https://github.com/kvcache-ai/Mooncake/issues/2127

I think we can collaborate with @yokinoshitayoki on this feature.

@ykwd commented on GitHub (May 19, 2026): @CAICAIIs Sorry for the late reply. I think the solution you proposed is very detailed, and the various issues were considered quite thoroughly. However, @yokinoshitayoki later ran some benchmarks and found that, when the global memory pool becomes large enough, the probability that only part of the objects in a group are evicted while others remain is actually quite low. At the same time, considering that network communication in distributed systems can always fail, guaranteeing that all objects in a group can always be fetched together would require substantial changes — including major modifications on the SGLang/vLLM side as well. Taking everything into account, we proposed a best-effort strategy instead, aiming to support group semantics with minimal changes: [https://github.com/kvcache-ai/Mooncake/issues/2127](https://github.com/kvcache-ai/Mooncake/issues/2127) I think we can collaborate with @yokinoshitayoki on this feature.