kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2706] [RFC]: Consolidate Master Metadata Routing #707

New issue

Closed

opened 2026-10-29 21:18:29 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:18:29 +00:00

Owner

Originally created by @Aionw on GitHub (Jul 1, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2706

Changes proposed

RFC: Consolidate Master Metadata Routing

Summary

MasterService should keep the current tenant-isolated metadata layout:

metadata_shards_[shard].tenants[tenant_id].metadata[user_key]

This preserves the strong tenant boundary: each tenant owns an independent TenantState, and duplicate user_key values across tenants are naturally isolated.

The proposed change is not to flatten tenant storage. Instead, it is to consolidate routing and per-object access behind a canonical object identity:

ObjectIdentity{tenant_id, user_key}
  -> MetadataRoute{shard, tenant_state, object_state}
  -> ObjectState{metadata, processing state, tasks, group state}

The goal is to get most of the simplicity of a flat metadata model without weakening tenant isolation. Object operations should use one routing/accessor path and then read or mutate object-scoped state from one colocated object entry.

Motivation

The current lookup path is tenant_id + user_key -> shard -> TenantState -> metadata[user_key]. This tenant layer is useful and should remain, but object-scoped state is spread across several tenant-local maps such as metadata, processing_keys, replication_tasks, offloading_tasks, promotion_tasks, and group_members.

That means a single object operation can repeat the same user_key lookup across multiple maps after it has already found the shard and tenant. It also makes routing logic easy to duplicate across call sites.

The master already has ObjectIdentity and MetadataAccessorRO/RW. This RFC proposes making that route/accessor layer the single way to reach object metadata and related state, instead of open-coding the route at individual call sites.

Goals

Preserve MetadataShard::tenants[tenant_id] as the isolation boundary.
Use ObjectIdentity{tenant_id, user_key} as the canonical input to routing and accessors.
Consolidate shard selection, tenant lookup, tenant creation, and per-object lookup in one route/accessor path.
Colocate object-scoped state inside a per-object ObjectState so common operations avoid several tenant-local map lookups.
Keep public APIs compatible: callers may continue passing tenant_id and user_key separately.
Preserve single-tenant mode by normalizing requests to the default tenant when multi-tenant mode is disabled.
Keep snapshot compatibility with the existing tenant-nested shape.

Non-Goals

Do not flatten MetadataShard::tenants.
Do not remove tenant semantics from Mooncake Store.
Do not require globally unique user keys.
Do not redesign tenant quota policy storage.
Do not introduce auth or access control.

Proposed Model

Keep the shard and tenant structure:

struct MetadataShard {
    mutable SharedMutex mutex;
    std::unordered_map<std::string, TenantState> tenants GUARDED_BY(mutex);
    long disk_object_count GUARDED_BY(mutex) = 0;
};

Use ObjectIdentity as the canonical routing key:

struct ObjectIdentity {
    std::string tenant_id;
    std::string user_key;
};

Normalize tenant IDs at the API boundary:

ObjectIdentity MasterService::MakeObjectIdentityForRequest(
    const std::string& user_key,
    const std::string& tenant_id) const {
    return {NormalizeRequestTenantId(tenant_id), user_key};
}

Inside each TenantState, aggregate per-object state by user_key:

struct ObjectState {
    std::optional<ObjectMetadata> metadata;
    bool processing = false;
    std::optional<ReplicationTask> replication_task;
    std::optional<OffloadingTask> offloading_task;
    std::optional<PromotionTask> promotion_task;
    std::optional<std::string> group_id;
};

struct TenantState {
    std::unordered_map<std::string, ObjectState> objects;
    std::unordered_map<std::string, std::unordered_set<std::string>>
        group_members;  // group_id -> set of user keys in this tenant
};

This keeps tenant isolation explicit while avoiding parallel lookups in metadata, processing_keys, replication_tasks, offloading_tasks, and promotion_tasks.

Routing and Access

Object operations should use one route:

request tenant_id + user_key
  -> NormalizeRequestTenantId()
  -> ObjectIdentity{tenant_id, user_key}
  -> getMetadataShardIndex(object_id)
  -> shard.tenants[object_id.tenant_id]
  -> tenant_state.objects[object_id.user_key]
  -> ObjectState

MetadataAccessorRO/RW should own this route. Call sites should not manually repeat shard lookup, tenant lookup, and object-state lookup unless they are doing bulk iteration.

For compatibility, the default tenant may keep the current hash(user_key) routing. Other tenants should combine tenant and key when choosing the shard.

Tenant-Aware Behavior

Tenant-aware subsystems keep using tenant ID explicitly:

Quota remains keyed by tenant ID. Create/delete paths take the tenant from object_id.tenant_id.
Metrics continue aggregating by tenant.
Tenant admin operations can still find and remove an entire TenantState per shard.
Snapshot/restore can keep the existing tenant-nested shape.

Migration Plan

Keep MetadataShard::tenants and TenantState as the storage boundary.
Introduce ObjectState inside TenantState.
Update MetadataAccessorRO/RW to route from ObjectIdentity to one ObjectState.
Move metadata, processing_keys, replication_tasks, offloading_tasks, and promotion_tasks into ObjectState fields.
Keep group_members tenant-local, but store the per-object group link in ObjectState::group_id.
Update iteration paths: quota reconciliation, eviction, snapshot, admin remove/list, and cache accounting.
Remove the old parallel tenant-local maps after all call sites use ObjectState.

Testing Plan

Same user_key under two tenants maps to two isolated TenantState entries.
Single-tenant mode still normalizes to default.
Object create/delete quota accounting remains correct.
Tenant remove/list operations still work without scanning a global object map.
Snapshot/restore preserves tenant-nested metadata.
Group IDs are isolated per tenant.
Replication/offloading/promotion state is found through the same ObjectState as metadata.
Eviction and disk-object accounting remain correct after object deletion.

Risks

A partial migration could create a hybrid model where metadata is in ObjectState but tasks still live in parallel maps, losing the lookup reduction.
ObjectState must not blur tenant boundaries; it should always live under one TenantState.
Existing snapshot code may need a compatibility layer if TenantState serialization changes from parallel maps to objects[user_key] -> ObjectState.

Conclusion

Keep the tenant-nested metadata layout for strong isolation, but make ObjectIdentity and MetadataAccessorRO/RW the only normal route into object state. Within each TenantState, consolidate per-object metadata, processing flags, and tasks into ObjectState so common object operations get a flat-access experience without giving up tenant isolation.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Aionw on GitHub (Jul 1, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2706 ### Changes proposed # RFC: Consolidate Master Metadata Routing ## Summary `MasterService` should keep the current tenant-isolated metadata layout: ```text metadata_shards_[shard].tenants[tenant_id].metadata[user_key] ``` This preserves the strong tenant boundary: each tenant owns an independent `TenantState`, and duplicate `user_key` values across tenants are naturally isolated. The proposed change is not to flatten tenant storage. Instead, it is to consolidate routing and per-object access behind a canonical object identity: ```text ObjectIdentity{tenant_id, user_key} -> MetadataRoute{shard, tenant_state, object_state} -> ObjectState{metadata, processing state, tasks, group state} ``` The goal is to get most of the simplicity of a flat metadata model without weakening tenant isolation. Object operations should use one routing/accessor path and then read or mutate object-scoped state from one colocated object entry. ## Motivation The current lookup path is `tenant_id + user_key -> shard -> TenantState -> metadata[user_key]`. This tenant layer is useful and should remain, but object-scoped state is spread across several tenant-local maps such as `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, `promotion_tasks`, and `group_members`. That means a single object operation can repeat the same `user_key` lookup across multiple maps after it has already found the shard and tenant. It also makes routing logic easy to duplicate across call sites. The master already has `ObjectIdentity` and `MetadataAccessorRO/RW`. This RFC proposes making that route/accessor layer the single way to reach object metadata and related state, instead of open-coding the route at individual call sites. ## Goals - Preserve `MetadataShard::tenants[tenant_id]` as the isolation boundary. - Use `ObjectIdentity{tenant_id, user_key}` as the canonical input to routing and accessors. - Consolidate shard selection, tenant lookup, tenant creation, and per-object lookup in one route/accessor path. - Colocate object-scoped state inside a per-object `ObjectState` so common operations avoid several tenant-local map lookups. - Keep public APIs compatible: callers may continue passing `tenant_id` and `user_key` separately. - Preserve single-tenant mode by normalizing requests to the `default` tenant when multi-tenant mode is disabled. - Keep snapshot compatibility with the existing tenant-nested shape. ## Non-Goals - Do not flatten `MetadataShard::tenants`. - Do not remove tenant semantics from Mooncake Store. - Do not require globally unique user keys. - Do not redesign tenant quota policy storage. - Do not introduce auth or access control. ## Proposed Model Keep the shard and tenant structure: ```cpp struct MetadataShard { mutable SharedMutex mutex; std::unordered_map<std::string, TenantState> tenants GUARDED_BY(mutex); long disk_object_count GUARDED_BY(mutex) = 0; }; ``` Use `ObjectIdentity` as the canonical routing key: ```cpp struct ObjectIdentity { std::string tenant_id; std::string user_key; }; ``` Normalize tenant IDs at the API boundary: ```cpp ObjectIdentity MasterService::MakeObjectIdentityForRequest( const std::string& user_key, const std::string& tenant_id) const { return {NormalizeRequestTenantId(tenant_id), user_key}; } ``` Inside each `TenantState`, aggregate per-object state by `user_key`: ```cpp struct ObjectState { std::optional<ObjectMetadata> metadata; bool processing = false; std::optional<ReplicationTask> replication_task; std::optional<OffloadingTask> offloading_task; std::optional<PromotionTask> promotion_task; std::optional<std::string> group_id; }; struct TenantState { std::unordered_map<std::string, ObjectState> objects; std::unordered_map<std::string, std::unordered_set<std::string>> group_members; // group_id -> set of user keys in this tenant }; ``` This keeps tenant isolation explicit while avoiding parallel lookups in `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, and `promotion_tasks`. ## Routing and Access Object operations should use one route: ```text request tenant_id + user_key -> NormalizeRequestTenantId() -> ObjectIdentity{tenant_id, user_key} -> getMetadataShardIndex(object_id) -> shard.tenants[object_id.tenant_id] -> tenant_state.objects[object_id.user_key] -> ObjectState ``` `MetadataAccessorRO/RW` should own this route. Call sites should not manually repeat shard lookup, tenant lookup, and object-state lookup unless they are doing bulk iteration. For compatibility, the `default` tenant may keep the current `hash(user_key)` routing. Other tenants should combine tenant and key when choosing the shard. ## Tenant-Aware Behavior Tenant-aware subsystems keep using tenant ID explicitly: - Quota remains keyed by tenant ID. Create/delete paths take the tenant from `object_id.tenant_id`. - Metrics continue aggregating by tenant. - Tenant admin operations can still find and remove an entire `TenantState` per shard. - Snapshot/restore can keep the existing tenant-nested shape. ## Migration Plan 1. Keep `MetadataShard::tenants` and `TenantState` as the storage boundary. 2. Introduce `ObjectState` inside `TenantState`. 3. Update `MetadataAccessorRO/RW` to route from `ObjectIdentity` to one `ObjectState`. 4. Move `metadata`, `processing_keys`, `replication_tasks`, `offloading_tasks`, and `promotion_tasks` into `ObjectState` fields. 5. Keep `group_members` tenant-local, but store the per-object group link in `ObjectState::group_id`. 6. Update iteration paths: quota reconciliation, eviction, snapshot, admin remove/list, and cache accounting. 7. Remove the old parallel tenant-local maps after all call sites use `ObjectState`. ## Testing Plan - Same `user_key` under two tenants maps to two isolated `TenantState` entries. - Single-tenant mode still normalizes to `default`. - Object create/delete quota accounting remains correct. - Tenant remove/list operations still work without scanning a global object map. - Snapshot/restore preserves tenant-nested metadata. - Group IDs are isolated per tenant. - Replication/offloading/promotion state is found through the same `ObjectState` as metadata. - Eviction and disk-object accounting remain correct after object deletion. ## Risks - A partial migration could create a hybrid model where metadata is in `ObjectState` but tasks still live in parallel maps, losing the lookup reduction. - `ObjectState` must not blur tenant boundaries; it should always live under one `TenantState`. - Existing snapshot code may need a compatibility layer if `TenantState` serialization changes from parallel maps to `objects[user_key] -> ObjectState`. ## Conclusion Keep the tenant-nested metadata layout for strong isolation, but make `ObjectIdentity` and `MetadataAccessorRO/RW` the only normal route into object state. Within each `TenantState`, consolidate per-object metadata, processing flags, and tasks into `ObjectState` so common object operations get a flat-access experience without giving up tenant isolation. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:18:29 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@Aionw commented on GitHub (Jul 1, 2026):

Would you mind taking a look when you have time @Lin-z-w

@Aionw commented on GitHub (Jul 1, 2026): Would you mind taking a look when you have time @Lin-z-w

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 1, 2026):

Thanks for opening this issue, @Aionw!

Field	Value
Issue	#2706
GitHub user ID	`41376987`
Reporter	@Aionw

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 1, 2026): Thanks for opening this issue, @Aionw! | Field | Value | |-------|-------| | **Issue** | #2706 | | **GitHub user ID** | `41376987` | | **Reporter** | @Aionw | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 1, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 1, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 8, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:08 +00:00

[PR #707] [MERGED] docs: add troubleshooting steps for QP allocation error #1763