[GH-ISSUE #2979] [RFC]: [Store] Entity Ownership Decoupling, SegmentView, and Recoverable Lifecycle #770

New issue

Closed

opened 2026-10-29 21:19:19 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented 2026-10-29 21:19:19 +00:00

Owner

Copy link

Originally created by @Icedcoco on GitHub (Jul 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2979

Changes proposed

Summary

This RFC proposes a first-version redesign of Mooncake Store Master metadata ownership and resource lifecycle. The proposal separates logical object metadata, replica state, allocation handles, and client/segment liveness while keeping the implementation deliberately small:

• Object owns its Replica instances.
• Replica owns one AllocatedBuffer.
• AllocatedBuffer points to its containing Segment.
• Segment owns the allocator and publishes its lifecycle through an independent SegmentView.
• Raw pointers express ownership or observation; no global Registry and no shared_ptr ownership graph are introduced.
• Object has no lifecycle state. When its replica collection becomes empty, the object is removed from the object map and then destroyed.
• Replica changes are represented by a small set of primitive events and replayed in order. Composite operations such as Move are not new oplog operations.

The design is intended to complement RFC #2953 rather than replace its client liveness semantics. RFC #2953 supplies the client-level recovery window; this RFC defines how that signal affects Segments, Replicas, allocation handles, metadata visibility, and HA promotion.

Motivation

The current Master implementation keeps resource ownership and lifecycle information spread across Object, Replica, Segment, allocator, task, and HA paths. This creates several practical risks:

1. A Replica can retain information that is owned by another entity, making deletion and recovery ordering difficult to reason about.
2. Asynchronous callbacks may outlive the structure they reference if they capture entity pointers directly.
3. Segment state, connection information, and allocator state can be restored from different persistence paths and therefore disagree.
4. Composite operations write business-specific oplog entries in several places, making ordering and replay behavior difficult to prove.
5. Promotion currently needs an allocator even when the complete Segment description is not available yet.

The first version should solve these problems without introducing a general object registry, a new Replica state machine framework, or a new generation protocol for Objects.

Goals

1. Make ownership and pointer lifetime explicit for Object, Replica, AllocatedBuffer, Segment, and allocator.
2. Keep Replica identifiers stable enough for replay and asynchronous lookup, while allowing callers to use ObjectKey and ReplicaId without retaining a Replica*.
3. Define a fixed Segment lifecycle and its event transitions.
4. Centralize primitive Replica mutations and their oplog mapping.
5. Make removal safe: an AllocatedBuffer must be released before its Replica is removed from Object.
6. Make SegmentView the only persistent authority for Segment information.
7. Define a deterministic standby/promotion order that works before a real allocator is available.
8. Preserve the existing behavior where possible and keep the first version reviewable.

Non-goals

• No global Registry for Replica ownership or lookup.
• No shared_ptr ownership graph for core entities.
• No formal Replica state-machine framework in the first version.
• No Object generation field.
• No SegmentId reuse. New mounts receive a new client-generated UUID.
• No periodic forced deletion of PROCESSING Replicas. They complete naturally; cleanup is triggered by mutation callbacks or explicit removal paths.
• No change to the RPC schema in this RFC.
• No claim that a non-transactional backend can provide the same HA ordering guarantees.

Entity model

Object
  owns vector<Replica*>

Replica
  owns AllocatedBuffer*

AllocatedBuffer
  observes Segment*

Segment
  owns allocator

Object

Object retains ObjectKey, logical size, data type, tenant/group metadata, lease/pin/quota/accounting information, a per-object monotonic ReplicaId counter, and vector<Replica*>.

Object does not retain client id or Segment id. It has no independent lifecycle state. A Replica is removed from the vector only after its buffer has been released. Once the vector is empty, the Object is first erased from the object map and then destroyed.

group_id is immutable while the Object exists, matching the current code's validation. size may change during destructive Upsert; cleanup and quota accounting must use the sizes recorded in each AllocatedBuffer rather than the current Object size.

Replica

Each Replica retains its ObjectKey and a 32-bit ReplicaId. IDs are allocated monotonically per Object and are not reused while the Object exists. Copy and Move create a new target Replica with a new ID; the source ID does not change.

The first version keeps the current status values:

UNDEFINED, INITIALIZED, PROCESSING, COMPLETE, REMOVED, FAILED

It does not add a generic Replica state-machine abstraction. Replica methods should own common getters, descriptor access, read/write eligibility checks, reference-count changes, and the small set of legal status changes. They do not write HA storage directly.

AllocatedBuffer

AllocatedBuffer is a logical allocation handle, not ownership of the whole client memory region. It stores its containing Segment pointer, requested and reserved sizes, and the medium-specific allocation descriptor (offset, address, file locator, or equivalent).

Its destruction synchronously returns the allocation to the local allocator and decrements Segment.active_allocations. A separate released_ flag is not required. Local release is expected to be infallible; an error is an invariant failure rather than an ordinary business result.

Segment and SegmentView

Segment owns the allocator and runtime connection object. Its frequently read lifecycle state is an atomic value. The fixed states are:

INIT
ACTIVE
DRAINING
GRACEFUL_UNMOUNTING
SUSPECTED
UNMOUNTING
UNMOUNTED

DRAINED is intentionally not a state; a drained Segment proceeds to UNMOUNTING.

All durable Segment information is stored in an independent SegmentView store. SegmentView is not part of Object/Replica oplog or snapshot state and is the sole authority for Segment identity, medium, endpoint/connection information, lifecycle state, deadlines, reason, and connection generation.

The primary writes SegmentView first and publishes the new local atomic state only after the backend write succeeds. A failed write leaves the old local state in place and is retried. Ordinary GET/PUT/Copy/Move continue using the previous state; they do not receive an artificial retry result merely because the state publication backend is temporarily unavailable.

Segment lifecycle

The normal transitions are:

INIT --registration/heartbeat--> ACTIVE
ACTIVE --active drain request--> DRAINING
DRAINING --drain complete--> UNMOUNTING
ACTIVE --graceful unmount--> GRACEFUL_UNMOUNTING
GRACEFUL_UNMOUNTING --deadline--> UNMOUNTING
GRACEFUL_UNMOUNTING --temporary heartbeat loss--> SUSPECTED
INIT --recovered graceful registration--> GRACEFUL_UNMOUNTING
INIT --registration timeout--> SUSPECTED
SUSPECTED --heartbeat recovery--> ACTIVE or GRACEFUL_UNMOUNTING
SUSPECTED --long disconnect--> UNMOUNTING
UNMOUNTING --active_allocations == 0 and cleanup complete--> UNMOUNTED

GRACEFUL_UNMOUNTING never silently returns to ACTIVE. Reconnection keeps the Segment in graceful unmounting; cancelling that process requires an explicit future management event.

SUSPECTED rejects new reads, writes, allocation, Copy, Move, and task dispatch. Already-running tasks may finish and report their result so that Replica status, references, and allocation accounting converge.

UNMOUNTING restores existing Object/Replica/task metadata but rejects new work. PROCESSING Replicas are allowed to return normally. The first version does not periodically cancel them or force cleanup after a timeout.

UNMOUNTED means the allocator has been destroyed, every AllocatedBuffer has been released, active_allocations == 0, and the final SegmentView state has been persisted. A future mount uses a new Segment UUID and is unrelated to the old Segment.

Primitive Replica events and oplog

The system uses a finite set of primitive events:

ALLOCATE
WRITE_SUCCEEDED
WRITE_RETRY
WRITE_FAILED
WRITE_FAILED_REMOVE
REMOVE
REMOVE_DURABLE
FINALIZE
INC_REF
DEC_REF

The durable oplog mapping is:

ALLOCATE success       -> REPLICA_CREATE
WRITE_SUCCEEDED        -> REPLICA_COMPLETE
WRITE_FAILED          -> REPLICA_FAILED
WRITE_FAILED_REMOVE   -> REPLICA_REMOVE
REMOVE                -> REPLICA_REMOVE

REMOVE_DURABLE, FINALIZE, and reference-count changes are local cleanup steps and do not create another logical oplog entry.

Object/Replica methods validate and mutate in-memory state. A shared Master mutation helper owns HA/non-HA persistence, ordered submission, retry policy, durable callbacks, and finalization. This avoids scattering oplog writes through Put, Copy, Move, Evict, and cleanup implementations.

Composite operations are expressed using primitive events:

• Put: allocate, write, complete.
• Copy: allocate and complete a new Replica, then adjust references.
• Move: complete the target Replica, then remove the source Replica.
• Evict: remove the selected Replica.

Move must submit REPLICA_COMPLETE(target) before REPLICA_REMOVE(source) to the same ordered writer. It need not wait for the first durable callback. The ordered writer guarantees that a durable remove implies that the preceding complete was accepted.

Upsert is intentionally destructive in the first version. Regardless of size, old Replicas are marked removed and their remove events are submitted first; only then is the Object's logical size updated and a new processing Replica created. This permits a short interval with no readable Replica and accepts the corresponding data-loss risk on failure. The old Object structure is not deleted before old Replica cleanup completes.

HA recovery and promotion

SegmentView is a strict recovery prerequisite. Standby/promotion must not recover Object/Replica resources from a missing or incomplete SegmentView. When SegmentView cannot be read, the standby retries with backoff. If the old Master lease is lost during this period, the partial recovery is discarded and the process returns to standby instead of entering serving.

The promotion sequence is:

1. Replay Object/Replica oplog into memory.
2. Use DummyAllocator while real Segment allocators are unavailable. It stores logical allocation records in original allocation order.
3. Acquire leader fencing.
4. Read a complete consistent SegmentView set.
5. Create real Segments and allocators.
6. Pause subsequent oplog catch-up.
7. Traverse all DummyAllocator records and perform real allocation in order.
8. Resume oplog catch-up only after the existing allocation rebuild completes.
9. Catch up to the current oplog head and validate pointer chains and active_allocations.
10. Enter serving only after both allocation rebuild and oplog catch-up have completed.

Allocation conflicts, capacity mismatches, or descriptor mismatches are treated as program/link errors in this first version. The affected Replica and the conflicting Replica are marked for removal and cleaned. The design relies on recorded SegmentId, ReplicaId, requested/reserved size, descriptor, and ordered replay to make such conflicts unreachable in normal operation.

The long-term direction is hot-standby promotion. Snapshot allocator internals are compatibility data only and should eventually be removed from snapshots.

Concurrency and pointer safety

Version one keeps metadata shard locks for Object/Replica lifecycle, status, reference counts, vector mutation, and object accounting. SegmentManager's lock protects Segment and allocator lifetime. The lock order is:

metadata shard -> SegmentManager -> allocator

Async tasks do not retain entity pointers. They carry ObjectKey, ReplicaId, SegmentId, and task-specific descriptors, then re-lookup the entity under the appropriate lock. Replica* is therefore not an asynchronous ownership mechanism.

Relationship to RFC #2953

RFC #2953 proposes a useful client-level Active -> Suspected -> Offline recovery window and a common serving-eligibility decision. This RFC keeps that concept, but places the resource lifecycle boundary at Segment and Replica:

Concern	RFC #2953 emphasis	This RFC
Recovery identity	Client incarnation	Segment UUID plus client liveness
Temporary failure	Client Suspected	Segment Suspected, propagated to admission checks
Terminal cleanup	ClientOffboardingJob	Segment Unmounting plus Replica/buffer convergence
Routing decision	ServingEligibility seam	Entity predicates backed by Segment state and Replica status
Persistence	Snapshot/oplog compatibility	Independent SegmentView; primitive Replica oplog
Async safety	Job object	ID-based task relookup and owner-first destruction

The two designs can coexist: a client liveness record can drive Segment state transitions, while SegmentView and Replica events define the exact resource cleanup and replay behavior.

Rollout

1. Introduce entity ownership and ID-based asynchronous lookup without changing external APIs.
2. Centralize Replica mutation and oplog mapping.
3. Add Segment atomic lifecycle and SegmentView persistence.
4. Add DummyAllocator-based promotion ordering and remove dependence on snapshot allocator internals.
5. Migrate old duplicate Segment persistence fields and retain compatibility readers only for the transition period.

Open questions

These are implementation details, not blockers for the architecture:

1. Should SegmentView use an independent HA backend interface or a dedicated keyspace in the existing backend?
2. Is a cross-key version barrier required between SegmentView and Replica oplog commits, or is ordered replay sufficient for the first version?
3. Should standby watch SegmentView continuously or read it only during promotion/recovery?
4. Which compatibility release can remove allocator internals from snapshots?

Any feedback is appreciated!

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Icedcoco on GitHub (Jul 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2979 ### Changes proposed ## Summary This RFC proposes a first-version redesign of Mooncake Store Master metadata ownership and resource lifecycle. The proposal separates logical object metadata, replica state, allocation handles, and client/segment liveness while keeping the implementation deliberately small: - `Object` owns its `Replica` instances. - `Replica` owns one `AllocatedBuffer`. - `AllocatedBuffer` points to its containing `Segment`. - `Segment` owns the allocator and publishes its lifecycle through an independent `SegmentView`. - Raw pointers express ownership or observation; no global Registry and no `shared_ptr` ownership graph are introduced. - `Object` has no lifecycle state. When its replica collection becomes empty, the object is removed from the object map and then destroyed. - Replica changes are represented by a small set of primitive events and replayed in order. Composite operations such as Move are not new oplog operations. The design is intended to complement RFC #2953 rather than replace its client liveness semantics. RFC #2953 supplies the client-level recovery window; this RFC defines how that signal affects Segments, Replicas, allocation handles, metadata visibility, and HA promotion. ## Motivation The current Master implementation keeps resource ownership and lifecycle information spread across Object, Replica, Segment, allocator, task, and HA paths. This creates several practical risks: 1. A Replica can retain information that is owned by another entity, making deletion and recovery ordering difficult to reason about. 2. Asynchronous callbacks may outlive the structure they reference if they capture entity pointers directly. 3. Segment state, connection information, and allocator state can be restored from different persistence paths and therefore disagree. 4. Composite operations write business-specific oplog entries in several places, making ordering and replay behavior difficult to prove. 5. Promotion currently needs an allocator even when the complete Segment description is not available yet. The first version should solve these problems without introducing a general object registry, a new Replica state machine framework, or a new generation protocol for Objects. ## Goals 1. Make ownership and pointer lifetime explicit for Object, Replica, AllocatedBuffer, Segment, and allocator. 2. Keep Replica identifiers stable enough for replay and asynchronous lookup, while allowing callers to use ObjectKey and ReplicaId without retaining a `Replica*`. 3. Define a fixed Segment lifecycle and its event transitions. 4. Centralize primitive Replica mutations and their oplog mapping. 5. Make removal safe: an AllocatedBuffer must be released before its Replica is removed from Object. 6. Make SegmentView the only persistent authority for Segment information. 7. Define a deterministic standby/promotion order that works before a real allocator is available. 8. Preserve the existing behavior where possible and keep the first version reviewable. ## Non-goals - No global Registry for Replica ownership or lookup. - No `shared_ptr` ownership graph for core entities. - No formal Replica state-machine framework in the first version. - No Object generation field. - No SegmentId reuse. New mounts receive a new client-generated UUID. - No periodic forced deletion of PROCESSING Replicas. They complete naturally; cleanup is triggered by mutation callbacks or explicit removal paths. - No change to the RPC schema in this RFC. - No claim that a non-transactional backend can provide the same HA ordering guarantees. ## Entity model ```text Object owns vector<Replica*> Replica owns AllocatedBuffer* AllocatedBuffer observes Segment* Segment owns allocator ``` ### Object `Object` retains `ObjectKey`, logical size, data type, tenant/group metadata, lease/pin/quota/accounting information, a per-object monotonic ReplicaId counter, and `vector<Replica*>`. Object does not retain client id or Segment id. It has no independent lifecycle state. A Replica is removed from the vector only after its buffer has been released. Once the vector is empty, the Object is first erased from the object map and then destroyed. `group_id` is immutable while the Object exists, matching the current code's validation. `size` may change during destructive Upsert; cleanup and quota accounting must use the sizes recorded in each AllocatedBuffer rather than the current Object size. ### Replica Each Replica retains its ObjectKey and a 32-bit ReplicaId. IDs are allocated monotonically per Object and are not reused while the Object exists. Copy and Move create a new target Replica with a new ID; the source ID does not change. The first version keeps the current status values: ```text UNDEFINED, INITIALIZED, PROCESSING, COMPLETE, REMOVED, FAILED ``` It does not add a generic Replica state-machine abstraction. Replica methods should own common getters, descriptor access, read/write eligibility checks, reference-count changes, and the small set of legal status changes. They do not write HA storage directly. ### AllocatedBuffer `AllocatedBuffer` is a logical allocation handle, not ownership of the whole client memory region. It stores its containing Segment pointer, requested and reserved sizes, and the medium-specific allocation descriptor (offset, address, file locator, or equivalent). Its destruction synchronously returns the allocation to the local allocator and decrements `Segment.active_allocations`. A separate `released_` flag is not required. Local release is expected to be infallible; an error is an invariant failure rather than an ordinary business result. ### Segment and SegmentView `Segment` owns the allocator and runtime connection object. Its frequently read lifecycle state is an atomic value. The fixed states are: ```text INIT ACTIVE DRAINING GRACEFUL_UNMOUNTING SUSPECTED UNMOUNTING UNMOUNTED ``` `DRAINED` is intentionally not a state; a drained Segment proceeds to `UNMOUNTING`. All durable Segment information is stored in an independent SegmentView store. SegmentView is not part of Object/Replica oplog or snapshot state and is the sole authority for Segment identity, medium, endpoint/connection information, lifecycle state, deadlines, reason, and connection generation. The primary writes SegmentView first and publishes the new local atomic state only after the backend write succeeds. A failed write leaves the old local state in place and is retried. Ordinary GET/PUT/Copy/Move continue using the previous state; they do not receive an artificial retry result merely because the state publication backend is temporarily unavailable. ## Segment lifecycle The normal transitions are: ```text INIT --registration/heartbeat--> ACTIVE ACTIVE --active drain request--> DRAINING DRAINING --drain complete--> UNMOUNTING ACTIVE --graceful unmount--> GRACEFUL_UNMOUNTING GRACEFUL_UNMOUNTING --deadline--> UNMOUNTING GRACEFUL_UNMOUNTING --temporary heartbeat loss--> SUSPECTED INIT --recovered graceful registration--> GRACEFUL_UNMOUNTING INIT --registration timeout--> SUSPECTED SUSPECTED --heartbeat recovery--> ACTIVE or GRACEFUL_UNMOUNTING SUSPECTED --long disconnect--> UNMOUNTING UNMOUNTING --active_allocations == 0 and cleanup complete--> UNMOUNTED ``` `GRACEFUL_UNMOUNTING` never silently returns to `ACTIVE`. Reconnection keeps the Segment in graceful unmounting; cancelling that process requires an explicit future management event. `SUSPECTED` rejects new reads, writes, allocation, Copy, Move, and task dispatch. Already-running tasks may finish and report their result so that Replica status, references, and allocation accounting converge. `UNMOUNTING` restores existing Object/Replica/task metadata but rejects new work. PROCESSING Replicas are allowed to return normally. The first version does not periodically cancel them or force cleanup after a timeout. `UNMOUNTED` means the allocator has been destroyed, every AllocatedBuffer has been released, `active_allocations == 0`, and the final SegmentView state has been persisted. A future mount uses a new Segment UUID and is unrelated to the old Segment. ## Primitive Replica events and oplog The system uses a finite set of primitive events: ```text ALLOCATE WRITE_SUCCEEDED WRITE_RETRY WRITE_FAILED WRITE_FAILED_REMOVE REMOVE REMOVE_DURABLE FINALIZE INC_REF DEC_REF ``` The durable oplog mapping is: ```text ALLOCATE success -> REPLICA_CREATE WRITE_SUCCEEDED -> REPLICA_COMPLETE WRITE_FAILED -> REPLICA_FAILED WRITE_FAILED_REMOVE -> REPLICA_REMOVE REMOVE -> REPLICA_REMOVE ``` `REMOVE_DURABLE`, `FINALIZE`, and reference-count changes are local cleanup steps and do not create another logical oplog entry. Object/Replica methods validate and mutate in-memory state. A shared Master mutation helper owns HA/non-HA persistence, ordered submission, retry policy, durable callbacks, and finalization. This avoids scattering oplog writes through Put, Copy, Move, Evict, and cleanup implementations. Composite operations are expressed using primitive events: - Put: allocate, write, complete. - Copy: allocate and complete a new Replica, then adjust references. - Move: complete the target Replica, then remove the source Replica. - Evict: remove the selected Replica. Move must submit `REPLICA_COMPLETE(target)` before `REPLICA_REMOVE(source)` to the same ordered writer. It need not wait for the first durable callback. The ordered writer guarantees that a durable remove implies that the preceding complete was accepted. Upsert is intentionally destructive in the first version. Regardless of size, old Replicas are marked removed and their remove events are submitted first; only then is the Object's logical size updated and a new processing Replica created. This permits a short interval with no readable Replica and accepts the corresponding data-loss risk on failure. The old Object structure is not deleted before old Replica cleanup completes. ## HA recovery and promotion SegmentView is a strict recovery prerequisite. Standby/promotion must not recover Object/Replica resources from a missing or incomplete SegmentView. When SegmentView cannot be read, the standby retries with backoff. If the old Master lease is lost during this period, the partial recovery is discarded and the process returns to standby instead of entering serving. The promotion sequence is: 1. Replay Object/Replica oplog into memory. 2. Use DummyAllocator while real Segment allocators are unavailable. It stores logical allocation records in original allocation order. 3. Acquire leader fencing. 4. Read a complete consistent SegmentView set. 5. Create real Segments and allocators. 6. Pause subsequent oplog catch-up. 7. Traverse all DummyAllocator records and perform real allocation in order. 8. Resume oplog catch-up only after the existing allocation rebuild completes. 9. Catch up to the current oplog head and validate pointer chains and `active_allocations`. 10. Enter serving only after both allocation rebuild and oplog catch-up have completed. Allocation conflicts, capacity mismatches, or descriptor mismatches are treated as program/link errors in this first version. The affected Replica and the conflicting Replica are marked for removal and cleaned. The design relies on recorded SegmentId, ReplicaId, requested/reserved size, descriptor, and ordered replay to make such conflicts unreachable in normal operation. The long-term direction is hot-standby promotion. Snapshot allocator internals are compatibility data only and should eventually be removed from snapshots. ## Concurrency and pointer safety Version one keeps metadata shard locks for Object/Replica lifecycle, status, reference counts, vector mutation, and object accounting. SegmentManager's lock protects Segment and allocator lifetime. The lock order is: ```text metadata shard -> SegmentManager -> allocator ``` Async tasks do not retain entity pointers. They carry ObjectKey, ReplicaId, SegmentId, and task-specific descriptors, then re-lookup the entity under the appropriate lock. `Replica*` is therefore not an asynchronous ownership mechanism. ## Relationship to RFC #2953 RFC #2953 proposes a useful client-level `Active -> Suspected -> Offline` recovery window and a common serving-eligibility decision. This RFC keeps that concept, but places the resource lifecycle boundary at Segment and Replica: | Concern | RFC #2953 emphasis | This RFC | |---|---|---| | Recovery identity | Client incarnation | Segment UUID plus client liveness | | Temporary failure | Client Suspected | Segment Suspected, propagated to admission checks | | Terminal cleanup | ClientOffboardingJob | Segment Unmounting plus Replica/buffer convergence | | Routing decision | ServingEligibility seam | Entity predicates backed by Segment state and Replica status | | Persistence | Snapshot/oplog compatibility | Independent SegmentView; primitive Replica oplog | | Async safety | Job object | ID-based task relookup and owner-first destruction | The two designs can coexist: a client liveness record can drive Segment state transitions, while SegmentView and Replica events define the exact resource cleanup and replay behavior. ## Rollout 1. Introduce entity ownership and ID-based asynchronous lookup without changing external APIs. 2. Centralize Replica mutation and oplog mapping. 3. Add Segment atomic lifecycle and SegmentView persistence. 4. Add DummyAllocator-based promotion ordering and remove dependence on snapshot allocator internals. 5. Migrate old duplicate Segment persistence fields and retain compatibility readers only for the transition period. ## Open questions These are implementation details, not blockers for the architecture: 1. Should SegmentView use an independent HA backend interface or a dedicated keyspace in the existing backend? 2. Is a cross-key version barrier required between SegmentView and Replica oplog commits, or is ordered replay sufficient for the first version? 3. Should standby watch SegmentView continuously or read it only during promotion/recovery? 4. Which compatibility release can remove allocator internals from snapshots? Any feedback is appreciated! ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:19:19 +00:00

• closed this issue
• added the
  RFC
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:19:20 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 17, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#2979
GitHub user ID	102317026
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:5002438850 --> @github-actions[bot] commented on GitHub (Jul 17, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #2979 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:19:20 +00:00

Author

Owner

Copy link

@Aionw commented on GitHub (Jul 23, 2026):

Thanks for putting this RFC together. I like the overall direction, especially making entity lifetimes easier to reason about and avoiding long-lived entity pointers in asynchronous tasks. I have a few suggestions and questions.

1. Separate the two main concerns more clearly

The RFC currently covers both:

1. In-process ownership and lifecycle management for Segment, Object, Replica, AllocatedBuffer, and allocator.
2. HA-related persistence and recovery, including SegmentView, oplog replay, allocator reconstruction, and promotion.

I think the proposal would be easier to review if these were presented as two clearly separated parts, even if they remain in the same RFC.

The first part could focus on the entity ownership graph, destruction order, identity, locking, and in-memory mutation semantics. The second part could focus on persistence, replay, fencing, and recovery. The connection between them would then be the stable identities and primitive mutation events.

2. Use explicit ownership types for owning relationships

For relationships that represent ownership, I would prefer value semantics or std::unique_ptr rather than raw pointers. For example:

Object          -> unique_ptr<Replica>
Replica         -> unique_ptr<AllocatedBuffer>
SegmentManager  -> unique_ptr<Segment>
Segment         -> unique_ptr<BufferAllocatorBase>

I agree that a shared_ptr ownership graph may not be necessary. However, using unique_ptr for exclusive ownership would make the intended destruction order explicit and reduce the amount of lifetime reasoning required from callers.

For non-owning relationships, scoped references or ID-based lookup may be more appropriate. It would be helpful if the RFC explicitly distinguished owning relationships, temporary borrows, and asynchronous references.

3. Clarify the locking and performance model for ID-based lookup

Using IDs in asynchronous tasks is safer than retaining entity pointers, but I think the RFC should clarify the expected locking model and its performance implications.

The current Copy, Move, and Promotion completion paths already acquire a metadata shard lock and resolve Replicas by ID, so ID lookup itself may not add much overhead. The larger concern would be an implementation that acquires the lock separately for every primitive event.

Ideally, one task completion would:

acquire the metadata shard lock once
→ look up the Object once
→ resolve all referenced Replicas
→ apply the complete mutation batch
→ release the lock once

It would also be good to avoid requiring a global SegmentManager lookup for every Replica lookup. Some basic performance validation around lock contention, callback latency, and hot-key workloads would help confirm that the new abstraction does not introduce a regression.

4. Revisit the Segment and Replica state definitions

I am not yet fully convinced that the proposed state sets capture the right concepts.

For Segment, the current model appears to combine lifecycle, client or connection health, drain policy, and recovery readiness. For example, SUSPECTED may belong to client/connection health rather than Segment lifecycle. Otherwise, a Segment could conceptually be both DRAINING and SUSPECTED, but a single enum cannot represent that combination.

Similarly, GRACEFUL_UNMOUNTING may be a draining mode with a reason and deadline rather than a separate lifecycle state. On the other hand, removing DRAINED changes the current distinction between “drain completed” and “unmount started,” so it would be useful to explain whether this semantic change is intentional.

For Replica, some retained states such as INITIALIZED, REMOVED, and FAILED do not yet have clearly defined transition paths, while HA-related phases such as “write completed but not yet durable” or “removal requested but not yet durable” may need to be represented.

A transition table for both Segment and Replica would make this much easier to evaluate. It could describe, for each state and transition:

• whether the entity is visible or serviceable;
• whether it still owns resources;
• whether the change is durable;
• which transitions are retryable;
• how the state is reconstructed during recovery.

Overall, I agree with the direction of the RFC. Clarifying these boundaries would make the ownership and recovery model easier to reason about and implement incrementally.

<!-- gh-comment-id:5055510407 --> @Aionw commented on GitHub (Jul 23, 2026): Thanks for putting this RFC together. I like the overall direction, especially making entity lifetimes easier to reason about and avoiding long-lived entity pointers in asynchronous tasks. I have a few suggestions and questions. ### 1. Separate the two main concerns more clearly The RFC currently covers both: 1. In-process ownership and lifecycle management for Segment, Object, Replica, AllocatedBuffer, and allocator. 2. HA-related persistence and recovery, including SegmentView, oplog replay, allocator reconstruction, and promotion. I think the proposal would be easier to review if these were presented as two clearly separated parts, even if they remain in the same RFC. The first part could focus on the entity ownership graph, destruction order, identity, locking, and in-memory mutation semantics. The second part could focus on persistence, replay, fencing, and recovery. The connection between them would then be the stable identities and primitive mutation events. ### 2. Use explicit ownership types for owning relationships For relationships that represent ownership, I would prefer value semantics or `std::unique_ptr` rather than raw pointers. For example: ```text Object -> unique_ptr<Replica> Replica -> unique_ptr<AllocatedBuffer> SegmentManager -> unique_ptr<Segment> Segment -> unique_ptr<BufferAllocatorBase> ``` I agree that a `shared_ptr` ownership graph may not be necessary. However, using `unique_ptr` for exclusive ownership would make the intended destruction order explicit and reduce the amount of lifetime reasoning required from callers. For non-owning relationships, scoped references or ID-based lookup may be more appropriate. It would be helpful if the RFC explicitly distinguished owning relationships, temporary borrows, and asynchronous references. ### 3. Clarify the locking and performance model for ID-based lookup Using IDs in asynchronous tasks is safer than retaining entity pointers, but I think the RFC should clarify the expected locking model and its performance implications. The current Copy, Move, and Promotion completion paths already acquire a metadata shard lock and resolve Replicas by ID, so ID lookup itself may not add much overhead. The larger concern would be an implementation that acquires the lock separately for every primitive event. Ideally, one task completion would: ```text acquire the metadata shard lock once → look up the Object once → resolve all referenced Replicas → apply the complete mutation batch → release the lock once ``` It would also be good to avoid requiring a global SegmentManager lookup for every Replica lookup. Some basic performance validation around lock contention, callback latency, and hot-key workloads would help confirm that the new abstraction does not introduce a regression. ### 4. Revisit the Segment and Replica state definitions I am not yet fully convinced that the proposed state sets capture the right concepts. For Segment, the current model appears to combine lifecycle, client or connection health, drain policy, and recovery readiness. For example, `SUSPECTED` may belong to client/connection health rather than Segment lifecycle. Otherwise, a Segment could conceptually be both `DRAINING` and `SUSPECTED`, but a single enum cannot represent that combination. Similarly, `GRACEFUL_UNMOUNTING` may be a draining mode with a reason and deadline rather than a separate lifecycle state. On the other hand, removing `DRAINED` changes the current distinction between “drain completed” and “unmount started,” so it would be useful to explain whether this semantic change is intentional. For Replica, some retained states such as `INITIALIZED`, `REMOVED`, and `FAILED` do not yet have clearly defined transition paths, while HA-related phases such as “write completed but not yet durable” or “removal requested but not yet durable” may need to be represented. A transition table for both Segment and Replica would make this much easier to evaluate. It could describe, for each state and transition: - whether the entity is visible or serviceable; - whether it still owns resources; - whether the change is durable; - which transitions are retryable; - how the state is reconstructed during recovery. Overall, I agree with the direction of the RFC. Clarifying these boundaries would make the ownership and recovery model easier to reason about and implement incrementally.

bsakaguchi commented 2026-10-29 21:19:20 +00:00

Author

Owner

Copy link

@Icedcoco commented on GitHub (Jul 24, 2026):

Hi @Aionw . Thanks for the detailed comments. I agree with the main concerns. The RFC is still in planning, so I’ll first clarify the intended direction here before rewriting the RFC text.

1. Splitting ownership and HA concerns

I agree the current RFC mixes two related but different layers:

1. in-process ownership/lifetime management;
2. HA persistence, replay, fencing, promotion, and recovery.

I plan to keep them in the same RFC, but split the document into two clearly separated sections later. The first section will focus on Object/Replica/AllocatedBuffer/SegmentRuntime ownership, destruction order, borrow rules, lock scope, and async identity lookup. The second section will focus on lifecycle events, oplog replay, durable callbacks, Segment persistence, rollout, and rollback.

The connection between the two sections should be stable identities and primitive mutation events, not long-lived entity pointers in async tasks.

2. Ownership types

I agree owning relationships should be explicit. For Replica ownership, unique_ptr makes sense.

The target direction is:

Object -> owns Replicas
Replica -> owns AllocatedBuffer
SegmentManager -> owns SegmentRuntime
SegmentRuntime -> owns allocator

For the concrete C++ representation, I currently lean toward:

Object replicas: vector<unique_ptr<Replica>>
Replica buffer: unique_ptr<AllocatedBuffer>
SegmentRuntime map: unordered_map<SegmentId, unique_ptr<SegmentRuntime>>

But high-frequency observation should still use raw non-owning pointers where the lifetime is guaranteed. For example, AllocatedBuffer can keep a raw SegmentRuntime* plus SegmentId for validation/debug/recovery. The raw pointer is not ownership; it is a hot-path observer.

The allocator model should eventually be cleaned up too. Instead of relying on shared_ptr/weak_ptr to detect whether the allocator is still alive, the goal is to make the lifecycle explicit:

unmount starts -> remove from allocatable indexes -> reject new allocations
active_allocations reaches 0 / finalizer releases remaining buffers
then destroy allocator and erase SegmentRuntime

So shared_ptr/weak_ptr may be a current implementation detail, but not the target ownership model.

3. Locking and ID-based lookup

I agree with the locking concern. The intended model is not one lock acquisition per primitive event.

For one completion or mutation batch, the expected flow is:

acquire metadata shard lock once
look up Object once
resolve referenced Replicas
apply the complete in-memory mutation batch
release lock

Long operations must stay outside metadata/segment locks, including disk/network IO, etcd transactions, snapshot encoding/upload, and durable waits. If a path releases the lock and later resumes, it must re-check by stable identity rather than retaining an entity pointer.

So async tasks should carry identity, while scoped synchronous code may borrow raw pointers under the owning lock.

4. Segment and Replica states

I agree the RFC should clarify this with transition tables.

For Segment, I think DRAINED should stay. It has a useful meaning distinct from UNMOUNTING: drain is complete, but the segment has not been removed yet.

I also agree SUSPECTED is better modeled as a health/admission overlay rather than a lifecycle state. So instead of encoding product states like SUSPECTED_DRAINING, the intended model is closer to:

SegmentLifecycle:
  OK
  DRAINING
  DRAINED
  GRACEFULLY_UNMOUNTING
  UNMOUNTING

SegmentHealth:
  HEALTHY
  SUSPECTED
  OFFLINE  // Placeholder example, may not be used

For now, SUSPECTED only needs to define behavior for OK, DRAINING, DRAINED, and GRACEFULLY_UNMOUNTING. Helper predicates such as CanAllocate, CanServeRead, and CanContinueCleanup can hide the two-field checks from callers.

For Replica, I do not intend to introduce a heavy state-machine framework. The state set should be small and monotonic:

PROCESSING -> COMPLETE
PROCESSING -> FAILED -> REMOVED
PROCESSING -> REMOVED
COMPLETE   -> REMOVED

FAILED is not intended as a long-lived serving state; it should converge to REMOVED.

Durability should not become extra Replica states. For example, “complete submitted but not durable” is a pending lifecycle event, not a separate ReplicaStatus. The durable oplog event/callback drives the next action:

REPLICA_CREATE durable -> descriptor may be returned
REPLICA_COMPLETE submitted/durable -> completion path proceeds; no extra Replica durable state
REPLICA_REMOVE durable -> finalizer releases buffer and erases Replica

For Segment persistence / DurableSegmentView, I do not want to commit to an independent keyspace too early. The first step should be persist-before-publish and reader/apply hardening. Whether to split out a separate DurableSegmentView should be decided later based on actual recovery ambiguity, write frequency, allocator rebuild behavior, and rollout data.

<!-- gh-comment-id:5065766608 --> @Icedcoco commented on GitHub (Jul 24, 2026): Hi @Aionw . Thanks for the detailed comments. I agree with the main concerns. The RFC is still in planning, so I’ll first clarify the intended direction here before rewriting the RFC text. ### 1. Splitting ownership and HA concerns I agree the current RFC mixes two related but different layers: 1. in-process ownership/lifetime management; 2. HA persistence, replay, fencing, promotion, and recovery. I plan to keep them in the same RFC, but split the document into two clearly separated sections later. The first section will focus on Object/Replica/AllocatedBuffer/SegmentRuntime ownership, destruction order, borrow rules, lock scope, and async identity lookup. The second section will focus on lifecycle events, oplog replay, durable callbacks, Segment persistence, rollout, and rollback. The connection between the two sections should be stable identities and primitive mutation events, not long-lived entity pointers in async tasks. ### 2. Ownership types I agree owning relationships should be explicit. For Replica ownership, `unique_ptr` makes sense. The target direction is: ```text Object -> owns Replicas Replica -> owns AllocatedBuffer SegmentManager -> owns SegmentRuntime SegmentRuntime -> owns allocator ``` For the concrete C++ representation, I currently lean toward: ```text Object replicas: vector<unique_ptr<Replica>> Replica buffer: unique_ptr<AllocatedBuffer> SegmentRuntime map: unordered_map<SegmentId, unique_ptr<SegmentRuntime>> ``` But high-frequency observation should still use raw non-owning pointers where the lifetime is guaranteed. For example, `AllocatedBuffer` can keep a raw `SegmentRuntime*` plus `SegmentId` for validation/debug/recovery. The raw pointer is not ownership; it is a hot-path observer. The allocator model should eventually be cleaned up too. Instead of relying on `shared_ptr/weak_ptr` to detect whether the allocator is still alive, the goal is to make the lifecycle explicit: ```text unmount starts -> remove from allocatable indexes -> reject new allocations active_allocations reaches 0 / finalizer releases remaining buffers then destroy allocator and erase SegmentRuntime ``` So `shared_ptr/weak_ptr` may be a current implementation detail, but not the target ownership model. ### 3. Locking and ID-based lookup I agree with the locking concern. The intended model is not one lock acquisition per primitive event. For one completion or mutation batch, the expected flow is: ```text acquire metadata shard lock once look up Object once resolve referenced Replicas apply the complete in-memory mutation batch release lock ``` Long operations must stay outside metadata/segment locks, including disk/network IO, etcd transactions, snapshot encoding/upload, and durable waits. If a path releases the lock and later resumes, it must re-check by stable identity rather than retaining an entity pointer. So async tasks should carry identity, while scoped synchronous code may borrow raw pointers under the owning lock. ### 4. Segment and Replica states I agree the RFC should clarify this with transition tables. For Segment, I think `DRAINED` should stay. It has a useful meaning distinct from `UNMOUNTING`: drain is complete, but the segment has not been removed yet. I also agree `SUSPECTED` is better modeled as a health/admission overlay rather than a lifecycle state. So instead of encoding product states like `SUSPECTED_DRAINING`, the intended model is closer to: ```text SegmentLifecycle: OK DRAINING DRAINED GRACEFULLY_UNMOUNTING UNMOUNTING SegmentHealth: HEALTHY SUSPECTED OFFLINE // Placeholder example, may not be used ``` For now, `SUSPECTED` only needs to define behavior for `OK`, `DRAINING`, `DRAINED`, and `GRACEFULLY_UNMOUNTING`. Helper predicates such as `CanAllocate`, `CanServeRead`, and `CanContinueCleanup` can hide the two-field checks from callers. For Replica, I do not intend to introduce a heavy state-machine framework. The state set should be small and monotonic: ```text PROCESSING -> COMPLETE PROCESSING -> FAILED -> REMOVED PROCESSING -> REMOVED COMPLETE -> REMOVED ``` `FAILED` is not intended as a long-lived serving state; it should converge to `REMOVED`. Durability should not become extra Replica states. For example, “complete submitted but not durable” is a pending lifecycle event, not a separate `ReplicaStatus`. The durable oplog event/callback drives the next action: ```text REPLICA_CREATE durable -> descriptor may be returned REPLICA_COMPLETE submitted/durable -> completion path proceeds; no extra Replica durable state REPLICA_REMOVE durable -> finalizer releases buffer and erases Replica ``` For Segment persistence / DurableSegmentView, I do not want to commit to an independent keyspace too early. The first step should be persist-before-publish and reader/apply hardening. Whether to split out a separate DurableSegmentView should be decided later based on actual recovery ambiguity, write frequency, allocator rebuild behavior, and rollout data.

bsakaguchi commented 2026-10-29 21:19:20 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 22, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6331822589 --> @github-actions[bot] commented on GitHub (Oct 22, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:19:20 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 29, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:6487046592 --> @github-actions[bot] commented on GitHub (Oct 29, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:30:35 +00:00

[PR #779] [MERGED] [TE] Fix notifs problems in C wrapper #1816

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#770

No description provided.