[GH-ISSUE #4296] [RFC]: [Store] Add object storage as a first-class replica tier #1076

New issue

Open

opened 2026-10-29 21:22:36 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented 2026-10-29 21:22:36 +00:00

Owner

Copy link

Originally created by @particle128 on GitHub (Sep 23, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4296

Changes proposed

[RFC]: [Store] Add object storage as a first-class replica tier

This proposal introduces object storage as a first-class replica tier, reusing the existing OSS adapter. The new types, configuration options, and protocol extensions below are proposed changes.

1. Goals and scope

The current object storage integration follows the FileStorage / LOCAL_DISK path: the MEMORY owner performs uploads, and reads also depend on that owner. This proposal introduces an OBJECT_STORAGE replica alongside MEMORY, LOCAL_DISK, DFS, and NOF_SSD:

• The Master manages object locations, replica states, and lifecycle; Clients access shared object storage directly using descriptors.
• The Master configures one of three write modes: synchronous writes, asynchronous replication after a memory write, or direct offload on memory eviction.
• Object replicas support deletion and background reclamation.
• Once an object replica is committed, other Clients can read it even after the original writer/owner goes offline.

The initial implementation reuses the OSS adapter while keeping the interface provider-neutral. Each Mooncake object maps to one backend object. Object aggregation, cascading MEMORY → SSD → object storage, a unified redesign of all external storage tiers, and general idempotency enhancements to the shared Start/End/Revoke protocol are out of scope.

2. Architecture

flowchart TB
    subgraph MP["Master process"]
        M["Object directory, write mode, lease / pin"]
        M --> D["Existing MEMORY / LOCAL_DISK / DFS / NOF_SSD"]
        M --> O["New OBJECT_STORAGE<br/>object_key + object_size"]
        M --> G["Pending deletions / retries<br/>Existing background maintenance"]
        G -->|"DELETE after protection ends"| GA["ObjectStorageAdapter<br/>Master-side instance"]
    end
    C["Requesting Client / SDK"] <-->|"PutStart / PutEnd / Query"| M
    C -->|"Synchronous writes / direct reads"| B["ObjectStorageBackend"]
    W["MEMORY owner<br/>Offload executor"] -. "Fetch tasks / report completion" .-> M
    W -->|"Background upload from MEMORY"| B
    B --> A["ObjectStorageAdapter<br/>Client-side GetBatch / PutBatch"]
    A --> S["Shared object storage"]
    GA --> S
    classDef added fill:#fff0d6,stroke:#d97706,stroke-width:2px;
    class O,B,G added;

• Backends and adapters are in-process modules, with separate instances in each participating process. The Master does not carry object GET/PUT data traffic, but uses its own adapter for background DELETE operations.
• Deletion and retries reuse the Master's existing background maintenance mechanism, without requiring a dedicated GC thread or service. Only known pending deletions are processed; the entire bucket is not scanned.
• Both background modes reuse the owner's offload queue, heartbeat, and source-replica protection, extending them with an object-storage target and completion reporting. Reads no longer depend on the owner after the object replica is committed.
• As with DFS, the Master manages replicas while Clients access shared storage directly. Objects are addressed by key, without file-offset allocation or inheritance from the DFS allocator interface.

3. Metadata and configuration

3.1 Replica descriptor

struct ObjectStorageDescriptor {
    std::string object_key;
    uint64_t object_size;
};

// Added to ReplicateConfig
size_t os_replica_num{0};

ObjectStorageDescriptor is added to the existing Replica::Descriptor::descriptor_variant. Replica id and status remain in the shared outer descriptor rather than being duplicated in the object descriptor.

Enabling object replicas requires support in the Master and all participating Clients. The initial implementation does not promise mixed-version operation with older releases.

• os_replica_num initially supports 0 or 1, defaulting to 0. A value of 1 requests one object replica according to the Master's policy. It is neither the object service's internal replication factor nor an indication that the replica is already complete.
• The initial write requires at least one MEMORY replica. After the object replica is committed, MEMORY replicas may be evicted under the normal protection rules.
• object_key is a backend object name allocated by the Master, separate from the application's key. The adapter handles prefixes and encoding.
• Upsert, Put after Remove, and reallocation after abandoning an old task use new object names, so late uploads or old deletion tasks cannot affect new data. Ordinary retries of the same task reuse its object_key. This does not introduce a user-facing multi-version KV API.
• Existing shared object-checksum metadata is reused; provider ETags are not treated as a generic content checksum.

Using a separate object name isolates successive values of the same application key. For example, if Put(K) follows Remove(K) before the old background DELETE finishes, using K for both backend objects would let the delayed DELETE remove the new data. The Master maps the application key to an object_key, so old tasks operate only on the old object. The same isolation applies to Upsert and late uploads.

3.2 Cluster-wide backend configuration

Initially, each cluster uses a single object storage backend. The Master publishes non-secret configuration to Clients: provider, endpoint, bucket/namespace, and prefix. Clients fetch and cache it during initialization. The Master uses the same storage target for reclamation; descriptors do not duplicate this configuration.

An object's full location is the cluster-wide endpoint / bucket / prefix plus object_key.

Client-side and Master-side adapters use credentials configured in their respective processes, reusing the existing OSS environment variables without introducing a credential service:

MOONCAKE_OSS_ACCESS_KEY_ID
MOONCAKE_OSS_ACCESS_KEY_SECRET
MOONCAKE_OSS_SECURITY_TOKEN    # Optional, for temporary credentials

Clients need read/write access; Master-side reclamation needs delete access within the target bucket/prefix. They may use different identities. Secrets are not included in shared configuration or descriptors. The initial implementation retains credential loading at adapter initialization and does not introduce automatic credential refresh.

Backend configuration is fixed at cluster startup. Runtime target switching and multi-backend routing are not supported. Existing objects and upload/deletion tasks must not be reinterpreted against another namespace; descriptors, configuration caches, and tasks must not be reused across rebuilt clusters.

Object access does not require a local SSD mount. Background modes require an owner capable of fetching tasks and uploading objects.

The backend must make fully committed objects readable across Clients and support reads and repeated deletes by object_key. The adapter handles provider-specific differences.

3.3 Master write mode

A new Master option, object_storage_write_mode, defaults to sync and can be set through the configuration file or command line:

--object_storage_write_mode=sync
# Options: sync / async / offload_on_evict

Mode	Object upload timing	Foreground Put succeeds when	MEMORY handling
sync	The requesting Client uploads during the foreground Put	MEMORY, object storage, and all other required replicas are complete and committed by the Master	Normal eviction rules apply
async	After MEMORY is committed, its owner fetches an upload task	Foreground replicas are complete, and the Master has recorded the object task and protected its source	Task protection ends after object completion; MEMORY is not immediately evicted
offload_on_evict	The owner uploads when memory pressure or the high watermark triggers eviction	Foreground replicas are complete and the deferred object policy is recorded	MEMORY is reclaimed only after the object becomes COMPLETE and other protection permits it

The option applies only to writes with os_replica_num=1; 0 preserves existing behavior. The mode is bound to the write. PutStart returns the required foreground replicas and deferred target, rather than reinterpreting in-flight tasks after a configuration change.

Both background modes verify during foreground admission that at least one MEMORY owner can fetch tasks and upload objects. The sync mode only requires object-write capability on the requesting Client.

In either background mode, a successful Put does not guarantee that the data is already in object storage. Losing all valid replicas before the object is committed can still make the data unavailable. Only the object replica is deferred; other requested foreground replicas must still complete.

For these keys, the object policy controls automatic offload scheduling to avoid duplicate dispatch through the existing LOCAL_DISK path. Tasks explicitly target OBJECT_STORAGE and do not register completion as LOCAL_DISK. Existing LOCAL_DISK replicas are unaffected.

4. Write paths

4.1 Synchronous writes

sequenceDiagram
    participant C as Requesting Client
    participant M as Master
    participant S as Object storage
    C->>M: PutStart(key, os_replica_num=1)
    M-->>C: PROCESSING object replica descriptor
    C->>S: PutV / PutBatch
    S-->>C: Per-object write results
    C->>M: PutEnd (all required replicas complete)
    M-->>C: Publish COMPLETE and return success

Data transfers for MEMORY and other replicas are omitted from the diagram. The Master creates object-replica metadata and allocates an object_key; the Client uploads through the existing adapter. Completion is determined per key, including per-key results within a batch. Failure of any required replica must not be reported as success for that key.

4.2 Asynchronous replication after a memory write

sequenceDiagram
    participant C as Requesting Client
    participant M as Master
    participant O as MEMORY owner
    participant S as Object storage
    C->>M: PutStart(os_replica_num=1)
    M-->>C: MEMORY locations + deferred object target
    C->>O: Write MEMORY
    C->>M: PutEnd (foreground replicas complete)
    M->>M: Commit MEMORY, record task, protect source
    M-->>C: Put succeeds
    O->>M: Fetch tasks through offload heartbeat
    M-->>O: Source replica + target replica id / object_key
    O->>S: Upload from protected MEMORY
    S-->>O: Upload complete
    O->>M: Report target replica id / object_key and result
    M->>M: Mark object COMPLETE and release task protection
    M-->>O: Acknowledge commit

• Before PutEnd returns success, foreground replicas must be committed, the task recorded, and its source protected. Failure to register the task must not be reported as success.
• The owner is a Client holding the source MEMORY replica and capable of object uploads; it need not be the Client that initiated Put.
• The existing offload heartbeat is reused, with a default 10-second interval after each work round finishes. While queued or uploading, the object replica is PROCESSING and MEMORY remains readable.
• Successful replication releases only task protection. Later memory eviction can use the existing COMPLETE object for the same value without another upload.

4.3 Offload on memory eviction

flowchart LR
    A["MEMORY high watermark / allocation pressure"] --> B["Master selects candidates<br/>Reserves object and task; protects source"]
    B --> C["Owner fetches task<br/>Uploads MEMORY directly to object storage"]
    C --> D["Master commits<br/>OBJECT_STORAGE COMPLETE"]
    D --> E["Wait for other leases / pins<br/>Reclaim MEMORY"]
    C -. "Failure / unknown outcome" .-> F["Keep source; retry<br/>Backpressure / alert on backlog"]

The foreground Put records only the deferred policy. Object and task reservation occurs when eviction is triggered. Existing memory-eviction candidate rules are reused. An existing COMPLETE object for the same value avoids another upload, and an existing task avoids duplicate scheduling.

Source memory must not be released merely to lower the watermark before the object is committed or while read/write tasks still reference it. Memory watermarks must leave upload headroom, with bounds on queue size and protected bytes. If object storage cannot keep up, new Puts may be backpressured or fail; data is not automatically redirected to SSD.

5. Read path

sequenceDiagram
    participant C as Requesting Client
    participant M as Master
    participant S as Object storage
    C->>M: Query / BatchQuery(key)
    M-->>C: COMPLETE object descriptor + lease
    C->>S: Get / GetBatch(object_key)
    S-->>C: Data written into destination buffer

All three write modes share this read path, retaining MEMORY preference. Uncommitted objects are not selected for reads.

• Support ordinary, batched, Python-buffer, and ranged-session entry points. Paths requiring staging use a byte budget; long reads must finish within their lease or renew it.
• Distinguish NotFound, permission errors, timeouts, short reads, and checksum failures. Report a replica as missing only after confirming that the exact object does not exist; network errors are not evidence of deletion.
• Foreground operations use the caller's execution context; background operations use the owner's offload executor. This proposal does not require a dedicated OSS thread pool.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @particle128 on GitHub (Sep 23, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4296 ### Changes proposed # [RFC]: [Store] Add object storage as a first-class replica tier This proposal introduces object storage as a first-class replica tier, reusing the [existing OSS adapter](https://kvcache-ai.github.io/Mooncake/design/store/oss-backend.html). The new types, configuration options, and protocol extensions below are proposed changes. ## 1. Goals and scope The current object storage integration follows the FileStorage / LOCAL_DISK path: the MEMORY owner performs uploads, and reads also depend on that owner. This proposal introduces an `OBJECT_STORAGE` replica alongside MEMORY, LOCAL_DISK, DFS, and NOF_SSD: - The Master manages object locations, replica states, and lifecycle; Clients access shared object storage directly using descriptors. - The Master configures one of three write modes: synchronous writes, asynchronous replication after a memory write, or direct offload on memory eviction. - Object replicas support deletion and background reclamation. - Once an object replica is committed, other Clients can read it even after the original writer/owner goes offline. The initial implementation reuses the OSS adapter while keeping the interface provider-neutral. Each Mooncake object maps to one backend object. Object aggregation, cascading MEMORY → SSD → object storage, a unified redesign of all external storage tiers, and general idempotency enhancements to the shared Start/End/Revoke protocol are out of scope. ## 2. Architecture ```mermaid flowchart TB subgraph MP["Master process"] M["Object directory, write mode, lease / pin"] M --> D["Existing MEMORY / LOCAL_DISK / DFS / NOF_SSD"] M --> O["New OBJECT_STORAGE<br/>object_key + object_size"] M --> G["Pending deletions / retries<br/>Existing background maintenance"] G -->|"DELETE after protection ends"| GA["ObjectStorageAdapter<br/>Master-side instance"] end C["Requesting Client / SDK"] <-->|"PutStart / PutEnd / Query"| M C -->|"Synchronous writes / direct reads"| B["ObjectStorageBackend"] W["MEMORY owner<br/>Offload executor"] -. "Fetch tasks / report completion" .-> M W -->|"Background upload from MEMORY"| B B --> A["ObjectStorageAdapter<br/>Client-side GetBatch / PutBatch"] A --> S["Shared object storage"] GA --> S classDef added fill:#fff0d6,stroke:#d97706,stroke-width:2px; class O,B,G added; ``` - Backends and adapters are in-process modules, with separate instances in each participating process. The Master does not carry object GET/PUT data traffic, but uses its own adapter for background DELETE operations. - Deletion and retries reuse the Master's existing background maintenance mechanism, without requiring a dedicated GC thread or service. Only known pending deletions are processed; the entire bucket is not scanned. - Both background modes reuse the owner's offload queue, heartbeat, and source-replica protection, extending them with an object-storage target and completion reporting. Reads no longer depend on the owner after the object replica is committed. - As with DFS, the Master manages replicas while Clients access shared storage directly. Objects are addressed by key, without file-offset allocation or inheritance from the DFS allocator interface. ## 3. Metadata and configuration ### 3.1 Replica descriptor ```cpp struct ObjectStorageDescriptor { std::string object_key; uint64_t object_size; }; // Added to ReplicateConfig size_t os_replica_num{0}; ``` `ObjectStorageDescriptor` is added to the existing `Replica::Descriptor::descriptor_variant`. Replica `id` and `status` remain in the shared outer descriptor rather than being duplicated in the object descriptor. Enabling object replicas requires support in the Master and all participating Clients. The initial implementation does not promise mixed-version operation with older releases. - `os_replica_num` initially supports 0 or 1, defaulting to 0. A value of 1 requests one object replica according to the Master's policy. It is neither the object service's internal replication factor nor an indication that the replica is already complete. - The initial write requires at least one MEMORY replica. After the object replica is committed, MEMORY replicas may be evicted under the normal protection rules. - `object_key` is a backend object name allocated by the Master, separate from the application's key. The adapter handles prefixes and encoding. - Upsert, Put after Remove, and reallocation after abandoning an old task use new object names, so late uploads or old deletion tasks cannot affect new data. Ordinary retries of the same task reuse its `object_key`. This does not introduce a user-facing multi-version KV API. - Existing shared object-checksum metadata is reused; provider ETags are not treated as a generic content checksum. Using a separate object name isolates successive values of the same application key. For example, if `Put(K)` follows `Remove(K)` before the old background DELETE finishes, using K for both backend objects would let the delayed DELETE remove the new data. The Master maps the application key to an `object_key`, so old tasks operate only on the old object. The same isolation applies to Upsert and late uploads. ### 3.2 Cluster-wide backend configuration Initially, each cluster uses a single object storage backend. The Master publishes non-secret configuration to Clients: provider, endpoint, bucket/namespace, and prefix. Clients fetch and cache it during initialization. The Master uses the same storage target for reclamation; descriptors do not duplicate this configuration. An object's full location is **the cluster-wide endpoint / bucket / prefix plus `object_key`**. Client-side and Master-side adapters use credentials configured in their respective processes, reusing the existing OSS environment variables without introducing a credential service: ```text MOONCAKE_OSS_ACCESS_KEY_ID MOONCAKE_OSS_ACCESS_KEY_SECRET MOONCAKE_OSS_SECURITY_TOKEN # Optional, for temporary credentials ``` Clients need read/write access; Master-side reclamation needs delete access within the target bucket/prefix. They may use different identities. Secrets are not included in shared configuration or descriptors. The initial implementation retains credential loading at adapter initialization and does not introduce automatic credential refresh. Backend configuration is fixed at cluster startup. Runtime target switching and multi-backend routing are not supported. Existing objects and upload/deletion tasks must not be reinterpreted against another namespace; descriptors, configuration caches, and tasks must not be reused across rebuilt clusters. Object access does not require a local SSD mount. Background modes require an owner capable of fetching tasks and uploading objects. The backend must make fully committed objects readable across Clients and support reads and repeated deletes by `object_key`. The adapter handles provider-specific differences. ### 3.3 Master write mode A new Master option, `object_storage_write_mode`, defaults to `sync` and can be set through the configuration file or command line: ```text --object_storage_write_mode=sync # Options: sync / async / offload_on_evict ``` | Mode | Object upload timing | Foreground Put succeeds when | MEMORY handling | | --- | --- | --- | --- | | `sync` | The requesting Client uploads during the foreground Put | MEMORY, object storage, and all other required replicas are complete and committed by the Master | Normal eviction rules apply | | `async` | After MEMORY is committed, its owner fetches an upload task | Foreground replicas are complete, and the Master has recorded the object task and protected its source | Task protection ends after object completion; MEMORY is not immediately evicted | | `offload_on_evict` | The owner uploads when memory pressure or the high watermark triggers eviction | Foreground replicas are complete and the deferred object policy is recorded | MEMORY is reclaimed only after the object becomes COMPLETE and other protection permits it | The option applies only to writes with `os_replica_num=1`; 0 preserves existing behavior. The mode is bound to the write. PutStart returns the required foreground replicas and deferred target, rather than reinterpreting in-flight tasks after a configuration change. Both background modes verify during foreground admission that at least one MEMORY owner can fetch tasks and upload objects. The `sync` mode only requires object-write capability on the requesting Client. **In either background mode, a successful Put does not guarantee that the data is already in object storage. Losing all valid replicas before the object is committed can still make the data unavailable.** Only the object replica is deferred; other requested foreground replicas must still complete. For these keys, the object policy controls automatic offload scheduling to avoid duplicate dispatch through the existing LOCAL_DISK path. Tasks explicitly target OBJECT_STORAGE and do not register completion as LOCAL_DISK. Existing LOCAL_DISK replicas are unaffected. ## 4. Write paths ### 4.1 Synchronous writes ```mermaid sequenceDiagram participant C as Requesting Client participant M as Master participant S as Object storage C->>M: PutStart(key, os_replica_num=1) M-->>C: PROCESSING object replica descriptor C->>S: PutV / PutBatch S-->>C: Per-object write results C->>M: PutEnd (all required replicas complete) M-->>C: Publish COMPLETE and return success ``` Data transfers for MEMORY and other replicas are omitted from the diagram. The Master creates object-replica metadata and allocates an `object_key`; the Client uploads through the existing adapter. Completion is determined per key, including per-key results within a batch. Failure of any required replica must not be reported as success for that key. ### 4.2 Asynchronous replication after a memory write ```mermaid sequenceDiagram participant C as Requesting Client participant M as Master participant O as MEMORY owner participant S as Object storage C->>M: PutStart(os_replica_num=1) M-->>C: MEMORY locations + deferred object target C->>O: Write MEMORY C->>M: PutEnd (foreground replicas complete) M->>M: Commit MEMORY, record task, protect source M-->>C: Put succeeds O->>M: Fetch tasks through offload heartbeat M-->>O: Source replica + target replica id / object_key O->>S: Upload from protected MEMORY S-->>O: Upload complete O->>M: Report target replica id / object_key and result M->>M: Mark object COMPLETE and release task protection M-->>O: Acknowledge commit ``` - Before PutEnd returns success, foreground replicas must be committed, the task recorded, and its source protected. Failure to register the task must not be reported as success. - The owner is a Client holding the source MEMORY replica and capable of object uploads; it need not be the Client that initiated Put. - The existing offload heartbeat is reused, with a default 10-second interval after each work round finishes. While queued or uploading, the object replica is PROCESSING and MEMORY remains readable. - Successful replication releases only task protection. Later memory eviction can use the existing COMPLETE object for the same value without another upload. ### 4.3 Offload on memory eviction ```mermaid flowchart LR A["MEMORY high watermark / allocation pressure"] --> B["Master selects candidates<br/>Reserves object and task; protects source"] B --> C["Owner fetches task<br/>Uploads MEMORY directly to object storage"] C --> D["Master commits<br/>OBJECT_STORAGE COMPLETE"] D --> E["Wait for other leases / pins<br/>Reclaim MEMORY"] C -. "Failure / unknown outcome" .-> F["Keep source; retry<br/>Backpressure / alert on backlog"] ``` The foreground Put records only the deferred policy. Object and task reservation occurs when eviction is triggered. Existing memory-eviction candidate rules are reused. An existing COMPLETE object for the same value avoids another upload, and an existing task avoids duplicate scheduling. **Source memory must not be released merely to lower the watermark before the object is committed or while read/write tasks still reference it.** Memory watermarks must leave upload headroom, with bounds on queue size and protected bytes. If object storage cannot keep up, new Puts may be backpressured or fail; data is not automatically redirected to SSD. ## 5. Read path ```mermaid sequenceDiagram participant C as Requesting Client participant M as Master participant S as Object storage C->>M: Query / BatchQuery(key) M-->>C: COMPLETE object descriptor + lease C->>S: Get / GetBatch(object_key) S-->>C: Data written into destination buffer ``` All three write modes share this read path, retaining MEMORY preference. Uncommitted objects are not selected for reads. - Support ordinary, batched, Python-buffer, and ranged-session entry points. Paths requiring staging use a byte budget; long reads must finish within their lease or renew it. - Distinguish NotFound, permission errors, timeouts, short reads, and checksum failures. Report a replica as missing only after confirming that the exact object does not exist; network errors are not evidence of deletion. - Foreground operations use the caller's execution context; background operations use the owner's offload executor. This proposal does not require a dedicated OSS thread pool. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label 2026-10-29 21:22:36 +00:00

bsakaguchi commented 2026-10-29 21:22:37 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Sep 23, 2026):

Thanks for opening this issue, @particle128!

Field	Value
Issue	#4296
GitHub user ID	3041842
Reporter	@particle128

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:5795308168 --> @github-actions[bot] commented on GitHub (Sep 23, 2026): Thanks for opening this issue, @particle128! | Field | Value | |-------|-------| | **Issue** | #4296 | | **GitHub user ID** | `3041842` | | **Reporter** | @particle128 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:22:37 +00:00

Author

Owner

Copy link

@fcczzz commented on GitHub (Sep 24, 2026):

@particle128 Thanks for the proposal! Making object-storage replicas readable independently of the original owner would be a useful improvement.

Have you considered extending the existing DFS abstraction instead of introducing a new top-level OBJECT_STORAGE replica type? DFS already supports shard and bucket allocators, so perhaps an object-storage allocator could handle unique object-key allocation and deferred deletion, with the backend routing reads and writes through the existing object-storage adapter.

That would still require descriptor and I/O-path extensions, but both approaches seem to share the same core model: Master-managed replica metadata with direct Client access to shared storage. It would be helpful to include a comparison in the RFC—are there lifecycle or API requirements that make a separate replica type preferable, or is the main motivation to keep this change scoped?

<!-- gh-comment-id:5806831509 --> @fcczzz commented on GitHub (Sep 24, 2026): @particle128 Thanks for the proposal! Making object-storage replicas readable independently of the original owner would be a useful improvement. Have you considered extending the existing DFS abstraction instead of introducing a new top-level `OBJECT_STORAGE` replica type? DFS already supports shard and bucket allocators, so perhaps an object-storage allocator could handle unique object-key allocation and deferred deletion, with the backend routing reads and writes through the existing object-storage adapter. That would still require descriptor and I/O-path extensions, but both approaches seem to share the same core model: Master-managed replica metadata with direct Client access to shared storage. It would be helpful to include a comparison in the RFC—are there lifecycle or API requirements that make a separate replica type preferable, or is the main motivation to keep this change scoped?

bsakaguchi commented 2026-10-29 21:22:37 +00:00

Author

Owner

Copy link

@hongzhi-gao commented on GitHub (Oct 16, 2026):

Revised review units and phase acceptance criteria

Updated after fcczzz's clarification: A separate OBJECT_STORAGE replica type is acceptable. The requirement is to share common replica-management semantics, rather than choose an enum as a substitute for code reuse. The earlier recommendation to require both locations under one replica category is superseded.

There are still three user-visible phases. The review split now separates bounded I/O, the catalog/recovery boundary, and synchronous integration; the number of PRs is secondary to a coherent correctness boundary.

Review unit	Scope	Acceptance
1 — #4727	Bounded object I/O, exact ranges, staging limits, single-attempt adapter capability and HTTP regressions	Independently buildable I/O component; no catalog, replica publication or owner-independent Store-read claim
2 — #4936 (Draft)	Extracted persistent upload/cleanup catalog and review of its ownership/recovery contract	Clarify Master ownership of logical heads/visibility/commit versus object attempt/retirement obligations, durable acknowledgement/fencing, and ordered log/standby/snapshot integration before subsequent integration relies on it
3	Complete synchronous first-class object replicas through common Master/Client/RPC/SDK management	Complete Phase 1 below, including removal and durable recovery; only sync is enabled
4	Asynchronous replication using common task/source management	Complete Phase 2 below
5	Offload on ordinary and tenant-quota eviction	Complete Phase 3 below

Implementation status: #4727 has been narrowed to I/O. #4936 preserves the extracted catalog implementation and component tests for focused review. Its independent logical heads/commit flags are not an agreed recovery authority, and it has not implemented unified Master log/snapshot recovery. The full integration, async and eviction review units remain future work.

Proposed initial deployment scope: First-class OSS requires a supported durable recovery configuration with DFS disabled. The first phase does not promise DFS/OSS coexistence in one cluster or both replicas for one key. Current upstream rejects DFS with snapshot/oplog recovery and cannot restore DFS allocator state; that gap must be resolved before expanding coexistence. Unsupported combinations must be rejected explicitly. This is a proposed integration scope, not a new restriction implemented by the I/O-only PR.

Durable upload/cleanup obligations remain necessary. The preferred ownership proposal extends the existing ordered recovery authority, rather than introducing a second authoritative logical-head directory. The exact interface is for the separate catalog review. Existing DFS recovery and PutEnd are insufficient unchanged. Object visibility, immutable attempt fencing, late-upload cleanup and recovery guarantees must accompany the first complete synchronous feature.

Phase 1 — Complete synchronous object replicas (review units 1–3)

Acceptance criteria:

• When an object replica is requested, Put succeeds only after every required foreground transfer, the object upload and the durable Master commit complete. The request/configuration spelling remains subject to the shared-storage interface decision.
• After the MEMORY owner exits, another client can read the committed object directly. Supported C++/Python entry points, including batches, buffers and get-session ranges, have integration coverage; unsupported entry points reject the configuration explicitly.
• File ranges and object keys have distinct, truthful location descriptors. Completion/visibility, leases, pins and logical removal reuse common management paths; backend-specific allocation and physical reclamation remain explicit.
• Remove and Upsert isolate versions and attempts. Completion/revoke notifications identify the exact write; cleanup targets immutable physical names. Old uploads, completion notifications and deletion tasks cannot affect a replacement value.
• Existing read leases delay physical reclamation. The failover contract must explicitly choose a conservative read-TTL grace period or durable lease bounds before enabling reclamation after promotion.
• Upload timeout, lost completion replies, late PUT after DELETE and deletion retries are covered. Notification retries do not repeat a physical PUT into the same attempt name; uncertain uploads retain bounded cleanup obligations.
• Restart and HA recovery restore committed object-only values and preserve all unresolved upload/retirement records, including records for logically deleted keys. Snapshot capture and log pruning preserve the same consistent recovery boundary; deleted or replaced values cannot reappear.
• The first durable feature requires a supported persistence/recovery configuration. This does not establish an intrinsic requirement for a separate etcd catalog or promise unsupported Redis snapshot/HA combinations.

Phase 2 — Complete asynchronous replication (review unit 4)

Acceptance criteria:

• Put succeeds after required foreground replicas complete, the upload intent is durable, and the exact MEMORY source is protected. This success does not promise object-storage durability before upload completion.
• An upload-capable MEMORY owner performs the upload, including when the writer is a different client. The object becomes readable only after accepted completion.
• Source protection covers Remove/Upsert, Move, segment clearing, cleanup, eviction and graceful unmount. Completion releases only the corresponding attempt's protection.
• Fault/recovery tests cover lost claims or completion replies, fresh-name retries, pending-budget exhaustion, owner loss, restart and HA promotion. Loss of an upload source must not erase a surviving readable replica in another tier.

Phase 3 — Complete offload on eviction (review unit 5)

Acceptance criteria:

• Put records a deferred policy; ordinary maintenance does not eagerly turn it into an upload. Allocation pressure or eviction admits the task within the configured pending budget.
• The last MEMORY copy remains protected until the object replica is COMPLETE and other leases/pins permit reclamation.
• Both ordinary and tenant-quota eviction follow this rule. An already-complete object copy avoids redundant legacy LOCAL_DISK offload.
• Upload failure or exhausted budget preserves the readable source and does not prevent unrelated eligible objects from being evicted. Integration tests demonstrate eventual memory reclamation after successful upload.

Delivery constraints

• Keep tests and documentation with the PR introducing the behavior. Deletion, source lifetime and recovery guarantees belong to the first phase that needs them.
• Aim for roughly 1,000 LOC of core changes per PR, allowing a modest overrun for a coherent review unit. Do not split an indivisible persistent transition or defer correctness guarantees to meet a line count.
• Port and validate against the actual upstream base. Historical test results apply only to their recorded code/configuration; they do not validate the proposed interfaces or their HA integration.
• This is a revised proposal, not a claim that the community has accepted the architecture or that the existing PR code has already been adapted.

<!-- gh-comment-id:6213875798 --> @hongzhi-gao commented on GitHub (Oct 16, 2026): ### Revised review units and phase acceptance criteria **Updated after [fcczzz's clarification](https://github.com/kvcache-ai/Mooncake/issues/4296#issuecomment-6393757368):** A separate `OBJECT_STORAGE` replica type is acceptable. The requirement is to share common replica-management semantics, rather than choose an enum as a substitute for code reuse. The earlier recommendation to require both locations under one replica category is superseded. There are still **three user-visible phases**. The review split now separates bounded I/O, the catalog/recovery boundary, and synchronous integration; the number of PRs is secondary to a coherent correctness boundary. | Review unit | Scope | Acceptance | | --- | --- | --- | | 1 — #4727 | Bounded object I/O, exact ranges, staging limits, single-attempt adapter capability and HTTP regressions | Independently buildable I/O component; no catalog, replica publication or owner-independent Store-read claim | | 2 — #4936 (Draft) | Extracted persistent upload/cleanup catalog and review of its ownership/recovery contract | Clarify Master ownership of logical heads/visibility/commit versus object attempt/retirement obligations, durable acknowledgement/fencing, and ordered log/standby/snapshot integration before subsequent integration relies on it | | 3 | Complete synchronous first-class object replicas through common Master/Client/RPC/SDK management | Complete Phase 1 below, including removal and durable recovery; only sync is enabled | | 4 | Asynchronous replication using common task/source management | Complete Phase 2 below | | 5 | Offload on ordinary and tenant-quota eviction | Complete Phase 3 below | **Implementation status:** #4727 has been narrowed to I/O. #4936 preserves the extracted catalog implementation and component tests for focused review. Its independent logical heads/commit flags are not an agreed recovery authority, and it has not implemented unified Master log/snapshot recovery. The full integration, async and eviction review units remain future work. **Proposed initial deployment scope:** First-class OSS requires a supported durable recovery configuration with DFS disabled. The first phase does not promise DFS/OSS coexistence in one cluster or both replicas for one key. Current upstream rejects DFS with snapshot/oplog recovery and cannot restore DFS allocator state; that gap must be resolved before expanding coexistence. Unsupported combinations must be rejected explicitly. This is a proposed integration scope, not a new restriction implemented by the I/O-only PR. Durable upload/cleanup obligations remain necessary. The preferred ownership proposal extends the existing ordered recovery authority, rather than introducing a second authoritative logical-head directory. The exact interface is for the separate catalog review. Existing DFS recovery and PutEnd are insufficient unchanged. Object visibility, immutable attempt fencing, late-upload cleanup and recovery guarantees must accompany the first complete synchronous feature. **Phase 1 — Complete synchronous object replicas (review units 1–3)** Acceptance criteria: - When an object replica is requested, Put succeeds only after every required foreground transfer, the object upload and the durable Master commit complete. The request/configuration spelling remains subject to the shared-storage interface decision. - After the MEMORY owner exits, another client can read the committed object directly. Supported C++/Python entry points, including batches, buffers and get-session ranges, have integration coverage; unsupported entry points reject the configuration explicitly. - File ranges and object keys have distinct, truthful location descriptors. Completion/visibility, leases, pins and logical removal reuse common management paths; backend-specific allocation and physical reclamation remain explicit. - Remove and Upsert isolate versions and attempts. Completion/revoke notifications identify the exact write; cleanup targets immutable physical names. Old uploads, completion notifications and deletion tasks cannot affect a replacement value. - Existing read leases delay physical reclamation. The failover contract must explicitly choose a conservative read-TTL grace period or durable lease bounds before enabling reclamation after promotion. - Upload timeout, lost completion replies, late PUT after DELETE and deletion retries are covered. Notification retries do not repeat a physical PUT into the same attempt name; uncertain uploads retain bounded cleanup obligations. - Restart and HA recovery restore committed object-only values and preserve all unresolved upload/retirement records, including records for logically deleted keys. Snapshot capture and log pruning preserve the same consistent recovery boundary; deleted or replaced values cannot reappear. - The first durable feature requires a supported persistence/recovery configuration. This does not establish an intrinsic requirement for a separate etcd catalog or promise unsupported Redis snapshot/HA combinations. **Phase 2 — Complete asynchronous replication (review unit 4)** Acceptance criteria: - Put succeeds after required foreground replicas complete, the upload intent is durable, and the exact MEMORY source is protected. This success does not promise object-storage durability before upload completion. - An upload-capable MEMORY owner performs the upload, including when the writer is a different client. The object becomes readable only after accepted completion. - Source protection covers Remove/Upsert, Move, segment clearing, cleanup, eviction and graceful unmount. Completion releases only the corresponding attempt's protection. - Fault/recovery tests cover lost claims or completion replies, fresh-name retries, pending-budget exhaustion, owner loss, restart and HA promotion. Loss of an upload source must not erase a surviving readable replica in another tier. **Phase 3 — Complete offload on eviction (review unit 5)** Acceptance criteria: - Put records a deferred policy; ordinary maintenance does not eagerly turn it into an upload. Allocation pressure or eviction admits the task within the configured pending budget. - The last MEMORY copy remains protected until the object replica is COMPLETE and other leases/pins permit reclamation. - Both ordinary and tenant-quota eviction follow this rule. An already-complete object copy avoids redundant legacy LOCAL_DISK offload. - Upload failure or exhausted budget preserves the readable source and does not prevent unrelated eligible objects from being evicted. Integration tests demonstrate eventual memory reclamation after successful upload. **Delivery constraints** - Keep tests and documentation with the PR introducing the behavior. Deletion, source lifetime and recovery guarantees belong to the first phase that needs them. - Aim for roughly 1,000 LOC of core changes per PR, allowing a modest overrun for a coherent review unit. Do not split an indivisible persistent transition or defer correctness guarantees to meet a line count. - Port and validate against the actual upstream base. Historical test results apply only to their recorded code/configuration; they do not validate the proposed interfaces or their HA integration. - This is a revised proposal, not a claim that the community has accepted the architecture or that the existing PR code has already been adapted.

bsakaguchi commented 2026-10-29 21:22:37 +00:00

Author

Owner

Copy link

@fcczzz commented on GitHub (Oct 23, 2026):

@hongzhi-gao Hi, thanks for explaining. My main concern is preserving a common abstraction and reusing replica-management logic.

From Mooncake’s perspective, DFS and object storage seem to serve the same role: the Master manages replica locations and states, while Clients access shared storage directly. File-and-offset addressing versus object-key addressing clearly requires different descriptors and I/O implementations, but could those differences be handled through an object allocator and backend extensions?

Uncertain upload outcomes, late completion, and deferred deletion need explicit handling in either design. These requirements alone don’t seem to establish why a separate top-level replica type is preferable. My concern is that adding OBJECT_STORAGE could introduce parallel branches across allocation, completion, read selection, lease protection, reclamation, and recovery, even though much of the lifecycle is shared.

Could the RFC compare the two approaches, identifying which management paths remain shared and where genuinely different behavior is required? In particular, what would be difficult to express through allocator/backend interfaces?

If the main benefit of a separate type is limiting changes to existing DFS code for the initial implementation, that is understandable. It would help to make that tradeoff explicit and explain how we would avoid duplicating the common lifecycle logic.

<!-- gh-comment-id:6356137997 --> @fcczzz commented on GitHub (Oct 23, 2026): @hongzhi-gao Hi, thanks for explaining. My main concern is preserving a common abstraction and reusing replica-management logic. From Mooncake’s perspective, DFS and object storage seem to serve the same role: the Master manages replica locations and states, while Clients access shared storage directly. File-and-offset addressing versus object-key addressing clearly requires different descriptors and I/O implementations, but could those differences be handled through an object allocator and backend extensions? Uncertain upload outcomes, late completion, and deferred deletion need explicit handling in either design. These requirements alone don’t seem to establish why a separate top-level replica type is preferable. My concern is that adding OBJECT_STORAGE could introduce parallel branches across allocation, completion, read selection, lease protection, reclamation, and recovery, even though much of the lifecycle is shared. Could the RFC compare the two approaches, identifying which management paths remain shared and where genuinely different behavior is required? In particular, what would be difficult to express through allocator/backend interfaces? If the main benefit of a separate type is limiting changes to existing DFS code for the initial implementation, that is understandable. It would help to make that tradeoff explicit and explain how we would avoid duplicating the common lifecycle logic.

bsakaguchi commented 2026-10-29 21:22:38 +00:00

Author

Owner

Copy link

@hongzhi-gao commented on GitHub (Oct 23, 2026):

Update (2026-10-26): Following fcczzz's clarification, keeping a separate OBJECT_STORAGE type is acceptable; my earlier recommendation below to require one shared replica category is superseded. See the current scope/management-boundary reply and updated delivery plan. #4727 is now I/O-only, with catalog review in draft #4936.

---

Thanks, @fcczzz. After tracing the allocator, Master/Client lifecycle, and current recovery paths, I think extending the shared-storage abstraction is the better direction.

I did not find a lifecycle requirement that inherently needs a separate top-level OBJECT_STORAGE type. File ranges and object keys need distinct location descriptors and I/O implementations, but allocation bookkeeping, replica completion/visibility, read leases, pins, and logical removal can remain shared. Upload uncertainty and deferred deletion require explicit handling in either design.

This needs interface changes: the allocator currently returns only DistributedFSDescriptor, and Free() cannot report asynchronous reclamation. I propose tagged file/object locations, exact write-attempt tokens, and separate logical retirement and physical reclamation contracts, while preserving shard/bucket-specific allocation and deletion units.

I would also revise the catalog split. Durable upload and cleanup records are necessary, but a second authoritative logical-head catalog is not inherently required. Current upstream provides ordered batch logging and materialized snapshots; these need extensions for pending attempts and reclamation records, plus durable acknowledgement before committing object replicas. Existing DFS recovery and PutEnd cannot simply be reused unchanged.

Would you agree with keeping both location variants under the shared-storage replica category, using the existing log/snapshot authority for visibility and recovery, and requiring durable recovery configuration for the first synchronous feature? The remaining lease decision is whether failover uses a conservative read-TTL grace period or persists lease bounds.

This is a source-level design comparison; I have not implemented or runtime-validated the proposed interfaces.

<!-- gh-comment-id:6356684936 --> @hongzhi-gao commented on GitHub (Oct 23, 2026): **Update (2026-10-26):** Following [fcczzz's clarification](https://github.com/kvcache-ai/Mooncake/issues/4296#issuecomment-6393757368), keeping a separate `OBJECT_STORAGE` type is acceptable; my earlier recommendation below to require one shared replica category is superseded. See [the current scope/management-boundary reply](https://github.com/kvcache-ai/Mooncake/issues/4296#issuecomment-6407163757) and [updated delivery plan](https://github.com/kvcache-ai/Mooncake/issues/4296#issuecomment-6213875798). #4727 is now I/O-only, with catalog review in draft #4936. --- Thanks, @fcczzz. After tracing the allocator, Master/Client lifecycle, and current recovery paths, I think extending the shared-storage abstraction is the better direction. I did not find a lifecycle requirement that inherently needs a separate top-level `OBJECT_STORAGE` type. File ranges and object keys need distinct location descriptors and I/O implementations, but allocation bookkeeping, replica completion/visibility, read leases, pins, and logical removal can remain shared. Upload uncertainty and deferred deletion require explicit handling in either design. This needs interface changes: the allocator currently returns only `DistributedFSDescriptor`, and `Free()` cannot report asynchronous reclamation. I propose tagged file/object locations, exact write-attempt tokens, and separate logical retirement and physical reclamation contracts, while preserving shard/bucket-specific allocation and deletion units. I would also revise the catalog split. Durable upload and cleanup records are necessary, but a second authoritative logical-head catalog is not inherently required. Current upstream provides ordered batch logging and materialized snapshots; these need extensions for pending attempts and reclamation records, plus durable acknowledgement before committing object replicas. Existing DFS recovery and PutEnd cannot simply be reused unchanged. Would you agree with keeping both location variants under the shared-storage replica category, using the existing log/snapshot authority for visibility and recovery, and requiring durable recovery configuration for the first synchronous feature? The remaining lease decision is whether failover uses a conservative read-TTL grace period or persists lease bounds. This is a source-level design comparison; I have not implemented or runtime-validated the proposed interfaces.

bsakaguchi commented 2026-10-29 21:22:38 +00:00

Author

Owner

Copy link

@fcczzz commented on GitHub (Oct 25, 2026):

@hongzhi-gao Hi, thanks for tracing the existing paths and updating the proposal. I'd like to clarify my earlier suggestion: extending DFS was an option to explore, not a requirement to avoid a separate OBJECT_STORAGE type. I'm comfortable proceeding with the separate type.

My main concern is reusing common replica-management logic where the semantics match, so introducing another backend does not require parallel allocation, completion, revoke, lease, removal, and recovery flows across Put, BatchPut, and Upsert. Sharing the same enum would not, by itself, prevent that duplication either.

As we proceed, could you clarify:

• Whether the initial scope supports DFS and OSS in the same cluster, and whether one key can request a replica in each. This is a scope clarification, not a requirement to support both initially.
• Which management paths will remain shared, and which operations need object-specific handling.

I support the phased delivery plan. For #4727, could we keep the object-storage I/O portion in this PR and move the persistent catalog/lifecycle management into a separate PR for focused review?

The catalog tracks upload and cleanup obligations, but also logical head and commit state, which overlaps with the Master's existing metadata and recovery responsibilities. In that separate PR, we should clarify ownership of these facts, what can extend the existing log/snapshot mechanisms, and how recovery handles discrepancies and unfinished operations. Object-specific persistent records may be necessary; the important question is how they fit into the overall recovery contract.

We can start reviewing the I/O portion now, while agreeing on the catalog boundary before subsequent integration builds on it. There is no need to change the replica type solely to accommodate my earlier suggestion.

<!-- gh-comment-id:6393757368 --> @fcczzz commented on GitHub (Oct 25, 2026): @hongzhi-gao Hi, thanks for tracing the existing paths and updating the proposal. I'd like to clarify my earlier suggestion: extending DFS was an option to explore, not a requirement to avoid a separate OBJECT_STORAGE type. I'm comfortable proceeding with the separate type. My main concern is reusing common replica-management logic where the semantics match, so introducing another backend does not require parallel allocation, completion, revoke, lease, removal, and recovery flows across Put, BatchPut, and Upsert. Sharing the same enum would not, by itself, prevent that duplication either. As we proceed, could you clarify: - Whether the initial scope supports DFS and OSS in the same cluster, and whether one key can request a replica in each. This is a scope clarification, not a requirement to support both initially. - Which management paths will remain shared, and which operations need object-specific handling. I support the phased delivery plan. For #4727, could we keep the object-storage I/O portion in this PR and move the persistent catalog/lifecycle management into a separate PR for focused review? The catalog tracks upload and cleanup obligations, but also logical head and commit state, which overlaps with the Master's existing metadata and recovery responsibilities. In that separate PR, we should clarify ownership of these facts, what can extend the existing log/snapshot mechanisms, and how recovery handles discrepancies and unfinished operations. Object-specific persistent records may be necessary; the important question is how they fit into the overall recovery contract. We can start reviewing the I/O portion now, while agreeing on the catalog boundary before subsequent integration builds on it. There is no need to change the replica type solely to accommodate my earlier suggestion.

bsakaguchi commented 2026-10-29 21:22:38 +00:00

Author

Owner

Copy link

@hongzhi-gao commented on GitHub (Oct 26, 2026):

@hongzhi-gao Hi, thanks for tracing the existing paths and updating the proposal. I'd like to clarify my earlier suggestion: extending DFS was an option to explore, not a requirement to avoid a separate OBJECT_STORAGE type. I'm comfortable proceeding with the separate type.

My main concern is reusing common replica-management logic where the semantics match, so introducing another backend does not require parallel allocation, completion, revoke, lease, removal, and recovery flows across Put, BatchPut, and Upsert. Sharing the same enum would not, by itself, prevent that duplication either.

As we proceed, could you clarify:

• Whether the initial scope supports DFS and OSS in the same cluster, and whether one key can request a replica in each. This is a scope clarification, not a requirement to support both initially.
• Which management paths will remain shared, and which operations need object-specific handling.

I support the phased delivery plan. For #4727, could we keep the object-storage I/O portion in this PR and move the persistent catalog/lifecycle management into a separate PR for focused review?

The catalog tracks upload and cleanup obligations, but also logical head and commit state, which overlaps with the Master's existing metadata and recovery responsibilities. In that separate PR, we should clarify ownership of these facts, what can extend the existing log/snapshot mechanisms, and how recovery handles discrepancies and unfinished operations. Object-specific persistent records may be necessary; the important question is how they fit into the overall recovery contract.

We can start reviewing the I/O portion now, while agreeing on the catalog boundary before subsequent integration builds on it. There is no need to change the replica type solely to accommodate my earlier suggestion.
@fcczzz
Thanks for clarifying. I plan to retain a separate OBJECT_STORAGE type and reuse common replica-management logic where the semantics match. #4727 is now scoped to the I/O component; the integration details will be discussed separately.

<!-- gh-comment-id:6407545158 --> @hongzhi-gao commented on GitHub (Oct 26, 2026): > [@hongzhi-gao](https://github.com/hongzhi-gao) Hi, thanks for tracing the existing paths and updating the proposal. I'd like to clarify my earlier suggestion: extending DFS was an option to explore, not a requirement to avoid a separate OBJECT_STORAGE type. I'm comfortable proceeding with the separate type. > > My main concern is reusing common replica-management logic where the semantics match, so introducing another backend does not require parallel allocation, completion, revoke, lease, removal, and recovery flows across Put, BatchPut, and Upsert. Sharing the same enum would not, by itself, prevent that duplication either. > > As we proceed, could you clarify: > > * Whether the initial scope supports DFS and OSS in the same cluster, and whether one key can request a replica in each. This is a scope clarification, not a requirement to support both initially. > * Which management paths will remain shared, and which operations need object-specific handling. > > I support the phased delivery plan. For [#4727](https://github.com/kvcache-ai/Mooncake/pull/4727), could we keep the object-storage I/O portion in this PR and move the persistent catalog/lifecycle management into a separate PR for focused review? > > The catalog tracks upload and cleanup obligations, but also logical head and commit state, which overlaps with the Master's existing metadata and recovery responsibilities. In that separate PR, we should clarify ownership of these facts, what can extend the existing log/snapshot mechanisms, and how recovery handles discrepancies and unfinished operations. Object-specific persistent records may be necessary; the important question is how they fit into the overall recovery contract. > > We can start reviewing the I/O portion now, while agreeing on the catalog boundary before subsequent integration builds on it. There is no need to change the replica type solely to accommodate my earlier suggestion. @fcczzz Thanks for clarifying. I plan to retain a separate OBJECT_STORAGE type and reuse common replica-management logic where the semantics match. #4727 is now scoped to the I/O component; the integration details will be discussed separately.

bsakaguchi referenced this issue 2026-10-29 21:31:59 +00:00

[PR #1076] [MERGED] feat: add PCIe Relaxed Ordering (RO) support and RDMA traffic class (… #1995

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1076

No description provided.