[GH-ISSUE #1054] [RFC] Contribute Local Storage to Distributed Pool #347

New issue

Closed

opened 2026-10-29 21:13:07 +00:00 by bsakaguchi · 11 comments

bsakaguchi commented 2026-10-29 21:13:07 +00:00

Owner

Copy link

Originally created by @ykwd on GitHub (Nov 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1054

Changes proposed

This RFC proposes the design of a new feature: support contributing local SSD storage to the distributed store pool, which introduces a unified storage backend interface with multiple backends, file storage Get/Put/Eviction workflows, and master coordination.

Motivation

Currently, Mooncake relies heavily on in-memory storage for fast access. However, memory capacity is limited and expensive. Adding support for local SSD-based storage enables:

• Larger total storage capacity.
• Tiered data access
• Improved data persistence and recovery.
• Flexibility in resource allocation between heterogeneous clients.

Design Overview

1. Storage Backend: Unified Interface

All backends will implement a unified storage interface to simplify integration and extension. @zhangzuo21

We plan to support three backend implementations:

1. File-per-key Backend (Already supported)

   • Each key-value pair is stored in a separate file.
   • Simple but inefficient for large-scale data.

2. Bucket Backend @zhuxinjie-nz

   • Multiple key-value pairs are grouped into a single file (“bucket”).
   • Write and eviction are performed at the bucket level, improving I/O efficiency.

3. OffsetAllocator Backend

   • A large pre-allocated file acts as a storage arena.
   • Space is allocated and released using an OffsetAllocator.
   • Enables efficient space management and fewer file operations.

2. Get Workflow @zhuxinjie-nz

1. Client A issues a Get request.

2. Master returns the replica information.

3. Client A attempts to read from a memory replica first.

4. If only remote file replica exists (located on Client B), then:

   • A sends an RPC to B.
   • B reads the requested data from local SSD into its local buffer memory.
   • B returns the buffer address to A. B will ensure this address will not write other data before a certain time (e.g. 5s).
   • A performs an RDMA read to fetch the data directly.
   • If the read completes before timeout → A returns OK.
   • If it times out → A returns Error.

3. Put Workflow @zhuxinjie-nz

1. Client A issues a Put request.
2. Master assigns the target replica to Client B.
3. A performs an RDMA write to B.
4. Once the write succeeds, A sends a PutEnd notification to Master.
5. Upon receiving PutEnd, Master adds the key to B’s persistence queue.
6. B periodically requests pending persistence tasks from Master.
7. B obtains the persistence request, performs a BatchGetReplica to acquire the lease, and writes the data to local SSD.
8. Once successfully persisted, B notifies Master with another PutEnd message.

Load Balancing Considerations

Clients may have heterogeneous resource configurations:

• Client B: large memory, limited or no SSD.
• Client C: minimal memory, large SSD.

To optimize for this diversity, future versions will enhance Master’s scheduling logic to assign persistence tasks to the most suitable clients. The rest of the flow remains unchanged.

4. Eviction Workflow

1. Each client manages its local SSD storage usage.

2. When nearing capacity, the client initiates Eviction:

   • The client sends a Remove request to Master.
   • Upon successful confirmation, the client deletes the local data.

5. Initialization Workflow

Upon startup, each client:

• Reads local file metadata.
• Validates file integrity.
• Reports valid replicas back to Master via Put requests.

This ensures that valid persisted data is re-registered after restarts or failures.

6. Master Modifications

The Master component requires several extensions:

1. Associate SSD segment information to each client.
2. If a client has an SSD, associate it with a persistence queue.
3. The replica metadata structure will record ip address and replica size, which provides necessary information for clients to issue remote file read.

---

This feature is under active development. Suggestions and contributions are appreciated.

Related PR: https://github.com/kvcache-ai/Mooncake/pull/968 https://github.com/kvcache-ai/Mooncake/pull/1028 https://github.com/kvcache-ai/Mooncake/pull/1031

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @ykwd on GitHub (Nov 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1054 ### Changes proposed This RFC proposes the design of a new feature: support contributing local SSD storage to the distributed store pool, which introduces a unified storage backend interface with multiple backends, file storage Get/Put/Eviction workflows, and master coordination. ## Motivation Currently, Mooncake relies heavily on in-memory storage for fast access. However, memory capacity is limited and expensive. Adding support for **local SSD-based storage** enables: * Larger total storage capacity. * Tiered data access * Improved data persistence and recovery. * Flexibility in resource allocation between heterogeneous clients. ## Design Overview ### 1. Storage Backend: Unified Interface All backends will implement a unified storage interface to simplify integration and extension. @zhangzuo21 We plan to support three backend implementations: 1. **File-per-key Backend** (Already supported) * Each key-value pair is stored in a separate file. * Simple but inefficient for large-scale data. 2. **Bucket Backend** @zhuxinjie-nz * Multiple key-value pairs are grouped into a single file (“bucket”). * Write and eviction are performed at the bucket level, improving I/O efficiency. 3. **OffsetAllocator Backend** * A large pre-allocated file acts as a storage arena. * Space is allocated and released using an `OffsetAllocator`. * Enables efficient space management and fewer file operations. ### 2. Get Workflow @zhuxinjie-nz 1. **Client A** issues a `Get` request. 2. **Master** returns the replica information. 3. Client A attempts to read from a **memory replica** first. 4. If only **remote file replica** exists (located on Client B), then: * A sends an **RPC** to B. * B reads the requested data from **local SSD** into its **local buffer memory**. * B returns the **buffer address** to A. B will ensure this address will not write other data before a certain time (e.g. 5s). * A performs an **RDMA read** to fetch the data directly. * If the read completes before timeout → A returns **OK**. * If it times out → A returns **Error**. ### 3. Put Workflow @zhuxinjie-nz 1. **Client A** issues a `Put` request. 2. **Master** assigns the target replica to **Client B**. 3. A performs an **RDMA write** to B. 4. Once the write succeeds, A sends a **PutEnd** notification to Master. 5. Upon receiving `PutEnd`, Master adds the key to B’s **persistence queue**. 6. B periodically requests pending persistence tasks from Master. 7. B obtains the persistence request, performs a **BatchGetReplica** to acquire the **lease**, and writes the data to **local SSD**. 8. Once successfully persisted, B notifies Master with another **PutEnd** message. #### Load Balancing Considerations Clients may have heterogeneous resource configurations: * Client B: large memory, limited or no SSD. * Client C: minimal memory, large SSD. To optimize for this diversity, future versions will enhance Master’s scheduling logic to assign persistence tasks to the most suitable clients. The rest of the flow remains unchanged. ### 4. Eviction Workflow 1. Each client manages its **local SSD storage** usage. 2. When nearing capacity, the client initiates **Eviction**: * The client sends a `Remove` request to Master. * Upon successful confirmation, the client deletes the local data. ### 5. Initialization Workflow Upon startup, each client: * Reads **local file metadata**. * Validates file integrity. * Reports valid replicas back to Master via `Put` requests. This ensures that valid persisted data is re-registered after restarts or failures. ### 6. Master Modifications The Master component requires several extensions: 1. Associate **SSD segment information** to each client. 2. If a client has an SSD, associate it with a **persistence queue**. 3. The **replica metadata** structure will record ip address and replica size, which provides necessary information for clients to issue remote file read. --- This feature is under active development. Suggestions and contributions are appreciated. Related PR: https://github.com/kvcache-ai/Mooncake/pull/968 https://github.com/kvcache-ai/Mooncake/pull/1028 https://github.com/kvcache-ai/Mooncake/pull/1031 ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:13:07 +00:00

• closed this issue
• added the
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:13:08 +00:00

Author

Owner

Copy link

@maheshrbapatu commented on GitHub (Dec 1, 2025):

3. OffsetAllocator Backend

• A large pre-allocated file acts as a storage arena.
• Space is allocated and released using an OffsetAllocator.
• Enables efficient space management and fewer file operations.

Below is the document outlining my thoughts on tackling the OffsetAllocator backend.
It includes few key questions that should be addressed before moving forward with an implementation, along with a rough initial design proposal.
The document is open for comments and suggestions.

https://docs.google.com/document/d/1-pRFFAMcii8Hb1YmW0r0zTEd9xooP9nIC26ncct7Ub8/edit?tab=t.0

<!-- gh-comment-id:3596668533 --> @maheshrbapatu commented on GitHub (Dec 1, 2025): 3. OffsetAllocator Backend - A large pre-allocated file acts as a storage arena. - Space is allocated and released using an OffsetAllocator. - Enables efficient space management and fewer file operations. Below is the document outlining my thoughts on tackling the OffsetAllocator backend. It includes few key questions that should be addressed before moving forward with an implementation, along with a rough initial design proposal. The document is open for comments and suggestions. https://docs.google.com/document/d/1-pRFFAMcii8Hb1YmW0r0zTEd9xooP9nIC26ncct7Ub8/edit?tab=t.0

bsakaguchi commented 2026-10-29 21:13:08 +00:00

Author

Owner

Copy link

@maheshrbapatu commented on GitHub (Dec 21, 2025):

Hi Team,
https://github.com/kvcache-ai/Mooncake/issues/1054 - RFC
https://docs.google.com/document/d/1ImklK3hY0iYOQvi2Ar5-k0dZ2sl24OxsC9Klx1rIFTM/edit?tab=t.0
This design doc mainly focused on an offset-allocator–backed single-file KV cache to enable fine-grained deletion and space reuse.
While writing it, I realized the design has fundamental issues around loss of sequential writes and fragmentation over time, we can discuss and see if we want to go ahead with this. If yes I can implement it.
The doc also proposes an alternate direction that evolves the existing bucket model into a GC-aware, append-only design, preserving sequential IO while fixing the eviction and dead-space problems.
Details and trade-offs are in the doc — happy to discuss.

<!-- gh-comment-id:3679123327 --> @maheshrbapatu commented on GitHub (Dec 21, 2025): Hi Team, https://github.com/kvcache-ai/Mooncake/issues/1054 - RFC https://docs.google.com/document/d/1ImklK3hY0iYOQvi2Ar5-k0dZ2sl24OxsC9Klx1rIFTM/edit?tab=t.0 This design doc mainly focused on an offset-allocator–backed single-file KV cache to enable fine-grained deletion and space reuse. While writing it, I realized the design has fundamental issues around loss of sequential writes and fragmentation over time, we can discuss and see if we want to go ahead with this. If yes I can implement it. The doc also proposes an alternate direction that evolves the existing bucket model into a GC-aware, append-only design, preserving sequential IO while fixing the eviction and dead-space problems. Details and trade-offs are in the doc — happy to discuss.

bsakaguchi commented 2026-10-29 21:13:08 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Dec 23, 2025):

@maheshrbapatu This document is very detailed. Thank you!

Regarding the improvements to the bucket backend mentioned in it, I find the overall approach quite insightful. However, there seems to be a trade-off between write amplification and space utilization. If the GC threshold is 20%, it means that as the system runs over time, the long-term space utilization may drop to close to 20%. If the threshold is 80%, that would imply a 5× write amplification. Is there any way to improve this trade-off?

On metadata persistence, I completely agree that it makes sense to start with an in-memory solution only, and then add metadata persistence later once we have a more complete version. I’ll also think more about possible approaches for metadata persistence to see if there’s a good solution.

Regarding offset allocator fragmentation, I don’t think this is a concern. Mooncake Store already uses an offset allocator for memory management: each object corresponds to a contiguous memory region allocated by the offset allocator. The size of the data stored on SSD is the same as the size of the object in memory. If there is no fragmentation in memory, the data on SSD should also be fine, especially since memory is more precious than SSD.

As for the “grows the file endlessly or forces frequent compaction” concern, the offset allocator implements an automatic merge mechanism. When a region is freed, the allocator automatically merges it with adjacent free spaces on both sides. We have run a benchmark for reference:
https://kvcache-ai.github.io/Mooncake/performance/allocator-benchmark-result.html

Additionally, here is a slide I made that introduces the KVCache memory allocation issue in LLM inference, which may also be helpful:

<!-- gh-comment-id:3686630350 --> @ykwd commented on GitHub (Dec 23, 2025): @maheshrbapatu This document is very detailed. Thank you! Regarding the improvements to the bucket backend mentioned in it, I find the overall approach quite insightful. However, there seems to be a trade-off between write amplification and space utilization. If the GC threshold is 20%, it means that as the system runs over time, the long-term space utilization may drop to close to 20%. If the threshold is 80%, that would imply a 5× write amplification. Is there any way to improve this trade-off? On metadata persistence, I completely agree that it makes sense to start with an in-memory solution only, and then add metadata persistence later once we have a more complete version. I’ll also think more about possible approaches for metadata persistence to see if there’s a good solution. Regarding offset allocator fragmentation, I don’t think this is a concern. Mooncake Store already uses an offset allocator for memory management: each object corresponds to a contiguous memory region allocated by the offset allocator. The size of the data stored on SSD is the same as the size of the object in memory. If there is no fragmentation in memory, the data on SSD should also be fine, especially since memory is more precious than SSD. As for the “grows the file endlessly or forces frequent compaction” concern, the offset allocator implements an automatic merge mechanism. When a region is freed, the allocator automatically merges it with adjacent free spaces on both sides. We have run a benchmark for reference: [https://kvcache-ai.github.io/Mooncake/performance/allocator-benchmark-result.html](https://kvcache-ai.github.io/Mooncake/performance/allocator-benchmark-result.html) Additionally, here is a slide I made that introduces the KVCache memory allocation issue in LLM inference, which may also be helpful: <img width="490" height="237" alt="Image" src="https://github.com/user-attachments/assets/4f4a91e3-1c32-480c-86e0-92254af9167b" />

bsakaguchi commented 2026-10-29 21:13:08 +00:00

Author

Owner

Copy link

@maheshrbapatu commented on GitHub (Dec 24, 2025):

Thanks for the detailed feedback — this is very helpful and I agree with a lot of the points raised.

Clarifying the utilization baseline (90% quota already exists)

One important clarification that may help frame the GC discussion: today we already operate under a ~90% disk quota, even in the current FilePerKey backend. Eviction starts once disk usage crosses this quota, so steady-state operation is intentionally capped well below 100%.

In other words, we are not trying to squeeze the last byte out of disk capacity; we already keep headroom by design. From that perspective, the goal of the bucket/segment approach is not to maximize absolute utilization, but to:

maximize effective cache capacity within the quota band (e.g., stay around ~80–90% used), and

maintain predictable IO behavior under steady-state eviction + write pressure.

Seen this way, the concern that a GC “threshold” (e.g., 20%) would drive long-term utilization down toward that value doesn’t really apply if GC is driven by quota / pressure, rather than a fixed live-ratio rule.

Improving the WA vs utilization trade-off

I agree that a naive policy like “clean any bucket with live_ratio < T” creates an artificial-looking trade-off (e.g., T=20% vs T=80%). The intent isn’t to use a single global threshold, but to combine a few simple mechanisms that are commonly used in log-structured systems:

Quota / pressure-driven GC (watermarks)

Start GC only when usage approaches the quota (e.g., ~90%).

Stop GC once usage drops to a lower watermark (e.g., 80–85%).
This ensures GC runs only when capacity pressure exists, not continuously.

Cost/benefit-based victim selection

Instead of enforcing a hard live-ratio cutoff, select buckets that offer the best “bytes freed per bytes copied” (i.e., high garbage, low live data).

This naturally bounds write amplification without forcing utilization down.

(Optional) Hot/cold separation during GC

Rewriting stable (“cold”) KV entries repeatedly is what really drives up WA.

Promoting cold entries into dense buckets that are rarely re-cleaned significantly reduces rewrite churn, while hot buckets clean cheaply.

With these policies, long-run utilization can remain close to the existing quota target (near ~90%), while keeping write amplification manageable and avoiding pathological GC behavior.

On offset allocator fragmentation and merging

I agree that fragmentation in the sense of “can I allocate contiguous space?” may not be a major issue here, especially given:

uniform KV sizes in many inference scenarios, and

the allocator’s ability to merge adjacent free regions, as shown in the benchmark results.

The motivation for the bucket/segment approach is less about allocator correctness and more about long-term IO behavior under steady-state eviction, especially when operating close to quota. Append-only writes (both foreground writes and GC rewrites) help preserve predictable write patterns at high utilization, which is orthogonal to allocator fragmentation per se

Meanwhile, I am implementing the offsetAllocator backend and we can keep thinking about the above discussion.

<!-- gh-comment-id:3688358754 --> @maheshrbapatu commented on GitHub (Dec 24, 2025): Thanks for the detailed feedback — this is very helpful and I agree with a lot of the points raised. Clarifying the utilization baseline (90% quota already exists) One important clarification that may help frame the GC discussion: today we already operate under a ~90% disk quota, even in the current FilePerKey backend. Eviction starts once disk usage crosses this quota, so steady-state operation is intentionally capped well below 100%. In other words, we are not trying to squeeze the last byte out of disk capacity; we already keep headroom by design. From that perspective, the goal of the bucket/segment approach is not to maximize absolute utilization, but to: maximize effective cache capacity within the quota band (e.g., stay around ~80–90% used), and maintain predictable IO behavior under steady-state eviction + write pressure. Seen this way, the concern that a GC “threshold” (e.g., 20%) would drive long-term utilization down toward that value doesn’t really apply if GC is driven by quota / pressure, rather than a fixed live-ratio rule. Improving the WA vs utilization trade-off I agree that a naive policy like “clean any bucket with live_ratio < T” creates an artificial-looking trade-off (e.g., T=20% vs T=80%). The intent isn’t to use a single global threshold, but to combine a few simple mechanisms that are commonly used in log-structured systems: Quota / pressure-driven GC (watermarks) Start GC only when usage approaches the quota (e.g., ~90%). Stop GC once usage drops to a lower watermark (e.g., 80–85%). This ensures GC runs only when capacity pressure exists, not continuously. Cost/benefit-based victim selection Instead of enforcing a hard live-ratio cutoff, select buckets that offer the best “bytes freed per bytes copied” (i.e., high garbage, low live data). This naturally bounds write amplification without forcing utilization down. (Optional) Hot/cold separation during GC Rewriting stable (“cold”) KV entries repeatedly is what really drives up WA. Promoting cold entries into dense buckets that are rarely re-cleaned significantly reduces rewrite churn, while hot buckets clean cheaply. With these policies, long-run utilization can remain close to the existing quota target (near ~90%), while keeping write amplification manageable and avoiding pathological GC behavior. On offset allocator fragmentation and merging I agree that fragmentation in the sense of “can I allocate contiguous space?” may not be a major issue here, especially given: uniform KV sizes in many inference scenarios, and the allocator’s ability to merge adjacent free regions, as shown in the benchmark results. The motivation for the bucket/segment approach is less about allocator correctness and more about long-term IO behavior under steady-state eviction, especially when operating close to quota. Append-only writes (both foreground writes and GC rewrites) help preserve predictable write patterns at high utilization, which is orthogonal to allocator fragmentation per se Meanwhile, I am implementing the offsetAllocator backend and we can keep thinking about the above discussion.

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Dec 24, 2025):

@maheshrbapatu I agree that trying to squeeze every last bit of SSD capacity is not realistic. What we really need is to maintain a relatively high space utilization.

From an amortized analysis perspective, if the threshold is T (with 0 < T < 1) and deletions are purely random, then after the system has been running in steady state for some time, buckets—ordered by creation time from old to new—will have their space utilization roughly uniformly distributed between T and 1. On average, the utilization is (1 + T) / 2, and the write amplification is 1 / (1 - T).

Even if we adjust the timing of global GC, for example, “start GC only when usage approaches the quota (e.g., ~90%)”, this is essentially just adjusting T. The formulas above do not change.

The optimization of “promoting cold entries into dense buckets that are rarely re-cleaned, while hot buckets clean cheaply” is indeed effective. However, for an external storage scenario, I am concerned that most objects may not actually be very hot.

My personal thought is that the fundamental issue lies in the incompatibility between a bucket-based implementation and object-granularity eviction. Perhaps we could look at this from a different angle and avoid having buckets support object-granularity eviction altogether. Put differently, in the original proposal, when we evict an object, we mark it as deleted. At that point, it still occupies space but can no longer be read. We could instead change this so that when an object is selected for eviction, it is only marked as evictable, while still remaining readable.

In this way, all data in all buckets would remain usable. When space pressure requires deleting buckets, we could then apply the strategy you mentioned: select cold buckets to delete, and preserve the hot data within them via rewriting. Would this be a better approach?

<!-- gh-comment-id:3688661780 --> @ykwd commented on GitHub (Dec 24, 2025): @maheshrbapatu I agree that trying to squeeze every last bit of SSD capacity is not realistic. What we really need is to maintain a relatively high space utilization. From an amortized analysis perspective, if the threshold is T (with 0 < T < 1) and deletions are purely random, then after the system has been running in steady state for some time, buckets—ordered by creation time from old to new—will have their space utilization roughly uniformly distributed between T and 1. On average, the utilization is (1 + T) / 2, and the write amplification is 1 / (1 - T). Even if we adjust the timing of global GC, for example, “start GC only when usage approaches the quota (e.g., ~90%)”, this is essentially just adjusting T. The formulas above do not change. The optimization of “promoting cold entries into dense buckets that are rarely re-cleaned, while hot buckets clean cheaply” is indeed effective. However, for an external storage scenario, I am concerned that most objects may not actually be very hot. My personal thought is that the fundamental issue lies in the incompatibility between a bucket-based implementation and object-granularity eviction. Perhaps we could look at this from a different angle and avoid having buckets support object-granularity eviction altogether. Put differently, in the original proposal, when we evict an object, we mark it as deleted. At that point, it still occupies space but can no longer be read. We could instead change this so that when an object is selected for eviction, it is only marked as *evictable*, while still remaining readable. In this way, all data in all buckets would remain usable. When space pressure requires deleting buckets, we could then apply the strategy you mentioned: select cold buckets to delete, and preserve the hot data within them via rewriting. Would this be a better approach?

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@MMuzzammil1 commented on GitHub (Dec 24, 2025):

3. **OffsetAllocatorBackend&*: I'm curious about the benefit of this backend over BucketBackend. I guess Mooncake Store is primarily intended as a KVCache store, no? Assuming this and also assuming the FIFO eviction order introduced in this PR: https://github.com/kvcache-ai/Mooncake/pull/1028, the Bucket based backend would be simpler (with the only complexity of finding an optimal bucket size for caching, such that eviction would be "bucket level").

As for the problems of the current design listed by @maheshrbapatu in the Google doc, I agree with the second point of "Recovery requiring scanning all the metadata files", but don't understand the internal fragmentation part. Could anybody help me with it?

<!-- gh-comment-id:3689213605 --> @MMuzzammil1 commented on GitHub (Dec 24, 2025): 3. **OffsetAllocatorBackend&*: I'm curious about the benefit of this backend over **BucketBackend**. I guess Mooncake Store is primarily intended as a KVCache store, no? Assuming this and also assuming the *FIFO* eviction order introduced in this PR: https://github.com/kvcache-ai/Mooncake/pull/1028, the Bucket based backend would be simpler (with the only complexity of finding an optimal bucket size for caching, such that eviction would be "bucket level"). As for the problems of the current design listed by @maheshrbapatu in the [Google doc](https://docs.google.com/document/d/1ImklK3hY0iYOQvi2Ar5-k0dZ2sl24OxsC9Klx1rIFTM/edit?tab=t.0#heading=h.g5dfohxhhce0), I agree with the second point of "Recovery requiring scanning all the metadata files", but don't understand the internal fragmentation part. Could anybody help me with it?

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@MMuzzammil1 commented on GitHub (Jan 6, 2026):

Another question is the efficacy of the SSD offloading for KVCache. Do we have any data on if SSD based offloading is effective in reducing TTFT? Because usually R/W speed for SSDs is much slower compared to DRAM based approaches.

<!-- gh-comment-id:3713627942 --> @MMuzzammil1 commented on GitHub (Jan 6, 2026): Another question is the efficacy of the SSD offloading for KVCache. Do we have any data on if SSD based offloading is effective in reducing TTFT? Because usually R/W speed for SSDs is much slower compared to DRAM based approaches.

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Jan 6, 2026):

Another question is the efficacy of the SSD offloading for KVCache. Do we have any data on if SSD based offloading is effective in reducing TTFT? Because usually R/W speed for SSDs is much slower compared to DRAM based approaches.

@zhuxinjie-nz had an end2end experiment using sglang hicache, using no memory segment in Mooncake. With SSD approach, the ttft reduces to 27% compared to recomputing (no cache hit).

We also plan to make a benchmark with our open-sourced implementation when this feature is ready.

<!-- gh-comment-id:3713678296 --> @ykwd commented on GitHub (Jan 6, 2026): > Another question is the efficacy of the SSD offloading for KVCache. Do we have any data on if SSD based offloading is effective in reducing TTFT? Because usually R/W speed for SSDs is much slower compared to DRAM based approaches. @zhuxinjie-nz had an end2end experiment using sglang hicache, using no memory segment in Mooncake. With SSD approach, the ttft reduces to 27% compared to recomputing (no cache hit). We also plan to make a benchmark with our open-sourced implementation when this feature is ready.

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@ykwd commented on GitHub (Jan 6, 2026):

As for the problems of the current design listed by @maheshrbapatu in the Google doc, I agree with the second point of "Recovery requiring scanning all the metadata files", but don't understand the internal fragmentation part. Could anybody help me with it?

For the offset allocator approach, it starts with a single large contiguous free storage region. As storage space is continuously allocated and freed, the available space can become fragmented. However, as discussed in https://github.com/kvcache-ai/Mooncake/issues/1054#issuecomment-3686630350, the offset allocator has built-in algorithms to automatically merge adjacent free fragments. Based on testing, the overall space utilization remains relatively high.

<!-- gh-comment-id:3713687354 --> @ykwd commented on GitHub (Jan 6, 2026): > As for the problems of the current design listed by [@maheshrbapatu](https://github.com/maheshrbapatu) in the [Google doc](https://docs.google.com/document/d/1ImklK3hY0iYOQvi2Ar5-k0dZ2sl24OxsC9Klx1rIFTM/edit?tab=t.0#heading=h.g5dfohxhhce0), I agree with the second point of "Recovery requiring scanning all the metadata files", but don't understand the internal fragmentation part. Could anybody help me with it? For the offset allocator approach, it starts with a single large contiguous free storage region. As storage space is continuously allocated and freed, the available space can become fragmented. However, as discussed in [https://github.com/kvcache-ai/Mooncake/issues/1054#issuecomment-3686630350](https://github.com/kvcache-ai/Mooncake/issues/1054#issuecomment-3686630350), the offset allocator has built-in algorithms to automatically merge adjacent free fragments. Based on testing, the overall space utilization remains relatively high.

bsakaguchi commented 2026-10-29 21:13:09 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 19, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4748522530 --> @github-actions[bot] commented on GitHub (Jun 19, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:13:10 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jun 26, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4806400326 --> @github-actions[bot] commented on GitHub (Jun 26, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:27:58 +00:00

[PR #347] [CLOSED] fix(store): Return error put while OBJECT_ALREADY_EXISTS #1499

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#347

No description provided.