kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1990] [RFC]: Native SSD KV Storage Backend for Mooncake Store #552

New issue

Closed

opened 2026-10-29 21:16:17 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:16:17 +00:00

Owner

Originally created by @kangyifei on GitHub (Apr 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1990

1. Introduction

We would like to propose integrating SSD KV Engine into Mooncake Store as a new SSD storage backend. SSD KV Engine is a high-performance, SSD-optimized single-node KV engine specifically designed for KV Cache offload workloads. It will be released as an independent open-source repository and jointly compiled with Mooncake Store as a static library.

The integration is implemented as SsdKvStorageBackend, a concrete implementation of Mooncake's StorageBackendInterface. It plugs seamlessly into the existing FileStorage / FileBackend pipeline, requiring no changes to upper-layer application code.

The goal is to give Mooncake Store a low-cost, large-capacity SSD tier that complements DRAM-based storage without introducing remote target mounting or mixed-layer placement complexity.

The initial implementation focuses on single-node KV Cache offload. Cluster-wide placement, replication, and rebalancing remain the responsibility of Mooncake Store rather than the engine itself.

2. Background and Motivation

As long-context scenarios proliferate — multi-turn conversation, coding assistants, document understanding — the size of KV Caches continues to expand. DRAM is both expensive and capacity-constrained, making it insufficient to satisfy the demands of large-scale KV Cache storage.

SSD offers an effective cost/capacity tradeoff. However, Mooncake's existing storage backends (FilePerKeyBackend, BucketBackend, OffsetAllocatorBackend) all operate through the filesystem layer — each offload/load translates into file creation, open, write/read, and close syscalls. This per-file overhead becomes a bottleneck when KV Cache workloads demand high-frequency, small-granularity I/O. Furthermore, filesystem-based backends provide no built-in persistence semantics beyond what the OS page cache offers, requiring external mechanisms to guarantee data durability.

Meanwhile, existing community SSD approaches introduce additional operational friction:

Some SSD I/O frameworks (e.g., SPDK-based solutions) rely on CPU core affinity binding, competing with inference engines for cores and reducing deployment flexibility.
Some approaches require manual discovery and mounting of remote storage targets, increasing operational complexity.
Some approaches manage SSD and memory in the same SegmentManager, making replica placement and access latency unpredictable.

SSD KV Engine addresses all of the above:

Native KV interface, low overhead. The engine provides direct Put / Get / Del batch APIs that bypass the filesystem entirely. Data is written to and read from the block device through an optimized I/O path, eliminating per-file syscall overhead and delivering significantly higher throughput than filesystem-based backends.
Built-in persistence. All writes are persisted to SSD with metadata checksums. On restart, the engine recovers its index from on-device metadata — no external WAL, journaling filesystem, or page cache flush is needed.
LRU eviction. When storage approaches capacity, stale items are automatically reclaimed via LRU, transparent to the caller.
Independent open-source library. SSD KV Engine evolves independently in its own repository, reducing maintenance burden on the Mooncake main tree.
Simple deployment. Linked as a static library; no extra daemon process, no remote target to mount, no external control-plane dependency.

3. Technical Design

3.1 Architecture Overview

The integration follows Mooncake's existing storage backend abstraction. SsdKvStorageBackend sits behind StorageBackendInterface and forwards operations to SSD KV Engine.

flowchart LR
    A[Application / FileStorage] --> B[StorageBackendInterface]
    B --> C[SsdKvStorageBackend]
    C --> D[SSD KV Engine]
    D --> E[SSD / NVMe Device]

SSD KV Engine is both a standalone application and a linkable library. When built with Mooncake, it is compiled as a static library and linked directly — there is no runtime subprocess or network hop between SsdKvStorageBackend and the engine.

3.2 SSD KV Engine Interface

SSD KV Engine exposes the following interfaces to upper layers:

API	Description
`batch_put(batch_ctx, batch_size, iter_fn, cpl_fn)`	Batch write — persist multiple KV pairs to SSD
`batch_get(batch_ctx, batch_size, iter_fn, cpl_fn)`	Batch read — retrieve multiple values by key
`batch_del(batch_ctx, batch_size, iter_fn, cpl_fn)`	Batch delete — remove multiple KV pairs
`batch_scan(scan_ctx, iter_fn)`	Scan — iterate over all stored keys, invoking `iter_fn` for each entry
`count()` / `size()` / `capacity()`	Query current item count, used size, and total capacity

All batch APIs follow an iterator / completion-callback model:

iter_fn(batch_ctx, index, &key, &value, &value_size) — called by the engine to pull the next item from the caller's buffer.
cpl_fn(batch_ctx, index, status) — called by the engine when each item's I/O completes.

This design decouples data supply from result handling, allowing SsdKvStorageBackend to feed slices and collect results asynchronously.

3.3 Key and Value Flexibility

SSD KV Engine accepts variable-length keys and variable-length values. Internally, the engine hashes incoming keys into fixed-width representations for O(1) index lookup, so callers can pass arbitrary-length string paths directly without external mapping. Values of any size are handled natively by the engine.

3.4 StorageBackendInterface Integration

SsdKvStorageBackend implements StorageBackendInterface to bridge Mooncake's FileStorage pipeline with SSD KV Engine:

Init() initializes the engine runtime and opens the underlying block device.
BatchOffload() passes string paths and data directly to engine batch_put; the engine handles key hashing internally.
BatchLoad() passes paths to engine batch_get and fills the output buffers.
IsExist() checks whether a key exists in the engine.
IsEnableOffloading() always returns true — the engine manages capacity internally via LRU eviction, so it is always ready to accept new writes.
ScanMeta() delegates to the engine's built-in key enumeration interface and invokes the provided callback for each stored entry.

Backend registration:

A new enum value StorageBackendType::kSsdKv is added.
CreateStorageBackend() factory instantiates SsdKvStorageBackend when the type matches.
The backend is selected via environment variable: MOONCAKE_STORAGE_BACKEND_TYPE=ssd_kv_storage_backend.

3.5 Build Integration

CMake option USE_SSD_KV_BACKEND guards all SSD KV Engine code. When disabled, zero additional compilation or runtime overhead is added to Mooncake.
The path to SSD KV Engine's source tree is supplied via SSD_KV_ENGINE_PATH.
SSD KV Engine is compiled as a static library and linked into the Mooncake build. No separate process or daemon is required at runtime.

4. Compatibility

Interface compatibility. SsdKvStorageBackend implements StorageBackendInterface in full. All callers that use FileStorage continue to work without modification.
Non-invasive. Existing backends — BucketBackend, FilePerKeyBackend, OffsetAllocatorBackend — are completely unaffected.
Zero-cost when disabled. When USE_SSD_KV_BACKEND is off, no symbols, headers, or link dependencies from SSD KV Engine are included.
Runtime switching. The backend is selected via a single environment variable, enabling A/B comparison with other backends at deployment time without recompilation.

5. Alternatives Considered

SSD pool over NVMe-oF

This approach expands KV Cache capacity using remote NVMe-oF targets. However, it requires users to discover and mount remote NVMe-oF targets manually before the service starts, and the underlying SPDK-based I/O framework relies on CPU core affinity, which competes with inference engines for cores. In contrast, SSD KV Engine is built on standard pthreads without any core pinning, coexisting naturally with inference engines. Operational complexity is higher than a locally-linked library approach.

Standards-aligned NVMe KV backend

This design introduces a full backend + connector + executor three-layer stack with a dedicated control-plane service for device state and cleanup orchestration. While architecturally clean, the added complexity and the control-plane dependency raise the barrier for single-node or low-ops deployments. The SSD KV approach achieves the same SSD offload goal with a simpler integration path.

Mixed SegmentManager for SSD and memory

Managing SSD and DRAM segments in the same SegmentManager makes replica placement and access latency non-deterministic — callers cannot reliably predict whether a given read will hit DRAM or SSD. Keeping SSD as a dedicated StorageBackend behind FileStorage preserves explicit tiering semantics.

SSD KV Engine takes the simplest path that is still correct and efficient: a self-contained static library with a single-file integration point via StorageBackendInterface. We believe this is a natural addition to the Mooncake storage backend ecosystem and look forward to community feedback.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @kangyifei on GitHub (Apr 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1990 ## 1. Introduction We would like to propose integrating **SSD KV Engine** into Mooncake Store as a new SSD storage backend. SSD KV Engine is a high-performance, SSD-optimized single-node KV engine specifically designed for KV Cache offload workloads. It will be released as an independent open-source repository and jointly compiled with Mooncake Store as a static library. The integration is implemented as `SsdKvStorageBackend`, a concrete implementation of Mooncake's `StorageBackendInterface`. It plugs seamlessly into the existing `FileStorage` / `FileBackend` pipeline, requiring no changes to upper-layer application code. The goal is to give Mooncake Store a low-cost, large-capacity SSD tier that complements DRAM-based storage without introducing remote target mounting or mixed-layer placement complexity. The initial implementation focuses on single-node KV Cache offload. Cluster-wide placement, replication, and rebalancing remain the responsibility of Mooncake Store rather than the engine itself. ## 2. Background and Motivation As long-context scenarios proliferate — multi-turn conversation, coding assistants, document understanding — the size of KV Caches continues to expand. DRAM is both expensive and capacity-constrained, making it insufficient to satisfy the demands of large-scale KV Cache storage. SSD offers an effective cost/capacity tradeoff. However, Mooncake's existing storage backends (`FilePerKeyBackend`, `BucketBackend`, `OffsetAllocatorBackend`) all operate through the filesystem layer — each offload/load translates into file creation, open, write/read, and close syscalls. This per-file overhead becomes a bottleneck when KV Cache workloads demand high-frequency, small-granularity I/O. Furthermore, filesystem-based backends provide no built-in persistence semantics beyond what the OS page cache offers, requiring external mechanisms to guarantee data durability. Meanwhile, existing community SSD approaches introduce additional operational friction: - Some SSD I/O frameworks (e.g., SPDK-based solutions) rely on **CPU core affinity binding**, competing with inference engines for cores and reducing deployment flexibility. - Some approaches require **manual discovery and mounting of remote storage targets**, increasing operational complexity. - Some approaches manage SSD and memory in the **same SegmentManager**, making replica placement and access latency unpredictable. SSD KV Engine addresses all of the above: - **Native KV interface, low overhead.** The engine provides direct `Put` / `Get` / `Del` batch APIs that bypass the filesystem entirely. Data is written to and read from the block device through an optimized I/O path, eliminating per-file syscall overhead and delivering significantly higher throughput than filesystem-based backends. - **Built-in persistence.** All writes are persisted to SSD with metadata checksums. On restart, the engine recovers its index from on-device metadata — no external WAL, journaling filesystem, or page cache flush is needed. - **LRU eviction.** When storage approaches capacity, stale items are automatically reclaimed via LRU, transparent to the caller. - **Independent open-source library.** SSD KV Engine evolves independently in its own repository, reducing maintenance burden on the Mooncake main tree. - **Simple deployment.** Linked as a static library; no extra daemon process, no remote target to mount, no external control-plane dependency. ## 3. Technical Design ### 3.1 Architecture Overview The integration follows Mooncake's existing storage backend abstraction. `SsdKvStorageBackend` sits behind `StorageBackendInterface` and forwards operations to SSD KV Engine. ```mermaid flowchart LR A[Application / FileStorage] --> B[StorageBackendInterface] B --> C[SsdKvStorageBackend] C --> D[SSD KV Engine] D --> E[SSD / NVMe Device] ``` SSD KV Engine is both a standalone application and a linkable library. When built with Mooncake, it is compiled as a static library and linked directly — there is no runtime subprocess or network hop between `SsdKvStorageBackend` and the engine. ### 3.2 SSD KV Engine Interface SSD KV Engine exposes the following interfaces to upper layers: | API | Description | |---|---| | `batch_put(batch_ctx, batch_size, iter_fn, cpl_fn)` | Batch write — persist multiple KV pairs to SSD | | `batch_get(batch_ctx, batch_size, iter_fn, cpl_fn)` | Batch read — retrieve multiple values by key | | `batch_del(batch_ctx, batch_size, iter_fn, cpl_fn)` | Batch delete — remove multiple KV pairs | | `batch_scan(scan_ctx, iter_fn)` | Scan — iterate over all stored keys, invoking `iter_fn` for each entry | | `count()` / `size()` / `capacity()` | Query current item count, used size, and total capacity | All batch APIs follow an **iterator / completion-callback** model: - **`iter_fn(batch_ctx, index, &key, &value, &value_size)`** — called by the engine to pull the next item from the caller's buffer. - **`cpl_fn(batch_ctx, index, status)`** — called by the engine when each item's I/O completes. This design decouples data supply from result handling, allowing `SsdKvStorageBackend` to feed slices and collect results asynchronously. ### 3.3 Key and Value Flexibility SSD KV Engine accepts **variable-length keys** and **variable-length values**. Internally, the engine hashes incoming keys into fixed-width representations for O(1) index lookup, so callers can pass arbitrary-length string paths directly without external mapping. Values of any size are handled natively by the engine. ### 3.4 StorageBackendInterface Integration `SsdKvStorageBackend` implements `StorageBackendInterface` to bridge Mooncake's FileStorage pipeline with SSD KV Engine: - **`Init()`** initializes the engine runtime and opens the underlying block device. - **`BatchOffload()`** passes string paths and data directly to engine `batch_put`; the engine handles key hashing internally. - **`BatchLoad()`** passes paths to engine `batch_get` and fills the output buffers. - **`IsExist()`** checks whether a key exists in the engine. - **`IsEnableOffloading()`** always returns `true` — the engine manages capacity internally via LRU eviction, so it is always ready to accept new writes. - **`ScanMeta()`** delegates to the engine's built-in key enumeration interface and invokes the provided callback for each stored entry. **Backend registration:** - A new enum value `StorageBackendType::kSsdKv` is added. - `CreateStorageBackend()` factory instantiates `SsdKvStorageBackend` when the type matches. - The backend is selected via environment variable: `MOONCAKE_STORAGE_BACKEND_TYPE=ssd_kv_storage_backend`. ### 3.5 Build Integration - CMake option `USE_SSD_KV_BACKEND` guards all SSD KV Engine code. When disabled, zero additional compilation or runtime overhead is added to Mooncake. - The path to SSD KV Engine's source tree is supplied via `SSD_KV_ENGINE_PATH`. - SSD KV Engine is compiled as a static library and linked into the Mooncake build. No separate process or daemon is required at runtime. ## 4. Compatibility - **Interface compatibility.** `SsdKvStorageBackend` implements `StorageBackendInterface` in full. All callers that use `FileStorage` continue to work without modification. - **Non-invasive.** Existing backends — `BucketBackend`, `FilePerKeyBackend`, `OffsetAllocatorBackend` — are completely unaffected. - **Zero-cost when disabled.** When `USE_SSD_KV_BACKEND` is off, no symbols, headers, or link dependencies from SSD KV Engine are included. - **Runtime switching.** The backend is selected via a single environment variable, enabling A/B comparison with other backends at deployment time without recompilation. ## 5. Alternatives Considered ### SSD pool over NVMe-oF This approach expands KV Cache capacity using remote NVMe-oF targets. However, it requires users to discover and mount remote NVMe-oF targets manually before the service starts, and the underlying SPDK-based I/O framework relies on CPU core affinity, which competes with inference engines for cores. In contrast, SSD KV Engine is built on standard pthreads without any core pinning, coexisting naturally with inference engines. Operational complexity is higher than a locally-linked library approach. ### Standards-aligned NVMe KV backend This design introduces a full backend + connector + executor three-layer stack with a dedicated control-plane service for device state and cleanup orchestration. While architecturally clean, the added complexity and the control-plane dependency raise the barrier for single-node or low-ops deployments. The SSD KV approach achieves the same SSD offload goal with a simpler integration path. ### Mixed SegmentManager for SSD and memory Managing SSD and DRAM segments in the same `SegmentManager` makes replica placement and access latency non-deterministic — callers cannot reliably predict whether a given read will hit DRAM or SSD. Keeping SSD as a dedicated `StorageBackend` behind `FileStorage` preserves explicit tiering semantics. --- SSD KV Engine takes the simplest path that is still correct and efficient: a self-contained static library with a single-file integration point via `StorageBackendInterface`. We believe this is a natural addition to the Mooncake storage backend ecosystem and look forward to community feedback. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:16:17 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:16:18 +00:00

Author

Owner

@stmatengss commented on GitHub (Apr 29, 2026):

Please consider this RFC: #1957

@stmatengss commented on GitHub (Apr 29, 2026): Please consider this RFC: #1957

bsakaguchi commented

2026-10-29 21:16:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 29, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 29, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:16:19 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 6, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Aug 6, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:29:10 +00:00

[PR #552] [MERGED] [TransferEngine] Fix side effect of wild location registration #1645