[GH-ISSUE #2219] [Feature Request]: Disk Watermark Eviction: Proactive Disk Usage Management #610

New issue

Closed

opened 2026-10-29 21:17:10 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:17:10 +00:00

Owner

Originally created by @Colors-111 on GitHub (May 25, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2219

Describe your feature request

Problem

Disk eviction is currently triggered only reactively — when a write finds insufficient space via EnsureDiskSpace. There is no proactive watermark-based eviction mechanism. This causes disk usage to remain at ~100% after the disk fills up, unlike memory which oscillates between watermark thresholds.

Specific impacts:

No usage oscillation: Once full, disk usage stays at 100%. Memory eviction uses eviction_high_watermark_ratio (default 0.95) with a background eviction thread, keeping usage fluctuating between ~90%–95%. Disk has no equivalent mechanism.
Write latency spikes: When the disk is full, every write must first evict then write sequentially, causing latency spikes.
Existing TODO: file_storage.cpp:597 marks // TODO(eviction): Implement an LRU eviction mechanism to manage local storage capacity., which remains unimplemented.

Current vs Expected Behavior

Memory watermarks (existing):      Disk usage (current):
95% ──── high watermark            100% ──────────── stays at 100% after filling
  |  ↗  ↘  ↗  ↘                     |
90% ──── oscillation range           0% ──────────── no watermark control

Disk watermarks (expected):
90% ──── high watermark
  |  ↗  ↘  ↗  ↘
80% ──── low watermark

Suggested Approach

Core Idea

Add disk watermark checks in the FileStorage::Heartbeat() periodic thread. When usage exceeds the high watermark, proactively evict until usage drops below the low watermark. Reuse the existing EnsureDiskSpace() / EvictFile() logic in StorageBackend — no need to rewrite eviction strategies.

Configuration

Add watermark config fields to FileStorageConfig:

Field	Type	Default	Description
`disk_eviction_high_watermark_ratio`	`double`	`0.90`	Disk usage ratio that triggers proactive eviction
`disk_eviction_low_watermark_ratio`	`double`	`0.80`	Target usage ratio to evict down to

Support environment variable overrides (MOONCAKE_DISK_EVICTION_HIGH_WATERMARK_RATIO / MOONCAKE_DISK_EVICTION_LOW_WATERMARK_RATIO).

Eviction Trigger Point

Add a new STEP 3 in Heartbeat() after the existing STEP 2 (persist offloaded objects):

// STEP 3: Disk watermark eviction
if (disk_eviction_enabled) {
    double used_ratio = used_space_ / total_space_;
    if (used_ratio > disk_eviction_high_watermark_ratio) {
        size_t target_used = total_space_ * disk_eviction_low_watermark_ratio;
        size_t space_to_free = used_space_ - target_used;
        storage_backend_->EnsureDiskSpace(space_to_free);
        // Notify master of evicted keys so it updates metadata
    }
}

Points to Address

Interface extension: StorageBackendInterface needs a GetSpaceUsage() method for FileStorage to query current disk usage.
Master notification: EnsureDiskSpace already returns evicted_keys, but these are not currently reported to the master. After watermark eviction, the master must be notified to clean up LOCAL_DISK replica info for evicted objects — otherwise the master still believes the data resides on SSD, and subsequent reads will fail.
Backend adaptation: FilePerKey and Bucket backends can reuse existing eviction logic. The OffsetAllocator backend currently does not support eviction; watermark mechanism should not cover it initially.
Thread safety: Watermark eviction (Heartbeat thread) and write-time reactive eviction may run concurrently. EnsureDiskSpace must remain thread-safe (currently protected by file_queue_mutex_ and space_mutex_).
Heartbeat interval: Defaults to 10s, which limits watermark check frequency. Reusing the heartbeat thread is sufficient initially; a dedicated eviction thread can be considered later if needed.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @Colors-111 on GitHub (May 25, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2219 ### Describe your feature request ## Problem Disk eviction is currently triggered only reactively — when a write finds insufficient space via `EnsureDiskSpace`. There is no proactive watermark-based eviction mechanism. This causes disk usage to remain at ~100% after the disk fills up, unlike memory which oscillates between watermark thresholds. Specific impacts: 1. **No usage oscillation**: Once full, disk usage stays at 100%. Memory eviction uses `eviction_high_watermark_ratio` (default 0.95) with a background eviction thread, keeping usage fluctuating between ~90%–95%. Disk has no equivalent mechanism. 2. **Write latency spikes**: When the disk is full, every write must first evict then write sequentially, causing latency spikes. 3. **Existing TODO**: `file_storage.cpp:597` marks `// TODO(eviction): Implement an LRU eviction mechanism to manage local storage capacity.`, which remains unimplemented. ## Current vs Expected Behavior ``` Memory watermarks (existing): Disk usage (current): 95% ──── high watermark 100% ──────────── stays at 100% after filling | ↗ ↘ ↗ ↘ | 90% ──── oscillation range 0% ──────────── no watermark control Disk watermarks (expected): 90% ──── high watermark | ↗ ↘ ↗ ↘ 80% ──── low watermark ``` ## Suggested Approach ### Core Idea Add disk watermark checks in the `FileStorage::Heartbeat()` periodic thread. When usage exceeds the high watermark, proactively evict until usage drops below the low watermark. Reuse the existing `EnsureDiskSpace()` / `EvictFile()` logic in `StorageBackend` — no need to rewrite eviction strategies. ### Configuration Add watermark config fields to `FileStorageConfig`: | Field | Type | Default | Description | |-------|------|---------|-------------| | `disk_eviction_high_watermark_ratio` | `double` | `0.90` | Disk usage ratio that triggers proactive eviction | | `disk_eviction_low_watermark_ratio` | `double` | `0.80` | Target usage ratio to evict down to | Support environment variable overrides (`MOONCAKE_DISK_EVICTION_HIGH_WATERMARK_RATIO` / `MOONCAKE_DISK_EVICTION_LOW_WATERMARK_RATIO`). ### Eviction Trigger Point Add a new STEP 3 in `Heartbeat()` after the existing STEP 2 (persist offloaded objects): ``` // STEP 3: Disk watermark eviction if (disk_eviction_enabled) { double used_ratio = used_space_ / total_space_; if (used_ratio > disk_eviction_high_watermark_ratio) { size_t target_used = total_space_ * disk_eviction_low_watermark_ratio; size_t space_to_free = used_space_ - target_used; storage_backend_->EnsureDiskSpace(space_to_free); // Notify master of evicted keys so it updates metadata } } ``` ### Points to Address - **Interface extension**: `StorageBackendInterface` needs a `GetSpaceUsage()` method for `FileStorage` to query current disk usage. - **Master notification**: `EnsureDiskSpace` already returns `evicted_keys`, but these are not currently reported to the master. After watermark eviction, the master must be notified to clean up LOCAL_DISK replica info for evicted objects — otherwise the master still believes the data resides on SSD, and subsequent reads will fail. - **Backend adaptation**: FilePerKey and Bucket backends can reuse existing eviction logic. The OffsetAllocator backend currently does not support eviction; watermark mechanism should not cover it initially. - **Thread safety**: Watermark eviction (Heartbeat thread) and write-time reactive eviction may run concurrently. `EnsureDiskSpace` must remain thread-safe (currently protected by `file_queue_mutex_` and `space_mutex_`). - **Heartbeat interval**: Defaults to 10s, which limits watermark check frequency. Reusing the heartbeat thread is sufficient initially; a dedicated eviction thread can be considered later if needed. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:17:10 +00:00

bsakaguchi commented

2026-10-29 21:17:11 +00:00

Author

Owner

@CAICAIIs commented on GitHub (May 28, 2026):

hi, I’d like to take this one.

I’m planning to implement the first slice as proactive disk watermark eviction in FileStorage::Heartbeat(), covering file_per_key_storage_backend and bucket_storage_backend, and keeping offset_allocator_storage_backend as no-op for the initial version.

I’ll also include config/env docs and tests for the watermark behavior and master-side LOCAL_DISK replica cleanup.

@CAICAIIs commented on GitHub (May 28, 2026): hi, I’d like to take this one. I’m planning to implement the first slice as proactive disk watermark eviction in `FileStorage::Heartbeat()`, covering `file_per_key_storage_backend` and `bucket_storage_backend`, and keeping `offset_allocator_storage_backend` as no-op for the initial version. I’ll also include config/env docs and tests for the watermark behavior and master-side LOCAL_DISK replica cleanup.

bsakaguchi commented

2026-10-29 21:17:11 +00:00

Author

Owner

@Colors-111 commented on GitHub (May 28, 2026):

hi, I’d like to take this one.

I’m planning to implement the first slice as proactive disk watermark eviction in FileStorage::Heartbeat(), covering file_per_key_storage_backend and bucket_storage_backend, and keeping offset_allocator_storage_backend as no-op for the initial version.

I’ll also include config/env docs and tests for the watermark behavior and master-side LOCAL_DISK replica cleanup.

Thanks for driving this. The iterative approach makes a lot of sense. Looking forward to the PR.

@Colors-111 commented on GitHub (May 28, 2026): > hi, I’d like to take this one. > > I’m planning to implement the first slice as proactive disk watermark eviction in `FileStorage::Heartbeat()`, covering `file_per_key_storage_backend` and `bucket_storage_backend`, and keeping `offset_allocator_storage_backend` as no-op for the initial version. > > I’ll also include config/env docs and tests for the watermark behavior and master-side LOCAL_DISK replica cleanup. Thanks for driving this. The iterative approach makes a lot of sense. Looking forward to the PR.

bsakaguchi referenced this issue

2026-10-29 21:29:31 +00:00

[PR #610] [MERGED] [Store]feat: Add 3fs native api plugin for KVCache storage persistence #1688

bsakaguchi referenced this issue

2026-10-29 21:30:00 +00:00

[PR #690] [MERGED] [Store]feat: Migrate Persistence Metadata from Client to Master Service #1747