kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4561] [RFC]: [Store] NoF Storage Device Control Plane (Metadata Model, Health State Machine, and Admin Observability) #1163

New issue

Open

opened 2026-10-29 21:23:30 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:30 +00:00

Owner

Originally created by @zxpdemonio on GitHub (Oct 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4561

Changes proposed

Summary

Mooncake Store supports offloading objects to NVMe-oF (NoF) SSD targets and local SSDs. While the master server manages segments via NoFSegmentManager and periodically executes heartbeat probes against mounted targets, it previously managed storage primarily at the coarse segment/allocator level without a formalized storage device control plane.

Storage device health lifecycles, real capacity utilization across active allocators, degradation detection, and maintenance planning were unexposed to cluster operators and automated maintenance pipelines.

This RFC proposes a unified, lightweight NoF Storage Device Control Plane for Mooncake Store:

Device Metadata & Health State Machine: Formalized StorageDeviceHealth states with a pure-function derivation model and automated maintenance planning (recovery vs. GC candidacy).
Master & Segment Engine Integration: Device-level capacity and allocated-bytes accounting via NoFSegmentManager::GetSegmentUsages(), coupled with on-demand operator probe triggering.
Admin Observability & Operations Endpoints: REST APIs (/api/v1/devices, /api/v1/devices/maintenance_plan, /api/v1/devices/probe) for integration with Kubernetes operators, cluster dashboards, and automated node evacuations.

An implementation of Phase 1 is already available in PR #4560.

Motivation & Gaps

Current State

Segment-Level Management Only: The master tracks MountedSegment and allocates chunks via allocators, but does not correlate these segments back to their underlying storage device identity, health status, or physical capacity limits.
Binary / Coarse Heartbeat Failure Handling: Heartbeat probes run in the background. Once failure counts exceed a threshold, unmounting may occur, but there is no intermediate DEGRADED state to warn operators or trigger graceful draining before unmounting.
Lack of Observability: Storage administrators and cluster orchestrators cannot query the storage device inventory or retrieve pending maintenance plans through standard Admin REST APIs.
No On-Demand Probe Trigger: If an operator suspects a hardware issue on a specific NVMe target, there is no mechanism to trigger an immediate diagnostic probe without waiting for the next scheduled heartbeat interval.

Proposed Design

1. Storage Device Metadata & Health State Machine

Health Lifecycle

A storage device transitions across five explicit health states:

   +------------+
   |  UNKNOWN   | (initial / unprobed)
   +-----+------+
         |
         v
   +------------+  probe failure >= threshold / 2   +------------+
   |  HEALTHY   | --------------------------------> |  DEGRADED  |
   +------------+                                   +-----+------+
         ^                                                |
         | probe success                                  | probe failure >= threshold
         +------------------------------------------------+
                                                          v
                                                    +------------+
                                                    |   FAILED   |
                                                    +------------+

   +-------------------------------------------------------------+
   |  UNMOUNTING (terminal / unmounting in progress)             |
   +-------------------------------------------------------------+

DeriveStorageDeviceHealth: Pure function mapping probe accounting (is_unmounting, consecutive_probe_failures, consecutive_probe_successes) and threshold policies (degraded_threshold, failed_threshold) to StorageDeviceHealth.
Dynamic Threshold Derivation: By default, failed_threshold = nof_heartbeat_failures_threshold_ and degraded_threshold = max(1, failed_threshold / 2).

Maintenance Planner

BuildStorageDeviceMaintenancePlan() inspects active devices and categorizes them into two actionable sets:

recovery_candidates: Devices in DEGRADED or FAILED state, detailing consecutive probe failures, last error message, and schedulability.
gc_candidates: Devices flagged as UNMOUNTING, or devices whose allocated capacity utilization exceeds the critical high-water mark (\ge 95\%).

2. MasterService & Segment Engine Integration

Allocator Usage Aggregation:
NoFSegmentManager::GetSegmentUsages() inspects each active segment allocator, returning:

struct SegmentUsageInfo {
    std::string segment_name;
    uint64_t total_bytes;
    uint64_t allocated_bytes;
};

Master Device Inventory:
MasterService::ListStorageDevices() combines:
1. Mounted NoF target snapshots (device_id, target_name, uri, capacity).
2. Aggregated segment capacity and allocated bytes.
3. Background heartbeat probe health, failure counters, and last probe timestamp.
Operator-Triggered Probing:
MasterService::RequestStorageDeviceProbe(device_id) schedules an immediate asynchronous heartbeat probe against the specified device.

3. Admin REST APIs

Endpoint	Method	Description
`/api/v1/devices`	`GET`	Lists NoF storage devices with identity, capacity, allocated bytes, health, and probe stats. Supports query filter `?health=healthy
`/api/v1/devices/maintenance_plan`	`GET`	Returns structured JSON containing `recovery_candidates` and `gc_candidates` for maintenance schedulers.
`/api/v1/devices/probe?device_id=<id>`	`POST`	Submits an immediate on-demand probe request for a target device.

Sample Maintenance Plan Output

{
  "recovery_candidates": [
    {
      "device_id": "target-rack1-ssd0",
      "health": "DEGRADED",
      "consecutive_probe_failures": 3,
      "failed_threshold": 5,
      "last_error": "timeout reading heartbeat block",
      "schedulable": false
    }
  ],
  "gc_candidates": [
    {
      "device_id": "target-rack2-ssd1",
      "reason": "high_usage",
      "usage_percentage": 96.5,
      "allocated_bytes": 1060960534528,
      "capacity_bytes": 1099511627776
    }
  ]
}

Phased Implementation Plan

Phase 1 (Complete — PR #4560):
- Add storage_device_metadata.h/.cpp with core data types and pure derivation logic.
- Implement NoFSegmentManager::GetSegmentUsages and MasterService::ListStorageDevices.
- Add Admin REST endpoints (/api/v1/devices, /api/v1/devices/maintenance_plan, /api/v1/devices/probe).
- Unit and integration tests (storage_device_metadata_test, master_admin_server_test, nof_heartbeat_test).
Phase 2 (Follow-up):
- Automate graceful drain / evacuation workflows triggered by maintenance_plan (e.g. automatically trigger segment drain when a device enters DEGRADED).
Phase 3 (Follow-up):
- Integrate device health and active IO load into client-side multi-replica read balancing and write offload throttling.

Compatibility & Impact

Backward Compatibility: All new REST endpoints and C++ APIs are additive. Non-NoF builds (USE_NOF=OFF) remain fully functional and tested.
Performance Overhead: ListStorageDevices uses cached probe accounting and reader locks over segment managers, avoiding locks on the hot I/O data path.
Reference Implementation: Implemented and tested in PR #4560.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @zxpdemonio on GitHub (Oct 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4561 ### Changes proposed ## Summary Mooncake Store supports offloading objects to NVMe-oF (NoF) SSD targets and local SSDs. While the master server manages segments via `NoFSegmentManager` and periodically executes heartbeat probes against mounted targets, it previously managed storage primarily at the coarse segment/allocator level without a formalized **storage device control plane**. Storage device health lifecycles, real capacity utilization across active allocators, degradation detection, and maintenance planning were unexposed to cluster operators and automated maintenance pipelines. This RFC proposes a unified, lightweight **NoF Storage Device Control Plane** for Mooncake Store: 1. **Device Metadata & Health State Machine**: Formalized `StorageDeviceHealth` states with a pure-function derivation model and automated maintenance planning (recovery vs. GC candidacy). 2. **Master & Segment Engine Integration**: Device-level capacity and allocated-bytes accounting via `NoFSegmentManager::GetSegmentUsages()`, coupled with on-demand operator probe triggering. 3. **Admin Observability & Operations Endpoints**: REST APIs (`/api/v1/devices`, `/api/v1/devices/maintenance_plan`, `/api/v1/devices/probe`) for integration with Kubernetes operators, cluster dashboards, and automated node evacuations. An implementation of Phase 1 is already available in **PR #4560**. --- ## Motivation & Gaps ### Current State - **Segment-Level Management Only**: The master tracks `MountedSegment` and allocates chunks via allocators, but does not correlate these segments back to their underlying storage device identity, health status, or physical capacity limits. - **Binary / Coarse Heartbeat Failure Handling**: Heartbeat probes run in the background. Once failure counts exceed a threshold, unmounting may occur, but there is no intermediate `DEGRADED` state to warn operators or trigger graceful draining before unmounting. - **Lack of Observability**: Storage administrators and cluster orchestrators cannot query the storage device inventory or retrieve pending maintenance plans through standard Admin REST APIs. - **No On-Demand Probe Trigger**: If an operator suspects a hardware issue on a specific NVMe target, there is no mechanism to trigger an immediate diagnostic probe without waiting for the next scheduled heartbeat interval. --- ## Proposed Design ### 1. Storage Device Metadata & Health State Machine #### Health Lifecycle A storage device transitions across five explicit health states: ```text +------------+ | UNKNOWN | (initial / unprobed) +-----+------+ | v +------------+ probe failure >= threshold / 2 +------------+ | HEALTHY | --------------------------------> | DEGRADED | +------------+ +-----+------+ ^ | | probe success | probe failure >= threshold +------------------------------------------------+ v +------------+ | FAILED | +------------+ +-------------------------------------------------------------+ | UNMOUNTING (terminal / unmounting in progress) | +-------------------------------------------------------------+ ``` - **`DeriveStorageDeviceHealth`**: Pure function mapping probe accounting (`is_unmounting`, `consecutive_probe_failures`, `consecutive_probe_successes`) and threshold policies (`degraded_threshold`, `failed_threshold`) to `StorageDeviceHealth`. - **Dynamic Threshold Derivation**: By default, `failed_threshold = nof_heartbeat_failures_threshold_` and `degraded_threshold = max(1, failed_threshold / 2)`. #### Maintenance Planner `BuildStorageDeviceMaintenancePlan()` inspects active devices and categorizes them into two actionable sets: - **`recovery_candidates`**: Devices in `DEGRADED` or `FAILED` state, detailing consecutive probe failures, last error message, and schedulability. - **`gc_candidates`**: Devices flagged as `UNMOUNTING`, or devices whose allocated capacity utilization exceeds the critical high-water mark (\(\ge 95\%\)). ### 2. MasterService & Segment Engine Integration - **Allocator Usage Aggregation**: `NoFSegmentManager::GetSegmentUsages()` inspects each active segment allocator, returning: ```cpp struct SegmentUsageInfo { std::string segment_name; uint64_t total_bytes; uint64_t allocated_bytes; }; ``` - **Master Device Inventory**: `MasterService::ListStorageDevices()` combines: 1. Mounted NoF target snapshots (`device_id`, `target_name`, `uri`, `capacity`). 2. Aggregated segment capacity and allocated bytes. 3. Background heartbeat probe health, failure counters, and last probe timestamp. - **Operator-Triggered Probing**: `MasterService::RequestStorageDeviceProbe(device_id)` schedules an immediate asynchronous heartbeat probe against the specified device. ### 3. Admin REST APIs | Endpoint | Method | Description | | --- | --- | --- | | `/api/v1/devices` | `GET` | Lists NoF storage devices with identity, capacity, allocated bytes, health, and probe stats. Supports query filter `?health=healthy|degraded|failed|unmounting`. | | `/api/v1/devices/maintenance_plan` | `GET` | Returns structured JSON containing `recovery_candidates` and `gc_candidates` for maintenance schedulers. | | `/api/v1/devices/probe?device_id=<id>` | `POST` | Submits an immediate on-demand probe request for a target device. | #### Sample Maintenance Plan Output ```json { "recovery_candidates": [ { "device_id": "target-rack1-ssd0", "health": "DEGRADED", "consecutive_probe_failures": 3, "failed_threshold": 5, "last_error": "timeout reading heartbeat block", "schedulable": false } ], "gc_candidates": [ { "device_id": "target-rack2-ssd1", "reason": "high_usage", "usage_percentage": 96.5, "allocated_bytes": 1060960534528, "capacity_bytes": 1099511627776 } ] } ``` --- ## Phased Implementation Plan - **Phase 1 (Complete — PR #4560)**: - Add `storage_device_metadata.h/.cpp` with core data types and pure derivation logic. - Implement `NoFSegmentManager::GetSegmentUsages` and `MasterService::ListStorageDevices`. - Add Admin REST endpoints (`/api/v1/devices`, `/api/v1/devices/maintenance_plan`, `/api/v1/devices/probe`). - Unit and integration tests (`storage_device_metadata_test`, `master_admin_server_test`, `nof_heartbeat_test`). - **Phase 2 (Follow-up)**: - Automate graceful drain / evacuation workflows triggered by `maintenance_plan` (e.g. automatically trigger segment drain when a device enters `DEGRADED`). - **Phase 3 (Follow-up)**: - Integrate device health and active IO load into client-side multi-replica read balancing and write offload throttling. --- ## Compatibility & Impact - **Backward Compatibility**: All new REST endpoints and C++ APIs are additive. Non-NoF builds (`USE_NOF=OFF`) remain fully functional and tested. - **Performance Overhead**: `ListStorageDevices` uses cached probe accounting and reader locks over segment managers, avoiding locks on the hot I/O data path. - **Reference Implementation**: Implemented and tested in [PR #4560](https://github.com/kvcache-ai/Mooncake/pull/4560). --- ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

Store

RFC

labels

2026-10-29 21:23:30 +00:00

bsakaguchi commented

2026-10-29 21:23:31 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 10, 2026):

Thanks for opening this issue, @zxpdemonio!

Field	Value
Issue	#4561
GitHub user ID	`16206312`
Reporter	@zxpdemonio

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 10, 2026): Thanks for opening this issue, @zxpdemonio! | Field | Value | |-------|-------| | **Issue** | #4561 | | **GitHub user ID** | `16206312` | | **Reporter** | @zxpdemonio | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:23:31 +00:00

Author

Owner

@zxpdemonio commented on GitHub (Oct 10, 2026):

Phase 2 from the phased plan above is now specified in #4565 and implemented in #4566, stacked on #4560.

Scope delivered:

Automatic fault isolation at DEGRADED, self-healing on the first successful probe.
Device-level graceful drain bound to the existing DrainJob machinery, so no second migration lifecycle is introduced.
maintenance_plan integration: device_isolated in recovery candidates, draining in GC candidates.
Admin endpoints isolate, unisolate, drain, drain_status, plus isolated / draining / draining_job_id on GET /api/v1/devices.
--nof_auto_isolate_degraded_devices (default true) as an operator off-switch.

One refinement relative to the state machine drawn above is worth flagging for reviewers of #4560: the fence triggers at DEGRADED rather than at FAILED. FAILED is derived from the same threshold as the alive timeout, so a device that is only fenced at FAILED is unmounted in the same heartbeat iteration and the fence never gets a window to act. The alive-timeout unmount rule itself is unchanged, and fencing a device never delays it.

@zxpdemonio commented on GitHub (Oct 10, 2026): Phase 2 from the phased plan above is now specified in #4565 and implemented in #4566, stacked on #4560. Scope delivered: - Automatic fault isolation at `DEGRADED`, self-healing on the first successful probe. - Device-level graceful drain bound to the existing `DrainJob` machinery, so no second migration lifecycle is introduced. - `maintenance_plan` integration: `device_isolated` in recovery candidates, `draining` in GC candidates. - Admin endpoints `isolate`, `unisolate`, `drain`, `drain_status`, plus `isolated` / `draining` / `draining_job_id` on `GET /api/v1/devices`. - `--nof_auto_isolate_degraded_devices` (default `true`) as an operator off-switch. One refinement relative to the state machine drawn above is worth flagging for reviewers of #4560: the fence triggers at `DEGRADED` rather than at `FAILED`. `FAILED` is derived from the same threshold as the alive timeout, so a device that is only fenced at `FAILED` is unmounted in the same heartbeat iteration and the fence never gets a window to act. The alive-timeout unmount rule itself is unchanged, and fencing a device never delays it.