[GH-ISSUE #2783] [Bug]: ssd total capacity shows 0 after master restart with snapshot and offload enabled #722

New issue

Closed

opened 2026-10-29 21:18:42 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:18:42 +00:00

Owner

Originally created by @liangxu2000 on GitHub (Jul 7, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2783

Bug Report

Environment

Mooncake version: master branch
Master config: enable_snapshot=true, enable_offloading=true
Client: 1 client, writing 100 KV pairs with SSD offload

Description

After master restarts with snapshot restore, the SSD total capacity (denominator)
in metrics shows 0, while the used space (numerator) is correct:

Before restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 2TB
After restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 0 B
^^^

The issue persists until the client also restarts (re-executes FileStorage::Init).

Root Cause (from source code analysis)

Two things happen:

1. ssd_total_capacity_bytes is not serialized in snapshot

In mooncake-store/src/segment.cpp, the SegmentSerializer serializes LocalDiskSegment with only enable_offloading and the offloading objects list (line ~589-591). The field ssd_total_capacity_bytes is not included.

After deserialize (line ~927-928), a new LocalDiskSegment is created with ssd_total_capacity_bytes = 0 (default value, segment.h:88).

2. file_total_capacity_ gauge is not rebuilt after restore

In TryRestoreStateFromSnapshot() (master_service.cpp), memory capacity is rebuilt after restore (lines 5001-5034), but there is no equivalent logic for file capacity. The file_total_capacity_ gauge remains at its initial value 0.

Note: ReportSsdCapacity RPC is the only runtime path that sets file_total_capacity_, but the client only calls it once at FileStorage::Init() and does not re-report after master restart.

Steps to Reproduce

Start master with enable_snapshot=true and enable_offloading=true
Start 1 client with SSD offload configured (e.g. 2TB capacity)
Write 100 KV pairs to trigger offloading to SSD
Observe metrics: SSD Storage: X / 2TB (correct)
Restart the master process (client continues running)
Master restores from snapshot
Observe metrics: SSD Storage: X / 0 B (denominator is 0)

Suggested Fix

Add ssd_total_capacity_bytes to LocalDiskSegment serialization/deserialization in segment.cpp
After snapshot restore in master_service.cpp, rebuild file_total_capacity_ gauge from restored segments, similar to the existing mem capacity rebuild logic (lines 5001-5034)

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @liangxu2000 on GitHub (Jul 7, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2783 ### Bug Report ## Environment - Mooncake version: `master` branch - Master config: `enable_snapshot=true`, `enable_offloading=true` - Client: 1 client, writing 100 KV pairs with SSD offload ## Description After master restarts with snapshot restore, the SSD total capacity (denominator) in metrics shows 0, while the used space (numerator) is correct: Before restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 2TB After restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 0 B ^^^ The issue persists until the client also restarts (re-executes `FileStorage::Init`). ## Root Cause (from source code analysis) Two things happen: **1. `ssd_total_capacity_bytes` is not serialized in snapshot** In `mooncake-store/src/segment.cpp`, the `SegmentSerializer` serializes `LocalDiskSegment` with only `enable_offloading` and the offloading objects list (line ~589-591). The field `ssd_total_capacity_bytes` is not included. After deserialize (line ~927-928), a new `LocalDiskSegment` is created with `ssd_total_capacity_bytes = 0` (default value, `segment.h:88`). **2. `file_total_capacity_` gauge is not rebuilt after restore** In `TryRestoreStateFromSnapshot()` (`master_service.cpp`), memory capacity is rebuilt after restore (lines 5001-5034), but there is no equivalent logic for file capacity. The `file_total_capacity_` gauge remains at its initial value 0. Note: `ReportSsdCapacity` RPC is the only runtime path that sets `file_total_capacity_`, but the client only calls it once at `FileStorage::Init()` and does not re-report after master restart. ## Steps to Reproduce 1. Start master with `enable_snapshot=true` and `enable_offloading=true` 2. Start 1 client with SSD offload configured (e.g. 2TB capacity) 3. Write 100 KV pairs to trigger offloading to SSD 4. Observe metrics: `SSD Storage: X / 2TB` (correct) 5. Restart the master process (client continues running) 6. Master restores from snapshot 7. Observe metrics: `SSD Storage: X / 0 B` (denominator is 0) ## Suggested Fix 1. Add `ssd_total_capacity_bytes` to `LocalDiskSegment` serialization/deserialization in `segment.cpp` 2. After snapshot restore in `master_service.cpp`, rebuild `file_total_capacity_` gauge from restored segments, similar to the existing mem capacity rebuild logic (lines 5001-5034) ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:18:42 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:18:43 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 7, 2026):

Thanks for opening this issue, @liangxu2000!

Field	Value
Issue	#2783
GitHub user ID	`258133064`
Reporter	@liangxu2000

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 7, 2026): Thanks for opening this issue, @liangxu2000! | Field | Value | |-------|-------| | **Issue** | #2783 | | **GitHub user ID** | `258133064` | | **Reporter** | @liangxu2000 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:43 +00:00

Author

Owner

@he-yufeng commented on GitHub (Jul 15, 2026):

Thanks for the precise root-cause analysis — the missing field in the LocalDiskSegment serializer was exactly it. Opened #2927, which appends ssd_total_capacity_bytes to the packed array and reads it back backward-compatibly (old snapshots keep the current behavior until the client re-reports).

@he-yufeng commented on GitHub (Jul 15, 2026): Thanks for the precise root-cause analysis — the missing field in the `LocalDiskSegment` serializer was exactly it. Opened #2927, which appends `ssd_total_capacity_bytes` to the packed array and reads it back backward-compatibly (old snapshots keep the current behavior until the client re-reports).

bsakaguchi referenced this issue

2026-10-29 21:30:12 +00:00

[PR #722] [CLOSED] [Bugfix]Fix 'invalid argument' error when setting QP state to INIT vi… #1771