[PR #1140] [MERGED] [mooncake-store]: prevent orphaned bucket data files from leaking dis…

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#2035

bsakaguchi commented

2026-10-29 21:32:17 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/1140
Author: @maheshrbapatu
Created: 11/29/2025
Status: ✅ Merged
Merged: 12/3/2025
Merged by: @ykwd

Base: main ← Head: issue_1139

📝 Commits (5)

b8a5e78 [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
641e1ea [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
7bd5a24 [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
1648d1e [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
d225c9d [mooncake-store]: prevent orphaned bucket data files from leaking disk space.

📊 Changes

3 files changed (+200 additions, -12 deletions)

View changed files

📝 mooncake-store/include/storage_backend.h (+1 -0)
📝 mooncake-store/src/storage_backend.cpp (+84 -11)
📝 mooncake-store/tests/storage_backend_test.cpp (+115 -1)

📄 Description

Description

Add cleanup mechanism for bucket data files that are left orphaned when the process crashes between writing the data file and its metadata file.

Problem:

BucketStorageBackend writes data file first, then .meta file
If crash/failure occurs between these steps, data file is orphaned
Init() only scans .meta files, never discovers orphans
Orphaned files accumulate indefinitely, leaking disk space

Solution:

Add immediate cleanup in WriteBucket() when metadata write fails
Add orphan detection during Init() to clean up past crashes
Make cleanup opt-in (disabled by default) via constructor flag
Identify orphans by: numeric filename, no extension, no .meta pair

Changes:

BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
WriteBucket(): Clean up data file if StoreBucketMetadata() fails
Init(): Scan for and remove orphaned data files when enabled
Add comprehensive test coverage for orphan cleanup

Issue link: https://github.com/kvcache-ai/Mooncake/issues/1139

Type of Change

Types
- Bug fix
- New feature
  - Transfer Engine
  - Mooncake Store
  - Mooncake EP
  - Integration
  - P2P Store
  - Python Wheel
- Breaking change
- CI/CD
- Documentation update
- Other

How Has This Been Tested?

OrphanedDataFileCleanup: Creates valid bucket, manually orphans files, verifies cleanup
Tests cleanup enabled vs disabled (default behavior)
Validates valid buckets preserved during orphan cleanup
Tests file count reduction after cleanup

Checklist

I have performed a self-review of my own code.
I have updated the documentation.
I have added tests to prove my changes are effective.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/1140 **Author:** [@maheshrbapatu](https://github.com/maheshrbapatu) **Created:** 11/29/2025 **Status:** ✅ Merged **Merged:** 12/3/2025 **Merged by:** [@ykwd](https://github.com/ykwd) **Base:** `main` ← **Head:** `issue_1139` --- ### 📝 Commits (5) - [`b8a5e78`](https://github.com/kvcache-ai/Mooncake/commit/b8a5e7830bfb0050c0abdfe6c2311a46e399cc84) [mooncake-store]: prevent orphaned bucket data files from leaking disk space. - [`641e1ea`](https://github.com/kvcache-ai/Mooncake/commit/641e1eadeb9f6f85bbe7dc9931f8271985eb4021) [mooncake-store]: prevent orphaned bucket data files from leaking disk space. - [`7bd5a24`](https://github.com/kvcache-ai/Mooncake/commit/7bd5a241daf9cf83c24595a7859683c13e91efde) [mooncake-store]: prevent orphaned bucket data files from leaking disk space. - [`1648d1e`](https://github.com/kvcache-ai/Mooncake/commit/1648d1e7de7426cb2efe9d1a810f2abfd9e777b1) [mooncake-store]: prevent orphaned bucket data files from leaking disk space. - [`d225c9d`](https://github.com/kvcache-ai/Mooncake/commit/d225c9dead4bc0a4cf19597d5088abf010ec3315) [mooncake-store]: prevent orphaned bucket data files from leaking disk space. ### 📊 Changes **3 files changed** (+200 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-store/include/storage_backend.h` (+1 -0) 📝 `mooncake-store/src/storage_backend.cpp` (+84 -11) 📝 `mooncake-store/tests/storage_backend_test.cpp` (+115 -1) </details> ### 📄 Description ## Description  Add cleanup mechanism for bucket data files that are left orphaned when the process crashes between writing the data file and its metadata file. Problem: - BucketStorageBackend writes data file first, then .meta file - If crash/failure occurs between these steps, data file is orphaned - Init() only scans .meta files, never discovers orphans - Orphaned files accumulate indefinitely, leaking disk space Solution: - Add immediate cleanup in WriteBucket() when metadata write fails - Add orphan detection during Init() to clean up past crashes - Make cleanup opt-in (disabled by default) via constructor flag - Identify orphans by: numeric filename, no extension, no .meta pair Changes: - BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false) - WriteBucket(): Clean up data file if StoreBucketMetadata() fails - Init(): Scan for and remove orphaned data files when enabled - Add comprehensive test coverage for orphan cleanup Issue link: https://github.com/kvcache-ai/Mooncake/issues/1139 ## Type of Change * Types - [X] Bug fix - [ ] New feature - [ ] Transfer Engine - [x] Mooncake Store - [ ] Mooncake EP - [ ] Integration - [ ] P2P Store - [ ] Python Wheel - [ ] Breaking change - [ ] CI/CD - [ ] Documentation update - [ ] Other ## How Has This Been Tested?  - OrphanedDataFileCleanup: Creates valid bucket, manually orphans files, verifies cleanup - Tests cleanup enabled vs disabled (default behavior) - Validates valid buckets preserved during orphan cleanup - Tests file count reduction after cleanup ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [x] I have added tests to prove my changes are effective. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

closed this issue
added the
pull-request
label

Rows
Columns

[PR #1140] [MERGED] [mooncake-store]: prevent orphaned bucket data files from leaking dis… #2035