[PR #251] [CLOSED] bugfix: let meta_cache.remove happen after meta_store.write #341

New issue

Closed

opened 2026-10-26 09:48:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:48:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/3FS/pull/251
Author: @YuqqiZhou
Created: 4/22/2025
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (1)

73cebcf bugfix: let meta_cache.remove happen after meta_store.write

📊 Changes

1 file changed (+13 additions, -7 deletions)

View changed files

📝 src/storage/chunk_engine/src/core/engine.rs (+13 -7)

📄 Description

We found meta data in RocksDB inconsistent under these conditions:

RemoveTarget is called (batch_remove in engine.rs)
Some write request , which trigger chunk allocate
system restart
The main issue is the meta_cache in batch_remove. After meta_store.remove_mut is called, meta_cache.remove is performed, which will dealloacte the chunk Position and the Position may be reused , but the write_batch is not persisted. See an example:
[thread 1]batch_remove remove chunk1<->pos1
[thread 1]the deletion of key chunk1->pos1 and pos1->chunk1 is written in write_batch, but not persisted into disk
[thread 2]another write operation use pos2, add chunk2->pos1 and pos1->chunk2, and persisted into disk
[thread 1] persist write_batch
Now the rocksDB has an inconsistent state, only chunk2->pos1 map is reserved, when pos1 is allocated again, it will cause fatal err. as shown in https://github.com/deepseek-ai/3FS/issues/238

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/3FS/pull/251 **Author:** [@YuqqiZhou](https://github.com/YuqqiZhou) **Created:** 4/22/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`73cebcf`](https://github.com/deepseek-ai/3FS/commit/73cebcf537ea6bfc36359cae213f9af9b2ba5b17) bugfix: let meta_cache.remove happen after meta_store.write ### 📊 Changes **1 file changed** (+13 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `src/storage/chunk_engine/src/core/engine.rs` (+13 -7) </details> ### 📄 Description We found meta data in RocksDB inconsistent under these conditions: 1. RemoveTarget is called (batch_remove in engine.rs) 2. Some write request , which trigger chunk allocate 3. system restart The main issue is the meta_cache in batch_remove. After meta_store.remove_mut is called, meta_cache.remove is performed, which will dealloacte the chunk Position and the Position may be reused , but the write_batch is not persisted. See an example: 1. [thread 1]batch_remove remove chunk1<->pos1 2. [thread 1]the deletion of key chunk1->pos1 and pos1->chunk1 is written in write_batch, but not persisted into disk 3. [thread 2]another write operation use pos2, add chunk2->pos1 and pos1->chunk2, and persisted into disk 4. [thread 1] persist write_batch Now the rocksDB has an inconsistent state, only chunk2->pos1 map is reserved, when pos1 is allocated again, it will cause fatal err. as shown in https://github.com/deepseek-ai/3FS/issues/238 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 09:48:51 +00:00