[PR #610] [MERGED] [Store]feat: Add 3fs native api plugin for KVCache storage persistence #1688

New issue

Closed

opened 2026-10-29 21:29:31 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:31 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/610
Author: @SgtPepperr
Created: 7/10/2025
Status: ✅ Merged
Merged: 7/24/2025
Merged by: @stmatengss

Base: main ← Head: 3fs_native

📝 Commits (10+)

190cf83 initial version for 3fs native api
1d4c992 Merge branch 'main' into 3fs_native
20faa75 delete redundant code in file_interface and storage_backend. Only keep api in use
f110847 refactor some problem
63c16f2 refactor file interface, add lockRAII
73c4060 fix some problem
09b7be7 fix conflict
049aa8d fix test problem
9eb931a rm char file interface , edit filemode
4c78b01 edit cmake file , mkdir hf3fs in store

📊 Changes

20 files changed (+1342 additions, -673 deletions)

View changed files

📝 mooncake-common/common.cmake (+5 -0)
📝 mooncake-store/include/client.h (+2 -5)
➕ mooncake-store/include/file_interface.h (+162 -0)
➕ mooncake-store/include/hf3fs/hf3fs.h (+101 -0)
➖ mooncake-store/include/local_file.h (+0 -97)
📝 mooncake-store/include/storage_backend.h (+66 -29)
📝 mooncake-store/src/CMakeLists.txt (+17 -2)
📝 mooncake-store/src/client.cpp (+31 -16)
➕ mooncake-store/src/hf3fs/CMakeLists.txt (+8 -0)
➕ mooncake-store/src/hf3fs/README.md (+42 -0)
➕ mooncake-store/src/hf3fs/hf3fs_file.cpp (+363 -0)
➕ mooncake-store/src/hf3fs/hf3fs_resource_manager.cpp (+92 -0)
➖ mooncake-store/src/local_file.cpp (+0 -224)
➕ mooncake-store/src/posix_file.cpp (+151 -0)
📝 mooncake-store/src/storage_backend.cpp (+109 -121)
📝 mooncake-store/src/transfer_task.cpp (+11 -11)
📝 mooncake-store/tests/CMakeLists.txt (+3 -3)
➖ mooncake-store/tests/local_file_test.cpp (+0 -165)
➕ mooncake-store/tests/posix_file_test.cpp (+174 -0)
📝 mooncake-store/tests/stress_cluster_benchmark.py (+5 -0)

📄 Description

Background

In the previous PR #437, the initial version of KV cache persistence and tiering functionality was implemented. However, since POSIX interfaces were used for read/write operations, the performance in testing was relatively modest.

Changes Introduced

To improve file read/write performance and make the tiered caching functionality fully viable, this PR introduces the 3fs native API（USRBIO) into the store project as a plugin, significantly reducing the latency of performance-sensitive operations such as get and batchget when reading files.

Key Updates:

Added 3fs native API support
Enables high-performance read operations in 3fs scenarios
Refactored file_interface
Improved support for both POSIX and 3fs read/write modes
Enhanced KV cache persistence
Added batchput mode support for KV cache persistence
Benchmark tool update
Added persistence path specification option to stress_cluster_benchmark.py

Performance Testing

Test Setup

Tool: stress_cluster_benchmark.py (testing batch_put_from and batch_get_into)
Workflow:
1. Prefill on one node
2. Decode on another node
Configuration:
- file_read_thread: 10 (consistent across tests)
- Nodes used: 3
  - Node A: General server running:
    - 3fs meta service
    - Mooncake master + meta services
  - Node B: Storage server running:
    - 3fs storage service
    - Mooncake client (prefill/decode)
  - Node C: Storage server running:
    - 3fs storage service
    - Mooncake client (prefill/decode)
- Data flow: Bidirectional data transfer between Nodes B and C

Test Results（3fs Native API BatchGet Throughput Comparison）

Configuration	Batch Count	BatchGet Throughput (MB/s)
Value Size: 16MB
put: mem+disk get: disk	1	3113
	4	5247
	8	6206
	16	6778
Value Size: 64MB
put: mem+disk get: disk	1	3913
	4	7114
	8	8169
	16	8476

Problem

1. Asynchronous Write Performance Issue

Summary: Identified performance degradation in 3FS asynchronous writes and proposed solution.

Currently, persistence is achieved through asynchronous writes, but before asynchronous writing in 3FS, significant performance degradation may occur due to data copying. Profiling reveals that the number of page faults triggered in this scenario is nearly double the normal count. Future plans include introducing a reuse buffer list to address this performance degradation issue.

2. Read Path Copy Overhead

Summary: Existing read path copy mechanism and deferred optimization rationale.

Currently, the read path incurs an additional copy (iov → slice) due to 3FS's native API mechanism. This copy could be eliminated by ensuring the upper-layer slice uses shared memory (shm) and directly reuses it as the iov's address space, enabling zero-copy. However, this optimization has been temporarily postponed because it would require:

Modifications to the upper Python interface
Exposure of 3FS implementation details to the application layer
codebase intrusiveness

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/610 **Author:** [@SgtPepperr](https://github.com/SgtPepperr) **Created:** 7/10/2025 **Status:** ✅ Merged **Merged:** 7/24/2025 **Merged by:** [@stmatengss](https://github.com/stmatengss) **Base:** `main` ← **Head:** `3fs_native` --- ### 📝 Commits (10+) - [`190cf83`](https://github.com/kvcache-ai/Mooncake/commit/190cf8334259d25e38ff1f4411cac2701d57d0ae) initial version for 3fs native api - [`1d4c992`](https://github.com/kvcache-ai/Mooncake/commit/1d4c992f52391f0e87414e9ce84530385bef8e96) Merge branch 'main' into 3fs_native - [`20faa75`](https://github.com/kvcache-ai/Mooncake/commit/20faa75b84079860a773dc8346539549102305c9) delete redundant code in file_interface and storage_backend. Only keep api in use - [`f110847`](https://github.com/kvcache-ai/Mooncake/commit/f1108473e4d21fc1a499629885996ca035f146f1) refactor some problem - [`63c16f2`](https://github.com/kvcache-ai/Mooncake/commit/63c16f20f3137ae8084604c7f869b58fc91c6ee0) refactor file interface, add lockRAII - [`73c4060`](https://github.com/kvcache-ai/Mooncake/commit/73c4060fceeb1736fc257f8a701d15e5ffaa4f9e) fix some problem - [`09b7be7`](https://github.com/kvcache-ai/Mooncake/commit/09b7be7dc3a42f0c5e4d40652f86b3754e6b4e1c) fix conflict - [`049aa8d`](https://github.com/kvcache-ai/Mooncake/commit/049aa8d2126516871cb9fe0009e8e9eb775c08b2) fix test problem - [`9eb931a`](https://github.com/kvcache-ai/Mooncake/commit/9eb931a5f66c97fa2f852a53def8a410001e00bf) rm char file interface , edit filemode - [`4c78b01`](https://github.com/kvcache-ai/Mooncake/commit/4c78b014b21768e8312fa77a3f5a72119f0ec7f6) edit cmake file , mkdir hf3fs in store ### 📊 Changes **20 files changed** (+1342 additions, -673 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-common/common.cmake` (+5 -0) 📝 `mooncake-store/include/client.h` (+2 -5) ➕ `mooncake-store/include/file_interface.h` (+162 -0) ➕ `mooncake-store/include/hf3fs/hf3fs.h` (+101 -0) ➖ `mooncake-store/include/local_file.h` (+0 -97) 📝 `mooncake-store/include/storage_backend.h` (+66 -29) 📝 `mooncake-store/src/CMakeLists.txt` (+17 -2) 📝 `mooncake-store/src/client.cpp` (+31 -16) ➕ `mooncake-store/src/hf3fs/CMakeLists.txt` (+8 -0) ➕ `mooncake-store/src/hf3fs/README.md` (+42 -0) ➕ `mooncake-store/src/hf3fs/hf3fs_file.cpp` (+363 -0) ➕ `mooncake-store/src/hf3fs/hf3fs_resource_manager.cpp` (+92 -0) ➖ `mooncake-store/src/local_file.cpp` (+0 -224) ➕ `mooncake-store/src/posix_file.cpp` (+151 -0) 📝 `mooncake-store/src/storage_backend.cpp` (+109 -121) 📝 `mooncake-store/src/transfer_task.cpp` (+11 -11) 📝 `mooncake-store/tests/CMakeLists.txt` (+3 -3) ➖ `mooncake-store/tests/local_file_test.cpp` (+0 -165) ➕ `mooncake-store/tests/posix_file_test.cpp` (+174 -0) 📝 `mooncake-store/tests/stress_cluster_benchmark.py` (+5 -0) </details> ### 📄 Description ## Background In the previous PR #437, the initial version of KV cache persistence and tiering functionality was implemented. However, since POSIX interfaces were used for read/write operations, the performance in testing was relatively modest. ## Changes Introduced To improve file read/write performance and make the tiered caching functionality fully viable, this PR introduces the 3fs native API（`USRBIO`) into the store project as a plugin, significantly reducing the latency of performance-sensitive operations such as `get` and `batchget` when reading files. ### Key Updates: - **Added 3fs native API support** Enables high-performance read operations in 3fs scenarios - **Refactored `file_interface`** Improved support for both POSIX and 3fs read/write modes - **Enhanced KV cache persistence** Added batchput mode support for KV cache persistence - **Benchmark tool update** Added persistence path specification option to `stress_cluster_benchmark.py` ## Performance Testing ### Test Setup - **Tool**: `stress_cluster_benchmark.py` (testing `batch_put_from` and `batch_get_into`) - **Workflow**: 1. Prefill on one node 2. Decode on another node - **Configuration**: - `file_read_thread`: 10 (consistent across tests) - **Nodes used**: 3 - **Node A**: General server running: - 3fs meta service - Mooncake master + meta services - **Node B**: Storage server running: - 3fs storage service - Mooncake client (prefill/decode) - **Node C**: Storage server running: - 3fs storage service - Mooncake client (prefill/decode) - **Data flow**: Bidirectional data transfer between Nodes B and C ### Test Results（3fs Native API BatchGet Throughput Comparison） | Configuration | Batch Count | BatchGet Throughput (MB/s) | |----------------------------:|:----------:|:--------------------------:| | **Value Size: 16MB** | | | | put: mem+disk<br>get: disk | 1 | 3113 | | | 4 | 5247 | | | 8 | 6206 | | | 16 | 6778 | | **Value Size: 64MB** | | | | put: mem+disk<br>get: disk | 1 | 3913 | | | 4 | 7114 | | | 8 | 8169 | | | 16 | 8476 | ## Problem ### 1. Asynchronous Write Performance Issue **Summary**: Identified performance degradation in 3FS asynchronous writes and proposed solution. Currently, persistence is achieved through asynchronous writes, but before asynchronous writing in 3FS, significant performance degradation may occur due to data copying. Profiling reveals that the number of page faults triggered in this scenario is nearly double the normal count. Future plans include introducing a reuse buffer list to address this performance degradation issue. ## 2. Read Path Copy Overhead **Summary**: Existing read path copy mechanism and deferred optimization rationale. Currently, the read path incurs an additional copy (iov → slice) due to 3FS's native API mechanism. This copy could be eliminated by ensuring the upper-layer slice uses shared memory (shm) and directly reuses it as the iov's address space, enabling zero-copy. However, this optimization has been temporarily postponed because it would require: - Modifications to the upper Python interface - Exposure of 3FS implementation details to the application layer - codebase intrusiveness --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:31 +00:00