[GH-ISSUE #2715] [Bug]: mooncake-store offload SSD some question #708

New issue

Open

opened 2026-10-29 21:18:29 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:18:29 +00:00

Owner

Originally created by @weibingo on GitHub (Jul 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2715

Bug Report

When SSD offload is enabled, I noticed that after writing some keys on Node 1, they get offloaded to the SSD. Later, when I try to read these keys from Node 2, if the key has already been evicted from Node 1’s DRAM but still resides on its SSD, Node 2 will attempt to fetch it. However, Node 2 tries to read directly via the local SSD file path of the key. Since Node 2 obviously does not have this file locally, the read operation fails.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @weibingo on GitHub (Jul 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2715 ### Bug Report When SSD offload is enabled, I noticed that after writing some keys on Node 1, they get offloaded to the SSD. Later, when I try to read these keys from Node 2, if the key has already been evicted from Node 1’s DRAM but still resides on its SSD, Node 2 will attempt to fetch it. However, Node 2 tries to read directly via the local SSD file path of the key. Since Node 2 obviously does not have this file locally, the read operation fails. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

stale

bug

labels

2026-10-29 21:18:29 +00:00

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 2, 2026):

Thanks for opening this issue, @weibingo!

Field	Value
Issue	#2715
GitHub user ID	`10209205`
Reporter	@weibingo

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 2, 2026): Thanks for opening this issue, @weibingo! | Field | Value | |-------|-------| | **Issue** | #2715 | | **GitHub user ID** | `10209205` | | **Reporter** | @weibingo | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@LujhCoconut commented on GitHub (Jul 2, 2026):

Hi, did you set root_fs_dir in your master config? That's a legacy path designed for shared filesystems. When set, each key gets a DISK replica with a local file_path — remote nodes will try to ::open() that path locally and fail.

The recommended way is --enable_offload=true only, which uses LOCAL_DISK replicas: the SSD owner reads data from disk into a registered DRAM buffer, then the remote node pulls it via RDMA.

@LujhCoconut commented on GitHub (Jul 2, 2026): Hi, did you set `root_fs_dir` in your master config? That's a legacy path designed for shared filesystems. When set, each key gets a DISK replica with a local `file_path` — remote nodes will try to `::open()` that path locally and fail. The recommended way is `--enable_offload=true` only, which uses `LOCAL_DISK` replicas: the SSD owner reads data from disk into a registered DRAM buffer, then the remote node pulls it via RDMA.

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@weibingo commented on GitHub (Jul 29, 2026):

yes , we test it with new docs

@weibingo commented on GitHub (Jul 29, 2026): yes , we test it with new docs

bsakaguchi commented

2026-10-29 21:18:30 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 28, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 28, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.