kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4345] [Bug][Store] BucketStorageBackend: "O_DIRECT mode enabled" is logged but bucket files are opened buffered (O_DIRECT not set) #1087

New issue

Open

opened 2026-10-29 21:22:45 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:22:45 +00:00

Owner

Originally created by @cloudomate on GitHub (Sep 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4345

Bug Report

Version: mooncake-transfer-engine-cuda13 / mooncake-transfer-engine 0.3.13.post1 (PyPI), mooncake_client real client, x86_64, Ubuntu 24.04 (kernel 6.8), ext4 on NVMe.

Config (store / real client):

MOONCAKE_OFFLOAD_ENABLED=true
MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend
MOONCAKE_OFFLOAD_USE_URING=true
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=1073741824   # registers fine as a fixed buffer
MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/kvcache/mooncake
master: --enable_offload=true --offload_on_evict=true --promotion_on_hit=true

Expected: bucket files opened with O_DIRECT, so the SSD tier bypasses the page cache.
This is what #3611 describes as the correct behaviour for BucketStorageBackend.

Actual:

The log says it's enabled, once per bucket and meta file:

uring_file.cpp:590] [UringFile] O_DIRECT mode enabled for /kvcache/mooncake/7333952934034.bucket

But the kernel's open-file flags say otherwise. Sampling /proc/<mooncake_client pid>/fdinfo/<fd> every 200 ms during a 110 GiB spill, on 3 nodes: 69 distinct open bucket files, all with the same flags:
```
flags:	02100001      # O_CLOEXEC | O_LARGEFILE | O_WRONLY  (no O_DIRECT = 040000)
```
Consistent with that, each node's page cache grows while spilling, capped only by the container's memory cgroup.

Looks like: the same pattern as #3611 (the flag is stored and logged, but open()/fcntl() never sets it), here on the bucket backend's write path. PR #3615 fixes the OffsetAllocator vector-I/O path only, and isn't in a release yet (v0.3.14-rc1 predates it).

How to reproduce:

Run a real client with the config above.
Put more than the DRAM segment holds, so that offload-on-evict writes buckets.
While it writes: for fd in /proc/$(pgrep -x mooncake_client)/fd/*; do readlink $fd | grep -q '\.bucket$' && grep flags /proc/$(pgrep -x mooncake_client)/fdinfo/${fd##*/}; done

Impact:

SSD-tier reads can be served from the page cache, which inflates SSD hit latency and throughput in benchmarks.
Page cache competes with other workloads on shared nodes. Here it's bounded only by the container memory limit.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @cloudomate on GitHub (Sep 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4345 ### Bug Report **Version:** `mooncake-transfer-engine-cuda13` / `mooncake-transfer-engine` **0.3.13.post1** (PyPI), `mooncake_client` real client, x86_64, Ubuntu 24.04 (kernel 6.8), ext4 on NVMe. **Config (store / real client):** ``` MOONCAKE_OFFLOAD_ENABLED=true MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend MOONCAKE_OFFLOAD_USE_URING=true MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=1073741824 # registers fine as a fixed buffer MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/kvcache/mooncake master: --enable_offload=true --offload_on_evict=true --promotion_on_hit=true ``` **Expected:** bucket files opened with `O_DIRECT`, so the SSD tier bypasses the page cache. This is what #3611 describes as the correct behaviour for `BucketStorageBackend`. **Actual:** - The log says it's enabled, once per bucket and meta file: ``` uring_file.cpp:590] [UringFile] O_DIRECT mode enabled for /kvcache/mooncake/7333952934034.bucket ``` - But the kernel's open-file flags say otherwise. Sampling `/proc/<mooncake_client pid>/fdinfo/<fd>` every 200 ms during a 110 GiB spill, on 3 nodes: 69 distinct open bucket files, all with the **same** flags: ``` flags: 02100001 # O_CLOEXEC | O_LARGEFILE | O_WRONLY (no O_DIRECT = 040000) ``` - Consistent with that, each node's page cache grows while spilling, capped only by the container's memory cgroup. **Looks like:** the same pattern as #3611 (the flag is stored and logged, but `open()`/`fcntl()` never sets it), here on the bucket backend's write path. PR #3615 fixes the OffsetAllocator vector-I/O path only, and isn't in a release yet (v0.3.14-rc1 predates it). **How to reproduce:** 1. Run a real client with the config above. 2. Put more than the DRAM segment holds, so that offload-on-evict writes buckets. 3. While it writes: `for fd in /proc/$(pgrep -x mooncake_client)/fd/*; do readlink $fd | grep -q '\.bucket$' && grep flags /proc/$(pgrep -x mooncake_client)/fdinfo/${fd##*/}; done` **Impact:** - SSD-tier reads can be served from the page cache, which inflates SSD hit latency and throughput in benchmarks. - Page cache competes with other workloads on shared nodes. Here it's bounded only by the container memory limit. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:22:45 +00:00

bsakaguchi commented

2026-10-29 21:22:46 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 27, 2026):

Thanks for opening this issue, @cloudomate!

Field	Value
Issue	#4345
GitHub user ID	`38678537`
Reporter	@cloudomate

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 27, 2026): Thanks for opening this issue, @cloudomate! | Field | Value | |-------|-------| | **Issue** | #4345 | | **GitHub user ID** | `38678537` | | **Reporter** | @cloudomate | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:22:46 +00:00

Author

Owner

@he-yufeng commented on GitHub (Sep 27, 2026):

Read the bucket write path on current main (and the 0.3.13.post1 tag, which matches it here). What I found:

The log line is intent, not state: UringFile's ctor prints "O_DIRECT mode enabled" from its use_direct_io argument without ever checking the fd. Any caller that passes true without O_DIRECT in the open flags logs a lie. Worth a small honesty fix (log from fcntl(fd, F_GETFL) instead of the flag); I can do that PR.
On your actual spill symptom: bucket writes deliberately do not get O_DIRECT today. BucketStorageBackend::OpenFile only adds O_DIRECT and constructs a UringFile for FileMode::Read (storage_backend.cpp:3452-3470); writes go through plain PosixFile. So the O_WRONLY fds you sampled are by design, and the page-cache growth you measured is consistent with the code's own comment: O_DIRECT writes are avoided because the 4096-byte alignment padding would corrupt meta-file parsing and waste space on data files. The log lines you saw for bucket files came from the read-mode opens (promotion read-backs), not the write fds you sampled.
The interesting discovery while tracing this: WriteBucket already contains a complete aligned O_DIRECT write implementation (write_aligned with align_up + zero-padding into aligned_io_buffer_), gated on dynamic_cast<UringFile*>(file.get()) — which can never succeed today, because OpenFile never creates a write-mode UringFile. So the O_DIRECT bucket-write path is fully built but dormant. Enabling it for bucket data writes (keeping meta files buffered, since the padding breaks their parsing) is a real option if the store owners want it, and it matches the direction #3615 took for the allocator backend. That's a storage-hot-path behavior change though, so I'd want @stmatengss's call before writing it.

@he-yufeng commented on GitHub (Sep 27, 2026): Read the bucket write path on current main (and the 0.3.13.post1 tag, which matches it here). What I found: 1. The log line is intent, not state: `UringFile`'s ctor prints "O_DIRECT mode enabled" from its `use_direct_io` argument without ever checking the fd. Any caller that passes `true` without `O_DIRECT` in the open flags logs a lie. Worth a small honesty fix (log from `fcntl(fd, F_GETFL)` instead of the flag); I can do that PR. 2. On your actual spill symptom: bucket *writes* deliberately do not get O_DIRECT today. `BucketStorageBackend::OpenFile` only adds `O_DIRECT` and constructs a `UringFile` for `FileMode::Read` (storage_backend.cpp:3452-3470); writes go through plain `PosixFile`. So the O_WRONLY fds you sampled are by design, and the page-cache growth you measured is consistent with the code's own comment: O_DIRECT writes are avoided because the 4096-byte alignment padding would corrupt meta-file parsing and waste space on data files. The log lines you saw for bucket files came from the read-mode opens (promotion read-backs), not the write fds you sampled. 3. The interesting discovery while tracing this: `WriteBucket` already contains a complete aligned O_DIRECT write implementation (`write_aligned` with `align_up` + zero-padding into `aligned_io_buffer_`), gated on `dynamic_cast<UringFile*>(file.get())` — which can never succeed today, because `OpenFile` never creates a write-mode `UringFile`. So the O_DIRECT bucket-write path is fully built but dormant. Enabling it for bucket *data* writes (keeping meta files buffered, since the padding breaks their parsing) is a real option if the store owners want it, and it matches the direction #3615 took for the allocator backend. That's a storage-hot-path behavior change though, so I'd want @stmatengss's call before writing it.

bsakaguchi commented

2026-10-29 21:22:46 +00:00

Author

Owner

@he-yufeng commented on GitHub (Sep 27, 2026):

One correction to my point 1 above, after auditing every UringFile(..., true) call site on current main: all five set O_DIRECT in the open flags first (storage_backend.cpp:747, 3469, 3828, 4620, 5546), so on main the log does not actually lie. The version you run (0.3.13.post1) did have the real mismatch in the allocator backend — that was #3611, already fixed and merged in #3615. For bucket files the log only fires on read-mode opens, which do carry O_DIRECT; the fds you sampled with O_WRONLY|O_CLOEXEC were the write-mode opens, which are deliberately buffered and never logged. So the "enabled" log and the no-O_DIRECT fd are two different opens of the same path, not one lying open.

That leaves the real actionable from your report: the dormant aligned O_DIRECT write path in WriteBucket, and the design question of whether bucket data writes should bypass the page cache at all (the code comment currently argues no, for meta-file and padding reasons). That's the store owners' call; the instrumentation angle for you would be sampling fds in read mode during promotion hits to confirm O_DIRECT on those.

@he-yufeng commented on GitHub (Sep 27, 2026): One correction to my point 1 above, after auditing every `UringFile(..., true)` call site on current main: all five set O_DIRECT in the open flags first (storage_backend.cpp:747, 3469, 3828, 4620, 5546), so on main the log does not actually lie. The version you run (0.3.13.post1) did have the real mismatch in the allocator backend — that was #3611, already fixed and merged in #3615. For bucket files the log only fires on read-mode opens, which do carry O_DIRECT; the fds you sampled with O_WRONLY|O_CLOEXEC were the write-mode opens, which are deliberately buffered and never logged. So the "enabled" log and the no-O_DIRECT fd are two different opens of the same path, not one lying open. That leaves the real actionable from your report: the dormant aligned O_DIRECT write path in WriteBucket, and the design question of whether bucket data writes should bypass the page cache at all (the code comment currently argues no, for meta-file and padding reasons). That's the store owners' call; the instrumentation angle for you would be sampling fds in read mode during promotion hits to confirm O_DIRECT on those.

bsakaguchi commented

2026-10-29 21:22:46 +00:00

Author

Owner

@cloudomate commented on GitHub (Sep 27, 2026):

Thank you @he-yufeng for tracing this so carefully, and for the follow-up audit of the UringFile(..., true) call sites. It cleared up my misreading. My notes, point by point:

The correction is right. My sampler only ran during the spill, so it only saw the write-mode opens (O_WRONLY|O_CLOEXEC, 69 bucket files across 3 nodes). It never sampled the read-mode opens, which log "O_DIRECT mode enabled" and do carry O_DIRECT. So these were two different opens of the same path, not a log that lies. I'll retitle the issue to match.
The meta-file concern makes sense. Padding to 4096 bytes would break meta-file parsing. Meta files are small, so keeping them buffered costs almost nothing.
For data files the padding cost looks small next to the page cache.

Padding is under 4 KiB per bucket write. With our 64 MiB objects, that's under 0.01%.
Buffered bucket writes, on the other hand, held about 8 GiB of page cache per node during a sustained spill, capped only by the store container's memory limit.
Since bucket reads already use O_DIRECT, reads never use those cached pages. They mostly compete for memory with the store's own DRAM segment on the same node.
If the padding matters, the file could be truncated back to its true length after writing, or readers could rely on the lengths recorded in the meta, which they already use for offsets.

Two possible directions, both fine for us, and entirely your call:

(a) An opt-in flag (default off) that enables the dormant aligned O_DIRECT path in WriteBucket for data files only, with meta files staying buffered. This would match the direction #3615 took for the allocator backend.
(b) A smaller change: keep writes buffered and call sync_file_range() + posix_fadvise(POSIX_FADV_DONTNEED) after each bucket write. There are no alignment changes, no meta-file risk, and the page cache is released instead of building up.

Why this matters in our deployment. We run the store as a shared L3 tier behind SGLang HiCache on GPU nodes that also serve production inference. So predictable memory use on the store nodes matters more to us than peak write throughput.
Offer to help. @stmatengss, if either direction sounds reasonable, I'm happy to test a patch on our setup and report before/after write throughput and page-cache use. The setup is 0.3.13.post1, 3 nodes, ext4 on NVMe, io_uring, RDMA.

@cloudomate commented on GitHub (Sep 27, 2026): Thank you @he-yufeng for tracing this so carefully, and for the follow-up audit of the UringFile(..., true) call sites. It cleared up my misreading. My notes, point by point: 1. The correction is right. My sampler only ran during the spill, so it only saw the write-mode opens (O_WRONLY|O_CLOEXEC, 69 bucket files across 3 nodes). It never sampled the read-mode opens, which log "O_DIRECT mode enabled" and do carry O_DIRECT. So these were two different opens of the same path, not a log that lies. I'll retitle the issue to match. 2. The meta-file concern makes sense. Padding to 4096 bytes would break meta-file parsing. Meta files are small, so keeping them buffered costs almost nothing. 3. For data files the padding cost looks small next to the page cache. - Padding is under 4 KiB per bucket write. With our 64 MiB objects, that's under 0.01%. - Buffered bucket writes, on the other hand, held about 8 GiB of page cache per node during a sustained spill, capped only by the store container's memory limit. - Since bucket reads already use O_DIRECT, reads never use those cached pages. They mostly compete for memory with the store's own DRAM segment on the same node. - If the padding matters, the file could be truncated back to its true length after writing, or readers could rely on the lengths recorded in the meta, which they already use for offsets. 4. Two possible directions, both fine for us, and entirely your call: - (a) An opt-in flag (default off) that enables the dormant aligned O_DIRECT path in WriteBucket for data files only, with meta files staying buffered. This would match the direction #3615 took for the allocator backend. - (b) A smaller change: keep writes buffered and call sync_file_range() + posix_fadvise(POSIX_FADV_DONTNEED) after each bucket write. There are no alignment changes, no meta-file risk, and the page cache is released instead of building up. 5. Why this matters in our deployment. We run the store as a shared L3 tier behind SGLang HiCache on GPU nodes that also serve production inference. So predictable memory use on the store nodes matters more to us than peak write throughput. 6. Offer to help. @stmatengss, if either direction sounds reasonable, I'm happy to test a patch on our setup and report before/after write throughput and page-cache use. The setup is 0.3.13.post1, 3 nodes, ext4 on NVMe, io_uring, RDMA.

bsakaguchi referenced this issue

2026-10-29 21:32:04 +00:00

[PR #1087] [MERGED] [TE/Topology] Support device filtering when dumping topology #2007

bsakaguchi referenced this issue

2026-10-29 21:32:27 +00:00

[PR #1180] [CLOSED] [CI] add sglang e2e integration test #2057