kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4758] [Store][Offload] SSD reads fail with BUFFER_OVERFLOW when concurrent readers share the default offload staging buffer (1280 MiB); the log gives no hint #1214

New issue

Open

opened 2026-10-29 21:24:02 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:24:02 +00:00

Owner

Originally created by @sivabreddy on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4758

Setup: Mooncake 0.3.13.post1. The master runs --enable_offload=true --offload_on_evict=true. There are 2 store pods: mooncake_client --protocol=rdma --global_segment_size=40GB --local_buffer_size=16GB --enable_offload=true, with bucket_storage_backend on local NVMe. The engines use global_segment_size 0: vLLM 0.31.0 MooncakeStoreConnector (standalone-store, enable_offload) and SGLang 0.5.20 HiCache, with Llama-3.1-70B TP4. Pod A stores 16 prompts of 28,096 tokens each (≈ 8.6 GB KV per prompt, ≈ 138 GB in total, RAM pool 80 GB). Pod B then reads the 4 oldest prompts, which are now on SSD.

Store pod setting	Engine	4 prompts from SSD
default	vLLM	0 of 4 gave output (default `kv_load_failure_policy="fail"`). The engine logged 1,804 `SSD read failed ... BUFFER_OVERFLOW` lines. The store pods logged only `Failed to allocate batch objects`
`--local_buffer_size=16GB`	vLLM	same result (0 of 4)
default offload buffer	SGLang	8,256 / 16,384 / 28,032 / 0 of 28,096 tokens from storage; SGLang computed the rest
`MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=17179869184` (16 GiB)	both	4 of 4 fully from SSD; TTFT 0.92–1.00 s against 2.66–4.93 s compute; 0 `BUFFER_OVERFLOW`

The SSD owner serves batch_get_offload_object from one arena (FileStorageConfig::local_buffer_size, default 1280 * kMB, storage_backend.h L213). All concurrent readers share this arena. vLLM limits each TP rank's read batch to 1280 MiB (DEFAULT_MOONCAKE_DISK_STAGING_BUFFER_BYTES), so 4 ranks that read at the same time can need 4 × 1280 MiB. ssd-offload.md lists the variable as "Client-side staging buffer size". It does not say that the variable must be set on the process that owns the SSD, that all readers share the buffer, or that --local_buffer_size does not change it. #3331 already sets 64 GiB, so the need to raise it seems known but is not documented. #2119 (closed without a stated cause) shows the same Failed to allocate batch objects / BUFFER_OVERFLOW lines on another platform; this may be the same cause.

Suggestions: (1) when BUFFER_OVERFLOW occurs, log the requested bytes, the arena size and the variable name; (2) add these points and a troubleshooting entry to ssd-offload.md. We can send both as a PR.

Originally created by @sivabreddy on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4758 Setup: Mooncake 0.3.13.post1. The master runs `--enable_offload=true --offload_on_evict=true`. There are 2 store pods: `mooncake_client --protocol=rdma --global_segment_size=40GB --local_buffer_size=16GB --enable_offload=true`, with `bucket_storage_backend` on local NVMe. The engines use `global_segment_size 0`: vLLM 0.31.0 `MooncakeStoreConnector` (`standalone-store`, `enable_offload`) and SGLang 0.5.20 HiCache, with Llama-3.1-70B TP4. Pod A stores 16 prompts of 28,096 tokens each (≈ 8.6 GB KV per prompt, ≈ 138 GB in total, RAM pool 80 GB). Pod B then reads the 4 oldest prompts, which are now on SSD. | Store pod setting | Engine | 4 prompts from SSD | |---|---|---| | default | vLLM | 0 of 4 gave output (default `kv_load_failure_policy="fail"`). The engine logged 1,804 `SSD read failed ... BUFFER_OVERFLOW` lines. The store pods logged only `Failed to allocate batch objects` | | `--local_buffer_size=16GB` | vLLM | same result (0 of 4) | | default offload buffer | SGLang | 8,256 / 16,384 / 28,032 / 0 of 28,096 tokens from storage; SGLang computed the rest | | `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=17179869184` (16 GiB) | both | 4 of 4 fully from SSD; TTFT 0.92–1.00 s against 2.66–4.93 s compute; 0 `BUFFER_OVERFLOW` | The SSD owner serves `batch_get_offload_object` from one arena (`FileStorageConfig::local_buffer_size`, default `1280 * kMB`, `storage_backend.h` L213). All concurrent readers share this arena. vLLM limits each TP rank's read batch to 1280 MiB (`DEFAULT_MOONCAKE_DISK_STAGING_BUFFER_BYTES`), so 4 ranks that read at the same time can need 4 × 1280 MiB. `ssd-offload.md` lists the variable as "Client-side staging buffer size". It does not say that the variable must be set on the process that owns the SSD, that all readers share the buffer, or that `--local_buffer_size` does not change it. #3331 already sets 64 GiB, so the need to raise it seems known but is not documented. #2119 (closed without a stated cause) shows the same `Failed to allocate batch objects` / `BUFFER_OVERFLOW` lines on another platform; this may be the same cause. Suggestions: (1) when `BUFFER_OVERFLOW` occurs, log the requested bytes, the arena size and the variable name; (2) add these points and a troubleshooting entry to `ssd-offload.md`. We can send both as a PR.

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Thanks for opening this issue, @sivabreddy!

Field	Value
Issue	#4758
GitHub user ID	`113816799`
Reporter	@sivabreddy

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Thanks for opening this issue, @sivabreddy! | Field | Value | |-------|-------| | **Issue** | #4758 | | **GitHub user ID** | `113816799` | | **Reporter** | @sivabreddy | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 19, 2026):

Excellent writeup — your reading of the failure is correct. The offload read arena lives in the store pod that owns the SSD (FileStorage sizes client_buffer_allocator_ from FileStorageConfig::local_buffer_size, default 1280 MiB), every concurrent reader's staging buffers come out of that one arena, and --local_buffer_size on the engine side does not change it. With 4 TP ranks each staging up to 1280 MiB against a 1280 MiB arena, the BUFFER_OVERFLOW / Failed to allocate batch objects pair is exactly what an exhausted arena produces, and raising MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES on the SSD-owning pod is the right workaround. #2119 shows the same signature, so this is very likely the same cause.

Both of your suggestions are welcome as a PR — please do send it.

For the diagnostic: the cheapest honest spot is the failure branch of FileStorage::LoadBatch (file_storage.cpp, where it currently logs a bare "Failed to allocate batch objects"). At that point the requested per-batch size is in scope and the allocator can report its capacity, so the log can carry requested bytes, arena size, and a pointer at MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES. One request: keep the arena-capacity read cheap (a cached/config value, not a locking walk of the allocator on the error path).
For the docs: adding the ownership rule (the variable is read by the process that owns the SSD, not by the engines) and the concurrency math (concurrent readers × per-rank staging cap vs arena size) to ssd-offload.md, plus a troubleshooting entry mapping BUFFER_OVERFLOW → arena sizing, would close the gap you hit.

I'll review when it lands.

@he-yufeng commented on GitHub (Oct 19, 2026): Excellent writeup — your reading of the failure is correct. The offload read arena lives in the store pod that owns the SSD (`FileStorage` sizes `client_buffer_allocator_` from `FileStorageConfig::local_buffer_size`, default 1280 MiB), every concurrent reader's staging buffers come out of that one arena, and `--local_buffer_size` on the engine side does not change it. With 4 TP ranks each staging up to 1280 MiB against a 1280 MiB arena, the `BUFFER_OVERFLOW` / `Failed to allocate batch objects` pair is exactly what an exhausted arena produces, and raising `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES` on the SSD-owning pod is the right workaround. #2119 shows the same signature, so this is very likely the same cause. Both of your suggestions are welcome as a PR — please do send it. 1. For the diagnostic: the cheapest honest spot is the failure branch of `FileStorage::LoadBatch` (file_storage.cpp, where it currently logs a bare "Failed to allocate batch objects"). At that point the requested per-batch size is in scope and the allocator can report its capacity, so the log can carry requested bytes, arena size, and a pointer at `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES`. One request: keep the arena-capacity read cheap (a cached/config value, not a locking walk of the allocator on the error path). 2. For the docs: adding the ownership rule (the variable is read by the process that owns the SSD, not by the engines) and the concurrency math (concurrent readers × per-rank staging cap vs arena size) to `ssd-offload.md`, plus a troubleshooting entry mapping `BUFFER_OVERFLOW` → arena sizing, would close the gap you hit. I'll review when it lands.

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@he-yufeng commented on GitHub (Oct 20, 2026):

Opened #4795 for this. On arena exhaustion the read now waits (bounded by a new client_buffer_alloc_wait_ms, default 5 s — paired with the 5 s lease-GC TTL so a dead holder is always reaped within the budget) and wakes the moment a batch releases, instead of failing with BUFFER_OVERFLOW at once. The log also says what it is doing now: a WARNING with the arena size and budget when the wait engages, and an INFO on success after waiting. Two regression tests cover wait-then-succeed and wait-then-bounded-timeout, verified in a Linux container.

@he-yufeng commented on GitHub (Oct 20, 2026): Opened #4795 for this. On arena exhaustion the read now waits (bounded by a new `client_buffer_alloc_wait_ms`, default 5 s — paired with the 5 s lease-GC TTL so a dead holder is always reaped within the budget) and wakes the moment a batch releases, instead of failing with `BUFFER_OVERFLOW` at once. The log also says what it is doing now: a WARNING with the arena size and budget when the wait engages, and an INFO on success after waiting. Two regression tests cover wait-then-succeed and wait-then-bounded-timeout, verified in a Linux container.

bsakaguchi commented

2026-10-29 21:24:03 +00:00

Author

Owner

@sususama commented on GitHub (Oct 20, 2026):

I posted about 13 seconds after you, so this landed right on top of you — sorry about that.
Withdrawing my offer on the code side: #4795 already covers the bounded wait, and the new WARNING carries the arena size, which was most of what I was going to add to the LoadBatch failure branch.
What is still missing is the documentation. ssd-offload.md has none of the three items from your comment, and the new knob adds a fourth:

the ownership rule — MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES is read by the process that owns the SSD, not by the engines, and --local_buffer_size on the engine side does not affect it
the concurrency math — concurrent readers × per-rank staging cap vs arena size
a troubleshooting entry mapping BUFFER_OVERFLOW → arena sizing
MOONCAKE_OFFLOAD_CLIENT_BUFFER_ALLOC_WAIT_MS itself: the default, what the wait does, and that 0 restores the previous fail-fast behaviour
Can I take that one? Docs-only, no code, and nothing that would overlap with #4795. If you would rather it land together with #4795, say so and I will just review instead.
@sivabreddy — thank you for the writeup. It is what got this fixed.

@sususama commented on GitHub (Oct 20, 2026): I posted about 13 seconds after you, so this landed right on top of you — sorry about that. Withdrawing my offer on the code side: #4795 already covers the bounded wait, and the new WARNING carries the arena size, which was most of what I was going to add to the LoadBatch failure branch. What is still missing is the documentation. ssd-offload.md has none of the three items from your comment, and the new knob adds a fourth: - the ownership rule — MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES is read by the process that owns the SSD, not by the engines, and --local_buffer_size on the engine side does not affect it - the concurrency math — concurrent readers × per-rank staging cap vs arena size - a troubleshooting entry mapping BUFFER_OVERFLOW → arena sizing - MOONCAKE_OFFLOAD_CLIENT_BUFFER_ALLOC_WAIT_MS itself: the default, what the wait does, and that 0 restores the previous fail-fast behaviour Can I take that one? Docs-only, no code, and nothing that would overlap with #4795. If you would rather it land together with #4795, say so and I will just review instead. @sivabreddy — thank you for the writeup. It is what got this fixed.

bsakaguchi commented

2026-10-29 21:24:04 +00:00

Author

Owner

@sivabreddy commented on GitHub (Oct 20, 2026):

Thank you @he-yufeng for the quick confirmation and for #4795. A bounded wait with a WARNING that names the arena size is a better fix than the log-only change we suggested, so we will not send a separate code PR.

@sususama, please go ahead with the ssd-offload.md part. The four points in your list match what we needed in our tests. One number that may help for the concurrency example: with vLLM at TP4, each rank staged up to 1280 MiB per read batch, so 4 concurrent readers needed about 5 GiB against the default 1280 MiB arena, and 16 GiB (MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=17179869184) removed all BUFFER_OVERFLOW errors in our setup.

When #4795 is ready for testing, we can run it on the same setup (Mooncake 0.3.13.post1, 2 store pods with SSD offload, vLLM and SGLang readers) with the default arena, and report whether the 4-prompt case completes and what the wait adds to TTFT.

@sivabreddy commented on GitHub (Oct 20, 2026): Thank you @he-yufeng for the quick confirmation and for #4795. A bounded wait with a WARNING that names the arena size is a better fix than the log-only change we suggested, so we will not send a separate code PR. @sususama, please go ahead with the `ssd-offload.md` part. The four points in your list match what we needed in our tests. One number that may help for the concurrency example: with vLLM at TP4, each rank staged up to 1280 MiB per read batch, so 4 concurrent readers needed about 5 GiB against the default 1280 MiB arena, and 16 GiB (`MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=17179869184`) removed all `BUFFER_OVERFLOW` errors in our setup. When #4795 is ready for testing, we can run it on the same setup (Mooncake 0.3.13.post1, 2 store pods with SSD offload, vLLM and SGLang readers) with the default arena, and report whether the 4-prompt case completes and what the wait adds to TTFT.

bsakaguchi referenced this issue

2026-10-29 21:32:36 +00:00

[PR #1214] [MERGED] [Misc] add new dockerfile #2078