[GH-ISSUE #4740] [Bug]: Partial SSD offload buckets lose task ownership across heartbeat cycles #1207

New issue

Open

opened 2026-10-29 21:23:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:58 +00:00

Owner

Originally created by @MarinaMiao on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4740

Supplemental reproduction for #3006. Proposed default-on tail-flush fix: #4739. Related alternatives: #3009 (carry tasks across cycles) and #4673 (opt-in tail flushing).

We found a bucket offload lifecycle problem in a Mooncake Store checkout derived from 0.3.13, with additional SSD prefetch work. Small offload batches can retain only keys and sizes in the ungrouped pool while the current batch's full task metadata is released. The objects then fail to become usable SSD replicas. Separately, after fixing persistence, a sequential benchmark can still see no SSD hits because eager writes evict cached data before it is reused.

Environment and scope

Mooncake Store 0.3.13 derived checkout, Linux aarch64, Python 3.11, Ascend direct transport, bucket storage backend, LRU bucket eviction. This is a report against the tested checkout, with native test results referring to that checkout. The same grouping/task-cleanup interaction was also inspected on upstream main at d750cddafa4b62387ca135949595008c51e05815; a native integration run of the current-main PR rebase remains pending.

The persistence reproducer uses a standalone client and Master with 11 objects of 1 MiB each, bucket_keys_limit=500, and a 256 MiB bucket byte limit. Prefetch and promotion are disabled.

Reproduction

Start an isolated Master with SSD offload enabled and an empty backend.
Put 11 known objects and allow the offload heartbeat to run.
Observe the bucket allocation output, persisted object metadata, and Master SSD byte count.
Remove only registered DRAM cache replicas while preserving the SSD backend and the live client.
Get the same 11 objects and verify their SHA256 checksums.

With the original grouping path, the incomplete group remains in the ungrouped pool. Setting the key limit to 1 makes all 11 objects persist and allows all 11 checksum checks after clearing DRAM.

With the tail flush patch and the key limit restored to 500, all 11 objects persist. Master SSD bytes are 11,534,336 before and after clearing DRAM, DRAM bytes are zero, all 11 checksums match, and offload RPC reads increase by 11. This functional reproducer used shared storage and establishes correctness only; it is not an SSD bandwidth benchmark.

Expected behavior

A nonempty offload batch should either persist its objects with their task ownership intact or report a retryable failure. An incomplete tail should not cross the task lifetime boundary as an orphaned key list.

A bucket is a storage grouping unit. Its lifecycle is associated with the offload heartbeat batch, rather than with a model forward call.

Root cause in the tested checkout

BucketStorageBackend::GroupOffloadingKeysByBucket, called by AllocateOffloadingBuckets, moves the current incomplete group into ungrouped_offloading_objects and returns without emitting a bucket when the input iterator reaches the end before the key limit.

FileStorage::OffloadObjects builds task_by_storage_key from the current batch. Its success callback needs this mapping to call NotifyOffloadSuccess with the original task identity. Keys not included in actual bucket writes are handled by the batch's failure cleanup. Keeping only keys and sizes for a later call does not preserve this task identity or the source replica pin.

Consequently, repeated small heartbeat batches can lose most or all useful persistence opportunities. This is not limited to one partially filled bucket at the end of a forward.

Proposed correctness fix

Stop filling when the iterator is exhausted, then emit the nonempty tail bucket during the same call. Reuse the existing bucket write, success notification, and failure cleanup paths. Retain the key count limit, byte limit, oversized object handling, and empty input behavior.

The patch is limited to mooncake-store/src/storage_backend.cpp and mooncake-store/tests/file_storage_test.cpp.

Validation completed on the tested checkout:

The real offload persistence regression fails before the fix and passes afterward.
FileStorage tests: 29 passed.
StorageBackend tests: 98 passed.
Standalone wheel test: 11 of 11 SSD objects and checksum reads passed with the original key limit of 500.
Full repository tests were not run.

Flushing small tails can produce smaller buckets and more I/O operations. A future cross-batch aggregation design would need to retain complete task ownership and release it correctly on all success, timeout, and failure paths.

Separate performance observation

Fixing persistence does not guarantee a speedup for a repeated sequential scan. In our workload, the unique KV working set was approximately 1,973 GiB, while available SSD cache was much smaller. Under eager Put-triggered offload and LRU eviction, the beginning of a new scan overwrote SSD objects from the previous scan before their requests were reached.

A controlled observation found all 152 SSD objects present before a DRAM-only reset were evicted about 54 seconds into the following scan, before the matching late request. The benchmark consequently recorded zero SSD lookup hits. An isolated request for an SSD-resident prompt did successfully read 152 objects, so this was distinct from a broken SSD read path.

Optional admission policy evaluated separately

We also evaluated an experimental local default-off policy controlled by MOONCAKE_OFFLOAD_READ_ADMISSION_THRESHOLD:

0 preserves eager Put-triggered offload.
A positive value counts valid demand metadata lookups for resident memory replicas and queues offload at that threshold. Our experiments use 2.
Duplicate keys within a batch count once.
Internal offload source lookups and idempotent Put healing lookups do not count as demand reads.
Existing SSD replicas and existing offload tasks are excluded.
The existing queue, source pinning, notification, and failure paths are reused.

These counts represent demand metadata lookups, rather than confirmed successful payload reads. Counters are volatile and reset when metadata is removed or the Master restarts. This policy is incompatible with offload-on-evict and is rejected in that combination.

Forty distinct targeted tests passed for admission behavior, client read-only queries, and existing offload scenarios, including a real RPC regression for idempotent Put. This optimization is a separate policy proposal; the tail flush correctness fix does not require it.

Discussion requested

Would maintainers prefer an immediate partial-bucket flush, or an aggregation design that explicitly retains full offload task ownership across batches? PR #4739 contains only the tail-flush correctness fix. The optional admission policy is outside that PR and is not claimed to be available upstream.

Originally created by @MarinaMiao on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4740 Supplemental reproduction for #3006. Proposed default-on tail-flush fix: #4739. Related alternatives: #3009 (carry tasks across cycles) and #4673 (opt-in tail flushing). We found a bucket offload lifecycle problem in a Mooncake Store checkout derived from 0.3.13, with additional SSD prefetch work. Small offload batches can retain only keys and sizes in the ungrouped pool while the current batch's full task metadata is released. The objects then fail to become usable SSD replicas. Separately, after fixing persistence, a sequential benchmark can still see no SSD hits because eager writes evict cached data before it is reused. # Environment and scope Mooncake Store 0.3.13 derived checkout, Linux aarch64, Python 3.11, Ascend direct transport, bucket storage backend, LRU bucket eviction. This is a report against the tested checkout, with native test results referring to that checkout. The same grouping/task-cleanup interaction was also inspected on upstream main at `d750cddafa4b62387ca135949595008c51e05815`; a native integration run of the current-main PR rebase remains pending. The persistence reproducer uses a standalone client and Master with 11 objects of 1 MiB each, `bucket_keys_limit=500`, and a 256 MiB bucket byte limit. Prefetch and promotion are disabled. # Reproduction 1. Start an isolated Master with SSD offload enabled and an empty backend. 2. Put 11 known objects and allow the offload heartbeat to run. 3. Observe the bucket allocation output, persisted object metadata, and Master SSD byte count. 4. Remove only registered DRAM cache replicas while preserving the SSD backend and the live client. 5. Get the same 11 objects and verify their SHA256 checksums. With the original grouping path, the incomplete group remains in the ungrouped pool. Setting the key limit to 1 makes all 11 objects persist and allows all 11 checksum checks after clearing DRAM. With the tail flush patch and the key limit restored to 500, all 11 objects persist. Master SSD bytes are 11,534,336 before and after clearing DRAM, DRAM bytes are zero, all 11 checksums match, and offload RPC reads increase by 11. This functional reproducer used shared storage and establishes correctness only; it is not an SSD bandwidth benchmark. # Expected behavior A nonempty offload batch should either persist its objects with their task ownership intact or report a retryable failure. An incomplete tail should not cross the task lifetime boundary as an orphaned key list. A bucket is a storage grouping unit. Its lifecycle is associated with the offload heartbeat batch, rather than with a model forward call. # Root cause in the tested checkout `BucketStorageBackend::GroupOffloadingKeysByBucket`, called by `AllocateOffloadingBuckets`, moves the current incomplete group into `ungrouped_offloading_objects` and returns without emitting a bucket when the input iterator reaches the end before the key limit. `FileStorage::OffloadObjects` builds `task_by_storage_key` from the current batch. Its success callback needs this mapping to call `NotifyOffloadSuccess` with the original task identity. Keys not included in actual bucket writes are handled by the batch's failure cleanup. Keeping only keys and sizes for a later call does not preserve this task identity or the source replica pin. Consequently, repeated small heartbeat batches can lose most or all useful persistence opportunities. This is not limited to one partially filled bucket at the end of a forward. # Proposed correctness fix Stop filling when the iterator is exhausted, then emit the nonempty tail bucket during the same call. Reuse the existing bucket write, success notification, and failure cleanup paths. Retain the key count limit, byte limit, oversized object handling, and empty input behavior. The patch is limited to `mooncake-store/src/storage_backend.cpp` and `mooncake-store/tests/file_storage_test.cpp`. Validation completed on the tested checkout: - The real offload persistence regression fails before the fix and passes afterward. - FileStorage tests: 29 passed. - StorageBackend tests: 98 passed. - Standalone wheel test: 11 of 11 SSD objects and checksum reads passed with the original key limit of 500. - Full repository tests were not run. Flushing small tails can produce smaller buckets and more I/O operations. A future cross-batch aggregation design would need to retain complete task ownership and release it correctly on all success, timeout, and failure paths. # Separate performance observation Fixing persistence does not guarantee a speedup for a repeated sequential scan. In our workload, the unique KV working set was approximately 1,973 GiB, while available SSD cache was much smaller. Under eager Put-triggered offload and LRU eviction, the beginning of a new scan overwrote SSD objects from the previous scan before their requests were reached. A controlled observation found all 152 SSD objects present before a DRAM-only reset were evicted about 54 seconds into the following scan, before the matching late request. The benchmark consequently recorded zero SSD lookup hits. An isolated request for an SSD-resident prompt did successfully read 152 objects, so this was distinct from a broken SSD read path. # Optional admission policy evaluated separately We also evaluated an experimental local default-off policy controlled by `MOONCAKE_OFFLOAD_READ_ADMISSION_THRESHOLD`: - 0 preserves eager Put-triggered offload. - A positive value counts valid demand metadata lookups for resident memory replicas and queues offload at that threshold. Our experiments use 2. - Duplicate keys within a batch count once. - Internal offload source lookups and idempotent Put healing lookups do not count as demand reads. - Existing SSD replicas and existing offload tasks are excluded. - The existing queue, source pinning, notification, and failure paths are reused. These counts represent demand metadata lookups, rather than confirmed successful payload reads. Counters are volatile and reset when metadata is removed or the Master restarts. This policy is incompatible with offload-on-evict and is rejected in that combination. Forty distinct targeted tests passed for admission behavior, client read-only queries, and existing offload scenarios, including a real RPC regression for idempotent Put. This optimization is a separate policy proposal; the tail flush correctness fix does not require it. # Discussion requested Would maintainers prefer an immediate partial-bucket flush, or an aggregation design that explicitly retains full offload task ownership across batches? PR #4739 contains only the tail-flush correctness fix. The optional admission policy is outside that PR and is not claimed to be available upstream.

bsakaguchi commented

2026-10-29 21:23:59 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 17, 2026):

Thanks for opening this issue, @MarinaMiao!

Field	Value
Issue	#4740
GitHub user ID	`46422913`
Reporter	@MarinaMiao

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 17, 2026): Thanks for opening this issue, @MarinaMiao! | Field | Value | |-------|-------| | **Issue** | #4740 | | **GitHub user ID** | `46422913` | | **Reporter** | @MarinaMiao | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.