[GH-ISSUE #2963] [Bug]: OffsetAllocator eviction loses local replicas when notification fails #768

New issue

Closed

opened 2026-10-29 21:19:17 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:19:17 +00:00

Owner

Originally created by @feichai0017 on GitHub (Jul 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2963

Bug report

OffsetAllocatorStorageBackend::EvictToMakeRoom() removes FIFO victims from the shard map and counters before BatchOffload() calls the eviction handler. If that handler returns an error, BatchOffload() returns immediately without restoring the victims.

This leaves the master free to retain a LOCAL_DISK replica for an object that the storage backend no longer contains. The released extent can also be reused by a later write.

Reproduction

Enable FIFO eviction with a small byte watermark.
Write enough objects to cross the high watermark.
Make the eviction handler return INTERNAL_ERROR.
The offload fails as expected, but IsExist() and BatchLoad() for the selected FIFO victim now fail.

A focused regression test reproduces this deterministically: after the failed notification, key_a is missing instead of remaining readable.

Expected behavior

Keep selected victims and their allocation handles pending until the master accepts the replica-removal notification. On notification failure, restore the shard map, FIFO index, byte count, and key count; on success, release the pending allocations before retrying allocation.

Existing work

I searched open issues and PRs for OffsetAllocator eviction notification/rollback and found no existing fix. #2932 touches the same backend for persistence, but its current eviction path preserves the same return-without-rollback behavior, so it does not address this bug.

Originally created by @feichai0017 on GitHub (Jul 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2963 ## Bug report `OffsetAllocatorStorageBackend::EvictToMakeRoom()` removes FIFO victims from the shard map and counters before `BatchOffload()` calls the eviction handler. If that handler returns an error, `BatchOffload()` returns immediately without restoring the victims. This leaves the master free to retain a `LOCAL_DISK` replica for an object that the storage backend no longer contains. The released extent can also be reused by a later write. ## Reproduction 1. Enable FIFO eviction with a small byte watermark. 2. Write enough objects to cross the high watermark. 3. Make the eviction handler return `INTERNAL_ERROR`. 4. The offload fails as expected, but `IsExist()` and `BatchLoad()` for the selected FIFO victim now fail. A focused regression test reproduces this deterministically: after the failed notification, `key_a` is missing instead of remaining readable. ## Expected behavior Keep selected victims and their allocation handles pending until the master accepts the replica-removal notification. On notification failure, restore the shard map, FIFO index, byte count, and key count; on success, release the pending allocations before retrying allocation. ## Existing work I searched open issues and PRs for OffsetAllocator eviction notification/rollback and found no existing fix. #2932 touches the same backend for persistence, but its current eviction path preserves the same return-without-rollback behavior, so it does not address this bug.

bsakaguchi closed this issue

2026-10-29 21:19:17 +00:00

bsakaguchi commented

2026-10-29 21:19:18 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Thanks for opening this issue, @feichai0017!

Field	Value
Issue	#2963
GitHub user ID	`132118805`
Reporter	@feichai0017

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 16, 2026): Thanks for opening this issue, @feichai0017! | Field | Value | |-------|-------| | **Issue** | #2963 | | **GitHub user ID** | `132118805` | | **Reporter** | @feichai0017 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:30:31 +00:00

[PR #768] [MERGED] [TE][EndpointStore]: Fix hand_ assignment after evict #1809