[GH-ISSUE #850] [Feature Request]: Avoid data corruption in slow `Get` #258

New issue

Closed

opened 2026-10-29 21:11:37 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:11:37 +00:00

Owner

Originally created by @nickyc975 on GitHub (Sep 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/850

Describe your feature request

Currently Mooncake Store relies on lease to protect objects from eviction and removing during data transferring. However, if data transfer takes too long (because of huge object or poor network), objects may be evicted or removed before the transfer is completed, causing the client to read corrupted data.

A simple solution is to check whether the objects are still valid after completing a long-time transmission, then discard read data of invalid objects. This may slightly degrade the performance, so maybe we should make it configurable.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @nickyc975 on GitHub (Sep 17, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/850 ### Describe your feature request Currently Mooncake Store relies on lease to protect objects from eviction and removing during data transferring. However, if data transfer takes too long (because of huge object or poor network), objects may be evicted or removed before the transfer is completed, causing the client to read corrupted data. A simple solution is to check whether the objects are still valid after completing a long-time transmission, then discard read data of invalid objects. This may slightly degrade the performance, so maybe we should make it configurable. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:11:37 +00:00

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 18, 2025):

That’s a great suggestion. In fact, this work is already in progress: #778. It was just delayed due to some other tasks, but I’ll make sure to complete it soon.

@ykwd commented on GitHub (Sep 18, 2025): That’s a great suggestion. In fact, this work is already in progress: #778. It was just delayed due to some other tasks, but I’ll make sure to complete it soon.

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@friday0513 commented on GitHub (Sep 19, 2025):

"Double check" may not be a perfect solution: between transferring the the post check, many things may happen.
i.e., that slice of memory may be allocated for another token, or the evicted KVCache may be rewrite to a new address by other clients, both may cause mistakes for the checking.

@friday0513 commented on GitHub (Sep 19, 2025): "Double check" may not be a perfect solution: between transferring the the post check, many things may happen. i.e., that slice of memory may be allocated for another token, or the evicted KVCache may be rewrite to a new address by other clients, both may cause mistakes for the checking.

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 25, 2025):

Thanks for the suggestion! This feature has now been implemented #778

@ykwd commented on GitHub (Sep 25, 2025): Thanks for the suggestion! This feature has now been implemented #778

bsakaguchi referenced this issue

2026-10-29 21:27:23 +00:00

[PR #258] [MERGED] [TransferEngine] bugfix: fix regression in #244 #1431