[GH-ISSUE #849] [Feature Request]: Cleanup invalid objects when client terminated right after `PutStart` #257

New issue

Closed

opened 2026-10-29 21:11:37 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:11:37 +00:00

Owner

Originally created by @nickyc975 on GitHub (Sep 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/849

Describe your feature request

Currently in Mooncake Store, if a client got killed right after PutStart, the incomplete object would be left in master forever, and later PutStart operations on the same key would all fail.

A possible solution is to record the 'owner client' of each processing object, and clear these objects when the corresponding client is expired due to pinging timedout.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @nickyc975 on GitHub (Sep 17, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/849 ### Describe your feature request Currently in Mooncake Store, if a client got killed right after `PutStart`, the incomplete object would be left in master forever, and later `PutStart` operations on the same key would all fail. A possible solution is to record the 'owner client' of each processing object, and clear these objects when the corresponding client is expired due to pinging timedout. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi

2026-10-29 21:11:37 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@ykwd commented on GitHub (Sep 18, 2025):

Thanks for bringing this up. A simple temporary workaround is to set a sufficiently long timeout (e.g., 10 minutes). However, this still cannot fully guarantee data integrity, since there’s no reliable mechanism to ensure that the RDMA write operation has stopped before the given timeout. Another potential approach is to switch from a put model to a get model—i.e., when A wants to put a slice to B, it instead notifies B to fetch the slice from A. But this would significantly increase system complexity and could also impact performance. We do plan to address this issue, though the exact solution is not yet finalized.

@ykwd commented on GitHub (Sep 18, 2025): Thanks for bringing this up. A simple temporary workaround is to set a sufficiently long timeout (e.g., 10 minutes). However, this still cannot fully guarantee data integrity, since there’s no reliable mechanism to ensure that the RDMA write operation has stopped before the given timeout. Another potential approach is to switch from a put model to a get model—i.e., when A wants to put a slice to B, it instead notifies B to fetch the slice from A. But this would significantly increase system complexity and could also impact performance. We do plan to address this issue, though the exact solution is not yet finalized.

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 27, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 27, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:11:39 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 5, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 5, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.