[GH-ISSUE #3401] [Bug]: Replication tasks retain targets and reserved quota after source cleanup #870

New issue

Open

opened 2026-10-29 21:20:22 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:20:22 +00:00

Owner

Originally created by @Aionw on GitHub (Aug 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3401

Bug Report

While reviewing #2877, I found a pre-existing replication-task lifecycle gap. This is not caused by making segment cleanup asynchronous, but async cleanup makes the retained state easier to observe.

When an in-flight Copy/Move source becomes unavailable, CleanupStaleHandles() removes the invalid COMPLETE source replica. It only cancels promotion tasks for removed replica IDs, so a replication task whose source_id was removed is left behind together with its PROCESSING target replicas and reserved tenant quota. Those resources remain until the client calls CopyEnd/MoveEnd/CopyRevoke/MoveRevoke or DiscardExpiredProcessingReplicas() reaches the task timeout.

Reproduction

A deterministic unit test can use the existing MasterServiceTest fixture:

Create a non-HA MasterService with a tenant quota large enough for exactly the source and one temporary target.
Mount source_segment and target_segment.
PutStart + PutEnd a key on source_segment.
Call either:
- CopyStart(client, key, ..., "source_segment", {"target_segment"}), or
- MoveStart(client, key, ..., "source_segment", "target_segment").
Unmount source_segment and wait until invalid-handle cleanup has removed the source replica.
Do not call the corresponding End/Revoke API.
Inspect the key's TenantState, or attempt another allocation under the strict quota.

Observed state after step 5:

the source replica is absent;
the target replica is still PROCESSING;
tenant_state.replication_tasks still contains the key;
reserved_quota_charge_bytes is still held;
physical metadata remains until the replication-task timeout.

A compact assertion sketch:

ASSERT_TRUE(service.CopyStart(client, key, tenant, src, {dst}).has_value());
ASSERT_TRUE(service.UnmountSegment(src_id, src_owner).has_value());
WaitUntil([&] { return source_replica_is_gone(); });

EXPECT_TRUE(tenant_state.replication_tasks.contains(key));
EXPECT_TRUE(target_replica_is_processing());
EXPECT_GT(tenant_state.replication_tasks.at(key)
              .reserved_quota_charge_bytes,
          0);

Expected behavior

Removing a replica that is the source of a Copy/Move task should invalidate that task promptly, discard its PROCESSING targets, release its reserved quota, and balance the source reference if it still exists.

Relevant code

MasterService::CleanupStaleHandles()
MasterService::CancelPromotionTaskForRemovedReplicas()
MasterService::DiscardExpiredProcessingReplicas()

This should be handled separately from #2877 so that the asynchronous-unmount PR can remain focused on RPC latency and immediate logical invisibility.

Originally created by @Aionw on GitHub (Aug 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3401 ## Bug Report While reviewing #2877, I found a pre-existing replication-task lifecycle gap. This is not caused by making segment cleanup asynchronous, but async cleanup makes the retained state easier to observe. When an in-flight Copy/Move source becomes unavailable, `CleanupStaleHandles()` removes the invalid COMPLETE source replica. It only cancels promotion tasks for removed replica IDs, so a replication task whose `source_id` was removed is left behind together with its PROCESSING target replicas and reserved tenant quota. Those resources remain until the client calls `CopyEnd/MoveEnd/CopyRevoke/MoveRevoke` or `DiscardExpiredProcessingReplicas()` reaches the task timeout. ### Reproduction A deterministic unit test can use the existing `MasterServiceTest` fixture: 1. Create a non-HA `MasterService` with a tenant quota large enough for exactly the source and one temporary target. 2. Mount `source_segment` and `target_segment`. 3. `PutStart` + `PutEnd` a key on `source_segment`. 4. Call either: - `CopyStart(client, key, ..., "source_segment", {"target_segment"})`, or - `MoveStart(client, key, ..., "source_segment", "target_segment")`. 5. Unmount `source_segment` and wait until invalid-handle cleanup has removed the source replica. 6. Do not call the corresponding End/Revoke API. 7. Inspect the key's `TenantState`, or attempt another allocation under the strict quota. Observed state after step 5: - the source replica is absent; - the target replica is still PROCESSING; - `tenant_state.replication_tasks` still contains the key; - `reserved_quota_charge_bytes` is still held; - physical metadata remains until the replication-task timeout. A compact assertion sketch: ```cpp ASSERT_TRUE(service.CopyStart(client, key, tenant, src, {dst}).has_value()); ASSERT_TRUE(service.UnmountSegment(src_id, src_owner).has_value()); WaitUntil([&] { return source_replica_is_gone(); }); EXPECT_TRUE(tenant_state.replication_tasks.contains(key)); EXPECT_TRUE(target_replica_is_processing()); EXPECT_GT(tenant_state.replication_tasks.at(key) .reserved_quota_charge_bytes, 0); ``` ### Expected behavior Removing a replica that is the source of a Copy/Move task should invalidate that task promptly, discard its PROCESSING targets, release its reserved quota, and balance the source reference if it still exists. ### Relevant code - `MasterService::CleanupStaleHandles()` - `MasterService::CancelPromotionTaskForRemovedReplicas()` - `MasterService::DiscardExpiredProcessingReplicas()` This should be handled separately from #2877 so that the asynchronous-unmount PR can remain focused on RPC latency and immediate logical invisibility.

bsakaguchi added the

bug

label

2026-10-29 21:20:22 +00:00

bsakaguchi commented

2026-10-29 21:20:23 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 12, 2026):

Thanks for opening this issue, @Aionw!

Field	Value
Issue	#3401
GitHub user ID	`41376987`
Reporter	@Aionw

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 12, 2026): Thanks for opening this issue, @Aionw! | Field | Value | |-------|-------| | **Issue** | #3401 | | **GitHub user ID** | `41376987` | | **Reporter** | @Aionw | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:31:03 +00:00

[PR #870] [MERGED] chore: bump version to 0.3.6.post1 in pyproject.toml #1878