kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4522] [Bug]: HA promotion can lose its lease during large Oplog catch-up and repeat recovery #1153

New issue

Closed

opened 2026-10-29 21:23:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:23 +00:00

Owner

Originally created by @Icedcoco on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4522

Bug Report

Summary

When a standby with a large Oplog backlog is promoted, Mooncake acquires the candidate leadership lease before completing Oplog catch-up, restore, and serving warmup. If recovery takes longer than the lease protection window, the candidate can lose leadership during warmup and return to the candidate/standby state. The already completed catch-up may then be discarded, causing the next attempt to replay the same history again.

This creates election churn and can leave the service unavailable for much longer than the time required for a single recovery attempt.

Observed behavior

During a controlled HA switch with snapshot creation disabled, repeated object cleanup had produced a large historical Oplog. The Oplog grew to approximately 255 MB, with a Raft index of about 95,443. The candidate followed this sequence:

PROMOTING -> PROMOTED
Entering warmup phase
lease ... keep alive channel closed
Leadership expired during warmup phase

Clients observed Connection refused while the node repeatedly cycled through candidate and leader-warmup states. The affected performance run could not be used as a valid result.

The failure is not limited to a slow first attempt. In a later recovery observation, the standby applied sequence progressed as follows while targeting the same durable Oplog prefix:

832,185
-> 1,200,182
-> 1,444,015
-> 205,405

The standby had nearly caught up with the durable prefix, then returned to an earlier recovery position after promotion or warmup failed. This indicates that completed catch-up progress was not retained for the next attempt.

Current recovery flow

The current supervisor flow is approximately:

Read the current master_view.
If leadership is available, call TryAcquireLeadership().
Grant an etcd lease and publish a lease-backed warming value.
Claim the producer view for serving.
Promote the standby and export its state.
Allow promotion even when the standby has a substantial replication lag.
Perform final Oplog catch-up during PreparePromotionLocked().
Enter leader_warmup, then restore state, register RPC, and run serving preflight.
Publish a routable serving endpoint only after warmup completes.

In other words, the candidate obtains leadership before it has confirmed that its applied sequence has reached the durable prefix. A large catch-up or restore can therefore outlive the lease's effective protection window. If warmup or keepalive fails, the candidate gives up leadership and the next attempt may restart from an older cursor.

Root cause

The leadership lease lifetime is not coordinated with the duration and progress of Oplog catch-up and restore. The recovery state also does not provide a monotonic handoff point that preserves completed catch-up work after a failed promotion attempt.

The current IsReadyForPromotion() behavior permits promotion with a large lag and expects the remaining Oplog to be synchronized during promotion. That policy is unsafe when the candidate's fencing or leadership lease can expire before synchronization completes.

This is not evidence that HA recovery is permanently impossible. It is a recovery-state and lease-management defect that is amplified by a large Oplog and a short lease protection window.

Expected behavior

A node must not publish a routable leader endpoint until the required Oplog catch-up, restore, producer-view handoff, and serving preflight have completed.
If the design requires a candidate to reserve leadership before catch-up, the candidate fencing lease and the serving lease should be separated, or the recovery lease must be renewed reliably for the entire catch-up and restore period.
The warming master_view must remain explicitly non-routable until the node is ready to serve.
Recovery progress must be durable and resumable. A failed promotion must not reset the standby to an older cursor when already applied Oplog entries are still valid.
Recovery must distinguish “still catching up”, “no progress”, “lease renewal failure”, and “lease definitively lost”, instead of collapsing them into a generic election retry.
A failed attempt must return safely to standby/candidate without skipping or replaying Oplog entries incorrectly.

Proposed validation

Add a regression test with:

a standby that is substantially behind a large durable Oplog;
a controlled promotion and warmup sequence;
instrumentation for applied sequence, durable-prefix sequence, lease TTL/keepalive state, promotion duration, and first serving time;
an injected warmup or lease-renewal failure after the standby has nearly caught up.

The test should verify that:

no routable leader endpoint is published before recovery is complete;
a valid lease remains active throughout the intended catch-up window, or the candidate fails with a clear diagnostic before claiming service;
a retry resumes from the latest valid applied cursor rather than an earlier recovery baseline;
after successful recovery, service_ready=true, the published master_view contains the serving endpoint, and the applied sequence remains at or beyond the durable prefix without falling back;
the client becomes usable after one successful recovery rather than requiring repeated full-history replay.

For performance and reliability tests, use an isolated Oplog namespace or compact/snapshot between cases. Otherwise, cleanup records from earlier cases can artificially inflate the recovery history and obscure the behavior being measured.

Acceptance criteria

A large Oplog catch-up cannot cause a healthy candidate to lose leadership solely because recovery outlasts the normal lease window.
A candidate that has not completed required catch-up never advertises a routable serving endpoint.
Promotion failure does not discard valid catch-up progress.
Recovery retries are monotonic from the latest valid cursor and do not repeatedly replay the same history without a state change.
Metrics and logs expose catch-up progress, durable-prefix sequence, lease state, recovery duration, retry reason, and first-serving time.
A regression test covers both a successful large-backlog promotion and a failed-then-retried promotion.

The issue concerns the interaction between the HA leadership supervisor, standby promotion and restore, the etcd lease coordinator, Oplog catch-up, and recovery cursor persistence. It is separate from data corruption: the observed problem is repeated recovery and an extended unavailable window when promotion does not complete.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @Icedcoco on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4522 ### Bug Report ### Summary When a standby with a large Oplog backlog is promoted, Mooncake acquires the candidate leadership lease before completing Oplog catch-up, restore, and serving warmup. If recovery takes longer than the lease protection window, the candidate can lose leadership during warmup and return to the candidate/standby state. The already completed catch-up may then be discarded, causing the next attempt to replay the same history again. This creates election churn and can leave the service unavailable for much longer than the time required for a single recovery attempt. ### Observed behavior During a controlled HA switch with snapshot creation disabled, repeated object cleanup had produced a large historical Oplog. The Oplog grew to approximately 255 MB, with a Raft index of about 95,443. The candidate followed this sequence: ```text PROMOTING -> PROMOTED Entering warmup phase lease ... keep alive channel closed Leadership expired during warmup phase ``` Clients observed `Connection refused` while the node repeatedly cycled through candidate and leader-warmup states. The affected performance run could not be used as a valid result. The failure is not limited to a slow first attempt. In a later recovery observation, the standby applied sequence progressed as follows while targeting the same durable Oplog prefix: ```text 832,185 -> 1,200,182 -> 1,444,015 -> 205,405 ``` The standby had nearly caught up with the durable prefix, then returned to an earlier recovery position after promotion or warmup failed. This indicates that completed catch-up progress was not retained for the next attempt. ### Current recovery flow The current supervisor flow is approximately: 1. Read the current `master_view`. 2. If leadership is available, call `TryAcquireLeadership()`. 3. Grant an etcd lease and publish a lease-backed warming value. 4. Claim the producer view for serving. 5. Promote the standby and export its state. 6. Allow promotion even when the standby has a substantial replication lag. 7. Perform final Oplog catch-up during `PreparePromotionLocked()`. 8. Enter `leader_warmup`, then restore state, register RPC, and run serving preflight. 9. Publish a routable serving endpoint only after warmup completes. In other words, the candidate obtains leadership before it has confirmed that its applied sequence has reached the durable prefix. A large catch-up or restore can therefore outlive the lease's effective protection window. If warmup or keepalive fails, the candidate gives up leadership and the next attempt may restart from an older cursor. ### Root cause The leadership lease lifetime is not coordinated with the duration and progress of Oplog catch-up and restore. The recovery state also does not provide a monotonic handoff point that preserves completed catch-up work after a failed promotion attempt. The current `IsReadyForPromotion()` behavior permits promotion with a large lag and expects the remaining Oplog to be synchronized during promotion. That policy is unsafe when the candidate's fencing or leadership lease can expire before synchronization completes. This is not evidence that HA recovery is permanently impossible. It is a recovery-state and lease-management defect that is amplified by a large Oplog and a short lease protection window. ### Expected behavior 1. A node must not publish a routable leader endpoint until the required Oplog catch-up, restore, producer-view handoff, and serving preflight have completed. 2. If the design requires a candidate to reserve leadership before catch-up, the candidate fencing lease and the serving lease should be separated, or the recovery lease must be renewed reliably for the entire catch-up and restore period. 3. The warming `master_view` must remain explicitly non-routable until the node is ready to serve. 4. Recovery progress must be durable and resumable. A failed promotion must not reset the standby to an older cursor when already applied Oplog entries are still valid. 5. Recovery must distinguish “still catching up”, “no progress”, “lease renewal failure”, and “lease definitively lost”, instead of collapsing them into a generic election retry. 6. A failed attempt must return safely to standby/candidate without skipping or replaying Oplog entries incorrectly. ### Proposed validation Add a regression test with: - a standby that is substantially behind a large durable Oplog; - a controlled promotion and warmup sequence; - instrumentation for applied sequence, durable-prefix sequence, lease TTL/keepalive state, promotion duration, and first serving time; - an injected warmup or lease-renewal failure after the standby has nearly caught up. The test should verify that: - no routable leader endpoint is published before recovery is complete; - a valid lease remains active throughout the intended catch-up window, or the candidate fails with a clear diagnostic before claiming service; - a retry resumes from the latest valid applied cursor rather than an earlier recovery baseline; - after successful recovery, `service_ready=true`, the published `master_view` contains the serving endpoint, and the applied sequence remains at or beyond the durable prefix without falling back; - the client becomes usable after one successful recovery rather than requiring repeated full-history replay. For performance and reliability tests, use an isolated Oplog namespace or compact/snapshot between cases. Otherwise, cleanup records from earlier cases can artificially inflate the recovery history and obscure the behavior being measured. ### Acceptance criteria - A large Oplog catch-up cannot cause a healthy candidate to lose leadership solely because recovery outlasts the normal lease window. - A candidate that has not completed required catch-up never advertises a routable serving endpoint. - Promotion failure does not discard valid catch-up progress. - Recovery retries are monotonic from the latest valid cursor and do not repeatedly replay the same history without a state change. - Metrics and logs expose catch-up progress, durable-prefix sequence, lease state, recovery duration, retry reason, and first-serving time. - A regression test covers both a successful large-backlog promotion and a failed-then-retried promotion. The issue concerns the interaction between the HA leadership supervisor, standby promotion and restore, the etcd lease coordinator, Oplog catch-up, and recovery cursor persistence. It is separate from data corruption: the observed problem is repeated recovery and an extended unavailable window when promotion does not complete. ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:23:23 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:23:25 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 9, 2026):

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#4522
GitHub user ID	`102317026`
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 9, 2026): Thanks for opening this issue, @Icedcoco! | Field | Value | |-------|-------| | **Issue** | #4522 | | **GitHub user ID** | `102317026` | | **Reporter** | @Icedcoco | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:19 +00:00

[PR #1153] [MERGED] [doc]merge doc to docs，and change all internal links to blog. #2041