[GH-ISSUE #3561] Regarding Mooncake's HA (High Availability) capability, there are still some issues to be resolved as of now. #913

New issue

Open

opened 2026-10-29 21:20:51 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:20:51 +00:00

Owner

Originally created by @sun-baoshan-sunshine on GitHub (Aug 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3561

While reviewing the code on the main branch, we analyzed Mooncake's HA capability and identified several outstanding issues:

The oplog only supports etcd mode, and each time it must perform a cold start from seq 0. When the oplog volume is large, this becomes a serious problem.
Due to the issue above, there is currently no GC (garbage collection) mechanism for oplog data in etcd.
The snapshot is only read once during standby initialization and is never read again afterward. If a failover occurs, this will cause a significant gap between the standby and the leader.
Currently, snapshot and oplog operate in a mutually exclusive (either/or) mode. When both are enabled simultaneously, the leader/master will stop writing snapshots entirely.
Could the community share its specific plans to address these issues? At this point, it doesn't appear that the current HA implementation offers a robust solution to mitigate these shortcomings. We would greatly appreciate any feedback or insights!

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @sun-baoshan-sunshine on GitHub (Aug 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3561 While reviewing the code on the main branch, we analyzed Mooncake's HA capability and identified several outstanding issues: 1. The oplog only supports etcd mode, and each time it must perform a cold start from seq 0. When the oplog volume is large, this becomes a serious problem. 2. Due to the issue above, there is currently no GC (garbage collection) mechanism for oplog data in etcd. 3. The snapshot is only read once during standby initialization and is never read again afterward. If a failover occurs, this will cause a significant gap between the standby and the leader. 4. Currently, snapshot and oplog operate in a mutually exclusive (either/or) mode. When both are enabled simultaneously, the leader/master will stop writing snapshots entirely. Could the community share its specific plans to address these issues? At this point, it doesn't appear that the current HA implementation offers a robust solution to mitigate these shortcomings. We would greatly appreciate any feedback or insights! ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:20:51 +00:00

bsakaguchi commented

2026-10-29 21:20:52 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 21, 2026):

Thanks for opening this issue, @sun-baoshan-sunshine!

Field	Value
Issue	#3561
GitHub user ID	`78401405`
Reporter	@sun-baoshan-sunshine

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Aug 21, 2026): Thanks for opening this issue, @sun-baoshan-sunshine! | Field | Value | |-------|-------| | **Issue** | #3561 | | **GitHub user ID** | `78401405` | | **Reporter** | @sun-baoshan-sunshine | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:20:52 +00:00

Author

Owner

@Icedcoco commented on GitHub (Aug 21, 2026):

Thanks for raising these points. Your assessment is accurate for the current legacy HA path on main. These limitations are also the motivation for RFC #3167, and the replacement path is under active development. It is not production-complete yet.

The work completed so far includes:

Batch-record OpLog persistence and writer safety improvements, including durable-prefix validation, retry/failure handling, and producer-view fencing: #2826, #3165, #3166, #3201, #3204, and #3384.
The new batch-OpLog snapshot metadata protocol: #3178.
Bounded snapshot capture from standby state at a complete applied batch boundary: #3326.
Chunked, checksummed snapshot artifact generation for local/S3 object storage: #3447.
Explicit restore error propagation: #3354. Gating HA serving on successful standby restore is currently under review in #3497.

The remaining RFC work is still in development. It includes fenced publication through independent latest/fallback snapshot pointers, bootstrap from a snapshot followed by OpLog suffix replay, periodic standby snapshot scheduling, production wiring and real-etcd E2E validation, floor-aware online rebootstrap, guarded OpLog pruning, snapshot object GC, and etcd capacity operations.

More specifically:

The new recovery path will restore a snapshot baseline and replay only the OpLog suffix after its complete batch cursor. Full replay from batch 1 is retained only as a validated fallback when complete history is still available.
OpLog deletion will be enabled only after two independently verified snapshots exist and all eligible readers understand the published compaction floor. The floor is published before covered batches are deleted.
A healthy standby is expected to keep applying the batch OpLog rather than periodically reload snapshots. Snapshots are used for initial bootstrap and for online rebootstrap when a standby falls behind the compaction floor. Promotion must also complete recovery before the node starts serving.
We are not combining the two existing legacy flags. The new implementation is a separate standby-generated snapshot mode integrated with the batch OpLog path. The legacy snapshot and new batch-OpLog snapshot systems remain isolated to avoid ambiguous recovery authority.

The first implementation remains etcd-based; support for additional coordination backends is outside the initial scope of RFC #3167.

Therefore, bounded OpLog retention and integrated snapshot-plus-suffix recovery should not yet be considered available on main, even though several foundational PRs have landed. We are landing the work incrementally and will expose the new mode only after the recovery, pruning, failure-safety, and E2E stages are complete.

@Icedcoco commented on GitHub (Aug 21, 2026): Thanks for raising these points. Your assessment is accurate for the current legacy HA path on `main`. These limitations are also the motivation for RFC #3167, and the replacement path is under active development. It is not production-complete yet. The work completed so far includes: - Batch-record OpLog persistence and writer safety improvements, including durable-prefix validation, retry/failure handling, and producer-view fencing: #2826, #3165, #3166, #3201, #3204, and #3384. - The new batch-OpLog snapshot metadata protocol: #3178. - Bounded snapshot capture from standby state at a complete applied batch boundary: #3326. - Chunked, checksummed snapshot artifact generation for local/S3 object storage: #3447. - Explicit restore error propagation: #3354. Gating HA serving on successful standby restore is currently under review in #3497. The remaining RFC work is still in development. It includes fenced publication through independent `latest`/`fallback` snapshot pointers, bootstrap from a snapshot followed by OpLog suffix replay, periodic standby snapshot scheduling, production wiring and real-etcd E2E validation, floor-aware online rebootstrap, guarded OpLog pruning, snapshot object GC, and etcd capacity operations. More specifically: 1. The new recovery path will restore a snapshot baseline and replay only the OpLog suffix after its complete batch cursor. Full replay from batch 1 is retained only as a validated fallback when complete history is still available. 2. OpLog deletion will be enabled only after two independently verified snapshots exist and all eligible readers understand the published compaction floor. The floor is published before covered batches are deleted. 3. A healthy standby is expected to keep applying the batch OpLog rather than periodically reload snapshots. Snapshots are used for initial bootstrap and for online rebootstrap when a standby falls behind the compaction floor. Promotion must also complete recovery before the node starts serving. 4. We are not combining the two existing legacy flags. The new implementation is a separate standby-generated snapshot mode integrated with the batch OpLog path. The legacy snapshot and new batch-OpLog snapshot systems remain isolated to avoid ambiguous recovery authority. The first implementation remains etcd-based; support for additional coordination backends is outside the initial scope of RFC #3167. Therefore, bounded OpLog retention and integrated snapshot-plus-suffix recovery should not yet be considered available on `main`, even though several foundational PRs have landed. We are landing the work incrementally and will expose the new mode only after the recovery, pruning, failure-safety, and E2E stages are complete.

bsakaguchi referenced this issue

2026-10-29 21:31:13 +00:00

[PR #913] [MERGED] [Doc] Add Mooncake x SGLang Hicache Design and Some Updates #1899