kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2807] [Roadmap] data consistency sync between etcd leader/follower in Master HA mode? #730

New issue

Open

opened 2026-10-29 21:18:48 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:18:48 +00:00

Owner

Originally created by @gitgaoqian on GitHub (Jul 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2807

Originally assigned to: @Icedcoco on GitHub.

Hi Mooncake team,

I'm currently exploring the Master HA capabilities and have a specific question regarding the roadmap for etcd integration.

Could you share if there are plans to implement strict data consistency synchronization between the etcd leader and followers within the Master HA architecture? If so, is there an estimated timeline or a target release version for this feature?

Understanding this would be very helpful for evaluating the reliability of the HA setup in production environments.

By the way, When is the next release （0.3.12） planned?
Thanks!

Originally created by @gitgaoqian on GitHub (Jul 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2807 Originally assigned to: @Icedcoco on GitHub. Hi Mooncake team, I'm currently exploring the Master HA capabilities and have a specific question regarding the roadmap for etcd integration. Could you share if there are plans to implement strict data consistency synchronization between the etcd leader and followers within the Master HA architecture? If so, is there an estimated timeline or a target release version for this feature? Understanding this would be very helpful for evaluating the reliability of the HA setup in production environments. By the way, When is the next release （0.3.12） planned? Thanks!

bsakaguchi commented

2026-10-29 21:18:49 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 9, 2026):

Thanks for opening this issue, @gitgaoqian!

Field	Value
Issue	#2807
GitHub user ID	`33947447`
Reporter	@gitgaoqian

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 9, 2026): Thanks for opening this issue, @gitgaoqian! | Field | Value | |-------|-------| | **Issue** | #2807 | | **GitHub user ID** | `33947447` | | **Reporter** | @gitgaoqian | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:49 +00:00

Author

Owner

@Icedcoco commented on GitHub (Jul 9, 2026):

Hi @gitgaoqian , Thanks for asking.

For Master HA consistency, the short answer is: the current main branch already has several HA building blocks, but I would not call it fully strict-consistent HA yet.

Current main already has:

HA leader election / master discovery through the HA backend abstraction;
etcd-backed leadership as the most complete current backend;
OpLogStore and OpLogChangeNotifier abstraction;
standby OpLog following mainly for the etcd path;
snapshot bootstrap / restore infrastructure;
standby promotion context handoff in the ongoing HA work.

The main in-progress PR is #2331. It adapts the earlier HA hot-standby / OpLog work to the current main branch and adds several missing pieces: tenant-aware OpLog metadata, segment lifecycle replication, standby snapshot export, promoted-primary restore, and more HA mutation-path tests.

There is also an active RFC, #2756, for the next consistency step. That RFC proposes replacing the current per-entry OpLog write path with an ordered batch-record writer and a durable prefix. The target is to make the etcd-backed HA path stricter around:

durable OpLog ordering;
no-gap standby replay;
promotion only up to a verified durable boundary;
fail-closed behavior when OpLog persistence is unavailable.

We expect the development work for RFC #2756 to be completed within about two weeks.

So the direction is clear: etcd-backed Master HA is the first path we are trying to make strict enough for production-style failover. Redis/K8s currently should be understood more as leadership/discovery backends, not as equivalent full OpLog-following HA backends.

For release timing, 0.3.12 does not have a confirmed release date yet. I need to check with the release maintainers before giving a concrete date.

@Icedcoco commented on GitHub (Jul 9, 2026): Hi @gitgaoqian , Thanks for asking. For Master HA consistency, the short answer is: the current main branch already has several HA building blocks, but I would not call it fully strict-consistent HA yet. Current main already has: - HA leader election / master discovery through the HA backend abstraction; - etcd-backed leadership as the most complete current backend; - OpLogStore and OpLogChangeNotifier abstraction; - standby OpLog following mainly for the etcd path; - snapshot bootstrap / restore infrastructure; - standby promotion context handoff in the ongoing HA work. The main in-progress PR is #2331. It adapts the earlier HA hot-standby / OpLog work to the current main branch and adds several missing pieces: tenant-aware OpLog metadata, segment lifecycle replication, standby snapshot export, promoted-primary restore, and more HA mutation-path tests. There is also an active RFC, #2756, for the next consistency step. That RFC proposes replacing the current per-entry OpLog write path with an ordered batch-record writer and a durable prefix. The target is to make the etcd-backed HA path stricter around: - durable OpLog ordering; - no-gap standby replay; - promotion only up to a verified durable boundary; - fail-closed behavior when OpLog persistence is unavailable. We expect the development work for RFC #2756 to be completed within about two weeks. So the direction is clear: etcd-backed Master HA is the first path we are trying to make strict enough for production-style failover. Redis/K8s currently should be understood more as leadership/discovery backends, not as equivalent full OpLog-following HA backends. For release timing, 0.3.12 does not have a confirmed release date yet. I need to check with the release maintainers before giving a concrete date.

bsakaguchi commented

2026-10-29 21:18:49 +00:00

Author

Owner

@sun-baoshan-sunshine commented on GitHub (Aug 10, 2026):

@Icedcoco
Hello, we would like to use Redis as the backend for Mooncake HA. However, during actual usage, we found that the latest version of Redis does not support HA functionality for the master node. We would like to ask whether etcd is more recommended for master HA in this case? It seems that Redis's support in this area still has many aspects that need to be improved.

@sun-baoshan-sunshine commented on GitHub (Aug 10, 2026): @Icedcoco Hello, we would like to use Redis as the backend for Mooncake HA. However, during actual usage, we found that the latest version of Redis does not support HA functionality for the master node. We would like to ask whether etcd is more recommended for master HA in this case? It seems that Redis's support in this area still has many aspects that need to be improved.

bsakaguchi commented

2026-10-29 21:18:49 +00:00

Author

Owner

@Icedcoco commented on GitHub (Aug 10, 2026):

Hi @sun-baoshan-sunshine . Thank you for reaching out! Currently, etcd is the only supported backend for reading and writing high-availability oplog information. We haven't had the capacity to build out full support for other backends yet, as integrating Redis requires implementing features like KV operations, transactions, and more.
We plan to gradually expand support for additional backends, including Redis, in future updates. In the meantime, we recommend using etcd for master HA. Hope this helps, and feel free to let us know if you have any other questions!

@Icedcoco commented on GitHub (Aug 10, 2026): Hi @sun-baoshan-sunshine . Thank you for reaching out! Currently, etcd is the only supported backend for reading and writing high-availability oplog information. We haven't had the capacity to build out full support for other backends yet, as integrating Redis requires implementing features like KV operations, transactions, and more. We plan to gradually expand support for additional backends, including Redis, in future updates. In the meantime, we recommend using etcd for master HA. Hope this helps, and feel free to let us know if you have any other questions!

bsakaguchi referenced this issue

2026-10-29 21:30:15 +00:00

[PR #730] [CLOSED] [Bugfix] GetReplicaList does not grant a lease when GC is enabled #1776