kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2971] [Bug]: ETCD OpLog is enabled implicitly for etcd HA but lacks retention and complete metadata replication #767

New issue

Open

opened 2026-10-29 21:19:17 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:19:17 +00:00

Owner

Originally created by @no-teet-on on GitHub (Jul 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2971

Originally assigned to: @Icedcoco on GitHub.

Bug Report

Description

Before the commit that introduced the MC_ENABLE_HA_METADATA_REPLICATION feature gate, selecting etcd as the HA backend automatically enabled OpLog following:

capabilities.has_oplog_following =
spec.type == HABackendType::ETCD;

As a result, every deployment using etcd for leader election also entered the etcd OpLog replication path, with no way to use etcd only for leadership coordination.

Current problems

Leader election is tightly coupled with metadata replication

Using ha_backend_type=etcd implicitly enables OpLog following. Users cannot independently enable etcd leader election while disabling the experimental metadata replication path.
OpLog entries do not have a TTL

Entries under the following prefix are written using regular etcd Put or BatchCreate operations without an attached lease:

/oplog/<cluster_id>/<sequence_id>

The following metadata keys also have no TTL:

/oplog/<cluster_id>/latest
/oplog/<cluster_id>/snapshot/<snapshot_id>/sequence_id
There is no automatic OpLog cleanup

EtcdOpLogStore::CleanupOpLogBefore() can delete entries below a sequence boundary, but it is not called by any production path. It is currently only exercised by tests.

Therefore, once production OpLog writes are enabled, the number of keys and the etcd database size can grow indefinitely.
The primary write path is not fully integrated

The production request path does not appear to:
- Create an etcd OpLog store in writer mode.
- Attach it through OpLogManager::SetOpLogStore().
- Append production metadata mutations to the OpLog.
Consequently, a standby may start an OpLog watcher even though the active primary is not publishing the required records.
Promotion does not hand replicated metadata to the serving master

HotStandbyService maintains an internal metadata store and provides ExportMetadataSnapshot(). However, the promotion path creates a new WrappedMasterService without transferring the replicated
standby metadata or its last applied sequence ID.

This means that successfully applying OpLog records in the standby does not necessarily make that state available to the newly promoted serving master.

Impact

etcd HA deployments unexpectedly depend on an incomplete replication path.
Enabling OpLog may give users a false expectation of lossless metadata failover.
Stale or empty metadata may be served after failover.
OpLog keys may accumulate indefinitely and eventually consume the etcd storage quota.
A failure to initialize the OpLog reader can prevent a standby from being safely promoted.

Expected behavior

Leader election and metadata replication should be independently configurable.

Until the metadata replication path is complete, etcd OpLog following should be disabled by default and enabled only through an explicit feature gate such as:

MC_ENABLE_HA_METADATA_REPLICATION=1

The complete implementation should also:

Wire all primary metadata mutations into an etcd OpLog writer.
Transfer replicated standby state into the promoted master.
Preserve and restore the last applied sequence ID.
Introduce a safe retention policy based on snapshot boundaries and standby progress.
Periodically invoke OpLog cleanup.
Clean obsolete snapshot sequence markers.
Expose metrics for OpLog key count, retained sequence range, and cleanup failures.
Document the etcd storage and permission requirements.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @no-teet-on on GitHub (Jul 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2971 Originally assigned to: @Icedcoco on GitHub. ### Bug Report ### Description Before the commit that introduced the MC_ENABLE_HA_METADATA_REPLICATION feature gate, selecting etcd as the HA backend automatically enabled OpLog following: capabilities.has_oplog_following = spec.type == HABackendType::ETCD; As a result, every deployment using etcd for leader election also entered the etcd OpLog replication path, with no way to use etcd only for leadership coordination. ### Current problems 1. Leader election is tightly coupled with metadata replication Using ha_backend_type=etcd implicitly enables OpLog following. Users cannot independently enable etcd leader election while disabling the experimental metadata replication path. 2. OpLog entries do not have a TTL Entries under the following prefix are written using regular etcd Put or BatchCreate operations without an attached lease: /oplog/<cluster_id>/<sequence_id> The following metadata keys also have no TTL: /oplog/<cluster_id>/latest /oplog/<cluster_id>/snapshot/<snapshot_id>/sequence_id 3. There is no automatic OpLog cleanup EtcdOpLogStore::CleanupOpLogBefore() can delete entries below a sequence boundary, but it is not called by any production path. It is currently only exercised by tests. Therefore, once production OpLog writes are enabled, the number of keys and the etcd database size can grow indefinitely. 4. The primary write path is not fully integrated The production request path does not appear to: - Create an etcd OpLog store in writer mode. - Attach it through OpLogManager::SetOpLogStore(). - Append production metadata mutations to the OpLog. Consequently, a standby may start an OpLog watcher even though the active primary is not publishing the required records. 5. Promotion does not hand replicated metadata to the serving master HotStandbyService maintains an internal metadata store and provides ExportMetadataSnapshot(). However, the promotion path creates a new WrappedMasterService without transferring the replicated standby metadata or its last applied sequence ID. This means that successfully applying OpLog records in the standby does not necessarily make that state available to the newly promoted serving master. ### Impact - etcd HA deployments unexpectedly depend on an incomplete replication path. - Enabling OpLog may give users a false expectation of lossless metadata failover. - Stale or empty metadata may be served after failover. - OpLog keys may accumulate indefinitely and eventually consume the etcd storage quota. - A failure to initialize the OpLog reader can prevent a standby from being safely promoted. ### Expected behavior Leader election and metadata replication should be independently configurable. Until the metadata replication path is complete, etcd OpLog following should be disabled by default and enabled only through an explicit feature gate such as: MC_ENABLE_HA_METADATA_REPLICATION=1 The complete implementation should also: - Wire all primary metadata mutations into an etcd OpLog writer. - Transfer replicated standby state into the promoted master. - Preserve and restore the last applied sequence ID. - Introduce a safe retention policy based on snapshot boundaries and standby progress. - Periodically invoke OpLog cleanup. - Clean obsolete snapshot sequence markers. - Expose metrics for OpLog key count, retained sequence range, and cleanup failures. - Document the etcd storage and permission requirements. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi added the

bug

label

2026-10-29 21:19:17 +00:00

bsakaguchi commented

2026-10-29 21:19:18 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 17, 2026):

Thanks for opening this issue, @no-teet-on!

Field	Value
Issue	#2971
GitHub user ID	`305664233`
Reporter	@no-teet-on

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 17, 2026): Thanks for opening this issue, @no-teet-on! | Field | Value | |-------|-------| | **Issue** | #2971 | | **GitHub user ID** | `305664233` | | **Reporter** | @no-teet-on | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:18 +00:00

Author

Owner

@Icedcoco commented on GitHub (Jul 17, 2026):

Thanks for the detailed write-up. I checked this against the current main branch, and the concerns in points 1 and 4 are valid. Leadership and metadata replication are still coupled: choosing ha_backend_type=etcd also enables OpLog following, without an independent option to turn replication off. At the same time, the primary write path is not fully wired while the standby reader can already be started. We’ll get these two issues fixed as soon as possible.

For point 2, the lack of TTL is intentional. OpLog entries are durable recovery data, so expiring them through etcd leases could leave gaps in the replay history. They should instead be removed through an explicit retention policy once a durable snapshot covers the relevant sequence range.

Point 3 is also valid: automatic cleanup has not been implemented yet. The intended approach is to clean up old OpLog entries after a snapshot is safely persisted, while making sure lagging standbys can either continue replaying or bootstrap from a retained snapshot.

Point 5 is being addressed in #2826, which passes the replicated metadata, segment state, and last applied sequence ID into the promoted master. Please feel free to take a look and leave comments if you spot anything we missed.

@Icedcoco commented on GitHub (Jul 17, 2026): Thanks for the detailed write-up. I checked this against the current `main` branch, and the concerns in points 1 and 4 are valid. Leadership and metadata replication are still coupled: choosing `ha_backend_type=etcd` also enables OpLog following, without an independent option to turn replication off. At the same time, the primary write path is not fully wired while the standby reader can already be started. We’ll get these two issues fixed as soon as possible. For point 2, the lack of TTL is intentional. OpLog entries are durable recovery data, so expiring them through etcd leases could leave gaps in the replay history. They should instead be removed through an explicit retention policy once a durable snapshot covers the relevant sequence range. Point 3 is also valid: automatic cleanup has not been implemented yet. The intended approach is to clean up old OpLog entries after a snapshot is safely persisted, while making sure lagging standbys can either continue replaying or bootstrap from a retained snapshot. Point 5 is being addressed in #2826, which passes the replicated metadata, segment state, and last applied sequence ID into the promoted master. Please feel free to take a look and leave comments if you spot anything we missed.

bsakaguchi commented

2026-10-29 21:19:18 +00:00

Author

Owner

@Colors-111 commented on GitHub (Aug 21, 2026):

Thanks for the detailed write-up. I checked this against the current main branch, and the concerns in points 1 and 4 are valid. Leadership and metadata replication are still coupled: choosing ha_backend_type=etcd also enables OpLog following, without an independent option to turn replication off. At the same time, the primary write path is not fully wired while the standby reader can already be started. We’ll get these two issues fixed as soon as possible.

For point 2, the lack of TTL is intentional. OpLog entries are durable recovery data, so expiring them through etcd leases could leave gaps in the replay history. They should instead be removed through an explicit retention policy once a durable snapshot covers the relevant sequence range.

Point 3 is also valid: automatic cleanup has not been implemented yet. The intended approach is to clean up old OpLog entries after a snapshot is safely persisted, while making sure lagging standbys can either continue replaying or bootstrap from a retained snapshot.

Point 5 is being addressed in #2826, which passes the replicated metadata, segment state, and last applied sequence ID into the promoted master. Please feel free to take a look and leave comments if you spot anything we missed.

I found the issue problem 1 still reproduces in the current codebase f90ae691f109e49a60920e0c8abbf7e572826d8c: with only --enable_ha=true, Mooncake fails to start up normally — OpLog must also be enabled. Could you confirm whether this is by design? I think it would be better make --enable_ha work without requiring OpLog.

@Colors-111 commented on GitHub (Aug 21, 2026): > Thanks for the detailed write-up. I checked this against the current `main` branch, and the concerns in points 1 and 4 are valid. Leadership and metadata replication are still coupled: choosing `ha_backend_type=etcd` also enables OpLog following, without an independent option to turn replication off. At the same time, the primary write path is not fully wired while the standby reader can already be started. We’ll get these two issues fixed as soon as possible. > > For point 2, the lack of TTL is intentional. OpLog entries are durable recovery data, so expiring them through etcd leases could leave gaps in the replay history. They should instead be removed through an explicit retention policy once a durable snapshot covers the relevant sequence range. > > Point 3 is also valid: automatic cleanup has not been implemented yet. The intended approach is to clean up old OpLog entries after a snapshot is safely persisted, while making sure lagging standbys can either continue replaying or bootstrap from a retained snapshot. > > Point 5 is being addressed in [#2826](https://github.com/kvcache-ai/Mooncake/pull/2826), which passes the replicated metadata, segment state, and last applied sequence ID into the promoted master. Please feel free to take a look and leave comments if you spot anything we missed. I found the issue problem 1 still reproduces in the current codebase `f90ae691f109e49a60920e0c8abbf7e572826d8c`: with only `--enable_ha=true`, Mooncake fails to start up normally — OpLog must also be enabled. Could you confirm whether this is by design? I think it would be better make `--enable_ha` work without requiring OpLog.

bsakaguchi commented

2026-10-29 21:19:18 +00:00

Author

Owner

@Icedcoco commented on GitHub (Aug 21, 2026):

I found the issue problem 1 still reproduces in the current codebase f90ae691f109e49a60920e0c8abbf7e572826d8c: with only --enable_ha=true, Mooncake fails to start up normally — OpLog must also be enabled. Could you confirm whether this is by design? I think it would be better make --enable_ha work without requiring OpLog.

Thanks for catching this. I confirmed that the failure is in the promotion path, not caused by OpLog enablement. When only enable_ha=true is set, the no-op standby controller cannot complete PromoteStandbyAndExport(), so the elected leader never enters the serving state.

This is not intended: enable_ha should work independently without requiring OpLog. We will fix it immediately and add a regression test for the HA-only configuration.

@Icedcoco commented on GitHub (Aug 21, 2026): > I found the issue problem 1 still reproduces in the current codebase `f90ae691f109e49a60920e0c8abbf7e572826d8c`: with only `--enable_ha=true`, Mooncake fails to start up normally — OpLog must also be enabled. Could you confirm whether this is by design? I think it would be better make `--enable_ha` work without requiring OpLog. Thanks for catching this. I confirmed that the failure is in the promotion path, not caused by OpLog enablement. When only `enable_ha=true` is set, the no-op standby controller cannot complete `PromoteStandbyAndExport()`, so the elected leader never enters the serving state. This is not intended: `enable_ha` should work independently without requiring OpLog. We will fix it immediately and add a regression test for the HA-only configuration.

bsakaguchi referenced this issue

2026-10-29 21:30:31 +00:00

[PR #767] [MERGED] [TransferEngine] Fix SIEVE eviction algorithm for RDMA endpoint store #1808