[GH-ISSUE #4722] [Bug][Store] Tenant-watermark eviction removes objects without an OpLog record and releases quota before durability #1201

New issue

Open

opened 2026-10-29 21:23:54 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:54 +00:00

Owner

Originally created by @apex-mochen on GitHub (Oct 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4722

Summary

With enable_ha + enable_oplog + enable_multi_tenants, the eviction thread's tenant pass (EvictTenantsOverWatermark → EvictTenantMemoryForQuota, on by default at tenant_eviction_high_watermark_ratio = 0.90) removes memory replicas, releases the tenant's committed quota and erases the metadata without appending any OpLog entry.

BatchEvict handles the same removal differently: persist_evict_oplog_or_skip marks the replicas REMOVED, appends REMOVE / PUT_END with a durable finalize, and frees the replicas and quota only in FinalizeRemovedReplicasAfterDurable ("HA strong-consistency: persist the post-eviction state before the helper mutates metadata"). The tenant pass has no equivalent step.

Checked on main 20fc0bc and again on 129c961.

Reproduction (deterministic)

Regression test in master_service_evict_scenario_test.cpp (in the linked PR):

HA config, tenant-a quota = 1 KiB, one completed, lease-expired 1 KiB object cold;
no explicit eviction: wait 2 s so the tenant pass runs;
cold no longer exists, but OpLog batch 3 never appears (ETCD_KEY_NOT_EXIST).

On unfixed main the test fails 5/5, both on 20fc0bc and on 129c961.

Where it already shows up

MasterServiceEvictScenarioTest.OpLogDurabilityGatesTenantQuotaReclamation fails intermittently: nightly run 37339233838, triaged in #4485; a PR run on 2026-09-15 is also mentioned in #4136.
The failure always produces the same three messages:
- before-durable succeeded
- batch 3 ETCD_KEY_NOT_EXIST
- after-durable TENANT_QUOTA_EXCEEDED
With 32 CPU-burning processes (on 20fc0bc), 33 of 50 repeats fail with exactly these messages; idle, 50/50 pass. No failing iteration contains [EVICT-RESULT], so the explicit EvictMemory found nothing to evict, because the tenant pass had already removed cold.
#4155 isolated two HA tests from the same interaction ("Background eviction can remove that object and release its quota before the tested operation, bypassing the blocked OpLog transaction"). It treated this as a test issue and left production unchanged.

Possible consequence (not reproduced end to end)

A standby replays only the OpLog, so it would keep the evicted key with its old memory descriptor. If the primary later reuses that address, the standby could hold two live-looking descriptors for one range. That is the "overlapping memory descriptors" condition from #3760, where the maintainer note says overlap "is not expected during normal operation" and the source of the stale descriptor was still open. #3760's cluster ran with -enable_multi_tenants=true. I don't know whether a tenant quota was configured there, so I can't say this path caused it; it is one path that can produce the overlap.

Proposed fix

Route the tenant pass through the same persist-before-mutate step as BatchEvict (shared helper), and leave quota release, metadata erase and the KV-removed publish to the durable finalize when the OpLog is on. I have this fix plus the regression test ready and will open a PR shortly.

Originally created by @apex-mochen on GitHub (Oct 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4722 ### Summary With `enable_ha` + `enable_oplog` + `enable_multi_tenants`, the eviction thread's tenant pass (`EvictTenantsOverWatermark` → `EvictTenantMemoryForQuota`, on by default at `tenant_eviction_high_watermark_ratio = 0.90`) removes memory replicas, releases the tenant's committed quota and erases the metadata **without appending any OpLog entry**. `BatchEvict` handles the same removal differently: `persist_evict_oplog_or_skip` marks the replicas `REMOVED`, appends `REMOVE` / `PUT_END` with a durable finalize, and frees the replicas and quota only in `FinalizeRemovedReplicasAfterDurable` ("HA strong-consistency: persist the post-eviction state before the helper mutates metadata"). The tenant pass has no equivalent step. Checked on main `20fc0bc` and again on `129c961`. ### Reproduction (deterministic) Regression test in `master_service_evict_scenario_test.cpp` (in the linked PR): - HA config, `tenant-a` quota = 1 KiB, one completed, lease-expired 1 KiB object `cold`; - no explicit eviction: wait 2 s so the tenant pass runs; - `cold` no longer exists, but OpLog batch 3 never appears (`ETCD_KEY_NOT_EXIST`). On unfixed main the test fails 5/5, both on `20fc0bc` and on `129c961`. ### Where it already shows up - `MasterServiceEvictScenarioTest.OpLogDurabilityGatesTenantQuotaReclamation` fails intermittently: nightly run 37339233838, triaged in #4485; a PR run on 2026-09-15 is also mentioned in #4136. - The failure always produces the same three messages: - `before-durable succeeded` - batch 3 `ETCD_KEY_NOT_EXIST` - `after-durable TENANT_QUOTA_EXCEEDED` - With 32 CPU-burning processes (on `20fc0bc`), 33 of 50 repeats fail with exactly these messages; idle, 50/50 pass. No failing iteration contains `[EVICT-RESULT]`, so the explicit `EvictMemory` found nothing to evict, because the tenant pass had already removed `cold`. - #4155 isolated two HA tests from the same interaction ("Background eviction can remove that object and release its quota before the tested operation, bypassing the blocked OpLog transaction"). It treated this as a test issue and left production unchanged. ### Possible consequence (not reproduced end to end) A standby replays only the OpLog, so it would keep the evicted key with its old memory descriptor. If the primary later reuses that address, the standby could hold two live-looking descriptors for one range. That is the "overlapping memory descriptors" condition from #3760, where the maintainer note says overlap "is not expected during normal operation" and the source of the stale descriptor was still open. #3760's cluster ran with `-enable_multi_tenants=true`. I don't know whether a tenant quota was configured there, so I can't say this path caused it; it is one path that can produce the overlap. ### Proposed fix Route the tenant pass through the same persist-before-mutate step as `BatchEvict` (shared helper), and leave quota release, metadata erase and the KV-removed publish to the durable finalize when the OpLog is on. I have this fix plus the regression test ready and will open a PR shortly.

bsakaguchi commented

2026-10-29 21:23:54 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 16, 2026):

Thanks for opening this issue, @apex-mochen!

Field	Value
Issue	#4722
GitHub user ID	`265981728`
Reporter	@apex-mochen

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 16, 2026): Thanks for opening this issue, @apex-mochen! | Field | Value | |-------|-------| | **Issue** | #4722 | | **GitHub user ID** | `265981728` | | **Reporter** | @apex-mochen | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:32 +00:00

[PR #1201] [MERGED] [Store]: Add master_bench for benchmarking QPS of MasterService #2068