[GH-ISSUE #3157] [RFC]: Lock-Free Tenant Quota Charge/Release with Unified Charged Bytes #810

New issue

Open

opened 2026-10-29 21:19:46 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented 2026-10-29 21:19:46 +00:00

Owner

Copy link

Originally created by @Lin-z-w on GitHub (Jul 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3157

Changes proposed

Changes proposed

Summary

This RFC proposes replacing Mooncake Store's per-tenant used_bytes and reserved_bytes quota-accounting state with one atomic charged_bytes counter.

Memory is charged when Put, Copy, Move, or promotion admission starts. A successful operation keeps the charge, while a failed, revoked, partially completed, or expired operation refunds the unused portion. Completion no longer moves bytes from a global reserved counter to a global used counter.

The tenant quota control plane remains lock-based. Policy persistence, effective-quota recomputation, tenant registration, snapshots, and administrative mutation continue to use the existing control-plane locks. Only the high-frequency charge and release operations become independent of the quota-table shard mutex.

In this RFC, "lock-free" describes only the tenant quota charge/release primitive. A complete Put, Copy, Move, or promotion still uses its existing metadata, allocator, and task locks. Move is included because it shares the same ReplicationTask quota-settlement path as Copy; this RFC does not otherwise broaden the lock-free scope.

The proposed design:

• uses one compare-and-swap loop for quota admission;
• keeps operation-local pending_charge_bytes for exact rollback;
• stores a stable quota-account handle in each metadata TenantState;
• blocks new admissions through a bit in the same atomic charge state;
• preserves strict policy-deletion semantics, including zero-memory objects;
• rebuilds charged usage from authoritative metadata after restore;
• replaces the old quota API and metrics with a single charged-byte model; and
• does not provide a dual implementation, compatibility aliases, or downgrade guarantees.

Motivation

The current quota state distinguishes:

used_bytes      completed memory-replica charge
reserved_bytes  admitted but not yet completed memory-replica charge

Admission checks:

used_bytes + reserved_bytes + incoming_bytes <= effective_quota_bytes

Reserve, Commit, CommitAdditional, Abort, Release, and ReleasePartial all acquire the tenant quota shard mutex. Although the quota table is sharded, every operation for the same tenant maps to the same shard and therefore serializes on one mutex. A large tenant writing keys distributed across many metadata shards can still be bottlenecked by its single quota mutex.

The global reserved/used distinction is not required for quota admission. Admission only needs the total number of bytes already charged to the tenant. The distinction is useful when settling one operation, but that information is already present in ObjectMetadata, ReplicationTask, and PromotionTask.

Combining both global counters allows admission to become:

load charged bytes
check charged + incoming against effective quota
CAS charged to charged + incoming

Completion normally performs no aggregate byte update. Failure and partial completion only subtract the corresponding operation-local pending charge.

Goals

1. Remove the quota-table shard mutex from charge and release hot paths.
2. Preserve strict per-tenant admission: successful concurrent admissions must not exceed the effective quota at their linearization points.
3. Preserve policy deletion, dynamic policy updates, orphan-account cleanup, zero-memory metadata, timeout cleanup, and HA restore behavior.
4. Preserve the existing peak-charge behavior of size-changing Upsert, Copy, Move, and promotion.
5. Detect accounting underflow instead of allowing unsigned wraparound.
6. Avoid introducing a lock-free map or general-purpose memory-reclamation scheme in the first implementation.
7. Remove accounting fields, APIs, and metrics that are unnecessary once backward compatibility is not required.

Non-goals

• Making the entire Master request path lock-free.
• Removing metadata-shard, allocator, snapshot, or policy-store locks.
• Changing tenant effective-quota allocation policy.
• Changing eviction selection or retry policy.
• Persisting in-flight quota charge across Master restart or failover.
• Introducing per-metadata-shard quota credits in the first implementation.
• Providing a transactionally consistent multi-tenant metrics snapshot.
• Supporting mixed-version Masters, rolling downgrade, or legacy quota API and metric consumers.

Terminology

Term	Meaning
Requested quota	The configured quota requested by the tenant policy.
Effective quota	The tenant's current quota after cluster-capacity scaling.
Charged bytes	All bytes currently counted against admission, including committed and in-flight memory replicas.
Pending charge	The portion owned by one in-flight operation and refundable on failure.
Committed charge	The portion owned by completed memory replicas in object metadata.
Admission closed	The quota account rejects new positive and zero-byte admissions. Releases remain allowed.
Orphan account	An account with metadata or charge but no explicit tenant policy.

charged_bytes is an accounting term. A charge can be installed immediately before physical allocation and refunded if allocation fails. It does not mean that every charged byte is already readable or physically allocated at the instant a metrics request observes it.

Current behavior

The current quota state machine performs the following transitions:

Reserve(N):          reserved += N
Commit(N):           reserved -= N; used += N
CommitAdditional(N): reserved -= N; used += N
Abort(N):            reserved -= N
Release(N):          used -= N
ReleasePartial(N):   used -= N

The quota table lock provides three different kinds of protection:

1. atomic check-and-increment during admission;
2. consistent movement between multiple accounting fields; and
3. tenant entry and policy lifetime protection.

Removing reserved_bytes eliminates the second requirement, but it does not by itself solve policy deletion or tenant-entry lifetime. The proposed design addresses those requirements explicitly.

Proposed semantics

Unified charge

The tenant aggregate becomes:

charged_bytes = committed memory charge + in-flight pending memory charge

The state transitions become:

TryCharge(N):                CAS charged += N
SettleSuccess(pending, used): charged -= pending - used
RefundPending(N):            charged -= N
ReleaseCommitted(N):         charged -= N

PutStart therefore has the same admission behavior as the current Reserve. The visible difference is that the aggregate counter is immediately reported as charged rather than split between used and reserved.

Breaking API and metric changes

This RFC is an intentional breaking refactor. The implementation removes the old tenant-ID-based Reserve, Commit, CommitAdditional, Abort, Release, and ReleasePartial accounting API and replaces it with handle-based TryCharge and Release plus operation-settlement helpers.

The admin response and Prometheus surface remove used_bytes, reserved_bytes, committed_count, and metadata_object_count. They expose charged_bytes, effective and requested quota, and admission state directly. No field aliases, zero-valued placeholders, dual accounting mode, or compatibility feature flag are provided.

The change must therefore be deployed as one control-plane cutover rather than as a mixed-version rolling upgrade.

Operation-local pending charge

This RFC removes reserved_bytes from the tenant quota account. It does not remove knowledge of the charge owned by an in-flight operation.

The existing fields are renamed:

reserved_quota_charge_bytes -> pending_quota_charge_bytes

They remain in:

• ObjectMetadata;
• ReplicationTask; and
• PromotionTask.

These fields are protected by the existing metadata-shard lock. End, Revoke, cleanup, and durable-finalization paths consume them exactly once, preferably through a centralized helper using std::exchange(field, 0).

Without operation-local pending charge, an aborting path cannot determine the correct refund after partial completion, replica loss, Upsert replacement, or promotion timeout.

Quota account representation

Atomic charge state

The preferred representation is:

class TenantQuotaAccount {
   public:
    static constexpr uint64_t kAdmissionClosed = 1ULL << 63;
    static constexpr uint64_t kChargedBytesMask = kAdmissionClosed - 1;
    static constexpr uint64_t kMaxChargedBytes = kChargedBytesMask;

   private:
    // Bit 63: admission is closed.
    // Bits 0..62: charged bytes.
    alignas(64) std::atomic<uint64_t> charged_state_{
        kAdmissionClosed};

    std::atomic<uint64_t> effective_quota_bytes_{0};
    std::atomic<uint64_t> policy_sequence_{0};

    // Accessed only under control-plane locks.
    uint64_t requested_quota_bytes_{0};
    bool has_explicit_policy_{false};
};

Encoding the admission gate and byte count in one atomic word gives policy deletion and admission a single, exact ordering point.

This reserves the high bit and limits one tenant to 2^63 - 1 charged bytes, approximately 8 EiB. The limit is far above practical Mooncake deployments, but it is an observable validation change and must be documented. Policy input, capacity recomputation, requested replica charge, and rebuild code must reject values above this limit rather than truncate them.

Policy sequence

policy_sequence_ is even while the policy is stable and odd while an effective quota or registration state is being changed.

A charge operation records the sequence before its CAS and checks it again afterward. If the sequence changed, the operation refunds its charge and retries against the new policy.

This prevents an old request from succeeding across:

• effective-quota shrink;
• connector policy removal;
• policy delete followed by re-registration; or
• disabling and later re-enabling an orphan account.

Only control-plane updates write the sequence, so it is not a contended data-plane CAS location.

Which fields require atomic operations

Only charged_state_ needs a compare-and-swap loop for quota correctness. The other fields do not form one atomic quota equation:

Field	Data-plane access	Reason
charged_state_	CAS for charge, release, and admission close	The limit check and byte update must linearize together; the gate must not race policy deletion.
effective_quota_bytes_	Atomic load	The control plane stores it under existing locks; charge reads it without taking those locks.
policy_sequence_	Two atomic loads per charge; control-plane increment on update	Detects policy-generation changes and ABA without joining the hot CAS cache line.
Requested quota, registration flags, account registry	Existing control-plane locks	These paths are administrative and do not justify a lock-free registry.

committed_count and metadata_object_count are removed rather than made atomic. They are not required for admission, release, or policy deletion when the control plane can perform an authoritative metadata scan.

The hot-path quota interface is intentionally small:

TenantQuotaChargeResult TryCharge(TenantQuotaHandle account,
                                  uint64_t bytes);

TenantQuotaResult Release(TenantQuotaHandle account,
                          uint64_t bytes);

Neither method accepts a tenant ID or accesses the account registry. Tenant-ID-based lookup, policy upsert/delete, effective-quota recomputation, listing, and admin snapshots remain control-plane interfaces protected by the existing locks.

Stable account lifetime

Quota accounts are stored behind stable pointers:

using TenantQuotaHandle = TenantQuotaAccount*;

std::map<TenantId, std::unique_ptr<TenantQuotaAccount>> accounts_;

An account is not freed or erased while the Master is running. Removing a policy leaves a closed tombstone. This avoids RCU, hazard pointers, and shared-pointer reference-count traffic in charge/release paths.

Each metadata-shard TenantState stores the handle:

struct TenantState {
    TenantQuotaHandle quota_account{nullptr};
    // Existing metadata and task maps.
};

All TenantState instances for the same tenant point to the same quota account. A normal write obtains the handle while validating the tenant policy and binds it when creating the per-shard TenantState. Existing-object paths reuse the bound handle and do not perform a quota-map lookup.

Creating the first TenantState for a tenant in one metadata shard may take the existing quota-registry shard lock to resolve or create the stable handle. This is a cold lifetime event, not part of TryCharge or Release. If the same workload repeatedly removes and recreates otherwise-empty TenantStates, that lookup may still appear in profiles; avoiding it would require retaining per-shard tombstones or a lock-free registry and is outside this RFC.

After a charge succeeds, every owner that may refund it must retain access to the same handle. Normally the owning TenantState provides it. Any deferred callback or cleanup record that can outlive removal of that TenantState must capture TenantQuotaHandle directly. Hot settlement code must not fall back to a tenant-ID registry lookup.

Tombstone memory grows with the number of distinct tenant IDs seen during the Master process lifetime. The first implementation accepts this tradeoff. Bounded reclamation can be added later if high-rate tenant churn becomes a real deployment requirement.

Admission algorithm

The charge result includes the deficit observed by the failing attempt:

struct TenantQuotaChargeFailure {
    TenantQuotaError error;
    uint64_t deficit_bytes{0};
};

using TenantQuotaChargeResult =
    tl::expected<void, TenantQuotaChargeFailure>;

At a high level, TryCharge(bytes) performs:

loop:
    read stable policy sequence
    read charged state
    reject if admission is closed
    read effective quota
    reject if charged + incoming exceeds quota
    CAS charged state
    verify policy sequence and admission gate
    if policy changed:
        refund and retry
    return success

The overflow-safe admission check is:

if (current > limit || incoming > limit - current) {
    return quota_exceeded;
}

Successful CAS is the admission linearization point.

Illustrative pseudocode, with error plumbing omitted, is:

for (;;) {
    const uint64_t sequence_before =
        account->policy_sequence_.load(std::memory_order_acquire);
    if (sequence_before & 1) {
        continue;
    }

    uint64_t expected =
        account->charged_state_.load(std::memory_order_acquire);
    if (expected & kAdmissionClosed) {
        return TENANT_NOT_REGISTERED;
    }

    const uint64_t charged = expected & kChargedBytesMask;
    const uint64_t limit =
        account->effective_quota_bytes_.load(std::memory_order_acquire);

    if (bytes != 0 &&
        (charged > limit || bytes > limit - charged)) {
        return QuotaExceeded(Deficit(charged, bytes, limit));
    }

    if (bytes != 0) {
        const uint64_t desired = charged + bytes;
        if (!account->charged_state_.compare_exchange_weak(
                expected, desired, std::memory_order_acq_rel,
                std::memory_order_acquire)) {
            continue;
        }
    }

    const uint64_t state_after =
        account->charged_state_.load(std::memory_order_acquire);
    const uint64_t sequence_after =
        account->policy_sequence_.load(std::memory_order_acquire);

    if (sequence_before == sequence_after &&
        !(sequence_after & 1) &&
        !(state_after & kAdmissionClosed)) {
        return Success();
    }

    if (bytes != 0) {
        Release(account, bytes);
    }
    // Retry to return the result of the new policy generation.
}

Every control-plane change that can alter registration or admission semantics must bracket its atomic stores and admission-gate mutation by changing policy_sequence_ from even to odd and back to even. Control-plane writers are already serialized by the existing locks.

For a registered tenant, a zero-byte admission continues to succeed even when the account is over quota. It still verifies that admission is open and that the policy sequence is stable.

Returning the deficit from the failed charge attempt replaces the current separate ComputeDeficit lookup in the Put eviction/retry path. The deficit is a point-in-time eviction target; concurrent release can make it conservative, which is already acceptable to the retry loop.

Release algorithm

Release must work even when admission is closed, because orphaned or administratively disabled accounts still need to drain.

loop:
    load charged state
    extract charged byte count
    reject if refund exceeds charged bytes
    preserve admission bit
    CAS charged state to charged - refund

A CAS loop is preferred over unchecked fetch_sub so an accounting bug cannot underflow the byte field and corrupt the admission bit.

Illustrative release pseudocode is:

uint64_t expected =
    account->charged_state_.load(std::memory_order_acquire);
for (;;) {
    const uint64_t charged = expected & kChargedBytesMask;
    if (bytes > charged) {
        return ACCOUNTING_UNDERFLOW;
    }

    const uint64_t desired =
        (expected & kAdmissionClosed) | (charged - bytes);
    if (account->charged_state_.compare_exchange_weak(
            expected, desired, std::memory_order_acq_rel,
            std::memory_order_acquire)) {
        return Success();
    }
}

No registration or effective-quota check is performed during release.

Control-plane behavior

The existing control-plane locking remains:

• tenant_quota_policy_mutex_ serializes policy persistence and tenant registration changes;
• tenant_quota_recompute_mutex_ serializes the capacity snapshot with effective-quota recomputation;
• quota-table shard locks protect account registry and control-only fields; and
• the quota-table recompute mutex protects cross-shard policy assignment.

Charge and release methods accept a TenantQuotaHandle and do not acquire any of these quota-table locks.

Effective-quota update

An effective quota update uses a sequence write:

increment sequence to odd
store effective quota
increment sequence to even

An admission completed before the odd sequence begins can be linearized before the update. An admission overlapping the update detects the sequence change, refunds, and retries.

Delete policy if empty

Policy deletion is a control-plane operation and may take the existing policy and metadata locks. It does not need data-plane counters.

Under tenant_quota_policy_mutex_, deletion:

1. opens a short odd-sequence update window;
2. closes admission only if the account has zero charged bytes;
3. completes the sequence update;
4. calls the authoritative TenantHasObjects(tenant_id) metadata scan while admission remains closed;
5. persists policy removal; and
6. updates the control-only policy fields while leaving a closed tombstone.

Admission close uses:

uint64_t expected = 0;
charged_state_.compare_exchange_strong(
    expected, kAdmissionClosed);

There are only two outcomes:

1. charge wins first, so delete observes nonzero charge and returns TENANT_NOT_EMPTY; or
2. close wins first, so the charge observes the closed bit and returns TENANT_NOT_REGISTERED.

If TenantHasObjects finds any metadata, or if policy persistence fails, deletion opens another short sequence window, restores the policy, reopens admission, completes the sequence update, and returns an error. The scan holds tenant_quota_policy_mutex_ and visits metadata shards one at a time, consistent with the existing lock order.

Odd sequence windows contain only in-memory atomic and policy-field updates. Connector I/O and metadata scans run with admission closed but the sequence even, so TryCharge returns TENANT_NOT_REGISTERED instead of spinning for the duration of a slow control-plane operation.

Connector policy removal with existing objects

Applying a connector snapshot can remove an explicit policy while metadata still exists. This creates an orphan account rather than deleting its state.

The control plane atomically sets the admission-closed bit while preserving the low charged-byte bits. Existing charges remain visible and releasable, but new admissions fail. Re-registering the tenant updates its effective quota, advances the policy sequence, and reopens admission.

Zero-charge metadata

NOF-only and local-disk-only metadata may have zero memory quota charge. charged_bytes therefore cannot by itself prevent policy deletion from racing with creation of such metadata.

The first implementation retains the current slow-path rule:

• creation of new zero-charge metadata holds tenant_quota_policy_mutex_ until metadata insertion completes; and
• policy deletion holds the same mutex.

Policy deletion closes admission and scans authoritative metadata while holding the same outer lock. It therefore cannot miss a concurrent zero-charge creation. This RFC does not add an object counter or a data-plane reader counter solely to accelerate this cold administrative path.

Every metadata-creation path must satisfy one of two publication rules:

1. a positive charge remains installed from successful admission until the new metadata is visible; or
2. tenant_quota_policy_mutex_ remains held from policy validation until the new metadata is visible.

A path that initially charges positive bytes but refunds the charge to zero before publishing metadata must use the second rule. It must acquire the policy mutex before the metadata-shard lock, or drop the inner lock and retry through the protected creation path; it must not invert the existing lock order. This prevents policy deletion from closing admission, observing an empty tenant, and then being followed by publication of uncharged metadata.

Operation settlement

Put and new-object Upsert

PutStart:

1. Compute the maximum requested memory-replica charge.
2. Call TryCharge(requested).
3. Allocate replicas.
4. On allocation or metadata-insertion failure, refund the full charge.
5. Store the successful amount in ObjectMetadata::pending_quota_charge_bytes.

PutEnd:

1. Compute charge for completed memory replicas.
2. Compute the newly committed portion relative to committed_quota_charge_bytes.
3. Refund pending - newly_committed.
4. Clear pending charge.
5. Update committed charge.

Completion does not add newly committed bytes to the tenant aggregate; they were already charged at Start.

PutRevoke, stale-processing cleanup, and object erase refund the remaining pending amount exactly once.

Same-size Upsert

Same-size Upsert reuses existing memory buffers and does not add quota charge. The existing committed charge remains installed while replicas temporarily move from COMPLETE to PROCESSING.

Size-changing Upsert

Size-changing Upsert preserves the current peak-accounting behavior:

charged during transfer = old committed charge + new pending charge

If new allocation fails after old metadata has been removed, both the failed new charge and the no-longer-owned old charge are released.

If the replacement succeeds, PutEnd:

• settles the new pending charge;
• releases pending_replaced_quota_charge_bytes; and
• clears replacement state.

Copy

CopyStart charges:

object_size * number_of_new_memory_targets

CopyEnd keeps the charge for successfully completed targets and refunds failed or missing targets. It adds the successful portion to the object's committed charge but does not modify aggregate charged bytes again.

CopyRevoke and replication-task expiry refund the full remaining pending charge.

Move

MoveStart temporarily charges the target allocation while the source remains charged. On successful completion, the source's existing committed charge becomes ownership of the target and the entire temporary pending charge is refunded.

The object's total committed charge therefore remains unchanged after a same-sized Move.

Move failure, revoke, and expiry also refund the temporary pending charge.

Promotion

Promotion allocation charges one object size before allocating a MEMORY replica.

On success, the pending charge becomes committed without another aggregate increment. On failure, notification loss, replica loss, or reaper expiry, the pending charge is refunded.

Replica removal and eviction

Partial replica removal computes completed memory charge before and after the metadata mutation. The difference is released from both the object's committed charge and the tenant aggregate.

No tenant-level committed-object counter is updated. The object's committed_quota_charge_bytes remains the authoritative ownership record used for subsequent release and reconciliation.

Centralized settlement helpers

The implementation should centralize settlement rather than reproduce charge arithmetic across every error path:

void RefundPendingCharge(TenantState&, uint64_t& pending);

void SettleInitialCharge(TenantState&, ObjectMetadata&,
                         uint64_t actual_charge);

void SettleAdditionalCharge(TenantState&, ObjectMetadata&,
                            uint64_t& task_pending,
                            uint64_t actual_charge);

All Put, Copy, Move, promotion, erase, durable-finalize, and reaper paths must use these helpers.

Snapshot and metrics semantics

Snapshot consistency

The replacement admin snapshot is:

struct TenantQuotaSnapshot {
    TenantId tenant_id;
    uint64_t requested_quota_bytes;
    uint64_t effective_quota_bytes;
    uint64_t charged_bytes;
    bool admission_closed;
    bool has_explicit_policy;
    bool over_quota;
};

Quota snapshots atomically load:

• charged bytes;
• effective quota; and
• admission-closed state.

Requested quota and explicit-policy state are read under the existing control-plane lock. The resulting snapshot is point-in-time approximate across fields. This is sufficient for admin and Prometheus output. Policy deletion does not rely on this snapshot; it uses the admission gate and authoritative metadata scan.

over_quota becomes a derived value:

charged_bytes > effective_quota_bytes

An orphan account is represented explicitly by has_explicit_policy == false and admission_closed == true, including when its objects have zero memory charge.

Breaking metrics replacement

Add:

mooncake_tenant_quota_charged_bytes
mooncake_tenant_quota_admission_closed

Remove the old used, reserved, committed-count, and metadata-object-count quota metrics and response fields in the same change. Dashboards and clients must move directly to charged_bytes and admission_closed.

The old used/reserved split is intentionally not reconstructed. If operators later need in-flight visibility, it should be added as telemetry derived from operation state and must not become a second quota source of truth.

HA, snapshot restore, and rebuild

charged_state_, quota handles, and operation-local pending charge are derived runtime state and are not added to the snapshot or oplog wire format.

Startup remains:

restore metadata
load tenant policies
rebuild quota usage from metadata
start serving requests

Rebuild:

1. creates or finds the stable account for every tenant;
2. binds each restored TenantState to the account;
3. clears pending and replaced charge;
4. recomputes committed charge from completed MEMORY replicas;
5. rebuilds charged bytes; and
6. marks tenants without a loaded policy as closed orphan accounts.

Rebuild must run while request processing is quiescent. Runtime code must not overwrite charged_state_ while charge or release operations are active.

All metadata-creation paths must bind a quota handle, including:

• normal Put and Upsert;
• MetadataAccessorRW::Create;
• snapshot deserialization;
• standby restore;
• oplog application; and
• creation of local-disk-only metadata.

Quota accounting remains derived from metadata, so the new runtime counters are not serialized. However, this RFC does not guarantee mixed-version operation or downgrade: the quota admin API, metrics, and internal interfaces change in one cutover.

Code structure

File	Proposed change
mooncake-store/include/tenant_quota.h	Replace the old accounting API and state with TenantQuotaAccount, handle, charge-result, and minimal snapshot types.
mooncake-store/src/tenant_quota.cpp	Implement CAS admission, guarded release, policy gate, minimal snapshots, and rebuild; remove used/reserved and count bookkeeping.
mooncake-store/include/tenant_quota_sharded.h	Retain control-plane sharding and stable account registry; expose handle-based hot APIs.
mooncake-store/include/tenant_quota_sharded_impl.h	Delete tenant-ID-based hot accounting methods; keep locks only for policy, list, handle lookup, and recompute.
mooncake-store/include/master_service.h	Bind quota handles to TenantState; replace reserved fields with pending charge ownership; add settlement helpers.
mooncake-store/src/master_service.cpp	Migrate Put, Upsert, Copy, Move, promotion, erase, eviction, reaper, durable-finalize, and rebuild paths; remove count updates.
mooncake-store/src/master_admin_service.cpp	Replace used/reserved and count output with charged bytes and admission state.
mooncake-store/tests/tenant_quota_test.cpp	Add atomic-accounting and policy-race tests.
mooncake-store/tests/master_service_tenant_quota_test.cpp	Add operation-settlement, zero-charge, policy, and HA tests.

Correctness invariants

The implementation must maintain:

1. The low bits of charged_state_ never exceed kMaxChargedBytes.

2. A successful positive admission has one CAS linearization point.

3. Policy deletion and a new positive admission cannot both succeed.

4. Admission-closed accounts reject charge but accept release.

5. Every pending charge has exactly one owner in metadata or a task.

6. Every pending charge is consumed exactly once by success, failure, revoke, erase, durable finalization, or reaper expiry.

7. Object committed charge equals charge for its completed MEMORY replicas after an operation settles.

8. After the system becomes quiescent:

   tenant charged bytes
     = sum(object committed charge)
     + sum(live operation pending charge)
   

9. After restart or failover, pending charge is zero and charged bytes equal completed MEMORY charge reconstructed from restored metadata.

10. Zero-charge metadata prevents successful policy deletion.

11. New metadata is published only while protected by a positive charge or tenant_quota_policy_mutex_.

Testing plan

Quota-account unit tests

• Concurrent charge admits exactly the effective quota.
• Concurrent charge and release never underflow or exceed the admitted total.
• Charge overflow is rejected without wrapping.
• Release greater than charged bytes reports accounting mismatch.
• Release works while admission is closed.
• Policy shrink racing charge either orders before the charge or causes retry.
• Policy delete racing charge cannot allow both operations to succeed.
• Closed orphan accounts drain to zero.
• Disable/re-enable does not allow an old admission to cross a policy generation.
• 2^63 - 1 boundary behavior is explicit.
• Snapshot reads correctly mask the admission bit.
• Minimal snapshots contain no used/reserved or object-count compatibility fields.

MasterService tests

• PutStart immediately increases charged bytes.
• PutEnd does not double-charge and refunds unused replica charge.
• Allocation and metadata-insertion failures refund the full amount.
• PutRevoke and processing timeout refund exactly once.
• Same-size Upsert preserves charged bytes.
• Size-changing Upsert charges old and new allocations concurrently.
• Size-changing Upsert success and failure settle both ownerships correctly.
• Copy partial success retains only completed target charge.
• Move success leaves total committed charge unchanged.
• Promotion success, failure, client notification loss, and timeout settle correctly.
• Replica eviction, segment loss, stale-handle cleanup, and object removal release committed charge.
• Zero-charge Put and policy delete remain serialized without metadata counters.
• A creation that can refund its positive charge before metadata publication acquires policy-mutex protection without inverting lock order.
• Policy deletion closes admission, finds zero-charge metadata through TenantHasObjects, and reopens admission when deletion is rejected.
• Connector-save failure reopens admission.
• Policy removal leaves a releasable orphan.
• Restore rebuilds charge from completed replicas and drops pending charge.
• Durable callbacks and reapers cannot double-refund.
• Admin responses and Prometheus output expose only the new quota model.

Concurrency and performance validation

• Run quota and policy race tests under ThreadSanitizer.
• Add a benchmark with one tenant writing keys across many metadata shards.
• Compare throughput and tail latency against the shard-mutex implementation.
• Record CAS retries to identify pathological contention or policy-update retry loops.
• Verify from profiling that quota shard mutex acquisition is absent from charge and release call stacks.

Implementation and cutover plan

The work can be split into reviewable commits, but the final branch is deployed as one breaking control-plane cutover:

1. Quota account core
   • add the stable account and atomic charged state;
   • add unit and race tests.
2. Master hot-path migration
   • bind handles;
   • migrate operation settlement;
   • migrate reaper and HA rebuild;
   • remove hot-path quota shard locking.
3. Breaking cleanup
   • delete used/reserved and tenant-level count state;
   • delete tenant-ID-based legacy accounting methods;
   • replace admin responses and metrics;
   • add performance benchmarks.

A test-only reconciliation pass should compare atomic charged bytes against:

sum(committed metadata charge) + sum(pending operation charge)

There is no dual accounting implementation, compatibility feature flag, or mixed-version mode. Operators must update quota API consumers and dashboards as part of the same deployment.

Performance expectations

For one tenant, the existing implementation serializes quota updates through a mutex. The proposed design replaces that critical section with a CAS loop over one cache line.

This removes scheduler blocking and mutex handoff, but it does not remove cache-line contention. At very high core counts, a single tenant's exact global quota is inherently a shared synchronization point.

If the charged-state cache line remains a bottleneck after this change, a future RFC can consider distributing quota credits to metadata shards and refilling them from a global account in batches. That design has more complex fairness, reclamation, and policy-update semantics and is intentionally not part of this RFC.

Alternatives considered

Keep reserved bytes and add an atomic total

An atomic used + reserved total could drive admission while the old counters remain for observability. This reduces semantic change, but every operation must update multiple counters and exact snapshots still require locking or a sequence protocol. It also retains two sources of truth for the same quota.

Make used and reserved independently atomic

Admission cannot atomically check and update the sum of two independent counters. Concurrent commits and reservations can observe inconsistent pairs, so another combined counter or a 128-bit atomic is still required.

Keep tenant object and committed counters

Tenant-level counters make policy deletion and some metrics O(1), but every object creation, removal, and committed-charge transition performs another cross-shard atomic update. Because policy deletion is a cold administrative path and already performs an authoritative metadata check, this RFC removes the counters and pays the scan cost there.

Separate atomic admission gate and reader counter

Policy deletion could close a separate gate and wait for active admission readers to drain. This preserves the full 64-bit byte range, but adds at least two additional atomic operations to every charge attempt and introduces a second contended cache line.

128-bit compare-and-swap

A 128-bit state can hold the full byte count, registration state, and policy generation. Lock-free 128-bit atomics are not uniformly guaranteed across Mooncake's supported platforms and toolchains; some implementations fall back to a library lock. The proposed 64-bit state has more predictable behavior.

Keep the quota shard mutex

This remains the simplest correctness model and may be sufficient for tenants with low write concurrency. It does not address the target workload where one tenant writes across many metadata shards.

Lock-free tenant registry with reclamation

RCU or hazard-pointer lookup would also remove the policy lookup lock. The selected scope removes locking only from byte charge/release and uses stable tombstones for lifetime. A general lock-free registry is not necessary for the expected benefit.

Per-shard quota credits

Local credits can reduce cache-line contention further, but require reclaiming unused credits during policy shrink, segment changes, tenant deletion, and failover. CAS on the exact tenant total is a substantially smaller first step.

Risks and mitigations

Risk	Mitigation
Missing refund on a rare error path	Central settlement helpers, reconciliation tests, and reaper/HA coverage.
Double refund	Consume operation-local pending charge exactly once under the metadata shard lock.
Policy ABA	Stable accounts plus policy_sequence_ validation.
Use-after-free	Never erase quota accounts during process lifetime.
Tombstone growth	Track account count; add bounded reclamation only if tenant churn requires it.
New 8 EiB per-tenant limit	Validate explicitly and document; consider 128-bit state only if a real deployment requires it.
Breaking admin API and metrics	Announce the cutover and update in-tree dashboards and clients in the same change.
Slow policy deletion scan	Accept the authoritative metadata scan on this cold control-plane path; admission and release remain unaffected.
CAS contention remains high	Measure retries; consider per-shard credits in a later RFC.
Runtime rebuild races admission	Restrict rebuild to initialization/failover quiescence.

Feedback requested

This RFC especially requests feedback on:

• whether charging at Start and reporting the amount as charged_bytes is the right operator-facing semantic;
• reserving the high bit of a 64-bit charge word and introducing an explicit 2^63 - 1 per-tenant limit;
• keeping operation-local pending charge while removing global reserved state;
• stable process-lifetime tombstones versus adding account reclamation;
• removing tenant-level object and committed counts in favor of a policy-deletion metadata scan;
• preserving the existing policy mutex only for zero-charge metadata creation;
• accepting a direct breaking replacement of the quota API and metrics;
• whether strict policy-sequence retry is preferable to allowing overlapping admissions to linearize before a quota update; and
• whether the expected same-tenant write concurrency justifies replacing the current shard mutex before considering per-shard quota credits.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @Lin-z-w on GitHub (Jul 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/3157 ### Changes proposed ## Changes proposed ### Summary This RFC proposes replacing Mooncake Store's per-tenant `used_bytes` and `reserved_bytes` quota-accounting state with one atomic `charged_bytes` counter. Memory is charged when Put, Copy, Move, or promotion admission starts. A successful operation keeps the charge, while a failed, revoked, partially completed, or expired operation refunds the unused portion. Completion no longer moves bytes from a global reserved counter to a global used counter. The tenant quota control plane remains lock-based. Policy persistence, effective-quota recomputation, tenant registration, snapshots, and administrative mutation continue to use the existing control-plane locks. Only the high-frequency charge and release operations become independent of the quota-table shard mutex. In this RFC, "lock-free" describes only the tenant quota charge/release primitive. A complete Put, Copy, Move, or promotion still uses its existing metadata, allocator, and task locks. Move is included because it shares the same `ReplicationTask` quota-settlement path as Copy; this RFC does not otherwise broaden the lock-free scope. The proposed design: - uses one compare-and-swap loop for quota admission; - keeps operation-local `pending_charge_bytes` for exact rollback; - stores a stable quota-account handle in each metadata `TenantState`; - blocks new admissions through a bit in the same atomic charge state; - preserves strict policy-deletion semantics, including zero-memory objects; - rebuilds charged usage from authoritative metadata after restore; - replaces the old quota API and metrics with a single charged-byte model; and - does not provide a dual implementation, compatibility aliases, or downgrade guarantees. ### Motivation The current quota state distinguishes: ```text used_bytes completed memory-replica charge reserved_bytes admitted but not yet completed memory-replica charge ``` Admission checks: ```text used_bytes + reserved_bytes + incoming_bytes <= effective_quota_bytes ``` `Reserve`, `Commit`, `CommitAdditional`, `Abort`, `Release`, and `ReleasePartial` all acquire the tenant quota shard mutex. Although the quota table is sharded, every operation for the same tenant maps to the same shard and therefore serializes on one mutex. A large tenant writing keys distributed across many metadata shards can still be bottlenecked by its single quota mutex. The global reserved/used distinction is not required for quota admission. Admission only needs the total number of bytes already charged to the tenant. The distinction is useful when settling one operation, but that information is already present in `ObjectMetadata`, `ReplicationTask`, and `PromotionTask`. Combining both global counters allows admission to become: ```text load charged bytes check charged + incoming against effective quota CAS charged to charged + incoming ``` Completion normally performs no aggregate byte update. Failure and partial completion only subtract the corresponding operation-local pending charge. ### Goals 1. Remove the quota-table shard mutex from charge and release hot paths. 2. Preserve strict per-tenant admission: successful concurrent admissions must not exceed the effective quota at their linearization points. 3. Preserve policy deletion, dynamic policy updates, orphan-account cleanup, zero-memory metadata, timeout cleanup, and HA restore behavior. 4. Preserve the existing peak-charge behavior of size-changing Upsert, Copy, Move, and promotion. 5. Detect accounting underflow instead of allowing unsigned wraparound. 6. Avoid introducing a lock-free map or general-purpose memory-reclamation scheme in the first implementation. 7. Remove accounting fields, APIs, and metrics that are unnecessary once backward compatibility is not required. ### Non-goals - Making the entire Master request path lock-free. - Removing metadata-shard, allocator, snapshot, or policy-store locks. - Changing tenant effective-quota allocation policy. - Changing eviction selection or retry policy. - Persisting in-flight quota charge across Master restart or failover. - Introducing per-metadata-shard quota credits in the first implementation. - Providing a transactionally consistent multi-tenant metrics snapshot. - Supporting mixed-version Masters, rolling downgrade, or legacy quota API and metric consumers. ## Terminology | Term | Meaning | |------|---------| | Requested quota | The configured quota requested by the tenant policy. | | Effective quota | The tenant's current quota after cluster-capacity scaling. | | Charged bytes | All bytes currently counted against admission, including committed and in-flight memory replicas. | | Pending charge | The portion owned by one in-flight operation and refundable on failure. | | Committed charge | The portion owned by completed memory replicas in object metadata. | | Admission closed | The quota account rejects new positive and zero-byte admissions. Releases remain allowed. | | Orphan account | An account with metadata or charge but no explicit tenant policy. | `charged_bytes` is an accounting term. A charge can be installed immediately before physical allocation and refunded if allocation fails. It does not mean that every charged byte is already readable or physically allocated at the instant a metrics request observes it. ## Current behavior The current quota state machine performs the following transitions: ```text Reserve(N): reserved += N Commit(N): reserved -= N; used += N CommitAdditional(N): reserved -= N; used += N Abort(N): reserved -= N Release(N): used -= N ReleasePartial(N): used -= N ``` The quota table lock provides three different kinds of protection: 1. atomic check-and-increment during admission; 2. consistent movement between multiple accounting fields; and 3. tenant entry and policy lifetime protection. Removing `reserved_bytes` eliminates the second requirement, but it does not by itself solve policy deletion or tenant-entry lifetime. The proposed design addresses those requirements explicitly. ## Proposed semantics ### Unified charge The tenant aggregate becomes: ```text charged_bytes = committed memory charge + in-flight pending memory charge ``` The state transitions become: ```text TryCharge(N): CAS charged += N SettleSuccess(pending, used): charged -= pending - used RefundPending(N): charged -= N ReleaseCommitted(N): charged -= N ``` PutStart therefore has the same admission behavior as the current `Reserve`. The visible difference is that the aggregate counter is immediately reported as charged rather than split between used and reserved. ### Breaking API and metric changes This RFC is an intentional breaking refactor. The implementation removes the old tenant-ID-based `Reserve`, `Commit`, `CommitAdditional`, `Abort`, `Release`, and `ReleasePartial` accounting API and replaces it with handle-based `TryCharge` and `Release` plus operation-settlement helpers. The admin response and Prometheus surface remove `used_bytes`, `reserved_bytes`, `committed_count`, and `metadata_object_count`. They expose `charged_bytes`, effective and requested quota, and admission state directly. No field aliases, zero-valued placeholders, dual accounting mode, or compatibility feature flag are provided. The change must therefore be deployed as one control-plane cutover rather than as a mixed-version rolling upgrade. ### Operation-local pending charge This RFC removes `reserved_bytes` from the tenant quota account. It does not remove knowledge of the charge owned by an in-flight operation. The existing fields are renamed: ```text reserved_quota_charge_bytes -> pending_quota_charge_bytes ``` They remain in: - `ObjectMetadata`; - `ReplicationTask`; and - `PromotionTask`. These fields are protected by the existing metadata-shard lock. End, Revoke, cleanup, and durable-finalization paths consume them exactly once, preferably through a centralized helper using `std::exchange(field, 0)`. Without operation-local pending charge, an aborting path cannot determine the correct refund after partial completion, replica loss, Upsert replacement, or promotion timeout. ## Quota account representation ### Atomic charge state The preferred representation is: ```cpp class TenantQuotaAccount { public: static constexpr uint64_t kAdmissionClosed = 1ULL << 63; static constexpr uint64_t kChargedBytesMask = kAdmissionClosed - 1; static constexpr uint64_t kMaxChargedBytes = kChargedBytesMask; private: // Bit 63: admission is closed. // Bits 0..62: charged bytes. alignas(64) std::atomic<uint64_t> charged_state_{ kAdmissionClosed}; std::atomic<uint64_t> effective_quota_bytes_{0}; std::atomic<uint64_t> policy_sequence_{0}; // Accessed only under control-plane locks. uint64_t requested_quota_bytes_{0}; bool has_explicit_policy_{false}; }; ``` Encoding the admission gate and byte count in one atomic word gives policy deletion and admission a single, exact ordering point. This reserves the high bit and limits one tenant to `2^63 - 1` charged bytes, approximately 8 EiB. The limit is far above practical Mooncake deployments, but it is an observable validation change and must be documented. Policy input, capacity recomputation, requested replica charge, and rebuild code must reject values above this limit rather than truncate them. ### Policy sequence `policy_sequence_` is even while the policy is stable and odd while an effective quota or registration state is being changed. A charge operation records the sequence before its CAS and checks it again afterward. If the sequence changed, the operation refunds its charge and retries against the new policy. This prevents an old request from succeeding across: - effective-quota shrink; - connector policy removal; - policy delete followed by re-registration; or - disabling and later re-enabling an orphan account. Only control-plane updates write the sequence, so it is not a contended data-plane CAS location. ### Which fields require atomic operations Only `charged_state_` needs a compare-and-swap loop for quota correctness. The other fields do not form one atomic quota equation: | Field | Data-plane access | Reason | |-------|-------------------|--------| | `charged_state_` | CAS for charge, release, and admission close | The limit check and byte update must linearize together; the gate must not race policy deletion. | | `effective_quota_bytes_` | Atomic load | The control plane stores it under existing locks; charge reads it without taking those locks. | | `policy_sequence_` | Two atomic loads per charge; control-plane increment on update | Detects policy-generation changes and ABA without joining the hot CAS cache line. | | Requested quota, registration flags, account registry | Existing control-plane locks | These paths are administrative and do not justify a lock-free registry. | `committed_count` and `metadata_object_count` are removed rather than made atomic. They are not required for admission, release, or policy deletion when the control plane can perform an authoritative metadata scan. The hot-path quota interface is intentionally small: ```cpp TenantQuotaChargeResult TryCharge(TenantQuotaHandle account, uint64_t bytes); TenantQuotaResult Release(TenantQuotaHandle account, uint64_t bytes); ``` Neither method accepts a tenant ID or accesses the account registry. Tenant-ID-based lookup, policy upsert/delete, effective-quota recomputation, listing, and admin snapshots remain control-plane interfaces protected by the existing locks. ### Stable account lifetime Quota accounts are stored behind stable pointers: ```cpp using TenantQuotaHandle = TenantQuotaAccount*; std::map<TenantId, std::unique_ptr<TenantQuotaAccount>> accounts_; ``` An account is not freed or erased while the Master is running. Removing a policy leaves a closed tombstone. This avoids RCU, hazard pointers, and shared-pointer reference-count traffic in charge/release paths. Each metadata-shard `TenantState` stores the handle: ```cpp struct TenantState { TenantQuotaHandle quota_account{nullptr}; // Existing metadata and task maps. }; ``` All `TenantState` instances for the same tenant point to the same quota account. A normal write obtains the handle while validating the tenant policy and binds it when creating the per-shard `TenantState`. Existing-object paths reuse the bound handle and do not perform a quota-map lookup. Creating the first `TenantState` for a tenant in one metadata shard may take the existing quota-registry shard lock to resolve or create the stable handle. This is a cold lifetime event, not part of `TryCharge` or `Release`. If the same workload repeatedly removes and recreates otherwise-empty `TenantState`s, that lookup may still appear in profiles; avoiding it would require retaining per-shard tombstones or a lock-free registry and is outside this RFC. After a charge succeeds, every owner that may refund it must retain access to the same handle. Normally the owning `TenantState` provides it. Any deferred callback or cleanup record that can outlive removal of that `TenantState` must capture `TenantQuotaHandle` directly. Hot settlement code must not fall back to a tenant-ID registry lookup. Tombstone memory grows with the number of distinct tenant IDs seen during the Master process lifetime. The first implementation accepts this tradeoff. Bounded reclamation can be added later if high-rate tenant churn becomes a real deployment requirement. ## Admission algorithm The charge result includes the deficit observed by the failing attempt: ```cpp struct TenantQuotaChargeFailure { TenantQuotaError error; uint64_t deficit_bytes{0}; }; using TenantQuotaChargeResult = tl::expected<void, TenantQuotaChargeFailure>; ``` At a high level, `TryCharge(bytes)` performs: ```text loop: read stable policy sequence read charged state reject if admission is closed read effective quota reject if charged + incoming exceeds quota CAS charged state verify policy sequence and admission gate if policy changed: refund and retry return success ``` The overflow-safe admission check is: ```cpp if (current > limit || incoming > limit - current) { return quota_exceeded; } ``` Successful CAS is the admission linearization point. Illustrative pseudocode, with error plumbing omitted, is: ```cpp for (;;) { const uint64_t sequence_before = account->policy_sequence_.load(std::memory_order_acquire); if (sequence_before & 1) { continue; } uint64_t expected = account->charged_state_.load(std::memory_order_acquire); if (expected & kAdmissionClosed) { return TENANT_NOT_REGISTERED; } const uint64_t charged = expected & kChargedBytesMask; const uint64_t limit = account->effective_quota_bytes_.load(std::memory_order_acquire); if (bytes != 0 && (charged > limit || bytes > limit - charged)) { return QuotaExceeded(Deficit(charged, bytes, limit)); } if (bytes != 0) { const uint64_t desired = charged + bytes; if (!account->charged_state_.compare_exchange_weak( expected, desired, std::memory_order_acq_rel, std::memory_order_acquire)) { continue; } } const uint64_t state_after = account->charged_state_.load(std::memory_order_acquire); const uint64_t sequence_after = account->policy_sequence_.load(std::memory_order_acquire); if (sequence_before == sequence_after && !(sequence_after & 1) && !(state_after & kAdmissionClosed)) { return Success(); } if (bytes != 0) { Release(account, bytes); } // Retry to return the result of the new policy generation. } ``` Every control-plane change that can alter registration or admission semantics must bracket its atomic stores and admission-gate mutation by changing `policy_sequence_` from even to odd and back to even. Control-plane writers are already serialized by the existing locks. For a registered tenant, a zero-byte admission continues to succeed even when the account is over quota. It still verifies that admission is open and that the policy sequence is stable. Returning the deficit from the failed charge attempt replaces the current separate `ComputeDeficit` lookup in the Put eviction/retry path. The deficit is a point-in-time eviction target; concurrent release can make it conservative, which is already acceptable to the retry loop. ### Release algorithm Release must work even when admission is closed, because orphaned or administratively disabled accounts still need to drain. ```text loop: load charged state extract charged byte count reject if refund exceeds charged bytes preserve admission bit CAS charged state to charged - refund ``` A CAS loop is preferred over unchecked `fetch_sub` so an accounting bug cannot underflow the byte field and corrupt the admission bit. Illustrative release pseudocode is: ```cpp uint64_t expected = account->charged_state_.load(std::memory_order_acquire); for (;;) { const uint64_t charged = expected & kChargedBytesMask; if (bytes > charged) { return ACCOUNTING_UNDERFLOW; } const uint64_t desired = (expected & kAdmissionClosed) | (charged - bytes); if (account->charged_state_.compare_exchange_weak( expected, desired, std::memory_order_acq_rel, std::memory_order_acquire)) { return Success(); } } ``` No registration or effective-quota check is performed during release. ## Control-plane behavior The existing control-plane locking remains: - `tenant_quota_policy_mutex_` serializes policy persistence and tenant registration changes; - `tenant_quota_recompute_mutex_` serializes the capacity snapshot with effective-quota recomputation; - quota-table shard locks protect account registry and control-only fields; and - the quota-table recompute mutex protects cross-shard policy assignment. Charge and release methods accept a `TenantQuotaHandle` and do not acquire any of these quota-table locks. ### Effective-quota update An effective quota update uses a sequence write: ```text increment sequence to odd store effective quota increment sequence to even ``` An admission completed before the odd sequence begins can be linearized before the update. An admission overlapping the update detects the sequence change, refunds, and retries. ### Delete policy if empty Policy deletion is a control-plane operation and may take the existing policy and metadata locks. It does not need data-plane counters. Under `tenant_quota_policy_mutex_`, deletion: 1. opens a short odd-sequence update window; 2. closes admission only if the account has zero charged bytes; 3. completes the sequence update; 4. calls the authoritative `TenantHasObjects(tenant_id)` metadata scan while admission remains closed; 5. persists policy removal; and 6. updates the control-only policy fields while leaving a closed tombstone. Admission close uses: ```cpp uint64_t expected = 0; charged_state_.compare_exchange_strong( expected, kAdmissionClosed); ``` There are only two outcomes: 1. charge wins first, so delete observes nonzero charge and returns `TENANT_NOT_EMPTY`; or 2. close wins first, so the charge observes the closed bit and returns `TENANT_NOT_REGISTERED`. If `TenantHasObjects` finds any metadata, or if policy persistence fails, deletion opens another short sequence window, restores the policy, reopens admission, completes the sequence update, and returns an error. The scan holds `tenant_quota_policy_mutex_` and visits metadata shards one at a time, consistent with the existing lock order. Odd sequence windows contain only in-memory atomic and policy-field updates. Connector I/O and metadata scans run with admission closed but the sequence even, so `TryCharge` returns `TENANT_NOT_REGISTERED` instead of spinning for the duration of a slow control-plane operation. ### Connector policy removal with existing objects Applying a connector snapshot can remove an explicit policy while metadata still exists. This creates an orphan account rather than deleting its state. The control plane atomically sets the admission-closed bit while preserving the low charged-byte bits. Existing charges remain visible and releasable, but new admissions fail. Re-registering the tenant updates its effective quota, advances the policy sequence, and reopens admission. ## Zero-charge metadata NOF-only and local-disk-only metadata may have zero memory quota charge. `charged_bytes` therefore cannot by itself prevent policy deletion from racing with creation of such metadata. The first implementation retains the current slow-path rule: - creation of new zero-charge metadata holds `tenant_quota_policy_mutex_` until metadata insertion completes; and - policy deletion holds the same mutex. Policy deletion closes admission and scans authoritative metadata while holding the same outer lock. It therefore cannot miss a concurrent zero-charge creation. This RFC does not add an object counter or a data-plane reader counter solely to accelerate this cold administrative path. Every metadata-creation path must satisfy one of two publication rules: 1. a positive charge remains installed from successful admission until the new metadata is visible; or 2. `tenant_quota_policy_mutex_` remains held from policy validation until the new metadata is visible. A path that initially charges positive bytes but refunds the charge to zero before publishing metadata must use the second rule. It must acquire the policy mutex before the metadata-shard lock, or drop the inner lock and retry through the protected creation path; it must not invert the existing lock order. This prevents policy deletion from closing admission, observing an empty tenant, and then being followed by publication of uncharged metadata. ## Operation settlement ### Put and new-object Upsert PutStart: 1. Compute the maximum requested memory-replica charge. 2. Call `TryCharge(requested)`. 3. Allocate replicas. 4. On allocation or metadata-insertion failure, refund the full charge. 5. Store the successful amount in `ObjectMetadata::pending_quota_charge_bytes`. PutEnd: 1. Compute charge for completed memory replicas. 2. Compute the newly committed portion relative to `committed_quota_charge_bytes`. 3. Refund `pending - newly_committed`. 4. Clear pending charge. 5. Update committed charge. Completion does not add newly committed bytes to the tenant aggregate; they were already charged at Start. PutRevoke, stale-processing cleanup, and object erase refund the remaining pending amount exactly once. ### Same-size Upsert Same-size Upsert reuses existing memory buffers and does not add quota charge. The existing committed charge remains installed while replicas temporarily move from COMPLETE to PROCESSING. ### Size-changing Upsert Size-changing Upsert preserves the current peak-accounting behavior: ```text charged during transfer = old committed charge + new pending charge ``` If new allocation fails after old metadata has been removed, both the failed new charge and the no-longer-owned old charge are released. If the replacement succeeds, PutEnd: - settles the new pending charge; - releases `pending_replaced_quota_charge_bytes`; and - clears replacement state. ### Copy CopyStart charges: ```text object_size * number_of_new_memory_targets ``` CopyEnd keeps the charge for successfully completed targets and refunds failed or missing targets. It adds the successful portion to the object's committed charge but does not modify aggregate charged bytes again. CopyRevoke and replication-task expiry refund the full remaining pending charge. ### Move MoveStart temporarily charges the target allocation while the source remains charged. On successful completion, the source's existing committed charge becomes ownership of the target and the entire temporary pending charge is refunded. The object's total committed charge therefore remains unchanged after a same-sized Move. Move failure, revoke, and expiry also refund the temporary pending charge. ### Promotion Promotion allocation charges one object size before allocating a MEMORY replica. On success, the pending charge becomes committed without another aggregate increment. On failure, notification loss, replica loss, or reaper expiry, the pending charge is refunded. ### Replica removal and eviction Partial replica removal computes completed memory charge before and after the metadata mutation. The difference is released from both the object's committed charge and the tenant aggregate. No tenant-level committed-object counter is updated. The object's `committed_quota_charge_bytes` remains the authoritative ownership record used for subsequent release and reconciliation. ### Centralized settlement helpers The implementation should centralize settlement rather than reproduce charge arithmetic across every error path: ```cpp void RefundPendingCharge(TenantState&, uint64_t& pending); void SettleInitialCharge(TenantState&, ObjectMetadata&, uint64_t actual_charge); void SettleAdditionalCharge(TenantState&, ObjectMetadata&, uint64_t& task_pending, uint64_t actual_charge); ``` All Put, Copy, Move, promotion, erase, durable-finalize, and reaper paths must use these helpers. ## Snapshot and metrics semantics ### Snapshot consistency The replacement admin snapshot is: ```cpp struct TenantQuotaSnapshot { TenantId tenant_id; uint64_t requested_quota_bytes; uint64_t effective_quota_bytes; uint64_t charged_bytes; bool admission_closed; bool has_explicit_policy; bool over_quota; }; ``` Quota snapshots atomically load: - charged bytes; - effective quota; and - admission-closed state. Requested quota and explicit-policy state are read under the existing control-plane lock. The resulting snapshot is point-in-time approximate across fields. This is sufficient for admin and Prometheus output. Policy deletion does not rely on this snapshot; it uses the admission gate and authoritative metadata scan. `over_quota` becomes a derived value: ```text charged_bytes > effective_quota_bytes ``` An orphan account is represented explicitly by `has_explicit_policy == false` and `admission_closed == true`, including when its objects have zero memory charge. ### Breaking metrics replacement Add: ```text mooncake_tenant_quota_charged_bytes mooncake_tenant_quota_admission_closed ``` Remove the old used, reserved, committed-count, and metadata-object-count quota metrics and response fields in the same change. Dashboards and clients must move directly to `charged_bytes` and `admission_closed`. The old used/reserved split is intentionally not reconstructed. If operators later need in-flight visibility, it should be added as telemetry derived from operation state and must not become a second quota source of truth. ## HA, snapshot restore, and rebuild `charged_state_`, quota handles, and operation-local pending charge are derived runtime state and are not added to the snapshot or oplog wire format. Startup remains: ```text restore metadata load tenant policies rebuild quota usage from metadata start serving requests ``` Rebuild: 1. creates or finds the stable account for every tenant; 2. binds each restored `TenantState` to the account; 3. clears pending and replaced charge; 4. recomputes committed charge from completed MEMORY replicas; 5. rebuilds charged bytes; and 6. marks tenants without a loaded policy as closed orphan accounts. Rebuild must run while request processing is quiescent. Runtime code must not overwrite `charged_state_` while charge or release operations are active. All metadata-creation paths must bind a quota handle, including: - normal Put and Upsert; - `MetadataAccessorRW::Create`; - snapshot deserialization; - standby restore; - oplog application; and - creation of local-disk-only metadata. Quota accounting remains derived from metadata, so the new runtime counters are not serialized. However, this RFC does not guarantee mixed-version operation or downgrade: the quota admin API, metrics, and internal interfaces change in one cutover. ## Code structure | File | Proposed change | |------|-----------------| | `mooncake-store/include/tenant_quota.h` | Replace the old accounting API and state with `TenantQuotaAccount`, handle, charge-result, and minimal snapshot types. | | `mooncake-store/src/tenant_quota.cpp` | Implement CAS admission, guarded release, policy gate, minimal snapshots, and rebuild; remove used/reserved and count bookkeeping. | | `mooncake-store/include/tenant_quota_sharded.h` | Retain control-plane sharding and stable account registry; expose handle-based hot APIs. | | `mooncake-store/include/tenant_quota_sharded_impl.h` | Delete tenant-ID-based hot accounting methods; keep locks only for policy, list, handle lookup, and recompute. | | `mooncake-store/include/master_service.h` | Bind quota handles to `TenantState`; replace reserved fields with pending charge ownership; add settlement helpers. | | `mooncake-store/src/master_service.cpp` | Migrate Put, Upsert, Copy, Move, promotion, erase, eviction, reaper, durable-finalize, and rebuild paths; remove count updates. | | `mooncake-store/src/master_admin_service.cpp` | Replace used/reserved and count output with charged bytes and admission state. | | `mooncake-store/tests/tenant_quota_test.cpp` | Add atomic-accounting and policy-race tests. | | `mooncake-store/tests/master_service_tenant_quota_test.cpp` | Add operation-settlement, zero-charge, policy, and HA tests. | ## Correctness invariants The implementation must maintain: 1. The low bits of `charged_state_` never exceed `kMaxChargedBytes`. 2. A successful positive admission has one CAS linearization point. 3. Policy deletion and a new positive admission cannot both succeed. 4. Admission-closed accounts reject charge but accept release. 5. Every pending charge has exactly one owner in metadata or a task. 6. Every pending charge is consumed exactly once by success, failure, revoke, erase, durable finalization, or reaper expiry. 7. Object committed charge equals charge for its completed MEMORY replicas after an operation settles. 8. After the system becomes quiescent: ```text tenant charged bytes = sum(object committed charge) + sum(live operation pending charge) ``` 9. After restart or failover, pending charge is zero and charged bytes equal completed MEMORY charge reconstructed from restored metadata. 10. Zero-charge metadata prevents successful policy deletion. 11. New metadata is published only while protected by a positive charge or `tenant_quota_policy_mutex_`. ## Testing plan ### Quota-account unit tests - Concurrent charge admits exactly the effective quota. - Concurrent charge and release never underflow or exceed the admitted total. - Charge overflow is rejected without wrapping. - Release greater than charged bytes reports accounting mismatch. - Release works while admission is closed. - Policy shrink racing charge either orders before the charge or causes retry. - Policy delete racing charge cannot allow both operations to succeed. - Closed orphan accounts drain to zero. - Disable/re-enable does not allow an old admission to cross a policy generation. - `2^63 - 1` boundary behavior is explicit. - Snapshot reads correctly mask the admission bit. - Minimal snapshots contain no used/reserved or object-count compatibility fields. ### MasterService tests - PutStart immediately increases charged bytes. - PutEnd does not double-charge and refunds unused replica charge. - Allocation and metadata-insertion failures refund the full amount. - PutRevoke and processing timeout refund exactly once. - Same-size Upsert preserves charged bytes. - Size-changing Upsert charges old and new allocations concurrently. - Size-changing Upsert success and failure settle both ownerships correctly. - Copy partial success retains only completed target charge. - Move success leaves total committed charge unchanged. - Promotion success, failure, client notification loss, and timeout settle correctly. - Replica eviction, segment loss, stale-handle cleanup, and object removal release committed charge. - Zero-charge Put and policy delete remain serialized without metadata counters. - A creation that can refund its positive charge before metadata publication acquires policy-mutex protection without inverting lock order. - Policy deletion closes admission, finds zero-charge metadata through `TenantHasObjects`, and reopens admission when deletion is rejected. - Connector-save failure reopens admission. - Policy removal leaves a releasable orphan. - Restore rebuilds charge from completed replicas and drops pending charge. - Durable callbacks and reapers cannot double-refund. - Admin responses and Prometheus output expose only the new quota model. ### Concurrency and performance validation - Run quota and policy race tests under ThreadSanitizer. - Add a benchmark with one tenant writing keys across many metadata shards. - Compare throughput and tail latency against the shard-mutex implementation. - Record CAS retries to identify pathological contention or policy-update retry loops. - Verify from profiling that quota shard mutex acquisition is absent from charge and release call stacks. ## Implementation and cutover plan The work can be split into reviewable commits, but the final branch is deployed as one breaking control-plane cutover: 1. **Quota account core** - add the stable account and atomic charged state; - add unit and race tests. 2. **Master hot-path migration** - bind handles; - migrate operation settlement; - migrate reaper and HA rebuild; - remove hot-path quota shard locking. 3. **Breaking cleanup** - delete used/reserved and tenant-level count state; - delete tenant-ID-based legacy accounting methods; - replace admin responses and metrics; - add performance benchmarks. A test-only reconciliation pass should compare atomic charged bytes against: ```text sum(committed metadata charge) + sum(pending operation charge) ``` There is no dual accounting implementation, compatibility feature flag, or mixed-version mode. Operators must update quota API consumers and dashboards as part of the same deployment. ## Performance expectations For one tenant, the existing implementation serializes quota updates through a mutex. The proposed design replaces that critical section with a CAS loop over one cache line. This removes scheduler blocking and mutex handoff, but it does not remove cache-line contention. At very high core counts, a single tenant's exact global quota is inherently a shared synchronization point. If the charged-state cache line remains a bottleneck after this change, a future RFC can consider distributing quota credits to metadata shards and refilling them from a global account in batches. That design has more complex fairness, reclamation, and policy-update semantics and is intentionally not part of this RFC. ## Alternatives considered ### Keep reserved bytes and add an atomic total An atomic `used + reserved` total could drive admission while the old counters remain for observability. This reduces semantic change, but every operation must update multiple counters and exact snapshots still require locking or a sequence protocol. It also retains two sources of truth for the same quota. ### Make used and reserved independently atomic Admission cannot atomically check and update the sum of two independent counters. Concurrent commits and reservations can observe inconsistent pairs, so another combined counter or a 128-bit atomic is still required. ### Keep tenant object and committed counters Tenant-level counters make policy deletion and some metrics O(1), but every object creation, removal, and committed-charge transition performs another cross-shard atomic update. Because policy deletion is a cold administrative path and already performs an authoritative metadata check, this RFC removes the counters and pays the scan cost there. ### Separate atomic admission gate and reader counter Policy deletion could close a separate gate and wait for active admission readers to drain. This preserves the full 64-bit byte range, but adds at least two additional atomic operations to every charge attempt and introduces a second contended cache line. ### 128-bit compare-and-swap A 128-bit state can hold the full byte count, registration state, and policy generation. Lock-free 128-bit atomics are not uniformly guaranteed across Mooncake's supported platforms and toolchains; some implementations fall back to a library lock. The proposed 64-bit state has more predictable behavior. ### Keep the quota shard mutex This remains the simplest correctness model and may be sufficient for tenants with low write concurrency. It does not address the target workload where one tenant writes across many metadata shards. ### Lock-free tenant registry with reclamation RCU or hazard-pointer lookup would also remove the policy lookup lock. The selected scope removes locking only from byte charge/release and uses stable tombstones for lifetime. A general lock-free registry is not necessary for the expected benefit. ### Per-shard quota credits Local credits can reduce cache-line contention further, but require reclaiming unused credits during policy shrink, segment changes, tenant deletion, and failover. CAS on the exact tenant total is a substantially smaller first step. ## Risks and mitigations | Risk | Mitigation | |------|------------| | Missing refund on a rare error path | Central settlement helpers, reconciliation tests, and reaper/HA coverage. | | Double refund | Consume operation-local pending charge exactly once under the metadata shard lock. | | Policy ABA | Stable accounts plus `policy_sequence_` validation. | | Use-after-free | Never erase quota accounts during process lifetime. | | Tombstone growth | Track account count; add bounded reclamation only if tenant churn requires it. | | New 8 EiB per-tenant limit | Validate explicitly and document; consider 128-bit state only if a real deployment requires it. | | Breaking admin API and metrics | Announce the cutover and update in-tree dashboards and clients in the same change. | | Slow policy deletion scan | Accept the authoritative metadata scan on this cold control-plane path; admission and release remain unaffected. | | CAS contention remains high | Measure retries; consider per-shard credits in a later RFC. | | Runtime rebuild races admission | Restrict rebuild to initialization/failover quiescence. | ## Feedback requested This RFC especially requests feedback on: - whether charging at Start and reporting the amount as `charged_bytes` is the right operator-facing semantic; - reserving the high bit of a 64-bit charge word and introducing an explicit `2^63 - 1` per-tenant limit; - keeping operation-local pending charge while removing global reserved state; - stable process-lifetime tombstones versus adding account reclamation; - removing tenant-level object and committed counts in favor of a policy-deletion metadata scan; - preserving the existing policy mutex only for zero-charge metadata creation; - accepting a direct breaking replacement of the quota API and metrics; - whether strict policy-sequence retry is preferable to allowing overlapping admissions to linearize before a quota update; and - whether the expected same-tenant write concurrency justifies replacing the current shard mutex before considering per-shard quota credits. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

stale

labels 2026-10-29 21:19:46 +00:00

bsakaguchi commented 2026-10-29 21:19:47 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 28, 2026):

Thanks for opening this issue, @Lin-z-w!

Field	Value
Issue	#3157
GitHub user ID	99333079
Reporter	@Lin-z-w

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:5102152027 --> @github-actions[bot] commented on GitHub (Jul 28, 2026): Thanks for opening this issue, @Lin-z-w! | Field | Value | |-------|-------| | **Issue** | #3157 | | **GitHub user ID** | `99333079` | | **Reporter** | @Lin-z-w | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:19:47 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 27, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6433832786 --> @github-actions[bot] commented on GitHub (Oct 27, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi referenced this issue 2026-10-29 21:30:45 +00:00

[PR #810] [MERGED] [RL] Add dummy example of RL training on mooncake store #1838

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#810

No description provided.