kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2866] [RFC]: TENT Hierarchical QoS Budget Enforcement and Work-Conserving Scheduling #745

New issue

Closed

opened 2026-10-29 21:19:01 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:19:01 +00:00

Owner

Originally created by @catyans on GitHub (Jul 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2866

Summary

This RFC proposes hierarchical QoS budget enforcement for TENT: a default-disabled, work-conserving scheduler that turns an effective QoS Contract into bounded runtime service.

The QoS Contract in #2856/#2857 defines fields such as:

authorized priority;
minimum target bandwidth;
maximum bandwidth;
relative weight;
burst bytes;
maximum inflight bytes/requests;
allowed degraded actions.

Those fields currently describe policy but do not enforce a shared resource budget. Existing priority/deadline ordering, local Admission limits, sender-local byte credits, and receiver-advertised credits solve different parts of the problem:

Transfer Intent (#2865)
        |
        v
effective QoS Contract (#2856/#2857)
        |
        v
Execution Plan + ResourceCharge (#2863)
        |
        v
hierarchical QoS budget enforcement  <-- this RFC
        |
        +-- local Admission / dispatch (#2132/#2655)
        +-- receiver capacity grants (#2849/#2860-#2862)
        |
        v
existing transport / ProxyManager

The scheduler should provide, within an explicitly defined enforcement scope:

aggregate tenant/intent caps;
bounded bursts;
weighted excess sharing;
work-conserving borrowing of otherwise idle capacity;
bounded reclaim when a guaranteed class becomes active again;
no starvation for eligible background traffic;
EDF/aging inside a class without allowing deadlines to bypass authorization or caps;
early, explainable degrade/reject when a request is infeasible.

This is a design RFC. Production enforcement PRs are explicitly gated on the prerequisites listed below. The first implementation milestone should be a pure model/simulator and shadow accounting; it must not change default dispatch behavior.

Why this RFC is separate from the QoS Contract

#2856 answers:

What service is this tenant/intent authorized to request?

This RFC answers:

When multiple authorized classes compete for constrained resources, how does TENT account, schedule, borrow, cap, and explain their actual service?

Keeping the schema and enforcement separate allows:

policy parsing/validation to remain stable and testable without a scheduler;
enforcement algorithms to evolve without changing the public contract format;
a shadow-accounting phase before behavior changes;
explicit claims about local versus distributed guarantees.

Motivation

TENT already has useful QoS building blocks:

per-request priority and priority queues;
starvation prevention/aging mechanisms;
deadline fields, EDF/deadline-aware Admission, and infeasible degradation work;
a local runtime Admission Queue (#2132);
sender-local priority/byte-credit work (#2655);
live RDMA bandwidth input (#2816);
deadline-aware arbitration (#2794);
receiver-advertised multi-resource credits (#2849 and #2860-#2862);
QoS policy/intent binding (#2847);
QoS Contract parsing/effective-policy resolution (#2856/#2857).

These mechanisms do not yet provide a complete multi-tenant budget contract.

1. Priority is ordering, not isolation

Strict priority can protect foreground traffic, but it can starve background work indefinitely. Aging can reduce starvation, but promotion alone does not enforce a tenant cap or a long-term weighted share.

2. A cap must aggregate requests

Per-request or per-flow limits can be bypassed by splitting one workload into many requests, flows, policies, or sender processes. Enforcement must aggregate by an authenticated budget subject such as tenant + intent.

3. Idle capacity should not be wasted

Static reservations can leave NIC bandwidth unused when a guaranteed class is idle. A production scheduler should let other eligible classes borrow idle capacity while ensuring that the owner can reclaim service within a bounded interval.

4. Local capacity and receiver capacity are different

A sender may have local dispatch budget while the receiver has no staging/consumer/QP capacity. Conversely, receiver credit is a capacity reservation, not proof that the sender's tenant is authorized to consume that share.

5. Direct and Staged plans have different costs

Charging only Request.length cannot correctly account for staging buffers, repeated device copies, receiver/consumer slots, QP/CQ pressure, or fallback paths. Enforcement needs #2863's plan-level resource vector.

6. A local scheduler cannot claim a cluster-wide guarantee

If sixteen independent senders each enforce “tenant A may use 40 Gbps” locally, aggregate traffic can greatly exceed 40 Gbps at one receiver. v1 must state its enforcement scope honestly and use receiver-side allocation or future coordination for aggregate guarantees.

Prerequisites and implementation gate

Opening this RFC does not authorize immediate production scheduler changes.

Prerequisite	Why required	Minimum gate before enforcement PR
#2856/#2857 QoS Contract	stable guarantee/cap/weight/burst/inflight semantics	maintainer scope ACK and stable effective-policy fields
#2865 Transfer Intent API	stable intent and requested/effective distinction	normalized intent available internally; public connector adoption may follow later
#2863 Execution Plan	consistent plan-level resource charge	accepted `ResourceCharge` model for at least Direct and Staged paths
#2849/#2860-#2862 Receiver Credits	receiver-owned capacity and many-to-one gating	production control-path direction and credit scope agreed
authenticated tenant/subject identity	prevent tenant/priority spoofing	trusted `QosSubject` source or explicit single-tenant-only scope
#2845/tebench metrics baseline	prove isolation without hiding throughput loss	reusable multi-class result schema and offered-load sweep

Before these gates are met, acceptable work is limited to:

RFC discussion;
deterministic scheduler model/simulator;
trace replay;
shadow accounting with no dispatch effect;
benchmark/test fixtures.

Goals

Define an explicit enforcement scope and budget subject.
Enforce hierarchical aggregate caps and inflight limits.
Provide minimum target service when the configured capacity is feasible.
Share excess capacity by weight.
Borrow idle capacity without wasting throughput.
Bound reclaim latency when a guaranteed class becomes active.
Prevent starvation without allowing promotion to bypass caps.
Combine class budgets with EDF/aging inside each class.
Charge complete Execution Plan resources atomically.
Integrate receiver capacity without conflating credits and authorization.
Expose deterministic decisions, metrics, and explain output.
Preserve current behavior when enforcement is disabled.

Non-goals

A hard end-to-end bandwidth guarantee when physical capacity is insufficient.
A cluster-wide global scheduler in v1.
Per-RTT congestion control or replacement of ECN/DCQCN/HPCC/PFC.
Revoking bytes already posted to a transport.
Treating PFC or SL/TC as tenant isolation by itself.
Defining tenant authentication/RBAC in this RFC.
Replacing Receiver Credits, Admission Queue, transport queues, or ProxyManager.
Implementing a general multi-resource optimizer in the first enforcement PR.
Allowing a caller-supplied HIGH priority to bypass the effective contract.
Claiming performance or SLO improvements before the real-cluster matrix is complete.

Terminology

Budget subject

The authenticated entity charged by the scheduler:

QosSubject = enforcement-domain / tenant / policy / intent

The exact hierarchy may omit levels that are not configured, but tenant identity must come from a trusted connector/Store/control-plane context. An arbitrary request string is not sufficient authorization.

Enforcement scope

Where a budget is actually enforced:

ENGINE_LOCAL: one TENT engine/process;
RECEIVER_LOCAL: aggregate grants at one receiving TENT instance;
HOST_LOCAL: multiple engines coordinated on one host;
CLUSTER_GLOBAL: future coordinated scope.

Every configured budget and every metric must identify its scope. v1 should implement ENGINE_LOCAL; RECEIVER_LOCAL can follow through receiver-credit allocation. It must not label either one as cluster-global.

Guarantee, cap, and weight

min_bandwidth: target minimum service under contention when total guarantees and physical capacity are feasible;
max_bandwidth: long-term ceiling, subject to the configured burst;
weight: relative share of excess capacity after eligible guarantees are served;
burst_bytes: bounded short-term usage above the steady cap;
max_inflight_*: bounds admitted but unfinished work, independent of rate.

The term “guarantee” should be used carefully. If telemetry is missing, capacity drops, or configured minimums exceed capacity, TENT can provide deterministic proportional degradation and report a violation, not create bandwidth that does not exist.

Borrowing

Borrowing means an eligible class uses capacity that another class is not currently using. Borrowed service is opportunistic and is not a debt that must later be repaid by starving the borrower. Reclaim means stopping new excess dispatch for borrowers and resuming the owner's eligible service within a bounded dispatch quantum/inflight window.

Proposed hierarchy

root capacity domain (for example one NIC/rail group)
  |
  +-- tenant A
  |     +-- foreground_get
  |     +-- background_prefetch
  |     +-- checkpoint
  |
  +-- tenant B
        +-- foreground_get
        +-- migration

Limits apply at every configured level:

child service contributes to parent usage;
a child cannot exceed its own cap or make its parent exceed the parent cap;
sibling weights only divide the parent's available excess;
creating more child classes does not increase the parent's entitlement;
unconfigured requests resolve to an explicit default/best-effort class.

Proposed budget model

The exact layout is not fixed, but the model should cover:

struct QosBudgetConfig {
    ResourceScope resource_scope;
    uint64_t min_rate_per_sec;
    uint64_t max_rate_per_sec;
    uint32_t weight;
    uint64_t burst_units;
    uint64_t max_inflight_units;
    uint32_t max_inflight_requests;
};

struct QosBudgetState {
    uint64_t generation;
    int64_t cap_tokens;
    int64_t guarantee_deficit;
    int64_t excess_deficit;
    uint64_t inflight_units;
    uint32_t inflight_requests;
    uint64_t last_refill_ns;
    uint64_t last_service_ns;
};

“Units” are bytes in the first network-budget implementation. Later plan resources can use typed dimensions rather than pretending QP slots or staging slots are bytes.

All arithmetic must be overflow checked and use monotonic time. Refills should be computed lazily or in bounded scheduler ticks; a background timer per class is not acceptable.

Scheduling model

The proposed v1 algorithm combines three layers:

hierarchical token buckets
    enforce cap + bounded burst

guarantee/excess DRR or WDRR
    allocate service between eligible classes

EDF + aging inside one class
    order requests without changing the class entitlement

1. Eligibility and hard limits

A request is eligible only when:

the class and all parents are below inflight limits;
cap/burst tokens permit the charge;
the complete plan resource charge can be reserved;
required receiver credit is available;
policy permits the plan and any degraded action.

Admission of one submit call remains atomic. A request must not consume a tenant token while failing to obtain the staging/receiver resource needed to complete.

2. Guaranteed service

For active classes with configured minimum target rates, accrue a bounded guarantee deficit. Under contention, serve eligible guaranteed deficit before distributing excess.

The implementation must cap accumulated idle deficit. An idle class cannot return after a long absence with an unbounded token stockpile and block all other traffic.

3. Excess service

After currently eligible guaranteed service, distribute excess capacity among eligible classes using DRR/WDRR based on configured weights.

DRR quantum should be large enough to avoid excessive scheduler overhead but small enough to bound reclaim and latency. The quantum and maximum dispatch batch become part of the observable contract.

4. Work-conserving borrowing

If a class has no eligible work, its currently unused capacity is available to siblings subject to their caps. When the owner becomes active:

no new excess service is granted to borrowers ahead of eligible guaranteed work;
already-dispatched transfers are not revoked;
reclaim latency is bounded by maximum outstanding dispatch, transport progress, and scheduler quantum;
the runtime reports actual reclaim time.

5. Intra-class ordering

Within one budget class:

requests with deadlines use EDF or deadline slack;
requests without deadlines use FIFO/aging;
aging prevents indefinite wait;
deadline/aging changes order, not entitlement;
a deadline cannot bypass the class/parent cap unless an explicit authorized override class exists.

6. Infeasible requests

If a request cannot meet its deadline under the effective budget and current capacity:

apply only a degraded action allowed by the effective QoS Contract;
possible actions include delay, fallback transport, local recompute, compression, or reject;
record the budget/capacity evidence and selected action;
avoid consuming substantial network bytes before an inevitable reject.

The enforcement layer does not implement compression or recompute itself; it emits the authorized decision/hook.

Resource accounting

Phase 1: network-byte budget

The first enforcement PR should limit scope to one well-defined constrained resource, such as bytes dispatched through a selected NIC/rail capacity domain.

This phase validates:

hierarchy;
cap/burst;
guarantee/excess scheduling;
borrowing/reclaim;
EDF/aging interaction;
metrics and explain.

It should not pretend to solve multi-resource fairness.

Phase 2: plan resource vector

After #2863 stabilizes, a plan may charge:

NIC bytes
receiver bytes
request slots
staging slots and bytes
consumer slots
QP/CQ slots
CPU copy work
device-copy bytes

Resources have different owners:

Resource	Typical owner
local queue/inflight	Local Admission
NIC byte budget	sender/host budget scheduler
receiver bytes/slots	receiver-credit allocator
staging buffers	ProxyManager/receiver staging owner
QP/CQ slots	RDMA backend/pool
GPU/CPU copy	platform/staging executor

The plan computes the charge; each resource owner reserves its dimension. Eligibility requires an atomic all-or-nothing reservation protocol at the Admission boundary, with rollback on failure.

Avoiding double charge for Staged paths

A staged user request and its internal transfers must share one parent plan/attempt identity.

charge actual physical stages to the appropriate resources;
do not charge both the outer logical length and the same internal NIC bytes twice;
attribute internal staging work to the original subject unless a separately bounded internal control/staging budget is required for progress;
reserve a small bounded internal-progress class so payload exhaustion cannot prevent the staging work that releases resources.

Receiver-side allocation

Receiver Credits answer “what capacity has this receiver reserved for this sender/class?” They do not by themselves enforce tenant entitlement.

For many-to-one traffic, a future receiver-local allocator can divide receiver capacity by authenticated budget subject:

receiver capacity
    -> tenant guarantee/cap/weight
    -> intent class
    -> per-sender cumulative credit grants

Rules:

aggregate grants must never exceed receiver capacity;
multiple senders for the same tenant share the tenant budget rather than each receiving the full budget;
grants use the epoch/replay-safe protocol from #2849;
credit is non-revocable within a live session; reclaim stops new grants and waits for consumed reservations to finish/return;
missing/old peer support follows explicit compatibility policy, never “unlimited capacity” in required mode.

This is Phase 2+ and depends on agreement about the production receiver-credit control path.

Capacity input and feasibility

Rate enforcement requires an explicit capacity domain.

Capacity input should include:

value
scope/resource ID
observed interval
timestamp
freshness TTL
sample count
confidence
source
anomaly flag

Rules:

expired telemetry must not continue to authorize a larger guarantee;
unknown capacity uses a conservative configured baseline or disables guarantee claims while retaining hard caps;
capacity changes use hysteresis/minimum dwell time to avoid budget oscillation;
sum of configured minimum targets is validated against conservative capacity when possible;
oversubscription, if permitted, is explicit and uses deterministic proportional degradation;
enforcement is slower-timescale scheduling, not congestion control.

Configuration validation

Reject or warn on:

negative/zero-invalid rates, weights, bursts, and inflight limits;
child guarantee/cap inconsistent with parent;
min_rate > max_rate;
sum of child minimums greater than parent minimum/capacity without explicit oversubscription mode;
unlimited cap combined with strict isolation mode;
unknown resource scope;
high-cardinality or dynamically created policy classes beyond configured bounds;
guarantee configured without trusted subject identity;
degraded action requested but not authorized by the contract;
cluster-global guarantee configured when only engine-local enforcement exists.

tent check-config from #2864 should show the hierarchy, aggregate rates, oversubscription, enforcement scope, and restart/runtime requirements.

Priority authorization and tenant identity

The scheduler must never trust arbitrary request fields as authority.

caller request
    + trusted connector/session identity
    + configured QoS Contract
    -> authenticated QosSubject
    -> effective budget class

Required properties:

callers cannot self-assign another tenant;
HIGH priority is only effective if authorized;
explicit policy_name remains authorization checked;
splitting requests/flows/senders does not multiply the parent budget;
internal staging/control work cannot impersonate foreground user traffic;
unauthorized or missing identity uses a configured default/best-effort class or fails closed in strict mode;
audit/explain records the policy match without exposing unbounded sensitive labels.

The authentication mechanism itself is outside this RFC, but production multi-tenant enforcement is blocked until a trusted identity source exists. In single-tenant deployments, v1 may use an explicit fixed subject and clearly label the scope.

State bounds and lifecycle

Budget state must be bounded:

only configured/resolved classes create persistent state;
maximum tenants/classes per engine is configured;
idle state expires only when inflight/reservations are zero;
policy/config generation changes create an atomic new snapshot;
existing attempts retain the policy generation under which they were admitted;
counters migrate or reset according to explicit rules; cap tokens must not be minted twice during reload;
peer/engine restart resets local state but distributed credit epochs prevent stale grants from becoming usable;
metrics avoid per-request and unbounded raw-tenant labels.

Decision and explain model

Each Admission decision should have a bounded internal reason:

ADMIT_GUARANTEE
ADMIT_EXCESS
ADMIT_BORROWED
WAIT_CAP
WAIT_PARENT_CAP
WAIT_INFLIGHT
WAIT_RECEIVER_CREDIT
WAIT_STAGING
DEGRADE_DEADLINE
REJECT_UNAUTHORIZED
REJECT_INFEASIBLE

Example sanitized explain:

{
  "schema_version": 1,
  "scope": "engine_local:nic_group_0",
  "subject": "tenant_hash/foreground_get",
  "decision": "admit_guarantee",
  "plan_charge": {
    "nic_bytes": 1048576,
    "receiver_bytes": 1048576,
    "request_slots": 1
  },
  "budget": {
    "min_rate_bps": 80000000000,
    "max_rate_bps": 160000000000,
    "weight": 8,
    "borrowed": false
  },
  "capacity": {
    "value_bps": 190000000000,
    "fresh": true,
    "confidence": 0.91
  }
}

Per-request explain belongs in trace/sample/CLI queries, not as unbounded Prometheus labels.

Metrics

Required aggregate metrics per bounded configured class/scope:

admitted/dispatched/completed bytes and requests;
guaranteed, excess, and borrowed service bytes;
cap/burst throttled time and bytes;
inflight bytes/requests;
queue wait distribution;
reclaim latency;
starvation/maximum wait;
deadline miss and infeasible decisions;
degraded/rejected counts by bounded reason;
receiver-credit/staging/transport stall time;
capacity freshness/confidence;
configured versus achieved service rate;
cap violation amount/time;
scheduler iterations and decision time.

Benchmark-derived metrics:

guarantee attainment ratio;
cap error;
Jain fairness index for comparable weighted classes;
total utilization/goodput;
foreground P50/P95/P99 and TTFT where integrated;
background completion/starvation time;
wasted bytes before degrade/reject.

Compatibility and rollout

Suggested modes:

disabled  -> current behavior (default)
shadow    -> account and emit decisions; never delay/reorder/reject
enforced  -> apply budgets within the declared scope

Rules:

absent QoS Contract preserves current behavior;
unknown/missing subject maps to compatibility default unless strict mode requires rejection;
shadow mode must compare predicted decisions with actual dispatch without modifying it;
enforced mode is opt-in until real-cluster qualification is complete;
disabling enforcement does not disable independent hard safety limits such as receiver capacity;
mixed-version peers follow receiver-credit capability negotiation; local enforcement does not assume remote enforcement;
a runtime must advertise its actual scope, not imply cluster-global QoS.

Proposed PR sequence

PR0: Model, simulator, and trace replay

Allowed before production prerequisites merge:

pure hierarchical budget model;
deterministic virtual clock;
token/deficit arithmetic and invariants;
trace replay for offered-load sweeps;
no TENT submit/dispatch changes.

PR1: Shadow accounting adapter

After stable QoS Contract and minimal ResourceCharge:

resolve bounded QosSubject/effective policy;
compute network-byte charges;
emit shadow admit/wait/degrade reasons;
compare with actual dispatch;
no behavior change.

PR2: Engine-local inflight cap and rate cap

enforce aggregate tenant/intent inflight and max-rate/burst limits;
all-or-nothing Admission integration;
default disabled;
Direct network-byte path first.

PR3: Guarantee/excess DRR and borrowing

minimum target service for feasible configurations;
weighted excess sharing;
bounded borrowing/reclaim;
EDF/aging inside class;
scheduler cost/latency benchmarks.

PR4: Staged plan accounting

consume Direct/Staged ResourceCharge from #2863;
avoid double charge;
reserve internal-progress budget;
integrate causal metrics.

PR5: Receiver-local tenant allocation

allocate receiver credits by authenticated subject;
aggregate multiple senders;
preserve epoch/replay invariants;
requires agreed production credit control path.

Later work

host-local coordination across multiple TE instances;
multi-resource fairness beyond typed independent gates;
connector identity/auth integration;
cluster/global coordination if a real production requirement emerges;
hardware wire-QoS validation and policy mapping.

Correctness invariants

The model and implementation must prove/test:

No class or parent exceeds its configured inflight limit.
Long-term service does not exceed cap beyond the documented burst/quantum tolerance.
Request splitting does not increase parent entitlement.
Token/deficit arithmetic never overflows, underflows, or mints budget after duplicate updates.
In feasible stationary load, active guaranteed classes receive their configured target within bounded measurement tolerance.
Excess service is work-conserving subject to caps and resource availability.
A returning guaranteed class reclaims eligibility within a bounded quantum/inflight interval.
Eligible background traffic is not permanently starved.
Deadline ordering cannot bypass authorization, parent cap, or receiver capacity.
Atomic multi-resource reservation never exposes partial charge.
Late completion/release is charged to the correct plan/attempt/policy generation exactly once.
Receiver credit and QoS budget are both required where configured; neither is treated as the other.
Unknown/stale capacity never creates additional guaranteed budget.
Disabled mode preserves current behavior.

Validation plan

Deterministic model tests

virtual-time token refill and burst boundaries;
hierarchical parent/child caps;
idle guarantee accumulation bound;
weighted excess distribution;
borrowing and reclaim;
variable request sizes and DRR deficit;
request splitting/flow multiplication attacks;
EDF/aging inside class;
overload and proportional degradation;
policy generation change;
arithmetic/fuzz/property tests;
one million+ scheduling decisions without unbounded state growth.

CPU integration tests

Admission all-or-nothing rollback;
mixed Direct/Staged charge fixtures;
receiver-credit available/unavailable;
late completion/release and attempt generation;
strict/compatibility identity behavior;
shadow mode produces no dispatch differences;
enforced mode respects caps under concurrent submitters;
sanitizer and TSan coverage.

Real-cluster workload matrix

Use tebench on the H20/RoCE cluster with at least:

topology:       1P1D and 2P2D
senders:        1 / 4 / 16
paths:          Direct RDMA / forced Staged / TCP fallback
classes:        foreground_get / background_prefetch / checkpoint
offered load:   below capacity / near congestion point / overload
baselines:      QoS disabled / strict priority / budget scheduler

For each run report:

configured and measured capacity;
per-class offered/accepted/completed rate;
guarantee attainment and cap error;
borrowed bytes and reclaim time;
queue wait and P50/P95/P99 latency;
deadline miss/degrade/reject;
background maximum wait/starvation;
receiver-credit and staging stalls;
data integrity;
total utilization and goodput;
scheduler CPU and submit overhead.

Required scenarios

Foreground saturated, background active: foreground target protected, background bounded but not permanently starved.
Foreground idle: background borrows capacity up to cap.
Foreground returns: bounded reclaim without revoking posted work.
Tenant creates many small requests/flows: aggregate cap unchanged.
Two tenants with weights 8:2: excess share converges within defined tolerance.
Sum of guarantees infeasible: config rejects or explicit proportional-degrade mode reports violations.
Receiver slowdown: local tokens alone do not dispatch without receiver credit.
Stale bandwidth telemetry: conservative fallback, no oscillatory budget inflation.
Tight deadline under exhausted cap: authorized early degrade/reject, not cap bypass.
Rail/path failure: replan charge is revalidated; old and new attempts are not double charged.

Claims boundary

No performance/SLO claim should be made from light-load averages. Offered load must cross the congestion point, and results must include isolation, fairness, integrity, and total utilization.

Expected benefits

If prerequisites and invariants are satisfied, this RFC should provide:

bounded aggregate tenant/intent usage within the declared scope;
foreground protection without permanent background starvation;
work-conserving use of idle capacity;
explicit and measurable borrowing/reclaim;
consistent accounting across Direct/Staged/Fallback plans;
separation of authorization, sender budget, and receiver capacity;
early explainable overload decisions;
a path from policy schema to production QoS enforcement.

It does not provide a hard cluster-wide SLA in v1. Any result must be labeled by actual enforcement scope and capacity assumptions.

Relationship to existing work

#2856/#2857: source of effective QoS Contract fields; this RFC does not redefine the schema.
#2865: source of normalized request intent and requested/effective distinction.
#2863: source of Direct/Staged/Fallback plan and ResourceCharge.
#2864: configuration validation, status, and explain surface.
#2132: Local Admission Queue remains the dispatch ownership boundary.
#2655: coordinate with existing priority/byte-credit runtime queue work; do not create a second queue.
#2519/#2816: reuse deadline feasibility and live bandwidth inputs.
#2794: same-priority deadline arbitration remains a narrower mechanism inside authorized budget.
#2849/#2860-#2862: receiver-owned capacity and many-to-one gating; credits are not authorization.
#2845: reuse tebench QoS metrics/baselines.
#2568/#2847: QP pool/SL/TC and intent policy binding consume the effective class but do not enforce aggregate software budgets alone.
#2489/#1752: congestion/device controllers operate at a different timescale; this RFC does not replace them.

Open questions

Do maintainers agree that this enforcement design should remain a separate follow-up to #2856 rather than extending the schema RFC?
What is the accepted v1 enforcement scope: engine-local only, or should receiver-local allocation be required before any enforcement is enabled?
What trusted tenant/subject identity should TENT consume from Store/connectors?
Should min_bandwidth_gbps be named/documented as a target minimum rather than a hard guarantee when capacity is infeasible?
Is hierarchical token bucket + DRR/WDRR + intra-class EDF the preferred baseline, or should v1 start with cap/inflight enforcement only?
How should oversubscribed minimums behave: reject configuration, proportional degradation, or explicit best-effort mode?
What resource domain should own a network budget: individual rail, NIC group, NUMA domain, or policy-selected device set?
What is an acceptable dispatch quantum/reclaim bound for foreground latency?
Should borrowed service be reported only, or should a bounded borrow limit be independently configurable?
Which fields from #2856 are stable enough for a shadow-accounting prototype?
Should enforcement remain blocked until #2863 ResourceCharge is accepted, or may a network-byte-only shadow prototype proceed earlier?
How should configured class identity be exported to metrics without unbounded label cardinality?

Before submitting a new issue...

Searched existing issues and PRs for QoS budget, tenant policing, bandwidth guarantee/cap, WDRR/DRR, token bucket, Admission, receiver credits, and multi-tenant scheduling.
Reviewed #2132, #2519, #2655, #2794, #2849, #2856, #2857, #2860-#2865 and the relevant TENT roadmap items.

Originally created by @catyans on GitHub (Jul 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2866 ## Summary This RFC proposes hierarchical QoS budget enforcement for TENT: a default-disabled, work-conserving scheduler that turns an effective QoS Contract into bounded runtime service. The QoS Contract in #2856/#2857 defines fields such as: - authorized priority; - minimum target bandwidth; - maximum bandwidth; - relative weight; - burst bytes; - maximum inflight bytes/requests; - allowed degraded actions. Those fields currently describe policy but do not enforce a shared resource budget. Existing priority/deadline ordering, local Admission limits, sender-local byte credits, and receiver-advertised credits solve different parts of the problem: ```text Transfer Intent (#2865) | v effective QoS Contract (#2856/#2857) | v Execution Plan + ResourceCharge (#2863) | v hierarchical QoS budget enforcement <-- this RFC | +-- local Admission / dispatch (#2132/#2655) +-- receiver capacity grants (#2849/#2860-#2862) | v existing transport / ProxyManager ``` The scheduler should provide, within an explicitly defined enforcement scope: - aggregate tenant/intent caps; - bounded bursts; - weighted excess sharing; - work-conserving borrowing of otherwise idle capacity; - bounded reclaim when a guaranteed class becomes active again; - no starvation for eligible background traffic; - EDF/aging inside a class without allowing deadlines to bypass authorization or caps; - early, explainable degrade/reject when a request is infeasible. This is a design RFC. Production enforcement PRs are explicitly gated on the prerequisites listed below. The first implementation milestone should be a pure model/simulator and shadow accounting; it must not change default dispatch behavior. ## Why this RFC is separate from the QoS Contract #2856 answers: > What service is this tenant/intent authorized to request? This RFC answers: > When multiple authorized classes compete for constrained resources, how does TENT account, schedule, borrow, cap, and explain their actual service? Keeping the schema and enforcement separate allows: - policy parsing/validation to remain stable and testable without a scheduler; - enforcement algorithms to evolve without changing the public contract format; - a shadow-accounting phase before behavior changes; - explicit claims about local versus distributed guarantees. ## Motivation TENT already has useful QoS building blocks: - per-request priority and priority queues; - starvation prevention/aging mechanisms; - deadline fields, EDF/deadline-aware Admission, and infeasible degradation work; - a local runtime Admission Queue (#2132); - sender-local priority/byte-credit work (#2655); - live RDMA bandwidth input (#2816); - deadline-aware arbitration (#2794); - receiver-advertised multi-resource credits (#2849 and #2860-#2862); - QoS policy/intent binding (#2847); - QoS Contract parsing/effective-policy resolution (#2856/#2857). These mechanisms do not yet provide a complete multi-tenant budget contract. ### 1. Priority is ordering, not isolation Strict priority can protect foreground traffic, but it can starve background work indefinitely. Aging can reduce starvation, but promotion alone does not enforce a tenant cap or a long-term weighted share. ### 2. A cap must aggregate requests Per-request or per-flow limits can be bypassed by splitting one workload into many requests, flows, policies, or sender processes. Enforcement must aggregate by an authenticated budget subject such as tenant + intent. ### 3. Idle capacity should not be wasted Static reservations can leave NIC bandwidth unused when a guaranteed class is idle. A production scheduler should let other eligible classes borrow idle capacity while ensuring that the owner can reclaim service within a bounded interval. ### 4. Local capacity and receiver capacity are different A sender may have local dispatch budget while the receiver has no staging/consumer/QP capacity. Conversely, receiver credit is a capacity reservation, not proof that the sender's tenant is authorized to consume that share. ### 5. Direct and Staged plans have different costs Charging only `Request.length` cannot correctly account for staging buffers, repeated device copies, receiver/consumer slots, QP/CQ pressure, or fallback paths. Enforcement needs #2863's plan-level resource vector. ### 6. A local scheduler cannot claim a cluster-wide guarantee If sixteen independent senders each enforce “tenant A may use 40 Gbps” locally, aggregate traffic can greatly exceed 40 Gbps at one receiver. v1 must state its enforcement scope honestly and use receiver-side allocation or future coordination for aggregate guarantees. ## Prerequisites and implementation gate Opening this RFC does not authorize immediate production scheduler changes. | Prerequisite | Why required | Minimum gate before enforcement PR | |---|---|---| | #2856/#2857 QoS Contract | stable guarantee/cap/weight/burst/inflight semantics | maintainer scope ACK and stable effective-policy fields | | #2865 Transfer Intent API | stable intent and requested/effective distinction | normalized intent available internally; public connector adoption may follow later | | #2863 Execution Plan | consistent plan-level resource charge | accepted `ResourceCharge` model for at least Direct and Staged paths | | #2849/#2860-#2862 Receiver Credits | receiver-owned capacity and many-to-one gating | production control-path direction and credit scope agreed | | authenticated tenant/subject identity | prevent tenant/priority spoofing | trusted `QosSubject` source or explicit single-tenant-only scope | | #2845/tebench metrics baseline | prove isolation without hiding throughput loss | reusable multi-class result schema and offered-load sweep | Before these gates are met, acceptable work is limited to: - RFC discussion; - deterministic scheduler model/simulator; - trace replay; - shadow accounting with no dispatch effect; - benchmark/test fixtures. ## Goals 1. Define an explicit enforcement scope and budget subject. 2. Enforce hierarchical aggregate caps and inflight limits. 3. Provide minimum target service when the configured capacity is feasible. 4. Share excess capacity by weight. 5. Borrow idle capacity without wasting throughput. 6. Bound reclaim latency when a guaranteed class becomes active. 7. Prevent starvation without allowing promotion to bypass caps. 8. Combine class budgets with EDF/aging inside each class. 9. Charge complete Execution Plan resources atomically. 10. Integrate receiver capacity without conflating credits and authorization. 11. Expose deterministic decisions, metrics, and explain output. 12. Preserve current behavior when enforcement is disabled. ## Non-goals - A hard end-to-end bandwidth guarantee when physical capacity is insufficient. - A cluster-wide global scheduler in v1. - Per-RTT congestion control or replacement of ECN/DCQCN/HPCC/PFC. - Revoking bytes already posted to a transport. - Treating PFC or SL/TC as tenant isolation by itself. - Defining tenant authentication/RBAC in this RFC. - Replacing Receiver Credits, Admission Queue, transport queues, or ProxyManager. - Implementing a general multi-resource optimizer in the first enforcement PR. - Allowing a caller-supplied HIGH priority to bypass the effective contract. - Claiming performance or SLO improvements before the real-cluster matrix is complete. ## Terminology ### Budget subject The authenticated entity charged by the scheduler: ```text QosSubject = enforcement-domain / tenant / policy / intent ``` The exact hierarchy may omit levels that are not configured, but tenant identity must come from a trusted connector/Store/control-plane context. An arbitrary request string is not sufficient authorization. ### Enforcement scope Where a budget is actually enforced: - `ENGINE_LOCAL`: one TENT engine/process; - `RECEIVER_LOCAL`: aggregate grants at one receiving TENT instance; - `HOST_LOCAL`: multiple engines coordinated on one host; - `CLUSTER_GLOBAL`: future coordinated scope. Every configured budget and every metric must identify its scope. v1 should implement `ENGINE_LOCAL`; `RECEIVER_LOCAL` can follow through receiver-credit allocation. It must not label either one as cluster-global. ### Guarantee, cap, and weight - `min_bandwidth`: target minimum service under contention when total guarantees and physical capacity are feasible; - `max_bandwidth`: long-term ceiling, subject to the configured burst; - `weight`: relative share of excess capacity after eligible guarantees are served; - `burst_bytes`: bounded short-term usage above the steady cap; - `max_inflight_*`: bounds admitted but unfinished work, independent of rate. The term “guarantee” should be used carefully. If telemetry is missing, capacity drops, or configured minimums exceed capacity, TENT can provide deterministic proportional degradation and report a violation, not create bandwidth that does not exist. ### Borrowing Borrowing means an eligible class uses capacity that another class is not currently using. Borrowed service is opportunistic and is not a debt that must later be repaid by starving the borrower. Reclaim means stopping new excess dispatch for borrowers and resuming the owner's eligible service within a bounded dispatch quantum/inflight window. ## Proposed hierarchy ```text root capacity domain (for example one NIC/rail group) | +-- tenant A | +-- foreground_get | +-- background_prefetch | +-- checkpoint | +-- tenant B +-- foreground_get +-- migration ``` Limits apply at every configured level: - child service contributes to parent usage; - a child cannot exceed its own cap or make its parent exceed the parent cap; - sibling weights only divide the parent's available excess; - creating more child classes does not increase the parent's entitlement; - unconfigured requests resolve to an explicit default/best-effort class. ## Proposed budget model The exact layout is not fixed, but the model should cover: ```cpp struct QosBudgetConfig { ResourceScope resource_scope; uint64_t min_rate_per_sec; uint64_t max_rate_per_sec; uint32_t weight; uint64_t burst_units; uint64_t max_inflight_units; uint32_t max_inflight_requests; }; struct QosBudgetState { uint64_t generation; int64_t cap_tokens; int64_t guarantee_deficit; int64_t excess_deficit; uint64_t inflight_units; uint32_t inflight_requests; uint64_t last_refill_ns; uint64_t last_service_ns; }; ``` “Units” are bytes in the first network-budget implementation. Later plan resources can use typed dimensions rather than pretending QP slots or staging slots are bytes. All arithmetic must be overflow checked and use monotonic time. Refills should be computed lazily or in bounded scheduler ticks; a background timer per class is not acceptable. ## Scheduling model The proposed v1 algorithm combines three layers: ```text hierarchical token buckets enforce cap + bounded burst guarantee/excess DRR or WDRR allocate service between eligible classes EDF + aging inside one class order requests without changing the class entitlement ``` ### 1. Eligibility and hard limits A request is eligible only when: - the class and all parents are below inflight limits; - cap/burst tokens permit the charge; - the complete plan resource charge can be reserved; - required receiver credit is available; - policy permits the plan and any degraded action. Admission of one submit call remains atomic. A request must not consume a tenant token while failing to obtain the staging/receiver resource needed to complete. ### 2. Guaranteed service For active classes with configured minimum target rates, accrue a bounded guarantee deficit. Under contention, serve eligible guaranteed deficit before distributing excess. The implementation must cap accumulated idle deficit. An idle class cannot return after a long absence with an unbounded token stockpile and block all other traffic. ### 3. Excess service After currently eligible guaranteed service, distribute excess capacity among eligible classes using DRR/WDRR based on configured weights. DRR quantum should be large enough to avoid excessive scheduler overhead but small enough to bound reclaim and latency. The quantum and maximum dispatch batch become part of the observable contract. ### 4. Work-conserving borrowing If a class has no eligible work, its currently unused capacity is available to siblings subject to their caps. When the owner becomes active: - no new excess service is granted to borrowers ahead of eligible guaranteed work; - already-dispatched transfers are not revoked; - reclaim latency is bounded by maximum outstanding dispatch, transport progress, and scheduler quantum; - the runtime reports actual reclaim time. ### 5. Intra-class ordering Within one budget class: - requests with deadlines use EDF or deadline slack; - requests without deadlines use FIFO/aging; - aging prevents indefinite wait; - deadline/aging changes order, not entitlement; - a deadline cannot bypass the class/parent cap unless an explicit authorized override class exists. ### 6. Infeasible requests If a request cannot meet its deadline under the effective budget and current capacity: - apply only a degraded action allowed by the effective QoS Contract; - possible actions include delay, fallback transport, local recompute, compression, or reject; - record the budget/capacity evidence and selected action; - avoid consuming substantial network bytes before an inevitable reject. The enforcement layer does not implement compression or recompute itself; it emits the authorized decision/hook. ## Resource accounting ### Phase 1: network-byte budget The first enforcement PR should limit scope to one well-defined constrained resource, such as bytes dispatched through a selected NIC/rail capacity domain. This phase validates: - hierarchy; - cap/burst; - guarantee/excess scheduling; - borrowing/reclaim; - EDF/aging interaction; - metrics and explain. It should not pretend to solve multi-resource fairness. ### Phase 2: plan resource vector After #2863 stabilizes, a plan may charge: ```text NIC bytes receiver bytes request slots staging slots and bytes consumer slots QP/CQ slots CPU copy work device-copy bytes ``` Resources have different owners: | Resource | Typical owner | |---|---| | local queue/inflight | Local Admission | | NIC byte budget | sender/host budget scheduler | | receiver bytes/slots | receiver-credit allocator | | staging buffers | ProxyManager/receiver staging owner | | QP/CQ slots | RDMA backend/pool | | GPU/CPU copy | platform/staging executor | The plan computes the charge; each resource owner reserves its dimension. Eligibility requires an atomic all-or-nothing reservation protocol at the Admission boundary, with rollback on failure. ### Avoiding double charge for Staged paths A staged user request and its internal transfers must share one parent plan/attempt identity. - charge actual physical stages to the appropriate resources; - do not charge both the outer logical length and the same internal NIC bytes twice; - attribute internal staging work to the original subject unless a separately bounded internal control/staging budget is required for progress; - reserve a small bounded internal-progress class so payload exhaustion cannot prevent the staging work that releases resources. ## Receiver-side allocation Receiver Credits answer “what capacity has this receiver reserved for this sender/class?” They do not by themselves enforce tenant entitlement. For many-to-one traffic, a future receiver-local allocator can divide receiver capacity by authenticated budget subject: ```text receiver capacity -> tenant guarantee/cap/weight -> intent class -> per-sender cumulative credit grants ``` Rules: - aggregate grants must never exceed receiver capacity; - multiple senders for the same tenant share the tenant budget rather than each receiving the full budget; - grants use the epoch/replay-safe protocol from #2849; - credit is non-revocable within a live session; reclaim stops new grants and waits for consumed reservations to finish/return; - missing/old peer support follows explicit compatibility policy, never “unlimited capacity” in required mode. This is Phase 2+ and depends on agreement about the production receiver-credit control path. ## Capacity input and feasibility Rate enforcement requires an explicit capacity domain. Capacity input should include: ```text value scope/resource ID observed interval timestamp freshness TTL sample count confidence source anomaly flag ``` Rules: - expired telemetry must not continue to authorize a larger guarantee; - unknown capacity uses a conservative configured baseline or disables guarantee claims while retaining hard caps; - capacity changes use hysteresis/minimum dwell time to avoid budget oscillation; - sum of configured minimum targets is validated against conservative capacity when possible; - oversubscription, if permitted, is explicit and uses deterministic proportional degradation; - enforcement is slower-timescale scheduling, not congestion control. ## Configuration validation Reject or warn on: - negative/zero-invalid rates, weights, bursts, and inflight limits; - child guarantee/cap inconsistent with parent; - `min_rate > max_rate`; - sum of child minimums greater than parent minimum/capacity without explicit oversubscription mode; - unlimited cap combined with strict isolation mode; - unknown resource scope; - high-cardinality or dynamically created policy classes beyond configured bounds; - guarantee configured without trusted subject identity; - degraded action requested but not authorized by the contract; - cluster-global guarantee configured when only engine-local enforcement exists. `tent check-config` from #2864 should show the hierarchy, aggregate rates, oversubscription, enforcement scope, and restart/runtime requirements. ## Priority authorization and tenant identity The scheduler must never trust arbitrary request fields as authority. ```text caller request + trusted connector/session identity + configured QoS Contract -> authenticated QosSubject -> effective budget class ``` Required properties: - callers cannot self-assign another tenant; - HIGH priority is only effective if authorized; - explicit `policy_name` remains authorization checked; - splitting requests/flows/senders does not multiply the parent budget; - internal staging/control work cannot impersonate foreground user traffic; - unauthorized or missing identity uses a configured default/best-effort class or fails closed in strict mode; - audit/explain records the policy match without exposing unbounded sensitive labels. The authentication mechanism itself is outside this RFC, but production multi-tenant enforcement is blocked until a trusted identity source exists. In single-tenant deployments, v1 may use an explicit fixed subject and clearly label the scope. ## State bounds and lifecycle Budget state must be bounded: - only configured/resolved classes create persistent state; - maximum tenants/classes per engine is configured; - idle state expires only when inflight/reservations are zero; - policy/config generation changes create an atomic new snapshot; - existing attempts retain the policy generation under which they were admitted; - counters migrate or reset according to explicit rules; cap tokens must not be minted twice during reload; - peer/engine restart resets local state but distributed credit epochs prevent stale grants from becoming usable; - metrics avoid per-request and unbounded raw-tenant labels. ## Decision and explain model Each Admission decision should have a bounded internal reason: ```text ADMIT_GUARANTEE ADMIT_EXCESS ADMIT_BORROWED WAIT_CAP WAIT_PARENT_CAP WAIT_INFLIGHT WAIT_RECEIVER_CREDIT WAIT_STAGING DEGRADE_DEADLINE REJECT_UNAUTHORIZED REJECT_INFEASIBLE ``` Example sanitized explain: ```json { "schema_version": 1, "scope": "engine_local:nic_group_0", "subject": "tenant_hash/foreground_get", "decision": "admit_guarantee", "plan_charge": { "nic_bytes": 1048576, "receiver_bytes": 1048576, "request_slots": 1 }, "budget": { "min_rate_bps": 80000000000, "max_rate_bps": 160000000000, "weight": 8, "borrowed": false }, "capacity": { "value_bps": 190000000000, "fresh": true, "confidence": 0.91 } } ``` Per-request explain belongs in trace/sample/CLI queries, not as unbounded Prometheus labels. ## Metrics Required aggregate metrics per bounded configured class/scope: - admitted/dispatched/completed bytes and requests; - guaranteed, excess, and borrowed service bytes; - cap/burst throttled time and bytes; - inflight bytes/requests; - queue wait distribution; - reclaim latency; - starvation/maximum wait; - deadline miss and infeasible decisions; - degraded/rejected counts by bounded reason; - receiver-credit/staging/transport stall time; - capacity freshness/confidence; - configured versus achieved service rate; - cap violation amount/time; - scheduler iterations and decision time. Benchmark-derived metrics: - guarantee attainment ratio; - cap error; - Jain fairness index for comparable weighted classes; - total utilization/goodput; - foreground P50/P95/P99 and TTFT where integrated; - background completion/starvation time; - wasted bytes before degrade/reject. ## Compatibility and rollout Suggested modes: ```text disabled -> current behavior (default) shadow -> account and emit decisions; never delay/reorder/reject enforced -> apply budgets within the declared scope ``` Rules: - absent QoS Contract preserves current behavior; - unknown/missing subject maps to compatibility default unless strict mode requires rejection; - shadow mode must compare predicted decisions with actual dispatch without modifying it; - enforced mode is opt-in until real-cluster qualification is complete; - disabling enforcement does not disable independent hard safety limits such as receiver capacity; - mixed-version peers follow receiver-credit capability negotiation; local enforcement does not assume remote enforcement; - a runtime must advertise its actual scope, not imply cluster-global QoS. ## Proposed PR sequence ### PR0: Model, simulator, and trace replay Allowed before production prerequisites merge: - pure hierarchical budget model; - deterministic virtual clock; - token/deficit arithmetic and invariants; - trace replay for offered-load sweeps; - no TENT submit/dispatch changes. ### PR1: Shadow accounting adapter After stable QoS Contract and minimal ResourceCharge: - resolve bounded QosSubject/effective policy; - compute network-byte charges; - emit shadow admit/wait/degrade reasons; - compare with actual dispatch; - no behavior change. ### PR2: Engine-local inflight cap and rate cap - enforce aggregate tenant/intent inflight and max-rate/burst limits; - all-or-nothing Admission integration; - default disabled; - Direct network-byte path first. ### PR3: Guarantee/excess DRR and borrowing - minimum target service for feasible configurations; - weighted excess sharing; - bounded borrowing/reclaim; - EDF/aging inside class; - scheduler cost/latency benchmarks. ### PR4: Staged plan accounting - consume Direct/Staged ResourceCharge from #2863; - avoid double charge; - reserve internal-progress budget; - integrate causal metrics. ### PR5: Receiver-local tenant allocation - allocate receiver credits by authenticated subject; - aggregate multiple senders; - preserve epoch/replay invariants; - requires agreed production credit control path. ### Later work - host-local coordination across multiple TE instances; - multi-resource fairness beyond typed independent gates; - connector identity/auth integration; - cluster/global coordination if a real production requirement emerges; - hardware wire-QoS validation and policy mapping. ## Correctness invariants The model and implementation must prove/test: 1. No class or parent exceeds its configured inflight limit. 2. Long-term service does not exceed cap beyond the documented burst/quantum tolerance. 3. Request splitting does not increase parent entitlement. 4. Token/deficit arithmetic never overflows, underflows, or mints budget after duplicate updates. 5. In feasible stationary load, active guaranteed classes receive their configured target within bounded measurement tolerance. 6. Excess service is work-conserving subject to caps and resource availability. 7. A returning guaranteed class reclaims eligibility within a bounded quantum/inflight interval. 8. Eligible background traffic is not permanently starved. 9. Deadline ordering cannot bypass authorization, parent cap, or receiver capacity. 10. Atomic multi-resource reservation never exposes partial charge. 11. Late completion/release is charged to the correct plan/attempt/policy generation exactly once. 12. Receiver credit and QoS budget are both required where configured; neither is treated as the other. 13. Unknown/stale capacity never creates additional guaranteed budget. 14. Disabled mode preserves current behavior. ## Validation plan ### Deterministic model tests - virtual-time token refill and burst boundaries; - hierarchical parent/child caps; - idle guarantee accumulation bound; - weighted excess distribution; - borrowing and reclaim; - variable request sizes and DRR deficit; - request splitting/flow multiplication attacks; - EDF/aging inside class; - overload and proportional degradation; - policy generation change; - arithmetic/fuzz/property tests; - one million+ scheduling decisions without unbounded state growth. ### CPU integration tests - Admission all-or-nothing rollback; - mixed Direct/Staged charge fixtures; - receiver-credit available/unavailable; - late completion/release and attempt generation; - strict/compatibility identity behavior; - shadow mode produces no dispatch differences; - enforced mode respects caps under concurrent submitters; - sanitizer and TSan coverage. ### Real-cluster workload matrix Use `tebench` on the H20/RoCE cluster with at least: ```text topology: 1P1D and 2P2D senders: 1 / 4 / 16 paths: Direct RDMA / forced Staged / TCP fallback classes: foreground_get / background_prefetch / checkpoint offered load: below capacity / near congestion point / overload baselines: QoS disabled / strict priority / budget scheduler ``` For each run report: - configured and measured capacity; - per-class offered/accepted/completed rate; - guarantee attainment and cap error; - borrowed bytes and reclaim time; - queue wait and P50/P95/P99 latency; - deadline miss/degrade/reject; - background maximum wait/starvation; - receiver-credit and staging stalls; - data integrity; - total utilization and goodput; - scheduler CPU and submit overhead. ### Required scenarios 1. Foreground saturated, background active: foreground target protected, background bounded but not permanently starved. 2. Foreground idle: background borrows capacity up to cap. 3. Foreground returns: bounded reclaim without revoking posted work. 4. Tenant creates many small requests/flows: aggregate cap unchanged. 5. Two tenants with weights 8:2: excess share converges within defined tolerance. 6. Sum of guarantees infeasible: config rejects or explicit proportional-degrade mode reports violations. 7. Receiver slowdown: local tokens alone do not dispatch without receiver credit. 8. Stale bandwidth telemetry: conservative fallback, no oscillatory budget inflation. 9. Tight deadline under exhausted cap: authorized early degrade/reject, not cap bypass. 10. Rail/path failure: replan charge is revalidated; old and new attempts are not double charged. ### Claims boundary No performance/SLO claim should be made from light-load averages. Offered load must cross the congestion point, and results must include isolation, fairness, integrity, and total utilization. ## Expected benefits If prerequisites and invariants are satisfied, this RFC should provide: - bounded aggregate tenant/intent usage within the declared scope; - foreground protection without permanent background starvation; - work-conserving use of idle capacity; - explicit and measurable borrowing/reclaim; - consistent accounting across Direct/Staged/Fallback plans; - separation of authorization, sender budget, and receiver capacity; - early explainable overload decisions; - a path from policy schema to production QoS enforcement. It does not provide a hard cluster-wide SLA in v1. Any result must be labeled by actual enforcement scope and capacity assumptions. ## Relationship to existing work - #2856/#2857: source of effective QoS Contract fields; this RFC does not redefine the schema. - #2865: source of normalized request intent and requested/effective distinction. - #2863: source of Direct/Staged/Fallback plan and `ResourceCharge`. - #2864: configuration validation, status, and explain surface. - #2132: Local Admission Queue remains the dispatch ownership boundary. - #2655: coordinate with existing priority/byte-credit runtime queue work; do not create a second queue. - #2519/#2816: reuse deadline feasibility and live bandwidth inputs. - #2794: same-priority deadline arbitration remains a narrower mechanism inside authorized budget. - #2849/#2860-#2862: receiver-owned capacity and many-to-one gating; credits are not authorization. - #2845: reuse tebench QoS metrics/baselines. - #2568/#2847: QP pool/SL/TC and intent policy binding consume the effective class but do not enforce aggregate software budgets alone. - #2489/#1752: congestion/device controllers operate at a different timescale; this RFC does not replace them. ## Open questions 1. Do maintainers agree that this enforcement design should remain a separate follow-up to #2856 rather than extending the schema RFC? 2. What is the accepted v1 enforcement scope: engine-local only, or should receiver-local allocation be required before any enforcement is enabled? 3. What trusted tenant/subject identity should TENT consume from Store/connectors? 4. Should `min_bandwidth_gbps` be named/documented as a target minimum rather than a hard guarantee when capacity is infeasible? 5. Is hierarchical token bucket + DRR/WDRR + intra-class EDF the preferred baseline, or should v1 start with cap/inflight enforcement only? 6. How should oversubscribed minimums behave: reject configuration, proportional degradation, or explicit best-effort mode? 7. What resource domain should own a network budget: individual rail, NIC group, NUMA domain, or policy-selected device set? 8. What is an acceptable dispatch quantum/reclaim bound for foreground latency? 9. Should borrowed service be reported only, or should a bounded borrow limit be independently configurable? 10. Which fields from #2856 are stable enough for a shadow-accounting prototype? 11. Should enforcement remain blocked until #2863 `ResourceCharge` is accepted, or may a network-byte-only shadow prototype proceed earlier? 12. How should configured class identity be exported to metrics without unbounded label cardinality? ### Before submitting a new issue... - [x] Searched existing issues and PRs for QoS budget, tenant policing, bandwidth guarantee/cap, WDRR/DRR, token bucket, Admission, receiver credits, and multi-tenant scheduling. - [x] Reviewed #2132, #2519, #2655, #2794, #2849, #2856, #2857, #2860-#2865 and the relevant TENT roadmap items.

bsakaguchi

2026-10-29 21:19:01 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:02 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 12, 2026):

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2866
GitHub user ID	`18214026`
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 12, 2026): Thanks for opening this issue, @catyans! | Field | Value | |-------|-------| | **Issue** | #2866 | | **GitHub user ID** | `18214026` | | **Reporter** | @catyans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:02 +00:00

Author

Owner

@catyans commented on GitHub (Jul 12, 2026):

Deterministic shadow-scheduler experiment (2026-07-13) to test the core enforcement hypothesis before touching Admission. This is simulation evidence only.

Environment: Lingjun H20 lingjun-099, Python 3.12.7 / NumPy, fixed seed 20260713. Model: 10,000 x 1 ms epochs, capacity 100 work units/epoch; foreground arrivals Poisson(40/epoch), background Poisson(90/epoch), foreground deadline 5 epochs. Compared a single unbiased FIFO against a work-conserving 60/40 class budget; unused foreground capacity was borrowed by background.

Results:

mode	FG completed / arrived	FG miss	FG P99 wait	BG completed	utilization	final FG/BG backlog
FIFO	307,991 / 400,150	99.829%	2,286 epochs	692,009	100%	92,159 / 207,355
budget	400,150 / 400,150	0.000%	0 epochs	599,850	100%	0 / 299,514

Conclusion: under sustained aggregate overload, explicit class budgets can protect feasible foreground demand while remaining work-conserving; the benefit is not higher total throughput (both are 100% utilized), but controlled allocation. The cost is explicit: background service fell by 92,159 units and its backlog grew, so aging/reclaim and overload policy are mandatory. This does not validate real TENT timing, hierarchical identity, byte-vs-request accounting or distributed guarantees. A code PR should begin in shadow-accounting mode and reproduce this comparison with tebench and real RDMA before enforcement is enabled.

@catyans commented on GitHub (Jul 12, 2026): Deterministic shadow-scheduler experiment (2026-07-13) to test the core enforcement hypothesis before touching Admission. This is simulation evidence only. Environment: Lingjun H20 `lingjun-099`, Python 3.12.7 / NumPy, fixed seed 20260713. Model: 10,000 x 1 ms epochs, capacity 100 work units/epoch; foreground arrivals Poisson(40/epoch), background Poisson(90/epoch), foreground deadline 5 epochs. Compared a single unbiased FIFO against a work-conserving 60/40 class budget; unused foreground capacity was borrowed by background. Results: | mode | FG completed / arrived | FG miss | FG P99 wait | BG completed | utilization | final FG/BG backlog | |---|---:|---:|---:|---:|---:|---:| | FIFO | 307,991 / 400,150 | 99.829% | 2,286 epochs | 692,009 | 100% | 92,159 / 207,355 | | budget | 400,150 / 400,150 | 0.000% | 0 epochs | 599,850 | 100% | 0 / 299,514 | Conclusion: under sustained aggregate overload, explicit class budgets can protect feasible foreground demand while remaining work-conserving; the benefit is not higher total throughput (both are 100% utilized), but controlled allocation. The cost is explicit: background service fell by 92,159 units and its backlog grew, so aging/reclaim and overload policy are mandatory. This does not validate real TENT timing, hierarchical identity, byte-vs-request accounting or distributed guarantees. A code PR should begin in shadow-accounting mode and reproduce this comparison with tebench and real RDMA before enforcement is enabled.

bsakaguchi commented

2026-10-29 21:19:02 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 11, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 11, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:02 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.