[GH-ISSUE #2849] [RFC]: TENT Receiver-Advertised Credits and Upstream Backpressure #738

New issue

Closed

opened 2026-10-29 21:18:53 +00:00 by bsakaguchi · 11 comments

bsakaguchi commented 2026-10-29 21:18:53 +00:00

Owner

Copy link

Originally created by @catyans on GitHub (Jul 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2849

Changes proposed

Summary

This RFC proposes a receiver-advertised credit layer for TENT. It allows the node that will actually receive payload data to reserve and advertise bounded capacity before a sender dispatches work into a transport or remote staging path.

The proposal is a Phase 3 extension of #2132. It builds on the existing local runtime queue and dispatch window; it does not replace transport execution, RDMA congestion control, or the priority/byte-deficit work in #2655.

The key distinction is:

• DispatchByteCredit: sender-local scheduler accounting, as discussed in #2655.
• ReceiverAdvertisedCredit: receiver-owned resource reservation shared across nodes, proposed here.

The first implementation PR should add only the private credit state model, versioned message schema, and deterministic fault tests. It should not modify Admission scheduling or enable distributed credits by default.

This is the detailed follow-up for the receiver-credit part of #2132 Phase 3. It does not replace #2132's local queue design, and it deliberately does not duplicate #2655's sender-local dispatch byte credits.

Motivation

The current runtime queue can bound local outstanding owners/bytes and the amount of work handed to transports:

sender submit
  -> local queue admission
  -> local scheduling / dispatch window
  -> Transport::submitTransferTasks() or ProxyManager::submit()

This is necessary but insufficient for many-to-one traffic. With 16 Prefill senders targeting one Decode receiver, every sender can independently observe local capacity while the receiver has exhausted one of these resources:

• bytes that may safely be in flight;
• logical request/completion slots;
• remote staging chunks;
• consumer readiness or Decode-side commit capacity;
• receiver CPU/QP/CQ progress capacity.

PFC/ECN/DCQCN protect the fabric but do not express application buffer or consumer readiness. One-sided RDMA WRITE also does not synchronize the initiator with the remote consumer. TENT therefore needs an explicit receiver-side resource contract before remote dispatch.

Goals

1. Bound aggregate many-to-one work by receiver capacity.
2. Apply backpressure before work is buried in transport or ProxyManager queues.
3. Support Direct and Staged plans with different resource charges.
4. Make duplicate, reordered, stale, and post-restart credit updates safe.
5. Preserve old-peer and default-disabled behavior.
6. Expose structured backpressure to SGLang/vLLM without blocking scheduler threads.

Expected Benefits and Claims Boundary

If the implementation satisfies the invariants and acceptance criteria in this RFC, it provides the following enforceable properties. These properties do not depend on an assumed throughput or latency improvement:

1. Receiver-capacity safety for modeled resources. The sum of outstanding reservations across senders cannot exceed the receiver's configured capacity for DATA_BYTES, REQUEST_SLOTS, STAGING_SLOTS, and CONSUMER_SLOTS. A resource that is not modeled or charged is not covered by this guarantee.
2. Early, bounded backpressure. Work that lacks receiver capacity remains in the sender's bounded runtime queue instead of being buried in transport, ProxyManager, or remote-staging queues. This converts receiver overload into explicit credit stall rather than unbounded downstream accumulation.
3. Atomic multi-resource eligibility. A transfer is dispatched only after its complete charge vector is reserved. Partial reservations, such as bytes without a staging or consumer slot, are never exposed.
4. Staged-path progress isolation. Separately bounded control and staging-internal capacity prevents user payload from consuming the resources required to return credit or complete the inner work that releases an outer reservation.
5. Replay and restart safety. Duplicate, reordered, decreasing, overflowing, stale-epoch, and pre-restart updates cannot mint additional usable credit. Old sessions are fenced before their resources are reused.
6. Control-plane recovery at zero data credit. Credit, heartbeat, reset, cancel, and health traffic can make progress through a bounded control budget even when payload dispatch is stopped.
7. Actionable pressure attribution. The runtime can distinguish receiver-credit, local-queue, staging, and transport pressure. A later connector can use that typed signal to pause, retry, select another Decode node, fallback, or recompute instead of treating every condition as generic TooManyRequests.
8. Incremental deployment. Default-disabled behavior remains unchanged, capability negotiation supports mixed-version peers, and required mode fails or falls back explicitly rather than treating missing credit as unlimited capacity.

This RFC does not claim a numerical P99/TTFT reduction, higher total throughput, better SLO attainment, or tenant fairness before the real-cluster matrix below is completed. Those are experimental outcomes, not protocol invariants. Tenant guarantees, caps, weights, borrowing, and authorization also require a separate QoS-contract/policing design. Healthy-path utilization must be measured against a correctly tuned fixed window, while overload tests must report receiver occupancy, stalls, failures, and data integrity in addition to latency and throughput.

Non-goals

• A second WDRR/priority scheduler.
• Per-RTT congestion control or replacement of DCQCN/HPCC/ECN/PFC.
• A cluster-wide per-transfer executor.
• Revoking already-dispatched work.
• Solving tenant authentication or QoS contract policy in this RFC.
• Reusing Status::TooManyRequests without identifying the pressure source.

Terminology and Direction

receiver means the node whose memory/resource is receiving payload, not necessarily the API target:

Plan	Data receiver	Remote receiver credit?
RDMA WRITE to peer	remote target	yes
remote staging/delegate	remote staging/consumer	yes
RDMA READ from peer	local initiator	no; apply local resource limits
local SHM/NVLink	local process/host	local accounting only

A plan that does not consume remote receiver resources must not be stalled by an unrelated remote credit ledger.

Design Principles

1. A grant is a reservation, not an estimate

The receiver may use telemetry to decide how much capacity to reserve, but advertised credit itself means the receiver has committed that resource to the peer/class. Unknown telemetry must not create credit.

2. Absolute cumulative grants within an epoch

Delta grants can be double-applied after retransmission or reordering. Instead, for every credit key the receiver advertises monotonically increasing cumulative totals:

available(resource) = grant_total(resource) - consumed_total(resource)

The sender accepts an update only when:

• session_id and epoch match the active peer session;
• sequence is newer than the last accepted update;
• every cumulative grant is non-decreasing;
• subtraction and addition pass overflow/underflow checks.

Duplicate and old updates become no-ops.

3. Credits are non-revocable inside a live session

The receiver can stop granting more credit but cannot safely retract credit that the sender might already have consumed. Unused credit is returned explicitly or reclaimed after the peer session is fenced and old QPs/inflight operations are drained or failed.

The first version should keep initial grants small rather than implement unsafe timer-only reclamation.

4. Control traffic never consumes data credit

Credit update, usage, return, heartbeat, reset, cancel, and health messages use a bounded control budget. A zero data window must not prevent the message that reopens the window.

Credit Key and Resource Model

struct CreditKey {
    PeerSessionId receiver_session;
    PeerId sender_peer;
    uint32_t qos_class;
};

enum class CreditResource : uint16_t {
    DATA_BYTES = 1,
    REQUEST_SLOTS = 2,
    STAGING_SLOTS = 3,
    CONSUMER_SLOTS = 4,
};

struct CreditAmount {
    CreditResource resource;
    uint64_t grant_total;
};

The receiver owns a global capacity allocator. Per-peer/per-class grants are carved from that allocator so that the sum of outstanding reservations cannot exceed configured physical capacity.

An Execution Plan produces a charge vector:

struct CreditCharge {
    std::vector<std::pair<CreditResource, uint64_t>> resources;
};

Examples:

• Direct WRITE: DATA_BYTES + REQUEST_SLOTS + CONSUMER_SLOTS.
• Remote staged WRITE: the above plus STAGING_SLOTS.
• Local RDMA READ receive: local resource accounting, no remote ledger.

Proposed Wire Schema

The exact serialization can change, but the semantic fields must remain explicit and versioned:

struct ReceiverCreditUpdateV1 {
    uint16_t schema_version{1};
    uint16_t flags{0};
    uint32_t qos_class{0};

    UInt128 receiver_session_id;
    uint64_t epoch{0};
    uint64_t sequence{0};

    // Freshness hint, evaluated from the sender's local receive time.
    // It is not permission for the receiver to reclaim resources.
    uint32_t freshness_ttl_ms{0};

    std::vector<CreditAmount> grants;
};

struct SenderCreditUsageV1 {
    uint16_t schema_version{1};
    UInt128 receiver_session_id;
    uint64_t epoch{0};
    uint64_t sequence{0};
    uint32_t qos_class{0};
    std::vector<CreditAmount> consumed_totals;
    std::vector<CreditAmount> pending_demand;
};

Capability negotiation must happen during peer/bootstrap setup. The negotiated state is one of:

• Unsupported: peer has no receiver-credit capability.
• OptionalLegacy: configured rollout allows legacy bounded local dispatch.
• Required: unsupported or stale credit blocks/fallbacks instead of assuming infinity.
• Active(version): typed updates are accepted.

State Machine

Disabled ------------------------------> Legacy
   |
   v
Negotiating -- unsupported -----------> Legacy or Failed(required)
   |
   +-- compatible --------------------> Active
                                          |
                     stale/disconnect ----+----> Stale
                                          |       |
                                          |       +-- newer update -> Active
                                          |
                     new session/epoch ---+----> Fencing -> Active(new epoch)

Rules:

1. Active: new dispatch may reserve credit.
2. Stale: stop new credit-gated dispatch; existing transport-owned work continues to terminal state.
3. New epoch: invalidate unused old ledger state, but do not reuse old target resources until old QPs/inflight work are fenced.
4. Credit violation: fail/quarantine the peer and emit a protocol error metric; do not silently overdraw.
5. Counter overflow, unknown required resource, malformed schema, or decreasing cumulative total: reject the update.

Dispatch Integration

Receiver credit belongs at dispatch eligibility, not queue admission:

TryAdmit(local owners/bytes/lifetime)
  -> PickByLocalPolicy(priority/deadline/deficit)
  -> CreditLedger::TryReserve(plan.credit_charge)
       success: hand off to transport/proxy
       blocked: keep queued; record stall; request/update demand

The current pickForDispatch() transitions selected owners to Dispatching. A production integration therefore needs one agreed boundary:

1. an eligibility callback/filter before state transition; or
2. a safe requeueForDispatch() preserving original enqueue/aging time.

This should be coordinated with #2655 rather than implemented in the first credit PR.

Local dispatch capacity and remote credit are both required:

effective dispatch = local window
                   AND local scheduler eligibility
                   AND receiver credit for every required resource

Staging and Deadlock Avoidance

Credits should shape the physical inner transfers generated by ProxyManager, not only the outer user-visible staged request, matching #2132.

Staging-internal/control work needs reserved capacity that user payload cannot consume. Otherwise:

outer staged work reserves receiver capacity
  -> inner staging transfer waits for the same exhausted class
  -> outer work cannot complete and release capacity

The resource allocator must distinguish user payload, staging-internal progress, and control messages. Bypass is allowed only when separately bounded and accounted.

Control Transport

TENT's existing notification QP is a useful prototype path because it is separate from data QPs and already has configurable SL/TC. It is not yet the protocol contract:

• current notifications are string name/msg payloads;
• peer/session identity must be bound to the endpoint, not trusted from payload text;
• publisher enqueue must be nonblocking;
• updates need periodic refresh and failure metrics;
• control WR/CQ slots must remain available under payload saturation.

The first wire PR may use a fake transport and ControlService for tests. Using the notification QP in production should be a separate reviewed step.

Control transport alternatives

Alternative	Advantages	Risks / limitations	Recommendation
Extend BootstrapDesc	Reuses peer setup and capability negotiation	Bootstrap is not a recurring update channel; reconnect semantics become overloaded	Capability/version negotiation only
Typed ControlService RPC	Explicit schema, identity and error reporting; easy to fault-inject	Requires a bounded async client/server path and independent progress guarantees	Preferred first production transport
Typed notification message	Reuses the existing notification QP and configured control SL/TC	Current notification payload is string-based and its retry/identity contract is insufficient	Follow-up after typed framing and reserved WR/CQ budget
Piggyback on data completion	Low message count	Credit cannot recover when payload is stopped; creates a circular dependency	Do not use as the only update path

The protocol model must remain independent of the selected transport. PR 1 and PR 2 should test through a fake bounded transport so that transport selection cannot weaken epoch, ordering or capacity invariants.

Backpressure API

TooManyRequests currently conflates queue admission, batch capacity, transport pressure, and staging exhaustion. Receiver-credit stall should expose structured information:

struct BackpressureInfo {
    enum class Source { RECEIVER_CREDIT, LOCAL_QUEUE, STAGING, TRANSPORT };
    Source source;
    PeerId peer;
    uint32_t qos_class;
    std::vector<CreditResource> exhausted;
    uint64_t retry_after_us;
    bool fallback_allowed;
};

Initially this can remain internal and observable through metrics. A later connector API can surface BACKPRESSURE/RETRY_AFTER so SGLang/vLLM can pause submission, choose another Decode node, fallback, or recompute locally.

Metrics

Per bounded peer/class labels:

• granted/consumed/available by resource;
• credit-stalled owners/bytes and stall duration;
• update age, sequence gaps, duplicate/stale updates;
• epoch resets and fenced inflight work;
• explicit returns and idle reserved credit;
• protocol violations and legacy fallbacks;
• receiver global committed/free capacity.

Request IDs and unbounded tenant IDs belong in traces/sampling, not metric labels.

Compatibility and Rollout

• Default: disabled; current behavior unchanged.
• Optional mode: negotiate capability; legacy peers retain current bounded local dispatch with a visible metric.
• Required mode: unsupported/stale peers cannot enter the gated path; use configured fallback or fail explicitly.
• No existing public Request field changes in the first PR.
• Disablement is a config rollback; it does not require wire downgrade during an active session.

Proposed Code Placement and Concurrency Model

The exact file names are subject to maintainer feedback, but the ownership boundary should remain explicit:

Component	Tentative placement	Responsibility
CreditKey, CreditCharge, wire-neutral update types	tent/include/tent/runtime/receiver_credit.h	Private types and checked arithmetic
SenderCreditLedger	tent/src/runtime/receiver_credit.cpp	Apply cumulative grants and atomically reserve a complete charge vector
ReceiverCreditAllocator	tent/src/runtime/receiver_credit.cpp	Carve bounded grants from global receiver resources
Versioned wire codec	tent/include/tent/control/receiver_credit_protocol.h	Validate schema/version/length before constructing runtime types
Dispatch eligibility adapter	adjacent to Admission/runtime queue implementation	Gate transport handoff without changing queue admission semantics
Metrics	existing bounded TENT metrics registry	Stall, stale update, protocol violation and capacity gauges

Concurrency rules:

1. Control callbacks validate framing and enqueue updates into a bounded queue; they do not mutate Admission state or call a transport while holding a credit lock.
2. The runtime/dispatch owner drains updates and applies each credit-key mutation serially, or under a dedicated ledger shard lock if the current thread model requires it.
3. Multi-resource reserve is all-or-nothing under one ledger critical section. A partial reservation is never externally visible.
4. Receiver capacity allocation and resource release use the same ownership domain. Grant publication happens after reservation is committed.
5. No credit mutex may be held across transport submission, user callbacks, RPC, notification send, or blocking wait.
6. Completion only reports receiver resource release; it must not mint sender-side credit locally.

Memory is bounded by configured limits, not request count:

ledger entries <= max_peers * max_qos_classes * negotiated_resources
queued control updates <= control_queue_capacity
wire resources per update <= max_resources_per_update

Unknown peers/classes/resources, oversized updates and capacity multiplication overflow are rejected before allocation. Request IDs and tenant strings are not retained as unbounded ledger keys.

Security and Abuse Boundaries

• A sender cannot self-assign a QoS class or increase a grant; peer identity and authorized class come from the negotiated endpoint/policy context.
• A receiver never grants more than committed capacity, even if demand reports are forged or duplicated.
• Per-peer demand, update frequency and control-queue occupancy are bounded to prevent control-plane memory or CPU exhaustion.
• Malformed, decreasing, overflowing or wrong-epoch updates fail closed and increment a bounded protocol-error metric.
• A disconnected sender cannot retain reservations indefinitely: explicit return is preferred, while session fencing provides the safe reclamation boundary.
• Optional legacy mode is a rollout policy, not an implicit unlimited grant. Required mode must reject or take an explicit configured fallback.
• Debug traces may contain peer/request identifiers, but metrics must not use unbounded identifiers as labels.

Existing PG Mechanism: Cluster Evidence and Gaps

This RFC does not yet have a TENT implementation to benchmark. We therefore used Mooncake PG #1971 only as mechanism evidence, comparing upstream commit 4188e3a with the parent of #1971 (c3bafbed) on an 8×H20 host with four ~392 Gbit/s RDMA rails.

• The current receiver-driven PG passed 8/8 CPU/CUDA P2P functional tests; the pre-#1971 baseline passed 4/8.
• A single-rail isolation matrix passed 15/15 runs with wrong=0.
• The same high-risk cases on four rails had 5/15 abnormal runs: two silent data-corruption results and three P2P operation failures.
• A valid 32/64/128 MiB pool/chunk matrix passed 9/10 runs; one 64 MiB pool + 8 MiB chunk run lost progress and hit a 150-second watchdog.
• CUDA failed-rank and replacement-recovery tests both passed.
• A two-node run published the correct RDMA IPs but could not establish PG connectivity because the peer store value was shorter than sizeof(SegmentInfo); both sides timed out.

These results support receiver-owned credits as a correctness direction, but they do not validate TENT throughput, SLO, or isolation. Before dispatch gating is enabled, the acceptance gate must include strict data validation across one and multiple rails, bounded pool liveness, and a versioned cross-node metadata handshake. Full details are recorded in cluster-validation.md beside this draft.

Implementation Plan

PR 1 — Credit model and protocol invariants

• private CreditKey, CreditLedger, CreditUpdateV1;
• deterministic tests for duplicate, reorder, sequence gaps, stale epoch, reset, overflow, underflow, multi-resource atomic reserve and return;
• no background thread, network messages, Admission changes, or behavior change.

PR 2 — Capability and control transport

• version negotiation and optional/required rollout mode;
• fake transport fault injection: loss, delay, duplication, reordering, disconnect;
• endpoint-bound peer identity;
• bounded nonblocking publisher and freshness metrics.

PR 3 — Runtime dispatch gating

• coordinate eligibility/requeue contract with #2655;
• reserve all resource dimensions atomically before transport handoff;
• preserve original enqueue time while credit-stalled;
• direct WRITE first; RDMA READ remains local-only.

PR 4 — Staging and connector backpressure

• remote staging slots and consumer readiness;
• reserved staging-internal/control class;
• SGLang/vLLM typed backpressure and retry/fallback behavior.

Test and Experiment Matrix

Deterministic protocol tests

• duplicate/reordered/lost update;
• decreasing totals and integer overflow;
• sender/receiver restart and epoch mismatch;
• old completion after new epoch;
• control disconnect while payload is inflight;
• atomic multi-resource reservation failure;
• explicit unused-credit return;
• unsupported old peer in optional/required modes.

Real cluster experiments

Dimension	Values
Senders	1 / 4 / 16
Path	Direct / Staged / mixed
Mode	disabled / fixed local window / receiver credit
Receiver condition	normal / consumer slowdown / stage exhaustion / restart
Workload	foreground KV + checkpoint/background

Report:

• foreground P50/P99 and SLO attainment;
• background throughput and total utilization;
• receiver staging/CQ/queue peak occupancy;
• credit stall time and update overhead;
• failure/fallback/degraded-action counts;
• no data corruption and no unbounded receiver growth.

Acceptance Criteria

1. Aggregate reserved receiver resources never exceed configured capacity.
2. A sender never dispatches a required remote plan without all credit dimensions.
3. Duplicate, reordered, and stale updates never increase available credit.
4. Restarted peers cannot reuse old credit or old allocation generation.
5. Zero data credit cannot block control updates.
6. Under 1/4/16-sender incast, receiver queue/staging growth remains bounded.
7. Healthy-path utilization does not regress materially versus a correctly tuned fixed window.
8. Default-disabled and old-peer behavior remain compatible.

Related Work

• #2132 — local runtime admission queue and Phase 3 pressure/credits boundary.
• #2655 — sender-local priority lanes and byte deficit; deliberately not duplicated here.
• #1971 — Mooncake PG receiver-driven per-chunk credit protocol and epoch recovery.
• #2526 — configured SL/TC on TENT notification QP.
• RFC 9000 / RFC 9113 / RFC 9893 — receiver-owned flow-control window semantics.
• Homa / SIRD — receiver-driven scheduling and the need to account for shared bottlenecks.

Open Questions for Maintainers

1. Should the first wire transport extend BootstrapDesc, add a ControlService RPC, or define a typed notification protocol?
2. Should dispatch gating use an eligibility callback or a requeue transition in the runtime queue?
3. Which first path is most useful for production validation: direct RDMA WRITE or remote staging WRITE?
4. Should optional legacy mode remain available long term, or only during rollout?
5. Which component owns consumer readiness: TENT, Mooncake Store, or the SGLang/vLLM connector?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @catyans on GitHub (Jul 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2849 ### Changes proposed ## Summary This RFC proposes a receiver-advertised credit layer for TENT. It allows the node that will actually receive payload data to reserve and advertise bounded capacity before a sender dispatches work into a transport or remote staging path. The proposal is a Phase 3 extension of #2132. It builds on the existing local runtime queue and dispatch window; it does not replace transport execution, RDMA congestion control, or the priority/byte-deficit work in #2655. The key distinction is: - `DispatchByteCredit`: sender-local scheduler accounting, as discussed in #2655. - `ReceiverAdvertisedCredit`: receiver-owned resource reservation shared across nodes, proposed here. The first implementation PR should add only the private credit state model, versioned message schema, and deterministic fault tests. It should not modify Admission scheduling or enable distributed credits by default. This is the detailed follow-up for the receiver-credit part of #2132 Phase 3. It does not replace #2132's local queue design, and it deliberately does not duplicate #2655's sender-local dispatch byte credits. ## Motivation The current runtime queue can bound local outstanding owners/bytes and the amount of work handed to transports: ```text sender submit -> local queue admission -> local scheduling / dispatch window -> Transport::submitTransferTasks() or ProxyManager::submit() ``` This is necessary but insufficient for many-to-one traffic. With 16 Prefill senders targeting one Decode receiver, every sender can independently observe local capacity while the receiver has exhausted one of these resources: - bytes that may safely be in flight; - logical request/completion slots; - remote staging chunks; - consumer readiness or Decode-side commit capacity; - receiver CPU/QP/CQ progress capacity. PFC/ECN/DCQCN protect the fabric but do not express application buffer or consumer readiness. One-sided RDMA WRITE also does not synchronize the initiator with the remote consumer. TENT therefore needs an explicit receiver-side resource contract before remote dispatch. ## Goals 1. Bound aggregate many-to-one work by receiver capacity. 2. Apply backpressure before work is buried in transport or ProxyManager queues. 3. Support Direct and Staged plans with different resource charges. 4. Make duplicate, reordered, stale, and post-restart credit updates safe. 5. Preserve old-peer and default-disabled behavior. 6. Expose structured backpressure to SGLang/vLLM without blocking scheduler threads. ## Expected Benefits and Claims Boundary If the implementation satisfies the invariants and acceptance criteria in this RFC, it provides the following enforceable properties. These properties do not depend on an assumed throughput or latency improvement: 1. **Receiver-capacity safety for modeled resources.** The sum of outstanding reservations across senders cannot exceed the receiver's configured capacity for `DATA_BYTES`, `REQUEST_SLOTS`, `STAGING_SLOTS`, and `CONSUMER_SLOTS`. A resource that is not modeled or charged is not covered by this guarantee. 2. **Early, bounded backpressure.** Work that lacks receiver capacity remains in the sender's bounded runtime queue instead of being buried in transport, ProxyManager, or remote-staging queues. This converts receiver overload into explicit credit stall rather than unbounded downstream accumulation. 3. **Atomic multi-resource eligibility.** A transfer is dispatched only after its complete charge vector is reserved. Partial reservations, such as bytes without a staging or consumer slot, are never exposed. 4. **Staged-path progress isolation.** Separately bounded control and staging-internal capacity prevents user payload from consuming the resources required to return credit or complete the inner work that releases an outer reservation. 5. **Replay and restart safety.** Duplicate, reordered, decreasing, overflowing, stale-epoch, and pre-restart updates cannot mint additional usable credit. Old sessions are fenced before their resources are reused. 6. **Control-plane recovery at zero data credit.** Credit, heartbeat, reset, cancel, and health traffic can make progress through a bounded control budget even when payload dispatch is stopped. 7. **Actionable pressure attribution.** The runtime can distinguish receiver-credit, local-queue, staging, and transport pressure. A later connector can use that typed signal to pause, retry, select another Decode node, fallback, or recompute instead of treating every condition as generic `TooManyRequests`. 8. **Incremental deployment.** Default-disabled behavior remains unchanged, capability negotiation supports mixed-version peers, and required mode fails or falls back explicitly rather than treating missing credit as unlimited capacity. This RFC does **not** claim a numerical P99/TTFT reduction, higher total throughput, better SLO attainment, or tenant fairness before the real-cluster matrix below is completed. Those are experimental outcomes, not protocol invariants. Tenant guarantees, caps, weights, borrowing, and authorization also require a separate QoS-contract/policing design. Healthy-path utilization must be measured against a correctly tuned fixed window, while overload tests must report receiver occupancy, stalls, failures, and data integrity in addition to latency and throughput. ## Non-goals - A second WDRR/priority scheduler. - Per-RTT congestion control or replacement of DCQCN/HPCC/ECN/PFC. - A cluster-wide per-transfer executor. - Revoking already-dispatched work. - Solving tenant authentication or QoS contract policy in this RFC. - Reusing `Status::TooManyRequests` without identifying the pressure source. ## Terminology and Direction `receiver` means the node whose memory/resource is receiving payload, not necessarily the API target: | Plan | Data receiver | Remote receiver credit? | |---|---|---| | RDMA WRITE to peer | remote target | yes | | remote staging/delegate | remote staging/consumer | yes | | RDMA READ from peer | local initiator | no; apply local resource limits | | local SHM/NVLink | local process/host | local accounting only | A plan that does not consume remote receiver resources must not be stalled by an unrelated remote credit ledger. ## Design Principles ### 1. A grant is a reservation, not an estimate The receiver may use telemetry to decide how much capacity to reserve, but advertised credit itself means the receiver has committed that resource to the peer/class. Unknown telemetry must not create credit. ### 2. Absolute cumulative grants within an epoch Delta grants can be double-applied after retransmission or reordering. Instead, for every credit key the receiver advertises monotonically increasing cumulative totals: ```text available(resource) = grant_total(resource) - consumed_total(resource) ``` The sender accepts an update only when: - `session_id` and `epoch` match the active peer session; - `sequence` is newer than the last accepted update; - every cumulative grant is non-decreasing; - subtraction and addition pass overflow/underflow checks. Duplicate and old updates become no-ops. ### 3. Credits are non-revocable inside a live session The receiver can stop granting more credit but cannot safely retract credit that the sender might already have consumed. Unused credit is returned explicitly or reclaimed after the peer session is fenced and old QPs/inflight operations are drained or failed. The first version should keep initial grants small rather than implement unsafe timer-only reclamation. ### 4. Control traffic never consumes data credit Credit update, usage, return, heartbeat, reset, cancel, and health messages use a bounded control budget. A zero data window must not prevent the message that reopens the window. ## Credit Key and Resource Model ```cpp struct CreditKey { PeerSessionId receiver_session; PeerId sender_peer; uint32_t qos_class; }; enum class CreditResource : uint16_t { DATA_BYTES = 1, REQUEST_SLOTS = 2, STAGING_SLOTS = 3, CONSUMER_SLOTS = 4, }; struct CreditAmount { CreditResource resource; uint64_t grant_total; }; ``` The receiver owns a global capacity allocator. Per-peer/per-class grants are carved from that allocator so that the sum of outstanding reservations cannot exceed configured physical capacity. An Execution Plan produces a charge vector: ```cpp struct CreditCharge { std::vector<std::pair<CreditResource, uint64_t>> resources; }; ``` Examples: - Direct WRITE: `DATA_BYTES + REQUEST_SLOTS + CONSUMER_SLOTS`. - Remote staged WRITE: the above plus `STAGING_SLOTS`. - Local RDMA READ receive: local resource accounting, no remote ledger. ## Proposed Wire Schema The exact serialization can change, but the semantic fields must remain explicit and versioned: ```cpp struct ReceiverCreditUpdateV1 { uint16_t schema_version{1}; uint16_t flags{0}; uint32_t qos_class{0}; UInt128 receiver_session_id; uint64_t epoch{0}; uint64_t sequence{0}; // Freshness hint, evaluated from the sender's local receive time. // It is not permission for the receiver to reclaim resources. uint32_t freshness_ttl_ms{0}; std::vector<CreditAmount> grants; }; struct SenderCreditUsageV1 { uint16_t schema_version{1}; UInt128 receiver_session_id; uint64_t epoch{0}; uint64_t sequence{0}; uint32_t qos_class{0}; std::vector<CreditAmount> consumed_totals; std::vector<CreditAmount> pending_demand; }; ``` Capability negotiation must happen during peer/bootstrap setup. The negotiated state is one of: - `Unsupported`: peer has no receiver-credit capability. - `OptionalLegacy`: configured rollout allows legacy bounded local dispatch. - `Required`: unsupported or stale credit blocks/fallbacks instead of assuming infinity. - `Active(version)`: typed updates are accepted. ## State Machine ```text Disabled ------------------------------> Legacy | v Negotiating -- unsupported -----------> Legacy or Failed(required) | +-- compatible --------------------> Active | stale/disconnect ----+----> Stale | | | +-- newer update -> Active | new session/epoch ---+----> Fencing -> Active(new epoch) ``` Rules: 1. `Active`: new dispatch may reserve credit. 2. `Stale`: stop new credit-gated dispatch; existing transport-owned work continues to terminal state. 3. New epoch: invalidate unused old ledger state, but do not reuse old target resources until old QPs/inflight work are fenced. 4. Credit violation: fail/quarantine the peer and emit a protocol error metric; do not silently overdraw. 5. Counter overflow, unknown required resource, malformed schema, or decreasing cumulative total: reject the update. ## Dispatch Integration Receiver credit belongs at dispatch eligibility, not queue admission: ```text TryAdmit(local owners/bytes/lifetime) -> PickByLocalPolicy(priority/deadline/deficit) -> CreditLedger::TryReserve(plan.credit_charge) success: hand off to transport/proxy blocked: keep queued; record stall; request/update demand ``` The current `pickForDispatch()` transitions selected owners to Dispatching. A production integration therefore needs one agreed boundary: 1. an eligibility callback/filter before state transition; or 2. a safe `requeueForDispatch()` preserving original enqueue/aging time. This should be coordinated with #2655 rather than implemented in the first credit PR. Local dispatch capacity and remote credit are both required: ```text effective dispatch = local window AND local scheduler eligibility AND receiver credit for every required resource ``` ## Staging and Deadlock Avoidance Credits should shape the physical inner transfers generated by ProxyManager, not only the outer user-visible staged request, matching #2132. Staging-internal/control work needs reserved capacity that user payload cannot consume. Otherwise: ```text outer staged work reserves receiver capacity -> inner staging transfer waits for the same exhausted class -> outer work cannot complete and release capacity ``` The resource allocator must distinguish user payload, staging-internal progress, and control messages. Bypass is allowed only when separately bounded and accounted. ## Control Transport TENT's existing notification QP is a useful prototype path because it is separate from data QPs and already has configurable SL/TC. It is not yet the protocol contract: - current notifications are string `name/msg` payloads; - peer/session identity must be bound to the endpoint, not trusted from payload text; - publisher enqueue must be nonblocking; - updates need periodic refresh and failure metrics; - control WR/CQ slots must remain available under payload saturation. The first wire PR may use a fake transport and ControlService for tests. Using the notification QP in production should be a separate reviewed step. ### Control transport alternatives | Alternative | Advantages | Risks / limitations | Recommendation | |---|---|---|---| | Extend `BootstrapDesc` | Reuses peer setup and capability negotiation | Bootstrap is not a recurring update channel; reconnect semantics become overloaded | Capability/version negotiation only | | Typed `ControlService` RPC | Explicit schema, identity and error reporting; easy to fault-inject | Requires a bounded async client/server path and independent progress guarantees | Preferred first production transport | | Typed notification message | Reuses the existing notification QP and configured control SL/TC | Current notification payload is string-based and its retry/identity contract is insufficient | Follow-up after typed framing and reserved WR/CQ budget | | Piggyback on data completion | Low message count | Credit cannot recover when payload is stopped; creates a circular dependency | Do not use as the only update path | The protocol model must remain independent of the selected transport. PR 1 and PR 2 should test through a fake bounded transport so that transport selection cannot weaken epoch, ordering or capacity invariants. ## Backpressure API `TooManyRequests` currently conflates queue admission, batch capacity, transport pressure, and staging exhaustion. Receiver-credit stall should expose structured information: ```cpp struct BackpressureInfo { enum class Source { RECEIVER_CREDIT, LOCAL_QUEUE, STAGING, TRANSPORT }; Source source; PeerId peer; uint32_t qos_class; std::vector<CreditResource> exhausted; uint64_t retry_after_us; bool fallback_allowed; }; ``` Initially this can remain internal and observable through metrics. A later connector API can surface `BACKPRESSURE/RETRY_AFTER` so SGLang/vLLM can pause submission, choose another Decode node, fallback, or recompute locally. ## Metrics Per bounded peer/class labels: - granted/consumed/available by resource; - credit-stalled owners/bytes and stall duration; - update age, sequence gaps, duplicate/stale updates; - epoch resets and fenced inflight work; - explicit returns and idle reserved credit; - protocol violations and legacy fallbacks; - receiver global committed/free capacity. Request IDs and unbounded tenant IDs belong in traces/sampling, not metric labels. ## Compatibility and Rollout - Default: disabled; current behavior unchanged. - Optional mode: negotiate capability; legacy peers retain current bounded local dispatch with a visible metric. - Required mode: unsupported/stale peers cannot enter the gated path; use configured fallback or fail explicitly. - No existing public `Request` field changes in the first PR. - Disablement is a config rollback; it does not require wire downgrade during an active session. ## Proposed Code Placement and Concurrency Model The exact file names are subject to maintainer feedback, but the ownership boundary should remain explicit: | Component | Tentative placement | Responsibility | |---|---|---| | `CreditKey`, `CreditCharge`, wire-neutral update types | `tent/include/tent/runtime/receiver_credit.h` | Private types and checked arithmetic | | `SenderCreditLedger` | `tent/src/runtime/receiver_credit.cpp` | Apply cumulative grants and atomically reserve a complete charge vector | | `ReceiverCreditAllocator` | `tent/src/runtime/receiver_credit.cpp` | Carve bounded grants from global receiver resources | | Versioned wire codec | `tent/include/tent/control/receiver_credit_protocol.h` | Validate schema/version/length before constructing runtime types | | Dispatch eligibility adapter | adjacent to Admission/runtime queue implementation | Gate transport handoff without changing queue admission semantics | | Metrics | existing bounded TENT metrics registry | Stall, stale update, protocol violation and capacity gauges | Concurrency rules: 1. Control callbacks validate framing and enqueue updates into a bounded queue; they do not mutate Admission state or call a transport while holding a credit lock. 2. The runtime/dispatch owner drains updates and applies each credit-key mutation serially, or under a dedicated ledger shard lock if the current thread model requires it. 3. Multi-resource reserve is all-or-nothing under one ledger critical section. A partial reservation is never externally visible. 4. Receiver capacity allocation and resource release use the same ownership domain. Grant publication happens after reservation is committed. 5. No credit mutex may be held across transport submission, user callbacks, RPC, notification send, or blocking wait. 6. Completion only reports receiver resource release; it must not mint sender-side credit locally. Memory is bounded by configured limits, not request count: ```text ledger entries <= max_peers * max_qos_classes * negotiated_resources queued control updates <= control_queue_capacity wire resources per update <= max_resources_per_update ``` Unknown peers/classes/resources, oversized updates and capacity multiplication overflow are rejected before allocation. Request IDs and tenant strings are not retained as unbounded ledger keys. ## Security and Abuse Boundaries - A sender cannot self-assign a QoS class or increase a grant; peer identity and authorized class come from the negotiated endpoint/policy context. - A receiver never grants more than committed capacity, even if demand reports are forged or duplicated. - Per-peer demand, update frequency and control-queue occupancy are bounded to prevent control-plane memory or CPU exhaustion. - Malformed, decreasing, overflowing or wrong-epoch updates fail closed and increment a bounded protocol-error metric. - A disconnected sender cannot retain reservations indefinitely: explicit return is preferred, while session fencing provides the safe reclamation boundary. - Optional legacy mode is a rollout policy, not an implicit unlimited grant. Required mode must reject or take an explicit configured fallback. - Debug traces may contain peer/request identifiers, but metrics must not use unbounded identifiers as labels. ## Existing PG Mechanism: Cluster Evidence and Gaps This RFC does not yet have a TENT implementation to benchmark. We therefore used Mooncake PG #1971 only as mechanism evidence, comparing upstream commit `4188e3a` with the parent of #1971 (`c3bafbed`) on an 8×H20 host with four ~392 Gbit/s RDMA rails. - The current receiver-driven PG passed 8/8 CPU/CUDA P2P functional tests; the pre-#1971 baseline passed 4/8. - A single-rail isolation matrix passed 15/15 runs with `wrong=0`. - The same high-risk cases on four rails had 5/15 abnormal runs: two silent data-corruption results and three P2P operation failures. - A valid 32/64/128 MiB pool/chunk matrix passed 9/10 runs; one `64 MiB pool + 8 MiB chunk` run lost progress and hit a 150-second watchdog. - CUDA failed-rank and replacement-recovery tests both passed. - A two-node run published the correct RDMA IPs but could not establish PG connectivity because the peer store value was shorter than `sizeof(SegmentInfo)`; both sides timed out. These results support receiver-owned credits as a correctness direction, but they do not validate TENT throughput, SLO, or isolation. Before dispatch gating is enabled, the acceptance gate must include strict data validation across one and multiple rails, bounded pool liveness, and a versioned cross-node metadata handshake. Full details are recorded in `cluster-validation.md` beside this draft. ## Implementation Plan ### PR 1 — Credit model and protocol invariants - private `CreditKey`, `CreditLedger`, `CreditUpdateV1`; - deterministic tests for duplicate, reorder, sequence gaps, stale epoch, reset, overflow, underflow, multi-resource atomic reserve and return; - no background thread, network messages, Admission changes, or behavior change. ### PR 2 — Capability and control transport - version negotiation and optional/required rollout mode; - fake transport fault injection: loss, delay, duplication, reordering, disconnect; - endpoint-bound peer identity; - bounded nonblocking publisher and freshness metrics. ### PR 3 — Runtime dispatch gating - coordinate eligibility/requeue contract with #2655; - reserve all resource dimensions atomically before transport handoff; - preserve original enqueue time while credit-stalled; - direct WRITE first; RDMA READ remains local-only. ### PR 4 — Staging and connector backpressure - remote staging slots and consumer readiness; - reserved staging-internal/control class; - SGLang/vLLM typed backpressure and retry/fallback behavior. ## Test and Experiment Matrix ### Deterministic protocol tests - duplicate/reordered/lost update; - decreasing totals and integer overflow; - sender/receiver restart and epoch mismatch; - old completion after new epoch; - control disconnect while payload is inflight; - atomic multi-resource reservation failure; - explicit unused-credit return; - unsupported old peer in optional/required modes. ### Real cluster experiments | Dimension | Values | |---|---| | Senders | 1 / 4 / 16 | | Path | Direct / Staged / mixed | | Mode | disabled / fixed local window / receiver credit | | Receiver condition | normal / consumer slowdown / stage exhaustion / restart | | Workload | foreground KV + checkpoint/background | Report: - foreground P50/P99 and SLO attainment; - background throughput and total utilization; - receiver staging/CQ/queue peak occupancy; - credit stall time and update overhead; - failure/fallback/degraded-action counts; - no data corruption and no unbounded receiver growth. ## Acceptance Criteria 1. Aggregate reserved receiver resources never exceed configured capacity. 2. A sender never dispatches a required remote plan without all credit dimensions. 3. Duplicate, reordered, and stale updates never increase available credit. 4. Restarted peers cannot reuse old credit or old allocation generation. 5. Zero data credit cannot block control updates. 6. Under 1/4/16-sender incast, receiver queue/staging growth remains bounded. 7. Healthy-path utilization does not regress materially versus a correctly tuned fixed window. 8. Default-disabled and old-peer behavior remain compatible. ## Related Work - #2132 — local runtime admission queue and Phase 3 pressure/credits boundary. - #2655 — sender-local priority lanes and byte deficit; deliberately not duplicated here. - #1971 — Mooncake PG receiver-driven per-chunk credit protocol and epoch recovery. - #2526 — configured SL/TC on TENT notification QP. - RFC 9000 / RFC 9113 / RFC 9893 — receiver-owned flow-control window semantics. - Homa / SIRD — receiver-driven scheduling and the need to account for shared bottlenecks. ## Open Questions for Maintainers 1. Should the first wire transport extend `BootstrapDesc`, add a ControlService RPC, or define a typed notification protocol? 2. Should dispatch gating use an eligibility callback or a requeue transition in the runtime queue? 3. Which first path is most useful for production validation: direct RDMA WRITE or remote staging WRITE? 4. Should optional legacy mode remain available long term, or only during rollout? 5. Which component owns consumer readiness: TENT, Mooncake Store, or the SGLang/vLLM connector? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi 2026-10-29 21:18:53 +00:00

• closed this issue
• added the
  RFC
  label

bsakaguchi commented 2026-10-29 21:18:54 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 10, 2026):

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2849
GitHub user ID	18214026
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4936957895 --> @github-actions[bot] commented on GitHub (Jul 10, 2026): Thanks for opening this issue, @catyans! | Field | Value | |-------|-------| | **Issue** | #2849 | | **GitHub user ID** | `18214026` | | **Reporter** | @catyans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:18:54 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 11, 2026):

Final consolidated receiver-credit experiment report

This comment supersedes all earlier experiment comments on this RFC. The benchmark-only branch is receiver-credit-experiment, at commit 8e01c8d5. It does not modify production Admission or dispatch.

What was corrected

The receiver JSON p99_us starts at a multi-slot lease demand and ends as individual slots complete. It is a lease-lifecycle / receiver-backlog metric, not a per-transfer end-to-end latency metric, so it must not be compared directly with fixed per-transfer demand latency.

Commit 8e01c8d5 moves the sender timer before credit acquisition. The sender P99 below therefore covers credit wait + request construction + RDMA transfer. Earlier interpretations of receiver p99_us as a per-transfer SLO metric are superseded.

Direct capacity-safety evidence

On the backup RDMA hosts, sender-local fixed behavior exceeded receiver capacity, while receiver-owned credits enforced it:

Condition	Configured capacity	Fixed peak / violations	Credit peak / violations
4-sender incast	8 slots	11 / 657	8 / 0
16-sender incast	8 slots	22 / 14,657	8 / 0
16 senders, 5 ms consumer slowdown	64 slots	81 / 14,103	64 / 0

The restored main cluster also reproduced the risk after reducing benchmark control polling from 50 us to the retained 10 us setting: one 16-sender fixed run reached 16/8 slots with 9 violations. Every corresponding credit run remained within capacity.

Final main-cluster 16-sender A/B

Five interleaved repetitions used 16 senders, an 8-slot receiver cap, 4 MiB transfers, lease batch 2, and the corrected sender end-to-end timer:

Metric (median of 5)	Fixed	Receiver credit	Delta
Receiver throughput	7.363 GB/s	7.393 GB/s	+0.41%
Mean of 16 sender P99s	119.200 us	119.675 us	+0.40%
Worst-sender P99 per run	128.0 us	122.0 us	-4.69%
Sender Jain fairness	0.999896	0.999615	-0.000281

All credit runs completed all offered operations with zero data errors, zero capacity violations, zero invalid releases, and zero final occupancy.

Main-cluster throughput coverage

The broader three-repetition matrix produced these receiver-throughput medians:

Condition	Fixed	Receiver credit	Delta
1 sender, healthy, 1 rail	35.219 GB/s	35.756 GB/s	+1.53%
4-sender incast, 8-slot cap	16.269 GB/s	16.186 GB/s	-0.51%
16 senders, 5 ms consumer slowdown	6.720 GB/s	7.365 GB/s	+9.59%
4 senders / 4 RDMA rails	28.601 GB/s	28.778 GB/s	+0.62%

Three integrity repetitions completed 12,288 4 MiB WRITE→READ pairs with zero data errors.

Parameter selection

• batch 1 caused a 7.75% 16-sender throughput deficit in the initial main-cluster matrix;
• batch 2 removed that deficit while retaining near-perfect fairness;
• batch 4 increased variance and produced about 6.7k receiver stalls per run;
• batch 8 reduced median throughput and produced about 14k stalls per run;
• 1 us polling improved the lease-lifecycle metric by only about 3 us over 10 us while increasing busy-poll cost, so it was reverted.

The retained benchmark operating point is batch 2 with 10 us polling. Batch 2 is evidence for an operating point, not a proposed production constant; production should adapt grant size to capacity, active senders, and SLO pressure.

Verification and claims boundary

The focused C++ tests, Python result-verifier tests, JSONL mechanical checks, offered/completed equality, capacity guards, and integrity checks pass.

These results support the RFC's core claim: receiver-owned credits can enforce an aggregate resource bound without a material throughput, end-to-end P99, or fairness regression in the tested workload. They do not yet validate production restart fencing, replay/reorder handling, typed ControlService messages, STAGING_SLOTS / CONSUMER_SLOTS, connector backpressure, or SGLang/vLLM TTFT improvements.

<!-- gh-comment-id:4943967348 --> @catyans commented on GitHub (Jul 11, 2026): ## Final consolidated receiver-credit experiment report This comment supersedes all earlier experiment comments on this RFC. The benchmark-only branch is [`receiver-credit-experiment`](https://github.com/catyans/Mooncake/tree/receiver-credit-experiment), at commit `8e01c8d5`. It does **not** modify production Admission or dispatch. ### What was corrected The receiver JSON `p99_us` starts at a multi-slot lease demand and ends as individual slots complete. It is a **lease-lifecycle / receiver-backlog metric**, not a per-transfer end-to-end latency metric, so it must not be compared directly with fixed per-transfer demand latency. Commit `8e01c8d5` moves the sender timer before credit acquisition. The sender P99 below therefore covers **credit wait + request construction + RDMA transfer**. Earlier interpretations of receiver `p99_us` as a per-transfer SLO metric are superseded. ### Direct capacity-safety evidence On the backup RDMA hosts, sender-local fixed behavior exceeded receiver capacity, while receiver-owned credits enforced it: | Condition | Configured capacity | Fixed peak / violations | Credit peak / violations | |---|---:|---:|---:| | 4-sender incast | 8 slots | 11 / 657 | 8 / 0 | | 16-sender incast | 8 slots | 22 / 14,657 | 8 / 0 | | 16 senders, 5 ms consumer slowdown | 64 slots | 81 / 14,103 | 64 / 0 | The restored main cluster also reproduced the risk after reducing benchmark control polling from 50 us to the retained 10 us setting: one 16-sender fixed run reached **16/8 slots with 9 violations**. Every corresponding credit run remained within capacity. ### Final main-cluster 16-sender A/B Five interleaved repetitions used 16 senders, an 8-slot receiver cap, 4 MiB transfers, lease batch 2, and the corrected sender end-to-end timer: | Metric (median of 5) | Fixed | Receiver credit | Delta | |---|---:|---:|---:| | Receiver throughput | 7.363 GB/s | 7.393 GB/s | +0.41% | | Mean of 16 sender P99s | 119.200 us | 119.675 us | +0.40% | | Worst-sender P99 per run | 128.0 us | 122.0 us | -4.69% | | Sender Jain fairness | 0.999896 | 0.999615 | -0.000281 | All credit runs completed all offered operations with zero data errors, zero capacity violations, zero invalid releases, and zero final occupancy. ### Main-cluster throughput coverage The broader three-repetition matrix produced these receiver-throughput medians: | Condition | Fixed | Receiver credit | Delta | |---|---:|---:|---:| | 1 sender, healthy, 1 rail | 35.219 GB/s | 35.756 GB/s | +1.53% | | 4-sender incast, 8-slot cap | 16.269 GB/s | 16.186 GB/s | -0.51% | | 16 senders, 5 ms consumer slowdown | 6.720 GB/s | 7.365 GB/s | +9.59% | | 4 senders / 4 RDMA rails | 28.601 GB/s | 28.778 GB/s | +0.62% | Three integrity repetitions completed **12,288 4 MiB WRITE→READ pairs** with zero data errors. ### Parameter selection - batch 1 caused a 7.75% 16-sender throughput deficit in the initial main-cluster matrix; - batch 2 removed that deficit while retaining near-perfect fairness; - batch 4 increased variance and produced about 6.7k receiver stalls per run; - batch 8 reduced median throughput and produced about 14k stalls per run; - 1 us polling improved the lease-lifecycle metric by only about 3 us over 10 us while increasing busy-poll cost, so it was reverted. The retained benchmark operating point is **batch 2 with 10 us polling**. Batch 2 is evidence for an operating point, not a proposed production constant; production should adapt grant size to capacity, active senders, and SLO pressure. ### Verification and claims boundary The focused C++ tests, Python result-verifier tests, JSONL mechanical checks, offered/completed equality, capacity guards, and integrity checks pass. These results support the RFC's core claim: receiver-owned credits can enforce an aggregate resource bound without a material throughput, end-to-end P99, or fairness regression in the tested workload. They do not yet validate production restart fencing, replay/reorder handling, typed ControlService messages, `STAGING_SLOTS` / `CONSUMER_SLOTS`, connector backpressure, or SGLang/vLLM TTFT improvements.

bsakaguchi commented 2026-10-29 21:18:54 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 11, 2026):

Draft benchmark PR opened: #2859

It contains the cleaned, benchmark-only tebench harness used to validate this RFC: receiver byte/slot capacity tracking, fixed-vs-credit demand/grant/release experiments, fixed-count integrity runs, corrected end-to-end sender timing, versioned JSONL, and mechanical verification.

The PR deliberately excludes production Admission/dispatch integration and links back to this RFC for design discussion and experimental evidence.

<!-- gh-comment-id:4945073215 --> @catyans commented on GitHub (Jul 11, 2026): Draft benchmark PR opened: #2859 It contains the cleaned, benchmark-only `tebench` harness used to validate this RFC: receiver byte/slot capacity tracking, fixed-vs-credit demand/grant/release experiments, fixed-count integrity runs, corrected end-to-end sender timing, versioned JSONL, and mechanical verification. The PR deliberately excludes production Admission/dispatch integration and links back to this RFC for design discussion and experimental evidence.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 11, 2026):

Draft PR1 is now open: #2860. It is intentionally limited to the private cumulative-grant ledger, versioned semantic types, epoch/replay fencing, bounded state, atomic multi-resource reserve/rollback, and deterministic protocol/concurrency tests. It does not add a control transport, capability negotiation, Admission/dispatch integration, background threads, or default behavior changes. Cluster validation on two independent H20 nodes includes 2000 repeated Release runs total, TSan, ASan/LeakSanitizer, and a CUDA-enabled TENT runtime build. I will keep PR2 transport-neutral behind a fake bounded control channel until the control-path choice in this RFC is aligned.

<!-- gh-comment-id:4946563263 --> @catyans commented on GitHub (Jul 11, 2026): Draft PR1 is now open: #2860. It is intentionally limited to the private cumulative-grant ledger, versioned semantic types, epoch/replay fencing, bounded state, atomic multi-resource reserve/rollback, and deterministic protocol/concurrency tests. It does not add a control transport, capability negotiation, Admission/dispatch integration, background threads, or default behavior changes. Cluster validation on two independent H20 nodes includes 2000 repeated Release runs total, TSan, ASan/LeakSanitizer, and a CUDA-enabled TENT runtime build. I will keep PR2 transport-neutral behind a fake bounded control channel until the control-path choice in this RFC is aligned.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 11, 2026):

PR1 #2860 is now Ready for Review with the full upstream matrix green and all modified coverable lines covered. The transport-neutral PR2 prototype has also passed two-node Release (1000 repeats per node), ASan/LSan, TSan, CUDA runtime build, bounded-inbox saturation, receiver restart fencing, and real cross-host mixed-version negotiation.

One production control-path decision is now blocking a clean PR2 interface:

1. Receiver push: the receiver pushes cumulative grants to the sender. This requires the negotiation request/update envelope to bind sender identity (the current CoroRpcAgent handler does not expose caller identity).
2. Sender pull: the sender polls by target/QoS/last-sequence and receives activation/update in the response. This naturally preserves target context but introduces polling semantics.

Both can reuse the validated capability, activation, epoch, bounded ingress, and peer-context models. Is push or pull preferred for TENT ControlService? I will keep the production RPC/default-off integration out of PR2 until this identity/control-flow choice is aligned, rather than baking a test-only constant into the interface.

<!-- gh-comment-id:4948007484 --> @catyans commented on GitHub (Jul 11, 2026): PR1 #2860 is now Ready for Review with the full upstream matrix green and all modified coverable lines covered. The transport-neutral PR2 prototype has also passed two-node Release (1000 repeats per node), ASan/LSan, TSan, CUDA runtime build, bounded-inbox saturation, receiver restart fencing, and real cross-host mixed-version negotiation. One production control-path decision is now blocking a clean PR2 interface: 1. **Receiver push**: the receiver pushes cumulative grants to the sender. This requires the negotiation request/update envelope to bind sender identity (the current `CoroRpcAgent` handler does not expose caller identity). 2. **Sender pull**: the sender polls by target/QoS/last-sequence and receives activation/update in the response. This naturally preserves target context but introduces polling semantics. Both can reuse the validated capability, activation, epoch, bounded ingress, and peer-context models. Is push or pull preferred for TENT ControlService? I will keep the production RPC/default-off integration out of PR2 until this identity/control-flow choice is aligned, rather than baking a test-only constant into the interface.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 11, 2026):

Implementation progress update (2026-07-12)

This comment records the current implementation boundary for the RFC. The core
credit invariants and a sender-side dispatch prototype now exist, but the RFC is
not yet implemented end to end. In particular, production control transport,
receiver allocation/telemetry, staged resources, typed backpressure, and
connectors remain follow-up work.

PR stack

Stage	PR	State	Implemented boundary
Benchmark mechanism/evidence	#2859	Draft	Benchmark-only fixed-vs-credit capacity, integrity, and performance harness; no production dispatch changes
PR1: ledger/invariants	#2860	Ready for review; 26 successful checks	Cumulative grants, epoch/session fencing, bounded sender ledger, atomic multi-resource reserve/rollback, malformed/replay/overflow/concurrency tests
PR2: control model	#2861	Draft; retry CI has no failing checks, 2 jobs still running	Capability state/codec, activation, endpoint/QoS peer context, bounded nonblocking inbox, wire validation, fake control-channel fault tests, mixed-version harness
PR3: dispatch gate	#2862	Draft; CI running	Queued-owner identity/charge snapshot, pre-dispatch atomic reserve, exhaustion defer/retry, submit-success commit, exact submit-failure rollback, restart rejection, default-off opt-in API

PR3 is a single squashed delta (8160d44) over the PR2 head (d2a3333). The
upstream GitHub view temporarily includes prerequisite commits because a PR
cannot use a fork-only PR2 branch as its base; after the prerequisites merge,
the visible diff will collapse to the single PR3 commit.

What is implemented now

RFC property	Status	Evidence / qualification
Monotonic absolute cumulative grants	Implemented	Duplicate/old/decreasing/overflow updates cannot mint credit
Session/epoch replay and restart safety	Implemented in model and dispatch snapshot	Old queued snapshots fail before transport submission
Atomic charge-vector reserve	Implemented	All resources reserve or none reserve
Exact rollback on submission failure	Implemented	Reservation token restores exact bytes/slots; committed tokens cannot be rolled back
Backpressure before transport handoff	Implemented in PR3 prototype	Insufficient credit produces zero transport submissions and requeues the owner
Bounded multi-owner dispatch	Implemented for modeled sender ledger	Two owners with one-owner grant submit exactly one before the cumulative grant increases
Default-disabled behavior	Implemented	receiver_credit/enabled=false by default
Fail-closed opt-in setup	Implemented	Receiver credit requires runtime queue; enabled-but-uninstalled control state rejects before task publication
Context/ledger lifetime	Implemented	Formal installation API retains shared ownership
Configured default QoS	Implemented	Optional provider; otherwise receiver_credit/default_qos_class

Current opt-in surface:

enable_runtime_queue=true
receiver_credit/enabled=true
receiver_credit/default_qos_class=0

installReceiverCreditDispatch(contexts, ledger, qos_provider);

PR3 cluster validation

All validation was run remotely on the Lingjun H20 cluster, primarily on an
8 x H20-3e host (192 visible vCPUs, 2 TiB RAM, GCC 13.3, CMake 3.28.3,
NVIDIA driver 580.105.08). No workstation results are used as cluster evidence.

Mode / campaign	Executions	Passed	Failed	Notes
Final Release suite	55	55	0	3 suites, 0.58 s
Final ASan + LSan suite	55	55	0	0 memory errors, 0 leaks; 1.75 s
Final TSan suite	55	55	0	0 data races; 1.06 s
Final CUDA-enabled suite	55	55	0	USE_CUDA=ON; 1.40 s
Admission queue repeat	31,000	31,000	0	31 cases x 1000
Atomic dispatch gate repeat	4,000	4,000	0	4 cases x 1000
Formal API E2E repeat	17,000	17,000	0	17 cases x 1000
Opt-in/config API repeat	4,000	4,000	0	4 cases x 1000
Review concurrency regressions	2,000	2,000	0	2 cases x 1000
Repeated-test total	58,000	58,000	0	Excludes the four final one-shot build matrices

xychart-beta
    title "PR3 repeated test executions (all passed)"
    x-axis [Admission, Gate, E2E, OptInAPI]
    y-axis "case executions" 0 --> 32000
    bar [31000, 4000, 17000, 4000]

Key deterministic observations:

Scenario	Initial bytes/slots	Final bytes/slots	Transport submits	Result
Successful submit	4096/1	0/0	1	committed
Injected submit failure	4096/1	4096/1	1 failed call	exact rollback
Exhausted credit	0/0	0/0	0	deferred/retryable
Stale snapshot after restart	old session 0/0	old session 0/0	0	rejected
Two owners, cumulative grant 4096/1	4096/1	0/0	1	no overdraw
Cumulative grant increases to 8192/2	+4096/+1 usable	0/0	2 total	both complete

Important remaining work

RFC area	Status / next requirement
Production control transport	Not implemented. Need maintainer direction on receiver push vs sender pull, then a typed bounded ControlService path with endpoint-bound identity
Bootstrap negotiation wiring	Model/codec exists, but it is not connected to real peer bootstrap/session lifecycle
Receiver global capacity allocator	Not implemented. Production grants are not yet carved from a global receiver-owned capacity pool across senders/classes
Receiver telemetry -> grant production	Not implemented. Current updates are supplied by tests/harnesses
Plan-aware charging	Incomplete. PR3 currently demonstrates DATA_BYTES + REQUEST_SLOTS; production gating must classify remote Direct WRITE vs local/READ and avoid charging unrelated paths
CONSUMER_SLOTS	Type exists; no production execution-plan charge/release integration yet
STAGING_SLOTS / ProxyManager inner work	Not implemented. Belongs to the staged-path follow-up
Independent control budget	Model principle only; zero-data-credit recovery has not been proven through a production control channel
Structured BackpressureInfo and metrics	Not implemented. No typed connector-facing source/retry/fallback signal or production stall/capacity metric set yet
SGLang/vLLM connector behavior	Not implemented. Retry, alternate Decode selection, fallback, and recompute remain PR4 work
Production 1/4/16-sender acceptance matrix	Benchmark mechanism evidence exists, but the matrix has not been rerun through the production TENT control+dispatch path because that path is not connected yet

Revised next steps

1. Review/merge PR1 and align the control-flow question before productionizing PR2.
2. Implement the selected typed ControlService push/pull flow, endpoint-bound
   identity, bootstrap negotiation, and receiver global allocator.
3. Connect Direct remote WRITE plan charging first; explicitly leave RDMA READ
   and local SHM/NVLink under local accounting.
4. Add bounded metrics and internal structured pressure attribution.
5. Run the RFC's real 1/4/16-sender normal/slowdown/restart matrix through the
   connected production path.
6. Add staging/consumer resources and connector-facing backpressure in PR4.

Explicitly not implemented

The following items are not present in the production path today. Passing
PR1/PR2/PR3 tests must not be interpreted as completing them:

• A real receiver-to-sender credit update channel in ControlService.
• Maintainer-selected receiver-push or sender-pull control semantics.
• Endpoint-authenticated sender identity on production credit messages.
• Real bootstrap capability/version negotiation between TENT peers.
• Automatic activation/deactivation tied to connection/session lifecycle.
• A receiver-global allocator that proves the sum of grants across all
  senders and QoS classes is at most physical receiver capacity.
• Receiver telemetry sampling and policy that converts free capacity into
  committed cumulative grants.
• Sender usage/demand reports on a production wire path.
• Explicit unused-credit return on a production wire path.
• Safe receiver-side reclamation after disconnect, including QP/inflight
  fencing before old capacity is reused.
• Periodic refresh, heartbeat, update-age handling, and reconnect recovery
  through the real control transport.
• A separately reserved control WR/CQ/message budget proven to progress at
  zero data credit.
• Execution-plan classification that gates only remote receiver-consuming
  operations; the prototype has not finalized Direct WRITE vs READ/local
  accounting.
• Production CONSUMER_SLOTS charging and release.
• Production STAGING_SLOTS charging and release.
• ProxyManager inner-transfer credit ownership and deadlock avoidance.
• Separate bounded staging-internal/control credit classes.
• Typed BackpressureInfo with exhausted resources, peer, QoS, retry hint,
  and fallback permission.
• Receiver-credit stall duration, grant/consume/available, update-age,
  violation, fencing, and global-capacity production metrics.
• Protocol-violation quarantine/failure policy in the real peer path.
• Optional-legacy and required-mode production fallback behavior.
• SGLang/vLLM pause, retry, alternate Decode selection, fallback, or local
  recompute integration.
• Production-path 1/4/16-sender incast validation.
• Production-path consumer slowdown, staging exhaustion, disconnect, and
  restart validation.
• Healthy-path comparison against a tuned fixed window after real control
  and dispatch wiring.
• Production-path P50/P99, SLO, occupancy, stall overhead, failure counts,
  and data-integrity report.
• A demonstrated TENT TTFT, bandwidth, throughput, or fairness improvement.

RFC acceptance-criteria status

RFC acceptance criterion	Current status	Reason
1. Aggregate reserved receiver resources never exceed configured capacity	Not yet satisfied end to end	Benchmark and sender-ledger evidence exists, but production receiver-global allocation is absent
2. Required remote plan never dispatches without every credit dimension	Partially satisfied	Atomic gate works for modeled bytes/slots; plan classification, consumer and staging dimensions remain
3. Duplicate/reordered/stale updates never increase available credit	Satisfied in protocol model	PR1/PR2 deterministic tests; production transport wiring remains
4. Restarted peers cannot reuse old credit/allocation generation	Partially satisfied	Sender snapshot/session fencing is tested; receiver QP/inflight reclamation is not implemented
5. Zero data credit cannot block control updates	Not yet satisfied in production	Bounded inbox model exists; no independent production control transport/budget
6. Receiver growth remains bounded under 1/4/16-sender incast	Not yet validated on production path	Benchmark harness evidence is not the connected TENT implementation
7. Healthy utilization does not regress materially vs tuned fixed window	Not yet validated on production path	Requires real control transport and receiver allocator
8. Default-disabled and old-peer behavior remain compatible	Default-disabled implemented; old-peer rollout partial	Default-off PR3 tests pass; real bootstrap optional/required negotiation is not wired

Current claims boundary

The implemented stack now validates the credit arithmetic, replay/restart
fencing, bounded sender-side dispatch decision, and exact rollback behavior. It
does not yet establish end-to-end receiver-capacity safety in production,
because real receiver capacity allocation and the production update transport
are not connected. PR3 also does not claim a new TCP/RDMA throughput or TTFT
gain; those measurements become meaningful only after that end-to-end path is
complete.

<!-- gh-comment-id:4948350049 --> @catyans commented on GitHub (Jul 11, 2026): ## Implementation progress update (2026-07-12) This comment records the current implementation boundary for the RFC. The core credit invariants and a sender-side dispatch prototype now exist, but the RFC is **not yet implemented end to end**. In particular, production control transport, receiver allocation/telemetry, staged resources, typed backpressure, and connectors remain follow-up work. ### PR stack | Stage | PR | State | Implemented boundary | |---|---|---|---| | Benchmark mechanism/evidence | #2859 | Draft | Benchmark-only fixed-vs-credit capacity, integrity, and performance harness; no production dispatch changes | | PR1: ledger/invariants | #2860 | Ready for review; 26 successful checks | Cumulative grants, epoch/session fencing, bounded sender ledger, atomic multi-resource reserve/rollback, malformed/replay/overflow/concurrency tests | | PR2: control model | #2861 | Draft; retry CI has no failing checks, 2 jobs still running | Capability state/codec, activation, endpoint/QoS peer context, bounded nonblocking inbox, wire validation, fake control-channel fault tests, mixed-version harness | | PR3: dispatch gate | #2862 | Draft; CI running | Queued-owner identity/charge snapshot, pre-dispatch atomic reserve, exhaustion defer/retry, submit-success commit, exact submit-failure rollback, restart rejection, default-off opt-in API | PR3 is a single squashed delta (`8160d44`) over the PR2 head (`d2a3333`). The upstream GitHub view temporarily includes prerequisite commits because a PR cannot use a fork-only PR2 branch as its base; after the prerequisites merge, the visible diff will collapse to the single PR3 commit. ### What is implemented now | RFC property | Status | Evidence / qualification | |---|---|---| | Monotonic absolute cumulative grants | Implemented | Duplicate/old/decreasing/overflow updates cannot mint credit | | Session/epoch replay and restart safety | Implemented in model and dispatch snapshot | Old queued snapshots fail before transport submission | | Atomic charge-vector reserve | Implemented | All resources reserve or none reserve | | Exact rollback on submission failure | Implemented | Reservation token restores exact bytes/slots; committed tokens cannot be rolled back | | Backpressure before transport handoff | Implemented in PR3 prototype | Insufficient credit produces zero transport submissions and requeues the owner | | Bounded multi-owner dispatch | Implemented for modeled sender ledger | Two owners with one-owner grant submit exactly one before the cumulative grant increases | | Default-disabled behavior | Implemented | `receiver_credit/enabled=false` by default | | Fail-closed opt-in setup | Implemented | Receiver credit requires runtime queue; enabled-but-uninstalled control state rejects before task publication | | Context/ledger lifetime | Implemented | Formal installation API retains shared ownership | | Configured default QoS | Implemented | Optional provider; otherwise `receiver_credit/default_qos_class` | Current opt-in surface: ```text enable_runtime_queue=true receiver_credit/enabled=true receiver_credit/default_qos_class=0 ``` ```cpp installReceiverCreditDispatch(contexts, ledger, qos_provider); ``` ### PR3 cluster validation All validation was run remotely on the Lingjun H20 cluster, primarily on an 8 x H20-3e host (192 visible vCPUs, 2 TiB RAM, GCC 13.3, CMake 3.28.3, NVIDIA driver 580.105.08). No workstation results are used as cluster evidence. | Mode / campaign | Executions | Passed | Failed | Notes | |---|---:|---:|---:|---| | Final Release suite | 55 | 55 | 0 | 3 suites, 0.58 s | | Final ASan + LSan suite | 55 | 55 | 0 | 0 memory errors, 0 leaks; 1.75 s | | Final TSan suite | 55 | 55 | 0 | 0 data races; 1.06 s | | Final CUDA-enabled suite | 55 | 55 | 0 | `USE_CUDA=ON`; 1.40 s | | Admission queue repeat | 31,000 | 31,000 | 0 | 31 cases x 1000 | | Atomic dispatch gate repeat | 4,000 | 4,000 | 0 | 4 cases x 1000 | | Formal API E2E repeat | 17,000 | 17,000 | 0 | 17 cases x 1000 | | Opt-in/config API repeat | 4,000 | 4,000 | 0 | 4 cases x 1000 | | Review concurrency regressions | 2,000 | 2,000 | 0 | 2 cases x 1000 | | **Repeated-test total** | **58,000** | **58,000** | **0** | Excludes the four final one-shot build matrices | ```mermaid xychart-beta title "PR3 repeated test executions (all passed)" x-axis [Admission, Gate, E2E, OptInAPI] y-axis "case executions" 0 --> 32000 bar [31000, 4000, 17000, 4000] ``` Key deterministic observations: | Scenario | Initial bytes/slots | Final bytes/slots | Transport submits | Result | |---|---:|---:|---:|---| | Successful submit | 4096/1 | 0/0 | 1 | committed | | Injected submit failure | 4096/1 | 4096/1 | 1 failed call | exact rollback | | Exhausted credit | 0/0 | 0/0 | 0 | deferred/retryable | | Stale snapshot after restart | old session 0/0 | old session 0/0 | 0 | rejected | | Two owners, cumulative grant 4096/1 | 4096/1 | 0/0 | 1 | no overdraw | | Cumulative grant increases to 8192/2 | +4096/+1 usable | 0/0 | 2 total | both complete | ### Important remaining work | RFC area | Status / next requirement | |---|---| | Production control transport | **Not implemented.** Need maintainer direction on receiver push vs sender pull, then a typed bounded `ControlService` path with endpoint-bound identity | | Bootstrap negotiation wiring | Model/codec exists, but it is not connected to real peer bootstrap/session lifecycle | | Receiver global capacity allocator | **Not implemented.** Production grants are not yet carved from a global receiver-owned capacity pool across senders/classes | | Receiver telemetry -> grant production | **Not implemented.** Current updates are supplied by tests/harnesses | | Plan-aware charging | **Incomplete.** PR3 currently demonstrates `DATA_BYTES + REQUEST_SLOTS`; production gating must classify remote Direct WRITE vs local/READ and avoid charging unrelated paths | | `CONSUMER_SLOTS` | Type exists; no production execution-plan charge/release integration yet | | `STAGING_SLOTS` / ProxyManager inner work | **Not implemented.** Belongs to the staged-path follow-up | | Independent control budget | Model principle only; zero-data-credit recovery has not been proven through a production control channel | | Structured `BackpressureInfo` and metrics | **Not implemented.** No typed connector-facing source/retry/fallback signal or production stall/capacity metric set yet | | SGLang/vLLM connector behavior | **Not implemented.** Retry, alternate Decode selection, fallback, and recompute remain PR4 work | | Production 1/4/16-sender acceptance matrix | Benchmark mechanism evidence exists, but the matrix has not been rerun through the production TENT control+dispatch path because that path is not connected yet | ### Revised next steps 1. Review/merge PR1 and align the control-flow question before productionizing PR2. 2. Implement the selected typed `ControlService` push/pull flow, endpoint-bound identity, bootstrap negotiation, and receiver global allocator. 3. Connect Direct remote WRITE plan charging first; explicitly leave RDMA READ and local SHM/NVLink under local accounting. 4. Add bounded metrics and internal structured pressure attribution. 5. Run the RFC's real 1/4/16-sender normal/slowdown/restart matrix through the connected production path. 6. Add staging/consumer resources and connector-facing backpressure in PR4. ### Explicitly not implemented The following items are **not present in the production path today**. Passing PR1/PR2/PR3 tests must not be interpreted as completing them: - [ ] A real receiver-to-sender credit update channel in `ControlService`. - [ ] Maintainer-selected receiver-push or sender-pull control semantics. - [ ] Endpoint-authenticated sender identity on production credit messages. - [ ] Real bootstrap capability/version negotiation between TENT peers. - [ ] Automatic activation/deactivation tied to connection/session lifecycle. - [ ] A receiver-global allocator that proves the sum of grants across all senders and QoS classes is at most physical receiver capacity. - [ ] Receiver telemetry sampling and policy that converts free capacity into committed cumulative grants. - [ ] Sender usage/demand reports on a production wire path. - [ ] Explicit unused-credit return on a production wire path. - [ ] Safe receiver-side reclamation after disconnect, including QP/inflight fencing before old capacity is reused. - [ ] Periodic refresh, heartbeat, update-age handling, and reconnect recovery through the real control transport. - [ ] A separately reserved control WR/CQ/message budget proven to progress at zero data credit. - [ ] Execution-plan classification that gates only remote receiver-consuming operations; the prototype has not finalized Direct WRITE vs READ/local accounting. - [ ] Production `CONSUMER_SLOTS` charging and release. - [ ] Production `STAGING_SLOTS` charging and release. - [ ] ProxyManager inner-transfer credit ownership and deadlock avoidance. - [ ] Separate bounded staging-internal/control credit classes. - [ ] Typed `BackpressureInfo` with exhausted resources, peer, QoS, retry hint, and fallback permission. - [ ] Receiver-credit stall duration, grant/consume/available, update-age, violation, fencing, and global-capacity production metrics. - [ ] Protocol-violation quarantine/failure policy in the real peer path. - [ ] Optional-legacy and required-mode production fallback behavior. - [ ] SGLang/vLLM pause, retry, alternate Decode selection, fallback, or local recompute integration. - [ ] Production-path 1/4/16-sender incast validation. - [ ] Production-path consumer slowdown, staging exhaustion, disconnect, and restart validation. - [ ] Healthy-path comparison against a tuned fixed window after real control and dispatch wiring. - [ ] Production-path P50/P99, SLO, occupancy, stall overhead, failure counts, and data-integrity report. - [ ] A demonstrated TENT TTFT, bandwidth, throughput, or fairness improvement. ### RFC acceptance-criteria status | RFC acceptance criterion | Current status | Reason | |---|---|---| | 1. Aggregate reserved receiver resources never exceed configured capacity | **Not yet satisfied end to end** | Benchmark and sender-ledger evidence exists, but production receiver-global allocation is absent | | 2. Required remote plan never dispatches without every credit dimension | **Partially satisfied** | Atomic gate works for modeled bytes/slots; plan classification, consumer and staging dimensions remain | | 3. Duplicate/reordered/stale updates never increase available credit | **Satisfied in protocol model** | PR1/PR2 deterministic tests; production transport wiring remains | | 4. Restarted peers cannot reuse old credit/allocation generation | **Partially satisfied** | Sender snapshot/session fencing is tested; receiver QP/inflight reclamation is not implemented | | 5. Zero data credit cannot block control updates | **Not yet satisfied in production** | Bounded inbox model exists; no independent production control transport/budget | | 6. Receiver growth remains bounded under 1/4/16-sender incast | **Not yet validated on production path** | Benchmark harness evidence is not the connected TENT implementation | | 7. Healthy utilization does not regress materially vs tuned fixed window | **Not yet validated on production path** | Requires real control transport and receiver allocator | | 8. Default-disabled and old-peer behavior remain compatible | **Default-disabled implemented; old-peer rollout partial** | Default-off PR3 tests pass; real bootstrap optional/required negotiation is not wired | ### Current claims boundary The implemented stack now validates the credit arithmetic, replay/restart fencing, bounded sender-side dispatch decision, and exact rollback behavior. It does **not** yet establish end-to-end receiver-capacity safety in production, because real receiver capacity allocation and the production update transport are not connected. PR3 also does not claim a new TCP/RDMA throughput or TTFT gain; those measurements become meaningful only after that end-to-end path is complete.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 12, 2026):

Fresh feasibility rerun on 2026-07-13 against the stacked dispatch-gate head 8160d44a (feat/receiver-credit-dispatch-gate).

Environment: Lingjun H20 node lingjun-099, Linux 5.10.134, 2 x Intel Xeon Platinum 8575C (192 vCPUs), 8 x NVIDIA H20-3e, driver 580.105.08, five visible mlx5 HCAs, GCC 13.3, Release build, USE_TENT=ON, USE_CUDA=OFF.

Results:

• focused model/control/dispatch/admission/runtime suite: 5/5 passed in 0.67 s (receiver_credit_test, receiver_credit_control_test, receiver_credit_dispatch_test, admission_queue_test, tent_runtime_queue_dispatch_test);
• model + control + atomic-dispatch suites repeated 100 rounds: 300/300 test executions passed, 18.64 s total;
• exhausted credit caused no transport submission; multi-resource reserve remained atomic; submit failure returned the exact reservation; stale receiver/session state remained non-dispatchable (covered by the focused suites).

Conclusion: this rerun supports the protocol/state-machine and dispatch-boundary feasibility claims: receiver capacity can be enforced as an atomic pre-submit condition without changing the disabled path. It is not evidence of RDMA throughput or TTFT improvement. The remaining production gate is still a multi-sender real-RDMA incast experiment with receiver occupancy, data validation and healthy-path utilization.

<!-- gh-comment-id:4952110935 --> @catyans commented on GitHub (Jul 12, 2026): Fresh feasibility rerun on 2026-07-13 against the stacked dispatch-gate head `8160d44a` (`feat/receiver-credit-dispatch-gate`). Environment: Lingjun H20 node `lingjun-099`, Linux 5.10.134, 2 x Intel Xeon Platinum 8575C (192 vCPUs), 8 x NVIDIA H20-3e, driver 580.105.08, five visible mlx5 HCAs, GCC 13.3, Release build, `USE_TENT=ON`, `USE_CUDA=OFF`. Results: - focused model/control/dispatch/admission/runtime suite: **5/5 passed** in 0.67 s (`receiver_credit_test`, `receiver_credit_control_test`, `receiver_credit_dispatch_test`, `admission_queue_test`, `tent_runtime_queue_dispatch_test`); - model + control + atomic-dispatch suites repeated 100 rounds: **300/300 test executions passed**, 18.64 s total; - exhausted credit caused no transport submission; multi-resource reserve remained atomic; submit failure returned the exact reservation; stale receiver/session state remained non-dispatchable (covered by the focused suites). Conclusion: this rerun supports the protocol/state-machine and dispatch-boundary feasibility claims: receiver capacity can be enforced as an atomic pre-submit condition without changing the disabled path. It is not evidence of RDMA throughput or TTFT improvement. The remaining production gate is still a multi-sender real-RDMA incast experiment with receiver occupancy, data validation and healthy-path utilization.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 14, 2026):

Production adaptive dispatch follow-up (2026-07-14)

This is follow-up prototype evidence, not a result from the current #2859 or #2862 PR heads. The tested code is on receiver-credit-production-control, latest commit bbad07ab.

The prototype adds an engine-independent adaptive dispatch limiter shared by the credit pull controller and runtime queue. The RPC callback publishes only an atomic effective-owner limit; the runtime queue applies the smaller of its configured static limit and the adaptive limit. Defaults are conservative: initial/max owner window 2, 20 ms stall threshold, multiplicative decrease, additive recovery after 512 healthy pulls, two-event hysteresis before permanently lowering the learned ceiling, and a hard clamp to runtime_queue/max_dispatch_owners.

Final real-RDMA validation

Two-node H20/RoCE cluster, one NUMA node, one reachable rail (mlx5_0), four submit threads, 4 MiB WRITE, batch size 1, 20-second measurement interval:

Final case	Throughput	Pull maximum	Pulls over 100 ms	Final adaptive state
Runtime queue only, static owner limit 2	40.967 GB/s	n/a	n/a	n/a
Receiver credit, default adaptive limit 2	41.043 GB/s	1.764 ms	0 / 12,814	owners=2, ceiling=2, no reductions
Receiver credit, deliberately start/max at 3	39.362 GB/s	208.085 ms	2 / 11,992	owners=2, ceiling=2, 2 reductions, 3 increases
Receiver credit, old static owner limit 3	19.032 GB/s	209.819 ms	12 / 1,617	no automatic recovery

The default adaptive case was within +0.19% of the matched no-credit/static-2 baseline, with no pull over 100 ms. Starting from the known-unsafe level 3 improved throughput by 106.82% over the old static-3 behavior and reduced >100 ms pulls from 12 to 2 while learning and recovering to ceiling 2.

Remote regression guard: 9/9 suites passed (receiver_credit_test, control, protocol, allocator, config, control service, controller, dispatch, and runtime-queue dispatch).

Claims boundary

This supports automatic recovery from a persistently unsafe dispatch window without a measured healthy-path throughput regression in this setup. It does not guarantee zero first-loss when an operator explicitly raises the ceiling above a validated level: the two-event hysteresis can still observe the first two roughly 200 ms RTO samples before learning the lower ceiling. Preventing even that first control-path loss requires the independent typed control QP / reserved WR-CQ budget proposed by this RFC.

A separate reviewable follow-up PR is still required; these results must not be attributed to the existing #2859 or #2862 code until that branch is opened and reviewed.

Control RPC cost and tail evidence

The production-shaped path measured the actual ControlClient -> ControlService -> ReceiverCreditAllocator exchange between two RDMA nodes, rather than a fake/in-process channel.

RPC-only case	Aggregate pull rate	P99 / receiver CPU
1 sender	about 20,000 pulls/s	64.7 us P99
4 concurrent senders	about 79,400 pulls/s	receiver CPU within 5.8--7.3 us/pull
16 concurrent senders	about 165,000 pulls/s	receiver CPU within 5.8--7.3 us/pull

A two-resource credit request is 136 bytes and its response is 148 bytes at the credit codec layer. On the captured TCP wire, the framed request/response were 167/171 bytes. The unloaded result shows that codec and allocator work are not the healthy-path bottleneck.

When TCP control RPC and RDMA WRITE shared mlx5_0, the result depended strongly on data dispatch concurrency:

4 MiB WRITE case	Throughput	Pull maximum	Pulls over 100 ms
4 submit threads, dispatch window 64	25--32 GB/s	204--209 ms	2--8/run
1 submit thread	30.44 GB/s	10.16 ms	0
2 submit threads	40.85 GB/s	469 us	0
4 submit threads, dispatch window 2	40.99 GB/s	552 us	0
4 submit threads, dispatch window 3	19.03 GB/s	209.82 ms	12

Dual-endpoint packet capture established the approximately 200 ms tail mechanism: the sender capture contained the initial 167-byte framed request, but the receiver did not see that TCP sequence until retransmission 201--207 ms later. After receipt, the receiver generated the 171-byte response in about 10--50 us. Linux reported a 200 ms minimum TCP RTO. This rules out allocator CPU, receiver lock contention, and coroutine execution as the long-tail source in the sampled events; the control packet was lost after the sender capture point while TCP control and RDMA payload shared the mlx5 path.

With the safe two-owner window, three alternating baseline/credit repetitions gave median throughput 39.56/40.76 GB/s. The three credit runs had maximum pull latency 722/592/584 us, with no pull above 1 ms. Marking the TCP flow DSCP 46 did not fix the unsafe/unbounded case on this VPC.

Therefore the measured RPC conclusion is: healthy control RPC overhead is small, but TCP does not provide independent progress when the shared RDMA path is overdriven. The adaptive limiter prevents repeated collapse after detecting an unsafe level; a separate typed control QP with reserved WR/CQ capacity remains the durable design for preventing even the first shared-path TCP loss.

Benefit assessment

The primary benefit is receiver-capacity safety under many-to-one load; throughput improvement is a secondary overload-recovery benefit.

Dimension	Measured result	Assessment
Healthy default vs matched no-credit/static-2 baseline	41.043 vs 40.967 GB/s (+0.19%)	No material healthy-path regression measured
Adaptive recovery vs unsafe static window 3	39.362 vs 19.032 GB/s (+106.82%)	Large overload-path recovery
Default adaptive RPC tail	0 / 12,814 pulls over 100 ms; max 1.764 ms	No RTO-scale tail in the default run
Unsafe-start RPC tail	2 vs 12 pulls over 100 ms	Repeated damage is bounded after learning
Receiver capacity violations in credit experiments	0	The central receiver-owned bound held in the tested matrices
Receiver CPU cost	about 5.8--7.3 us/pull	Control processing is small relative to the observed 200 ms network RTO

Operationally, the design turns aggregate receiver overload into an explicit bounded credit stall before transport handoff, rather than allowing independent senders to bury excess work in RDMA, staging, or receiver queues. Under a validated safe window, this protection was obtained without a measurable throughput penalty; when deliberately started at an unsafe level, adaptive control avoided the persistent throughput collapse of a static setting.

The evidence does not yet establish a numerical SGLang/vLLM TTFT or application SLO improvement. It also does not guarantee zero first-loss after an operator raises the ceiling above a validated level. Those claims require an application-level experiment and, for independent control progress, the proposed typed control QP with reserved WR/CQ capacity.

<!-- gh-comment-id:4967194856 --> @catyans commented on GitHub (Jul 14, 2026): ## Production adaptive dispatch follow-up (2026-07-14) This is follow-up prototype evidence, not a result from the current #2859 or #2862 PR heads. The tested code is on [`receiver-credit-production-control`](https://github.com/catyans/Mooncake/tree/receiver-credit-production-control), latest commit [`bbad07ab`](https://github.com/catyans/Mooncake/commit/bbad07ab). The prototype adds an engine-independent adaptive dispatch limiter shared by the credit pull controller and runtime queue. The RPC callback publishes only an atomic effective-owner limit; the runtime queue applies the smaller of its configured static limit and the adaptive limit. Defaults are conservative: initial/max owner window 2, 20 ms stall threshold, multiplicative decrease, additive recovery after 512 healthy pulls, two-event hysteresis before permanently lowering the learned ceiling, and a hard clamp to `runtime_queue/max_dispatch_owners`. ### Final real-RDMA validation Two-node H20/RoCE cluster, one NUMA node, one reachable rail (`mlx5_0`), four submit threads, 4 MiB WRITE, batch size 1, 20-second measurement interval: | Final case | Throughput | Pull maximum | Pulls over 100 ms | Final adaptive state | |---|---:|---:|---:|---| | Runtime queue only, static owner limit 2 | 40.967 GB/s | n/a | n/a | n/a | | Receiver credit, default adaptive limit 2 | 41.043 GB/s | 1.764 ms | 0 / 12,814 | owners=2, ceiling=2, no reductions | | Receiver credit, deliberately start/max at 3 | 39.362 GB/s | 208.085 ms | 2 / 11,992 | owners=2, ceiling=2, 2 reductions, 3 increases | | Receiver credit, old static owner limit 3 | 19.032 GB/s | 209.819 ms | 12 / 1,617 | no automatic recovery | The default adaptive case was within **+0.19%** of the matched no-credit/static-2 baseline, with no pull over 100 ms. Starting from the known-unsafe level 3 improved throughput by **106.82%** over the old static-3 behavior and reduced >100 ms pulls from 12 to 2 while learning and recovering to ceiling 2. Remote regression guard: **9/9 suites passed** (`receiver_credit_test`, control, protocol, allocator, config, control service, controller, dispatch, and runtime-queue dispatch). ### Claims boundary This supports automatic recovery from a persistently unsafe dispatch window without a measured healthy-path throughput regression in this setup. It does not guarantee zero first-loss when an operator explicitly raises the ceiling above a validated level: the two-event hysteresis can still observe the first two roughly 200 ms RTO samples before learning the lower ceiling. Preventing even that first control-path loss requires the independent typed control QP / reserved WR-CQ budget proposed by this RFC. A separate reviewable follow-up PR is still required; these results must not be attributed to the existing #2859 or #2862 code until that branch is opened and reviewed. ### Control RPC cost and tail evidence The production-shaped path measured the actual `ControlClient -> ControlService -> ReceiverCreditAllocator` exchange between two RDMA nodes, rather than a fake/in-process channel. | RPC-only case | Aggregate pull rate | P99 / receiver CPU | |---|---:|---:| | 1 sender | about 20,000 pulls/s | 64.7 us P99 | | 4 concurrent senders | about 79,400 pulls/s | receiver CPU within 5.8--7.3 us/pull | | 16 concurrent senders | about 165,000 pulls/s | receiver CPU within 5.8--7.3 us/pull | A two-resource credit request is **136 bytes** and its response is **148 bytes** at the credit codec layer. On the captured TCP wire, the framed request/response were **167/171 bytes**. The unloaded result shows that codec and allocator work are not the healthy-path bottleneck. When TCP control RPC and RDMA WRITE shared `mlx5_0`, the result depended strongly on data dispatch concurrency: | 4 MiB WRITE case | Throughput | Pull maximum | Pulls over 100 ms | |---|---:|---:|---:| | 4 submit threads, dispatch window 64 | 25--32 GB/s | 204--209 ms | 2--8/run | | 1 submit thread | 30.44 GB/s | 10.16 ms | 0 | | 2 submit threads | 40.85 GB/s | 469 us | 0 | | 4 submit threads, dispatch window 2 | 40.99 GB/s | 552 us | 0 | | 4 submit threads, dispatch window 3 | 19.03 GB/s | 209.82 ms | 12 | Dual-endpoint packet capture established the approximately 200 ms tail mechanism: the sender capture contained the initial 167-byte framed request, but the receiver did not see that TCP sequence until retransmission **201--207 ms** later. After receipt, the receiver generated the 171-byte response in about **10--50 us**. Linux reported a **200 ms minimum TCP RTO**. This rules out allocator CPU, receiver lock contention, and coroutine execution as the long-tail source in the sampled events; the control packet was lost after the sender capture point while TCP control and RDMA payload shared the mlx5 path. With the safe two-owner window, three alternating baseline/credit repetitions gave median throughput **39.56/40.76 GB/s**. The three credit runs had maximum pull latency **722/592/584 us**, with no pull above 1 ms. Marking the TCP flow DSCP 46 did not fix the unsafe/unbounded case on this VPC. Therefore the measured RPC conclusion is: healthy control RPC overhead is small, but TCP does not provide independent progress when the shared RDMA path is overdriven. The adaptive limiter prevents repeated collapse after detecting an unsafe level; a separate typed control QP with reserved WR/CQ capacity remains the durable design for preventing even the first shared-path TCP loss. ### Benefit assessment The primary benefit is **receiver-capacity safety under many-to-one load**; throughput improvement is a secondary overload-recovery benefit. | Dimension | Measured result | Assessment | |---|---:|---| | Healthy default vs matched no-credit/static-2 baseline | 41.043 vs 40.967 GB/s (**+0.19%**) | No material healthy-path regression measured | | Adaptive recovery vs unsafe static window 3 | 39.362 vs 19.032 GB/s (**+106.82%**) | Large overload-path recovery | | Default adaptive RPC tail | 0 / 12,814 pulls over 100 ms; max 1.764 ms | No RTO-scale tail in the default run | | Unsafe-start RPC tail | 2 vs 12 pulls over 100 ms | Repeated damage is bounded after learning | | Receiver capacity violations in credit experiments | 0 | The central receiver-owned bound held in the tested matrices | | Receiver CPU cost | about 5.8--7.3 us/pull | Control processing is small relative to the observed 200 ms network RTO | Operationally, the design turns aggregate receiver overload into an explicit bounded credit stall before transport handoff, rather than allowing independent senders to bury excess work in RDMA, staging, or receiver queues. Under a validated safe window, this protection was obtained without a measurable throughput penalty; when deliberately started at an unsafe level, adaptive control avoided the persistent throughput collapse of a static setting. The evidence does **not** yet establish a numerical SGLang/vLLM TTFT or application SLO improvement. It also does not guarantee zero first-loss after an operator raises the ceiling above a validated level. Those claims require an application-level experiment and, for independent control progress, the proposed typed control QP with reserved WR/CQ capacity.

bsakaguchi commented 2026-10-29 21:18:55 +00:00

Author

Owner

Copy link

@alogfans commented on GitHub (Jul 16, 2026):

Thanks for the detailed proposal and validation. However, we do not recommend proceeding with the RPC-based receiver-credit design at this stage.

Our primary concern is system complexity. This proposal introduces a new distributed control protocol into the critical transfer path, including additional RPCs, credit allocation and reconciliation, session/epoch fencing, capability negotiation, stale-state handling, retries, timeout recovery, restart recovery, and compatibility/fallback modes. These mechanisms substantially increase the number of system states and failure combinations that we need to reason about, test, and maintain.

More importantly, once credit RPC becomes part of dispatch eligibility, the availability of the data path also depends on the availability and correctness of this new control path. The existing experiments already show that the RPC path may experience TCP retransmission delays when sharing resources with RDMA traffic. Adaptive limiting mitigates some symptoms, but adds another stateful feedback mechanism rather than removing the underlying complexity.

We acknowledge that receiver-side capacity control addresses a real many-to-one overload problem. Nevertheless, at the current stage, we prefer simpler sender-local admission and dispatch-window controls. They may provide weaker global guarantees, but their behavior, operational cost, and failure modes are considerably easier to understand and maintain.

Therefore, we suggest keeping the benchmark and protocol-model work for future reference, while pausing the runtime dispatch-gate and production RPC integration. We can revisit this direction if simpler mechanisms prove insufficient and the additional protocol complexity is clearly justified by application-level benefits.

<!-- gh-comment-id:4992242147 --> @alogfans commented on GitHub (Jul 16, 2026): Thanks for the detailed proposal and validation. However, we do not recommend proceeding with the RPC-based receiver-credit design at this stage. Our primary concern is system complexity. This proposal introduces a new distributed control protocol into the critical transfer path, including additional RPCs, credit allocation and reconciliation, session/epoch fencing, capability negotiation, stale-state handling, retries, timeout recovery, restart recovery, and compatibility/fallback modes. These mechanisms substantially increase the number of system states and failure combinations that we need to reason about, test, and maintain. More importantly, once credit RPC becomes part of dispatch eligibility, the availability of the data path also depends on the availability and correctness of this new control path. The existing experiments already show that the RPC path may experience TCP retransmission delays when sharing resources with RDMA traffic. Adaptive limiting mitigates some symptoms, but adds another stateful feedback mechanism rather than removing the underlying complexity. We acknowledge that receiver-side capacity control addresses a real many-to-one overload problem. Nevertheless, at the current stage, we prefer simpler sender-local admission and dispatch-window controls. They may provide weaker global guarantees, but their behavior, operational cost, and failure modes are considerably easier to understand and maintain. Therefore, we suggest keeping the benchmark and protocol-model work for future reference, while pausing the runtime dispatch-gate and production RPC integration. We can revisit this direction if simpler mechanisms prove insufficient and the additional protocol complexity is clearly justified by application-level benefits.

bsakaguchi commented 2026-10-29 21:18:56 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 21, 2026):

Thanks for the clear direction. Agreed — I will pause the runtime dispatch-gate and production RPC integration work in #2862 and #2925, and keep the benchmark and protocol-model work only as reference. I will refocus any follow-up on simpler sender-local admission and dispatch-window controls, without introducing a new distributed control-path dependency.

<!-- gh-comment-id:5032118107 --> @catyans commented on GitHub (Jul 21, 2026): Thanks for the clear direction. Agreed — I will pause the runtime dispatch-gate and production RPC integration work in #2862 and #2925, and keep the benchmark and protocol-model work only as reference. I will refocus any follow-up on simpler sender-local admission and dispatch-window controls, without introducing a new distributed control-path dependency.

bsakaguchi commented 2026-10-29 21:18:56 +00:00

Author

Owner

Copy link

@catyans commented on GitHub (Jul 31, 2026):

Closing as not planned after the maintainer decision above. The RPC-based receiver-credit PR stack has been closed; future work will focus on simpler sender-local admission and dispatch-window controls without a new distributed control-path dependency.

<!-- gh-comment-id:5140418896 --> @catyans commented on GitHub (Jul 31, 2026): Closing as not planned after the maintainer decision above. The RPC-based receiver-credit PR stack has been closed; future work will focus on simpler sender-local admission and dispatch-window controls without a new distributed control-path dependency.

bsakaguchi referenced this issue 2026-10-29 21:30:18 +00:00

[PR #738] [MERGED] [Transfer Engine] Metrics: Add total qp metrics #1783

bsakaguchi referenced this issue 2026-10-29 21:30:21 +00:00

[PR #742] [CLOSED] [Store] Add master side QP calculation #1786

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#738

No description provided.