kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2519] [RFC]: Deadline-aware admission & adaptive degradation for TENT transfers #670

New issue

Closed

opened 2026-10-29 21:17:55 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-29 21:17:55 +00:00

Owner

Originally created by @catyans on GitHub (Jun 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2519

Originally assigned to: @catyans on GitHub.

Verified against kvcache-ai/Mooncake main @ 94ac60df (2026-06-17).

Builds on #2132 (TENT Local Transfer Admission Queue). That RFC explicitly defers priority/deadline policy to a later milestone ("if a later queued submit path adds priority or deadline policy", "once retry budget or deadline is exhausted"). This RFC proposes that deadline layer concretely. It is purely additive: it reuses #2132's QueueOwner model and dispatch boundary, changes nothing when no deadline is supplied, and does not touch the WDRR default or the worker priority queues. Filing it separately (rather than as a comment on #2132) only because it introduces a new request field and an external degradation hook that are easier to track as their own thread; happy to merge the discussion into #2132 if maintainers prefer.

Motivation

TENT already prioritizes traffic well at the ordering level (documented in docs/source/design/tent/qos.md):

per-request priority (PRIO_HIGH/MEDIUM/LOW, tent/common/types.h)
per-priority slice queues with strict HIGH→LOW dequeue (workers.h, workers.cpp)
starvation prevention via timed promotion (promoteTimedOutRequests, workers.cpp)
cross-process coordination via time-sliced slots (SharedSlotManager, shared_quota.h)
priority-aware device selection (quota.h, enable_priority_filtering)

All of this answers "given a set of transfers, in what order do we send them?" What's still missing is the SLO question:

Will this P→D transfer finish before its TTFT deadline — and if not, what do we do instead of pushing it through and timing out?

Three concrete gaps:

No per-transfer deadline. Request has priority but no deadline, so the scheduler can't tell "HIGH with 8 ms left" from "HIGH with 80 ms left."
Admission is capacity-only, FIFO. LocalTransferAdmissionQueue::pickForDispatch (admission_queue.cpp:185) admits/dispatches purely on outstanding bytes/owners in FIFO order. It cannot reshape or reject a transfer that provably can't meet its deadline.
No degradation path. Under saturation the only outcomes are "send slower" or "time out." There's no wired-in "send a losslessly compressed version" or "skip transfer, recompute locally."

This is the SLO-guarantee capability implied by #1883 Milestone 3 that the ordering machinery alone cannot provide.

Scope

In scope:

An optional deadline_ns field on Request (backward compatible).
A feasibility metric (MLU) computed from the existing EWMA bandwidth estimate + slice timestamps — no new measurement subsystem.
Deadline-aware admission/dispatch as an opt-in policy inside LocalTransferAdmissionQueue; default FIFO behavior unchanged when no deadline is supplied.
A degradation hook interface (compression / local-decode signal).

Out of scope (kept as follow-up PRs to keep this mergeable):

The compression codec itself (dependency decision).
The LOCAL_DECODE implementation (lives in vLLM/SGLang; this RFC only defines the signal TENT raises).
Any change to worker priority queues or cross-process slots.

Design

1. Deadline on the request

// tent/common/types.h, struct Request
uint64_t deadline_ns = 0;  // absolute steady-clock ns; 0 = no deadline (default)

0 ⇒ behaves exactly as today (FIFO within priority). If the upper layer passes nothing, nothing changes. The connector (vLLM/SGLang) computes:

deadline_ns = now + (TTFT_SLO - t_queue_est - t_prefill_est - safety_margin)

TENT only consumes a deadline; it does not invent SLO numbers.

2. Feasibility metric (MLU)

MLU = remaining_bytes / (remaining_time × available_bw)

All three inputs already exist:

remaining_bytes — from RdmaTask accounting (transferred_bytes vs total).
remaining_time — deadline_ns - now.
available_bw — derived from the existing DeviceQuota::ewma_bandwidth_bps (quota.h:76) minus current in-flight load on the pinned device.

MLU < 1 feasible, MLU ≥ 1 infeasible. It is a standard work / (time × rate) feasibility ratio (effectively a laxity test). This direction is motivated by recent work on TTFT-SLO-aware flow scheduling for LLM serving — Multi-stage Flow Scheduling for LLM Serving (MFS, arXiv:2603.17456), which approximates Least-Laxity-First scheduling across reusable KV-block retrieval, collective communication, and P2D transfer to improve TTFT SLO attainment by 1.2–2.4×. MFS operates above the transfer layer; this RFC brings an equivalent laxity-driven decision into TENT's admission path, where it can also trigger degradation (compression / local decode) rather than only reorder.

3. Admission decision (one-time, at submit)

MLU	Action	Loss
`< θ_compress` (e.g. 0.3)	full transfer	0
`θ_compress … θ_local` (e.g. 0.3–1.5)	full transfer + lossless compression hook (if codec registered)	0
`≥ θ_local` (e.g. 1.5)	raise `LOCAL_DECODE_SUGGESTED`, do not transfer	0 (caller recomputes)

θ_compress / θ_local are config knobs. Non-deadline traffic (deadline_ns == 0, e.g. Store offload) skips MLU and falls through to today's path.

4. Dynamic re-evaluation (during transfer)

Re-compute MLU every N slices or M ms using live EWMA bandwidth:

MLU past θ_compress → compress not-yet-posted slices.
MLU past θ_local → cancel remaining transfer, raise LOCAL_DECODE_SUGGESTED.
Promotion is monotonic (promote-only) and reuses the existing promotion machinery in workers.cpp — no second mechanism.

5. Integration point

pickForDispatch(max_owners, max_bytes) is the single, clean injection site (currently FIFO, and this class is runtime-private, not yet on the hot path):

Add an optional DeadlinePolicy to QueueLimits.
When set: order queued owners by earliest deadline (EDF) and drop owners with MLU ≥ θ_local (raising the signal).
When unset (default): byte-for-byte the current FIFO.

6. Degradation hook (interface only)

struct DegradationHooks {
    // Lossless, reversible compression of a slice. Optional.
    std::function<Status(const Slice&, CompressedSlice&)> compress;
    // Raised when a transfer cannot meet its deadline. Optional.
    std::function<void(const Request&)> on_local_decode_suggested;
};

No hook registered ⇒ identical to current behavior: the θ_compress…θ_local band degrades to plain full transfer, and ≥ θ_local degrades to best-effort full transfer.

Estimated scope

Part	LoC
`Request.deadline_ns` + plumbing	~30
MLU computation (reusing EWMA + timestamps)	~80
Deadline policy in `pickForDispatch` (EDF + drop)	~120
Degradation hook interface + signal	~50
Total	~280 (codec & local-decode bodies excluded)

Relation to #1883

Milestone 3's ordering layer is satisfied by the shipped priority queues + cross-process slots. This RFC adds the SLO layer on top — admission and degradation driven by a deadline, which ordering cannot express. Strictly additive, gated behind an opt-in deadline.

Open questions

Deadline interface: deadline_ns on Request (proposed) vs a per-batch side-channel hint?
available_bw with multiple devices: a transfer is pinned to a device at allocate time — is per-device EWMA sufficient, or is an aggregated view needed?
Should the compression hook be split entirely into its own RFC/PR? (Leaning yes — the dependency decision shouldn't block the deadline mechanism.)
LOCAL_DECODE ownership: this RFC has TENT only signal; the decision/recompute stays in vLLM/SGLang. Agree?

Happy to open a PoC PR adding deadline_ns + the EDF pickForDispatch policy (no codec, no local-decode body) to validate the integration point.

Originally created by @catyans on GitHub (Jun 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2519 Originally assigned to: @catyans on GitHub. > Verified against `kvcache-ai/Mooncake` `main` @ `94ac60df` (2026-06-17). > > **Builds on #2132** (TENT Local Transfer Admission Queue). That RFC explicitly defers priority/deadline policy to a later milestone ("*if a later queued submit path adds priority or deadline policy*", "*once retry budget or deadline is exhausted*"). This RFC proposes that deadline layer concretely. It is purely additive: it reuses #2132's `QueueOwner` model and dispatch boundary, changes nothing when no deadline is supplied, and does not touch the WDRR default or the worker priority queues. Filing it separately (rather than as a comment on #2132) only because it introduces a new request field and an external degradation hook that are easier to track as their own thread; happy to merge the discussion into #2132 if maintainers prefer. ## Motivation TENT already prioritizes traffic well at the *ordering* level (documented in `docs/source/design/tent/qos.md`): - per-request priority (`PRIO_HIGH/MEDIUM/LOW`, `tent/common/types.h`) - per-priority slice queues with strict HIGH→LOW dequeue (`workers.h`, `workers.cpp`) - starvation prevention via timed promotion (`promoteTimedOutRequests`, `workers.cpp`) - cross-process coordination via time-sliced slots (`SharedSlotManager`, `shared_quota.h`) - priority-aware device selection (`quota.h`, `enable_priority_filtering`) All of this answers *"given a set of transfers, in what order do we send them?"* What's still missing is the SLO question: > **Will this P→D transfer finish before its TTFT deadline — and if not, what do we do instead of pushing it through and timing out?** Three concrete gaps: 1. **No per-transfer deadline.** `Request` has `priority` but no deadline, so the scheduler can't tell "HIGH with 8 ms left" from "HIGH with 80 ms left." 2. **Admission is capacity-only, FIFO.** `LocalTransferAdmissionQueue::pickForDispatch` (`admission_queue.cpp:185`) admits/dispatches purely on outstanding bytes/owners in FIFO order. It cannot reshape or reject a transfer that provably can't meet its deadline. 3. **No degradation path.** Under saturation the only outcomes are "send slower" or "time out." There's no wired-in "send a losslessly compressed version" or "skip transfer, recompute locally." This is the SLO-guarantee capability implied by #1883 Milestone 3 that the ordering machinery alone cannot provide. ## Scope In scope: - An optional `deadline_ns` field on `Request` (backward compatible). - A feasibility metric (MLU) computed from the **existing** EWMA bandwidth estimate + slice timestamps — no new measurement subsystem. - Deadline-aware admission/dispatch as an **opt-in policy inside `LocalTransferAdmissionQueue`**; default FIFO behavior unchanged when no deadline is supplied. - A degradation **hook interface** (compression / local-decode signal). Out of scope (kept as follow-up PRs to keep this mergeable): - The compression codec itself (dependency decision). - The LOCAL_DECODE implementation (lives in vLLM/SGLang; this RFC only defines the signal TENT raises). - Any change to worker priority queues or cross-process slots. ## Design ### 1. Deadline on the request ```cpp // tent/common/types.h, struct Request uint64_t deadline_ns = 0; // absolute steady-clock ns; 0 = no deadline (default) ``` `0` ⇒ behaves exactly as today (FIFO within priority). If the upper layer passes nothing, nothing changes. The connector (vLLM/SGLang) computes: ``` deadline_ns = now + (TTFT_SLO - t_queue_est - t_prefill_est - safety_margin) ``` TENT only consumes a deadline; it does not invent SLO numbers. ### 2. Feasibility metric (MLU) ``` MLU = remaining_bytes / (remaining_time × available_bw) ``` All three inputs already exist: - `remaining_bytes` — from `RdmaTask` accounting (`transferred_bytes` vs total). - `remaining_time` — `deadline_ns - now`. - `available_bw` — derived from the existing `DeviceQuota::ewma_bandwidth_bps` (`quota.h:76`) minus current in-flight load on the pinned device. `MLU < 1` feasible, `MLU ≥ 1` infeasible. It is a standard `work / (time × rate)` feasibility ratio (effectively a laxity test). This direction is motivated by recent work on TTFT-SLO-aware flow scheduling for LLM serving — *Multi-stage Flow Scheduling for LLM Serving* (MFS, arXiv:2603.17456), which approximates Least-Laxity-First scheduling across reusable KV-block retrieval, collective communication, and P2D transfer to improve TTFT SLO attainment by 1.2–2.4×. MFS operates above the transfer layer; this RFC brings an equivalent laxity-driven decision *into* TENT's admission path, where it can also trigger degradation (compression / local decode) rather than only reorder. ### 3. Admission decision (one-time, at submit) | MLU | Action | Loss | |---|---|---| | `< θ_compress` (e.g. 0.3) | full transfer | 0 | | `θ_compress … θ_local` (e.g. 0.3–1.5) | full transfer + lossless compression hook (if codec registered) | 0 | | `≥ θ_local` (e.g. 1.5) | raise `LOCAL_DECODE_SUGGESTED`, do not transfer | 0 (caller recomputes) | `θ_compress` / `θ_local` are config knobs. Non-deadline traffic (`deadline_ns == 0`, e.g. Store offload) skips MLU and falls through to today's path. ### 4. Dynamic re-evaluation (during transfer) Re-compute MLU every N slices or M ms using live EWMA bandwidth: - MLU past `θ_compress` → compress not-yet-posted slices. - MLU past `θ_local` → cancel remaining transfer, raise `LOCAL_DECODE_SUGGESTED`. - Promotion is monotonic (promote-only) and reuses the existing promotion machinery in `workers.cpp` — no second mechanism. ### 5. Integration point `pickForDispatch(max_owners, max_bytes)` is the single, clean injection site (currently FIFO, and this class is runtime-private, not yet on the hot path): - Add an optional `DeadlinePolicy` to `QueueLimits`. - When set: order queued owners by earliest deadline (EDF) and drop owners with MLU ≥ `θ_local` (raising the signal). - When unset (default): byte-for-byte the current FIFO. ### 6. Degradation hook (interface only) ```cpp struct DegradationHooks { // Lossless, reversible compression of a slice. Optional. std::function<Status(const Slice&, CompressedSlice&)> compress; // Raised when a transfer cannot meet its deadline. Optional. std::function<void(const Request&)> on_local_decode_suggested; }; ``` No hook registered ⇒ identical to current behavior: the `θ_compress…θ_local` band degrades to plain full transfer, and `≥ θ_local` degrades to best-effort full transfer. ## Estimated scope | Part | LoC | |---|---| | `Request.deadline_ns` + plumbing | ~30 | | MLU computation (reusing EWMA + timestamps) | ~80 | | Deadline policy in `pickForDispatch` (EDF + drop) | ~120 | | Degradation hook interface + signal | ~50 | | **Total** | **~280** (codec & local-decode bodies excluded) | ## Relation to #1883 Milestone 3's *ordering* layer is satisfied by the shipped priority queues + cross-process slots. This RFC adds the *SLO* layer on top — admission and degradation driven by a deadline, which ordering cannot express. Strictly additive, gated behind an opt-in deadline. ## Open questions 1. Deadline interface: `deadline_ns` on `Request` (proposed) vs a per-batch side-channel hint? 2. `available_bw` with multiple devices: a transfer is pinned to a device at allocate time — is per-device EWMA sufficient, or is an aggregated view needed? 3. Should the compression hook be split entirely into its own RFC/PR? (Leaning yes — the dependency decision shouldn't block the deadline mechanism.) 4. LOCAL_DECODE ownership: this RFC has TENT only *signal*; the decision/recompute stays in vLLM/SGLang. Agree? Happy to open a PoC PR adding `deadline_ns` + the EDF `pickForDispatch` policy (no codec, no local-decode body) to validate the integration point.

bsakaguchi closed this issue

2026-10-29 21:17:55 +00:00

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 17, 2026):

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2519
GitHub user ID	`18214026`
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jun 17, 2026): Thanks for opening this issue, @catyans! | Field | Value | |-------|-------| | **Issue** | #2519 | | **GitHub user ID** | `18214026` | | **Reporter** | @catyans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@catyans commented on GitHub (Jun 22, 2026):

Quick status update — #2519 has been quiet here, but two adjacent threads moved in directions that bear directly on this RFC, so I want to thread them in for context before proposing the smallest possible first step.

Scheduler boundary (#2132) — @zbchi confirmed in #2132 (comment):

"I'll try to keep the admission/dispatch side generic enough for priority, deadline, and pressure signals without creating separate scheduling paths."

That's almost exactly the surface this RFC needs — a deadline channel into the same LocalTransferAdmissionQueue rather than a separate scheduler. So #2519 should be able to land as a thin policy on top of #2132 rather than competing with it. I'll keep the integration point limited to what #2132 will already expose.

Congestion control vs. QoS (#2489) — @alogfans noted in #2489 (comment):

"They are not conflicted but should be considered together, and uses the uniform mechanism."

This RFC is intended to be a peer of #2489 under that uniform mechanism, not a parallel one: AIMD reshapes the per-flow rate at the sender, while deadline-aware admission decides at submit time whether the transfer is feasible at all (and if not, whether to compress or hand back LOCAL_DECODE_SUGGESTED). Both consume the same EWMA bandwidth estimator (DeviceQuota::ewma_bandwidth_bps, quota.h:76) and operate at distinct layers, so they should compose cleanly.

Proposed minimal first step

Given the alignment above, I'd like to start with a much smaller PR than the full RFC and hold back everything debatable for follow-ups. Concretely just two pieces, ~30 LoC total, with no behavioral change unless the caller opts in:

Add uint64_t deadline_ns = 0 to Request (tent/common/types.h:68). 0 ⇒ today's behavior bit-for-bit, no plumbing into TaskInfo or workers needed.
At submit time, when deadline_ns != 0, compute MLU as
```
deliverable_bytes = (deadline_ns - now_ns) / 1e9 × ewma_bandwidth_bps
MLU               = length / deliverable_bytes
```
(ewma_bandwidth_bps is in bytes/s, per quota.h:81's bw_gbps × 1e9 / 8.) Emit it as a histogram through the existing metrics module (tent/metrics/tent_metrics.h, alongside the read_latency_ / read_size_ precedents). Purely observability — no admission logic, no degradation hook, no pickForDispatch change.

That gives us a verifiable measurement of whether MLU is a useful signal on real workloads before any policy decisions land. The follow-up PRs — dynamic re-evaluation during transfer, EDF ordering inside pickForDispatch, the θ_compress/θ_local thresholds, the degradation hook, LOCAL_DECODE_SUGGESTED — each become independent review units that can be dropped or rethought without affecting the others.

@alogfans @stmatengss @zbchi — does this incremental shape work for you? If so I'll send the deadline-plumbing + MLU observability PR for review and follow up with each policy layer one at a time. Happy to drop or rework either part if the direction needs adjusting.

@catyans commented on GitHub (Jun 22, 2026): Quick status update — #2519 has been quiet here, but two adjacent threads moved in directions that bear directly on this RFC, so I want to thread them in for context before proposing the smallest possible first step. **Scheduler boundary (#2132)** — @zbchi confirmed in [#2132 (comment)](https://github.com/kvcache-ai/Mooncake/issues/2132#issuecomment-4742197840): > "I'll try to keep the admission/dispatch side generic enough for priority, deadline, and pressure signals without creating separate scheduling paths." That's almost exactly the surface this RFC needs — a deadline channel into the same `LocalTransferAdmissionQueue` rather than a separate scheduler. So #2519 should be able to land as a thin policy on top of #2132 rather than competing with it. I'll keep the integration point limited to what #2132 will already expose. **Congestion control vs. QoS (#2489)** — @alogfans noted in [#2489 (comment)](https://github.com/kvcache-ai/Mooncake/pull/2489#issuecomment-4741368929): > "They are not conflicted but should be considered together, and uses the uniform mechanism." This RFC is intended to be a peer of #2489 under that uniform mechanism, not a parallel one: AIMD reshapes the per-flow rate at the sender, while deadline-aware admission decides at submit time whether the transfer is feasible at all (and if not, whether to compress or hand back `LOCAL_DECODE_SUGGESTED`). Both consume the same EWMA bandwidth estimator (`DeviceQuota::ewma_bandwidth_bps`, `quota.h:76`) and operate at distinct layers, so they should compose cleanly. ### Proposed minimal first step Given the alignment above, I'd like to start with a much smaller PR than the full RFC and hold back everything debatable for follow-ups. Concretely just two pieces, ~30 LoC total, with no behavioral change unless the caller opts in: 1. Add `uint64_t deadline_ns = 0` to `Request` ([`tent/common/types.h:68`](https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/tent/include/tent/common/types.h#L68)). `0` ⇒ today's behavior bit-for-bit, no plumbing into `TaskInfo` or workers needed. 2. At submit time, when `deadline_ns != 0`, compute MLU as ``` deliverable_bytes = (deadline_ns - now_ns) / 1e9 × ewma_bandwidth_bps MLU = length / deliverable_bytes ``` (`ewma_bandwidth_bps` is in bytes/s, per `quota.h:81`'s `bw_gbps × 1e9 / 8`.) Emit it as a histogram through the existing metrics module ([`tent/metrics/tent_metrics.h`](https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/tent/include/tent/metrics/tent_metrics.h), alongside the `read_latency_` / `read_size_` precedents). Purely observability — no admission logic, no degradation hook, no `pickForDispatch` change. That gives us a verifiable measurement of whether MLU is a useful signal on real workloads before any policy decisions land. The follow-up PRs — dynamic re-evaluation during transfer, EDF ordering inside `pickForDispatch`, the `θ_compress`/`θ_local` thresholds, the degradation hook, `LOCAL_DECODE_SUGGESTED` — each become independent review units that can be dropped or rethought without affecting the others. @alogfans @stmatengss @zbchi — does this incremental shape work for you? If so I'll send the deadline-plumbing + MLU observability PR for review and follow up with each policy layer one at a time. Happy to drop or rework either part if the direction needs adjusting.

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@alogfans commented on GitHub (Jun 24, 2026):

LGTM for the minimal first step (deadline_ns field + MLU observability only).
Let's see the MLU metrics on real workloads before adding policy logic.

@alogfans commented on GitHub (Jun 24, 2026): LGTM for the minimal first step (deadline_ns field + MLU observability only). Let's see the MLU metrics on real workloads before adding policy logic.

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@catyans commented on GitHub (Jul 6, 2026):

Progress update.

Step 1 (deadline plumbing + MLU observability) is merged — #2618. It adds the optional Request.deadline_ns field (0 = no deadline = today's behavior) and emits a tent_deadline_mlu_permille histogram at task completion, with no admission/scheduling/degradation logic yet — exactly the "observability first" scope approved above.

MLU validated as a discriminating signal on H20. Built with -DUSE_TENT -DTENT_METRICS_ENABLED, ran transfer_engine_bench -backend tent P2P and scraped the histogram under two deadline regimes:

deadline window	mean MLU	histogram
tight (2 ms)	~472	all observations MLU ≫ 1 (correctly "will miss")
loose (5 s)	~0.075	~70% in the smallest bucket, all ≤ 0.25 (correctly "comfortably feasible")

Same metric cleanly separates the two regimes, so MLU behaves as a meaningful feasibility signal — which was the prerequisite for adding any policy on top. (Caveat: single-box P2P RoCE, so absolute numbers aren't representative; the point is MLU's discrimination.)

Proposed follow-ups (each an independent, opt-in PR, gated on real-workload MLU data):

EDF ordering inside pickForDispatch (the injection point wired by #2562), so deadline-tight transfers dispatch first — no change when no deadline is set.
Infeasible-transfer handling: raise LOCAL_DECODE_SUGGESTED (signal only; the recompute decision stays in vLLM/SGLang) when MLU ≥ threshold.
Degradation hook interface (lossless compression), kept as a separate dependency decision.

Keeping this RFC open as the umbrella; will open the step-2 PR once the MLU signal has been observed on a production-like workload, per the note above.

@catyans commented on GitHub (Jul 6, 2026): **Progress update.** **Step 1 (deadline plumbing + MLU observability) is merged** — #2618. It adds the optional `Request.deadline_ns` field (0 = no deadline = today's behavior) and emits a `tent_deadline_mlu_permille` histogram at task completion, with no admission/scheduling/degradation logic yet — exactly the "observability first" scope approved above. **MLU validated as a discriminating signal on H20.** Built with `-DUSE_TENT -DTENT_METRICS_ENABLED`, ran `transfer_engine_bench -backend tent` P2P and scraped the histogram under two deadline regimes: | deadline window | mean MLU | histogram | |---|---|---| | tight (2 ms) | ~472 | all observations MLU ≫ 1 (correctly "will miss") | | loose (5 s) | ~0.075 | ~70% in the smallest bucket, all ≤ 0.25 (correctly "comfortably feasible") | Same metric cleanly separates the two regimes, so MLU behaves as a meaningful feasibility signal — which was the prerequisite for adding any policy on top. (Caveat: single-box P2P RoCE, so absolute numbers aren't representative; the point is MLU's discrimination.) **Proposed follow-ups** (each an independent, opt-in PR, gated on real-workload MLU data): 1. EDF ordering inside `pickForDispatch` (the injection point wired by #2562), so deadline-tight transfers dispatch first — no change when no deadline is set. 2. Infeasible-transfer handling: raise `LOCAL_DECODE_SUGGESTED` (signal only; the recompute decision stays in vLLM/SGLang) when MLU ≥ threshold. 3. Degradation hook interface (lossless compression), kept as a separate dependency decision. Keeping this RFC open as the umbrella; will open the step-2 PR once the MLU signal has been observed on a production-like workload, per the note above.

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@catyans commented on GitHub (Jul 6, 2026):

Follow-up: full MLU-vs-deadline distribution, and one gap worth flagging.

Extending the earlier two-point sanity check into a proper sweep on the same H20 / CX-7 200G RoCEv2 box (2 nodes, TENT backend, 1 MB reads, batch=8 / threads=2 to create contention). One prerequisite worth noting: on this multi-bond dual-NUMA host the rail attempts unreachable cross-NUMA devices and every transfer fails modify QP to RTR (see #2758), which pins throughput at ~0.14 GB/s and makes MLU meaningless. Scoping the engine to the reachable NIC (via the MC_FILTER_NIC env I proposed in #2760) restores a clean ~24.7 GB/s single-stream baseline (~42 µs per 1 MB read) so the MLU numbers below are real.

MLU vs deadline (post-hoc tent_deadline_mlu_permille):

deadline	transfers	mean MLU	feasible rate (MLU < 1.0)
500 µs	166,598	0.52	most feasible
200 µs	152,009	1.31	12.8%
100 µs	152,361	2.63	~0%
60 µs	175,649	4.37	0%
30 µs	136,610	8.89	0%

Two takeaways for the policy step:

The signal is monotone and discriminating — as the deadline tightens, mean MLU rises 0.52 → 8.89 and the feasible rate falls monotonically. So the step-1 observability actually tracks deadline feasibility, as hoped.
There is a clear "transition band" (~200 µs here, 12.8% feasible) where feasible and missed transfers coexist. That band is exactly where EDF ordering / admission has leverage — reorder the tight ones ahead, or degrade the ones that are going to miss anyway. In the all-feasible (500 µs) or all-missed (≤60 µs) regimes, ordering buys nothing; the transition band is where step 2 pays off. Note the misses here are driven by concurrency queueing (single-stream is 42 µs, well under 100 µs), which is precisely what an EDF policy on the shared queue would target.

One gap I hit while measuring: MLU is only recorded on the read completion path — a write with a non-zero deadline_ns produces no tent_deadline_mlu_permille samples. All the numbers above are therefore from reads. Since P→D KV transfer is predominantly write, step 2 (and the observability itself) would want the MLU emission wired into the write completion path too. Happy to send a small PR for that if it's useful.

Caveats unchanged: single HW gen, synthetic fixed-size load, deadlines set by the bench rather than a real SLO source — so this is still a controlled measurement, not a production trace. But it does show the metric behaves as needed and points to where the policy logic earns its keep.

@catyans commented on GitHub (Jul 6, 2026): **Follow-up: full MLU-vs-deadline distribution, and one gap worth flagging.** Extending the earlier two-point sanity check into a proper sweep on the same H20 / CX-7 200G RoCEv2 box (2 nodes, TENT backend, 1 MB reads, batch=8 / threads=2 to create contention). One prerequisite worth noting: on this multi-bond dual-NUMA host the rail attempts unreachable cross-NUMA devices and every transfer fails `modify QP to RTR` (see #2758), which pins throughput at ~0.14 GB/s and makes MLU meaningless. Scoping the engine to the reachable NIC (via the `MC_FILTER_NIC` env I proposed in #2760) restores a clean ~24.7 GB/s single-stream baseline (~42 µs per 1 MB read) so the MLU numbers below are real. **MLU vs deadline (post-hoc `tent_deadline_mlu_permille`):** | deadline | transfers | mean MLU | feasible rate (MLU < 1.0) | |---|---|---|---| | 500 µs | 166,598 | 0.52 | most feasible | | 200 µs | 152,009 | 1.31 | **12.8%** | | 100 µs | 152,361 | 2.63 | ~0% | | 60 µs | 175,649 | 4.37 | 0% | | 30 µs | 136,610 | 8.89 | 0% | Two takeaways for the policy step: 1. **The signal is monotone and discriminating** — as the deadline tightens, mean MLU rises 0.52 → 8.89 and the feasible rate falls monotonically. So the step-1 observability actually tracks deadline feasibility, as hoped. 2. **There is a clear "transition band" (~200 µs here, 12.8% feasible)** where feasible and missed transfers coexist. That band is exactly where EDF ordering / admission has leverage — reorder the tight ones ahead, or degrade the ones that are going to miss anyway. In the all-feasible (500 µs) or all-missed (≤60 µs) regimes, ordering buys nothing; the transition band is where step 2 pays off. Note the misses here are driven by *concurrency queueing* (single-stream is 42 µs, well under 100 µs), which is precisely what an EDF policy on the shared queue would target. **One gap I hit while measuring:** MLU is only recorded on the **read** completion path — a `write` with a non-zero `deadline_ns` produces no `tent_deadline_mlu_permille` samples. All the numbers above are therefore from reads. Since P→D KV transfer is predominantly `write`, step 2 (and the observability itself) would want the MLU emission wired into the write completion path too. Happy to send a small PR for that if it's useful. Caveats unchanged: single HW gen, synthetic fixed-size load, deadlines set by the bench rather than a real SLO source — so this is still a controlled measurement, not a production trace. But it does show the metric behaves as needed and points to where the policy logic earns its keep.

bsakaguchi commented

2026-10-29 21:17:56 +00:00

Author

Owner

@catyans commented on GitHub (Jul 6, 2026):

Correction to my previous comment — write does record MLU.

I claimed above that MLU is only recorded on the read path. That was wrong, and I want to correct it before it misleads the step-2 design. On a clean re-test (same setup, --operation=write --deadline_us=200, MC_FILTER_NIC scoping so the rail bug doesn't interfere), writes emit MLU exactly like reads:

write, deadline=200us:  tent_deadline_mlu_permille_count = 164,628,  mean MLU = 1.31

which matches the read result at the same deadline (mean 1.31). The recording site (recordTaskCompletionMetrics) is opcode-agnostic — the recordDeadlineMLU call sits outside the READ/WRITE branch — so both paths are covered. My earlier "read-only" observation was a measurement artifact from an earlier run, not a real gap. Apologies for the noise.

So there is no write-side gap to fix before step 2 — the MLU signal is available for both directions, including the P→D KV write path. The rest of the previous comment (the deadline sweep, the ~200µs transition band where feasible/missed coexist, and EDF's leverage there) stands.

@catyans commented on GitHub (Jul 6, 2026): **Correction to my previous comment — write *does* record MLU.** I claimed above that MLU is only recorded on the read path. That was wrong, and I want to correct it before it misleads the step-2 design. On a clean re-test (same setup, `--operation=write --deadline_us=200`, `MC_FILTER_NIC` scoping so the rail bug doesn't interfere), writes emit MLU exactly like reads: ``` write, deadline=200us: tent_deadline_mlu_permille_count = 164,628, mean MLU = 1.31 ``` which matches the read result at the same deadline (mean 1.31). The recording site (`recordTaskCompletionMetrics`) is opcode-agnostic — the `recordDeadlineMLU` call sits outside the READ/WRITE branch — so both paths are covered. My earlier "read-only" observation was a measurement artifact from an earlier run, not a real gap. Apologies for the noise. So there is **no write-side gap** to fix before step 2 — the MLU signal is available for both directions, including the P→D KV write path. The rest of the previous comment (the deadline sweep, the ~200µs transition band where feasible/missed coexist, and EDF's leverage there) stands.

bsakaguchi commented

2026-10-29 21:17:57 +00:00

Author

Owner

@catyans commented on GitHub (Jul 31, 2026):

Closing as implemented. The staged plan in this RFC has landed: #2618 added deadline_ns and MLU observability, #2763 added opt-in EDF dispatch, and #2764 added deadline-infeasible drop plus the degradation hook. All changes preserve the no-deadline/default behavior. Any future policy expansion should be proposed separately with new workload evidence.

@catyans commented on GitHub (Jul 31, 2026): Closing as implemented. The staged plan in this RFC has landed: #2618 added `deadline_ns` and MLU observability, #2763 added opt-in EDF dispatch, and #2764 added deadline-infeasible drop plus the degradation hook. All changes preserve the no-deadline/default behavior. Any future policy expansion should be proposed separately with new workload evidence.

bsakaguchi referenced this issue

2026-10-29 21:29:55 +00:00

[PR #670] [MERGED] [TransferEngine] feat: Support CXL shared memory, and provide simple unit tests. #1736