[GH-ISSUE #2946] [RFC]: Store-to-TENT QoS Integration Boundary for Foreground Transfers and Background Offload #762

New issue

Closed

opened 2026-10-29 21:19:13 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented 2026-10-29 21:19:13 +00:00

Owner

Copy link

Originally created by @catyans on GitHub (Jul 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2946

Changes proposed

Summary

This RFC discusses how Mooncake Store traffic should participate in TENT QoS so that latency-sensitive transfers are protected from background offload traffic.

A representative serving workload has two priority classes:

• HIGH: foreground point-to-point KV handoff, such as Prefill-to-Decode (P2D), and foreground remote-prefix fetch.
• LOW: background writes to DRAM, local SSD, remote memory, or remote storage.

The main contention is on NIC TX and accelerator PCIe outbound bandwidth. A background Store write should not delay a foreground KV handoff that gates time-to-first-token. This RFC compares three integration boundaries:

1. Option A — route all storage data movement through TENT.
2. Option B — use layered QoS and unified admission while retaining existing executors.
3. Option C — route only the network portion of remote Store traffic through TENT.

The proposed rollout is C first, then B if PCIe-side interference remains measurable. Option A should remain a longer-term choice only if TENT is intentionally expanded into a general data-movement runtime.

This RFC is an integration proposal. It does not redefine the intent schema, QoS contract, budget scheduler, execution plan, or local admission queue proposed in #2865, #2856, #2866, #2863, and #2132. Instead, it asks how Mooncake Store should consume those mechanisms and where the ownership boundary should be.

Motivation

Consider the following traffic classes:

Traffic	Typical resources	Priority	Reason
Foreground P2D KV handoff	accelerator PCIe OUT + NIC TX	HIGH	decode cannot start until the KV state arrives
Foreground remote-prefix fetch	NIC RX + accelerator PCIe IN, depending on placement	HIGH	a hit avoids repeated prefill computation
DRAM offload	accelerator PCIe OUT + memory bandwidth	LOW	background cache management
Local SSD write	accelerator PCIe OUT + memory bandwidth + NVMe	LOW	background persistence/eviction
Remote Store write	accelerator PCIe OUT, depending on GDR/staging, + NIC TX	LOW	background persistence/write-through

Foreground prefix fetch often uses the opposite direction from background writes, so it should remain concurrent when the resources do not conflict. The scheduler must therefore reason about resources and direction; a single global HIGH/LOW gate would unnecessarily serialize independent RX/IN and TX/OUT work.

The required behavior is cooperative preemption:

foreground HIGH arrives
  -> stop admitting new conflicting LOW slices
  -> allow already-submitted DMA/WQE/I/O to complete
  -> run HIGH
  -> resume LOW using otherwise-idle capacity

Already-posted WQEs and already-submitted DMA or NVMe operations generally cannot be revoked. Therefore, the protection bound depends on both slice size and the global LOW in-flight window, not on queue ordering alone.

Current code shape

The observations below use commit 9c6125f.

Store traffic does not have one execution path

Store Put dispatches disk writes to PutToLocalFile, while memory and NoF replicas go through TransferSubmitter (client_service.cpp).

TransferSubmitter then selects local memcpy, Transfer Engine, SPDK NoF, or file-read execution depending on replica type and locality (transfer_task.cpp). These executors do not share one priority scheduler.

The file-put path performs synchronous accelerator-to-host copying into a pinned buffer before submitting StorageBackend::StoreObject to the write pool (client_service.cpp). Deferring this copy requires an explicit buffer-lifetime contract; queueing a raw pointer after the caller may reuse it would be unsafe.

Store priority is not propagated to TENT

Remote memory replica transfers construct the legacy Transport::TransferRequest (transfer_task.cpp). The legacy request has no intent or QoS priority field (transport.h), and the adapter does not populate those fields when creating tent::Request (transfer_engine.cpp). The effective result is the TENT default priority rather than Store-aware classification.

TENT already contains useful QoS primitives

TENT RDMA slicing preserves request priority and passes it into device selection and worker dispatch (rdma_transport.cpp). RDMA workers maintain three priority queues and select HIGH before MEDIUM before LOW (workers.cpp, workers.cpp).

However:

• SharedSlotManager provides cross-process slot visibility/gating, not a cross-process priority queue or global LOW in-flight budget.
• The optional local admission queue is process-local and currently owner/FIFO or deadline oriented rather than Store-resource aware (#2132).
• LOW priority promotion must be configured carefully; promoting old background work into HIGH can defeat strict foreground protection.
• Device selection fallback is availability-oriented, not a hard priority reservation.
• Rail monitoring provides health and rerouting, not business-priority arbitration.

Hardware QoS is a QP-pool decision

SL/traffic class is associated with configured QP pools and selected by policy. A practical design uses distinct HIGH and LOW named QP pools. It should not claim per-WQE hardware preemption. End-to-end network differentiation also requires matching switch DCB/PFC/ETS configuration.

Goals

1. Protect foreground handoff P95/P99 latency under background Store load.
2. Preserve concurrency for transfers that do not contend on the same directional resource.
3. Prevent LOW traffic in one process from blocking HIGH traffic in another process sharing a NIC or PCIe root complex.
4. Allow LOW traffic to consume unused bandwidth and guarantee minimum progress.
5. Expose unmanaged paths explicitly instead of claiming QoS coverage for black-box backends.
6. Keep a default-off, reversible rollout path.

Non-goals

• Hard cancellation of already-submitted RDMA, CUDA, SPDK, or NVMe work.
• Replacing the QoS Contract (#2856), Transfer Intent API (#2865), hierarchical budgets (#2866), Execution Plan (#2863), or Admission Queue (#2132).
• Using transport kernel duration as the only success metric. The primary metric should be the application-visible foreground handoff latency; transport timing is diagnostic.
• Requiring all storage backends to be rewritten in the first phase.

Common semantic contract

All three options need the same semantic plumbing:

enum class TransferIntent {
  FOREGROUND_HANDOFF,
  FOREGROUND_PREFIX_FETCH,
  DRAM_OFFLOAD,
  LOCAL_STORE_WRITE,
  REMOTE_STORE_WRITE,
  CHECKPOINT,
};

struct StoreQosContext {
  TransferIntent intent;
  PriorityClass priority;
  uint64_t bytes;
  ResourceCharge resources;
  TraceId trace_id;
};

The concrete names and ABI should reuse #2865 and #2856 rather than introduce a second public schema. The important integration requirement is end-to-end propagation:

Store operation
  -> legacy/public request compatibility layer
  -> tent::Request
  -> policy and QP-pool selection
  -> admission and worker queues
  -> metrics/completion

Unknown legacy callers should be visible as UNKNOWN in metrics. A compatibility default may be necessary during rollout, but silently treating every Store request as HIGH should not be the steady state.

Resource accounting should distinguish at least:

NIC_TX, NIC_RX,
ACCEL_PCIE_OUT, ACCEL_PCIE_IN,
HOST_MEMORY_BW,
NVME_READ, NVME_WRITE

A non-GDR remote write is a staged pipeline rather than one resource charge:

accelerator -> pinned host  [ACCEL_PCIE_OUT, HOST_MEMORY_BW]
pinned host -> remote       [HOST_MEMORY_BW, NIC_TX]

Option A: route all storage data movement through TENT

Design

Expand TENT into a general data-movement runtime. P2D, remote memory, remote SSD, local SSD, accelerator/host copies, and prefix fetches are represented as logical transfer DAGs scheduled by one resource-aware runtime.

P2D / prefix / DRAM / local SSD / remote Store
                         |
                         v
             TENT resource-aware scheduler
                    /       |       \
                RDMA      Copy      File/NoF
              executor  executor    executor

A logical request would contain ordered stages, stable buffer ownership, resource charges, priority/intent, completion, and cancellation metadata. The Execution Plan work in #2863 is the natural representation boundary.

Scheduling rules:

• maintain runnable queues per resource and priority;
• stop new conflicting LOW slice submission while HIGH is pending;
• permit non-conflicting directional work to continue;
• enforce global and per-process LOW bytes/slices/outstanding limits;
• guarantee LOW progress with a minimum budget, not by blindly promoting it to HIGH;
• reserve all resources required by the next composite stage atomically or in a fixed order.

Required implementation work

• Add Copy, file/NVMe, SPDK/NoF, and backend adapters/executors with priority-equivalent scheduling.
• Convert synchronous file-put D2H into a lifetime-safe asynchronous contract. Possible models are retaining the source allocation, copying to runtime-owned staging memory, or exposing an asynchronous completion contract.
• Extend local and cross-process admission to all declared resources.
• Ensure an executor failure cannot block unrelated foreground TENT work.

Benefits

• One scheduling domain can model NIC, PCIe, memory, and NVMe contention.
• Consistent fairness, observability, and cross-process accounting.
• Strong foundation for deadlines, tenant budgets, and composite transfer plans.

Risks

• Largest scope and failure-domain expansion.
• TENT becomes responsible for very different completion, retry, and cancellation models.
• Buffer allocator lifetime changes may be invasive.
• A central scheduler/shared lock can become a new bottleneck.
• Merely wrapping black-box backend calls in a TENT API would produce apparent, not effective, QoS.

Option B: layered QoS with unified admission and existing executors

Design

Keep TENT, memcpy workers, file workers, SPDK/NoF, and storage backends as execution owners. Extend the Admission Queue from #2132, or a shared layer above it, so every executor must acquire a resource-scoped permit before submitting its next slice.

L0  application: intent, priority, handoff deadline
                         |
L1  node admission: resource budgets and HIGH-pending state
              /          |           \
L2 executors: TENT     memcpy      file/NoF/backend
              |
L3 network: HIGH/LOW named QP pools + switch policy
              |
L4 feedback: handoff P99 and utilization adjust LOW budget

Example permit:

struct Permit {
  ResourceMask resources;
  uint64_t max_bytes;
  TimePoint expires_at;
  uint64_t policy_generation;
  bool single_use;
};

Rules:

• HIGH sets high_pending[resource]; new conflicting LOW permits stop.
• One LOW permit authorizes at most one bounded slice.
• Permits expire so unused grants do not strand capacity.
• HIGH may borrow idle LOW capacity; LOW cannot consume the HIGH reservation.
• Separate resources preserve useful RX/TX and IN/OUT concurrency.
• Composite stages declare and acquire their actual resource vectors.

Cross-process shared state should include:

high_pending[resource]
waiters[resource][priority]
inflight_bytes[resource][priority]
token_bucket[resource][priority]
owner_leases[process]
heartbeat[process]
policy_generation

The uncontended path should remain an atomic fast path. Heartbeats/generations reclaim leaked permits after process failure.

Buffer-lifetime boundary

This option still cannot defer the synchronous file-put D2H safely without source retention or owned staging memory. The path should remain explicitly unmanaged until that contract exists. Network and stable-buffer paths can be integrated first.

Benefits

• Unified scheduling authority while retaining mature executors.
• Can cover both NIC and PCIe contention incrementally.
• Reuses existing TENT priority queues, policies, admission work, and QP pools.
• Smaller failure domain than Option A.

Risks

• All executors must honor the permit contract; bypasses need auditing.
• Priority configuration can diverge across layers and cause inversion.
• Resource declarations may not match a backend's real pipeline.
• Cross-process leases and multi-resource acquisition remain non-trivial.

Option C: route only remote Store network traffic through TENT

Design

Address the clearest conflict first: foreground handoff and background remote Store writes sharing NIC TX.

foreground handoff HIGH ----\
                              > TENT priority/admission -> RDMA/NIC
remote Store write LOW ------/

DRAM / local SSD / opaque backend -> existing path, reported as unmanaged

“Remote Store” must first be classified:

1. Remote memory replica already using Transfer Engine/TENT. Propagate priority/intent through the legacy adapter and limit LOW in-flight work.
2. Remote SSD backend exposing a network-transfer stage. Add a TENT transport adapter for the data plane while leaving metadata/control RPCs in the backend.
3. Remote SSD network hidden inside StorageBackend. TENT cannot directly schedule it. The backend must expose bounded slices, completion, stable buffers, and endpoint/resource metadata; otherwise only coarse external rate limiting is possible.

Minimum changes for case 1:

• add backward-compatible intent/priority/trace propagation to the legacy request adapter;
• mark background remote writes LOW and foreground fetch/handoff HIGH;
• bound global_low_inflight_bytes, per-process LOW bytes, per-QP outstanding WRs, and LOW slice size;
• use distinct HIGH/LOW named QP pools;
• add cross-process high_pending for processes sharing the selected NIC;
• configure LOW aging so it cannot silently become foreground HIGH.

The cooperative yield bound is approximately:

yield time <= global conflicting LOW in-flight bytes / effective bandwidth
              + scheduler and completion jitter

Reducing slice size without limiting the total number of posted LOW WQEs does not provide this bound.

Coverage boundary

Option C protects the TENT-managed network portion. It does not guarantee that:

• a synchronous D2H stage before TENT cannot interfere on PCIe;
• local DRAM/SSD copies cannot interfere;
• an opaque backend's internal network traffic follows TENT priority;
• traffic classes receive end-to-end treatment without switch configuration.

These limitations must be visible in metrics and documentation.

Benefits

• Smallest and fastest change.
• Directly reuses TENT's existing priority slicing and worker queues.
• Easy to disable and benchmark independently.

Risks

• Partial QoS can be mistaken for end-to-end isolation.
• PCIe interference may remain, especially without GDR.
• Opaque remote storage backends may not be integrable without a new contract.

Comparison

Dimension	A: all through TENT	B: layered admission	C: remote Store only
TENT role	general data-movement runtime	network executor plus shared QoS control	network executor
Scope	largest	incremental	smallest
NIC TX protection	complete	complete	complete for managed remote Store
PCIe OUT protection	potentially complete	complete after copy paths join	only TENT-visible portion
Local DRAM/SSD	TENT executors	existing executors with permits	unmanaged
Opaque backends	must become executors	must expose admission/slicing	may remain unsupported
Buffer-lifetime change	broad and required	incremental	avoidable for stable/GDR paths
Failure domain	largest	medium	smallest
Recommended role	possible long-term redesign	target architecture	first validation phase

Proposed rollout

Phase 0: semantics and observability

• Reuse the intent and contract schemas from #2865 and #2856.
• Propagate intent, priority, trace ID, and resource charge through Store adapters.
• Add UNKNOWN and unmanaged_transfer_bytes metrics.
• Trace the actual data path for each deployed remote backend, including GDR and staging behavior.

Phase 1: Option C

• Integrate remote-memory Store writes with TENT LOW priority.
• Keep foreground handoff/fetch HIGH.
• Add process-local LOW slice/outstanding limits.
• Add node-wide high_pending and global LOW in-flight accounting.
• Validate separate named QP pools and switch configuration where available.

Phase 2: evaluate the remaining interference

• Run foreground handoff with remote Store, D2H, and local SSD interference independently.
• If PCIe-side background traffic still degrades foreground P99, proceed to Option B.

Phase 3: Option B

• Extend admission to accelerator copy and file/NoF executors.
• Add a buffer lease or owned-staging contract for deferred file-put D2H.
• Make black-box backends expose bounded asynchronous stages or leave them explicitly unmanaged.

Phase 4: reconsider Option A

Only proceed if maintainers want TENT to own general data movement and the executor/lifetime refactor is justified by broader use cases.

Metrics and validation

Required metrics

• application-visible foreground handoff P50/P95/P99;
• end-to-end prefix-fetch latency and TTFT where available;
• queue/permit wait by intent, priority, and resource;
• HIGH-pending duration and LOW throttle time;
• in-flight bytes and outstanding slices by resource/priority/process;
• TENT submit-to-completion duration and worker queue wait;
• NIC TX/RX, accelerator PCIe IN/OUT, and NVMe queue/bandwidth counters;
• QP-pool/traffic-class byte counters;
• priority promotion count by intent;
• unmanaged bytes by path and resource.

Validation matrix

1. Foreground handoff only.
2. Store only.
3. Foreground handoff + remote write-through.
4. Foreground handoff + write-back burst.
5. Foreground handoff + D2H/DRAM offload.
6. Foreground handoff + local SSD write.
7. Foreground prefix fetch + TX-heavy background traffic.
8. Cross-process: process A runs LOW Store, process B submits HIGH handoff.
9. Single-rail/multi-rail and rail failure/recovery.
10. GDR and staged/non-GDR paths where supported.

Proposed initial acceptance targets, subject to benchmark agreement:

• foreground handoff P99 regression under saturated LOW traffic: at most 5% versus foreground-only baseline;
• no new conflicting LOW submission within 100 microseconds after HIGH becomes pending;
• LOW-only throughput at least 95% of QoS-disabled throughput;
• no cross-process LOW-over-HIGH inversion;
• no unnecessary serialization of non-conflicting RX/IN and TX/OUT work;
• bounded LOW progress without promotion into foreground HIGH;
• default-off configuration and behavior-preserving rollback.

Relationship to existing RFCs

• #1883: implements the roadmap goals for QoS-aware slice spraying and transfer queues in a Store integration scenario.
• #2132: provides the local admission boundary; this RFC proposes the Store/resource inputs and cross-executor integration.
• #2568: related traffic-class hints; this RFC assumes QP-pool-based enforcement rather than per-WQE preemption.
• #2856: defines the effective QoS contract consumed here.
• #2865: defines the transfer intent API that Store should propagate.
• #2866: owns hierarchical budgets and work-conserving scheduling semantics.
• #2863: can represent staged Store paths and their resource charge vectors.

Open questions

1. Should TENT remain a network transport runtime, or eventually own all storage data movement?
2. Do maintainers agree with Option C as the first integration phase and Option B as the target architecture?
3. Should the legacy Store transfer request gain explicit intent/priority fields, or should Store move directly to the newer request/options API?
4. Should cross-process Store QoS extend the existing shared-slot state, or use the distributed admission mechanism proposed elsewhere?
5. Which LOW intents may age within LOW, and which, if any, may be promoted to another priority class?
6. Is HIGH/LOW named-QP-pool selection the preferred hardware boundary?
7. For deferred file-put D2H, should Mooncake retain the source allocation, use runtime-owned staging memory, or expose asynchronous completion to callers?
8. Should the primary acceptance metric be the application-visible handoff P99, with CQ/API/kernel timings treated as diagnostic breakdowns?

Before submitting a new issue...

• Searched existing issues for TENT QoS, transfer intent, admission, resource budgets, traffic classes, and execution plans.
• Read the TENT roadmap and the related RFCs listed above.

Originally created by @catyans on GitHub (Jul 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2946 ### Changes proposed ## Summary This RFC discusses how Mooncake Store traffic should participate in TENT QoS so that latency-sensitive transfers are protected from background offload traffic. A representative serving workload has two priority classes: - **HIGH**: foreground point-to-point KV handoff, such as Prefill-to-Decode (P2D), and foreground remote-prefix fetch. - **LOW**: background writes to DRAM, local SSD, remote memory, or remote storage. The main contention is on NIC TX and accelerator PCIe outbound bandwidth. A background Store write should not delay a foreground KV handoff that gates time-to-first-token. This RFC compares three integration boundaries: 1. **Option A — route all storage data movement through TENT.** 2. **Option B — use layered QoS and unified admission while retaining existing executors.** 3. **Option C — route only the network portion of remote Store traffic through TENT.** The proposed rollout is **C first, then B if PCIe-side interference remains measurable**. Option A should remain a longer-term choice only if TENT is intentionally expanded into a general data-movement runtime. This RFC is an integration proposal. It does not redefine the intent schema, QoS contract, budget scheduler, execution plan, or local admission queue proposed in #2865, #2856, #2866, #2863, and #2132. Instead, it asks how Mooncake Store should consume those mechanisms and where the ownership boundary should be. ## Motivation Consider the following traffic classes: | Traffic | Typical resources | Priority | Reason | | --- | --- | --- | --- | | Foreground P2D KV handoff | accelerator PCIe OUT + NIC TX | HIGH | decode cannot start until the KV state arrives | | Foreground remote-prefix fetch | NIC RX + accelerator PCIe IN, depending on placement | HIGH | a hit avoids repeated prefill computation | | DRAM offload | accelerator PCIe OUT + memory bandwidth | LOW | background cache management | | Local SSD write | accelerator PCIe OUT + memory bandwidth + NVMe | LOW | background persistence/eviction | | Remote Store write | accelerator PCIe OUT, depending on GDR/staging, + NIC TX | LOW | background persistence/write-through | Foreground prefix fetch often uses the opposite direction from background writes, so it should remain concurrent when the resources do not conflict. The scheduler must therefore reason about resources and direction; a single global HIGH/LOW gate would unnecessarily serialize independent RX/IN and TX/OUT work. The required behavior is cooperative preemption: ```text foreground HIGH arrives -> stop admitting new conflicting LOW slices -> allow already-submitted DMA/WQE/I/O to complete -> run HIGH -> resume LOW using otherwise-idle capacity ``` Already-posted WQEs and already-submitted DMA or NVMe operations generally cannot be revoked. Therefore, the protection bound depends on both slice size and the **global LOW in-flight window**, not on queue ordering alone. ## Current code shape The observations below use commit [`9c6125f`](https://github.com/kvcache-ai/Mooncake/commit/9c6125f3e9ecf667907f2fd3cea17a783a8a4791). ### Store traffic does not have one execution path Store `Put` dispatches disk writes to `PutToLocalFile`, while memory and NoF replicas go through `TransferSubmitter` ([`client_service.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-store/src/client_service.cpp#L2031-L2070)). `TransferSubmitter` then selects local memcpy, Transfer Engine, SPDK NoF, or file-read execution depending on replica type and locality ([`transfer_task.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-store/src/transfer_task.cpp#L981-L1028)). These executors do not share one priority scheduler. The file-put path performs synchronous accelerator-to-host copying into a pinned buffer before submitting `StorageBackend::StoreObject` to the write pool ([`client_service.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-store/src/client_service.cpp#L3361-L3415)). Deferring this copy requires an explicit buffer-lifetime contract; queueing a raw pointer after the caller may reuse it would be unsafe. ### Store priority is not propagated to TENT Remote memory replica transfers construct the legacy `Transport::TransferRequest` ([`transfer_task.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-store/src/transfer_task.cpp#L1198-L1235)). The legacy request has no intent or QoS priority field ([`transport.h`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-transfer-engine/include/transport/transport.h#L60-L71)), and the adapter does not populate those fields when creating `tent::Request` ([`transfer_engine.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/mooncake-transfer-engine/src/transfer_engine.cpp#L611-L626)). The effective result is the TENT default priority rather than Store-aware classification. ### TENT already contains useful QoS primitives TENT RDMA slicing preserves request priority and passes it into device selection and worker dispatch ([`rdma_transport.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/tent/src/transport/rdma/rdma_transport.cpp#L415-L529)). RDMA workers maintain three priority queues and select HIGH before MEDIUM before LOW ([`workers.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/tent/src/transport/rdma/workers.cpp#L240-L263), [`workers.cpp`](https://github.com/kvcache-ai/Mooncake/blob/9c6125f3e9ecf667907f2fd3cea17a783a8a4791/tent/src/transport/rdma/workers.cpp#L386-L400)). However: - `SharedSlotManager` provides cross-process slot visibility/gating, not a cross-process priority queue or global LOW in-flight budget. - The optional local admission queue is process-local and currently owner/FIFO or deadline oriented rather than Store-resource aware (#2132). - LOW priority promotion must be configured carefully; promoting old background work into HIGH can defeat strict foreground protection. - Device selection fallback is availability-oriented, not a hard priority reservation. - Rail monitoring provides health and rerouting, not business-priority arbitration. ### Hardware QoS is a QP-pool decision SL/traffic class is associated with configured QP pools and selected by policy. A practical design uses distinct HIGH and LOW named QP pools. It should not claim per-WQE hardware preemption. End-to-end network differentiation also requires matching switch DCB/PFC/ETS configuration. ## Goals 1. Protect foreground handoff P95/P99 latency under background Store load. 2. Preserve concurrency for transfers that do not contend on the same directional resource. 3. Prevent LOW traffic in one process from blocking HIGH traffic in another process sharing a NIC or PCIe root complex. 4. Allow LOW traffic to consume unused bandwidth and guarantee minimum progress. 5. Expose unmanaged paths explicitly instead of claiming QoS coverage for black-box backends. 6. Keep a default-off, reversible rollout path. ## Non-goals - Hard cancellation of already-submitted RDMA, CUDA, SPDK, or NVMe work. - Replacing the QoS Contract (#2856), Transfer Intent API (#2865), hierarchical budgets (#2866), Execution Plan (#2863), or Admission Queue (#2132). - Using transport kernel duration as the only success metric. The primary metric should be the application-visible foreground handoff latency; transport timing is diagnostic. - Requiring all storage backends to be rewritten in the first phase. ## Common semantic contract All three options need the same semantic plumbing: ```cpp enum class TransferIntent { FOREGROUND_HANDOFF, FOREGROUND_PREFIX_FETCH, DRAM_OFFLOAD, LOCAL_STORE_WRITE, REMOTE_STORE_WRITE, CHECKPOINT, }; struct StoreQosContext { TransferIntent intent; PriorityClass priority; uint64_t bytes; ResourceCharge resources; TraceId trace_id; }; ``` The concrete names and ABI should reuse #2865 and #2856 rather than introduce a second public schema. The important integration requirement is end-to-end propagation: ```text Store operation -> legacy/public request compatibility layer -> tent::Request -> policy and QP-pool selection -> admission and worker queues -> metrics/completion ``` Unknown legacy callers should be visible as `UNKNOWN` in metrics. A compatibility default may be necessary during rollout, but silently treating every Store request as HIGH should not be the steady state. Resource accounting should distinguish at least: ```text NIC_TX, NIC_RX, ACCEL_PCIE_OUT, ACCEL_PCIE_IN, HOST_MEMORY_BW, NVME_READ, NVME_WRITE ``` A non-GDR remote write is a staged pipeline rather than one resource charge: ```text accelerator -> pinned host [ACCEL_PCIE_OUT, HOST_MEMORY_BW] pinned host -> remote [HOST_MEMORY_BW, NIC_TX] ``` ## Option A: route all storage data movement through TENT ### Design Expand TENT into a general data-movement runtime. P2D, remote memory, remote SSD, local SSD, accelerator/host copies, and prefix fetches are represented as logical transfer DAGs scheduled by one resource-aware runtime. ```text P2D / prefix / DRAM / local SSD / remote Store | v TENT resource-aware scheduler / | \ RDMA Copy File/NoF executor executor executor ``` A logical request would contain ordered stages, stable buffer ownership, resource charges, priority/intent, completion, and cancellation metadata. The Execution Plan work in #2863 is the natural representation boundary. Scheduling rules: - maintain runnable queues per resource and priority; - stop new conflicting LOW slice submission while HIGH is pending; - permit non-conflicting directional work to continue; - enforce global and per-process LOW bytes/slices/outstanding limits; - guarantee LOW progress with a minimum budget, not by blindly promoting it to HIGH; - reserve all resources required by the next composite stage atomically or in a fixed order. ### Required implementation work - Add Copy, file/NVMe, SPDK/NoF, and backend adapters/executors with priority-equivalent scheduling. - Convert synchronous file-put D2H into a lifetime-safe asynchronous contract. Possible models are retaining the source allocation, copying to runtime-owned staging memory, or exposing an asynchronous completion contract. - Extend local and cross-process admission to all declared resources. - Ensure an executor failure cannot block unrelated foreground TENT work. ### Benefits - One scheduling domain can model NIC, PCIe, memory, and NVMe contention. - Consistent fairness, observability, and cross-process accounting. - Strong foundation for deadlines, tenant budgets, and composite transfer plans. ### Risks - Largest scope and failure-domain expansion. - TENT becomes responsible for very different completion, retry, and cancellation models. - Buffer allocator lifetime changes may be invasive. - A central scheduler/shared lock can become a new bottleneck. - Merely wrapping black-box backend calls in a TENT API would produce apparent, not effective, QoS. ## Option B: layered QoS with unified admission and existing executors ### Design Keep TENT, memcpy workers, file workers, SPDK/NoF, and storage backends as execution owners. Extend the Admission Queue from #2132, or a shared layer above it, so every executor must acquire a resource-scoped permit before submitting its next slice. ```text L0 application: intent, priority, handoff deadline | L1 node admission: resource budgets and HIGH-pending state / | \ L2 executors: TENT memcpy file/NoF/backend | L3 network: HIGH/LOW named QP pools + switch policy | L4 feedback: handoff P99 and utilization adjust LOW budget ``` Example permit: ```cpp struct Permit { ResourceMask resources; uint64_t max_bytes; TimePoint expires_at; uint64_t policy_generation; bool single_use; }; ``` Rules: - HIGH sets `high_pending[resource]`; new conflicting LOW permits stop. - One LOW permit authorizes at most one bounded slice. - Permits expire so unused grants do not strand capacity. - HIGH may borrow idle LOW capacity; LOW cannot consume the HIGH reservation. - Separate resources preserve useful RX/TX and IN/OUT concurrency. - Composite stages declare and acquire their actual resource vectors. Cross-process shared state should include: ```text high_pending[resource] waiters[resource][priority] inflight_bytes[resource][priority] token_bucket[resource][priority] owner_leases[process] heartbeat[process] policy_generation ``` The uncontended path should remain an atomic fast path. Heartbeats/generations reclaim leaked permits after process failure. ### Buffer-lifetime boundary This option still cannot defer the synchronous file-put D2H safely without source retention or owned staging memory. The path should remain explicitly `unmanaged` until that contract exists. Network and stable-buffer paths can be integrated first. ### Benefits - Unified scheduling authority while retaining mature executors. - Can cover both NIC and PCIe contention incrementally. - Reuses existing TENT priority queues, policies, admission work, and QP pools. - Smaller failure domain than Option A. ### Risks - All executors must honor the permit contract; bypasses need auditing. - Priority configuration can diverge across layers and cause inversion. - Resource declarations may not match a backend's real pipeline. - Cross-process leases and multi-resource acquisition remain non-trivial. ## Option C: route only remote Store network traffic through TENT ### Design Address the clearest conflict first: foreground handoff and background remote Store writes sharing NIC TX. ```text foreground handoff HIGH ----\ > TENT priority/admission -> RDMA/NIC remote Store write LOW ------/ DRAM / local SSD / opaque backend -> existing path, reported as unmanaged ``` “Remote Store” must first be classified: 1. **Remote memory replica already using Transfer Engine/TENT.** Propagate priority/intent through the legacy adapter and limit LOW in-flight work. 2. **Remote SSD backend exposing a network-transfer stage.** Add a TENT transport adapter for the data plane while leaving metadata/control RPCs in the backend. 3. **Remote SSD network hidden inside `StorageBackend`.** TENT cannot directly schedule it. The backend must expose bounded slices, completion, stable buffers, and endpoint/resource metadata; otherwise only coarse external rate limiting is possible. Minimum changes for case 1: - add backward-compatible intent/priority/trace propagation to the legacy request adapter; - mark background remote writes LOW and foreground fetch/handoff HIGH; - bound `global_low_inflight_bytes`, per-process LOW bytes, per-QP outstanding WRs, and LOW slice size; - use distinct HIGH/LOW named QP pools; - add cross-process `high_pending` for processes sharing the selected NIC; - configure LOW aging so it cannot silently become foreground HIGH. The cooperative yield bound is approximately: ```text yield time <= global conflicting LOW in-flight bytes / effective bandwidth + scheduler and completion jitter ``` Reducing slice size without limiting the total number of posted LOW WQEs does not provide this bound. ### Coverage boundary Option C protects the TENT-managed network portion. It does **not** guarantee that: - a synchronous D2H stage before TENT cannot interfere on PCIe; - local DRAM/SSD copies cannot interfere; - an opaque backend's internal network traffic follows TENT priority; - traffic classes receive end-to-end treatment without switch configuration. These limitations must be visible in metrics and documentation. ### Benefits - Smallest and fastest change. - Directly reuses TENT's existing priority slicing and worker queues. - Easy to disable and benchmark independently. ### Risks - Partial QoS can be mistaken for end-to-end isolation. - PCIe interference may remain, especially without GDR. - Opaque remote storage backends may not be integrable without a new contract. ## Comparison | Dimension | A: all through TENT | B: layered admission | C: remote Store only | | --- | --- | --- | --- | | TENT role | general data-movement runtime | network executor plus shared QoS control | network executor | | Scope | largest | incremental | smallest | | NIC TX protection | complete | complete | complete for managed remote Store | | PCIe OUT protection | potentially complete | complete after copy paths join | only TENT-visible portion | | Local DRAM/SSD | TENT executors | existing executors with permits | unmanaged | | Opaque backends | must become executors | must expose admission/slicing | may remain unsupported | | Buffer-lifetime change | broad and required | incremental | avoidable for stable/GDR paths | | Failure domain | largest | medium | smallest | | Recommended role | possible long-term redesign | target architecture | first validation phase | ## Proposed rollout ### Phase 0: semantics and observability - Reuse the intent and contract schemas from #2865 and #2856. - Propagate intent, priority, trace ID, and resource charge through Store adapters. - Add `UNKNOWN` and `unmanaged_transfer_bytes` metrics. - Trace the actual data path for each deployed remote backend, including GDR and staging behavior. ### Phase 1: Option C - Integrate remote-memory Store writes with TENT LOW priority. - Keep foreground handoff/fetch HIGH. - Add process-local LOW slice/outstanding limits. - Add node-wide `high_pending` and global LOW in-flight accounting. - Validate separate named QP pools and switch configuration where available. ### Phase 2: evaluate the remaining interference - Run foreground handoff with remote Store, D2H, and local SSD interference independently. - If PCIe-side background traffic still degrades foreground P99, proceed to Option B. ### Phase 3: Option B - Extend admission to accelerator copy and file/NoF executors. - Add a buffer lease or owned-staging contract for deferred file-put D2H. - Make black-box backends expose bounded asynchronous stages or leave them explicitly unmanaged. ### Phase 4: reconsider Option A Only proceed if maintainers want TENT to own general data movement and the executor/lifetime refactor is justified by broader use cases. ## Metrics and validation ### Required metrics - application-visible foreground handoff P50/P95/P99; - end-to-end prefix-fetch latency and TTFT where available; - queue/permit wait by intent, priority, and resource; - HIGH-pending duration and LOW throttle time; - in-flight bytes and outstanding slices by resource/priority/process; - TENT submit-to-completion duration and worker queue wait; - NIC TX/RX, accelerator PCIe IN/OUT, and NVMe queue/bandwidth counters; - QP-pool/traffic-class byte counters; - priority promotion count by intent; - unmanaged bytes by path and resource. ### Validation matrix 1. Foreground handoff only. 2. Store only. 3. Foreground handoff + remote write-through. 4. Foreground handoff + write-back burst. 5. Foreground handoff + D2H/DRAM offload. 6. Foreground handoff + local SSD write. 7. Foreground prefix fetch + TX-heavy background traffic. 8. Cross-process: process A runs LOW Store, process B submits HIGH handoff. 9. Single-rail/multi-rail and rail failure/recovery. 10. GDR and staged/non-GDR paths where supported. Proposed initial acceptance targets, subject to benchmark agreement: - foreground handoff P99 regression under saturated LOW traffic: at most 5% versus foreground-only baseline; - no new conflicting LOW submission within 100 microseconds after HIGH becomes pending; - LOW-only throughput at least 95% of QoS-disabled throughput; - no cross-process LOW-over-HIGH inversion; - no unnecessary serialization of non-conflicting RX/IN and TX/OUT work; - bounded LOW progress without promotion into foreground HIGH; - default-off configuration and behavior-preserving rollback. ## Relationship to existing RFCs - #1883: implements the roadmap goals for QoS-aware slice spraying and transfer queues in a Store integration scenario. - #2132: provides the local admission boundary; this RFC proposes the Store/resource inputs and cross-executor integration. - #2568: related traffic-class hints; this RFC assumes QP-pool-based enforcement rather than per-WQE preemption. - #2856: defines the effective QoS contract consumed here. - #2865: defines the transfer intent API that Store should propagate. - #2866: owns hierarchical budgets and work-conserving scheduling semantics. - #2863: can represent staged Store paths and their resource charge vectors. ## Open questions 1. Should TENT remain a network transport runtime, or eventually own all storage data movement? 2. Do maintainers agree with Option C as the first integration phase and Option B as the target architecture? 3. Should the legacy Store transfer request gain explicit intent/priority fields, or should Store move directly to the newer request/options API? 4. Should cross-process Store QoS extend the existing shared-slot state, or use the distributed admission mechanism proposed elsewhere? 5. Which LOW intents may age within LOW, and which, if any, may be promoted to another priority class? 6. Is HIGH/LOW named-QP-pool selection the preferred hardware boundary? 7. For deferred file-put D2H, should Mooncake retain the source allocation, use runtime-owned staging memory, or expose asynchronous completion to callers? 8. Should the primary acceptance metric be the application-visible handoff P99, with CQ/API/kernel timings treated as diagnostic breakdowns? ### Before submitting a new issue... - [x] Searched existing issues for TENT QoS, transfer intent, admission, resource budgets, traffic classes, and execution plans. - [x] Read the TENT roadmap and the related RFCs listed above.

bsakaguchi 2026-10-29 21:19:13 +00:00

• closed this issue
• added the
  RFC
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 16, 2026):

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2946
GitHub user ID	18214026
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4987943144 --> @github-actions[bot] commented on GitHub (Jul 16, 2026): Thanks for opening this issue, @catyans! | Field | Value | |-------|-------| | **Issue** | #2946 | | **GitHub user ID** | `18214026` | | **Reporter** | @catyans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 15, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6183565020 --> @github-actions[bot] commented on GitHub (Oct 15, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 22, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:6331824250 --> @github-actions[bot] commented on GitHub (Oct 22, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue 2026-10-29 21:30:29 +00:00

[PR #762] [MERGED] [Doc] Fix 3FS plugin file link problem #1803

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#762

No description provided.