kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2863] [RFC]: TENT Backend Capability Model and First-Class Execution Plan #744

New issue

Closed

opened 2026-10-29 21:18:59 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:18:59 +00:00

Owner

Originally created by @catyans on GitHub (Jul 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2863

Summary

This RFC proposes a versioned backend capability model and a first-class ExecutionPlan intermediate representation for TENT.

Today TENT can select a transport, choose devices, stage unsupported memory pairs through host memory, and retry with another transport. However, these decisions are represented across TransportSelector, TransferEngineImpl, ProxyManager, transport-specific capability booleans, and failover state. There is no single object that answers:

which physical path will execute this request;
whether it is direct or staged;
which backend, rail, QP pool, and resources it requires;
which policy and health observations were used;
what fallback/degraded alternatives are legal;
why another candidate was rejected.

The proposed model makes the result of path selection explicit and explainable before execution:

Transfer Intent + segment metadata + topology + policy + health
                              |
                              v
                  candidate plan generation
                              |
                              v
                 validation + deterministic ranking
                              |
                              v
                       ExecutionPlan
                              |
                  admission / resource reservation
                              |
                              v
                 existing transport / ProxyManager

The first implementation milestone is intentionally behavior-preserving: add the data model, validation, candidate generation, legacy-path adaptation, and JSON explain output. It must not change the default transport selection or scheduling behavior.

Motivation

This is intended to implement the declarative and dynamic-path parts of the TENT roadmap in #1058 without replacing the transport executors that already work.

The current building blocks are useful but incomplete as a planning model:

Transport::Capabilities currently expresses six memory-pair booleans (dram_to_dram, gpu_to_gpu, and related pairs).
#2079 added rule-based transport/device selection. The first matching policy authorizes a preference list, and the selector returns one transport plus a device mask and QoS attributes.
#1878 added bounded cross-transport failover by advancing to another transport after failure.
Direct and staged requests are prepared through different runtime paths; staged execution remains owned by ProxyManager.
#2849 needs a plan-dependent resource charge vector for receiver credits, but deliberately does not define an Execution Plan.
#2856 defines which QoS/degraded actions are authorized, not how a concrete path is constructed.

This creates several problems.

1. Selection is implicit

A request can be direct, staged, retried, or degraded, but there is no immutable record of the complete decision. Debugging requires reconstructing decisions from multiple logs and code paths.

2. Capabilities do not describe execution semantics

Two backends that both report gpu_to_gpu=true can still differ in:

local versus remote support;
READ/WRITE support;
alignment and maximum transfer/SGE limits;
registration and exported-handle requirements;
completion and remote-visibility semantics;
cancellation, notification, and peer-failure behavior;
hardware QoS support;
whether a fallback is safe after partial progress.

These differences currently live in backend code and special cases rather than in a common contract.

3. Admission cannot charge one consistent resource vector

Direct RDMA, staged GPU→Host→RDMA→Host→GPU, and TCP fallback consume different combinations of NIC bandwidth, QP/CQ capacity, staging buffers, receiver bytes/slots, CPU work, and device-copy bandwidth. Admission and receiver-credit logic need the chosen plan before they can account for those resources consistently.

4. Health and fallback are difficult to explain

A rail or backend may become unavailable after the initial choice. The runtime needs to know which plan dependency changed, which candidates remain valid, and whether the applicable QoS contract permits fallback or degradation.

5. New backends increase scheduler special cases

The long-term goal is for a new backend to register capabilities and execution primitives. Adding a backend should not require repeatedly editing the central planner for vendor-specific behavior.

Goals

Define a versioned BackendCapability schema for static execution semantics.
Represent Direct and Staged physical paths as first-class immutable plans.
Represent Primary, Fallback, and Degraded as plan roles rather than conflating them with physical path shape.
Generate a bounded candidate set from request intent, memory descriptors, topology, policy, backend capabilities, and operational state.
Attach an explicit resource charge vector to each plan.
Validate plans before admission or transport submission.
Produce stable human-readable and sanitized JSON explain output.
Preserve current behavior by default and support incremental shadow-mode rollout.

Non-goals

A cluster-wide global scheduler or per-transfer coordinator.
Replacing Transport, RDMA workers, ProxyManager, or existing staging state machines.
Implementing congestion control or per-RTT path switching.
Defining a second QoS Contract, receiver-credit protocol, or Admission Queue.
Adding a new transport backend in this RFC.
Guaranteeing that a cost estimate predicts latency or bandwidth exactly.
Enabling automatic plan enforcement in the first implementation PR.
Exposing the initial internal C++ structures as a stable public ABI immediately.

Terminology

Physical path shape and decision role are separate dimensions:

PathKind: DIRECT | STAGED
PlanRole: PRIMARY | FALLBACK | DEGRADED

Examples:

primary direct RDMA;
fallback direct TCP;
primary staged GPU→Host→RDMA→Host→GPU;
degraded staged path using an allowed lower-cost representation.

This avoids treating “Fallback” as if it were a transport topology. A fallback can itself be Direct or Staged.

Proposed capability model

The exact C++ layout is not fixed by this RFC, but the semantic model should cover at least:

struct BackendCapability {
    uint32_t schema_version;
    TransportType backend;

    bool supports_local;
    bool supports_remote;
    bool supports_read;
    bool supports_write;
    bool supports_notification;
    bool supports_cancel;

    std::vector<MemoryPair> memory_pairs;
    uint64_t required_alignment;
    uint64_t max_transfer_bytes;
    uint32_t max_sge;

    RegistrationSemantics registration;
    CompletionSemantics completion;
    FailureSemantics failure;
    QosCapabilities qos;
};

The capability describes relatively static backend semantics. Dynamic measurements and health should be separate versioned inputs rather than mutating the capability object:

struct BackendOperationalState {
    uint64_t generation;
    uint64_t observed_at_ns;
    uint64_t valid_for_ns;
    HealthState health;
    std::optional<double> bandwidth_bps;
    std::optional<double> latency_ns;
    double confidence;
};

Keeping static capability and dynamic state separate prevents a stale telemetry sample from changing what a backend fundamentally supports.

Unknown capability fields should be ignored when safe. An unknown schema major version must fail validation or fall back to the legacy selector; it must not silently enable an unsupported path.

Proposed Execution Plan model

An Execution Plan describes one logical transfer owner before it is split into transport-specific slices.

struct PlanStage {
    StageType type;  // TRANSFER, LOCAL_COPY, REMOTE_DELEGATE, COMMIT
    TransportType backend;
    MemoryType source_memory;
    MemoryType target_memory;
    uint64_t bytes;
    DeviceSelection devices;
    QosSelection qos;
};

struct ResourceCharge {
    uint64_t nic_bytes;
    uint64_t receiver_bytes;
    uint32_t request_slots;
    uint32_t staging_slots;
    uint32_t consumer_slots;
    uint32_t qp_slots;
    uint32_t cq_slots;
    uint64_t host_staging_bytes;
};

struct ExecutionPlan {
    uint32_t schema_version;
    PathKind path_kind;
    PlanRole role;
    std::vector<PlanStage> stages;
    ResourceCharge charge;

    std::string matched_policy;
    std::vector<DependencyGeneration> dependencies;
    CostEstimate estimate;
    std::vector<DecisionReason> reasons;
};

The actual implementation may use compact internal IDs rather than strings in hot paths. Explain strings should be produced outside the transport/slice completion hot path.

Immutability and lifetime

Once validated and admitted, the plan is immutable for one attempt. A retry or changed dependency creates a new plan/attempt generation. Late completion from an old attempt must not mutate the state of the new plan.

Plan dependencies may include:

local and remote segment descriptor generations;
backend capability generation;
policy/config generation;
topology generation;
health/operational-state generation.

If a required dependency changes before dispatch, the plan is revalidated or regenerated. Already-dispatched work follows the existing transport cancellation/failure contract; this RFC does not claim that all posted work can be revoked.

Candidate generation

Candidate generation should be bounded and deterministic:

Resolve the effective request intent and policy.
Resolve source/target descriptors and memory locations.
Enumerate backends present on both required sides.
Filter by static capability and policy authorization.
Generate Direct candidates.
If no suitable Direct candidate exists, generate bounded Staged candidates using registered staging primitives.
Filter candidates using current health and operational-state freshness.
Attach QoS selection and the full resource charge vector.
Validate fallback/degraded actions against the effective QoS Contract.
Rank using a stable score and deterministic tie-breaker.

The first version should not implement an unbounded graph search. It should use an explicit maximum stage count and prevent cycles. Initially supported compositions can be registered templates such as:

device -> host -> network -> host -> device
device -> host -> file
file -> host -> device

The planner should return rejection reasons for candidates it filtered out, not only the final winner.

Cost and freshness semantics

Cost is a ranking input, not a guarantee:

struct CostEstimate {
    std::optional<uint64_t> latency_ns;
    std::optional<uint64_t> bandwidth_bps;
    double confidence;
    uint64_t observed_at_ns;
    uint64_t valid_for_ns;
    CostSource source;
};

Rules:

expired estimates must not influence ranking;
missing telemetry uses a conservative configured baseline or the current legacy order;
the explain result records why an estimate was used or ignored;
no plan is rejected only because an optional estimate is missing unless strict policy explicitly requires it.

Resource accounting

The planner computes resource demand; resource owners decide whether to grant it.

Local Admission remains responsible for local queue and dispatch limits (#2132).
Receiver-advertised credits remain responsible for receiver-owned reservations (#2849).
QoS Contract remains responsible for authorization, caps, and allowed degraded actions (#2856).
The Execution Plan only provides a consistent charge vector to those components.

Reservation must be atomic for the full plan charge. A staged plan must not reserve network bytes while failing to reserve the staging/consumer slot required to finish it.

Internal staging work should carry a parent plan/attempt ID so its resources and causal metrics can be attributed to the user-visible request without making it an independent policy decision.

Fallback and replanning

The planner may return an ordered candidate set:

primary candidate
fallback candidate(s)
allowed degraded candidate(s)

The runtime can prevalidate cheap alternatives, but a fallback should be revalidated against current descriptor, policy, health, and credit generations before dispatch.

Fallback after partial progress requires backend-specific failure semantics. A backend that may have partially modified the destination cannot be retried blindly unless the operation is idempotent or the destination is reset/versioned. The capability model must make this explicit; the planner must not infer safety from the transport name.

Explain output

The planner should support human-readable output and sanitized, versioned JSON. Example:

{
  "schema_version": 1,
  "path_kind": "direct",
  "role": "primary",
  "matched_policy": "foreground_get",
  "stages": [
    {
      "type": "transfer",
      "backend": "rdma",
      "source_memory": "cuda",
      "target_memory": "cuda",
      "bytes": 1048576,
      "device": "mlx5_bond_0",
      "qp_pool": "foreground"
    }
  ],
  "resource_charge": {
    "nic_bytes": 1048576,
    "receiver_bytes": 1048576,
    "request_slots": 1
  },
  "reasons": [
    "policy authorizes rdma and tcp",
    "rdma supports remote cuda-to-cuda write",
    "selected rail is healthy",
    "lowest valid deterministic cost"
  ],
  "rejected_candidates": [
    {
      "backend": "nvlink",
      "reason": "remote target is not on the same machine"
    },
    {
      "backend": "tcp",
      "reason": "valid but ranked after rdma"
    }
  ]
}

Raw addresses, rkeys, fabric handles, credentials, and unbounded tenant/request identifiers must not appear in a diagnostic bundle by default.

Compatibility and rollout

Suggested rollout modes:

disabled  -> current behavior only (default)
shadow    -> generate/explain a plan and compare it with the legacy decision
enforced  -> execute the selected plan (future opt-in)

The first PR does not need to expose all three modes; it may contain only the model and pure tests. Before enforced mode is added, shadow mode should demonstrate:

the same selected transport and device mask for legacy-compatible cases;
the same Direct/Staged boundary;
stable fallback ordering;
explicit, reviewed differences for cases where current behavior is ambiguous or incorrect.

No per-slice plan allocation or JSON formatting is allowed in the RDMA completion hot path. Planning occurs at logical request/queue-owner preparation time, and compact IDs are propagated to slices.

Proposed PR sequence

PR1: Capability and plan model

Add internal versioned types.
Add validation and serialization tests.
Add legacy capability adapters for existing transports.
No runtime selection or execution changes.

PR2: Legacy decision adapter and explain

Convert the current selector/staging result into an ExecutionPlan.
Add stable human/JSON explain output.
Differential tests against current behavior.

PR3: Candidate generator in shadow mode

Generate bounded Direct/Staged candidates.
Rank deterministically.
Compare the generated winner with the legacy adapter.
No default behavior change.

PR4: Opt-in Direct plan enforcement

Execute validated Direct plans through existing transports.
Integrate plan IDs with tracing and causal metrics.
Keep Staged execution on the legacy adapter initially.

Later PRs

Staged plan enforcement through existing ProxyManager.
Receiver-credit and Admission charge consumption.
Health generation invalidation and safe replanning.
Fallback/degraded plan enforcement.
Public C/Python explain API after the internal schema stabilizes.

Validation plan

Model and property tests

invalid/unknown schema versions;
unsupported memory pairs and operations;
alignment/max-size violations;
cycle and maximum-stage rejection;
deterministic ranking and stable tie-breaking;
stale health/cost input handling;
fallback/degraded authorization;
atomic multi-resource charge validation;
serialization round trip and sanitized JSON.

Differential tests

For the existing policy matrix, shadow planning must match current behavior for:

CPU↔CPU and GPU↔GPU;
local and remote targets;
RDMA/TCP/SHM/NVLink/GDS/io_uring where built;
explicit policy_name and transport_hint;
Direct and Staged paths;
transport failover ordering.

Real-cluster validation

On the existing two-node H20/RoCE environment:

Direct GPU↔GPU RDMA READ and WRITE;
host↔host TCP fallback;
forced Staged GPU↔GPU path with the Direct backend disabled;
one unhealthy/unavailable rail;
receiver-credit enabled/disabled once integration is implemented;
foreground/background policies with distinct QP pools where available.

Report plan generation time, submit throughput, transfer throughput/P99, selected/rejected candidates, data integrity, and fallback behavior. Shadow mode should have a defined overhead budget and must not add work to transport completion polling.

Fault injection

Reuse FaultProxyTransport from #1907 to verify:

primary backend fails before dispatch;
primary fails after submission;
stale plan dependency forces revalidation;
late completion from an old attempt cannot overwrite the new attempt;
fallback is rejected when policy does not allow it.

Expected benefits and claims boundary

If implemented, this RFC should provide:

one inspectable representation of how a transfer will execute;
consistent resource accounting before dispatch;
clearer backend integration contracts;
deterministic fallback/degraded alternatives;
explainability for operators and upper-layer connectors;
a stable integration point for future health, QoS, and telemetry work.

This RFC does not claim higher throughput, lower P99, or better SLO attainment by itself. The initial value is architectural correctness, explainability, and the ability to test later planners against an explicit contract. Any performance claim must come from the real-cluster matrix above.

Relationship to existing work

#1058: implements declarative scheduling, composite staged paths, topology-aware cost, resilience, and diagnostics direction.
#2079: reuse existing policy parsing and selection constraints; do not create a second policy language.
#1878 and #1984: reuse existing failover/rail recovery execution mechanisms; the plan records and validates alternatives.
#2132: Admission consumes the selected plan charge; it remains the queue owner.
#2519: deadline/degradation policy is a planning input, not a second planner.
#2821 and #1850: tracing records plan/stage/slice causality; this RFC does not replace tracing.
#2849: receiver credits grant resources described by the plan charge vector.
#2856: QoS Contract authorizes priority, caps, and degraded actions used during planning.
#2832: capability identity can become a plan dependency generation; this RFC does not redefine allocation identity.

Open questions

Is an internal, versioned BackendCapability + ExecutionPlan model the preferred boundary, or should the first RFC be limited to an explain-only adapter over current decisions?
Do maintainers agree with separating physical PathKind (Direct/Staged) from decision PlanRole (Primary/Fallback/Degraded)?
Should v1 support only the currently implemented host-staging templates, or define a generic bounded DAG immediately?
Should fallback candidates be precomputed, regenerated on failure, or use a hybrid approach as proposed above?
Which capability fields must be stable in v1, and which should remain backend-specific opaque attributes?
Should plan explain initially be exposed only through an internal/CLI API until the C and Python ABI is versioned?

Before submitting a new issue...

Searched existing issues and PRs for Execution Plan, capability model, dynamic path, staging, fallback, and planner work.
Read the TENT roadmap and related policy, failover, Admission, receiver-credit, and QoS Contract discussions.

Originally created by @catyans on GitHub (Jul 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2863 ## Summary This RFC proposes a versioned backend capability model and a first-class `ExecutionPlan` intermediate representation for TENT. Today TENT can select a transport, choose devices, stage unsupported memory pairs through host memory, and retry with another transport. However, these decisions are represented across `TransportSelector`, `TransferEngineImpl`, `ProxyManager`, transport-specific capability booleans, and failover state. There is no single object that answers: - which physical path will execute this request; - whether it is direct or staged; - which backend, rail, QP pool, and resources it requires; - which policy and health observations were used; - what fallback/degraded alternatives are legal; - why another candidate was rejected. The proposed model makes the result of path selection explicit and explainable before execution: ```text Transfer Intent + segment metadata + topology + policy + health | v candidate plan generation | v validation + deterministic ranking | v ExecutionPlan | admission / resource reservation | v existing transport / ProxyManager ``` The first implementation milestone is intentionally behavior-preserving: add the data model, validation, candidate generation, legacy-path adaptation, and JSON explain output. It must not change the default transport selection or scheduling behavior. ## Motivation This is intended to implement the declarative and dynamic-path parts of the TENT roadmap in #1058 without replacing the transport executors that already work. The current building blocks are useful but incomplete as a planning model: - `Transport::Capabilities` currently expresses six memory-pair booleans (`dram_to_dram`, `gpu_to_gpu`, and related pairs). - #2079 added rule-based transport/device selection. The first matching policy authorizes a preference list, and the selector returns one transport plus a device mask and QoS attributes. - #1878 added bounded cross-transport failover by advancing to another transport after failure. - Direct and staged requests are prepared through different runtime paths; staged execution remains owned by `ProxyManager`. - #2849 needs a plan-dependent resource charge vector for receiver credits, but deliberately does not define an Execution Plan. - #2856 defines which QoS/degraded actions are authorized, not how a concrete path is constructed. This creates several problems. ### 1. Selection is implicit A request can be direct, staged, retried, or degraded, but there is no immutable record of the complete decision. Debugging requires reconstructing decisions from multiple logs and code paths. ### 2. Capabilities do not describe execution semantics Two backends that both report `gpu_to_gpu=true` can still differ in: - local versus remote support; - READ/WRITE support; - alignment and maximum transfer/SGE limits; - registration and exported-handle requirements; - completion and remote-visibility semantics; - cancellation, notification, and peer-failure behavior; - hardware QoS support; - whether a fallback is safe after partial progress. These differences currently live in backend code and special cases rather than in a common contract. ### 3. Admission cannot charge one consistent resource vector Direct RDMA, staged GPU→Host→RDMA→Host→GPU, and TCP fallback consume different combinations of NIC bandwidth, QP/CQ capacity, staging buffers, receiver bytes/slots, CPU work, and device-copy bandwidth. Admission and receiver-credit logic need the chosen plan before they can account for those resources consistently. ### 4. Health and fallback are difficult to explain A rail or backend may become unavailable after the initial choice. The runtime needs to know which plan dependency changed, which candidates remain valid, and whether the applicable QoS contract permits fallback or degradation. ### 5. New backends increase scheduler special cases The long-term goal is for a new backend to register capabilities and execution primitives. Adding a backend should not require repeatedly editing the central planner for vendor-specific behavior. ## Goals 1. Define a versioned `BackendCapability` schema for static execution semantics. 2. Represent Direct and Staged physical paths as first-class immutable plans. 3. Represent Primary, Fallback, and Degraded as plan roles rather than conflating them with physical path shape. 4. Generate a bounded candidate set from request intent, memory descriptors, topology, policy, backend capabilities, and operational state. 5. Attach an explicit resource charge vector to each plan. 6. Validate plans before admission or transport submission. 7. Produce stable human-readable and sanitized JSON explain output. 8. Preserve current behavior by default and support incremental shadow-mode rollout. ## Non-goals - A cluster-wide global scheduler or per-transfer coordinator. - Replacing `Transport`, RDMA workers, `ProxyManager`, or existing staging state machines. - Implementing congestion control or per-RTT path switching. - Defining a second QoS Contract, receiver-credit protocol, or Admission Queue. - Adding a new transport backend in this RFC. - Guaranteeing that a cost estimate predicts latency or bandwidth exactly. - Enabling automatic plan enforcement in the first implementation PR. - Exposing the initial internal C++ structures as a stable public ABI immediately. ## Terminology Physical path shape and decision role are separate dimensions: ```text PathKind: DIRECT | STAGED PlanRole: PRIMARY | FALLBACK | DEGRADED ``` Examples: - primary direct RDMA; - fallback direct TCP; - primary staged GPU→Host→RDMA→Host→GPU; - degraded staged path using an allowed lower-cost representation. This avoids treating “Fallback” as if it were a transport topology. A fallback can itself be Direct or Staged. ## Proposed capability model The exact C++ layout is not fixed by this RFC, but the semantic model should cover at least: ```cpp struct BackendCapability { uint32_t schema_version; TransportType backend; bool supports_local; bool supports_remote; bool supports_read; bool supports_write; bool supports_notification; bool supports_cancel; std::vector<MemoryPair> memory_pairs; uint64_t required_alignment; uint64_t max_transfer_bytes; uint32_t max_sge; RegistrationSemantics registration; CompletionSemantics completion; FailureSemantics failure; QosCapabilities qos; }; ``` The capability describes relatively static backend semantics. Dynamic measurements and health should be separate versioned inputs rather than mutating the capability object: ```cpp struct BackendOperationalState { uint64_t generation; uint64_t observed_at_ns; uint64_t valid_for_ns; HealthState health; std::optional<double> bandwidth_bps; std::optional<double> latency_ns; double confidence; }; ``` Keeping static capability and dynamic state separate prevents a stale telemetry sample from changing what a backend fundamentally supports. Unknown capability fields should be ignored when safe. An unknown schema major version must fail validation or fall back to the legacy selector; it must not silently enable an unsupported path. ## Proposed Execution Plan model An Execution Plan describes one logical transfer owner before it is split into transport-specific slices. ```cpp struct PlanStage { StageType type; // TRANSFER, LOCAL_COPY, REMOTE_DELEGATE, COMMIT TransportType backend; MemoryType source_memory; MemoryType target_memory; uint64_t bytes; DeviceSelection devices; QosSelection qos; }; struct ResourceCharge { uint64_t nic_bytes; uint64_t receiver_bytes; uint32_t request_slots; uint32_t staging_slots; uint32_t consumer_slots; uint32_t qp_slots; uint32_t cq_slots; uint64_t host_staging_bytes; }; struct ExecutionPlan { uint32_t schema_version; PathKind path_kind; PlanRole role; std::vector<PlanStage> stages; ResourceCharge charge; std::string matched_policy; std::vector<DependencyGeneration> dependencies; CostEstimate estimate; std::vector<DecisionReason> reasons; }; ``` The actual implementation may use compact internal IDs rather than strings in hot paths. Explain strings should be produced outside the transport/slice completion hot path. ### Immutability and lifetime Once validated and admitted, the plan is immutable for one attempt. A retry or changed dependency creates a new plan/attempt generation. Late completion from an old attempt must not mutate the state of the new plan. Plan dependencies may include: - local and remote segment descriptor generations; - backend capability generation; - policy/config generation; - topology generation; - health/operational-state generation. If a required dependency changes before dispatch, the plan is revalidated or regenerated. Already-dispatched work follows the existing transport cancellation/failure contract; this RFC does not claim that all posted work can be revoked. ## Candidate generation Candidate generation should be bounded and deterministic: 1. Resolve the effective request intent and policy. 2. Resolve source/target descriptors and memory locations. 3. Enumerate backends present on both required sides. 4. Filter by static capability and policy authorization. 5. Generate Direct candidates. 6. If no suitable Direct candidate exists, generate bounded Staged candidates using registered staging primitives. 7. Filter candidates using current health and operational-state freshness. 8. Attach QoS selection and the full resource charge vector. 9. Validate fallback/degraded actions against the effective QoS Contract. 10. Rank using a stable score and deterministic tie-breaker. The first version should not implement an unbounded graph search. It should use an explicit maximum stage count and prevent cycles. Initially supported compositions can be registered templates such as: ```text device -> host -> network -> host -> device device -> host -> file file -> host -> device ``` The planner should return rejection reasons for candidates it filtered out, not only the final winner. ## Cost and freshness semantics Cost is a ranking input, not a guarantee: ```cpp struct CostEstimate { std::optional<uint64_t> latency_ns; std::optional<uint64_t> bandwidth_bps; double confidence; uint64_t observed_at_ns; uint64_t valid_for_ns; CostSource source; }; ``` Rules: - expired estimates must not influence ranking; - missing telemetry uses a conservative configured baseline or the current legacy order; - the explain result records why an estimate was used or ignored; - no plan is rejected only because an optional estimate is missing unless strict policy explicitly requires it. ## Resource accounting The planner computes resource demand; resource owners decide whether to grant it. - Local Admission remains responsible for local queue and dispatch limits (#2132). - Receiver-advertised credits remain responsible for receiver-owned reservations (#2849). - QoS Contract remains responsible for authorization, caps, and allowed degraded actions (#2856). - The Execution Plan only provides a consistent charge vector to those components. Reservation must be atomic for the full plan charge. A staged plan must not reserve network bytes while failing to reserve the staging/consumer slot required to finish it. Internal staging work should carry a parent plan/attempt ID so its resources and causal metrics can be attributed to the user-visible request without making it an independent policy decision. ## Fallback and replanning The planner may return an ordered candidate set: ```text primary candidate fallback candidate(s) allowed degraded candidate(s) ``` The runtime can prevalidate cheap alternatives, but a fallback should be revalidated against current descriptor, policy, health, and credit generations before dispatch. Fallback after partial progress requires backend-specific failure semantics. A backend that may have partially modified the destination cannot be retried blindly unless the operation is idempotent or the destination is reset/versioned. The capability model must make this explicit; the planner must not infer safety from the transport name. ## Explain output The planner should support human-readable output and sanitized, versioned JSON. Example: ```json { "schema_version": 1, "path_kind": "direct", "role": "primary", "matched_policy": "foreground_get", "stages": [ { "type": "transfer", "backend": "rdma", "source_memory": "cuda", "target_memory": "cuda", "bytes": 1048576, "device": "mlx5_bond_0", "qp_pool": "foreground" } ], "resource_charge": { "nic_bytes": 1048576, "receiver_bytes": 1048576, "request_slots": 1 }, "reasons": [ "policy authorizes rdma and tcp", "rdma supports remote cuda-to-cuda write", "selected rail is healthy", "lowest valid deterministic cost" ], "rejected_candidates": [ { "backend": "nvlink", "reason": "remote target is not on the same machine" }, { "backend": "tcp", "reason": "valid but ranked after rdma" } ] } ``` Raw addresses, rkeys, fabric handles, credentials, and unbounded tenant/request identifiers must not appear in a diagnostic bundle by default. ## Compatibility and rollout Suggested rollout modes: ```text disabled -> current behavior only (default) shadow -> generate/explain a plan and compare it with the legacy decision enforced -> execute the selected plan (future opt-in) ``` The first PR does not need to expose all three modes; it may contain only the model and pure tests. Before enforced mode is added, shadow mode should demonstrate: - the same selected transport and device mask for legacy-compatible cases; - the same Direct/Staged boundary; - stable fallback ordering; - explicit, reviewed differences for cases where current behavior is ambiguous or incorrect. No per-slice plan allocation or JSON formatting is allowed in the RDMA completion hot path. Planning occurs at logical request/queue-owner preparation time, and compact IDs are propagated to slices. ## Proposed PR sequence ### PR1: Capability and plan model - Add internal versioned types. - Add validation and serialization tests. - Add legacy capability adapters for existing transports. - No runtime selection or execution changes. ### PR2: Legacy decision adapter and explain - Convert the current selector/staging result into an `ExecutionPlan`. - Add stable human/JSON explain output. - Differential tests against current behavior. ### PR3: Candidate generator in shadow mode - Generate bounded Direct/Staged candidates. - Rank deterministically. - Compare the generated winner with the legacy adapter. - No default behavior change. ### PR4: Opt-in Direct plan enforcement - Execute validated Direct plans through existing transports. - Integrate plan IDs with tracing and causal metrics. - Keep Staged execution on the legacy adapter initially. ### Later PRs - Staged plan enforcement through existing `ProxyManager`. - Receiver-credit and Admission charge consumption. - Health generation invalidation and safe replanning. - Fallback/degraded plan enforcement. - Public C/Python explain API after the internal schema stabilizes. ## Validation plan ### Model and property tests - invalid/unknown schema versions; - unsupported memory pairs and operations; - alignment/max-size violations; - cycle and maximum-stage rejection; - deterministic ranking and stable tie-breaking; - stale health/cost input handling; - fallback/degraded authorization; - atomic multi-resource charge validation; - serialization round trip and sanitized JSON. ### Differential tests For the existing policy matrix, shadow planning must match current behavior for: - CPU↔CPU and GPU↔GPU; - local and remote targets; - RDMA/TCP/SHM/NVLink/GDS/io_uring where built; - explicit `policy_name` and `transport_hint`; - Direct and Staged paths; - transport failover ordering. ### Real-cluster validation On the existing two-node H20/RoCE environment: - Direct GPU↔GPU RDMA READ and WRITE; - host↔host TCP fallback; - forced Staged GPU↔GPU path with the Direct backend disabled; - one unhealthy/unavailable rail; - receiver-credit enabled/disabled once integration is implemented; - foreground/background policies with distinct QP pools where available. Report plan generation time, submit throughput, transfer throughput/P99, selected/rejected candidates, data integrity, and fallback behavior. Shadow mode should have a defined overhead budget and must not add work to transport completion polling. ### Fault injection Reuse `FaultProxyTransport` from #1907 to verify: - primary backend fails before dispatch; - primary fails after submission; - stale plan dependency forces revalidation; - late completion from an old attempt cannot overwrite the new attempt; - fallback is rejected when policy does not allow it. ## Expected benefits and claims boundary If implemented, this RFC should provide: - one inspectable representation of how a transfer will execute; - consistent resource accounting before dispatch; - clearer backend integration contracts; - deterministic fallback/degraded alternatives; - explainability for operators and upper-layer connectors; - a stable integration point for future health, QoS, and telemetry work. This RFC does not claim higher throughput, lower P99, or better SLO attainment by itself. The initial value is architectural correctness, explainability, and the ability to test later planners against an explicit contract. Any performance claim must come from the real-cluster matrix above. ## Relationship to existing work - #1058: implements declarative scheduling, composite staged paths, topology-aware cost, resilience, and diagnostics direction. - #2079: reuse existing policy parsing and selection constraints; do not create a second policy language. - #1878 and #1984: reuse existing failover/rail recovery execution mechanisms; the plan records and validates alternatives. - #2132: Admission consumes the selected plan charge; it remains the queue owner. - #2519: deadline/degradation policy is a planning input, not a second planner. - #2821 and #1850: tracing records plan/stage/slice causality; this RFC does not replace tracing. - #2849: receiver credits grant resources described by the plan charge vector. - #2856: QoS Contract authorizes priority, caps, and degraded actions used during planning. - #2832: capability identity can become a plan dependency generation; this RFC does not redefine allocation identity. ## Open questions 1. Is an internal, versioned `BackendCapability` + `ExecutionPlan` model the preferred boundary, or should the first RFC be limited to an explain-only adapter over current decisions? 2. Do maintainers agree with separating physical `PathKind` (Direct/Staged) from decision `PlanRole` (Primary/Fallback/Degraded)? 3. Should v1 support only the currently implemented host-staging templates, or define a generic bounded DAG immediately? 4. Should fallback candidates be precomputed, regenerated on failure, or use a hybrid approach as proposed above? 5. Which capability fields must be stable in v1, and which should remain backend-specific opaque attributes? 6. Should plan explain initially be exposed only through an internal/CLI API until the C and Python ABI is versioned? ### Before submitting a new issue... - [x] Searched existing issues and PRs for Execution Plan, capability model, dynamic path, staging, fallback, and planner work. - [x] Read the TENT roadmap and related policy, failover, Admission, receiver-credit, and QoS Contract discussions.

bsakaguchi

2026-10-29 21:18:59 +00:00

closed this issue
added the
RFC

stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:19:00 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 12, 2026):

Thanks for opening this issue, @catyans!

Field	Value
Issue	#2863
GitHub user ID	`18214026`
Reporter	@catyans

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 12, 2026): Thanks for opening this issue, @catyans! | Field | Value | |-------|-------| | **Issue** | #2863 | | **GitHub user ID** | `18214026` | | **Reporter** | @catyans | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:19:00 +00:00

Author

Owner

@catyans commented on GitHub (Jul 12, 2026):

Shadow-prototype feasibility result (2026-07-13). This is intentionally not a production implementation benchmark.

Environment: Lingjun H20 lingjun-099, Linux 5.10.134, Intel Xeon Platinum 8575C, Python 3.12.7. The prototype modeled capability filtering and ranking for RDMA, SHM, TCP, staged CUDA->host+TCP, unhealthy-RDMA exclusion and local-recompute fallback.

Results:

five deterministic Direct/Staged/Fallback/degraded-health scenarios: 5/5 selected the expected plan;
1,000,000 capability-filter/rank decisions: 690.97 ms total, 691 ns/decision in Python.

Interpretation: candidate enumeration over a small registered backend set is computationally cheap enough to remain control-path work, and the proposed capability boundary can express direct, staged and fallback choices without backend-specific branches in the caller. The absolute 691 ns number must not be used as a C++ latency claim: this prototype omits topology graph traversal, live telemetry snapshots, allocation, explain serialization and concurrency. A production PR still needs deterministic cost ordering, generation-consistent telemetry and a bounded candidate count.

@catyans commented on GitHub (Jul 12, 2026): Shadow-prototype feasibility result (2026-07-13). This is intentionally not a production implementation benchmark. Environment: Lingjun H20 `lingjun-099`, Linux 5.10.134, Intel Xeon Platinum 8575C, Python 3.12.7. The prototype modeled capability filtering and ranking for RDMA, SHM, TCP, staged CUDA->host+TCP, unhealthy-RDMA exclusion and local-recompute fallback. Results: - five deterministic Direct/Staged/Fallback/degraded-health scenarios: **5/5 selected the expected plan**; - 1,000,000 capability-filter/rank decisions: 690.97 ms total, **691 ns/decision** in Python. Interpretation: candidate enumeration over a small registered backend set is computationally cheap enough to remain control-path work, and the proposed capability boundary can express direct, staged and fallback choices without backend-specific branches in the caller. The absolute 691 ns number must not be used as a C++ latency claim: this prototype omits topology graph traversal, live telemetry snapshots, allocation, explain serialization and concurrency. A production PR still needs deterministic cost ordering, generation-consistent telemetry and a bounded candidate count.

bsakaguchi commented

2026-10-29 21:19:00 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 11, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Oct 11, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:19:00 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 19, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Oct 19, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

bsakaguchi referenced this issue

2026-10-29 21:30:21 +00:00

[PR #744] [MERGED] [Store] Add replication guarantees #1789