kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4598] [RFC]: TENT Multi-Rail RDMA Selection Observability #1172

New issue

Open

opened 2026-10-29 21:23:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:34 +00:00

Owner

Originally created by @RuixiangMa on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4598

Status

This RFC documents the accompanying implementation in https://github.com/kvcache-ai/Mooncake/pull/4420

Summary

TENT's RDMA DeviceSelector chooses a local NIC for each transfer slice, but the previous implementation did not expose which NICs were selected during a benchmark. Consequently, a throughput or latency result could not distinguish an even multi-rail distribution from a run that concentrated traffic on one rail.

This RFC adds read-only selection observability. It records allocation intent per local NIC and emits a warmup-excluded JSONL measurement window fromtebench. It does not change device eligibility, scoring, rail admission, fallback, scheduling policy, or transfer execution.

Motivation

Multi-rail routing work needs a reproducible baseline before changing the allocation policy. The baseline must answer:

Which local NICs were selected?
How many slices and requested bytes were assigned to each NIC?
Was the request single-path, multi-path, or a scheduled exploration probe?
Does the reported window exclude warmup traffic?

These signals are allocation decisions, not proof of completed wire traffic.
They complement, rather than replace, source and target NIC port counters.

Goals

Expose a lightweight snapshot of RDMA allocation decisions that is safe
concurrently with allocation.
Produce machine-readable benchmark records for a bounded measurement window.
Preserve all existing allocation behavior and defaults.
Make counter semantics explicit enough to support later policy validation.

Non-Goals

Change the multi-rail selection or slice-distribution policy.
Claim wire-byte, posted-byte, completion, throughput, or latency attribution.
Add per-request tracing, RPC endpoints, or persistent metrics storage.
Infer temporal oscillation from cumulative selection counters.

Design

flowchart TB
    subgraph BENCH[tebench measurement lifecycle]
        direction LR
        W[Warmup] --> B[MeasurementBarrier]
        B --> S0[Start snapshot]
        S0 --> M[Measured transfers]
        M --> S1[End snapshot]
        S1 --> D[Snapshot delta]
    end

    subgraph TENT[TENT RDMA selector]
        direction LR
        M -. allocation decisions .-> DS[DeviceSelector]
        DS --> C[(Per-NIC atomic counters)]
        C -. read-only snapshot .-> S0
        C -. read-only snapshot .-> S1
    end

    D --> J[[rdma_selection_metrics JSONL]]

    style BENCH fill:#F8FAFC,stroke:#CBD5E1,color:#0F172A;
    style TENT fill:#F8FAFC,stroke:#CBD5E1,color:#0F172A;
    classDef warmup fill:#FFFFFF,stroke:#94A3B8,color:#334155;
    classDef measure fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E;
    classDef stats fill:#ECFDF5,stroke:#059669,color:#064E3B;
    classDef output fill:#F1F5F9,stroke:#475569,color:#0F172A;
    class W warmup;
    class B,S0,M,S1,D measure;
    class DS,C stats;
    class J output;

Selection Counters

DeviceSelector records a successful, non-empty allocation after it has
chosen slice device IDs. The snapshot includes:

total allocations;
single-path and multi-path allocations, classified by the distinct local NIC IDs actually selected rather than by requested slice count;
probe allocations (smart multi-slice only; every 100th successful, non-empty selection per selector thread);
per-device selected slice count and selected bytes;
local device availability, name, and NUMA node.

The final slice is charged with the request remainder, so per-device selected bytes sum to the requested allocation size, including a folded tail.

No counter is recorded for an empty or failed selection. Historical counters remain visible when a device later becomes unavailable.

Semantics and Concurrency

Counters use relaxed atomics. A snapshot is best-effort: its fields can span concurrent allocations and are not a transactionally consistent point in time. The counters are observational only and never participate in scoring, EWMA learning, in-flight accounting, or eligibility decisions.

Snapshots are safe concurrently with allocation only. They must not race selector topology initialization or replacement, transport teardown, or engine destruction.

Selected bytes describe the allocator's pre-fallback intent. A worker may later
retry or use a different transport/NIC, so selected bytes are not posted,
completed, or physical wire bytes.

TransferEngine::getSelectionStats() replaces its output with the current RDMA
snapshot. An engine without an RDMA transport returns OK with zero counters
and an empty device list.

Benchmark Measurement Window

tebench runs the existing one-second warmup before entering a one-shot MeasurementBarrier. The final warmup worker obtains the start snapshot before releasing any worker into the measured workload. The end snapshot is obtained only after the initiator tasks finish.

If warmup, the start snapshot, or a measured transfer fails, the barrier enters the failed state and the case emits no selection record. This prevents partial or warmup-contaminated JSONL output.

When --result_output_jsonl is set for the TENT backend, each successful case appends one additional rdma_selection_metrics record alongside its existing target-metric records:

{
  "schema_version": 1,
  "record_type": "rdma_selection_metrics",
  "measurement_window": "snapshot_delta",
  "backend": "tent",
  "op_type": "write",
  "block_size": 2097152,
  "batch_size": 32,
  "num_threads": 4,
  "selection": {
    "allocations": 100,
    "single_path_allocations": 0,
    "multi_path_allocations": 100,
    "probe_allocations": 1,
    "devices": []
  }
}

--tent_rail_topo_path is a benchmark-only convenience option that passes a rail-pair topology file to the TENT RDMA worker configuration. It allows a benchmark to make its eligible-rail policy reproducible without changing the production configuration contract.

Compatibility

Existing TENT users do not need configuration changes.
Counter collection does not alter allocation results.
JSONL output is opt-in through the existing --result_output_jsonl flag.
The API is C++ only; no C ABI or Python API is added.

Validation

The implementation includes tests for:

single-path and multi-path accounting, including folded tails;
unavailable-device history;
empty and failed allocations;
scheduled probe accounting;
concurrent snapshot readability;
warmup exclusion, snapshot failure, and failure races in the measurement
barrier;
JSONL delta serialization.

Acceptance for a real multi-rail run requires the JSONL selection delta and source/target NIC port-counter deltas to agree on the intended eligible rails. The JSONL record alone must not be used to claim balanced physical traffic.

Follow-up Work

This RFC intentionally separates observability from routing-policy changes. Any policy proposal must use equal workloads and report throughput, p50/p99 latency, per-rail selected shares, and port-counter deltas for equal-rail, unequal-rail, burst, and rail-exclusion cases. Temporal oscillation requires fixed-window or transition-level instrumentation; it cannot be derived from the cumulative counters introduced here.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @RuixiangMa on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4598 ## Status This RFC documents the accompanying implementation in https://github.com/kvcache-ai/Mooncake/pull/4420 ## Summary TENT's RDMA `DeviceSelector` chooses a local NIC for each transfer slice, but the previous implementation did not expose which NICs were selected during a benchmark. Consequently, a throughput or latency result could not distinguish an even multi-rail distribution from a run that concentrated traffic on one rail. This RFC adds read-only selection observability. It records allocation intent per local NIC and emits a warmup-excluded JSONL measurement window from`tebench`. It does **not** change device eligibility, scoring, rail admission, fallback, scheduling policy, or transfer execution. ## Motivation Multi-rail routing work needs a reproducible baseline before changing the allocation policy. The baseline must answer: - Which local NICs were selected? - How many slices and requested bytes were assigned to each NIC? - Was the request single-path, multi-path, or a scheduled exploration probe? - Does the reported window exclude warmup traffic? These signals are allocation decisions, not proof of completed wire traffic. They complement, rather than replace, source and target NIC port counters. ## Goals - Expose a lightweight snapshot of RDMA allocation decisions that is safe concurrently with allocation. - Produce machine-readable benchmark records for a bounded measurement window. - Preserve all existing allocation behavior and defaults. - Make counter semantics explicit enough to support later policy validation. ## Non-Goals - Change the multi-rail selection or slice-distribution policy. - Claim wire-byte, posted-byte, completion, throughput, or latency attribution. - Add per-request tracing, RPC endpoints, or persistent metrics storage. - Infer temporal oscillation from cumulative selection counters. ## Design ```mermaid flowchart TB subgraph BENCH[tebench measurement lifecycle] direction LR W[Warmup] --> B[MeasurementBarrier] B --> S0[Start snapshot] S0 --> M[Measured transfers] M --> S1[End snapshot] S1 --> D[Snapshot delta] end subgraph TENT[TENT RDMA selector] direction LR M -. allocation decisions .-> DS[DeviceSelector] DS --> C[(Per-NIC atomic counters)] C -. read-only snapshot .-> S0 C -. read-only snapshot .-> S1 end D --> J[[rdma_selection_metrics JSONL]] style BENCH fill:#F8FAFC,stroke:#CBD5E1,color:#0F172A; style TENT fill:#F8FAFC,stroke:#CBD5E1,color:#0F172A; classDef warmup fill:#FFFFFF,stroke:#94A3B8,color:#334155; classDef measure fill:#E0F2FE,stroke:#0284C7,color:#0C4A6E; classDef stats fill:#ECFDF5,stroke:#059669,color:#064E3B; classDef output fill:#F1F5F9,stroke:#475569,color:#0F172A; class W warmup; class B,S0,M,S1,D measure; class DS,C stats; class J output; ``` ### Selection Counters `DeviceSelector` records a successful, non-empty allocation after it has chosen slice device IDs. The snapshot includes: - total allocations; - single-path and multi-path allocations, classified by the distinct local NIC IDs actually selected rather than by requested slice count; - probe allocations (smart multi-slice only; every 100th successful, non-empty selection per selector thread); - per-device selected slice count and selected bytes; - local device availability, name, and NUMA node. The final slice is charged with the request remainder, so per-device selected bytes sum to the requested allocation size, including a folded tail. No counter is recorded for an empty or failed selection. Historical counters remain visible when a device later becomes unavailable. ### Semantics and Concurrency Counters use relaxed atomics. A snapshot is best-effort: its fields can span concurrent allocations and are not a transactionally consistent point in time. The counters are observational only and never participate in scoring, EWMA learning, in-flight accounting, or eligibility decisions. Snapshots are safe concurrently with allocation only. They must not race selector topology initialization or replacement, transport teardown, or engine destruction. Selected bytes describe the allocator's pre-fallback intent. A worker may later retry or use a different transport/NIC, so selected bytes are not posted, completed, or physical wire bytes. `TransferEngine::getSelectionStats()` replaces its output with the current RDMA snapshot. An engine without an RDMA transport returns `OK` with zero counters and an empty device list. ### Benchmark Measurement Window `tebench` runs the existing one-second warmup before entering a one-shot `MeasurementBarrier`. The final warmup worker obtains the start snapshot before releasing any worker into the measured workload. The end snapshot is obtained only after the initiator tasks finish. If warmup, the start snapshot, or a measured transfer fails, the barrier enters the failed state and the case emits no selection record. This prevents partial or warmup-contaminated JSONL output. When `--result_output_jsonl` is set for the TENT backend, each successful case appends one additional `rdma_selection_metrics` record alongside its existing target-metric records: ```json { "schema_version": 1, "record_type": "rdma_selection_metrics", "measurement_window": "snapshot_delta", "backend": "tent", "op_type": "write", "block_size": 2097152, "batch_size": 32, "num_threads": 4, "selection": { "allocations": 100, "single_path_allocations": 0, "multi_path_allocations": 100, "probe_allocations": 1, "devices": [] } } ``` `--tent_rail_topo_path` is a benchmark-only convenience option that passes a rail-pair topology file to the TENT RDMA worker configuration. It allows a benchmark to make its eligible-rail policy reproducible without changing the production configuration contract. ## Compatibility - Existing TENT users do not need configuration changes. - Counter collection does not alter allocation results. - JSONL output is opt-in through the existing `--result_output_jsonl` flag. - The API is C++ only; no C ABI or Python API is added. ## Validation The implementation includes tests for: - single-path and multi-path accounting, including folded tails; - unavailable-device history; - empty and failed allocations; - scheduled probe accounting; - concurrent snapshot readability; - warmup exclusion, snapshot failure, and failure races in the measurement barrier; - JSONL delta serialization. Acceptance for a real multi-rail run requires the JSONL selection delta and source/target NIC port-counter deltas to agree on the intended eligible rails. The JSONL record alone must not be used to claim balanced physical traffic. ## Follow-up Work This RFC intentionally separates observability from routing-policy changes. Any policy proposal must use equal workloads and report throughput, p50/p99 latency, per-rail selected shares, and port-counter deltas for equal-rail, unequal-rail, burst, and rail-exclusion cases. Temporal oscillation requires fixed-window or transition-level instrumentation; it cannot be derived from the cumulative counters introduced here. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

RFC

label

2026-10-29 21:23:34 +00:00

bsakaguchi commented

2026-10-29 21:23:36 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 12, 2026):

Thanks for opening this issue, @RuixiangMa!

Field	Value
Issue	#4598
GitHub user ID	`27672240`
Reporter	@RuixiangMa

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 12, 2026): Thanks for opening this issue, @RuixiangMa! | Field | Value | |-------|-------| | **Issue** | #4598 | | **GitHub user ID** | `27672240` | | **Reporter** | @RuixiangMa | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.