[GH-ISSUE #4534] [RFC][TENT] Evaluate opt-in peer-attributed RDMA feedback #1157

New issue

Open

opened 2026-10-29 21:23:25 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:25 +00:00

Owner

Originally created by @jacklin78911-collab on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4534

Problem and question

TENT currently learns selection bandwidth at NIC scope. A slow destination can lower that NIC's learned bandwidth for requests to a different destination. This RFC proposes an explicitly enabled experiment to answer two separate questions: does peer-specific completion feedback improve that attribution, and is peer-specific inflight accounting worth losing visibility of other peers competing for the same local NIC?

Proposed experiment

Keep the existing inverse-score allocator, topology/candidate rules, priority/admission, probing, lane dispatch and QP execution. Add a per-device, per-SegmentID ledger behind transports/rdma/peer_accounting (default false). Disabled operation does not maintain peer entries or take peer locks/read the peer clock.

All enabled variants share the same ledger:

Variant	Inflight used for scoring	Bandwidth feedback
V0	zero	nominal link rate
V1	NIC total	NIC completion EWMA
V2	NIC total	peer completion EWMA
V3	peer total	peer completion EWMA

V2/V3 fall back to NIC feedback for the entire candidate set if any peer sample is missing or expired. V3 still uses peer inflight during feedback fallback. Neither V2 nor V3 is selected as the winner in advance.

The history target is not admission control: active entries may exceed it and only idle entries are reclaimed. Existing worker charge ownership controls exactly-once release; a peer accounting error must not strand the corresponding NIC charge. Absent peer identity is distinct from peer ID zero. All variants preserve actual-slice byte accounting.

Evaluation and tradeoffs

A small native two-peer workload submits independent arrival streams with disjoint registered-memory slots, records scheduled arrival/submission/terminal visibility, and checks completion and readback. Compare ledger-disabled baseline against V1 for cost, V2 against V1 for feedback attribution, and V3 against V2 for the local-contention tradeoff. A 2 MiB WRITE with 64 KiB blocks checks the actual 32-slice case.

Calibrate TTL once per scenario from V1 diagnostic noncold aggregate decisions: nearest-rank P99 of the maximum candidate sample age, rounded up to milliseconds, at least 1 ms. Cold starts stay separate. Normal runs disable detailed tracing and retain fallback/error counters. Process-wide P/Q counters are not evidence of per-flow feedback use. Keep all planned paired blocks and failures independent of effect direction.

Current evidence is local correctness/control testing only. Real multi-node RDMA, contention cost and net benefit remain pending. Locks, peer history cost, feedback starvation and V3's deliberate local-competition blind spot remain explicit risks; no production throughput or stability improvement is claimed.

Relationship and scope

This is independent of #4362, which changes within-batch allocation; this experiment retains inverse-score allocation. It does not replace the peer rail policy correctness work in #4435 or receiver-credit/admission proposals such as #2849. A draft implementation is being posted for inspection, not merge approval. Source is the frozen experiment based on 0d58aacc, tree 06a8238cb20080bae35663964eb4c28d5f5e3b40.

AI assistance: Codex helped implementation, tests and this write-up; Claude reviewed the frozen patch. Human line-by-line review is not asserted.

Originally created by @jacklin78911-collab on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4534 ## Problem and question TENT currently learns selection bandwidth at NIC scope. A slow destination can lower that NIC's learned bandwidth for requests to a different destination. This RFC proposes an explicitly enabled experiment to answer two separate questions: does peer-specific completion feedback improve that attribution, and is peer-specific inflight accounting worth losing visibility of other peers competing for the same local NIC? ## Proposed experiment Keep the existing inverse-score allocator, topology/candidate rules, priority/admission, probing, lane dispatch and QP execution. Add a per-device, per-SegmentID ledger behind `transports/rdma/peer_accounting` (default false). Disabled operation does not maintain peer entries or take peer locks/read the peer clock. All enabled variants share the same ledger: | Variant | Inflight used for scoring | Bandwidth feedback | | --- | --- | --- | | V0 | zero | nominal link rate | | V1 | NIC total | NIC completion EWMA | | V2 | NIC total | peer completion EWMA | | V3 | peer total | peer completion EWMA | V2/V3 fall back to NIC feedback for the entire candidate set if any peer sample is missing or expired. V3 still uses peer inflight during feedback fallback. Neither V2 nor V3 is selected as the winner in advance. The history target is not admission control: active entries may exceed it and only idle entries are reclaimed. Existing worker charge ownership controls exactly-once release; a peer accounting error must not strand the corresponding NIC charge. Absent peer identity is distinct from peer ID zero. All variants preserve actual-slice byte accounting. ## Evaluation and tradeoffs A small native two-peer workload submits independent arrival streams with disjoint registered-memory slots, records scheduled arrival/submission/terminal visibility, and checks completion and readback. Compare ledger-disabled baseline against V1 for cost, V2 against V1 for feedback attribution, and V3 against V2 for the local-contention tradeoff. A 2 MiB WRITE with 64 KiB blocks checks the actual 32-slice case. Calibrate TTL once per scenario from V1 diagnostic noncold aggregate decisions: nearest-rank P99 of the maximum candidate sample age, rounded up to milliseconds, at least 1 ms. Cold starts stay separate. Normal runs disable detailed tracing and retain fallback/error counters. Process-wide P/Q counters are not evidence of per-flow feedback use. Keep all planned paired blocks and failures independent of effect direction. Current evidence is local correctness/control testing only. Real multi-node RDMA, contention cost and net benefit remain pending. Locks, peer history cost, feedback starvation and V3's deliberate local-competition blind spot remain explicit risks; no production throughput or stability improvement is claimed. ## Relationship and scope This is independent of #4362, which changes within-batch allocation; this experiment retains inverse-score allocation. It does not replace the peer rail policy correctness work in #4435 or receiver-credit/admission proposals such as #2849. A draft implementation is being posted for inspection, not merge approval. Source is the frozen experiment based on `0d58aacc`, tree `06a8238cb20080bae35663964eb4c28d5f5e3b40`. AI assistance: Codex helped implementation, tests and this write-up; Claude reviewed the frozen patch. Human line-by-line review is not asserted.

bsakaguchi commented

2026-10-29 21:23:26 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 9, 2026):

Thanks for opening this issue, @jacklin78911-collab!

Field	Value
Issue	#4534
GitHub user ID	`251732545`
Reporter	@jacklin78911-collab

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 9, 2026): Thanks for opening this issue, @jacklin78911-collab! | Field | Value | |-------|-------| | **Issue** | #4534 | | **GitHub user ID** | `251732545` | | **Reporter** | @jacklin78911-collab | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.