[GH-ISSUE #4734] [RFC]: Add an opt-in T-One RDMA performance regression gate #1206

New issue

Open

opened 2026-10-29 21:23:57 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:57 +00:00

Owner

Originally created by @stmatengss on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4734

Changes proposed

Extend the performance gate in #4639 with an opt-in T-One RDMA suite. GitHub-hosted runners do not provide a guaranteed RDMA hardware pair; Mooncake already uses T-One for eRDMA integration tests, but those tests do not compare native Transfer Engine performance against the PR base.

The implementation uses a fixed, exclusive two-host pair and the existing tebench JSONL output. Its initial workload is TENT DRAM read/write: 4 KiB batch-1 P99 latency and 1 MiB batch-16 throughput. GPU Direct, Store RPC, and serving benchmarks remain outside this scope.

Execution and comparison

The GitHub controller runs the reviewed base harness, submits full baseline/candidate/harness SHAs, and retrieves a machine-readable result envelope from T-One. Candidate code is built and executed only on the isolated pair.
A dedicated mooncake-rdma-perf tone-cli functional suite reports Pass/Fail while retaining the performance data. Its installable adapter and example configuration live alongside the Mooncake harness so the contract can be reviewed in one PR; deployment into tone-cli/T-One is a separate operator step.
Both hosts build both revisions in Release mode against the same frozen benchmark source and their provisioned RDMA userspace libraries. This avoids relying on tebench being packaged in a wheel. A small TENT-only --dram_numa_node benchmark option makes the DRAM placement explicit.
An integrity smoke precedes measurement. Five paired repetitions alternate A/B and B/A; each measured invocation uses tebench's built-in warmup. Initial limits are 5% throughput loss and 10% P99 growth, with relative MAD rejecting unstable samples.
GitHub independently verifies run/revision/policy/benchmark identity and recomputes the verdict. It publishes the comparison table, error annotations, and artifacts even on failure. Exit 0 is pass, 1 regression, and 2 invalid/incomplete/unstable execution. Completed jobs, missing reports, or stale successful reports cannot count as a pass.
Per-host leases, owned-process cleanup, per-command/target watchdogs, and T-One stop requests bound overlapping or cancelled work.

Rollout and validation boundary

Automatic PR execution stays off until the T-One template, fixed host/NIC/GID/NUMA configuration, matching dependencies, and reachable HTTPS result origin are commissioned. Use manual same-SHA A/A comparisons before selecting thresholds or making RDMA Performance Gate required. RXE/SIW, CPU fixtures, and a successful compile do not establish hardware RDMA performance. Fork PRs lack secrets and fail closed when the gate is enabled; reviewed manual comparisons do not automatically publish a required check on the fork PR.

Local validation covers the comparator, protocol fixtures, an injected 6% regression, stale/invalid provenance, missing results, timeouts, cancellation, exclusive leases, process cleanup, and template parsing. Actual T-One deployment and real-hardware A/A calibration remain pending; no new RDMA performance measurements are claimed.

Existing work

This extends #4639's CPU gate and reuses the current native benchmark. Searches found no open PR implementing this T-One regression gate. #2607 concerns MoE/EP incast workloads; the open RDMA scheduling/feedback PRs change transport behavior rather than this CI integration.

Before submitting a new issue

Searched relevant issues/PRs and read the contribution and existing T-One framework documentation.

AI assistance: Codex prepared the implementation and this RFC. Human review of the changed lines and infrastructure commissioning are still required.

Originally created by @stmatengss on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4734 ## Changes proposed Extend the performance gate in #4639 with an opt-in T-One RDMA suite. GitHub-hosted runners do not provide a guaranteed RDMA hardware pair; Mooncake already uses T-One for eRDMA integration tests, but those tests do not compare native Transfer Engine performance against the PR base. The implementation uses a fixed, exclusive two-host pair and the existing `tebench` JSONL output. Its initial workload is TENT DRAM read/write: 4 KiB batch-1 P99 latency and 1 MiB batch-16 throughput. GPU Direct, Store RPC, and serving benchmarks remain outside this scope. ### Execution and comparison - The GitHub controller runs the reviewed base harness, submits full baseline/candidate/harness SHAs, and retrieves a machine-readable result envelope from T-One. Candidate code is built and executed only on the isolated pair. - A dedicated `mooncake-rdma-perf` tone-cli functional suite reports Pass/Fail while retaining the performance data. Its installable adapter and example configuration live alongside the Mooncake harness so the contract can be reviewed in one PR; deployment into tone-cli/T-One is a separate operator step. - Both hosts build both revisions in Release mode against the same frozen benchmark source and their provisioned RDMA userspace libraries. This avoids relying on `tebench` being packaged in a wheel. A small TENT-only `--dram_numa_node` benchmark option makes the DRAM placement explicit. - An integrity smoke precedes measurement. Five paired repetitions alternate A/B and B/A; each measured invocation uses tebench's built-in warmup. Initial limits are 5% throughput loss and 10% P99 growth, with relative MAD rejecting unstable samples. - GitHub independently verifies run/revision/policy/benchmark identity and recomputes the verdict. It publishes the comparison table, error annotations, and artifacts even on failure. Exit 0 is pass, 1 regression, and 2 invalid/incomplete/unstable execution. Completed jobs, missing reports, or stale successful reports cannot count as a pass. - Per-host leases, owned-process cleanup, per-command/target watchdogs, and T-One stop requests bound overlapping or cancelled work. ### Rollout and validation boundary Automatic PR execution stays off until the T-One template, fixed host/NIC/GID/NUMA configuration, matching dependencies, and reachable HTTPS result origin are commissioned. Use manual same-SHA A/A comparisons before selecting thresholds or making `RDMA Performance Gate` required. RXE/SIW, CPU fixtures, and a successful compile do not establish hardware RDMA performance. Fork PRs lack secrets and fail closed when the gate is enabled; reviewed manual comparisons do not automatically publish a required check on the fork PR. Local validation covers the comparator, protocol fixtures, an injected 6% regression, stale/invalid provenance, missing results, timeouts, cancellation, exclusive leases, process cleanup, and template parsing. Actual T-One deployment and real-hardware A/A calibration remain pending; no new RDMA performance measurements are claimed. ### Existing work This extends #4639's CPU gate and reuses the current native benchmark. Searches found no open PR implementing this T-One regression gate. #2607 concerns MoE/EP incast workloads; the open RDMA scheduling/feedback PRs change transport behavior rather than this CI integration. ## Before submitting a new issue - [x] Searched relevant issues/PRs and read the contribution and existing T-One framework documentation. AI assistance: Codex prepared the implementation and this RFC. Human review of the changed lines and infrastructure commissioning are still required.

bsakaguchi added the

RFC

label

2026-10-29 21:23:57 +00:00

bsakaguchi commented

2026-10-29 21:23:58 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 17, 2026):

Thanks for opening this issue, @stmatengss!

Field	Value
Issue	#4734
GitHub user ID	`11641725`
Reporter	@stmatengss

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 17, 2026): Thanks for opening this issue, @stmatengss! | Field | Value | |-------|-------| | **Issue** | #4734 | | **GitHub user ID** | `11641725` | | **Reporter** | @stmatengss | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:34 +00:00

[PR #1206] [MERGED] feat[Store]: add multi shm support for dummy and real client #2072