[PR #243] [MERGED] ci(cursor-review): cut panel spend — 3-lab panel, xhigh judge, cell time cap, bot-bump skip #250

New issue

Closed

opened 2026-10-26 22:37:45 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:37:45 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/github-workflows/pull/243
Author: @mattmillerai
Created: 8/27/2026
Status: ✅ Merged
Merged: 8/27/2026
Merged by: @mattmillerai

Base: main ← Head: matt/cursor-review-spend-trim

📝 Commits (1)

ff4180c ci(cursor-review): cut panel spend — 3-lab panel, xhigh judge, cell time cap, bot-bump skip

📊 Changes

5 files changed (+129 additions, -34 deletions)

View changed files

📝 .github/cursor-review/README.md (+15 -9)
📝 .github/cursor-review/prompt-judge.md (+6 -3)
📝 .github/cursor-review/tests/test_catalog_drift.py (+8 -3)
📝 .github/workflows/cursor-review.yml (+95 -16)
📝 docs/callers/cursor-review.md (+5 -3)

📄 Description

ELI-5

The PR-review panel was paying four AI labs top-tier rates on every labeled PR, but a month of its own posted reviews shows one lab (Gemini) almost never found anything the other labs didn't, one cell (GPT edge-case) regularly hangs for 30 minutes producing nothing, and machine-generated pin-bump PRs were getting the full panel. This PR removes the parts that weren't earning their cost — the review keeps every finding category that has actually caught bugs, and gets faster.

What changed

Change	Spend effect	Evidence
Drop `gemini-3.1-pro` (panel is now OpenAI + Anthropic + Moonshot)	~−9%	Over 1,450 judge-kept findings on 172 PRs, Gemini's two cells were the sole raiser of 2.5% of findings and 3 of 213 critical/high; every other Gemini finding was also raised by an Opus or GPT cell. 45% of reviews already sit at the judge's 10-finding cap.
`kimi-k3-max` → `kimi-k3-high`	~−4%	Same model, one reasoning tier lower; keeps the third lab's diversity vote.
Judge default → `claude-opus-5-thinking-xhigh`	~−3%	The judge has no tools and a bounded payload; it ran at xhigh until 2026-07-18 with no recorded quality gap.
15-min step cap on each cell's `cursor-agent` run (`continue-on-error`, judge idiom)	reliability	Healthy cells finish in 3–7 min (p90 < 12). A stalled cell — 8.8% of runs, usually `gpt-5.6-sol-max edge-case` — ran to the 30-min job timeout, which GitHub reports as a cancelled run, and stretched reviews to 35+ min. The step cap produces the same status=error artifact the judge already tolerates, 15 min sooner and green.
New `skip_bot_branch_prefixes` input (default `ci/bump- chore/refresh- auto/refresh-`, Bot authors only)	~−1–2%	~30 full panels/month ran on machine-generated pin bumps and catalog refreshes. Author type is checked, so a human pushing to a `ci/bump-…` branch is still reviewed; `skip-cursor-review`/`cursor-review` labels still override per-PR. Set `''` to review every bot PR.
New `panel_models` input	enables A/Bs	JSON-array override of the panel list, shape-checked in preflight and still validated against the live catalog. Lets one repo trial e.g. `-xhigh` tiers for two weeks without forking the workflow.

Numbers come from a 30-day pull of every caller repo's runs (2,774 panel runs), the judge's per-finding "Raised by k of N" attribution, and per-cell job-log timings. Spend shares are estimated from agent-seconds × Cursor list prices (the CLI exposes no token counts).

Docs updated (README diagram/table, docs/callers/cursor-review.md), and the judge prompt now derives the panel size from the PANEL FINDINGS block instead of hardcoding "8 cells", so panel_models overrides render correct attribution.

Testing

python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' — 369 tests, OK (the 2026-08-10 catalog-drift acceptance case now pins its own era's panel/judge constants, since it regression-tests that historical report's rendering).
shellcheck on the two scripts CI checks — clean.
actionlint on cursor-review.yml — only the pre-existing job.workflow_sha warning.
Not exercised live: a real panel run with the new pins (Cursor API is currently rejecting requests on the billing hold). kimi-k3-high and claude-opus-5-thinking-xhigh are both present in the checked-in 2026-08-10 catalog fixture, and preflight fails loud on a delisted pin before any cell spends.

Rollout

Merging fans out via bump-cursor-review-callers.yml as usual. Follow-up (separate PR, after this lands): point one mid-volume caller's panel_models at the -xhigh tier ids as the measured A/B for the remaining Opus/GPT spend.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/github-workflows/pull/243 **Author:** [@mattmillerai](https://github.com/mattmillerai) **Created:** 8/27/2026 **Status:** ✅ Merged **Merged:** 8/27/2026 **Merged by:** [@mattmillerai](https://github.com/mattmillerai) **Base:** `main` ← **Head:** `matt/cursor-review-spend-trim` --- ### 📝 Commits (1) - [`ff4180c`](https://github.com/Comfy-Org/github-workflows/commit/ff4180c089c4011e3a26a2840209aa09a0a4a226) ci(cursor-review): cut panel spend — 3-lab panel, xhigh judge, cell time cap, bot-bump skip ### 📊 Changes **5 files changed** (+129 additions, -34 deletions) <details> <summary>View changed files</summary> 📝 `.github/cursor-review/README.md` (+15 -9) 📝 `.github/cursor-review/prompt-judge.md` (+6 -3) 📝 `.github/cursor-review/tests/test_catalog_drift.py` (+8 -3) 📝 `.github/workflows/cursor-review.yml` (+95 -16) 📝 `docs/callers/cursor-review.md` (+5 -3) </details> ### 📄 Description ## ELI-5 The PR-review panel was paying four AI labs top-tier rates on every labeled PR, but a month of its own posted reviews shows one lab (Gemini) almost never found anything the other labs didn't, one cell (GPT edge-case) regularly hangs for 30 minutes producing nothing, and machine-generated pin-bump PRs were getting the full panel. This PR removes the parts that weren't earning their cost — the review keeps every finding category that has actually caught bugs, and gets faster. ## What changed | Change | Spend effect | Evidence | |---|---|---| | Drop `gemini-3.1-pro` (panel is now OpenAI + Anthropic + Moonshot) | ~−9% | Over 1,450 judge-kept findings on 172 PRs, Gemini's two cells were the **sole** raiser of 2.5% of findings and 3 of 213 critical/high; every other Gemini finding was also raised by an Opus or GPT cell. 45% of reviews already sit at the judge's 10-finding cap. | | `kimi-k3-max` → `kimi-k3-high` | ~−4% | Same model, one reasoning tier lower; keeps the third lab's diversity vote. | | Judge default → `claude-opus-5-thinking-xhigh` | ~−3% | The judge has no tools and a bounded payload; it ran at xhigh until 2026-07-18 with no recorded quality gap. | | 15-min step cap on each cell's `cursor-agent` run (`continue-on-error`, judge idiom) | reliability | Healthy cells finish in 3–7 min (p90 < 12). A stalled cell — 8.8% of runs, usually `gpt-5.6-sol-max edge-case` — ran to the 30-min **job** timeout, which GitHub reports as a cancelled run, and stretched reviews to 35+ min. The step cap produces the same status=error artifact the judge already tolerates, 15 min sooner and green. | | New `skip_bot_branch_prefixes` input (default `ci/bump- chore/refresh- auto/refresh-`, Bot authors only) | ~−1–2% | ~30 full panels/month ran on machine-generated pin bumps and catalog refreshes. Author *type* is checked, so a human pushing to a `ci/bump-…` branch is still reviewed; `skip-cursor-review`/`cursor-review` labels still override per-PR. Set `''` to review every bot PR. | | New `panel_models` input | enables A/Bs | JSON-array override of the panel list, shape-checked in preflight and still validated against the live catalog. Lets one repo trial e.g. `-xhigh` tiers for two weeks without forking the workflow. | Numbers come from a 30-day pull of every caller repo's runs (2,774 panel runs), the judge's per-finding "Raised by k of N" attribution, and per-cell job-log timings. Spend shares are estimated from agent-seconds × Cursor list prices (the CLI exposes no token counts). Docs updated (README diagram/table, `docs/callers/cursor-review.md`), and the judge prompt now derives the panel size from the PANEL FINDINGS block instead of hardcoding "8 cells", so `panel_models` overrides render correct attribution. ## Testing - `python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'` — **369 tests, OK** (the 2026-08-10 catalog-drift acceptance case now pins its own era's panel/judge constants, since it regression-tests that historical report's rendering). - `shellcheck` on the two scripts CI checks — clean. - `actionlint` on `cursor-review.yml` — only the pre-existing `job.workflow_sha` warning. - Not exercised live: a real panel run with the new pins (Cursor API is currently rejecting requests on the billing hold). `kimi-k3-high` and `claude-opus-5-thinking-xhigh` are both present in the checked-in 2026-08-10 catalog fixture, and preflight fails loud on a delisted pin before any cell spends. ## Rollout Merging fans out via `bump-cursor-review-callers.yml` as usual. Follow-up (separate PR, after this lands): point one mid-volume caller's `panel_models` at the `-xhigh` tier ids as the measured A/B for the remaining Opus/GPT spend. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:37:45 +00:00