[PR #147] [MERGED] fix(cursor-review): collapse catalog-drift review-me list to one row per family (BE-6911) #158

New issue

Closed

opened 2026-10-26 22:36:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:36:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/github-workflows/pull/147
Author: @mattmillerai
Created: 8/10/2026
Status: ✅ Merged
Merged: 8/12/2026
Merged by: @mattmillerai

Base: main ← Head: matt/be-6911-catalog-drift-tier-collapse

📝 Commits (1)

5281099 fix(cursor-review): collapse catalog-drift review-me list to one row per family (BE-6911)

📊 Changes

4 files changed (+630 additions, -19 deletions)

View changed files

📝 .github/cursor-review/catalog-drift.py (+214 -16)
➕ .github/cursor-review/tests/fixtures/catalog-2026-08-10.txt (+197 -0)
📝 .github/cursor-review/tests/test_catalog_drift.py (+218 -2)
📝 README.md (+1 -1)

📄 Description

ELI-5

Every Monday a robot compares the four AI models our PR-review panel uses against Cursor's live model list, and files a report saying "here's everything you're not using — worth a look?". Cursor sells each model at half a dozen effort levels (-low, -high, -max, each optionally -fast), and we pin exactly one level per model — so the report was listing every other effort level of every model we already use, forever. Last Monday that came to 178 lines, cut off at 25 per vendor in whatever order Cursor happened to print them. The two brand-new models that actually shipped that week were on lines 26+, invisible. This groups the effort levels back into one line per model and puts the newest models first, so the report is 37 lines and the new stuff is at the top.

What changed

.github/cursor-review/catalog-drift.py, rendering only:

Tier collapse. New module-level TIER_SUFFIXES (none/minimal/low/medium/high/extra-high/xhigh/max, each optionally -fast, plus a bare -fast) and family_of() split an id into (family, tier). The same-lab review-me list now renders one row per (lab, family): the family, how many tiers the catalog offers and which, and one member's catalog note reproduced verbatim.
Signal ordering. collapse_tiers() sorts rows by _family_version() descending (every digit run in the family id, in order — labs write versions as 5.6, 4-8 and k3, so anything smarter would need a per-lab table), ties broken by catalog order. MAX_FAMILY_IDS now drops the oldest families instead of an arbitrary tail of Cursor's print order.
Bounds. New MAX_TIERS_PER_ROW = 8 keeps a collapsed row at roughly the char cost of the single id row it replaces, so the section's worst-case budget is unchanged and the raw catalog fold keeps its room. A truncated tier list still states the full count and drops the weakest tiers.
The stale MAX_FAMILY_IDS comment (the one claiming "a few dozen models" and "every list leads with its highest-signal rows" — neither of which was true of this list) is rewritten to describe the shipped behaviour.
README.md's catalog row gains a clause on the per-family rendering.

Untouched on purpose: analyze(), the JSON report, summary_line(), urgent / has_findings, the NO-ZDR handling on pins, and the unpinned-families fold (still one row per id in catalog order — it's short and already collapsed).

Result against the 2026-08-10 catalog (#144)

178 unpinned same-lab ids → 37 rows. The gpt group goes from 25 shown + "… and 57 more" to 11 rows with nothing hidden, and reads:

- `gpt-5.6-sol-*`   — 11 tiers …  · _panel already pins `gpt-5.6-sol-max`_
- `gpt-5.6-terra-*` — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-terra-max` — `- GPT-5.6 Terra 1M Max`
- `gpt-5.6-luna-*`  — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-luna-max`  — `- GPT-5.6 Luna 1M Max`
- `gpt-5.5-*` … `gpt-5.4-*` … `gpt-5.3-codex-*` … `gpt-5.2-*` … `gpt-5.1-*` … `gpt-5-mini`

claude 87 → 21 rows, gemini 6 → 3, kimi 3 → 2. No group emits a "… and N more" line. Rendered body drops from ~44 KB to ~16 KB.

Tests

python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py' — 175 pass (159 before + 16 new). The other three Python suites (agents-md-integrity, groom, refresh-reviewers) are green too.

New TierCollapseTest covers: the family/tier split (longest-tier-wins for extra-high, bare -fast, no-suffix ids joining their family), version ordering including the catalog-order tie-break and a version-less family, tier collapse in the rendered body, verbatim notes on a collapsed row, the capped-rows message, a synthetic brand-new family printed below the pre-change cut, the worst-case char budget, and the real #144 catalog.

The #144 catalog is committed verbatim as .github/cursor-review/tests/fixtures/catalog-2026-08-10.txt so the acceptance case is asserted against the real thing rather than a hand-built approximation. It's inside the suite's existing path filter.

Judgment calls

-thinking is deliberately NOT a collapsible suffix. It reads like a reasoning knob, but the panel pins on it (claude-opus-5-thinking-max vs claude-opus-5-max are different products for this purpose), so collapsing them would hide the distinction a promotion decision turns on. Consequence: claude-4.6-opus-high-thinking (suffixes in the reverse order) stays a row of its own. Cosmetic.
minimal and extra-high were added to the ticket's named suffix set — both appear in the live catalog (gemini-3.6-flash-minimal, gpt-5.5-extra-high) and are plainly the same kind of tier.
A collapsed row shows one member's note, not all of them. Notes are per-tier ("GPT-5.6 Terra 1M Max"), so a family row reproduces the strongest tier's verbatim and the section intro points at the raw fold for the rest. The trade: a NO-ZDR marker present on only a non-top tier of a family is no longer in the report body (it is still in the raw fold). Pre-change it was equally invisible for anything past row 25, and I did not add NO-ZDR detection to candidate rows because the acceptance criteria require notes stay verbatim and uninterpreted there — the NO-ZDR check on pins (the urgent path) is untouched.
Beyond scope, flagged not fixed: a delisted pin's "available for lab X" line still lists raw ids capped at MAX_FAMILY_IDS (catalog-drift.py line ~713). Same underlying noise, but it's the urgent path that gets read carefully and the ticket scopes only the review-me list. Happy to follow up.
Negative-claim falsification (self-review step e): trigger did not fire. The diff adds no "not supported"/"unavailable"/"STOP" string, no throw/deny dead-end, and flips no test to assert a dead-end. It renders strictly more families than before and points at the raw catalog fold for per-id detail; nothing is denied.

Not done (out of scope per the ticket)

No pin is bumped — all four were re-derived against this catalog and none warrants a change; that stays a human call. urgent / has_findings / sticky-issue lifecycle, the family→lab alias table (still derived from the pins), cursor-review.yml's pin list, and the unpinned-families fold are all untouched.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/github-workflows/pull/147 **Author:** [@mattmillerai](https://github.com/mattmillerai) **Created:** 8/10/2026 **Status:** ✅ Merged **Merged:** 8/12/2026 **Merged by:** [@mattmillerai](https://github.com/mattmillerai) **Base:** `main` ← **Head:** `matt/be-6911-catalog-drift-tier-collapse` --- ### 📝 Commits (1) - [`5281099`](https://github.com/Comfy-Org/github-workflows/commit/5281099ac60d897d60956433e42ca94412f11631) fix(cursor-review): collapse catalog-drift review-me list to one row per family (BE-6911) ### 📊 Changes **4 files changed** (+630 additions, -19 deletions) <details> <summary>View changed files</summary> 📝 `.github/cursor-review/catalog-drift.py` (+214 -16) ➕ `.github/cursor-review/tests/fixtures/catalog-2026-08-10.txt` (+197 -0) 📝 `.github/cursor-review/tests/test_catalog_drift.py` (+218 -2) 📝 `README.md` (+1 -1) </details> ### 📄 Description ## ELI-5 Every Monday a robot compares the four AI models our PR-review panel uses against Cursor's live model list, and files a report saying "here's everything you're not using — worth a look?". Cursor sells each model at half a dozen effort levels (`-low`, `-high`, `-max`, each optionally `-fast`), and we pin exactly one level per model — so the report was listing every *other* effort level of every model we already use, forever. Last Monday that came to 178 lines, cut off at 25 per vendor in whatever order Cursor happened to print them. The two brand-new models that actually shipped that week were on lines 26+, invisible. This groups the effort levels back into one line per model and puts the newest models first, so the report is 37 lines and the new stuff is at the top. ## What changed `.github/cursor-review/catalog-drift.py`, rendering only: - **Tier collapse.** New module-level `TIER_SUFFIXES` (`none`/`minimal`/`low`/`medium`/`high`/`extra-high`/`xhigh`/`max`, each optionally `-fast`, plus a bare `-fast`) and `family_of()` split an id into `(family, tier)`. The same-lab review-me list now renders one row per `(lab, family)`: the family, how many tiers the catalog offers and which, and one member's catalog note reproduced verbatim. - **Signal ordering.** `collapse_tiers()` sorts rows by `_family_version()` descending (every digit run in the family id, in order — labs write versions as `5.6`, `4-8` and `k3`, so anything smarter would need a per-lab table), ties broken by catalog order. `MAX_FAMILY_IDS` now drops the *oldest* families instead of an arbitrary tail of Cursor's print order. - **Bounds.** New `MAX_TIERS_PER_ROW = 8` keeps a collapsed row at roughly the char cost of the single id row it replaces, so the section's worst-case budget is unchanged and the raw catalog fold keeps its room. A truncated tier list still states the full count and drops the *weakest* tiers. - The stale `MAX_FAMILY_IDS` comment (the one claiming "a few dozen models" and "every list leads with its highest-signal rows" — neither of which was true of this list) is rewritten to describe the shipped behaviour. - `README.md`'s catalog row gains a clause on the per-family rendering. Untouched on purpose: `analyze()`, the JSON report, `summary_line()`, `urgent` / `has_findings`, the NO-ZDR handling on *pins*, and the unpinned-families fold (still one row per id in catalog order — it's short and already collapsed). ## Result against the 2026-08-10 catalog (#144) 178 unpinned same-lab ids → **37 rows**. The `gpt` group goes from 25 shown + "… and 57 more" to **11 rows with nothing hidden**, and reads: ``` - `gpt-5.6-sol-*` — 11 tiers … · _panel already pins `gpt-5.6-sol-max`_ - `gpt-5.6-terra-*` — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-terra-max` — `- GPT-5.6 Terra 1M Max` - `gpt-5.6-luna-*` — 12 tiers (`max`, `max-fast`, `xhigh`, …); top `gpt-5.6-luna-max` — `- GPT-5.6 Luna 1M Max` - `gpt-5.5-*` … `gpt-5.4-*` … `gpt-5.3-codex-*` … `gpt-5.2-*` … `gpt-5.1-*` … `gpt-5-mini` ``` `claude` 87 → 21 rows, `gemini` 6 → 3, `kimi` 3 → 2. No group emits a "… and N more" line. Rendered body drops from ~44 KB to ~16 KB. ## Tests `python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'` — **175 pass** (159 before + 16 new). The other three Python suites (agents-md-integrity, groom, refresh-reviewers) are green too. New `TierCollapseTest` covers: the family/tier split (longest-tier-wins for `extra-high`, bare `-fast`, no-suffix ids joining their family), version ordering including the catalog-order tie-break and a version-less family, tier collapse in the rendered body, verbatim notes on a collapsed row, the capped-rows message, a synthetic brand-new family printed below the pre-change cut, the worst-case char budget, and the real #144 catalog. The #144 catalog is committed verbatim as `.github/cursor-review/tests/fixtures/catalog-2026-08-10.txt` so the acceptance case is asserted against the real thing rather than a hand-built approximation. It's inside the suite's existing path filter. ## Judgment calls - **`-thinking` is deliberately NOT a collapsible suffix.** It reads like a reasoning knob, but the panel pins *on* it (`claude-opus-5-thinking-max` vs `claude-opus-5-max` are different products for this purpose), so collapsing them would hide the distinction a promotion decision turns on. Consequence: `claude-4.6-opus-high-thinking` (suffixes in the reverse order) stays a row of its own. Cosmetic. - **`minimal` and `extra-high` were added to the ticket's named suffix set** — both appear in the live catalog (`gemini-3.6-flash-minimal`, `gpt-5.5-extra-high`) and are plainly the same kind of tier. - **A collapsed row shows one member's note, not all of them.** Notes are per-tier ("GPT-5.6 Terra 1M Max"), so a family row reproduces the strongest tier's verbatim and the section intro points at the raw fold for the rest. The trade: a NO-ZDR marker present on *only* a non-top tier of a family is no longer in the report body (it is still in the raw fold). Pre-change it was equally invisible for anything past row 25, and I did not add NO-ZDR detection to candidate rows because the acceptance criteria require notes stay verbatim and uninterpreted there — the NO-ZDR check on **pins** (the urgent path) is untouched. - **Beyond scope, flagged not fixed:** a *delisted* pin's "available for lab X" line still lists raw ids capped at `MAX_FAMILY_IDS` (`catalog-drift.py` line ~713). Same underlying noise, but it's the urgent path that gets read carefully and the ticket scopes only the review-me list. Happy to follow up. - **Negative-claim falsification (self-review step e): trigger did not fire.** The diff adds no "not supported"/"unavailable"/"STOP" string, no throw/deny dead-end, and flips no test to assert a dead-end. It renders strictly more families than before and points at the raw catalog fold for per-id detail; nothing is denied. ## Not done (out of scope per the ticket) No pin is bumped — all four were re-derived against this catalog and none warrants a change; that stays a human call. `urgent` / `has_findings` / sticky-issue lifecycle, the family→lab alias table (still derived from the pins), `cursor-review.yml`'s pin list, and the unpinned-families fold are all untouched. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:36:53 +00:00