[PR #299] [MERGED] test(refresh-reviewers): extend input/docs drift check to caller blocks, README knobs, and the Default column #298

New issue

Closed

opened 2026-10-26 22:38:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:38:12 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/github-workflows/pull/299
Author: @mattmillerai
Created: 9/16/2026
Status: ✅ Merged
Merged: 9/17/2026
Merged by: @mattmillerai

Base: main ← Head: matt/be-14782-refresh-reviewers-drift-extend

📝 Commits (8)

7e5f4d3 docs(refresh-reviewers): add its own caller guide so the workflows_ref cross-check covers it
fc9f7ac docs(refresh-reviewers): tighten guide accuracy from cursor-review panel
1bbe0e7 test(refresh-reviewers): pin workflow_call inputs against the docs/callers Inputs table
2321bdc test(refresh-reviewers): run the drift test on the files it parses; harden its scanners
f6d5156 test(refresh-reviewers): extend drift check to caller with: blocks, README knobs, and the Default column
f4f7038 test(refresh-reviewers): scope drift scanners and harden Default parsing
5daa18f Merge remote-tracking branch 'origin/main' into matt/be-14782-refresh-reviewers-drift-extend
ceefaa1 fix(refresh-reviewers): stop comment-stripping from leaving stray quotes on a quoted default

📊 Changes

1 file changed (+394 additions, -6 deletions)

View changed files

📝 .github/refresh-reviewers/tests/test_inputs_docs.py (+394 -6)

📄 Description

STACKED — merging lands on matt/be-14761-refresh-reviewers-inputs-docs-drift-test (owned by @mattmillerai, PR #298), NOT main. This PR is stacked on #298, which is itself stacked on #296 (docs/callers/refresh-reviewers.md). The file edited here does not exist on main yet — it is added by #298. Not ready to merge until #298 and #296 land; GitHub retargets this PR to main as each foundation merges. Review sees only the net diff.

What this does

Extends .github/refresh-reviewers/tests/test_inputs_docs.py (added by #298) with three new drift assertions, covering the surfaces a consumer actually copies — the same #31 / BE-4691 drift class the file already polices (a phantom input in the docs is a zero-job startup_failure for whoever copies it), pushed one level in from the Inputs table to the copy-paste examples.

Fenced/comment with: blocks ⊆ declared inputs (test_example_with_blocks_are_declared_inputs). Both copy-paste callers this repo ships — the ## Caller fenced YAML in docs/callers/refresh-reviewers.md, and the # -prefixed example in the refresh-reviewers.yml header comment — pass inputs via with:. A new relative-indentation scanner collects the mapping keys directly under with: (skipping uses: / secrets:, which sit at the with: indent and end the block, and placeholder scalar values, which never start a line; for the header comment the leading # is stripped first so indentation is preserved) and asserts each set is a subset of the declared inputs. Subset, not equality — an example legitimately shows only a couple of inputs.
README phantom-only knob check (test_readme_knob_table_has_no_phantom_knob). .github/refresh-reviewers/README.md's "Knob defaults (and why)" table is intentionally partial and uses combined-cell rows (| `top_k` / `floor` |) the existing TABLE_KEY regex correctly does not capture. Reusing documented_inputs(README, "## Knob defaults (and why)"), this asserts only the phantom direction (documented - declared == set()) — never set-equality, which would be permanently red because the table omits many inputs by design.
Pin the guide's Default column (test_guide_default_column_matches_workflow). Both existing assertions are name-only, so the guide Inputs table's Default column could silently drift from the workflow's real default:. A new scanner pairs each input key with its 8-space default: line in refresh-reviewers.yml and compares against the normalized Default cell of the guide's ## Inputs table (strip backticks/markdown; '' ↔ default: ''; and the workflows_ref special case — required: true with no default, shown as — (**required**)), asserting per-input equality with a message naming the input and both values.

Each new scanner keeps the existing setUp guarding style — sanity-checked non-empty / still finding workflows_ref — so a parser gone quiet fails loudly instead of passing vacuously. Stdlib only, no PyYAML.

No CI wiring needed: test-refresh-reviewers.yml's paths: filter already includes .github/refresh-reviewers/**, and it runs python3 -m unittest discover, so the new methods run automatically.

Verification

python3 -m unittest discover -s .github/refresh-reviewers/tests -p 'test_*.py' -v → 53 tests OK (the 3 new ones included).
Falsified each new guard against injected drift on temp copies of the source files (tracked files untouched): a phantom with: key, a phantom README knob, a drifted guide Default cell, and a workflows_ref shown with a fake default instead of the required marker — all four go RED, confirming the checks are real, not vacuous.
Confirmed both example with: blocks resolve to {map_exclude, workflows_ref} (⊆ declared), the README knob scanner to {window_months, half_life_days, floor_min_touches, pr_branch} (combined-cell rows correctly excluded), and all 12 inputs' Default cells match the workflow.

Provenance

Authored by: agent-work loop
Verified: python3 -m unittest discover -s .github/refresh-reviewers/tests -p 'test_*.py' green (53 tests); each new guard falsified against injected drift; py_compile clean; diff is a single test file, +260 lines, additive only.
Deviations: Built as a stack on the unmerged #298 branch rather than waiting for it to land on main (per the standing always-stack directive — the file this edits exists only on that branch). See Residual.

Residual

Blocked base not on main. This change cannot merge until #298 (adds the test file) and transitively #296 (adds docs/callers/refresh-reviewers.md) merge. Both were open at authoring time. If #298 is revised in review in a way that reshapes test_inputs_docs.py, this stack may need a rebase; the net diff is purely additive (three new methods + supporting module functions), so conflicts should be mechanical.
Unexercised references. The source review thread (github-workflows#298 discussion_r4026892130) and the originating spike's findings comment are not reachable from the build sandbox; the three extensions were implemented directly from the recommended plan carried in the task. No behavioral surface of refresh-reviewers itself is touched — this is test-only coverage of existing docs/workflow content.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/github-workflows/pull/299 **Author:** [@mattmillerai](https://github.com/mattmillerai) **Created:** 9/16/2026 **Status:** ✅ Merged **Merged:** 9/17/2026 **Merged by:** [@mattmillerai](https://github.com/mattmillerai) **Base:** `main` ← **Head:** `matt/be-14782-refresh-reviewers-drift-extend` --- ### 📝 Commits (8) - [`7e5f4d3`](https://github.com/Comfy-Org/github-workflows/commit/7e5f4d3ef5814645f38b7f96b4e90aad91196206) docs(refresh-reviewers): add its own caller guide so the workflows_ref cross-check covers it - [`fc9f7ac`](https://github.com/Comfy-Org/github-workflows/commit/fc9f7ac3dd01fbcc857add10c8a4c3ef9e59edbf) docs(refresh-reviewers): tighten guide accuracy from cursor-review panel - [`1bbe0e7`](https://github.com/Comfy-Org/github-workflows/commit/1bbe0e7b164eb58110b360aba8efb45d74280d96) test(refresh-reviewers): pin workflow_call inputs against the docs/callers Inputs table - [`2321bdc`](https://github.com/Comfy-Org/github-workflows/commit/2321bdc284f1022f5e173c5a6bbac47f9a168075) test(refresh-reviewers): run the drift test on the files it parses; harden its scanners - [`f6d5156`](https://github.com/Comfy-Org/github-workflows/commit/f6d5156605f746c13d8da4c0d45f9b6fbdea43df) test(refresh-reviewers): extend drift check to caller with: blocks, README knobs, and the Default column - [`f4f7038`](https://github.com/Comfy-Org/github-workflows/commit/f4f70387856c4a6daeb9d6e59ea5cb4e9329442c) test(refresh-reviewers): scope drift scanners and harden Default parsing - [`5daa18f`](https://github.com/Comfy-Org/github-workflows/commit/5daa18fba1b4e1588eb6cc6adf55e1f8957a6271) Merge remote-tracking branch 'origin/main' into matt/be-14782-refresh-reviewers-drift-extend - [`ceefaa1`](https://github.com/Comfy-Org/github-workflows/commit/ceefaa125f46fcb1927541de3db1d1f31044322f) fix(refresh-reviewers): stop comment-stripping from leaving stray quotes on a quoted default ### 📊 Changes **1 file changed** (+394 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `.github/refresh-reviewers/tests/test_inputs_docs.py` (+394 -6) </details> ### 📄 Description > **STACKED — merging lands on `matt/be-14761-refresh-reviewers-inputs-docs-drift-test` (owned by @mattmillerai, PR #298), NOT `main`.** This PR is stacked on #298, which is itself stacked on #296 (`docs/callers/refresh-reviewers.md`). The file edited here does not exist on `main` yet — it is added by #298. Not ready to merge until #298 and #296 land; GitHub retargets this PR to `main` as each foundation merges. Review sees only the net diff. ## What this does Extends `.github/refresh-reviewers/tests/test_inputs_docs.py` (added by #298) with three new drift assertions, covering the surfaces a consumer actually copies — the same #31 / BE-4691 drift class the file already polices (a phantom input in the docs is a zero-job `startup_failure` for whoever copies it), pushed one level in from the Inputs table to the copy-paste examples. 1. **Fenced/comment `with:` blocks ⊆ declared inputs** (`test_example_with_blocks_are_declared_inputs`). Both copy-paste callers this repo ships — the `## Caller` fenced YAML in `docs/callers/refresh-reviewers.md`, and the `# `-prefixed example in the `refresh-reviewers.yml` header comment — pass inputs via `with:`. A new relative-indentation scanner collects the mapping keys directly under `with:` (skipping `uses:` / `secrets:`, which sit at the `with:` indent and end the block, and placeholder scalar values, which never start a line; for the header comment the leading `# ` is stripped first so indentation is preserved) and asserts each set is a **subset** of the declared inputs. Subset, not equality — an example legitimately shows only a couple of inputs. 2. **README phantom-only knob check** (`test_readme_knob_table_has_no_phantom_knob`). `.github/refresh-reviewers/README.md`'s "Knob defaults (and why)" table is intentionally partial and uses combined-cell rows (`` | `top_k` / `floor` | ``) the existing `TABLE_KEY` regex correctly does not capture. Reusing `documented_inputs(README, "## Knob defaults (and why)")`, this asserts **only** the phantom direction (`documented - declared == set()`) — never set-equality, which would be permanently red because the table omits many inputs by design. 3. **Pin the guide's Default column** (`test_guide_default_column_matches_workflow`). Both existing assertions are name-only, so the guide Inputs table's `Default` column could silently drift from the workflow's real `default:`. A new scanner pairs each input key with its 8-space `default:` line in `refresh-reviewers.yml` and compares against the normalized Default cell of the guide's `## Inputs` table (strip backticks/markdown; `''` ↔ `default: ''`; and the `workflows_ref` special case — `required: true` with no default, shown as `— (**required**)`), asserting per-input equality with a message naming the input and both values. Each new scanner keeps the existing `setUp` guarding style — sanity-checked non-empty / still finding `workflows_ref` — so a parser gone quiet fails loudly instead of passing vacuously. Stdlib only, no PyYAML. No CI wiring needed: `test-refresh-reviewers.yml`'s `paths:` filter already includes `.github/refresh-reviewers/**`, and it runs `python3 -m unittest discover`, so the new methods run automatically. ## Verification - `python3 -m unittest discover -s .github/refresh-reviewers/tests -p 'test_*.py' -v` → **53 tests OK** (the 3 new ones included). - Falsified each new guard against injected drift on temp copies of the source files (tracked files untouched): a phantom `with:` key, a phantom README knob, a drifted guide Default cell, and a `workflows_ref` shown with a fake default instead of the required marker — all four go **RED**, confirming the checks are real, not vacuous. - Confirmed both example `with:` blocks resolve to `{map_exclude, workflows_ref}` (⊆ declared), the README knob scanner to `{window_months, half_life_days, floor_min_touches, pr_branch}` (combined-cell rows correctly excluded), and all 12 inputs' Default cells match the workflow. ## Provenance - **Authored by:** agent-work loop - **Verified:** `python3 -m unittest discover -s .github/refresh-reviewers/tests -p 'test_*.py'` green (53 tests); each new guard falsified against injected drift; `py_compile` clean; diff is a single test file, +260 lines, additive only. - **Deviations:** Built as a **stack on the unmerged #298 branch** rather than waiting for it to land on `main` (per the standing always-stack directive — the file this edits exists only on that branch). See Residual. ## Residual - **Blocked base not on `main`.** This change cannot merge until **#298** (adds the test file) and transitively **#296** (adds `docs/callers/refresh-reviewers.md`) merge. Both were open at authoring time. If #298 is revised in review in a way that reshapes `test_inputs_docs.py`, this stack may need a rebase; the net diff is purely additive (three new methods + supporting module functions), so conflicts should be mechanical. - **Unexercised references.** The source review thread (`github-workflows#298 discussion_r4026892130`) and the originating spike's findings comment are not reachable from the build sandbox; the three extensions were implemented directly from the recommended plan carried in the task. No behavioral surface of `refresh-reviewers` itself is touched — this is test-only coverage of existing docs/workflow content. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:38:12 +00:00