mirror of
https://github.com/Comfy-Org/github-workflows.git
synced 2026-11-05 00:17:37 +00:00
[PR #306] [MERGED] test(workflow-pins): table-driven declared-vs-documented input drift harness for all 16 reusables #307
Labels
No labels
cursor-review-catalog-drift
groom
groom
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Comfy-Org/github-workflows#307
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
📋 Pull Request Information
Original PR: https://github.com/Comfy-Org/github-workflows/pull/306
Author: @mattmillerai
Created: 9/18/2026
Status: ✅ Merged
Merged: 9/18/2026
Merged by: @mattmillerai
Base:
main← Head:matt/be-15259-inputs-docs-drift-harness📝 Commits (2)
16734b4test(workflow-pins): table-driven declared-vs-documented input drift harness for all 16 reusables4f80a1dtest(workflow-pins): harden the drift harness's own parsers against the panel review📊 Changes
2 files changed (+1186 additions, -0 deletions)
View changed files
📝
.github/workflow-pins/README.md(+17 -0)➕
.github/workflow-pins/tests/test_inputs_docs_drift.py(+1169 -0)📄 Description
ELI-5
Every reusable workflow here has a caller guide that lists its inputs in a table. If the table names an input the workflow does not actually declare, anyone who copies that caller gets a workflow that dies at startup with zero jobs and no logs — which is exactly what happened in #31 and took weeks to notice. Two workflows already had a hand-written test guarding against that. The other fourteen had nothing. This adds one table-driven test that guards all sixteen, so the next drift fails in CI instead of in somebody else's repo.
What changed
.github/workflow-pins/tests/test_inputs_docs_drift.py(new) — aROWStuple with one entry per reusable, one set of stdlib-only parsers ported from.github/refresh-reviewers/tests/test_inputs_docs.py(CRLF handling and blank/comment tolerance included), and one generatedTestCaseper reusable so-vnames the workflow that drifted rather than burying it in a subTest label.It lives under
.github/workflow-pins/because that is the one directory whose CI already watches both.github/workflows/**anddocs/callers/**, the two surfaces this compares.test-workflow-pins.yml:53already runsunittest discover -s .github/workflow-pins/tests -p 'test_*.py', so the file is picked up with no CI edit. It does not touchcheck_workflow_pins.pyor itsworkflows_refdocs cross-check, and nodocs/callers/*.mdis edited.Per row it asserts: both name-scanners find the row's
sentinelinput (anti-vacuity); documented − declared is empty; declared − documented − allowlist is empty; noKNOWN_UNDOCUMENTEDentry has gone stale; and thewith:keys of every copy-paste caller it owns (every fence in the guide, plus the workflow's header comment) are a subset of the declared inputs. Whichwith:belongs to which reusable is decided by theuses:governing it, not by the heading above it. Plus a coverage test asserting theROWSnames equal the set of reusables on disk, andSharedDocExampleCallersTestcoveringREADME.mdanddocs/callers/README.md— the two copy-pasteable catalogs that belong to no row — so a 17th reusable fails until it gets a row and a phantom key in a catalog fails too.The two directions are deliberately asymmetric. Documented-but-undeclared has no allowlist and never will: every instance is a broken caller. Declared-but-undocumented is quieter — a knob nobody can discover — and 13 already exist, so they are pinned in
KNOWN_UNDOCUMENTED, modelled onKNOWN_EXEMPTincheck_workflow_pins.py:63with the same self-draining staleness rule: an entry fails once its input is no longer declared or once the guide does document it, so documenting one is "add the table row, delete the name" and a settled debt cannot linger and pre-exempt the next drift under that name.One parser note worth flagging, since it differs from the file it was ported from: table section bounds are
^#{2}\s, not^#{2,}\s, so a###sub-section is scanned as part of its parent and a table under one contributes names too. Exactly one exists today (### Choosing \approval-mode`under detect-unreviewed-merge's## Inputs) and it yields no phantom. Widening toward *more* documented names can only add a phantom failure, never hide one, so it fails safe. The examplewith:scan is not section-bounded at all — it anchors onuses:`..github/workflow-pins/README.mdgains one bullet, because its existingtests/line reads as though that directory holds only the pins-lint suite.How this was tested
Run from the repo root on
cf7b8d9. Numbers first, since the ticket predicted them: 0 phantoms across all 16 Inputs tables, 0 phantomwith:keys across the 64 example-caller keys now scanned (16 guides, 16 header comments, 2 shared catalogs), and exactly 13 declared-but-undocumented inputs — all 13 allowlisted, with no over-allowlisting (the staleness assertion is what proves the allowlist is not larger than reality). 16 rows, 16 reusables on disk, and the 3 files that mentionworkflow_callonly in prose or a heredoc are correctly excluded.Green suites:
unittest discover -s .github/workflow-pins/tests→ 458 tests, OK (32 skipped, being the two opt-in legs × 16 rows);check_workflow_pins.py→ OK, 11 workflows, 0 exempt;check_agents_md.py --root .→ passed (2 warnings, both pre-existing and untouched); the two existing suites this generalises,.github/cursor-review/testsand.github/refresh-reviewers/tests→ both OK.A green test that cannot go red is worthless, so each assertion was driven red against injected drift and then restored. The three the ticket asked for: a fake row added to a guide Inputs table; an allowlisted name deleted from
KNOWN_UNDOCUMENTED; a fake key added under a guidewith:. Four more: an allowlisted name the workflow no longer declares, and one the guide now does document (the two staleness halves, each failing with "delete it fromKNOWN_UNDOCUMENTED"); a 17th reusable dropped into.github/workflows/; and aKNOWN_UNDOCUMENTEDentry naming a workflow with no row.Self-review caught one real defect in my own first draft, worth naming because the suite was green while broken: the module-level
forloop that published the generated classes left its loop variable bound, and unittest's loader collects everyTestCasesubclass indir(module)with no_-prefix exemption — so the last row's class was reachable under a second name andstalewas silently checked twice (17 runs of each method across 16 rows). Publishing inside a function fixes it; the count is now 16, and 457 rather than 464. Same pass also madesection_linesfence-aware (a##inside a caller fence would otherwise truncate a section and turn a real check vacuous rather than red — verified it no longer does) and made a row-suppliedguideresolve against the repo root likedir_readmealready did.The two opt-in legs ship off on every row, which would make them untested dead code, so I temporarily switched them on to exercise them: with
dir_readme+readme_mode="two-way"pointed at cursor-review's## Configuration knobsandcheck_defaults=Trueon refresh-reviewers, all three reproduce what the two hand-rolled originals assert today and pass — and fail correctly on injected drift (a typo'd knob name → phantom; a deleted knob row → two-way; a changed Default cell →'99' != '12'). Reverted before commit; the shipped table has them off.Negative-claim falsification: not triggered. The diff adds no deny path, no dead-end, and no capability-denying string;
KNOWN_UNDOCUMENTEDrecords existing debt and fails when that debt is settled.Residual
The 13 undocumented inputs are recorded, not fixed. They are
agents-md-integrity:exclude_paths;groom:bail_sink,config,path,sink;pr-derisk:bot_logins,fleet_logins;pr-risk:check_name,check_run,enabled,pr_number,pr_numbers,sticky_comment. Allowlisting them is what let this land green without editing docs, which was the explicit scope; each is still a knob a caller cannot discover from its guide. Documenting one is two lines — add the row to that guide's Inputs table, delete the name fromKNOWN_UNDOCUMENTED— and the suite fails if you do only the first half.The two hand-rolled predecessors are still in place and still run.
.github/cursor-review/tests/test_workflow_inputs_docs.pyand.github/refresh-reviewers/tests/test_inputs_docs.pyare not deleted, and thedir_readme/readme_heading/readme_mode/check_defaultscolumns are off on every row, so until those two rows are switched on there is genuine duplicate coverage for those two workflows and the extra pins (the two directory-README knob tables, the Default-column comparison) exist only in the originals. The verification above shows the generalised legs reproduce them, so the fold-in is a table edit plus two deletions rather than a rewrite.Nothing here checks Default-column drift for the other 15 workflows.
check_defaultsis off everywhere, so today onlyworkflows_refdefaults are policed, bycheck_workflow_pins.py's separate cross-check. A guide that advertisesdiff_size_capas5000while the workflow says3000still passes this suite.A future reusable declaring zero inputs has no valid sentinel. The coverage test would demand a row for it, and whatever sentinel that row named would fail the anti-vacuity assertion with a message about a lost scanner rather than about the real situation. No such workflow exists today; the row model would need an explicit "no inputs" case.
Unexercised artifact. The upstream investigation notes this implementation was specified from were not readable from the build environment, so everything above was verified against the repository itself rather than against those notes. The concrete predictions they carried — the 16-workflow list, the 13-name allowlist, the heading overrides, the three prose-only exclusions, the expected all-green first run — were each re-derived from the tree and all matched.
Review round — panel findings
Ten findings from the cursor-review panel, all in this new file, all fixed in
4f80a1d. Each was reproduced against a fixture the old parser got wrong before being fixed, and the fix re-verified against the same fixture:^ \S, so a siblingworkflow_dispatch:left the input scan latched open['dispatch_only', 'real_input']; now scoped toworkflow_call:and correct in both declaration orderstext[:1] in "\"'"is a substring test, so a baredefault:hittext[0]IndexError: string index out of range; now canonicalizes to''with:scan read only one heading's fencespublic-repo-hygiene.md's tuning example and groom's environment/builder/deny-list examples were unchecked (+7 keys);README.md/docs/callers/README.mdhad no coverage at all (+3 blocks)cursor-review-auto-labelwas already passing vacuously this way; guard is now per block, each attributed via itsuses:**`foo`**and[`foo`](#foo)now register;**Note**still contributes nothingworkflow_call: {}, trailing space/comment,.yaml`scope_label` / `scope_desc`gave two names and zero Default entries; one sharedcell_input_nameson both sides now>-indicator, not the valuethemesnow reads its real text instead of'>-'; stale's and cursor-review's likewise-→_is not injective, so two rows could collide and drop a workflowfoo-bar+foo_barinstalled one class silently; now raises naming both#, splicing later blockscursor-review.yml,groom.yml,pr-size.yml; fixture showed the spliced block's key reported as a phantom inputThe rebuild in #3/#4 replaced heading-scoping with
uses:-anchoring, which removed thecaller_headingcolumn and its two special cases. A step'sactions/checkoutknobs can no longer be read as workflow inputs — on a fixture carrying both, the old scanner returned all five keys and the new one returns only the reusable's two. One subtlety that needed care: in the step form,- uses:measures two columns shallower than thewith:it governs, so the indent check must run after theuses:match.Net coverage: 64 example-caller keys across the 16 reusables (was 57), plus 3 blocks in the shared catalogs. No key that was checked before is dropped — verified by diffing the old and new scanners side by side over the real tree.
Provenance
unittest discover -s .github/workflow-pins/tests: 458 tests, OK (32 skipped);check_workflow_pins.py: OK, 11 workflows, 0 exempt;check_agents_md.py --root .: passed, 2 pre-existing warnings;unittest discover -s .github/cursor-review/tests: OK;unittest discover -s .github/refresh-reviewers/tests: OK. For the review round: each of the 10 findings reproduced on a synthetic fixture against the pre-fix parser and re-verified after, and the old and new scanners diffed side by side over the real tree to prove no previously-checked key was dropped (0 regressions, +7 per-row keys, +3 catalog blocks).^ {0,4}\S, but the scan is now scoped toworkflow_call:outright, which is also correct whenworkflow_dispatch:is declared first; findings #3/#4 asked to track keys per fence and anchor the guard on the fence carrying theuses:, and anchoring onuses:made the heading scoping redundant, socaller_headingwas removed and the scan widened to every fence plus the two shared catalogs. Finding #8 was fixed by consuming the folded body rather than by the offered alternative of skipping block-scalar inputs, so the check still covers them oncecheck_defaultsis enabled. One residual bullet ("the coverage glob is*.ymlonly") was deleted from this body because finding #6 fixed it.🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
uses:ref is SHA-pinned, and pin the last two #310