[PR #111] [MERGED] feat(pr-risk): advisory PR risk grader + label-only shadow check (reusable) #124

New issue

Closed

opened 2026-10-26 22:36:29 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:36:29 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/github-workflows/pull/111
Author: @mattmillerai
Created: 8/3/2026
Status: ✅ Merged
Merged: 8/3/2026
Merged by: @mattmillerai

Base: main ← Head: matt/pr-risk-shadow-label

📝 Commits (2)

9ea1a55 feat(pr-risk): advisory PR risk grader + label-only shadow check (reusable)
41f9dc1 fix(pr-risk): close the review findings on the risk grader (BE-5507)

📊 Changes

10 files changed (+1746 additions, -0 deletions)

View changed files

➕ .github/workflows/pr-risk.yml (+334 -0)
➕ .github/workflows/test-pr-risk.yml (+60 -0)
📝 README.md (+1 -0)
➕ scripts/pr-risk/README.md (+130 -0)
➕ scripts/pr-risk/apply-risk-label.sh (+127 -0)
➕ scripts/pr-risk/grade-pr-risk.sh (+590 -0)
➕ scripts/pr-risk/risk-map.v0.json (+98 -0)
➕ scripts/pr-risk/runbook-registry.v0.json (+48 -0)
➕ scripts/pr-risk/tests/test_apply_risk_label.sh (+97 -0)
➕ scripts/pr-risk/tests/test_grade_pr_risk.sh (+261 -0)

📄 Description

Summary

New reusable workflow pr-risk.yml: grades every PR event into an advisory risk tier R0 (safest) .. R3 (riskiest) and syncs exactly one label — risk:R0..risk:R3, or risk:ungraded when an input could not be read. The label is the entire product: nothing is gated, routed, commented on, or merged. Humans glance at the label and agree or disagree; disagreement is recorded with a human-owned risk-dispute label the grader never touches.

grade = worst(path_floor, provenance, reversibility) — three deterministic axes, worst wins, so no axis can move a PR into a safer lane than another axis put it. The whole path is gh + jq; no model call anywhere.

Details

scripts/pr-risk/grade-pr-risk.sh — the grader. Three axes: a versioned path-glob map (worst tier over every matched rule, so a docs file can never cancel a migration), provenance (registered runbook producers must assert identity AND diff shape; fork / first-time contributors are R3 with no exceptions), and reversibility (persistent-state mutation, deletions under sensitive classes, whether green checks covered the lines). Unreadable input grades unknown, never a confident tier.
scripts/pr-risk/apply-risk-label.sh — the one write. Owns exactly the five mapped labels: removes stale ones, applies the computed one, touches nothing else. Missing labels are created color-coded on first use. Label text is remappable via label_map.
Defaults are generic; consumers sharpen them. .github/risk.json / .github/risk-runbooks.json in the consumer repo are read from the PR's base ref, so a PR cannot edit the rules that judge it (and touching them at all grades R3 by the map's first rule). Absent falls back to the shipped defaults; present-but-invalid fails loudly.
The grading job excludes its own run from the check rollup it reads. Its own check is always in-flight at grade time, so the raw rollup could never be green mid-run and every live grade would floor at R2 as a measurement artifact. The job recomputes the rollup from individual contexts minus its own workflow, and re-polls up to wait_for_checks_minutes (default 10) for the rest of CI to settle before labeling.
Labels ride the plain GITHUB_TOKEN, which cannot fire labeled triggers — the shadow check is structurally unable to start a workflow cascade. No secrets required.
No PR code is ever checked out: logic + defaults load from this repo at the pinned workflows_ref.
test-pr-risk.yml: shellcheck + two hermetic suites (35 checks — synthetic records plus a stubbed gh for the live-PR path, including the self-excluding-rollup cases). README catalog row + feature README (scripts/pr-risk/README.md) included.

ELI5

Some pull requests are scary (they touch billing, or database migrations, or the CI itself) and some are boring (a typo fix in a README). Today a human has to open each one to tell the difference. This workflow reads the list of changed files, who made the PR, and whether the tests passed, and puts a colored sticker on it: green for boring, red for scary, gray for "couldn't tell".

That sticker doesn't DO anything — it can't merge, block, or approve. It exists so the team can spend a couple of weeks checking whether the stickers match their own judgement. If someone thinks a sticker is wrong, they slap on a risk-dispute sticker and say why. Only after the stickers prove themselves does anyone talk about acting on them.

Two design points that matter: the rules that decide the grade can't be changed by the PR being graded (they're read from the target branch, not the PR), and the checker leaves out its own still-running check when it asks "did the tests pass?" — otherwise it would always answer "not yet" about itself.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/github-workflows/pull/111 **Author:** [@mattmillerai](https://github.com/mattmillerai) **Created:** 8/3/2026 **Status:** ✅ Merged **Merged:** 8/3/2026 **Merged by:** [@mattmillerai](https://github.com/mattmillerai) **Base:** `main` ← **Head:** `matt/pr-risk-shadow-label` --- ### 📝 Commits (2) - [`9ea1a55`](https://github.com/Comfy-Org/github-workflows/commit/9ea1a552a872c88bd3f95dc2efef47d7c70a0073) feat(pr-risk): advisory PR risk grader + label-only shadow check (reusable) - [`41f9dc1`](https://github.com/Comfy-Org/github-workflows/commit/41f9dc13e295d8da33430c68f05f48a0ca90cc15) fix(pr-risk): close the review findings on the risk grader (BE-5507) ### 📊 Changes **10 files changed** (+1746 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/pr-risk.yml` (+334 -0) ➕ `.github/workflows/test-pr-risk.yml` (+60 -0) 📝 `README.md` (+1 -0) ➕ `scripts/pr-risk/README.md` (+130 -0) ➕ `scripts/pr-risk/apply-risk-label.sh` (+127 -0) ➕ `scripts/pr-risk/grade-pr-risk.sh` (+590 -0) ➕ `scripts/pr-risk/risk-map.v0.json` (+98 -0) ➕ `scripts/pr-risk/runbook-registry.v0.json` (+48 -0) ➕ `scripts/pr-risk/tests/test_apply_risk_label.sh` (+97 -0) ➕ `scripts/pr-risk/tests/test_grade_pr_risk.sh` (+261 -0) </details> ### 📄 Description ## Summary New reusable workflow `pr-risk.yml`: grades every PR event into an advisory risk tier **R0 (safest) .. R3 (riskiest)** and syncs exactly one label — `risk:R0`..`risk:R3`, or `risk:ungraded` when an input could not be read. The label is the entire product: nothing is gated, routed, commented on, or merged. Humans glance at the label and agree or disagree; disagreement is recorded with a human-owned `risk-dispute` label the grader never touches. `grade = worst(path_floor, provenance, reversibility)` — three deterministic axes, worst wins, so no axis can move a PR into a safer lane than another axis put it. The whole path is `gh` + `jq`; no model call anywhere. ## Details - `scripts/pr-risk/grade-pr-risk.sh` — the grader. Three axes: a versioned path-glob map (worst tier over every matched rule, so a docs file can never cancel a migration), provenance (registered runbook producers must assert identity AND diff shape; fork / first-time contributors are R3 with no exceptions), and reversibility (persistent-state mutation, deletions under sensitive classes, whether green checks covered the lines). Unreadable input grades `unknown`, never a confident tier. - `scripts/pr-risk/apply-risk-label.sh` — the one write. Owns exactly the five mapped labels: removes stale ones, applies the computed one, touches nothing else. Missing labels are created color-coded on first use. Label text is remappable via `label_map`. - **Defaults are generic; consumers sharpen them.** `.github/risk.json` / `.github/risk-runbooks.json` in the consumer repo are read from the PR's **base ref**, so a PR cannot edit the rules that judge it (and touching them at all grades R3 by the map's first rule). Absent falls back to the shipped defaults; present-but-invalid fails loudly. - **The grading job excludes its own run from the check rollup it reads.** Its own check is always in-flight at grade time, so the raw rollup could never be green mid-run and every live grade would floor at R2 as a measurement artifact. The job recomputes the rollup from individual contexts minus its own workflow, and re-polls up to `wait_for_checks_minutes` (default 10) for the rest of CI to settle before labeling. - **Labels ride the plain `GITHUB_TOKEN`**, which cannot fire `labeled` triggers — the shadow check is structurally unable to start a workflow cascade. No secrets required. - No PR code is ever checked out: logic + defaults load from this repo at the pinned `workflows_ref`. - `test-pr-risk.yml`: shellcheck + two hermetic suites (35 checks — synthetic records plus a stubbed `gh` for the live-PR path, including the self-excluding-rollup cases). README catalog row + feature README (`scripts/pr-risk/README.md`) included. ## ELI5 Some pull requests are scary (they touch billing, or database migrations, or the CI itself) and some are boring (a typo fix in a README). Today a human has to open each one to tell the difference. This workflow reads the list of changed files, who made the PR, and whether the tests passed, and puts a colored sticker on it: green for boring, red for scary, gray for "couldn't tell". That sticker doesn't DO anything — it can't merge, block, or approve. It exists so the team can spend a couple of weeks checking whether the stickers match their own judgement. If someone thinks a sticker is wrong, they slap on a `risk-dispute` sticker and say why. Only after the stickers prove themselves does anyone talk about acting on them. Two design points that matter: the rules that decide the grade can't be changed by the PR being graded (they're read from the target branch, not the PR), and the checker leaves out its own still-running check when it asks "did the tests pass?" — otherwise it would always answer "not yet" about itself. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:36:29 +00:00