mirror of
https://github.com/Comfy-Org/ComfyUI_frontend.git
synced 2026-05-11 08:20:53 +00:00
a7218b2922
## Summary Fixes three critical issues with the CI performance reporting pipeline that made perf reports useless on PRs (demonstrated by PR #9248 — deep watcher removal merged without useful perf signal). ## Changes ### 1. Fix z-score baseline variance collection (`0/5 runs`) **Root cause:** PR #9305 added z-score statistical analysis code to `perf-report.ts`, but the historical data download step was placed in the wrong workflow file. The report is generated in `pr-perf-report.yaml` (a `workflow_run`-triggered job), but the historical download was in `ci-perf-report.yaml` (the test runner) — different runners, different filesystems. **Fix:** Implement `perf-data` orphan branch storage: - On push to main: save `perf-metrics.json` to `perf-data` branch with timestamped filename - On PR report: fetch last 5 baselines from `perf-data` branch into `temp/perf-history/` - Rolling window of 20 baselines, oldest pruned automatically - Same pattern used by `github-action-benchmark` (33.7k repos) ### 2. Fix force-push comment staleness **Root cause:** `cancel-in-progress: true` kills the perf test run before it uploads artifacts. The downstream report workflow only triggers on `conclusion == 'success'` — cancelled runs are ignored, so the comment from the first successful run goes stale. **Fix:** - Change `cancel-in-progress: false` — with GitHub's queue depth of 1, rapid pushes (A,B,C,D) run A and D, skipping B and C - Add SHA validation in `pr-perf-report.yaml` — before posting, check if the workflow_run's head SHA still matches the PR's current head. Skip posting stale results. ### 3. Add permissions for baseline operations - `contents: write` on CI job (needed for pushing to perf-data branch) - `actions: read` on both workflows (needed for artifact/baseline access) ## One-time setup required After merging, create the `perf-data` orphan branch: ```bash git checkout --orphan perf-data git rm -rf . echo '# Performance Baselines' > README.md mkdir -p baselines git add README.md baselines git commit -m 'Initialize perf-data branch' git push origin perf-data ``` The first 2 pushes to main after setup will build up variance data, and z-scores will start appearing in PR reports (threshold is `historical.length >= 2`). ## Testing - YAML validated with `yaml.safe_load()` - `perf-report.ts` `loadHistoricalReports()` already reads from `temp/perf-history/<index>/perf-metrics.json` — no code changes needed - All new steps use `continue-on-error: true` for graceful degradation ┆Issue is synchronized with this [Notion page](https://www.notion.so/PR-9886-fix-fix-perf-CI-pipeline-z-score-baselines-force-push-staleness-baseline-storage-3226d73d365081538424c7945e71f308) by [Unito](https://www.unito.io)