git checkout - uses @{-1} which requires a previous branch switch.
Since we use 'git checkout origin/main -- .' (file checkout, not branch
switch), there is no @{-1} ref. Use HEAD to restore from current branch.
Also restore proper concurrency group.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass the comprehensive test plan from .claude/skills/comfy-qa/SKILL.md
to Gemini when generating test steps. This gives Gemini knowledge of all
12 QA categories (canvas, menus, sidebar, settings, etc.) so it picks
the most relevant tests for each PR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nx build runs typecheck as a prerequisite (via @nx/vite/plugin config).
Use vite build directly for the main branch comparison build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Main branch may have transient TS errors when built with the PR
branch's lockfile. Since we only need the dist for visual comparison,
run nx build directly instead of pnpm build (which includes typecheck).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the unreliable codex exec approach with a Playwright script
(qa-record.ts) that uses Gemini to generate targeted test steps from
the PR diff, then executes them deterministically via Playwright's API.
Key changes:
- New scripts/qa-record.ts: Gemini generates JSON test actions, Playwright
executes them with reliable helper functions (menu nav, dialog fill, etc.)
- Remove codex CLI and playwright-cli dependencies
- Remove 150+ lines of prompt templates from pr-qa.yaml
- Firefox headless with video recording (same approach proven locally)
- Fallback steps if Gemini fails
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Tighten BEFORE prompt to 15s snapshot (show old state only)
- Add qa-generate-test.ts: Gemini-powered Playwright test generator
- New workflow step: generate .spec.ts and push to {branch}-add-qa-test
- Tests assert UIUX behavior (tab names, dirty state, visibility)
- Build both main (dist-before/) and PR (dist/) frontends in focused mode
- Run QA twice: BEFORE on main branch frontend, AFTER on PR branch
- Send both videos to Gemini in one request for comparative analysis
- Side-by-side dashboard layout with Before (main) / After (PR) panels
- Comparative prompt evaluates whether before confirms old behavior
and after proves the fix works
- Falls back to single-video mode when no before video available
moov atom was at end of file (8.6MB offset) — browser had to download
the entire video before seeking. Keyframes were only every 10 seconds.
Add -movflags +faststart (moov before mdat) and -g 60 (keyframe every
2.4s at 25fps) to ffmpeg conversion.
- Remove autoplay/loop so video timeline is seekable
- Skip card generation for platforms without recordings
- Add --pr-context flag to qa-video-review.ts so Gemini evaluates
against PR purpose instead of just describing what happened
- Workflow now builds pr-context.txt from PR title/body/diff
The Codex agent was spending 35s browsing the "Getting Started" template
gallery instead of testing the PR's changes. Pre-seeding this setting
via the ComfyUI API ensures the agent lands directly in the graph editor.
The Codex agent was spending time on login flow, template browsing,
and general smoke testing instead of testing the PR's actual changes.
Changes:
- Add 30-second time budget for video recording
- Move video-start AFTER login and editor verification
- Explicitly prohibit template browsing and sidebar exploration
- Reduce test steps to 3-6 targeted actions
- Restructure prompt with clear Instructions/Rules sections
Replace crude sed-based markdown conversion with client-side
rendering via marked.js CDN. Adds proper table, list, and
code styling for the report section.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the OpenAI GPT-based frame extraction approach (ffmpeg + screenshots)
with Gemini 2.5 Flash's native video understanding. This eliminates false
positives from frame-based analysis (e.g. "black screen = critical bug" during
page transitions) and produces dramatically better QA reviews.
Changes:
- Remove ffmpeg frame extraction, ffprobe duration detection, and all related
logic (~365 lines removed)
- Add @google/generative-ai SDK for native video/mp4 upload to Gemini
- Update CLI: remove --max-frames, --min-interval-seconds, --keep-frames flags
- Update env: OPENAI_API_KEY → GEMINI_API_KEY
- Update workflow: swap API key secret and model in pr-qa.yaml
- Update report: replace "Frames analyzed" with "Video size"
- Add note in prompt that brief black frames during transitions are normal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
playwright-cli doesn't support 'evaluate' command. Instead, instruct
Codex to quickly fill the username input and click Next on user-select
page BEFORE starting video recording, so the video only shows actual
QA testing.
storageState config doesn't work with playwright-cli. Instead, use
evaluate to set Comfy.userId/userName after opening the page, then
navigate back. This skips user-select before video-start so the
recording only shows actual QA testing.
Write a Playwright storageState JSON with Comfy.userId/userName pre-set
so the app loads directly to the graph editor. Saves ~40s per QA run
that was wasted on navigating the user-select page.
The convert step was using find which picked up a 0-byte file from
playwright's videos/ directory instead of the valid qa-session.webm.
Now prefers qa-session.webm explicitly and skips empty files.
Codex was using pnpm dlx instead of the global playwright-cli.
Pre-install chromium in setup step and make prompt explicit about
using the global command directly without pnpm/npx.
Replace claude --print with codex exec for cheaper QA runs.
Uses codex-mini-latest model ($1.50/$6 vs Sonnet $3/$15).
Uses existing OPENAI_API_KEY secret (no new secrets needed).
- Replace saveVideo config (didn't produce video) with explicit
playwright-cli video-start/video-stop commands in QA prompt
- Remove apt-get install ffmpeg step (pre-installed on GH runners)
- Switch video review model from gpt-4o to gpt-4.1-mini
- Enable saveVideo in playwright-cli config for real video recording
- Replace screenshot stitching with webm→mp4 conversion
- Move video review step before deploy so reports are included
- Add GPT video review reports inline on the Cloudflare Pages site
- Each video card now has expandable "GPT Video Review" section
- Set .playwright/cli.config.json with outputDir pointing to screenshots/
- This way bare 'playwright-cli screenshot' auto-saves to the right place
- Create screenshot directory before Claude runs (don't rely on Claude)
- Collect step now searches working directory for stray PNGs
- Simplified prompt: no --filename needed, just 'playwright-cli screenshot'
Screenshots were saved to artifact root but stitch looked in frames/.
Now: prompt tells Claude to save to screenshots/ dir with numbered names,
collect step consolidates PNGs there, stitch step globs from screenshots/.
Removed video-start/video-stop (Claude doesn't use them).
- Add playwright-cli config with outputDir and saveVideo
- Use video-start/video-stop instead of relying on screenshot frames
- Add fallback artifact collection from .playwright-cli/ default dir
- Simplify prompts to focus on video recording workflow
The escaped \$QA_ARTIFACTS in the heredoc produced literal text
'$QA_ARTIFACTS' in the prompt. Claude's Bash tool didn't reliably
expand this env var, so no screenshots or reports were saved.
Remove the escapes so the heredoc expands the variable to the actual
path (e.g. /home/runner/work/_temp/qa-artifacts).
Backtick-wrapped playwright-cli examples in the unquoted heredoc were
being interpreted as bash command substitution, producing empty prompts.
Replace backtick syntax with plain "Run:" prefixed commands.
- Remove all Xvfb/ffmpeg screen recording infrastructure from qa job
(captured blank display since playwright-cli runs headless)
- Add screenshot instructions to QA prompts: Claude saves sequential
frames to $QA_ARTIFACTS/frames/ after every interaction
- Stitch screenshots into video via ffmpeg in report job (2fps)
- Merge video-review job into report job (4 jobs → 3 jobs)
- Unified PR comment with video links + video review in <details> collapse
- Clean up stale QA_VIDEO_REVIEW_COMMENT markers from prior runs
Add Claude Code skills and a label-triggered QA workflow:
- .claude/skills/comfy-qa/SKILL.md: 12-category QA test plan using
playwright-cli for browser automation
- .github/workflows/pr-qa.yaml: CI workflow triggered by qa-changes
(focused, Linux) or qa-full (3-OS matrix) labels. Records screen via
ffmpeg, runs Claude CLI with playwright-cli, deploys video gallery to
Cloudflare Pages, posts PR comment with GIF thumbnails, and runs
OpenAI vision-based video review
- scripts/qa-video-review.ts: frame extraction + GPT-4o analysis
- scripts/qa-video-review.test.ts: unit tests for video review
- knip.config.ts: resolve knip errors for ingest-types package
## Summary
Restrict PyPI publishing of `comfyui-frontend-package` to bi-weekly
ComfyUI release cycles only, instead of every nightly version bump.
## Changes
- **What**: Move `publish_pypi` job from `release-draft-create.yaml` to
`release-biweekly-comfyui.yaml`
1. Removed `publish_pypi` job from `release-draft-create.yaml` (no
longer publishes on every merged Release PR)
2. Added `publish-pypi` job to `release-biweekly-comfyui.yaml` with tag
polling, build, publish, and PyPI availability confirmation
3. Gated `create-comfyui-pr` on `publish-pypi` success so the ComfyUI
requirements bump PR is only created after the package is confirmed
available
4. Updated ComfyUI PR body to confirm PyPI availability instead of
warning about a pending release PR
- **Breaking**: None — nightly releases still create GitHub releases and
publish npm types; only PyPI publishing timing changes
- **Dependencies**: None
## Review Focus
- The `publish-pypi` job uses `if: always() &&
needs.resolve-version.result == 'success'` to run even when
`trigger-release-if-needed` is skipped (tag already exists)
- Tag polling (30min timeout) waits for the version bump PR to be merged
before building from the tagged commit
- PyPI propagation polling (15min timeout) confirms the package is
installable before creating the ComfyUI PR
Fixes COM-16778
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9948-ops-restrict-PyPI-publishing-to-bi-weekly-ComfyUI-releases-3246d73d36508198b00fcc247ac5b58c)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Summary
- Adds `gh pr merge --auto --squash` after backport PR creation in the
backport workflow, so backport PRs merge automatically once checks pass
- Uses `|| echo "::warning::..."` fallback to avoid failing the workflow
if auto-merge can't be enabled (e.g. repo setting not configured)
## Test plan
- [ ] Trigger backport workflow on a test PR with `needs-backport` label
- [ ] Verify auto-merge is enabled on the created backport PR
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10108-chore-enable-auto-merge-on-backport-PRs-3256d73d3650814eb6e5fb2bdf3c5ec7)
by [Unito](https://www.unito.io)
## Summary
Remove team assignments from CODEOWNERS to reduce notification noise for
internal PRs. Add a workflow that requests team review only when
external contributors open PRs.
## Changes
- **What**: Strip `@Comfy-org/comfy_frontend_devs` and
`@Comfy-Org/comfy_maintainer` from all CODEOWNERS entries (keep
individual user assignments). Add `pr-request-team-review.yaml` workflow
that uses `pull_request_target` to request team review for
non-collaborator PRs.
- **Dependencies**: None
## Review Focus
- The workflow uses `pull_request_target` but does not check out or
execute any untrusted code — it only runs `gh pr edit --add-reviewer`.
- The `author_association` check excludes OWNER, MEMBER, and
COLLABORATOR — internal PRs will not trigger team review requests.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10104-chore-replace-team-CODEOWNERS-with-external-PR-review-workflow-3256d73d3650813b887ac16b5e97b4c4)
by [Unito](https://www.unito.io)
## Summary
Fixes three critical issues with the CI performance reporting pipeline
that made perf reports useless on PRs (demonstrated by PR #9248 — deep
watcher removal merged without useful perf signal).
## Changes
### 1. Fix z-score baseline variance collection (`0/5 runs`)
**Root cause:** PR #9305 added z-score statistical analysis code to
`perf-report.ts`, but the historical data download step was placed in
the wrong workflow file. The report is generated in
`pr-perf-report.yaml` (a `workflow_run`-triggered job), but the
historical download was in `ci-perf-report.yaml` (the test runner) —
different runners, different filesystems.
**Fix:** Implement `perf-data` orphan branch storage:
- On push to main: save `perf-metrics.json` to `perf-data` branch with
timestamped filename
- On PR report: fetch last 5 baselines from `perf-data` branch into
`temp/perf-history/`
- Rolling window of 20 baselines, oldest pruned automatically
- Same pattern used by `github-action-benchmark` (33.7k repos)
### 2. Fix force-push comment staleness
**Root cause:** `cancel-in-progress: true` kills the perf test run
before it uploads artifacts. The downstream report workflow only
triggers on `conclusion == 'success'` — cancelled runs are ignored, so
the comment from the first successful run goes stale.
**Fix:**
- Change `cancel-in-progress: false` — with GitHub's queue depth of 1,
rapid pushes (A,B,C,D) run A and D, skipping B and C
- Add SHA validation in `pr-perf-report.yaml` — before posting, check if
the workflow_run's head SHA still matches the PR's current head. Skip
posting stale results.
### 3. Add permissions for baseline operations
- `contents: write` on CI job (needed for pushing to perf-data branch)
- `actions: read` on both workflows (needed for artifact/baseline
access)
## One-time setup required
After merging, create the `perf-data` orphan branch:
```bash
git checkout --orphan perf-data
git rm -rf .
echo '# Performance Baselines' > README.md
mkdir -p baselines
git add README.md baselines
git commit -m 'Initialize perf-data branch'
git push origin perf-data
```
The first 2 pushes to main after setup will build up variance data, and
z-scores will start appearing in PR reports (threshold is
`historical.length >= 2`).
## Testing
- YAML validated with `yaml.safe_load()`
- `perf-report.ts` `loadHistoricalReports()` already reads from
`temp/perf-history/<index>/perf-metrics.json` — no code changes needed
- All new steps use `continue-on-error: true` for graceful degradation
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9886-fix-fix-perf-CI-pipeline-z-score-baselines-force-push-staleness-baseline-storage-3226d73d365081538424c7945e71f308)
by [Unito](https://www.unito.io)
## Problem
The lint/format CI workflow was broken for fork PRs in two ways:
### 1. Node version mismatch in setup-frontend action
The `setup-frontend` shared action (created in #8377) was missed when
Node version was standardized to `.nvmrc` in #9521. It still used
`node-version: 'lts/*'` instead of `node-version-file: '.nvmrc'`.
### 2. Fork PRs with lint issues silently passed CI
Fork PRs with auto-fixable lint/format issues got a **green checkmark**
despite having unfixed issues:
1. Auto-fix steps (`lint:fix`, `format`) fix issues in the workspace
2. `Commit changes` is correctly skipped for forks (can't push to fork
branches)
3. `Final validation` passes because it runs on the already-fixed
workspace
4. The `Comment on PR about manual fix needed` step tries to post a
comment via `actions/github-script`, but fork PRs have a read-only
`GITHUB_TOKEN` — the comment silently fails (`continue-on-error: true`)
5. **Result**: workflow reports success, contributor thinks their code
is clean
## Fix
- **setup-frontend**: Use `node-version-file: '.nvmrc'` instead of
`node-version: 'lts/*'`
- **ci-lint-format**: Replace the broken fork comment step with an
explicit `exit 1` that fails CI and prints clear fix instructions in the
log. This follows the principle from `.github/AGENTS.md`: fork PRs can't
post comments, so don't try.
## Testing
- [ ] Verify fork PRs with clean code still pass
- [ ] Verify fork PRs with lint issues now properly fail (instead of
silently passing)
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9846-fix-restore-fork-PR-lint-format-CI-workflow-3226d73d3650811cb5bfe9f1f989cc0c)
by [Unito](https://www.unito.io)
