Compare commits

...

57 Commits

Author SHA1 Message Date
copilot-swe-agent[bot]
6b90510068 fix: address remaining shell injection and model name issues
- Move github.event.label.name to env var in Remove QA label step
- Move github.head_ref to env var RAW_BRANCH in Deploy videos step
- Update gpt-5.4 default model to gpt-4o in qa-video-review.ts

Co-authored-by: snomiao <7323030+snomiao@users.noreply.github.com>
2026-03-17 01:01:59 +00:00
copilot-swe-agent[bot]
8ced2683b5 Initial plan 2026-03-17 00:57:30 +00:00
snomiao
b7f3fdb6d2 fix: remove ephemeral QA reports from version control
CI-generated QA reports are already uploaded as GitHub Actions
artifacts and posted as PR comments. Keeping them in docs/qa/ just
creates stale files that will diverge over time.

Replace with .gitkeep so the directory remains tracked.
2026-03-17 00:47:59 +00:00
snomiao
b0ad8d3ea1 fix: update video review model from gpt-5.4 to gpt-4o
gpt-5.4 is not a valid OpenAI model name. Use gpt-4o which is a
stable, well-known model.
2026-03-17 00:47:48 +00:00
snomiao
03a6279ada fix: resolve knip errors from ingest-types package
- Add openapi-ts.config.ts as entry point so knip traces @hey-api/openapi-ts
- Remove packages/ingest-types/openapi-ts.config.ts from global ignore
- Add workspace-level ignoreBinaries for openapi-ts (provided by @hey-api/openapi-ts)
- Remove stale ignoreBinaries entry 'generate' (config hint)
- Remove stale ignoreDependencies entry '@iconify/utils' (config hint)
2026-03-16 14:18:55 +00:00
snomiao
7323fe84c0 fix: skip PR creation in CI, only commit/push report
When running in CI (CI=true), the workflow already handles posting results
as a PR comment. Instructing the agent to create a new PR would open
unwanted PRs on every automated QA run. Add a CI-specific branch that
only commits and pushes the report.
2026-03-16 14:14:35 +00:00
snomiao
e115619da7 fix: correct skills path reference in QA prompt
The comfy-qa skill lives at .claude/skills/comfy-qa/SKILL.md, not
skills/comfy-qa/SKILL.md. The incorrect path would cause Claude to
fail to load the skill during CI runs.
2026-03-16 14:14:35 +00:00
snomiao
8a61c9232c fix: prevent shell injection via github.event.label.name
Move github.event.label.name and other GitHub expression values into
step-level env: vars so they are never interpolated directly inside
the shell run: block. An attacker with triage permissions could craft
a label name with shell metacharacters to inject arbitrary commands.
2026-03-16 14:14:35 +00:00
snomiao
5b27cceef3 fix: remove push trigger on sno-skills branch
The push trigger was only for testing during development and was
annotated REMOVE BEFORE MERGE. Leaving it in would run the full QA
pipeline on every push, consuming API credits and Cloudflare deployments.
2026-03-16 14:14:35 +00:00
snomiao
d5ed5f1b6e fix: update download-artifact version comment from v7 to v7.0.0
Fixes validate-pins CI check which requires exact semver in action
version comments.
2026-03-16 14:14:35 +00:00
snomiao
85b51bd49b fix: address Copilot review feedback
- Remove duplicate skill files from skills/ (canonical is .claude/skills/)
- Add fork guard to report/video-review jobs (secrets unavailable in forks)
- Make recording steps best-effort with continue-on-error
- Upgrade download-artifact to v7 and setup-node to v6 (repo conventions)
- Use Node 24 to match package.json engines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
07777923bf feat: use qa-changes / qa-full labels to trigger QA
Renamed from claude-qa / claude-qa-full for clearer naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
c42719b803 fix: replace default X cursor with black background on Linux Xvfb
The bare Xvfb display shows a large X-shaped cursor before the browser
launches, which appears in the first seconds of the recording.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
b9ced27b38 test: trigger QA on push to sno-skills branch
REMOVE BEFORE MERGE — temporary trigger for testing the QA workflow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
5e3328f351 fix: grant macOS screen recording TCC permission before recording
Inserts kTCCServiceScreenCapture entries for ffmpeg and bash into
TCC.db before starting avfoundation recording. This prevents the
blocking permission dialog that was obstructing QA sessions on macOS.
Tries macOS 14+ schema first (17 columns), falls back to 13- (13 columns).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
8e55276571 fix: gracefully stop ffmpeg on Windows to prevent corrupt mp4
taskkill /F force-kills ffmpeg without letting it write the moov atom,
resulting in corrupt/unplayable videos. Now sends WM_CLOSE first, waits
5s for finalization, then force-kills as fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
6d3717e14a feat: use claude-qa / claude-qa-full labels to trigger QA
Replace automatic PR triggers (opened/synchronize) with label-based:
- claude-qa: focused QA (Linux-only)
- claude-qa-full: full QA (3-OS matrix)
Labels are auto-removed after the run completes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:35 +00:00
snomiao
afbf089580 fix: use branch-based Cloudflare Pages URL instead of hash
Uses sanitized PR branch name (e.g. sno-skills.comfyui-qa-videos.pages.dev)
for stable, readable URLs in PR comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
170a07f327 fix: add pnpm version to video-review job setup
The pnpm/action-setup action requires an explicit version since
package.json has no packageManager field.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
712b394d83 fix: autoplay videos on QA Cloudflare Pages gallery
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
0210d41229 fix: pin GitHub Actions to SHA hashes in pr-qa.yaml
Fixes pinact validation failure by pinning all actions to commit SHAs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
f3df5e8948 fix: add 404.html to Cloudflare Pages deploy to prevent SPA fallback
Missing files (e.g. failed GIF generation) were returning index.html
with 200 status instead of a proper 404, causing broken images in
GitHub PR comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
50a187506b feat: add QA video review script with OpenAI vision analysis
Adds a script that extracts frames from QA session recordings and sends
them to OpenAI's vision API for automated review. Includes unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
1a06b530f4 fix: improve QA video gallery page UI
Replace barebones dark page with GitHub-dark themed card layout
featuring video players, platform icons, download links, and
proper handling of missing recordings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
GitHub Action
11d438b2e3 [automated] Apply ESLint and Oxfmt fixes 2026-03-16 14:14:34 +00:00
snomiao
a19d768bf4 fix: use GIF thumbnails instead of <video> tags in QA PR comment
GitHub sanitizes <video> tags from external URLs, making the QA
comment appear empty. Generate animated GIF thumbnails from the
recording (8s clip, 480px, 8fps) and embed as clickable images
linking to the full video on Cloudflare Pages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
42bb381b91 chore: remove screenshots from repo, add .gitignore for docs/qa/screenshots
Screenshots should not be committed to the repo - they are ephemeral QA
artifacts. Add .gitignore to prevent future accidental commits.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
65e999c921 feat: pr-qa with focused/full modes, runs on every PR
Rename qa.yaml → pr-qa.yaml. Two modes:
- Focused (default, every PR): Linux-only, tests areas affected
  by PR diff, 30 max turns
- Full (qa-full label / qa-*/sno-skills branch): 3-OS matrix,
  full SKILL.md test plan, 128 max turns

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
c739ce5bd7 rename: qa-claude.yaml → qa.yaml
Not a CI gate — it's an on-demand exploratory QA tool,
so it doesn't fit the ci- prefix convention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
60e2dc6469 feat: embed QA videos in PR comment via Cloudflare Pages
Deploy screen recordings to Cloudflare Pages (comfyui-qa-videos
project) and embed inline <video> tags in the PR comment instead
of just linking to artifact downloads. Uses the same Cloudflare
credentials already configured for Playwright/Storybook deploys.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
9f29252378 fix: use named device for macOS screen capture in CI
AVFoundation device index "1:none" is invalid on macOS CI runners.
Use "Capture screen 0:none" by name for reliable screen recording.
Also list available devices for debugging.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
4bd579f34f revert: remove npx.cmd workaround, npx works natively on Windows
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
bea707005e fix: use npx.cmd on Windows for Playwright MCP server
On Windows with Git Bash, 'npx' is not resolved as a command.
Using 'npx.cmd' ensures the MCP server starts correctly,
enabling Playwright browser tools for QA testing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
GitHub Action
0defd1393d [automated] Apply ESLint and Oxfmt fixes 2026-03-16 14:14:34 +00:00
snomiao
40bf25198a fix: install ffmpeg on macOS and Windows for QA video recording
ffmpeg was not available on macOS/Windows runners, causing
"command not found" errors during screen recording steps.
Linux already installed it via the Xvfb setup step.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
bfea5a7ec7 docs: add QA report 2026-03-08 (Windows CI) 2026-03-16 14:14:34 +00:00
snomiao
1acc0f8894 fix: enable screen recording and non-headless Playwright on all platforms
Run Playwright without --headless on macOS/Windows so the browser window
is visible for screen capture. Restore avfoundation (macOS) and gdigrab
(Windows) recording steps. Remove 2>/dev/null from ffmpeg to surface
errors in CI logs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
6be36f4f52 fix: remove non-functional screen recording on macOS and Windows
Playwright runs --headless on macOS/Windows, so there is no visible
browser to record. Only Linux has a virtual display (Xvfb) that
makes screen capture meaningful. Removes dead code that silently
failed and produced no video artifacts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
GitHub Action
4f808b50f8 [automated] Apply ESLint and Oxfmt fixes 2026-03-16 14:14:34 +00:00
Anka
7b1eea5f96 docs: add QA report 2026-03-07 (macOS CI)
Automated QA run covering all frontend routes and features.
34 pass, 1 fail (404 route → download), 42 skipped (canvas/node ops require nodes).
Notable: persistent overlay bug after concurrent zoom dropdown + error dialog.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
snomiao
7d73327f0c fix: replace wait-for-it with cross-platform curl polling in setup-comfyui-server
The wait-for-it Python package uses signal.SIGALRM which is not available
on Windows, causing QA CI to fail on windows-latest runners. Replace with
a portable curl-based polling loop that works on all platforms.

Also remove the wait-for-it pip dependency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:34 +00:00
GitHub Action
0260647d03 [automated] Apply ESLint and Oxfmt fixes 2026-03-16 14:14:34 +00:00
snomiao
60eb18b97b fix: move runner.temp to step-level env for workflow_dispatch compat
The runner context is not available in job-level env blocks, which
causes workflow_dispatch validation to fail. Set QA_ARTIFACTS via
GITHUB_ENV in a step instead.
2026-03-16 14:14:33 +00:00
snomiao
3109e68877 fix: pin @anthropic-ai/claude-code to 2.1.71 for deterministic CI
Unpinned installs can drift between runs causing non-reproducible
CI behavior when new versions introduce breaking changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
eed68652ac fix: reduce QA workflow permissions to least-privilege
Remove unused permissions (issues: write, packages: read) to minimize
blast radius if the workflow or its dependencies are compromised.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
4115d5b3b6 fix: pin @playwright/mcp to 0.0.68 for reproducible CI builds
Mutable @latest dist-tag can resolve to different versions over time,
causing non-reproducible builds and unexpected breakage.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
c2806b2922 fix: prevent script injection from untrusted GitHub context values
Move github.head_ref/ref_name, pull_request.number, and github.sha
into step-level env vars to avoid shell injection from attacker-controlled
branch names. Guard PR_NUM with 'N/A' fallback for workflow_dispatch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
5c4296940e fix: make report NNN index collision-safe in comfy-qa SKILL.md
Replace fragile file-count approach with max(NNN)+1 calculation so
existing reports are never overwritten when numbering is sparse.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
6760013d2d fix: simplify matrix strategy, remove complex exclude expressions
The dynamic matrix exclude with ternary expressions caused workflow
validation failures. Simplify to always run all 3 OSes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
ee5a207407 feat: add multi-OS support to QA workflow (linux/macos/windows)
- Matrix strategy: ubuntu-latest, macos-latest, windows-latest
- workflow_dispatch input to select specific OS or run all
- Per-platform video recording:
  - Linux: Xvfb + ffmpeg x11grab (headed browser)
  - macOS: ffmpeg avfoundation
  - Windows: ffmpeg gdigrab
- Cross-platform: bash shell, runner.temp paths, curl wait loop
- Separate report job posts combined artifact links on PR
- OS name included in report filenames and artifact names

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
1e1fdaa253 feat: add video recording and artifact upload to QA workflow
- Run Playwright MCP in headed mode on Xvfb virtual display
- Record screen via ffmpeg (10fps, x264 ultrafast)
- Upload video + screenshots + report as GitHub artifacts (14-day retention)
- Post artifact download link as PR comment
- Remove PR commenting from Claude prompt (workflow handles it)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
1285049d03 docs: add QA report 2026-03-05 2026-03-16 14:14:33 +00:00
snomiao
684fbb473a fix: pipe QA prompt via file to fix stdin argument passing
The multi-line prompt wasn't being received by claude --print.
Write prompt to file first, then pipe it via cat.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
2c671fe2fd fix: use claude CLI directly instead of claude-code-action
claude-code-action requires the workflow file to exist on the default
branch (main) before it will execute. Since this is a new workflow,
switch to installing and running the claude CLI directly to bypass
the workflow validation check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
snomiao
ae55b95f5d feat: add QA workflow using Claude + Playwright MCP
- Triggers on PRs from sno-skills or qa-* branches to main
- Also triggerable via 'qa-run' label or workflow_dispatch
- Sets up frontend build + ComfyUI backend server
- Runs Claude with Playwright MCP to execute comfy-qa skill
- Claude navigates the app, tests features, generates report
- Report committed to docs/qa/ on the PR branch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
GitHub Action
122fb850fc [automated] Apply ESLint and Oxfmt fixes 2026-03-16 14:14:33 +00:00
snomiao
6bb19a4af6 feat: add skills/ folder with comfy-qa skill and symlink setup
- Create skills/ directory following Vercel Labs skills standard
- Move writing-playwright-tests from .claude/skills/ to skills/
- Symlink .claude/skills -> ../skills for Claude Code compatibility
- Add comfy-qa skill: comprehensive QA using Playwright MCP
  - Agent-agnostic (Claude/Codex/other agents)
  - Cross-platform (Windows/Mac/Linux)
  - CI and local environment support
  - Generates QA reports in docs/qa/
  - Creates draft PRs assigned to comfy-pr-bot
- Add docs/qa/ directory for QA report output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:14:33 +00:00
10 changed files with 1916 additions and 8 deletions

View File

@@ -0,0 +1,358 @@
---
name: comfy-qa
description: 'Comprehensive QA of ComfyUI frontend. Navigates all routes, tests all interactive features using Playwright MCP, generates a report, and submits a draft PR. Works in CI and local environments, cross-platform.'
---
# ComfyUI Frontend QA Skill
Perform comprehensive quality assurance of the ComfyUI frontend application by navigating all routes, clicking interactive elements, and testing features. Generate a structured report and submit it as a draft PR.
## Prerequisites
- Node.js 18+
- `pnpm` package manager
- `gh` CLI (authenticated)
- Playwright MCP server (browser automation)
## Step 1: Environment Detection & Setup
Detect the runtime environment and ensure the app is accessible.
### CI Environment
If `CI=true` is set:
1. The ComfyUI backend is pre-configured in the CI container (`ghcr.io/comfy-org/comfyui-ci-container`)
2. Frontend dist is already built and served by the backend
3. Server runs at `http://127.0.0.1:8188`
4. Skip user prompts — run fully automated
### Local Environment
If `CI` is not set:
1. **Ask the user**: "Is a ComfyUI server already running? If so, what URL? (default: http://127.0.0.1:8188)"
- If yes: use the provided URL
- If no: offer to start one:
```bash
# Option A: Use existing ComfyUI installation
# Ask for the path to ComfyUI, then:
cd <comfyui_path>
python main.py --cpu --multi-user --front-end-root <frontend_dist_path> &
# Option B: Build frontend and use preview server (no backend features)
pnpm build && pnpm preview &
```
2. Wait for server readiness by polling the URL (retry with 2s intervals, 60s timeout)
### Browser Automation Setup
Use **Playwright MCP** for browser interaction. The agent should use whichever Playwright MCP is available in its tool set:
- **Claude Code**: Uses `remote-chrome` MCP or built-in browser tools
- **Interactive mode**: User can opt for `npx @anthropic-ai/mcp-server-playwright` or `npx @playwright/mcp@latest --port 3100` for headless
- **User's real browser**: `npx @playwright/mcp@latest --extension` — lets QA happen in the user's actual browser session
Determine the server URL. For local dev servers behind proxies, adjust the URL accordingly (e.g., `https://[port].stukivx.xyz` pattern if configured).
## Step 2: QA Test Plan
Navigate to the application URL and systematically test each area below. For each test, record:
- **Status**: pass / fail / skip (with reason)
- **Notes**: any issues, unexpected behavior, or visual glitches
- **Screenshots**: take screenshots of failures or notable states
### 2.1 Application Load & Routes
| Test | Steps |
| ----------------- | ------------------------------------------------------------ |
| Root route loads | Navigate to `/` — GraphView should render with canvas |
| User select route | Navigate to `/user-select` — user selection UI should appear |
| Default redirect | If multi-user mode, `/` redirects to `/user-select` first |
| 404 handling | Navigate to `/nonexistent` — should handle gracefully |
### 2.2 Canvas & Graph View
| Test | Steps |
| ------------------------- | -------------------------------------------------------------- |
| Canvas renders | The LiteGraph canvas is visible and interactive |
| Pan canvas | Click and drag on empty canvas area |
| Zoom in/out | Use scroll wheel or Alt+=/Alt+- |
| Fit view | Press `.` key — canvas fits to content |
| Add node via double-click | Double-click canvas to open search, type "KSampler", select it |
| Add node via search | Open search box, find and add a node |
| Delete node | Select a node, press Delete key |
| Connect nodes | Drag from output slot to input slot |
| Disconnect nodes | Right-click a link and remove, or drag from connected slot |
| Multi-select | Shift+click or drag-select multiple nodes |
| Copy/Paste | Select nodes, Ctrl+C then Ctrl+V |
| Undo/Redo | Make changes, Ctrl+Z to undo, Ctrl+Y to redo |
| Node context menu | Right-click a node — menu appears with all expected options |
| Canvas context menu | Right-click empty canvas — menu appears |
### 2.3 Node Operations
| Test | Steps |
| ------------------- | ---------------------------------------------------------- |
| Bypass node | Select node, Ctrl+B — node shows bypass state |
| Mute node | Select node, Ctrl+M — node shows muted state |
| Collapse node | Select node, Alt+C — node collapses |
| Pin node | Select node, press P — node becomes pinned |
| Rename node | Double-click node title — edit mode activates |
| Node color | Right-click > Color — color picker works |
| Group nodes | Select multiple nodes, Ctrl+G — group created |
| Ungroup | Right-click group > Ungroup |
| Widget interactions | Toggle checkboxes, adjust sliders, type in text fields |
| Combo widget | Click dropdown widgets — options appear and are selectable |
### 2.4 Sidebar Tabs
| Test | Steps |
| ---------------------- | ------------------------------------------------------ |
| Workflows tab | Press W — workflows sidebar opens with saved workflows |
| Node Library tab | Press N — node library opens with categories |
| Model Library tab | Press M — model library opens |
| Assets tab | Press A — assets browser opens |
| Tab toggle | Press same key again — sidebar closes |
| Search in sidebar | Type in search box — results filter |
| Drag node from library | Drag a node from library onto canvas |
### 2.5 Topbar & Workflow Tabs
| Test | Steps |
| -------------------- | ------------------------------------------------------ |
| Workflow tab display | Current workflow name shown in tab bar |
| New workflow | Ctrl+N — new blank workflow created |
| Rename workflow | Double-click workflow tab |
| Tab context menu | Right-click workflow tab — menu with Close/Rename/etc. |
| Multiple tabs | Open multiple workflows, switch between them |
| Queue button | Click Queue/Run button — prompt queues |
| Batch count | Click batch count editor, change value |
| Menu hamburger | Click hamburger menu — options appear |
### 2.6 Settings Dialog
| Test | Steps |
| ---------------- | ---------------------------------------------------- |
| Open settings | Press Ctrl+, or click settings button |
| Settings tabs | Navigate through all setting categories |
| Change a setting | Toggle a boolean setting — it persists after closing |
| Search settings | Type in settings search box — results filter |
| Keybindings tab | Navigate to keybindings panel |
| About tab | Navigate to about panel — version info shown |
| Close settings | Press Escape or click close button |
### 2.7 Bottom Panel
| Test | Steps |
| ------------------- | -------------------------------------- |
| Toggle panel | Press Ctrl+` — bottom panel opens |
| Logs tab | Logs/terminal tab shows server output |
| Shortcuts tab | Shortcuts reference is displayed |
| Keybindings display | Press Ctrl+Shift+K — keybindings panel |
### 2.8 Execution & Queue
| Test | Steps |
| -------------- | ----------------------------------------------------- |
| Queue prompt | Load default workflow, click Queue — execution starts |
| Queue progress | Progress indicator shows during execution |
| Interrupt | Press Ctrl+Alt+Enter during execution — interrupts |
| Job history | Open job history sidebar — past executions listed |
| Clear history | Clear execution history via menu |
### 2.9 Workflow File Operations
| Test | Steps |
| --------------- | ------------------------------------------------- |
| Save workflow | Ctrl+S — workflow saves (check for prompt if new) |
| Open workflow | Ctrl+O — file picker or workflow browser opens |
| Export JSON | Menu > Export — workflow JSON downloads |
| Import workflow | Drag a .json workflow file onto canvas |
| Load default | Menu > Load Default — default workflow loads |
| Clear workflow | Menu > Clear — canvas clears (after confirmation) |
### 2.10 Advanced Features
| Test | Steps |
| --------------- | ------------------------------------------------- |
| Minimap | Alt+M — minimap toggle |
| Focus mode | Toggle focus mode |
| Canvas lock | Press H to lock, V to unlock |
| Link visibility | Ctrl+Shift+L — toggle links |
| Subgraph | Select nodes > Ctrl+Shift+E — convert to subgraph |
### 2.11 Error Handling
| Test | Steps |
| --------------------- | -------------------------------------------- |
| Missing nodes dialog | Load workflow with non-existent node types |
| Missing models dialog | Trigger missing model warning |
| Network error | Disconnect backend, verify graceful handling |
| Invalid workflow | Try loading malformed JSON |
### 2.12 Responsive & Accessibility
| Test | Steps |
| ------------------- | ------------------------------------- |
| Window resize | Resize browser window — layout adapts |
| Keyboard navigation | Tab through interactive elements |
| Sidebar resize | Drag sidebar edge to resize |
## Step 3: Generate Report
After completing all tests, generate a markdown report file.
### Report Location
```
docs/qa/YYYY-MM-DD-NNN-report.md
```
Where:
- `YYYY-MM-DD` is today's date
- `NNN` is a zero-padded increment index (001, 002, etc.)
To determine the increment, check existing files:
```bash
ls docs/qa/ | grep "$(date +%Y-%m-%d)" | wc -l
```
### Report Template
```markdown
# QA Report: ComfyUI Frontend
**Date**: YYYY-MM-DD
**Environment**: CI / Local (OS, Browser)
**Frontend Version**: (git sha or version)
**Agent**: Claude / Codex / Other
**Server URL**: http://...
## Summary
| Category | Pass | Fail | Skip | Total |
| --------------- | ---- | ---- | ---- | ----- |
| Routes & Load | | | | |
| Canvas | | | | |
| Node Operations | | | | |
| Sidebar | | | | |
| Topbar | | | | |
| Settings | | | | |
| Bottom Panel | | | | |
| Execution | | | | |
| File Operations | | | | |
| Advanced | | | | |
| Error Handling | | | | |
| Responsive | | | | |
| **Total** | | | | |
## Results
### Routes & Load
- [x] Root route loads — pass
- [ ] ...
### Canvas & Graph View
- [x] Canvas renders — pass
- [ ] ...
(repeat for each category)
## Issues Found
### Issue 1: [Title]
- **Severity**: critical / major / minor / cosmetic
- **Steps to reproduce**: ...
- **Expected**: ...
- **Actual**: ...
- **Screenshot**: (if available)
## Notes
Any additional observations, performance notes, or suggestions.
```
## Step 4: Commit and Push Report
### In CI (when `CI=true`)
Commit and push the report directly to the current branch. Do **not** create
a new PR — the CI workflow handles posting results as a PR comment.
```bash
git add docs/qa/
git commit -m "docs: add QA report $(date +%Y-%m-%d)"
git push
```
### Local / interactive use
When running locally, create a draft PR after committing:
```bash
# Ensure on a feature branch
BRANCH_NAME="qa/$(date +%Y-%m-%d)-$(git rev-parse --short HEAD)"
git checkout -b "$BRANCH_NAME" 2>/dev/null || git checkout "$BRANCH_NAME"
git add docs/qa/
git commit -m "docs: add QA report $(date +%Y-%m-%d)
Automated QA report covering all frontend routes and features."
git push -u origin "$BRANCH_NAME"
# Create draft PR assigned to comfy-pr-bot
gh pr create \
--draft \
--title "QA Report: $(date +%Y-%m-%d)" \
--body "## QA Report
Automated frontend QA run covering all routes and interactive features.
See \`docs/qa/\` for the full report.
/cc @comfy-pr-bot" \
--assignee comfy-pr-bot
```
## Execution Notes
### Cross-Platform Considerations
- **Windows**: Use `pwsh` or `cmd` equivalents for shell commands. `gh` CLI works on all platforms.
- **macOS**: Keyboard shortcuts use Cmd instead of Ctrl in the actual app, but Playwright sends OS-appropriate keys.
- **Linux**: Primary CI platform. Screenshot baselines are Linux-only.
### Agent Compatibility
This skill is written to be agent-agnostic:
- **Claude Code**: Has native Playwright MCP via `remote-chrome` or configured MCP servers
- **Codex**: Can use `npx @playwright/mcp@latest` for browser automation
- **Other agents**: Any agent with browser automation tools can follow these instructions
The key requirement is the ability to:
1. Navigate to URLs
2. Take snapshots/screenshots
3. Click elements
4. Type text
5. Press keyboard shortcuts
6. Read page content
### Tips for Reliable QA
1. **Wait for page stability** before interacting — check that elements are visible and enabled
2. **Take a snapshot after each major navigation** to verify state
3. **Don't use fixed timeouts** — poll for expected conditions
4. **Record the full page snapshot** at the start for baseline comparison
5. **If a test fails**, document it and continue — don't abort the entire QA run
6. **Group related tests** — complete one category before moving to the next

View File

@@ -44,7 +44,6 @@ runs:
python -m pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
pip install wait-for-it
- name: Start ComfyUI server
if: ${{ inputs.launch_server == 'true' }}
@@ -52,4 +51,8 @@ runs:
working-directory: ComfyUI
run: |
python main.py --cpu --multi-user --front-end-root ../dist ${{ inputs.extra_server_params }} &
wait-for-it --service 127.0.0.1:8188 -t 600
for i in $(seq 1 300); do
curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1 && echo "Server ready" && exit 0
sleep 2
done
echo "::error::ComfyUI server did not start within 600s" && exit 1

564
.github/workflows/pr-qa.yaml vendored Normal file
View File

@@ -0,0 +1,564 @@
# Automated QA of ComfyUI frontend using Claude CLI + Playwright MCP.
# Two modes:
# Focused (qa-changes label): Linux-only, tests areas affected by PR changes
# Full (qa-full label): 3-OS matrix, full test plan
name: 'PR: QA'
on:
pull_request:
types: [labeled]
branches: [main]
workflow_dispatch:
inputs:
mode:
description: 'QA mode'
type: choice
options: [focused, full]
default: focused
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
resolve-matrix:
runs-on: ubuntu-latest
outputs:
os: ${{ steps.set.outputs.os }}
mode: ${{ steps.set.outputs.mode }}
skip: ${{ steps.set.outputs.skip }}
steps:
- name: Determine QA mode
id: set
env:
LABEL: ${{ github.event.label.name }}
EVENT_ACTION: ${{ github.event.action }}
EVENT_NAME: ${{ github.event_name }}
INPUT_MODE: ${{ inputs.mode }}
run: |
FULL=false
# Only run on label events if it's one of our labels
if [ "$EVENT_ACTION" = "labeled" ] && \
[ "$LABEL" != "qa-changes" ] && [ "$LABEL" != "qa-full" ]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
fi
# Full QA triggers
if [ "$EVENT_NAME" = "workflow_dispatch" ] && \
[ "$INPUT_MODE" = "full" ]; then
FULL=true
fi
if [ "$LABEL" = "qa-full" ]; then
FULL=true
fi
if [ "$FULL" = "true" ]; then
echo 'os=["ubuntu-latest","macos-latest","windows-latest"]' >> "$GITHUB_OUTPUT"
echo "mode=full" >> "$GITHUB_OUTPUT"
else
echo 'os=["ubuntu-latest"]' >> "$GITHUB_OUTPUT"
echo "mode=focused" >> "$GITHUB_OUTPUT"
fi
echo "Mode: $([ "$FULL" = "true" ] && echo full || echo focused)"
qa:
needs: resolve-matrix
if: needs.resolve-matrix.outputs.skip != 'true'
strategy:
fail-fast: false
matrix:
os: ${{ fromJson(needs.resolve-matrix.outputs.os) }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
permissions:
contents: write
pull-requests: write
env:
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
steps:
- name: Set QA artifacts path
shell: bash
run: echo "QA_ARTIFACTS=$RUNNER_TEMP/qa-artifacts" >> "$GITHUB_ENV"
- name: Checkout repository
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
fetch-depth: 0
ref: ${{ github.head_ref || github.ref }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup frontend
uses: ./.github/actions/setup-frontend
with:
include_build_step: true
- name: Setup and start ComfyUI server
uses: ./.github/actions/setup-comfyui-server
with:
launch_server: 'true'
- name: Wait for ComfyUI server
shell: bash
run: |
echo "Waiting for ComfyUI server..."
for i in $(seq 1 60); do
if curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1; then
echo "Server ready"; exit 0
fi; sleep 2
done
echo "::error::Server timeout"; exit 1
- name: Install Playwright for MCP
shell: bash
run: pnpm exec playwright install chromium --with-deps
- name: Install Claude Code
shell: bash
run: npm install -g @anthropic-ai/claude-code@2.1.71
# --- Virtual display + recording: Linux ---
- name: Setup Xvfb (Linux)
if: runner.os == 'Linux'
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq xvfb ffmpeg x11-xserver-utils >/dev/null 2>&1
Xvfb :99 -screen 0 1280x720x24 -ac -nolisten tcp &
echo $! > "${{ runner.temp }}/xvfb.pid"
sleep 2
echo "DISPLAY=:99" >> "$GITHUB_ENV"
# Set black background and normal cursor (removes the default X cursor)
DISPLAY=:99 xsetroot -solid black -cursor_name left_ptr 2>/dev/null || true
- name: Start recording (Linux)
if: runner.os == 'Linux'
continue-on-error: true
run: |
mkdir -p "$QA_ARTIFACTS"
ffmpeg -y -f x11grab -video_size 1280x720 -framerate 10 \
-i :99.0 -c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 1
# --- Recording: macOS (avfoundation) ---
- name: Install ffmpeg (macOS)
if: runner.os == 'macOS'
run: brew install ffmpeg
- name: Grant screen recording permission (macOS)
if: runner.os == 'macOS'
run: |
# Grant kTCCServiceScreenCapture to ffmpeg and bash so avfoundation
# recording works without the blocking permission dialog.
FFMPEG_PATH=$(which ffmpeg)
echo "Granting screen recording permission to: $FFMPEG_PATH"
# macOS 14+ (Sonoma/Sequoia) TCC.db schema has 17 columns
for BIN in "$FFMPEG_PATH" "/bin/bash"; do
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,0,'UNUSED',NULL,0,$(date +%s),NULL,NULL,NULL,NULL);" 2>/dev/null \
|| sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,NULL,'UNUSED',NULL,0,$(date +%s));" 2>/dev/null \
|| echo "Warning: Could not grant TCC permission to ${BIN}"
done
- name: Start recording (macOS)
if: runner.os == 'macOS'
continue-on-error: true
run: |
mkdir -p "$QA_ARTIFACTS"
ffmpeg -f avfoundation -list_devices true -i "" 2>&1 || true
ffmpeg -y -f avfoundation -framerate 10 -capture_cursor 1 \
-i "Capture screen 0:none" -c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 2
# --- Recording: Windows (gdigrab) ---
- name: Install ffmpeg (Windows)
if: runner.os == 'Windows'
shell: pwsh
run: choco install ffmpeg -y --no-progress
- name: Start recording (Windows)
if: runner.os == 'Windows'
continue-on-error: true
shell: bash
run: |
mkdir -p "$QA_ARTIFACTS"
ffmpeg -y -f gdigrab -framerate 10 -i desktop \
-c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 1
- name: Create MCP config
shell: bash
run: |
cat > "${{ runner.temp }}/mcp-config.json" <<EOF
{"mcpServers":{"playwright":{"command":"npx","args":["@playwright/mcp@0.0.68"]}}}
EOF
- name: Get PR diff for focused QA
if: needs.resolve-matrix.outputs.mode == 'focused'
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr diff ${{ github.event.pull_request.number || '' }} \
--repo ${{ github.repository }} > "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || \
git diff origin/main...HEAD > "${{ runner.temp }}/pr-diff.txt"
# Summarize changed files for the prompt
echo "Changed files:"
grep '^diff --git' "${{ runner.temp }}/pr-diff.txt" | \
sed 's|diff --git a/||;s| b/.*||' | sort -u | tee "${{ runner.temp }}/changed-files.txt"
- name: Write QA prompt
shell: bash
env:
BRANCH: ${{ github.head_ref || github.ref_name }}
PR_NUM: ${{ github.event.pull_request.number || 'N/A' }}
SHA: ${{ github.sha }}
run: |
OS_LOWER=$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')
if [ "$QA_MODE" = "full" ]; then
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
You are running a FULL automated QA pass on the ComfyUI frontend.
Read the file .claude/skills/comfy-qa/SKILL.md and follow the FULL QA test plan.
Environment: CI=true, OS=${{ runner.os }}
Server URL: http://127.0.0.1:8188
Branch: ${BRANCH}
PR: #${PR_NUM}
Commit: ${SHA}
1. Use playwright MCP tools to navigate http://127.0.0.1:8188
2. Run the FULL QA test plan from the skill file
3. Take screenshots of failures or notable states
4. Save report to docs/qa/ as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
5. Commit and push the report to this branch
Do NOT create a new PR. Do NOT post PR comments.
Skip tests not available in CI (file dialogs, GPU execution).
PROMPT
else
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
You are running a FOCUSED QA pass on a pull request to the ComfyUI frontend.
Your goal is to verify that the changes in this PR work correctly and don't break related functionality.
Environment: CI=true, OS=${{ runner.os }}
Server URL: http://127.0.0.1:8188
Branch: ${BRANCH}
PR: #${PR_NUM}
Commit: ${SHA}
CHANGED FILES:
$(cat "${{ runner.temp }}/changed-files.txt" 2>/dev/null || echo "Unknown")
DIFF (truncated to 500 lines):
$(head -500 "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || echo "No diff available")
Instructions:
1. Read the diff above to understand what changed in this PR
2. Use playwright MCP tools to navigate http://127.0.0.1:8188
3. Test the specific UI areas affected by these changes
4. Also do a quick smoke test of core functionality (app loads, canvas renders, sidebar works)
5. Take screenshots of any failures or the areas you tested
6. Save a concise report to docs/qa/ as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
7. Commit and push the report to this branch
Focus on:
- Does the changed functionality work as expected?
- Are there visual regressions in affected areas?
- Do related features still work?
Do NOT run the full QA test plan. Do NOT create a new PR. Do NOT post PR comments.
Skip tests not available in CI (file dialogs, GPU execution).
PROMPT
fi
- name: Run Claude QA
shell: bash
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
CI: 'true'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
MAX_TURNS=128
if [ "$QA_MODE" = "focused" ]; then MAX_TURNS=30; fi
cat "${{ runner.temp }}/qa-prompt.txt" | claude --print --verbose \
--max-turns "$MAX_TURNS" \
--mcp-config "${{ runner.temp }}/mcp-config.json" \
--allowedTools "mcp__playwright__browser_navigate,mcp__playwright__browser_snapshot,mcp__playwright__browser_click,mcp__playwright__browser_type,mcp__playwright__browser_press_key,mcp__playwright__browser_take_screenshot,mcp__playwright__browser_hover,mcp__playwright__browser_drag,mcp__playwright__browser_select_option,mcp__playwright__browser_handle_dialog,mcp__playwright__browser_tab_list,mcp__playwright__browser_tab_new,mcp__playwright__browser_tab_select,mcp__playwright__browser_tab_close,mcp__playwright__browser_console_messages,mcp__playwright__browser_resize,mcp__playwright__browser_wait_for,Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git status:*),Bash(git log:*),Bash(git diff:*),Bash(date:*),Bash(ls:*),Bash(mkdir:*),Read,Write,Edit,Glob,Grep"
- name: Stop recording
if: always()
shell: bash
run: |
PID_FILE="${{ runner.temp }}/ffmpeg.pid"
if [ -f "$PID_FILE" ]; then
if [ "$RUNNER_OS" = "Windows" ]; then
# Graceful stop: taskkill without /F sends WM_CLOSE so ffmpeg
# can finalize the mp4 (write moov atom). Force-kill as fallback.
taskkill //PID $(cat "$PID_FILE") 2>/dev/null || true
sleep 5
taskkill //F //PID $(cat "$PID_FILE") 2>/dev/null || true
else
kill -INT $(cat "$PID_FILE") 2>/dev/null || true
sleep 3; kill $(cat "$PID_FILE") 2>/dev/null || true
fi
fi
[ -f "$QA_ARTIFACTS/qa-session.mp4" ] && \
echo "Video: $(du -h "$QA_ARTIFACTS/qa-session.mp4" | cut -f1)" || \
echo "No video (non-fatal)"
- name: Collect artifacts
if: always()
shell: bash
run: |
mkdir -p "$QA_ARTIFACTS"
cp -r docs/qa/* "$QA_ARTIFACTS/" 2>/dev/null || true
ls -la "$QA_ARTIFACTS/" || true
- name: Upload QA artifacts
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v6.2.0
with:
name: qa-report-${{ runner.os }}-${{ github.run_id }}
path: ${{ env.QA_ARTIFACTS }}/
retention-days: 14
- name: Cleanup (Linux)
if: always() && runner.os == 'Linux'
run: kill $(cat "${{ runner.temp }}/xvfb.pid") 2>/dev/null || true
report:
needs: [resolve-matrix, qa]
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Download QA artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
path: qa-artifacts
pattern: qa-report-*
- name: Install ffmpeg
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
- name: Deploy videos to Cloudflare Pages
id: deploy-videos
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
RAW_BRANCH: ${{ github.head_ref || github.ref_name }}
run: |
npm install -g wrangler@^4.0.0 >/dev/null 2>&1
DEPLOY_DIR=$(mktemp -d)
mkdir -p "$DEPLOY_DIR"
for os in Linux macOS Windows; do
VID="qa-artifacts/qa-report-${os}-${{ github.run_id }}/qa-session.mp4"
if [ -f "$VID" ]; then
cp "$VID" "$DEPLOY_DIR/qa-${os}.mp4"
echo "Found ${os} video ($(du -h "$VID" | cut -f1))"
# Generate GIF thumbnail: 8s starting at 10s, 480px wide, 8fps
ffmpeg -y -ss 10 -i "$VID" -t 8 \
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|| ffmpeg -y -i "$VID" -t 8 \
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|| echo "GIF generation failed for ${os} (non-fatal)"
fi
done
# Build video cards HTML
CARDS=""
ICONS_Linux="&#x1F427;" ICONS_macOS="&#x1F34E;" ICONS_Windows="&#x1FA9F;"
for os in Linux macOS Windows; do
eval "ICON=\$ICONS_${os}"
if [ -f "$DEPLOY_DIR/qa-${os}.mp4" ]; then
CARDS="${CARDS}<div class=card><video controls autoplay muted loop preload=metadata><source src=qa-${os}.mp4 type=video/mp4></video><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><a class=download href=qa-${os}.mp4 download>Download</a></div></div>"
else
CARDS="${CARDS}<div class=card><div class=empty-card>No recording available</div><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><span class='badge missing'>Missing</span></div></div>"
fi
done
cat > "$DEPLOY_DIR/index.html" <<INDEXEOF
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><title>QA Session Recordings</title>
<style>
*{margin:0;padding:0;box-sizing:border-box}body{background:#0d1117;color:#e6edf3;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;min-height:100vh;padding:2rem 1rem}.container{max-width:1200px;margin:0 auto}header{display:flex;align-items:center;gap:.75rem;margin-bottom:2rem;padding-bottom:1rem;border-bottom:1px solid #30363d}h1{font-size:1.5rem;font-weight:600}.meta{color:#8b949e;font-size:.875rem;margin-top:.25rem}.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(340px,1fr));gap:1.25rem}.card{background:#161b22;border:1px solid #30363d;border-radius:.5rem;overflow:hidden;transition:border-color .15s}.card:hover{border-color:#58a6ff}.card video{width:100%;display:block;background:#010409;aspect-ratio:16/9;object-fit:contain}.card-body{padding:.75rem 1rem;display:flex;align-items:center;justify-content:space-between}.platform{display:flex;align-items:center;gap:.5rem;font-weight:500}.icon{font-size:1.25rem}.badge{font-size:.75rem;padding:.125rem .5rem;border-radius:999px;background:#1f6feb33;color:#58a6ff;border:1px solid #1f6feb55}.badge.missing{background:#da363333;color:#f85149;border-color:#da363355}.empty-card{display:flex;align-items:center;justify-content:center;min-height:200px;color:#484f58;font-size:.875rem}a.download{color:#58a6ff;text-decoration:none;font-size:.8125rem}a.download:hover{text-decoration:underline}
</style></head><body><div class=container>
<header><svg width=28 height=28 viewBox="0 0 24 24" fill=none stroke=#58a6ff stroke-width=2 stroke-linecap=round stroke-linejoin=round><polygon points="23 7 16 12 23 17 23 7"/><rect x=1 y=5 width=15 height=14 rx=2 ry=2/></svg><div><h1>QA Session Recordings</h1><div class=meta>ComfyUI Frontend &middot; Automated QA</div></div></header>
<div class=grid>${CARDS}</div>
</div></body></html>
INDEXEOF
# 404 page so Cloudflare Pages returns proper 404 for missing files
# (instead of SPA fallback serving index.html)
cat > "$DEPLOY_DIR/404.html" <<'ERROREOF'
<!DOCTYPE html><html><head><meta charset=utf-8><title>404</title>
<style>body{background:#0d1117;color:#8b949e;font-family:sans-serif;display:flex;align-items:center;justify-content:center;min-height:100vh;margin:0}div{text-align:center}h1{color:#f85149;font-size:3rem;margin-bottom:.5rem}p{font-size:1rem}</style>
</head><body><div><h1>404</h1><p>File not found. The QA recording may have failed or been cancelled.</p></div></body></html>
ERROREOF
# Sanitize branch name for Cloudflare Pages URL (same rules CF uses)
BRANCH=$(echo "$RAW_BRANCH" | sed 's/[^a-zA-Z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | cut -c1-28)
URL=$(wrangler pages deploy "$DEPLOY_DIR" \
--project-name="comfyui-qa-videos" \
--branch="$BRANCH" 2>&1 \
| grep -oE 'https://[a-zA-Z0-9.-]+\.pages\.dev\S*' | head -1)
echo "url=${URL:-https://${BRANCH}.comfyui-qa-videos.pages.dev}" >> "$GITHUB_OUTPUT"
echo "Deployed to: ${URL}"
- name: Post QA comment on PR
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VIDEO_BASE: ${{ steps.deploy-videos.outputs.url }}
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
run: |
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
COMMENT_MARKER="<!-- QA_REPORT_COMMENT -->"
MODE_BADGE="🔍 Focused"
if [ "$QA_MODE" = "full" ]; then MODE_BADGE="🔬 Full (3-OS)"; fi
# Build video section with GIF thumbnails linking to full videos
VIDEO_SECTION=""
for os in Linux macOS Windows; do
GIF_URL="${VIDEO_BASE}/qa-${os}-thumb.gif"
VID_URL="${VIDEO_BASE}/qa-${os}.mp4"
if curl -sf --head "$VID_URL" >/dev/null 2>&1; then
if curl -sf --head "$GIF_URL" >/dev/null 2>&1; then
VIDEO_SECTION="${VIDEO_SECTION}[![${os} QA](${GIF_URL})](${VID_URL})"$'\n'
else
VIDEO_SECTION="${VIDEO_SECTION}[${os} video](${VID_URL})"$'\n'
fi
fi
done
BODY=$(cat <<EOF
${COMMENT_MARKER}
## QA ${MODE_BADGE}
${VIDEO_SECTION}
**Run**: [${RUN}](${RUN}) · [Download artifacts](${RUN}#artifacts) · [All videos](${VIDEO_BASE})
EOF
)
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
--jq ".[] | select(.body | contains(\"${COMMENT_MARKER}\")) | .id" | head -1)
if [ -n "$EXISTING" ]; then
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
--field body="$BODY"
else
gh pr comment ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} --body "$BODY"
fi
- name: Remove QA label
if: >-
github.event.label.name == 'qa-changes' ||
github.event.label.name == 'qa-full'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LABEL: ${{ github.event.label.name }}
run: |
gh pr edit ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} --remove-label "$LABEL"
video-review:
needs: [qa, report]
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Setup Node.js
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
with:
node-version: 24
- name: Install pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
with:
version: 10
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Install ffmpeg
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
- name: Download QA artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
path: qa-artifacts
pattern: qa-report-*
- name: Run video review
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
mkdir -p video-reviews
for vid in qa-artifacts/qa-report-*/qa-session.mp4; do
[ -f "$vid" ] || continue
echo "::group::Reviewing $vid"
pnpm exec tsx scripts/qa-video-review.ts \
--artifacts-dir qa-artifacts \
--output-dir video-reviews \
--video-file "$vid" \
--model gpt-4o || true
echo "::endgroup::"
done
- name: Post video review comment
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
MARKER="<!-- QA_VIDEO_REVIEW_COMMENT -->"
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
# Combine per-platform reports
REPORT=""
for f in video-reviews/*-qa-video-report.md; do
[ -f "$f" ] || continue
[ -n "$REPORT" ] && REPORT="${REPORT}
---
"
REPORT="${REPORT}$(cat "$f")"
done
[ -z "$REPORT" ] && REPORT="No video reports were generated."
BODY="${MARKER}
## QA Video Report
${REPORT}
---
**Run**: [${RUN}](${RUN})"
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
--jq ".[] | select(.body | contains(\"${MARKER}\")) | .id" | head -1)
if [ -n "$EXISTING" ]; then
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
--field body="$BODY"
else
gh pr comment ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} --body "$BODY"
fi

0
docs/qa/.gitkeep Normal file
View File

5
docs/qa/screenshots/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
*.png
*.jpg
*.jpeg
*.webp
*.gif

View File

View File

@@ -29,10 +29,12 @@ const config: KnipConfig = {
},
'packages/ingest-types': {
project: ['src/**/*.{js,ts}'],
entry: ['src/index.ts']
entry: ['src/index.ts', 'openapi-ts.config.ts'],
// openapi-ts binary is provided by @hey-api/openapi-ts devDependency
ignoreBinaries: ['openapi-ts']
}
},
ignoreBinaries: ['python3', 'gh', 'generate'],
ignoreBinaries: ['python3', 'gh'],
ignoreDependencies: [
// Weird importmap things
'@iconify-json/lucide',
@@ -40,9 +42,7 @@ const config: KnipConfig = {
'@primeuix/forms',
'@primeuix/styled',
'@primeuix/utils',
'@primevue/icons',
// Used by lucideStrokePlugin.js (CSS @plugin)
'@iconify/utils'
'@primevue/icons'
],
ignore: [
// Auto generated API types
@@ -50,7 +50,6 @@ const config: KnipConfig = {
'packages/registry-types/src/comfyRegistryTypes.ts',
'packages/ingest-types/src/types.gen.ts',
'packages/ingest-types/src/zod.gen.ts',
'packages/ingest-types/openapi-ts.config.ts',
// Used by a custom node (that should move off of this)
'src/scripts/ui/components/splitButton.ts',
// Used by stacked PR (feat/glsl-live-preview)

View File

@@ -39,6 +39,7 @@
"preinstall": "pnpm dlx only-allow pnpm",
"prepare": "husky || true && git config blame.ignoreRevsFile .git-blame-ignore-revs || true",
"preview": "nx preview",
"qa:video-review": "tsx scripts/qa-video-review.ts",
"storybook": "nx storybook",
"storybook:desktop": "nx run @comfyorg/desktop-ui:storybook",
"stylelint:fix": "stylelint --cache --fix '{apps,packages,src}/**/*.{css,vue}'",

View File

@@ -0,0 +1,183 @@
import { describe, expect, it } from 'vitest'
import {
extractOutputText,
extractPlatformFromArtifactDirName,
pickLatestVideosByPlatform,
selectVideoCandidateByFile
} from './qa-video-review'
describe('extractPlatformFromArtifactDirName', () => {
it('extracts and normalizes known qa artifact directory names', () => {
expect(
extractPlatformFromArtifactDirName('qa-report-Windows-22818315023')
).toBe('windows')
expect(
extractPlatformFromArtifactDirName('qa-report-macOS-22818315023')
).toBe('macos')
expect(
extractPlatformFromArtifactDirName('qa-report-Linux-22818315023')
).toBe('linux')
})
it('falls back to slugifying unknown directory names', () => {
expect(extractPlatformFromArtifactDirName('custom platform run')).toBe(
'custom-platform-run'
)
})
})
describe('pickLatestVideosByPlatform', () => {
it('keeps only the latest candidate per platform', () => {
const selected = pickLatestVideosByPlatform([
{
platformName: 'windows',
videoPath: '/tmp/windows-old.mp4',
mtimeMs: 100
},
{
platformName: 'windows',
videoPath: '/tmp/windows-new.mp4',
mtimeMs: 200
},
{
platformName: 'linux',
videoPath: '/tmp/linux.mp4',
mtimeMs: 150
}
])
expect(selected).toEqual([
{
platformName: 'linux',
videoPath: '/tmp/linux.mp4',
mtimeMs: 150
},
{
platformName: 'windows',
videoPath: '/tmp/windows-new.mp4',
mtimeMs: 200
}
])
})
})
describe('extractOutputText', () => {
it('reads output_text when available', () => {
expect(extractOutputText({ output_text: 'hello world' })).toBe(
'hello world'
)
})
it('reads nested output text chunks', () => {
const responseBody = {
output: [
{
content: [
{ type: 'output_text', text: 'first block' },
{ type: 'refusal', refusal: 'not relevant' }
]
},
{
content: [{ type: 'output_text', text: 'second block' }]
}
]
}
expect(extractOutputText(responseBody)).toBe('first block\n\nsecond block')
})
it('returns null when there is no usable text', () => {
expect(
extractOutputText({ output: [{ content: [{ type: 'tool_call' }] }] })
).toBeNull()
})
})
describe('selectVideoCandidateByFile', () => {
it('selects a single candidate by artifacts-relative path', () => {
const selected = selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
},
{
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-report-Linux-1/qa-session.mp4'
}
)
expect(selected).toEqual({
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
})
})
it('throws when basename matches multiple videos', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
},
{
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-session.mp4'
}
)
).toThrow('matched 2 videos')
})
it('throws when there is no matching video', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-report-macOS-1/qa-session.mp4'
}
)
).toThrow('No video matched')
})
it('throws when video file is missing', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: ' '
}
)
).toThrow('--video-file is required')
})
})

795
scripts/qa-video-review.ts Normal file
View File

@@ -0,0 +1,795 @@
#!/usr/bin/env tsx
import { spawn } from 'node:child_process'
import {
mkdtemp,
mkdir,
readdir,
readFile,
rm,
stat,
writeFile
} from 'node:fs/promises'
import { basename, dirname, join, relative, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { globSync } from 'glob'
interface CliOptions {
artifactsDir: string
videoFile: string
outputDir: string
model: string
minIntervalSeconds: number
maxFrames: number
requestTimeoutMs: number
dryRun: boolean
keepFrames: boolean
}
interface VideoCandidate {
platformName: string
videoPath: string
mtimeMs: number
}
interface ExtractedFrame {
index: number
timestampSeconds: number
filePath: string
dataUrl: string
}
type ResponseInputContent =
| { type: 'input_text'; text: string }
| { type: 'input_image'; image_url: string }
interface ResponseInputMessage {
role: 'system' | 'user'
content: ResponseInputContent[]
}
interface ResponsesCreatePayload {
model: string
input: ResponseInputMessage[]
max_output_tokens: number
}
interface OpenAIReviewRequest {
apiKey: string
baseUrl: string
model: string
platformName: string
videoPath: string
frames: ExtractedFrame[]
samplingIntervalSeconds: number
timeoutMs: number
}
const DEFAULT_OPTIONS: CliOptions = {
artifactsDir: './tmp/qa-artifacts',
videoFile: '',
outputDir: './tmp',
model: 'gpt-4o',
minIntervalSeconds: 5,
maxFrames: 36,
requestTimeoutMs: 300_000,
dryRun: false,
keepFrames: false
}
const USAGE = `Usage:
pnpm exec tsx scripts/qa-video-review.ts [options]
Options:
--artifacts-dir <path> Artifacts root directory
(default: ./tmp/qa-artifacts)
--video-file <name-or-path> Video file to analyze (required)
(supports basename or relative/absolute path)
--output-dir <path> Output directory for markdown reports
(default: ./tmp)
--model <name> OpenAI model
(default: gpt-4o)
--min-interval-seconds <n> Minimum frame sampling interval in seconds
(default: 5)
--max-frames <n> Max frames analyzed per video
(default: 36)
--request-timeout-ms <n> Request timeout in milliseconds
(default: 300000)
--dry-run Discover videos and output targets only
--keep-frames Keep extracted frames in ./tmp for inspection
--help Show this help text
Environment:
OPENAI_API_KEY Required unless --dry-run
OPENAI_BASE_URL Optional override for API base URL
`
function parsePositiveNumber(rawValue: string, flagName: string): number {
const parsedValue = Number(rawValue)
if (!Number.isFinite(parsedValue) || parsedValue <= 0) {
throw new Error(`Invalid value for ${flagName}: "${rawValue}"`)
}
return parsedValue
}
function parsePositiveInteger(rawValue: string, flagName: string): number {
const parsedValue = Number.parseInt(rawValue, 10)
if (!Number.isInteger(parsedValue) || parsedValue <= 0) {
throw new Error(`Invalid value for ${flagName}: "${rawValue}"`)
}
return parsedValue
}
function parseCliOptions(args: string[]): CliOptions {
const options: CliOptions = { ...DEFAULT_OPTIONS }
for (let index = 0; index < args.length; index += 1) {
const argument = args[index]
const nextValue = args[index + 1]
const requireValue = (flagName: string): string => {
if (!nextValue || nextValue.startsWith('--')) {
throw new Error(`Missing value for ${flagName}`)
}
index += 1
return nextValue
}
if (argument === '--help') {
process.stdout.write(USAGE)
process.exit(0)
}
if (argument === '--artifacts-dir') {
options.artifactsDir = requireValue(argument)
continue
}
if (argument === '--video-file') {
options.videoFile = requireValue(argument)
continue
}
if (argument === '--output-dir') {
options.outputDir = requireValue(argument)
continue
}
if (argument === '--model') {
options.model = requireValue(argument)
continue
}
if (argument === '--min-interval-seconds') {
options.minIntervalSeconds = parsePositiveNumber(
requireValue(argument),
argument
)
continue
}
if (argument === '--max-frames') {
options.maxFrames = parsePositiveInteger(requireValue(argument), argument)
continue
}
if (argument === '--request-timeout-ms') {
options.requestTimeoutMs = parsePositiveInteger(
requireValue(argument),
argument
)
continue
}
if (argument === '--dry-run') {
options.dryRun = true
continue
}
if (argument === '--keep-frames') {
options.keepFrames = true
continue
}
throw new Error(`Unknown argument: ${argument}`)
}
return options
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null
}
function normalizePlatformName(value: string): string {
const slug = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-+|-+$/g, '')
return slug.length > 0 ? slug : 'unknown-platform'
}
export function extractPlatformFromArtifactDirName(dirName: string): string {
const matchedValue = dirName.match(/^qa-report-(.+?)(?:-\d+)?$/i)?.[1]
return normalizePlatformName(matchedValue ?? dirName)
}
function extractPlatformFromVideoPath(videoPath: string): string {
const artifactDirName = basename(dirname(videoPath))
return extractPlatformFromArtifactDirName(artifactDirName)
}
export function pickLatestVideosByPlatform(
candidates: VideoCandidate[]
): VideoCandidate[] {
const latestByPlatform = new Map<string, VideoCandidate>()
for (const candidate of candidates) {
const current = latestByPlatform.get(candidate.platformName)
if (!current || candidate.mtimeMs > current.mtimeMs) {
latestByPlatform.set(candidate.platformName, candidate)
}
}
return [...latestByPlatform.values()].sort((a, b) =>
a.platformName.localeCompare(b.platformName)
)
}
function toProjectRelativePath(targetPath: string): string {
const relativePath = relative(process.cwd(), targetPath)
if (relativePath.startsWith('.')) {
return relativePath
}
return `./${relativePath}`
}
function errorToString(error: unknown): string {
return error instanceof Error ? error.message : String(error)
}
function normalizePathForMatch(value: string): string {
return value.replaceAll('\\', '/').replace(/^\.\/+/, '')
}
export function selectVideoCandidateByFile(
candidates: VideoCandidate[],
options: { artifactsDir: string; videoFile: string }
): VideoCandidate {
const requestedValue = options.videoFile.trim()
if (requestedValue.length === 0) {
throw new Error('--video-file is required')
}
const artifactsRoot = resolve(options.artifactsDir)
const requestedAbsolutePath = resolve(requestedValue)
const requestedPathKey = normalizePathForMatch(requestedValue)
const matches = candidates.filter((candidate) => {
const candidateAbsolutePath = resolve(candidate.videoPath)
if (candidateAbsolutePath === requestedAbsolutePath) {
return true
}
const candidateBaseName = basename(candidate.videoPath)
if (candidateBaseName === requestedValue) {
return true
}
const relativeToCwd = normalizePathForMatch(
relative(process.cwd(), candidateAbsolutePath)
)
if (relativeToCwd === requestedPathKey) {
return true
}
const relativeToArtifacts = normalizePathForMatch(
relative(artifactsRoot, candidateAbsolutePath)
)
return relativeToArtifacts === requestedPathKey
})
if (matches.length === 1) {
return matches[0]
}
if (matches.length === 0) {
const availableVideos = candidates.map((candidate) =>
toProjectRelativePath(candidate.videoPath)
)
throw new Error(
[
`No video matched --video-file "${options.videoFile}".`,
'Available videos:',
...availableVideos.map((videoPath) => `- ${videoPath}`)
].join('\n')
)
}
throw new Error(
[
`--video-file "${options.videoFile}" matched ${matches.length} videos.`,
'Please pass a more specific path.',
...matches.map((match) => `- ${toProjectRelativePath(match.videoPath)}`)
].join('\n')
)
}
async function runCommand(command: string, args: string[]): Promise<string> {
return await new Promise<string>((resolvePromise, rejectPromise) => {
const child = spawn(command, args, {
stdio: ['ignore', 'pipe', 'pipe']
})
let stdout = ''
let stderr = ''
child.stdout.on('data', (chunk: Buffer | string) => {
stdout += chunk.toString()
})
child.stderr.on('data', (chunk: Buffer | string) => {
stderr += chunk.toString()
})
child.on('error', (error) => {
rejectPromise(error)
})
child.on('close', (code) => {
if (code === 0) {
resolvePromise(stdout)
return
}
const details = stderr.trim() || stdout.trim() || 'unknown failure'
rejectPromise(
new Error(
`${command} ${args.join(' ')} failed with code ${String(code)}: ${details}`
)
)
})
})
}
async function ensureBinariesAvailable(): Promise<void> {
await runCommand('ffmpeg', ['-version'])
await runCommand('ffprobe', ['-version'])
}
async function getVideoDurationSeconds(
videoPath: string
): Promise<number | null> {
try {
const stdout = await runCommand('ffprobe', [
'-v',
'error',
'-show_entries',
'format=duration',
'-of',
'default=noprint_wrappers=1:nokey=1',
videoPath
])
const durationSeconds = Number.parseFloat(stdout.trim())
if (!Number.isFinite(durationSeconds) || durationSeconds <= 0) {
return null
}
return durationSeconds
} catch {
return null
}
}
function computeSamplingIntervalSeconds(
durationSeconds: number | null,
minIntervalSeconds: number,
maxFrames: number
): number {
if (durationSeconds === null) {
return minIntervalSeconds
}
const fullCoverageInterval = durationSeconds / maxFrames
return Math.max(minIntervalSeconds, fullCoverageInterval)
}
async function collectVideoCandidates(
artifactsDir: string
): Promise<VideoCandidate[]> {
const absoluteArtifactsDir = resolve(artifactsDir)
const videoPaths = globSync('**/qa-session.mp4', {
cwd: absoluteArtifactsDir,
absolute: true,
nodir: true
}).sort()
const candidates = await Promise.all(
videoPaths.map(async (videoPath) => {
const videoStat = await stat(videoPath)
return {
platformName: extractPlatformFromVideoPath(videoPath),
videoPath,
mtimeMs: videoStat.mtimeMs
}
})
)
return candidates
}
async function extractFramesFromVideo(
videoPath: string,
stagingDir: string,
minIntervalSeconds: number,
maxFrames: number
): Promise<{ frames: ExtractedFrame[]; samplingIntervalSeconds: number }> {
const durationSeconds = await getVideoDurationSeconds(videoPath)
const samplingIntervalSeconds = computeSamplingIntervalSeconds(
durationSeconds,
minIntervalSeconds,
maxFrames
)
const outputPattern = join(stagingDir, 'frame-%03d.jpg')
await runCommand('ffmpeg', [
'-hide_banner',
'-loglevel',
'error',
'-y',
'-i',
videoPath,
'-vf',
`fps=1/${samplingIntervalSeconds},scale=1024:-2:force_original_aspect_ratio=decrease`,
'-frames:v',
String(maxFrames),
outputPattern
])
const frameNames = (await readdir(stagingDir))
.filter((name) => name.startsWith('frame-') && name.endsWith('.jpg'))
.sort()
if (frameNames.length === 0) {
throw new Error(`No frames were extracted from ${videoPath}`)
}
const frames = await Promise.all(
frameNames.map(async (frameName, index) => {
const framePath = join(stagingDir, frameName)
const buffer = await readFile(framePath)
return {
index: index + 1,
timestampSeconds: Number((index * samplingIntervalSeconds).toFixed(2)),
filePath: framePath,
dataUrl: `data:image/jpeg;base64,${buffer.toString('base64')}`
}
})
)
return { frames, samplingIntervalSeconds }
}
function buildReviewMessages(
request: OpenAIReviewRequest
): ResponseInputMessage[] {
const systemPrompt = [
'You are a senior QA engineer reviewing a UI test session.',
'Report only concrete, visible problems and avoid speculation.',
'If confidence is low, mark it explicitly.'
].join(' ')
const introPrompt = [
`Review sampled frames from a QA session for platform "${request.platformName}".`,
`Source video: ${toProjectRelativePath(request.videoPath)}.`,
`Frames are presented in chronological order at ~${request.samplingIntervalSeconds.toFixed(2)}s intervals.`,
'Focus on UI regressions, broken states, visual glitches, unreadable text, missing labels/i18n, and clear workflow failures.',
'Return markdown with these sections exactly:',
'## Summary',
'## Confirmed Issues',
'## Possible Issues (Needs Human Verification)',
'## Overall Risk',
'Under Confirmed Issues include a markdown table with columns:',
'Severity | Timestamp (s) | Issue | Evidence | Confidence | Suggested Fix'
].join('\n')
const userContent: ResponseInputContent[] = [
{ type: 'input_text', text: introPrompt }
]
for (const frame of request.frames) {
userContent.push({
type: 'input_text',
text: `Frame ${frame.index} at approximately ${frame.timestampSeconds}s`
})
userContent.push({
type: 'input_image',
image_url: frame.dataUrl
})
}
return [
{
role: 'system',
content: [{ type: 'input_text', text: systemPrompt }]
},
{
role: 'user',
content: userContent
}
]
}
function normalizeBaseUrl(baseUrl: string): string {
return baseUrl.endsWith('/') ? baseUrl.slice(0, -1) : baseUrl
}
function extractApiErrorMessage(responseBody: unknown): string {
if (!isRecord(responseBody)) {
return 'Unknown API error'
}
const errorValue = responseBody.error
if (!isRecord(errorValue)) {
return 'Unknown API error'
}
const messageValue = errorValue.message
if (typeof messageValue !== 'string' || messageValue.length === 0) {
return 'Unknown API error'
}
return messageValue
}
export function extractOutputText(responseBody: unknown): string | null {
if (!isRecord(responseBody)) {
return null
}
const topLevelOutputText = responseBody.output_text
if (
typeof topLevelOutputText === 'string' &&
topLevelOutputText.trim().length > 0
) {
return topLevelOutputText.trim()
}
const outputValue = responseBody.output
if (!Array.isArray(outputValue)) {
return null
}
const collectedParts: string[] = []
for (const outputItem of outputValue) {
if (!isRecord(outputItem)) {
continue
}
const contentValue = outputItem.content
if (!Array.isArray(contentValue)) {
continue
}
for (const contentItem of contentValue) {
if (!isRecord(contentItem)) {
continue
}
if (contentItem.type !== 'output_text') {
continue
}
const textValue = contentItem.text
if (typeof textValue === 'string' && textValue.trim().length > 0) {
collectedParts.push(textValue.trim())
}
}
}
if (collectedParts.length === 0) {
return null
}
return collectedParts.join('\n\n')
}
async function requestOpenAIReview(
request: OpenAIReviewRequest
): Promise<string> {
const payload: ResponsesCreatePayload = {
model: request.model,
input: buildReviewMessages(request),
max_output_tokens: 3_000
}
const controller = new AbortController()
const timeoutHandle = setTimeout(() => controller.abort(), request.timeoutMs)
try {
const response = await fetch(
`${normalizeBaseUrl(request.baseUrl)}/responses`,
{
method: 'POST',
headers: {
Authorization: `Bearer ${request.apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(payload),
signal: controller.signal
}
)
const rawResponse = await response.text()
let responseBody: unknown
try {
responseBody = JSON.parse(rawResponse)
} catch {
throw new Error(
`OpenAI API returned a non-JSON response: ${rawResponse.slice(0, 400)}`
)
}
if (!response.ok) {
throw new Error(
`OpenAI API request failed (${response.status}): ${extractApiErrorMessage(responseBody)}`
)
}
const reviewText = extractOutputText(responseBody)
if (!reviewText) {
throw new Error('OpenAI API returned no output text')
}
return reviewText
} finally {
clearTimeout(timeoutHandle)
}
}
function buildReportMarkdown(input: {
platformName: string
model: string
videoPath: string
frames: ExtractedFrame[]
samplingIntervalSeconds: number
reviewText: string
}): string {
const header = [
`# ${input.platformName} QA Video Report`,
'',
`- Generated at: ${new Date().toISOString()}`,
`- Model: \`${input.model}\``,
`- Source video: \`${toProjectRelativePath(input.videoPath)}\``,
`- Frames analyzed: ${input.frames.length}`,
`- Sampling interval: ${input.samplingIntervalSeconds.toFixed(2)}s`,
'',
'## AI Review',
''
].join('\n')
return `${header}${input.reviewText.trim()}\n`
}
async function reviewVideo(
video: VideoCandidate,
options: CliOptions,
apiKey: string,
baseUrl: string
): Promise<void> {
const stagingRoot = resolve('./tmp')
await mkdir(stagingRoot, { recursive: true })
const stagingDir = await mkdtemp(join(stagingRoot, 'qa-video-review-'))
try {
process.stdout.write(
`[${video.platformName}] Extracting frames from ${toProjectRelativePath(video.videoPath)}\n`
)
const { frames, samplingIntervalSeconds } = await extractFramesFromVideo(
video.videoPath,
stagingDir,
options.minIntervalSeconds,
options.maxFrames
)
process.stdout.write(
`[${video.platformName}] Sending ${frames.length} frame(s) to ${options.model}\n`
)
const reviewText = await requestOpenAIReview({
apiKey,
baseUrl,
model: options.model,
platformName: video.platformName,
videoPath: video.videoPath,
frames,
samplingIntervalSeconds,
timeoutMs: options.requestTimeoutMs
})
const outputPath = resolve(
options.outputDir,
`${video.platformName}-qa-video-report.md`
)
const reportMarkdown = buildReportMarkdown({
platformName: video.platformName,
model: options.model,
videoPath: video.videoPath,
frames,
samplingIntervalSeconds,
reviewText
})
await mkdir(dirname(outputPath), { recursive: true })
await writeFile(outputPath, reportMarkdown, 'utf-8')
process.stdout.write(
`[${video.platformName}] Wrote ${toProjectRelativePath(outputPath)}\n`
)
if (options.keepFrames) {
const stagedFrames = (await readdir(stagingDir)).filter((name) =>
name.endsWith('.jpg')
).length
process.stdout.write(
`[${video.platformName}] Kept ${stagedFrames} frame(s) in ${toProjectRelativePath(stagingDir)}\n`
)
return
}
} finally {
if (!options.keepFrames) {
await rm(stagingDir, { recursive: true, force: true })
}
}
}
function isExecutedAsScript(metaUrl: string): boolean {
const modulePath = fileURLToPath(metaUrl)
const scriptPath = process.argv[1] ? resolve(process.argv[1]) : ''
return modulePath === scriptPath
}
async function main(): Promise<void> {
const options = parseCliOptions(process.argv.slice(2))
const candidates = await collectVideoCandidates(options.artifactsDir)
if (candidates.length === 0) {
process.stdout.write(
`No qa-session.mp4 files found under ${toProjectRelativePath(resolve(options.artifactsDir))}\n`
)
return
}
const selectedVideo = selectVideoCandidateByFile(candidates, {
artifactsDir: options.artifactsDir,
videoFile: options.videoFile
})
process.stdout.write(
`Selected ${selectedVideo.platformName}: ${toProjectRelativePath(selectedVideo.videoPath)}\n`
)
if (options.dryRun) {
process.stdout.write('\nDry run mode enabled, no API calls were made.\n')
return
}
const apiKey = process.env.OPENAI_API_KEY
if (!apiKey) {
throw new Error('OPENAI_API_KEY is required unless --dry-run is set')
}
await ensureBinariesAvailable()
const baseUrl = process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1'
await reviewVideo(selectedVideo, options, apiKey, baseUrl)
}
if (isExecutedAsScript(import.meta.url)) {
void main().catch((error: unknown) => {
const message = errorToString(error)
process.stderr.write(`qa-video-review failed: ${message}\n`)
process.exit(1)
})
}