mirror of
https://github.com/Comfy-Org/ComfyUI_frontend.git
synced 2026-04-20 06:20:11 +00:00
feat: add comfy-qa skill and automated QA CI pipeline
Add Claude Code skills and a label-triggered QA workflow: - .claude/skills/comfy-qa/SKILL.md: 12-category QA test plan using playwright-cli for browser automation - .github/workflows/pr-qa.yaml: CI workflow triggered by qa-changes (focused, Linux) or qa-full (3-OS matrix) labels. Records screen via ffmpeg, runs Claude CLI with playwright-cli, deploys video gallery to Cloudflare Pages, posts PR comment with GIF thumbnails, and runs OpenAI vision-based video review - scripts/qa-video-review.ts: frame extraction + GPT-4o analysis - scripts/qa-video-review.test.ts: unit tests for video review - knip.config.ts: resolve knip errors for ingest-types package
This commit is contained in:
snomiao
361
.claude/skills/comfy-qa/SKILL.md
Normal file
361
.claude/skills/comfy-qa/SKILL.md
Normal file
@@ -0,0 +1,361 @@
|
||||
---
|
||||
name: comfy-qa
|
||||
description: 'Comprehensive QA of ComfyUI frontend. Navigates all routes, tests all interactive features using playwright-cli, generates a report, and submits a draft PR. Works in CI and local environments, cross-platform.'
|
||||
---
|
||||
|
||||
# ComfyUI Frontend QA Skill
|
||||
|
||||
Perform comprehensive quality assurance of the ComfyUI frontend application by navigating all routes, clicking interactive elements, and testing features. Generate a structured report and submit it as a draft PR.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18+
|
||||
- `pnpm` package manager
|
||||
- `gh` CLI (authenticated)
|
||||
- `playwright-cli` (browser automation): `npm install -g @playwright/cli@latest`
|
||||
|
||||
## Step 1: Environment Detection & Setup
|
||||
|
||||
Detect the runtime environment and ensure the app is accessible.
|
||||
|
||||
### CI Environment
|
||||
|
||||
If `CI=true` is set:
|
||||
|
||||
1. The ComfyUI backend is pre-configured in the CI container (`ghcr.io/comfy-org/comfyui-ci-container`)
|
||||
2. Frontend dist is already built and served by the backend
|
||||
3. Server runs at `http://127.0.0.1:8188`
|
||||
4. Skip user prompts — run fully automated
|
||||
|
||||
### Local Environment
|
||||
|
||||
If `CI` is not set:
|
||||
|
||||
1. **Ask the user**: "Is a ComfyUI server already running? If so, what URL? (default: http://127.0.0.1:8188)"
|
||||
- If yes: use the provided URL
|
||||
- If no: offer to start one:
|
||||
|
||||
```bash
|
||||
# Option A: Use existing ComfyUI installation
|
||||
# Ask for the path to ComfyUI, then:
|
||||
cd <comfyui_path>
|
||||
python main.py --cpu --multi-user --front-end-root <frontend_dist_path> &
|
||||
|
||||
# Option B: Build frontend and use preview server (no backend features)
|
||||
pnpm build && pnpm preview &
|
||||
```
|
||||
|
||||
2. Wait for server readiness by polling the URL (retry with 2s intervals, 60s timeout)
|
||||
|
||||
### Browser Automation Setup
|
||||
|
||||
Use **playwright-cli** for browser interaction via Bash commands:
|
||||
|
||||
```bash
|
||||
playwright-cli open http://127.0.0.1:8188 # open browser and navigate
|
||||
playwright-cli snapshot # capture snapshot with element refs
|
||||
playwright-cli click e1 # click by element ref from snapshot
|
||||
playwright-cli press Tab # keyboard shortcuts
|
||||
playwright-cli screenshot --filename=f.png # save screenshot
|
||||
```
|
||||
|
||||
playwright-cli is headless by default (CI-friendly). Each command outputs the current page snapshot with element references (`e1`, `e2`, …) that you use for subsequent `click`, `fill`, `hover` commands. Always run `snapshot` before interacting to get fresh refs.
|
||||
|
||||
For local dev servers behind proxies, adjust the URL accordingly (e.g., `https://[port].stukivx.xyz` pattern if configured).
|
||||
|
||||
## Step 2: QA Test Plan
|
||||
|
||||
Navigate to the application URL and systematically test each area below. For each test, record:
|
||||
|
||||
- **Status**: pass / fail / skip (with reason)
|
||||
- **Notes**: any issues, unexpected behavior, or visual glitches
|
||||
- **Screenshots**: take screenshots of failures or notable states
|
||||
|
||||
### 2.1 Application Load & Routes
|
||||
|
||||
| Test | Steps |
|
||||
| ----------------- | ------------------------------------------------------------ |
|
||||
| Root route loads | Navigate to `/` — GraphView should render with canvas |
|
||||
| User select route | Navigate to `/user-select` — user selection UI should appear |
|
||||
| Default redirect | If multi-user mode, `/` redirects to `/user-select` first |
|
||||
| 404 handling | Navigate to `/nonexistent` — should handle gracefully |
|
||||
|
||||
### 2.2 Canvas & Graph View
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------------- | -------------------------------------------------------------- |
|
||||
| Canvas renders | The LiteGraph canvas is visible and interactive |
|
||||
| Pan canvas | Click and drag on empty canvas area |
|
||||
| Zoom in/out | Use scroll wheel or Alt+=/Alt+- |
|
||||
| Fit view | Press `.` key — canvas fits to content |
|
||||
| Add node via double-click | Double-click canvas to open search, type "KSampler", select it |
|
||||
| Add node via search | Open search box, find and add a node |
|
||||
| Delete node | Select a node, press Delete key |
|
||||
| Connect nodes | Drag from output slot to input slot |
|
||||
| Disconnect nodes | Right-click a link and remove, or drag from connected slot |
|
||||
| Multi-select | Shift+click or drag-select multiple nodes |
|
||||
| Copy/Paste | Select nodes, Ctrl+C then Ctrl+V |
|
||||
| Undo/Redo | Make changes, Ctrl+Z to undo, Ctrl+Y to redo |
|
||||
| Node context menu | Right-click a node — menu appears with all expected options |
|
||||
| Canvas context menu | Right-click empty canvas — menu appears |
|
||||
|
||||
### 2.3 Node Operations
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | ---------------------------------------------------------- |
|
||||
| Bypass node | Select node, Ctrl+B — node shows bypass state |
|
||||
| Mute node | Select node, Ctrl+M — node shows muted state |
|
||||
| Collapse node | Select node, Alt+C — node collapses |
|
||||
| Pin node | Select node, press P — node becomes pinned |
|
||||
| Rename node | Double-click node title — edit mode activates |
|
||||
| Node color | Right-click > Color — color picker works |
|
||||
| Group nodes | Select multiple nodes, Ctrl+G — group created |
|
||||
| Ungroup | Right-click group > Ungroup |
|
||||
| Widget interactions | Toggle checkboxes, adjust sliders, type in text fields |
|
||||
| Combo widget | Click dropdown widgets — options appear and are selectable |
|
||||
|
||||
### 2.4 Sidebar Tabs
|
||||
|
||||
| Test | Steps |
|
||||
| ---------------------- | ------------------------------------------------------ |
|
||||
| Workflows tab | Press W — workflows sidebar opens with saved workflows |
|
||||
| Node Library tab | Press N — node library opens with categories |
|
||||
| Model Library tab | Press M — model library opens |
|
||||
| Assets tab | Press A — assets browser opens |
|
||||
| Tab toggle | Press same key again — sidebar closes |
|
||||
| Search in sidebar | Type in search box — results filter |
|
||||
| Drag node from library | Drag a node from library onto canvas |
|
||||
|
||||
### 2.5 Topbar & Workflow Tabs
|
||||
|
||||
| Test | Steps |
|
||||
| -------------------- | ------------------------------------------------------ |
|
||||
| Workflow tab display | Current workflow name shown in tab bar |
|
||||
| New workflow | Ctrl+N — new blank workflow created |
|
||||
| Rename workflow | Double-click workflow tab |
|
||||
| Tab context menu | Right-click workflow tab — menu with Close/Rename/etc. |
|
||||
| Multiple tabs | Open multiple workflows, switch between them |
|
||||
| Queue button | Click Queue/Run button — prompt queues |
|
||||
| Batch count | Click batch count editor, change value |
|
||||
| Menu hamburger | Click hamburger menu — options appear |
|
||||
|
||||
### 2.6 Settings Dialog
|
||||
|
||||
| Test | Steps |
|
||||
| ---------------- | ---------------------------------------------------- |
|
||||
| Open settings | Press Ctrl+, or click settings button |
|
||||
| Settings tabs | Navigate through all setting categories |
|
||||
| Change a setting | Toggle a boolean setting — it persists after closing |
|
||||
| Search settings | Type in settings search box — results filter |
|
||||
| Keybindings tab | Navigate to keybindings panel |
|
||||
| About tab | Navigate to about panel — version info shown |
|
||||
| Close settings | Press Escape or click close button |
|
||||
|
||||
### 2.7 Bottom Panel
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | -------------------------------------- |
|
||||
| Toggle panel | Press Ctrl+` — bottom panel opens |
|
||||
| Logs tab | Logs/terminal tab shows server output |
|
||||
| Shortcuts tab | Shortcuts reference is displayed |
|
||||
| Keybindings display | Press Ctrl+Shift+K — keybindings panel |
|
||||
|
||||
### 2.8 Execution & Queue
|
||||
|
||||
| Test | Steps |
|
||||
| -------------- | ----------------------------------------------------- |
|
||||
| Queue prompt | Load default workflow, click Queue — execution starts |
|
||||
| Queue progress | Progress indicator shows during execution |
|
||||
| Interrupt | Press Ctrl+Alt+Enter during execution — interrupts |
|
||||
| Job history | Open job history sidebar — past executions listed |
|
||||
| Clear history | Clear execution history via menu |
|
||||
|
||||
### 2.9 Workflow File Operations
|
||||
|
||||
| Test | Steps |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| Save workflow | Ctrl+S — workflow saves (check for prompt if new) |
|
||||
| Open workflow | Ctrl+O — file picker or workflow browser opens |
|
||||
| Export JSON | Menu > Export — workflow JSON downloads |
|
||||
| Import workflow | Drag a .json workflow file onto canvas |
|
||||
| Load default | Menu > Load Default — default workflow loads |
|
||||
| Clear workflow | Menu > Clear — canvas clears (after confirmation) |
|
||||
|
||||
### 2.10 Advanced Features
|
||||
|
||||
| Test | Steps |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| Minimap | Alt+M — minimap toggle |
|
||||
| Focus mode | Toggle focus mode |
|
||||
| Canvas lock | Press H to lock, V to unlock |
|
||||
| Link visibility | Ctrl+Shift+L — toggle links |
|
||||
| Subgraph | Select nodes > Ctrl+Shift+E — convert to subgraph |
|
||||
|
||||
### 2.11 Error Handling
|
||||
|
||||
| Test | Steps |
|
||||
| --------------------- | -------------------------------------------- |
|
||||
| Missing nodes dialog | Load workflow with non-existent node types |
|
||||
| Missing models dialog | Trigger missing model warning |
|
||||
| Network error | Disconnect backend, verify graceful handling |
|
||||
| Invalid workflow | Try loading malformed JSON |
|
||||
|
||||
### 2.12 Responsive & Accessibility
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | ------------------------------------- |
|
||||
| Window resize | Resize browser window — layout adapts |
|
||||
| Keyboard navigation | Tab through interactive elements |
|
||||
| Sidebar resize | Drag sidebar edge to resize |
|
||||
|
||||
## Step 3: Generate Report
|
||||
|
||||
After completing all tests, generate a markdown report file.
|
||||
|
||||
### Report Location
|
||||
|
||||
```
|
||||
docs/qa/YYYY-MM-DD-NNN-report.md
|
||||
```
|
||||
|
||||
Where:
|
||||
|
||||
- `YYYY-MM-DD` is today's date
|
||||
- `NNN` is a zero-padded increment index (001, 002, etc.)
|
||||
|
||||
To determine the increment, check existing files:
|
||||
|
||||
```bash
|
||||
ls docs/qa/ | grep "$(date +%Y-%m-%d)" | wc -l
|
||||
```
|
||||
|
||||
### Report Template
|
||||
|
||||
```markdown
|
||||
# QA Report: ComfyUI Frontend
|
||||
|
||||
**Date**: YYYY-MM-DD
|
||||
**Environment**: CI / Local (OS, Browser)
|
||||
**Frontend Version**: (git sha or version)
|
||||
**Agent**: Claude / Codex / Other
|
||||
**Server URL**: http://...
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Pass | Fail | Skip | Total |
|
||||
| --------------- | ---- | ---- | ---- | ----- |
|
||||
| Routes & Load | | | | |
|
||||
| Canvas | | | | |
|
||||
| Node Operations | | | | |
|
||||
| Sidebar | | | | |
|
||||
| Topbar | | | | |
|
||||
| Settings | | | | |
|
||||
| Bottom Panel | | | | |
|
||||
| Execution | | | | |
|
||||
| File Operations | | | | |
|
||||
| Advanced | | | | |
|
||||
| Error Handling | | | | |
|
||||
| Responsive | | | | |
|
||||
| **Total** | | | | |
|
||||
|
||||
## Results
|
||||
|
||||
### Routes & Load
|
||||
|
||||
- [x] Root route loads — pass
|
||||
- [ ] ...
|
||||
|
||||
### Canvas & Graph View
|
||||
|
||||
- [x] Canvas renders — pass
|
||||
- [ ] ...
|
||||
|
||||
(repeat for each category)
|
||||
|
||||
## Issues Found
|
||||
|
||||
### Issue 1: [Title]
|
||||
|
||||
- **Severity**: critical / major / minor / cosmetic
|
||||
- **Steps to reproduce**: ...
|
||||
- **Expected**: ...
|
||||
- **Actual**: ...
|
||||
- **Screenshot**: (if available)
|
||||
|
||||
## Notes
|
||||
|
||||
Any additional observations, performance notes, or suggestions.
|
||||
```
|
||||
|
||||
## Step 4: Commit and Push Report
|
||||
|
||||
### In CI (when `CI=true`)
|
||||
|
||||
Save the report directly to `$QA_ARTIFACTS` (the CI workflow uploads this as
|
||||
an artifact and posts results as a PR comment). Do **not** commit, push, or
|
||||
create a new PR.
|
||||
|
||||
### Local / interactive use
|
||||
|
||||
When running locally, create a draft PR after committing:
|
||||
|
||||
```bash
|
||||
# Ensure on a feature branch
|
||||
BRANCH_NAME="qa/$(date +%Y-%m-%d)-$(git rev-parse --short HEAD)"
|
||||
git checkout -b "$BRANCH_NAME" 2>/dev/null || git checkout "$BRANCH_NAME"
|
||||
|
||||
git add docs/qa/
|
||||
git commit -m "docs: add QA report $(date +%Y-%m-%d)
|
||||
|
||||
Automated QA report covering all frontend routes and features."
|
||||
git push -u origin "$BRANCH_NAME"
|
||||
|
||||
# Create draft PR assigned to comfy-pr-bot
|
||||
gh pr create \
|
||||
--draft \
|
||||
--title "QA Report: $(date +%Y-%m-%d)" \
|
||||
--body "## QA Report
|
||||
|
||||
Automated frontend QA run covering all routes and interactive features.
|
||||
|
||||
See \`docs/qa/\` for the full report.
|
||||
|
||||
/cc @comfy-pr-bot" \
|
||||
--assignee comfy-pr-bot
|
||||
```
|
||||
|
||||
## Execution Notes
|
||||
|
||||
### Cross-Platform Considerations
|
||||
|
||||
- **Windows**: Use `pwsh` or `cmd` equivalents for shell commands. `gh` CLI works on all platforms.
|
||||
- **macOS**: Keyboard shortcuts use Cmd instead of Ctrl in the actual app, but Playwright sends OS-appropriate keys.
|
||||
- **Linux**: Primary CI platform. Screenshot baselines are Linux-only.
|
||||
|
||||
### Agent Compatibility
|
||||
|
||||
This skill uses **playwright-cli** (`@playwright/cli`) — a token-efficient CLI designed for coding agents. Install it once with `npm install -g @playwright/cli@latest`, then use `Bash` to run commands.
|
||||
|
||||
The key operations and their playwright-cli equivalents:
|
||||
|
||||
| Action | Command |
|
||||
| ---------------- | ------------------------------------------ |
|
||||
| Navigate to URL | `playwright-cli goto <url>` |
|
||||
| Get element refs | `playwright-cli snapshot` |
|
||||
| Click element | `playwright-cli click <ref>` |
|
||||
| Type text | `playwright-cli fill <ref> <text>` |
|
||||
| Press shortcut | `playwright-cli press <key>` |
|
||||
| Take screenshot | `playwright-cli screenshot --filename=f` |
|
||||
| Hover element | `playwright-cli hover <ref>` |
|
||||
| Select dropdown | `playwright-cli select <ref> <value>` |
|
||||
|
||||
Snapshots return element references (`e1`, `e2`, …). Always run `snapshot` after navigation or major interactions to refresh refs before acting.
|
||||
|
||||
### Tips for Reliable QA
|
||||
|
||||
1. **Wait for page stability** before interacting — check that elements are visible and enabled
|
||||
2. **Take a snapshot after each major navigation** to verify state
|
||||
3. **Don't use fixed timeouts** — poll for expected conditions
|
||||
4. **Record the full page snapshot** at the start for baseline comparison
|
||||
5. **If a test fails**, document it and continue — don't abort the entire QA run
|
||||
6. **Group related tests** — complete one category before moving to the next
|
||||
556
.github/workflows/pr-qa.yaml
vendored
Normal file
556
.github/workflows/pr-qa.yaml
vendored
Normal file
@@ -0,0 +1,556 @@
|
||||
# Automated QA of ComfyUI frontend using Claude CLI + Playwright MCP.
|
||||
# Two modes:
|
||||
# Focused (qa-changes label): Linux-only, tests areas affected by PR changes
|
||||
# Full (qa-full label): 3-OS matrix, full test plan
|
||||
name: 'PR: QA'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
mode:
|
||||
description: 'QA mode'
|
||||
type: choice
|
||||
options: [focused, full]
|
||||
default: focused
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
resolve-matrix:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
os: ${{ steps.set.outputs.os }}
|
||||
mode: ${{ steps.set.outputs.mode }}
|
||||
skip: ${{ steps.set.outputs.skip }}
|
||||
steps:
|
||||
- name: Determine QA mode
|
||||
id: set
|
||||
env:
|
||||
LABEL: ${{ github.event.label.name }}
|
||||
EVENT_ACTION: ${{ github.event.action }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
INPUT_MODE: ${{ inputs.mode }}
|
||||
run: |
|
||||
FULL=false
|
||||
|
||||
# Only run on label events if it's one of our labels
|
||||
if [ "$EVENT_ACTION" = "labeled" ] && \
|
||||
[ "$LABEL" != "qa-changes" ] && [ "$LABEL" != "qa-full" ]; then
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# Full QA triggers
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] && \
|
||||
[ "$INPUT_MODE" = "full" ]; then
|
||||
FULL=true
|
||||
fi
|
||||
if [ "$LABEL" = "qa-full" ]; then
|
||||
FULL=true
|
||||
fi
|
||||
|
||||
if [ "$FULL" = "true" ]; then
|
||||
echo 'os=["ubuntu-latest","macos-latest","windows-latest"]' >> "$GITHUB_OUTPUT"
|
||||
echo "mode=full" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo 'os=["ubuntu-latest"]' >> "$GITHUB_OUTPUT"
|
||||
echo "mode=focused" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
echo "Mode: $([ "$FULL" = "true" ] && echo full || echo focused)"
|
||||
|
||||
qa:
|
||||
needs: resolve-matrix
|
||||
if: needs.resolve-matrix.outputs.skip != 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: ${{ fromJson(needs.resolve-matrix.outputs.os) }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 60
|
||||
permissions:
|
||||
pull-requests: write
|
||||
env:
|
||||
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
|
||||
steps:
|
||||
- name: Set QA artifacts path
|
||||
shell: bash
|
||||
run: echo "QA_ARTIFACTS=$RUNNER_TEMP/qa-artifacts" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.head_ref || github.ref }}
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Setup frontend
|
||||
uses: ./.github/actions/setup-frontend
|
||||
with:
|
||||
include_build_step: true
|
||||
|
||||
- name: Setup and start ComfyUI server
|
||||
uses: ./.github/actions/setup-comfyui-server
|
||||
with:
|
||||
launch_server: 'true'
|
||||
|
||||
- name: Wait for ComfyUI server
|
||||
shell: bash
|
||||
run: |
|
||||
echo "Waiting for ComfyUI server..."
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1; then
|
||||
echo "Server ready"; exit 0
|
||||
fi; sleep 2
|
||||
done
|
||||
echo "::error::Server timeout"; exit 1
|
||||
|
||||
- name: Install playwright-cli and Claude Code
|
||||
shell: bash
|
||||
run: npm install -g @playwright/cli@latest @anthropic-ai/claude-code@2.1.71
|
||||
|
||||
# --- Virtual display + recording: Linux ---
|
||||
- name: Setup Xvfb (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq xvfb ffmpeg x11-xserver-utils >/dev/null 2>&1
|
||||
Xvfb :99 -screen 0 1280x720x24 -ac -nolisten tcp &
|
||||
echo $! > "${{ runner.temp }}/xvfb.pid"
|
||||
sleep 2
|
||||
echo "DISPLAY=:99" >> "$GITHUB_ENV"
|
||||
# Set black background and normal cursor (removes the default X cursor)
|
||||
DISPLAY=:99 xsetroot -solid black -cursor_name left_ptr 2>/dev/null || true
|
||||
|
||||
- name: Start recording (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -y -f x11grab -video_size 1280x720 -framerate 10 \
|
||||
-i :99.0 -c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 1
|
||||
|
||||
# --- Recording: macOS (avfoundation) ---
|
||||
- name: Install ffmpeg (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
run: brew install ffmpeg
|
||||
|
||||
- name: Grant screen recording permission (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
run: |
|
||||
# Grant kTCCServiceScreenCapture to ffmpeg and bash so avfoundation
|
||||
# recording works without the blocking permission dialog.
|
||||
FFMPEG_PATH=$(which ffmpeg)
|
||||
echo "Granting screen recording permission to: $FFMPEG_PATH"
|
||||
# macOS 14+ (Sonoma/Sequoia) TCC.db schema has 17 columns
|
||||
for BIN in "$FFMPEG_PATH" "/bin/bash"; do
|
||||
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
|
||||
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,0,'UNUSED',NULL,0,$(date +%s),NULL,NULL,NULL,NULL);" 2>/dev/null \
|
||||
|| sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
|
||||
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,NULL,'UNUSED',NULL,0,$(date +%s));" 2>/dev/null \
|
||||
|| echo "Warning: Could not grant TCC permission to ${BIN}"
|
||||
done
|
||||
|
||||
- name: Start recording (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -f avfoundation -list_devices true -i "" 2>&1 || true
|
||||
ffmpeg -y -f avfoundation -framerate 10 -capture_cursor 1 \
|
||||
-i "Capture screen 0:none" -c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 2
|
||||
|
||||
# --- Recording: Windows (gdigrab) ---
|
||||
- name: Install ffmpeg (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
shell: pwsh
|
||||
run: choco install ffmpeg -y --no-progress
|
||||
|
||||
- name: Start recording (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -y -f gdigrab -framerate 10 -i desktop \
|
||||
-c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 1
|
||||
|
||||
- name: Get PR diff for focused QA
|
||||
if: needs.resolve-matrix.outputs.mode == 'focused'
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
gh pr diff ${{ github.event.pull_request.number || '' }} \
|
||||
--repo ${{ github.repository }} > "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || \
|
||||
git diff origin/main...HEAD > "${{ runner.temp }}/pr-diff.txt"
|
||||
|
||||
# Summarize changed files for the prompt
|
||||
echo "Changed files:"
|
||||
grep '^diff --git' "${{ runner.temp }}/pr-diff.txt" | \
|
||||
sed 's|diff --git a/||;s| b/.*||' | sort -u | tee "${{ runner.temp }}/changed-files.txt"
|
||||
|
||||
- name: Write QA prompt
|
||||
shell: bash
|
||||
env:
|
||||
BRANCH: ${{ github.head_ref || github.ref_name }}
|
||||
PR_NUM: ${{ github.event.pull_request.number || 'N/A' }}
|
||||
SHA: ${{ github.sha }}
|
||||
run: |
|
||||
OS_LOWER=$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
if [ "$QA_MODE" = "full" ]; then
|
||||
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
|
||||
You are running a FULL automated QA pass on the ComfyUI frontend.
|
||||
Read the file .claude/skills/comfy-qa/SKILL.md and follow the FULL QA test plan.
|
||||
|
||||
Environment: CI=true, OS=${{ runner.os }}
|
||||
Server URL: http://127.0.0.1:8188
|
||||
Branch: ${BRANCH}
|
||||
PR: #${PR_NUM}
|
||||
Commit: ${SHA}
|
||||
|
||||
1. Use playwright-cli via Bash to navigate http://127.0.0.1:8188
|
||||
- Run `playwright-cli goto http://127.0.0.1:8188` to open the app
|
||||
- Run `playwright-cli snapshot` after each navigation to get element refs
|
||||
- Use `playwright-cli click <ref>`, `playwright-cli press <key>`, etc. for interactions
|
||||
- Run `playwright-cli screenshot --filename=<path>` to capture failures
|
||||
2. Run the FULL QA test plan from the skill file
|
||||
3. Take screenshots of failures or notable states into $QA_ARTIFACTS
|
||||
4. Save report to $QA_ARTIFACTS as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
|
||||
|
||||
Do NOT create a new PR. Do NOT post PR comments. Do NOT commit or push anything.
|
||||
Skip tests not available in CI (file dialogs, GPU execution).
|
||||
PROMPT
|
||||
else
|
||||
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
|
||||
You are running a FOCUSED QA pass on a pull request to the ComfyUI frontend.
|
||||
Your goal is to verify that the changes in this PR work correctly and don't break related functionality.
|
||||
|
||||
Environment: CI=true, OS=${{ runner.os }}
|
||||
Server URL: http://127.0.0.1:8188
|
||||
Branch: ${BRANCH}
|
||||
PR: #${PR_NUM}
|
||||
Commit: ${SHA}
|
||||
|
||||
CHANGED FILES:
|
||||
$(cat "${{ runner.temp }}/changed-files.txt" 2>/dev/null || echo "Unknown")
|
||||
|
||||
DIFF (truncated to 500 lines):
|
||||
$(head -500 "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || echo "No diff available")
|
||||
|
||||
Instructions:
|
||||
1. Read the diff above to understand what changed in this PR
|
||||
2. Use playwright-cli via Bash to navigate http://127.0.0.1:8188
|
||||
- Run `playwright-cli goto http://127.0.0.1:8188` to open the app
|
||||
- Run `playwright-cli snapshot` after each navigation to get element refs
|
||||
- Use `playwright-cli click <ref>`, `playwright-cli press <key>`, etc. for interactions
|
||||
- Run `playwright-cli screenshot --filename=<path>` to capture failures into $QA_ARTIFACTS
|
||||
3. Test the specific UI areas affected by these changes
|
||||
4. Also do a quick smoke test of core functionality (app loads, canvas renders, sidebar works)
|
||||
5. Save a concise report to $QA_ARTIFACTS as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
|
||||
|
||||
Focus on:
|
||||
- Does the changed functionality work as expected?
|
||||
- Are there visual regressions in affected areas?
|
||||
- Do related features still work?
|
||||
|
||||
Do NOT run the full QA test plan. Do NOT create a new PR. Do NOT post PR comments. Do NOT commit or push anything.
|
||||
Skip tests not available in CI (file dialogs, GPU execution).
|
||||
PROMPT
|
||||
fi
|
||||
|
||||
- name: Run Claude QA
|
||||
shell: bash
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
CI: 'true'
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
MAX_TURNS=128
|
||||
if [ "$QA_MODE" = "focused" ]; then MAX_TURNS=30; fi
|
||||
|
||||
cat "${{ runner.temp }}/qa-prompt.txt" | claude --print \
|
||||
--max-turns "$MAX_TURNS" \
|
||||
--allowedTools "Bash(playwright-cli:*),Bash(date:*),Bash(ls:*),Bash(mkdir:*),Read,Write,Edit,Glob,Grep"
|
||||
|
||||
- name: Stop recording
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
PID_FILE="${{ runner.temp }}/ffmpeg.pid"
|
||||
if [ -f "$PID_FILE" ]; then
|
||||
if [ "$RUNNER_OS" = "Windows" ]; then
|
||||
# Graceful stop: taskkill without /F sends WM_CLOSE so ffmpeg
|
||||
# can finalize the mp4 (write moov atom). Force-kill as fallback.
|
||||
taskkill //PID $(cat "$PID_FILE") 2>/dev/null || true
|
||||
sleep 5
|
||||
taskkill //F //PID $(cat "$PID_FILE") 2>/dev/null || true
|
||||
else
|
||||
kill -INT $(cat "$PID_FILE") 2>/dev/null || true
|
||||
sleep 3; kill $(cat "$PID_FILE") 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
[ -f "$QA_ARTIFACTS/qa-session.mp4" ] && \
|
||||
echo "Video: $(du -h "$QA_ARTIFACTS/qa-session.mp4" | cut -f1)" || \
|
||||
echo "No video (non-fatal)"
|
||||
|
||||
- name: Collect artifacts
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ls -la "$QA_ARTIFACTS/" || true
|
||||
|
||||
- name: Upload QA artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v6.2.0
|
||||
with:
|
||||
name: qa-report-${{ runner.os }}-${{ github.run_id }}
|
||||
path: ${{ env.QA_ARTIFACTS }}/
|
||||
retention-days: 14
|
||||
|
||||
- name: Cleanup (Linux)
|
||||
if: always() && runner.os == 'Linux'
|
||||
run: kill $(cat "${{ runner.temp }}/xvfb.pid") 2>/dev/null || true
|
||||
|
||||
report:
|
||||
needs: [resolve-matrix, qa]
|
||||
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Download QA artifacts
|
||||
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
|
||||
with:
|
||||
path: qa-artifacts
|
||||
pattern: qa-report-*
|
||||
|
||||
- name: Install ffmpeg
|
||||
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
|
||||
|
||||
- name: Deploy videos to Cloudflare Pages
|
||||
id: deploy-videos
|
||||
env:
|
||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
||||
RAW_BRANCH: ${{ github.head_ref || github.ref_name }}
|
||||
run: |
|
||||
npm install -g wrangler@4.74.0 >/dev/null 2>&1
|
||||
|
||||
DEPLOY_DIR=$(mktemp -d)
|
||||
mkdir -p "$DEPLOY_DIR"
|
||||
|
||||
for os in Linux macOS Windows; do
|
||||
VID="qa-artifacts/qa-report-${os}-${{ github.run_id }}/qa-session.mp4"
|
||||
if [ -f "$VID" ]; then
|
||||
cp "$VID" "$DEPLOY_DIR/qa-${os}.mp4"
|
||||
echo "Found ${os} video ($(du -h "$VID" | cut -f1))"
|
||||
|
||||
# Generate GIF thumbnail: 8s starting at 10s, 480px wide, 8fps
|
||||
ffmpeg -y -ss 10 -i "$VID" -t 8 \
|
||||
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
|
||||
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|
||||
|| ffmpeg -y -i "$VID" -t 8 \
|
||||
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
|
||||
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|
||||
|| echo "GIF generation failed for ${os} (non-fatal)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Build video cards HTML
|
||||
CARDS=""
|
||||
ICONS_Linux="🐧" ICONS_macOS="🍎" ICONS_Windows="🪟"
|
||||
for os in Linux macOS Windows; do
|
||||
eval "ICON=\$ICONS_${os}"
|
||||
if [ -f "$DEPLOY_DIR/qa-${os}.mp4" ]; then
|
||||
CARDS="${CARDS}<div class=card><video controls autoplay muted loop preload=metadata><source src=qa-${os}.mp4 type=video/mp4></video><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><a class=download href=qa-${os}.mp4 download>Download</a></div></div>"
|
||||
else
|
||||
CARDS="${CARDS}<div class=card><div class=empty-card>No recording available</div><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><span class='badge missing'>Missing</span></div></div>"
|
||||
fi
|
||||
done
|
||||
|
||||
cat > "$DEPLOY_DIR/index.html" <<INDEXEOF
|
||||
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><title>QA Session Recordings</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}body{background:#0d1117;color:#e6edf3;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;min-height:100vh;padding:2rem 1rem}.container{max-width:1200px;margin:0 auto}header{display:flex;align-items:center;gap:.75rem;margin-bottom:2rem;padding-bottom:1rem;border-bottom:1px solid #30363d}h1{font-size:1.5rem;font-weight:600}.meta{color:#8b949e;font-size:.875rem;margin-top:.25rem}.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(340px,1fr));gap:1.25rem}.card{background:#161b22;border:1px solid #30363d;border-radius:.5rem;overflow:hidden;transition:border-color .15s}.card:hover{border-color:#58a6ff}.card video{width:100%;display:block;background:#010409;aspect-ratio:16/9;object-fit:contain}.card-body{padding:.75rem 1rem;display:flex;align-items:center;justify-content:space-between}.platform{display:flex;align-items:center;gap:.5rem;font-weight:500}.icon{font-size:1.25rem}.badge{font-size:.75rem;padding:.125rem .5rem;border-radius:999px;background:#1f6feb33;color:#58a6ff;border:1px solid #1f6feb55}.badge.missing{background:#da363333;color:#f85149;border-color:#da363355}.empty-card{display:flex;align-items:center;justify-content:center;min-height:200px;color:#484f58;font-size:.875rem}a.download{color:#58a6ff;text-decoration:none;font-size:.8125rem}a.download:hover{text-decoration:underline}
|
||||
</style></head><body><div class=container>
|
||||
<header><svg width=28 height=28 viewBox="0 0 24 24" fill=none stroke=#58a6ff stroke-width=2 stroke-linecap=round stroke-linejoin=round><polygon points="23 7 16 12 23 17 23 7"/><rect x=1 y=5 width=15 height=14 rx=2 ry=2/></svg><div><h1>QA Session Recordings</h1><div class=meta>ComfyUI Frontend · Automated QA</div></div></header>
|
||||
<div class=grid>${CARDS}</div>
|
||||
</div></body></html>
|
||||
INDEXEOF
|
||||
|
||||
# 404 page so Cloudflare Pages returns proper 404 for missing files
|
||||
# (instead of SPA fallback serving index.html)
|
||||
cat > "$DEPLOY_DIR/404.html" <<'ERROREOF'
|
||||
<!DOCTYPE html><html><head><meta charset=utf-8><title>404</title>
|
||||
<style>body{background:#0d1117;color:#8b949e;font-family:sans-serif;display:flex;align-items:center;justify-content:center;min-height:100vh;margin:0}div{text-align:center}h1{color:#f85149;font-size:3rem;margin-bottom:.5rem}p{font-size:1rem}</style>
|
||||
</head><body><div><h1>404</h1><p>File not found. The QA recording may have failed or been cancelled.</p></div></body></html>
|
||||
ERROREOF
|
||||
|
||||
# Sanitize branch name for Cloudflare Pages URL (same rules CF uses)
|
||||
BRANCH=$(echo "$RAW_BRANCH" | sed 's/[^a-zA-Z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | cut -c1-28)
|
||||
URL=$(wrangler pages deploy "$DEPLOY_DIR" \
|
||||
--project-name="comfyui-qa-videos" \
|
||||
--branch="$BRANCH" 2>&1 \
|
||||
| grep -oE 'https://[a-zA-Z0-9.-]+\.pages\.dev\S*' | head -1)
|
||||
|
||||
echo "url=${URL:-https://${BRANCH}.comfyui-qa-videos.pages.dev}" >> "$GITHUB_OUTPUT"
|
||||
echo "Deployed to: ${URL}"
|
||||
|
||||
- name: Post QA comment on PR
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
VIDEO_BASE: ${{ steps.deploy-videos.outputs.url }}
|
||||
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
|
||||
run: |
|
||||
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
COMMENT_MARKER="<!-- QA_REPORT_COMMENT -->"
|
||||
|
||||
MODE_BADGE="🔍 Focused"
|
||||
if [ "$QA_MODE" = "full" ]; then MODE_BADGE="🔬 Full (3-OS)"; fi
|
||||
|
||||
# Build video section with GIF thumbnails linking to full videos
|
||||
VIDEO_SECTION=""
|
||||
for os in Linux macOS Windows; do
|
||||
GIF_URL="${VIDEO_BASE}/qa-${os}-thumb.gif"
|
||||
VID_URL="${VIDEO_BASE}/qa-${os}.mp4"
|
||||
if curl -sf --head "$VID_URL" >/dev/null 2>&1; then
|
||||
if curl -sf --head "$GIF_URL" >/dev/null 2>&1; then
|
||||
VIDEO_SECTION="${VIDEO_SECTION}[](${VID_URL})"$'\n'
|
||||
else
|
||||
VIDEO_SECTION="${VIDEO_SECTION}[${os} video](${VID_URL})"$'\n'
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
BODY=$(cat <<EOF
|
||||
${COMMENT_MARKER}
|
||||
## QA ${MODE_BADGE}
|
||||
|
||||
${VIDEO_SECTION}
|
||||
**Run**: [${RUN}](${RUN}) · [Download artifacts](${RUN}#artifacts) · [All videos](${VIDEO_BASE})
|
||||
EOF
|
||||
)
|
||||
|
||||
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
|
||||
--jq ".[] | select(.body | contains(\"${COMMENT_MARKER}\")) | .id" | head -1)
|
||||
|
||||
if [ -n "$EXISTING" ]; then
|
||||
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
|
||||
--field body="$BODY"
|
||||
else
|
||||
gh pr comment ${{ github.event.pull_request.number }} \
|
||||
--repo ${{ github.repository }} --body "$BODY"
|
||||
fi
|
||||
|
||||
- name: Remove QA label
|
||||
if: >-
|
||||
github.event.label.name == 'qa-changes' ||
|
||||
github.event.label.name == 'qa-full'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
LABEL_NAME: ${{ github.event.label.name }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
gh pr edit "$PR_NUMBER" --repo "$REPO" --remove-label "$LABEL_NAME"
|
||||
|
||||
video-review:
|
||||
needs: [qa, report]
|
||||
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
|
||||
with:
|
||||
node-version: 24
|
||||
|
||||
- name: Install pnpm
|
||||
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
|
||||
with:
|
||||
version: 10
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install ffmpeg
|
||||
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
|
||||
|
||||
- name: Download QA artifacts
|
||||
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
|
||||
with:
|
||||
path: qa-artifacts
|
||||
pattern: qa-report-*
|
||||
|
||||
- name: Run video review
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
run: |
|
||||
mkdir -p video-reviews
|
||||
for vid in qa-artifacts/qa-report-*/qa-session.mp4; do
|
||||
[ -f "$vid" ] || continue
|
||||
echo "::group::Reviewing $vid"
|
||||
pnpm exec tsx scripts/qa-video-review.ts \
|
||||
--artifacts-dir qa-artifacts \
|
||||
--output-dir video-reviews \
|
||||
--video-file "$vid" \
|
||||
--model gpt-4o || true
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
- name: Post video review comment
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
MARKER="<!-- QA_VIDEO_REVIEW_COMMENT -->"
|
||||
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
|
||||
# Combine per-platform reports
|
||||
REPORT=""
|
||||
for f in video-reviews/*-qa-video-report.md; do
|
||||
[ -f "$f" ] || continue
|
||||
[ -n "$REPORT" ] && REPORT="${REPORT}
|
||||
---
|
||||
"
|
||||
REPORT="${REPORT}$(cat "$f")"
|
||||
done
|
||||
|
||||
[ -z "$REPORT" ] && REPORT="No video reports were generated."
|
||||
|
||||
BODY="${MARKER}
|
||||
## QA Video Report
|
||||
|
||||
${REPORT}
|
||||
|
||||
---
|
||||
**Run**: [${RUN}](${RUN})"
|
||||
|
||||
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
|
||||
--jq ".[] | select(.body | contains(\"${MARKER}\")) | .id" | head -1)
|
||||
|
||||
if [ -n "$EXISTING" ]; then
|
||||
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
|
||||
--field body="$BODY"
|
||||
else
|
||||
gh pr comment ${{ github.event.pull_request.number }} \
|
||||
--repo ${{ github.repository }} --body "$BODY"
|
||||
fi
|
||||
@@ -26,7 +26,10 @@ const config: KnipConfig = {
|
||||
project: ['src/**/*.{js,ts}']
|
||||
},
|
||||
'packages/ingest-types': {
|
||||
project: ['src/**/*.{js,ts}']
|
||||
project: ['src/**/*.{js,ts}'],
|
||||
entry: ['src/index.ts', 'openapi-ts.config.ts'],
|
||||
// openapi-ts binary is provided by @hey-api/openapi-ts devDependency
|
||||
ignoreBinaries: ['openapi-ts']
|
||||
},
|
||||
'apps/website': {
|
||||
entry: [
|
||||
@@ -40,7 +43,7 @@ const config: KnipConfig = {
|
||||
ignoreDependencies: ['@comfyorg/design-system', '@vercel/analytics']
|
||||
}
|
||||
},
|
||||
ignoreBinaries: ['python3'],
|
||||
ignoreBinaries: ['python3', 'gh'],
|
||||
ignoreDependencies: [
|
||||
// Weird importmap things
|
||||
'@iconify-json/lucide',
|
||||
@@ -54,6 +57,8 @@ const config: KnipConfig = {
|
||||
// Auto generated API types
|
||||
'src/workbench/extensions/manager/types/generatedManagerTypes.ts',
|
||||
'packages/ingest-types/src/zod.gen.ts',
|
||||
// Used by a custom node (that should move off of this)
|
||||
'src/scripts/ui/components/splitButton.ts',
|
||||
// Used by stacked PR (feat/glsl-live-preview)
|
||||
'src/renderer/glsl/useGLSLRenderer.ts',
|
||||
// Workflow files contain license names that knip misinterprets as binaries
|
||||
|
||||
@@ -39,6 +39,7 @@
|
||||
"oxlint": "oxlint src --type-aware",
|
||||
"prepare": "husky || true && git config blame.ignoreRevsFile .git-blame-ignore-revs || true",
|
||||
"preview": "nx preview",
|
||||
"qa:video-review": "tsx scripts/qa-video-review.ts",
|
||||
"storybook": "nx storybook",
|
||||
"storybook:desktop": "nx run @comfyorg/desktop-ui:storybook",
|
||||
"stylelint:fix": "stylelint --cache --fix '{apps,packages,src}/**/*.{css,vue}'",
|
||||
|
||||
183
scripts/qa-video-review.test.ts
Normal file
183
scripts/qa-video-review.test.ts
Normal file
@@ -0,0 +1,183 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import {
|
||||
extractOutputText,
|
||||
extractPlatformFromArtifactDirName,
|
||||
pickLatestVideosByPlatform,
|
||||
selectVideoCandidateByFile
|
||||
} from './qa-video-review'
|
||||
|
||||
describe('extractPlatformFromArtifactDirName', () => {
|
||||
it('extracts and normalizes known qa artifact directory names', () => {
|
||||
expect(
|
||||
extractPlatformFromArtifactDirName('qa-report-Windows-22818315023')
|
||||
).toBe('windows')
|
||||
expect(
|
||||
extractPlatformFromArtifactDirName('qa-report-macOS-22818315023')
|
||||
).toBe('macos')
|
||||
expect(
|
||||
extractPlatformFromArtifactDirName('qa-report-Linux-22818315023')
|
||||
).toBe('linux')
|
||||
})
|
||||
|
||||
it('falls back to slugifying unknown directory names', () => {
|
||||
expect(extractPlatformFromArtifactDirName('custom platform run')).toBe(
|
||||
'custom-platform-run'
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('pickLatestVideosByPlatform', () => {
|
||||
it('keeps only the latest candidate per platform', () => {
|
||||
const selected = pickLatestVideosByPlatform([
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/windows-old.mp4',
|
||||
mtimeMs: 100
|
||||
},
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/windows-new.mp4',
|
||||
mtimeMs: 200
|
||||
},
|
||||
{
|
||||
platformName: 'linux',
|
||||
videoPath: '/tmp/linux.mp4',
|
||||
mtimeMs: 150
|
||||
}
|
||||
])
|
||||
|
||||
expect(selected).toEqual([
|
||||
{
|
||||
platformName: 'linux',
|
||||
videoPath: '/tmp/linux.mp4',
|
||||
mtimeMs: 150
|
||||
},
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/windows-new.mp4',
|
||||
mtimeMs: 200
|
||||
}
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('extractOutputText', () => {
|
||||
it('reads output_text when available', () => {
|
||||
expect(extractOutputText({ output_text: 'hello world' })).toBe(
|
||||
'hello world'
|
||||
)
|
||||
})
|
||||
|
||||
it('reads nested output text chunks', () => {
|
||||
const responseBody = {
|
||||
output: [
|
||||
{
|
||||
content: [
|
||||
{ type: 'output_text', text: 'first block' },
|
||||
{ type: 'refusal', refusal: 'not relevant' }
|
||||
]
|
||||
},
|
||||
{
|
||||
content: [{ type: 'output_text', text: 'second block' }]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
expect(extractOutputText(responseBody)).toBe('first block\n\nsecond block')
|
||||
})
|
||||
|
||||
it('returns null when there is no usable text', () => {
|
||||
expect(
|
||||
extractOutputText({ output: [{ content: [{ type: 'tool_call' }] }] })
|
||||
).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('selectVideoCandidateByFile', () => {
|
||||
it('selects a single candidate by artifacts-relative path', () => {
|
||||
const selected = selectVideoCandidateByFile(
|
||||
[
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
|
||||
mtimeMs: 100
|
||||
},
|
||||
{
|
||||
platformName: 'linux',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
|
||||
mtimeMs: 200
|
||||
}
|
||||
],
|
||||
{
|
||||
artifactsDir: '/tmp/qa-artifacts',
|
||||
videoFile: 'qa-report-Linux-1/qa-session.mp4'
|
||||
}
|
||||
)
|
||||
|
||||
expect(selected).toEqual({
|
||||
platformName: 'linux',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
|
||||
mtimeMs: 200
|
||||
})
|
||||
})
|
||||
|
||||
it('throws when basename matches multiple videos', () => {
|
||||
expect(() =>
|
||||
selectVideoCandidateByFile(
|
||||
[
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
|
||||
mtimeMs: 100
|
||||
},
|
||||
{
|
||||
platformName: 'linux',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
|
||||
mtimeMs: 200
|
||||
}
|
||||
],
|
||||
{
|
||||
artifactsDir: '/tmp/qa-artifacts',
|
||||
videoFile: 'qa-session.mp4'
|
||||
}
|
||||
)
|
||||
).toThrow('matched 2 videos')
|
||||
})
|
||||
|
||||
it('throws when there is no matching video', () => {
|
||||
expect(() =>
|
||||
selectVideoCandidateByFile(
|
||||
[
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
|
||||
mtimeMs: 100
|
||||
}
|
||||
],
|
||||
{
|
||||
artifactsDir: '/tmp/qa-artifacts',
|
||||
videoFile: 'qa-report-macOS-1/qa-session.mp4'
|
||||
}
|
||||
)
|
||||
).toThrow('No video matched')
|
||||
})
|
||||
|
||||
it('throws when video file is missing', () => {
|
||||
expect(() =>
|
||||
selectVideoCandidateByFile(
|
||||
[
|
||||
{
|
||||
platformName: 'windows',
|
||||
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
|
||||
mtimeMs: 100
|
||||
}
|
||||
],
|
||||
{
|
||||
artifactsDir: '/tmp/qa-artifacts',
|
||||
videoFile: ' '
|
||||
}
|
||||
)
|
||||
).toThrow('--video-file is required')
|
||||
})
|
||||
})
|
||||
795
scripts/qa-video-review.ts
Normal file
795
scripts/qa-video-review.ts
Normal file
@@ -0,0 +1,795 @@
|
||||
#!/usr/bin/env tsx
|
||||
import { spawn } from 'node:child_process'
|
||||
import {
|
||||
mkdtemp,
|
||||
mkdir,
|
||||
readdir,
|
||||
readFile,
|
||||
rm,
|
||||
stat,
|
||||
writeFile
|
||||
} from 'node:fs/promises'
|
||||
import { basename, dirname, join, relative, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { globSync } from 'glob'
|
||||
|
||||
interface CliOptions {
|
||||
artifactsDir: string
|
||||
videoFile: string
|
||||
outputDir: string
|
||||
model: string
|
||||
minIntervalSeconds: number
|
||||
maxFrames: number
|
||||
requestTimeoutMs: number
|
||||
dryRun: boolean
|
||||
keepFrames: boolean
|
||||
}
|
||||
|
||||
interface VideoCandidate {
|
||||
platformName: string
|
||||
videoPath: string
|
||||
mtimeMs: number
|
||||
}
|
||||
|
||||
interface ExtractedFrame {
|
||||
index: number
|
||||
timestampSeconds: number
|
||||
filePath: string
|
||||
dataUrl: string
|
||||
}
|
||||
|
||||
type ResponseInputContent =
|
||||
| { type: 'input_text'; text: string }
|
||||
| { type: 'input_image'; image_url: string }
|
||||
|
||||
interface ResponseInputMessage {
|
||||
role: 'system' | 'user'
|
||||
content: ResponseInputContent[]
|
||||
}
|
||||
|
||||
interface ResponsesCreatePayload {
|
||||
model: string
|
||||
input: ResponseInputMessage[]
|
||||
max_output_tokens: number
|
||||
}
|
||||
|
||||
interface OpenAIReviewRequest {
|
||||
apiKey: string
|
||||
baseUrl: string
|
||||
model: string
|
||||
platformName: string
|
||||
videoPath: string
|
||||
frames: ExtractedFrame[]
|
||||
samplingIntervalSeconds: number
|
||||
timeoutMs: number
|
||||
}
|
||||
|
||||
const DEFAULT_OPTIONS: CliOptions = {
|
||||
artifactsDir: './tmp/qa-artifacts',
|
||||
videoFile: '',
|
||||
outputDir: './tmp',
|
||||
model: 'gpt-4o',
|
||||
minIntervalSeconds: 5,
|
||||
maxFrames: 36,
|
||||
requestTimeoutMs: 300_000,
|
||||
dryRun: false,
|
||||
keepFrames: false
|
||||
}
|
||||
|
||||
const USAGE = `Usage:
|
||||
pnpm exec tsx scripts/qa-video-review.ts [options]
|
||||
|
||||
Options:
|
||||
--artifacts-dir <path> Artifacts root directory
|
||||
(default: ./tmp/qa-artifacts)
|
||||
--video-file <name-or-path> Video file to analyze (required)
|
||||
(supports basename or relative/absolute path)
|
||||
--output-dir <path> Output directory for markdown reports
|
||||
(default: ./tmp)
|
||||
--model <name> OpenAI model
|
||||
(default: gpt-4o)
|
||||
--min-interval-seconds <n> Minimum frame sampling interval in seconds
|
||||
(default: 5)
|
||||
--max-frames <n> Max frames analyzed per video
|
||||
(default: 36)
|
||||
--request-timeout-ms <n> Request timeout in milliseconds
|
||||
(default: 300000)
|
||||
--dry-run Discover videos and output targets only
|
||||
--keep-frames Keep extracted frames in ./tmp for inspection
|
||||
--help Show this help text
|
||||
|
||||
Environment:
|
||||
OPENAI_API_KEY Required unless --dry-run
|
||||
OPENAI_BASE_URL Optional override for API base URL
|
||||
`
|
||||
|
||||
function parsePositiveNumber(rawValue: string, flagName: string): number {
|
||||
const parsedValue = Number(rawValue)
|
||||
if (!Number.isFinite(parsedValue) || parsedValue <= 0) {
|
||||
throw new Error(`Invalid value for ${flagName}: "${rawValue}"`)
|
||||
}
|
||||
return parsedValue
|
||||
}
|
||||
|
||||
function parsePositiveInteger(rawValue: string, flagName: string): number {
|
||||
const parsedValue = Number.parseInt(rawValue, 10)
|
||||
if (!Number.isInteger(parsedValue) || parsedValue <= 0) {
|
||||
throw new Error(`Invalid value for ${flagName}: "${rawValue}"`)
|
||||
}
|
||||
return parsedValue
|
||||
}
|
||||
|
||||
function parseCliOptions(args: string[]): CliOptions {
|
||||
const options: CliOptions = { ...DEFAULT_OPTIONS }
|
||||
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const argument = args[index]
|
||||
const nextValue = args[index + 1]
|
||||
const requireValue = (flagName: string): string => {
|
||||
if (!nextValue || nextValue.startsWith('--')) {
|
||||
throw new Error(`Missing value for ${flagName}`)
|
||||
}
|
||||
index += 1
|
||||
return nextValue
|
||||
}
|
||||
|
||||
if (argument === '--help') {
|
||||
process.stdout.write(USAGE)
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
if (argument === '--artifacts-dir') {
|
||||
options.artifactsDir = requireValue(argument)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--video-file') {
|
||||
options.videoFile = requireValue(argument)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--output-dir') {
|
||||
options.outputDir = requireValue(argument)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--model') {
|
||||
options.model = requireValue(argument)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--min-interval-seconds') {
|
||||
options.minIntervalSeconds = parsePositiveNumber(
|
||||
requireValue(argument),
|
||||
argument
|
||||
)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--max-frames') {
|
||||
options.maxFrames = parsePositiveInteger(requireValue(argument), argument)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--request-timeout-ms') {
|
||||
options.requestTimeoutMs = parsePositiveInteger(
|
||||
requireValue(argument),
|
||||
argument
|
||||
)
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--dry-run') {
|
||||
options.dryRun = true
|
||||
continue
|
||||
}
|
||||
|
||||
if (argument === '--keep-frames') {
|
||||
options.keepFrames = true
|
||||
continue
|
||||
}
|
||||
|
||||
throw new Error(`Unknown argument: ${argument}`)
|
||||
}
|
||||
|
||||
return options
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null
|
||||
}
|
||||
|
||||
function normalizePlatformName(value: string): string {
|
||||
const slug = value
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
|
||||
return slug.length > 0 ? slug : 'unknown-platform'
|
||||
}
|
||||
|
||||
export function extractPlatformFromArtifactDirName(dirName: string): string {
|
||||
const matchedValue = dirName.match(/^qa-report-(.+?)(?:-\d+)?$/i)?.[1]
|
||||
return normalizePlatformName(matchedValue ?? dirName)
|
||||
}
|
||||
|
||||
function extractPlatformFromVideoPath(videoPath: string): string {
|
||||
const artifactDirName = basename(dirname(videoPath))
|
||||
return extractPlatformFromArtifactDirName(artifactDirName)
|
||||
}
|
||||
|
||||
export function pickLatestVideosByPlatform(
|
||||
candidates: VideoCandidate[]
|
||||
): VideoCandidate[] {
|
||||
const latestByPlatform = new Map<string, VideoCandidate>()
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const current = latestByPlatform.get(candidate.platformName)
|
||||
if (!current || candidate.mtimeMs > current.mtimeMs) {
|
||||
latestByPlatform.set(candidate.platformName, candidate)
|
||||
}
|
||||
}
|
||||
|
||||
return [...latestByPlatform.values()].sort((a, b) =>
|
||||
a.platformName.localeCompare(b.platformName)
|
||||
)
|
||||
}
|
||||
|
||||
function toProjectRelativePath(targetPath: string): string {
|
||||
const relativePath = relative(process.cwd(), targetPath)
|
||||
if (relativePath.startsWith('.')) {
|
||||
return relativePath
|
||||
}
|
||||
return `./${relativePath}`
|
||||
}
|
||||
|
||||
function errorToString(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
|
||||
function normalizePathForMatch(value: string): string {
|
||||
return value.replaceAll('\\', '/').replace(/^\.\/+/, '')
|
||||
}
|
||||
|
||||
export function selectVideoCandidateByFile(
|
||||
candidates: VideoCandidate[],
|
||||
options: { artifactsDir: string; videoFile: string }
|
||||
): VideoCandidate {
|
||||
const requestedValue = options.videoFile.trim()
|
||||
if (requestedValue.length === 0) {
|
||||
throw new Error('--video-file is required')
|
||||
}
|
||||
|
||||
const artifactsRoot = resolve(options.artifactsDir)
|
||||
const requestedAbsolutePath = resolve(requestedValue)
|
||||
const requestedPathKey = normalizePathForMatch(requestedValue)
|
||||
|
||||
const matches = candidates.filter((candidate) => {
|
||||
const candidateAbsolutePath = resolve(candidate.videoPath)
|
||||
if (candidateAbsolutePath === requestedAbsolutePath) {
|
||||
return true
|
||||
}
|
||||
|
||||
const candidateBaseName = basename(candidate.videoPath)
|
||||
if (candidateBaseName === requestedValue) {
|
||||
return true
|
||||
}
|
||||
|
||||
const relativeToCwd = normalizePathForMatch(
|
||||
relative(process.cwd(), candidateAbsolutePath)
|
||||
)
|
||||
if (relativeToCwd === requestedPathKey) {
|
||||
return true
|
||||
}
|
||||
|
||||
const relativeToArtifacts = normalizePathForMatch(
|
||||
relative(artifactsRoot, candidateAbsolutePath)
|
||||
)
|
||||
return relativeToArtifacts === requestedPathKey
|
||||
})
|
||||
|
||||
if (matches.length === 1) {
|
||||
return matches[0]
|
||||
}
|
||||
|
||||
if (matches.length === 0) {
|
||||
const availableVideos = candidates.map((candidate) =>
|
||||
toProjectRelativePath(candidate.videoPath)
|
||||
)
|
||||
throw new Error(
|
||||
[
|
||||
`No video matched --video-file "${options.videoFile}".`,
|
||||
'Available videos:',
|
||||
...availableVideos.map((videoPath) => `- ${videoPath}`)
|
||||
].join('\n')
|
||||
)
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
[
|
||||
`--video-file "${options.videoFile}" matched ${matches.length} videos.`,
|
||||
'Please pass a more specific path.',
|
||||
...matches.map((match) => `- ${toProjectRelativePath(match.videoPath)}`)
|
||||
].join('\n')
|
||||
)
|
||||
}
|
||||
|
||||
async function runCommand(command: string, args: string[]): Promise<string> {
|
||||
return await new Promise<string>((resolvePromise, rejectPromise) => {
|
||||
const child = spawn(command, args, {
|
||||
stdio: ['ignore', 'pipe', 'pipe']
|
||||
})
|
||||
|
||||
let stdout = ''
|
||||
let stderr = ''
|
||||
|
||||
child.stdout.on('data', (chunk: Buffer | string) => {
|
||||
stdout += chunk.toString()
|
||||
})
|
||||
|
||||
child.stderr.on('data', (chunk: Buffer | string) => {
|
||||
stderr += chunk.toString()
|
||||
})
|
||||
|
||||
child.on('error', (error) => {
|
||||
rejectPromise(error)
|
||||
})
|
||||
|
||||
child.on('close', (code) => {
|
||||
if (code === 0) {
|
||||
resolvePromise(stdout)
|
||||
return
|
||||
}
|
||||
|
||||
const details = stderr.trim() || stdout.trim() || 'unknown failure'
|
||||
rejectPromise(
|
||||
new Error(
|
||||
`${command} ${args.join(' ')} failed with code ${String(code)}: ${details}`
|
||||
)
|
||||
)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async function ensureBinariesAvailable(): Promise<void> {
|
||||
await runCommand('ffmpeg', ['-version'])
|
||||
await runCommand('ffprobe', ['-version'])
|
||||
}
|
||||
|
||||
async function getVideoDurationSeconds(
|
||||
videoPath: string
|
||||
): Promise<number | null> {
|
||||
try {
|
||||
const stdout = await runCommand('ffprobe', [
|
||||
'-v',
|
||||
'error',
|
||||
'-show_entries',
|
||||
'format=duration',
|
||||
'-of',
|
||||
'default=noprint_wrappers=1:nokey=1',
|
||||
videoPath
|
||||
])
|
||||
const durationSeconds = Number.parseFloat(stdout.trim())
|
||||
if (!Number.isFinite(durationSeconds) || durationSeconds <= 0) {
|
||||
return null
|
||||
}
|
||||
return durationSeconds
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function computeSamplingIntervalSeconds(
|
||||
durationSeconds: number | null,
|
||||
minIntervalSeconds: number,
|
||||
maxFrames: number
|
||||
): number {
|
||||
if (durationSeconds === null) {
|
||||
return minIntervalSeconds
|
||||
}
|
||||
|
||||
const fullCoverageInterval = durationSeconds / maxFrames
|
||||
return Math.max(minIntervalSeconds, fullCoverageInterval)
|
||||
}
|
||||
|
||||
async function collectVideoCandidates(
|
||||
artifactsDir: string
|
||||
): Promise<VideoCandidate[]> {
|
||||
const absoluteArtifactsDir = resolve(artifactsDir)
|
||||
const videoPaths = globSync('**/qa-session.mp4', {
|
||||
cwd: absoluteArtifactsDir,
|
||||
absolute: true,
|
||||
nodir: true
|
||||
}).sort()
|
||||
|
||||
const candidates = await Promise.all(
|
||||
videoPaths.map(async (videoPath) => {
|
||||
const videoStat = await stat(videoPath)
|
||||
return {
|
||||
platformName: extractPlatformFromVideoPath(videoPath),
|
||||
videoPath,
|
||||
mtimeMs: videoStat.mtimeMs
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
return candidates
|
||||
}
|
||||
|
||||
async function extractFramesFromVideo(
|
||||
videoPath: string,
|
||||
stagingDir: string,
|
||||
minIntervalSeconds: number,
|
||||
maxFrames: number
|
||||
): Promise<{ frames: ExtractedFrame[]; samplingIntervalSeconds: number }> {
|
||||
const durationSeconds = await getVideoDurationSeconds(videoPath)
|
||||
const samplingIntervalSeconds = computeSamplingIntervalSeconds(
|
||||
durationSeconds,
|
||||
minIntervalSeconds,
|
||||
maxFrames
|
||||
)
|
||||
const outputPattern = join(stagingDir, 'frame-%03d.jpg')
|
||||
|
||||
await runCommand('ffmpeg', [
|
||||
'-hide_banner',
|
||||
'-loglevel',
|
||||
'error',
|
||||
'-y',
|
||||
'-i',
|
||||
videoPath,
|
||||
'-vf',
|
||||
`fps=1/${samplingIntervalSeconds},scale=1024:-2:force_original_aspect_ratio=decrease`,
|
||||
'-frames:v',
|
||||
String(maxFrames),
|
||||
outputPattern
|
||||
])
|
||||
|
||||
const frameNames = (await readdir(stagingDir))
|
||||
.filter((name) => name.startsWith('frame-') && name.endsWith('.jpg'))
|
||||
.sort()
|
||||
|
||||
if (frameNames.length === 0) {
|
||||
throw new Error(`No frames were extracted from ${videoPath}`)
|
||||
}
|
||||
|
||||
const frames = await Promise.all(
|
||||
frameNames.map(async (frameName, index) => {
|
||||
const framePath = join(stagingDir, frameName)
|
||||
const buffer = await readFile(framePath)
|
||||
return {
|
||||
index: index + 1,
|
||||
timestampSeconds: Number((index * samplingIntervalSeconds).toFixed(2)),
|
||||
filePath: framePath,
|
||||
dataUrl: `data:image/jpeg;base64,${buffer.toString('base64')}`
|
||||
}
|
||||
})
|
||||
)
|
||||
|
||||
return { frames, samplingIntervalSeconds }
|
||||
}
|
||||
|
||||
function buildReviewMessages(
|
||||
request: OpenAIReviewRequest
|
||||
): ResponseInputMessage[] {
|
||||
const systemPrompt = [
|
||||
'You are a senior QA engineer reviewing a UI test session.',
|
||||
'Report only concrete, visible problems and avoid speculation.',
|
||||
'If confidence is low, mark it explicitly.'
|
||||
].join(' ')
|
||||
|
||||
const introPrompt = [
|
||||
`Review sampled frames from a QA session for platform "${request.platformName}".`,
|
||||
`Source video: ${toProjectRelativePath(request.videoPath)}.`,
|
||||
`Frames are presented in chronological order at ~${request.samplingIntervalSeconds.toFixed(2)}s intervals.`,
|
||||
'Focus on UI regressions, broken states, visual glitches, unreadable text, missing labels/i18n, and clear workflow failures.',
|
||||
'Return markdown with these sections exactly:',
|
||||
'## Summary',
|
||||
'## Confirmed Issues',
|
||||
'## Possible Issues (Needs Human Verification)',
|
||||
'## Overall Risk',
|
||||
'Under Confirmed Issues include a markdown table with columns:',
|
||||
'Severity | Timestamp (s) | Issue | Evidence | Confidence | Suggested Fix'
|
||||
].join('\n')
|
||||
|
||||
const userContent: ResponseInputContent[] = [
|
||||
{ type: 'input_text', text: introPrompt }
|
||||
]
|
||||
|
||||
for (const frame of request.frames) {
|
||||
userContent.push({
|
||||
type: 'input_text',
|
||||
text: `Frame ${frame.index} at approximately ${frame.timestampSeconds}s`
|
||||
})
|
||||
userContent.push({
|
||||
type: 'input_image',
|
||||
image_url: frame.dataUrl
|
||||
})
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
role: 'system',
|
||||
content: [{ type: 'input_text', text: systemPrompt }]
|
||||
},
|
||||
{
|
||||
role: 'user',
|
||||
content: userContent
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
function normalizeBaseUrl(baseUrl: string): string {
|
||||
return baseUrl.endsWith('/') ? baseUrl.slice(0, -1) : baseUrl
|
||||
}
|
||||
|
||||
function extractApiErrorMessage(responseBody: unknown): string {
|
||||
if (!isRecord(responseBody)) {
|
||||
return 'Unknown API error'
|
||||
}
|
||||
|
||||
const errorValue = responseBody.error
|
||||
if (!isRecord(errorValue)) {
|
||||
return 'Unknown API error'
|
||||
}
|
||||
|
||||
const messageValue = errorValue.message
|
||||
if (typeof messageValue !== 'string' || messageValue.length === 0) {
|
||||
return 'Unknown API error'
|
||||
}
|
||||
|
||||
return messageValue
|
||||
}
|
||||
|
||||
export function extractOutputText(responseBody: unknown): string | null {
|
||||
if (!isRecord(responseBody)) {
|
||||
return null
|
||||
}
|
||||
|
||||
const topLevelOutputText = responseBody.output_text
|
||||
if (
|
||||
typeof topLevelOutputText === 'string' &&
|
||||
topLevelOutputText.trim().length > 0
|
||||
) {
|
||||
return topLevelOutputText.trim()
|
||||
}
|
||||
|
||||
const outputValue = responseBody.output
|
||||
if (!Array.isArray(outputValue)) {
|
||||
return null
|
||||
}
|
||||
|
||||
const collectedParts: string[] = []
|
||||
for (const outputItem of outputValue) {
|
||||
if (!isRecord(outputItem)) {
|
||||
continue
|
||||
}
|
||||
|
||||
const contentValue = outputItem.content
|
||||
if (!Array.isArray(contentValue)) {
|
||||
continue
|
||||
}
|
||||
|
||||
for (const contentItem of contentValue) {
|
||||
if (!isRecord(contentItem)) {
|
||||
continue
|
||||
}
|
||||
|
||||
if (contentItem.type !== 'output_text') {
|
||||
continue
|
||||
}
|
||||
|
||||
const textValue = contentItem.text
|
||||
if (typeof textValue === 'string' && textValue.trim().length > 0) {
|
||||
collectedParts.push(textValue.trim())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (collectedParts.length === 0) {
|
||||
return null
|
||||
}
|
||||
|
||||
return collectedParts.join('\n\n')
|
||||
}
|
||||
|
||||
async function requestOpenAIReview(
|
||||
request: OpenAIReviewRequest
|
||||
): Promise<string> {
|
||||
const payload: ResponsesCreatePayload = {
|
||||
model: request.model,
|
||||
input: buildReviewMessages(request),
|
||||
max_output_tokens: 3_000
|
||||
}
|
||||
|
||||
const controller = new AbortController()
|
||||
const timeoutHandle = setTimeout(() => controller.abort(), request.timeoutMs)
|
||||
|
||||
try {
|
||||
const response = await fetch(
|
||||
`${normalizeBaseUrl(request.baseUrl)}/responses`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${request.apiKey}`,
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
body: JSON.stringify(payload),
|
||||
signal: controller.signal
|
||||
}
|
||||
)
|
||||
|
||||
const rawResponse = await response.text()
|
||||
let responseBody: unknown
|
||||
try {
|
||||
responseBody = JSON.parse(rawResponse)
|
||||
} catch {
|
||||
throw new Error(
|
||||
`OpenAI API returned a non-JSON response: ${rawResponse.slice(0, 400)}`
|
||||
)
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`OpenAI API request failed (${response.status}): ${extractApiErrorMessage(responseBody)}`
|
||||
)
|
||||
}
|
||||
|
||||
const reviewText = extractOutputText(responseBody)
|
||||
if (!reviewText) {
|
||||
throw new Error('OpenAI API returned no output text')
|
||||
}
|
||||
|
||||
return reviewText
|
||||
} finally {
|
||||
clearTimeout(timeoutHandle)
|
||||
}
|
||||
}
|
||||
|
||||
function buildReportMarkdown(input: {
|
||||
platformName: string
|
||||
model: string
|
||||
videoPath: string
|
||||
frames: ExtractedFrame[]
|
||||
samplingIntervalSeconds: number
|
||||
reviewText: string
|
||||
}): string {
|
||||
const header = [
|
||||
`# ${input.platformName} QA Video Report`,
|
||||
'',
|
||||
`- Generated at: ${new Date().toISOString()}`,
|
||||
`- Model: \`${input.model}\``,
|
||||
`- Source video: \`${toProjectRelativePath(input.videoPath)}\``,
|
||||
`- Frames analyzed: ${input.frames.length}`,
|
||||
`- Sampling interval: ${input.samplingIntervalSeconds.toFixed(2)}s`,
|
||||
'',
|
||||
'## AI Review',
|
||||
''
|
||||
].join('\n')
|
||||
|
||||
return `${header}${input.reviewText.trim()}\n`
|
||||
}
|
||||
|
||||
async function reviewVideo(
|
||||
video: VideoCandidate,
|
||||
options: CliOptions,
|
||||
apiKey: string,
|
||||
baseUrl: string
|
||||
): Promise<void> {
|
||||
const stagingRoot = resolve('./tmp')
|
||||
await mkdir(stagingRoot, { recursive: true })
|
||||
const stagingDir = await mkdtemp(join(stagingRoot, 'qa-video-review-'))
|
||||
|
||||
try {
|
||||
process.stdout.write(
|
||||
`[${video.platformName}] Extracting frames from ${toProjectRelativePath(video.videoPath)}\n`
|
||||
)
|
||||
|
||||
const { frames, samplingIntervalSeconds } = await extractFramesFromVideo(
|
||||
video.videoPath,
|
||||
stagingDir,
|
||||
options.minIntervalSeconds,
|
||||
options.maxFrames
|
||||
)
|
||||
|
||||
process.stdout.write(
|
||||
`[${video.platformName}] Sending ${frames.length} frame(s) to ${options.model}\n`
|
||||
)
|
||||
|
||||
const reviewText = await requestOpenAIReview({
|
||||
apiKey,
|
||||
baseUrl,
|
||||
model: options.model,
|
||||
platformName: video.platformName,
|
||||
videoPath: video.videoPath,
|
||||
frames,
|
||||
samplingIntervalSeconds,
|
||||
timeoutMs: options.requestTimeoutMs
|
||||
})
|
||||
|
||||
const outputPath = resolve(
|
||||
options.outputDir,
|
||||
`${video.platformName}-qa-video-report.md`
|
||||
)
|
||||
const reportMarkdown = buildReportMarkdown({
|
||||
platformName: video.platformName,
|
||||
model: options.model,
|
||||
videoPath: video.videoPath,
|
||||
frames,
|
||||
samplingIntervalSeconds,
|
||||
reviewText
|
||||
})
|
||||
|
||||
await mkdir(dirname(outputPath), { recursive: true })
|
||||
await writeFile(outputPath, reportMarkdown, 'utf-8')
|
||||
|
||||
process.stdout.write(
|
||||
`[${video.platformName}] Wrote ${toProjectRelativePath(outputPath)}\n`
|
||||
)
|
||||
|
||||
if (options.keepFrames) {
|
||||
const stagedFrames = (await readdir(stagingDir)).filter((name) =>
|
||||
name.endsWith('.jpg')
|
||||
).length
|
||||
process.stdout.write(
|
||||
`[${video.platformName}] Kept ${stagedFrames} frame(s) in ${toProjectRelativePath(stagingDir)}\n`
|
||||
)
|
||||
return
|
||||
}
|
||||
} finally {
|
||||
if (!options.keepFrames) {
|
||||
await rm(stagingDir, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function isExecutedAsScript(metaUrl: string): boolean {
|
||||
const modulePath = fileURLToPath(metaUrl)
|
||||
const scriptPath = process.argv[1] ? resolve(process.argv[1]) : ''
|
||||
return modulePath === scriptPath
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const options = parseCliOptions(process.argv.slice(2))
|
||||
const candidates = await collectVideoCandidates(options.artifactsDir)
|
||||
|
||||
if (candidates.length === 0) {
|
||||
process.stdout.write(
|
||||
`No qa-session.mp4 files found under ${toProjectRelativePath(resolve(options.artifactsDir))}\n`
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
const selectedVideo = selectVideoCandidateByFile(candidates, {
|
||||
artifactsDir: options.artifactsDir,
|
||||
videoFile: options.videoFile
|
||||
})
|
||||
|
||||
process.stdout.write(
|
||||
`Selected ${selectedVideo.platformName}: ${toProjectRelativePath(selectedVideo.videoPath)}\n`
|
||||
)
|
||||
|
||||
if (options.dryRun) {
|
||||
process.stdout.write('\nDry run mode enabled, no API calls were made.\n')
|
||||
return
|
||||
}
|
||||
|
||||
const apiKey = process.env.OPENAI_API_KEY
|
||||
if (!apiKey) {
|
||||
throw new Error('OPENAI_API_KEY is required unless --dry-run is set')
|
||||
}
|
||||
|
||||
await ensureBinariesAvailable()
|
||||
|
||||
const baseUrl = process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1'
|
||||
await reviewVideo(selectedVideo, options, apiKey, baseUrl)
|
||||
}
|
||||
|
||||
if (isExecutedAsScript(import.meta.url)) {
|
||||
void main().catch((error: unknown) => {
|
||||
const message = errorToString(error)
|
||||
process.stderr.write(`qa-video-review failed: ${message}\n`)
|
||||
process.exit(1)
|
||||
})
|
||||
}
|
||||
Reference in New Issue
Block a user