Compare commits

..

51 Commits

Author SHA1 Message Date
snomiao
fb5813dddb feat: richer fixture API hints — i18n, queue mocks, subgraph helpers
- Add Comfy.Locale setting ID for i18n language switching tests
- Add queue/assets mock API (mockOutputHistory, runButton)
- Add subgraph workflow asset paths for loadWorkflow
- Add SubgraphHelper API docs (slot ops, navigation, conversion)
- Add VueNodeHelpers enterSubgraph/selectNode
2026-04-15 16:40:27 +00:00
snomiao
e153e58c91 feat: smarter agent — precondition reasoning + issue comments
- Prompt agent to reason about hidden preconditions before writing tests
  (e.g. z-index bugs need crowded canvas, not empty default workflow)
- Fetch issue comments with reproduction hints (repro/step/workaround)
- Better error analysis: different strategy on retry, not same code
- Both CI workflow and pnpm qa CLI fetch comments
2026-04-15 16:31:20 +00:00
snomiao
cdea8bf2c9 fix: Opus escalation graceful fallback on credit exhaustion
When Opus API call fails (credit balance, rate limit), keep Sonnet's
result instead of overwriting with INCONCLUSIVE API error. Only use
Opus result if it's actually better than Sonnet's attempt.
2026-04-14 15:37:07 +00:00
snomiao
a2da58eb0f feat: Opus escalation for INCONCLUSIVE issues
Sonnet tries first. If INCONCLUSIVE, automatically retries with
claude-opus-4-6 (30 turns). Disable with QA_OPUS_ESCALATION=0.
Also: model param added to ResearchOptions for flexibility.
2026-04-14 13:14:33 +00:00
snomiao
3154865ce2 feat: Phase 1 improvements — concurrency, auto-trigger, better prompts
- B1: Fix concurrency group to use ref_name (parallel sno-qa-* branches)
- D1: Auto-trigger QA on 'Potential Bug' and 'verified bug' labels
- A4: Prompt agent to read existing tests first before writing
- Turn budget enforcement from previous commit
2026-04-14 13:12:49 +00:00
snomiao
ff6034e2ee fix: reduce INCONCLUSIVE rate — enforce turn budget and fail-fast
- 3 consecutive test failures → call done(NOT_REPRODUCIBLE)
- Turn budget: ~3 inspect, 2 write, 3 fix = ~10 tool calls max
- Prevents 20+ tool call retry loops that waste CI time
2026-04-13 19:41:54 +00:00
snomiao
529ac3cea4 trigger: re-run cancelled batch 2 2026-04-13 18:42:20 +00:00
snomiao
f95eebf3db trigger: re-run cancelled QA batches 2026-04-13 17:49:03 +00:00
snomiao
1dd66315ed docs: update SKILL.md with current CLI and architecture 2026-04-13 16:21:02 +00:00
snomiao
83de5a222e feat: use demowright showTitleCard API for early title overlay
- Import showTitleCard/hideTitleCard from demowright/video-script
- Replace page.evaluate() hack with official demowright API
- CI clones demowright feat/show-title-card-api branch
- demowright PR: https://github.com/snomiao/demowright/pull/11
2026-04-13 09:24:44 +00:00
snomiao
2faadaeab0 fix: early title card covers setup, remove unstable ffmpeg trim
Show title card via page.evaluate() IMMEDIATELY before setup code runs.
Setup (setSetting, setupWorkflowsDirectory) executes behind the card.
Card is removed before createVideoScript() renders its own title.
This ensures the title card is visible from the first frame of the video.
2026-04-13 09:18:43 +00:00
snomiao
cb921ada71 fix: remove autoplay so browser plays video with sound 2026-04-13 07:21:29 +00:00
snomiao
884270c46f feat: show verdict banner with failure reason for non-reproduced bugs
NOT_REPRODUCIBLE and INCONCLUSIVE verdicts now display a prominent
banner with the agent's summary and evidence explaining why the bug
could not be reproduced. Default video playback speed changed to 1x.
2026-04-13 06:27:05 +00:00
snomiao
51e48b55b3 fix: default video playback speed 1x instead of 0.5x 2026-04-13 06:19:56 +00:00
snomiao
07f6611cc8 trigger: re-run QA for #10766 2026-04-12 13:51:17 +00:00
snomiao
55acfdd43b feat: video trim, deploy video-script, require videoScript, verdict mismatch
1. ffmpeg -ss 7: trim ComfyUI loading screen from demo video
2. Deploy video-script.spec.ts to report site + show in HTML
3. videoScript is REQUIRED when verdict=REPRODUCED (prompt + tool desc)
4. Log warning when E2E and video review verdicts disagree
2026-04-12 10:25:18 +00:00
snomiao
09ea4ae302 fix: title card first — setup before createVideoScript
- Move all setup code (setSetting, setupWorkflowsDirectory) BEFORE
  createVideoScript() so title card is first frame in video
- Add explicit rules: pace() first, waitForTimeout after actions,
  5s hold on final evidence
2026-04-11 23:51:18 +00:00
snomiao
d698347dfd fix: narrate-then-act pattern for video scripts
Call pace() FIRST in each segment so narration finishes before actions.
Add waitForTimeout(2000) after actions for viewer comprehension.
2026-04-11 23:17:17 +00:00
snomiao
7129c2d702 fix: add demowright/video-script exports + separate videoScript tool
- Patch demowright package.json exports in CI to add ./video-script entry
- Add videoScript param to done() tool so Claude provides separate video script
- Phase 2 uses video-script.spec.ts when available
2026-04-11 21:40:02 +00:00
snomiao
2d8c71d683 feat: separate videoScript in done() tool for Phase 2
- Add videoScript optional param to done() tool
- Claude writes test code (no demowright) + video script (with demowright) separately
- Phase 2 uses video-script.spec.ts if agent provided it, else fallback
- Fixes demowright import failures in Phase 1 (test env has no demowright config)
2026-04-11 20:40:52 +00:00
snomiao
231878918d feat: teach Claude to use demowright createVideoScript
- Add createVideoScript API docs to qa-agent system prompt
- Claude now writes tests with native video scripts (title, segment, outro)
- Phase 2 detects createVideoScript in test code → skip regex injection
- Fallback: still injects annotate() for legacy tests without video scripts
2026-04-11 18:28:57 +00:00
snomiao
5228245561 fix: safe step narration + research log in report
- Replace broken annotate() code-wrapping with safe narrate-before-step
  (inserts narrate call before each step comment, no code wrapping)
- Show research-log.json and reproduce.spec.ts in report HTML
- Fix Phase 2 "No tests found" caused by syntax errors in injected code
2026-04-11 16:37:43 +00:00
snomiao
03f2e1d183 fix: intro subtitle + slower demo video
- Use annotate() instead of narrate() for intro (shows subtitle + TTS)
- Increase actionDelay from 500ms to 1500ms for human-readable pacing
- Add testcloud.comfy.org as cloud fallback server
- Fix knip ignoreDependencies
2026-04-10 23:21:52 +00:00
snomiao
a3271cfde8 fix: remove stale knip ignoreDependencies for QA deps 2026-04-10 22:29:22 +00:00
snomiao
3eb8ed0748 feat: auto-clone ComfyUI + findComfyUIDir fallback chain
Server startup: comfy-cli → TEST_COMFYUI_DIR → .comfy-qa/ComfyUI →
auto-clone from GitHub as last resort. Health check before QA run.
2026-04-10 22:27:28 +00:00
snomiao
faac186ebe feat: auto-start ComfyUI server via comfy-cli or python
Tries: comfy launch --background --cpu → python main.py --cpu →
clear error with setup instructions. Health check with 120s timeout.
2026-04-10 22:24:51 +00:00
snomiao
d96d582ce2 fix: use REST API for issue/PR detection
gh pr view --json number falsely matches issues. Use gh api
repos/{repo}/issues/{n} with has("pull_request") for reliable detection.
2026-04-10 21:35:23 +00:00
snomiao
76ef88f18b refactor: tidy QA CLI — extract functions, validate target, dedup
- Extract fetchIssue/fetchPR/resolveTarget/detectType helpers
- Move constants to top (SCRIPT_DIR, RECORD_SCRIPT, DEFAULT_REPO)
- Add VALID_TARGETS validation for -t/--target
- Shared writeTmpFile/resolveOutputDir/logHeader/exit utilities
- Dispatch via runUncommitted/runTarget for clearer flow
2026-04-10 21:14:46 +00:00
snomiao
b42fd03213 fix: show help on parse errors for agent-friendly CLI 2026-04-10 20:44:43 +00:00
snomiao
e22daf415a feat: rewrite QA CLI with node:util parseArgs
- Replace hand-written arg parser with parseArgs (zero deps)
- Add -t/--target <head|base|both> for PR phase selection
- Add --uncommitted for testing local changes
- Cleaner help output with examples for all modes
2026-04-10 20:43:22 +00:00
snomiao
86e61d7f16 feat: add --base flag for PR before/after testing
PR default: test head only (demonstrate the fix)
PR --base: also reproduce bug on base before testing head
Issue: always runs reproduce mode
2026-04-10 20:36:16 +00:00
snomiao
218e44d112 feat: auto-detect issue vs PR + save PR base/head refs
- Remove pr: prefix, just `pnpm qa 10253` auto-detects via gh API
- Fetch PR baseRefOid/headRefOid and save to refs.json
- Output dir simplified to .comfy-qa/<number>/
- Extract runQaRecord() for future before/after dual-phase PR testing
2026-04-10 20:34:11 +00:00
snomiao
14187df785 feat: simplified QA CLI + move scripts to skill folder
- Add `pnpm qa <url|number>` entry point with dotenv auto-loading
  Accepts GitHub issue/PR URLs, shorthand (10253, pr:10270)
  Auto-fetches data via `gh` CLI, outputs to .comfy-qa/
- Move qa-* scripts from scripts/ to .claude/skills/comfy-qa/scripts/
- Fix TypeScript errors: add type annotations, fix browserTestFile ref
- Add .comfy-qa/ to .gitignore
2026-04-10 20:12:08 +00:00
snomiao
d821a94c59 docs: add QA skill env vars to .env_example
GEMINI_API_KEY (required) — PR analysis, video review, TTS narration
ANTHROPIC_API_KEY (optional locally) — Claude Agent SDK auto-detects
Claude Code session; required in CI as GitHub Actions secret

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 19:54:42 +00:00
snomiao
4fd9d723e0 feat: annotate() wraps each test step with TTS narration + subtitle
Uses demowright's annotate(page, text, callback) pattern to narrate
each "// Step N: ..." block while running the actions in parallel.
The viewer hears "Step 1: Save the workflow" while seeing the save
happen. Falls back to running the code directly if annotate fails.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 19:50:16 +00:00
snomiao
448c76c318 feat: autoplay with sound, remove muted default
Videos now autoplay with TTS narration audible by default.
Removed muted attribute from video tags and the JS pause-on-load.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 19:47:46 +00:00
snomiao
691242afcd feat: enable demowright TTS narration + audio capture + autoAnnotate
- audio: true — captures browser audio + TTS narration
- autoAnnotate: true — auto-generates narration from test steps
- actionDelay: 500ms (was 300ms) for more readable pacing
- Gemini TTS auto-detected via GEMINI_API_KEY in register.cjs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 17:20:50 +00:00
snomiao
49efcfe36a fix: regenerate lockfile with new deps in workspace catalog
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 11:27:13 +00:00
snomiao
60bf788050 fix: use withDemowright() config + add @google/generative-ai dep
- Switch from broken register.cjs to withDemowright() config wrapper
  that creates temp playwright.qa.config.ts for Phase 2 recording
- Add @google/generative-ai as explicit dev dependency (was only
  transitive, causing ESLint import-x/no-unresolved error)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 11:22:26 +00:00
snomiao
14e194ab34 fix: add --no-sandbox to PLAYWRIGHT_LOCAL launch options
Headless Chrome crashes in some environments without sandbox flags.
Only applies when PLAYWRIGHT_LOCAL=1 (QA Phase 2 recording).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 08:56:26 +00:00
snomiao
8664d2201e fix: use demowright from npm (^2.0.9) instead of local link
The local link (link:/tmp/demowright-test) doesn't exist in CI.
Changed to npm package for HUD overlay in QA test videos:
- Visible cursor with click ripples
- Keystroke display as HUD badges
- Auto-slowdown between actions
- Subtitles and TTS narration support

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 08:51:10 +00:00
sno
c1f7e03c1c feat: enable demowright TTS audio capture in QA videos
- Set QA_HUD_AUDIO=1 to enable demowright audio pipeline
- Install ffmpeg in qa-before/after jobs for audio muxing
- After Phase 2, check .demowright/ for rendered MP4 with audio
- Add narration detection to Gemini video review prompt
- Increase Phase 2 timeout for TTS fetch latency

Amp-Thread-ID: https://ampcode.com/threads/T-019d7181-cd11-7255-9f13-fecc658d2751
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
10d6e93197 fix: patch demowright register.cjs to use dist/setup.mjs, simplify Phase 2
register.cjs imports ./src/setup.ts (TS source) which doesn't work
when installed as a dependency. Patch it to ./dist/setup.mjs after build.

Simplify Phase 2 back to --require demowright/register approach (avoids
ESM/CJS issues with withDemowright config).
2026-04-10 08:51:10 +00:00
snomiao
cc85e699ae feat: use demowright withDemowright() config for proper HUD integration
Uses withDemowright() to wrap a dedicated Playwright config instead of
NODE_OPTIONS register approach. This properly patches Browser.newContext
so cursor overlay, keystroke badges, and action delays work reliably.

Injects annotate() title card showing the issue title at video start.
Falls back to register approach if demowright config fails.

Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
2394016036 feat: inject demowright annotate() title card into reproduce videos
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
72abaefe42 chore: retrigger QA runs 2026-04-10 08:51:10 +00:00
snomiao
d5b6ceead7 fix: install demowright from GitHub repo for latest unpublished changes
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
94561270ef fix: make analyze-pr non-blocking, use apt for ffmpeg
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
e04d050ad3 fix: use apt-get for ffmpeg install (johnvansickle.com unreliable)
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
c930bbaf8e feat: use demowright for reproduce videos, combine E2E+video verdicts
- Replace hand-rolled cursor overlay and waitForTimeout injection with
  demowright's automatic HUD (cursor, keystroke display, 300ms delay)
- Install demowright in both qa-before and qa-after CI jobs
- Always check video review verdicts (not just as fallback)
- Upgrade reproducedBy to 'both' when E2E and video agree
- Prevent false REPRODUCED from discovery/debug tests
- Ban discovery tests in agent system prompt

Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
snomiao
7520a47b5a feat: add automated QA pipeline with E2E test-driven bug reproduction
Three-phase pipeline triggered by labels (qa-changes, qa-full, qa-issue):
1. Research: Claude writes Playwright E2E tests to reproduce reported bugs
2. Reproduce: Deterministic replay with video recording
3. Report: Deploy results to Cloudflare Pages with badges

Key design decisions:
- Playwright assertions are source of truth (not AI vision)
- Agent has readFixture/readTest tools to discover project patterns
- Bug-specific assertions required (trivial assertions banned)
- Main branch dist cached by SHA to speed up before/after comparisons
- QA deps installed inline in CI (no package.json changes needed)

Verified across 48 runs (22 PRs + 26 issues) with 0 false positives.

Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019d519b-004f-71ce-b970-96edd971fbe0
Co-authored-by: Amp <amp@ampcode.com>
2026-04-10 08:51:10 +00:00
1247 changed files with 21976 additions and 86788 deletions

View File

@@ -1,86 +1,9 @@
{
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"if": "Bash(tsc *)",
"command": "echo 'Use `pnpm typecheck` instead of running tsc directly.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(vue-tsc *)",
"command": "echo 'Use `pnpm typecheck` instead of running vue-tsc directly.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx tsc *)",
"command": "echo 'Use `pnpm typecheck` instead of running tsc via npx.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpx tsc *)",
"command": "echo 'Use `pnpm typecheck` instead of running tsc via pnpx.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpm exec tsc *)",
"command": "echo 'Use `pnpm typecheck` instead of `pnpm exec tsc`.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx vitest *)",
"command": "echo 'Use `pnpm test:unit` (or `pnpm test:unit -- <path>`) instead of npx vitest.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpx vitest *)",
"command": "echo 'Use `pnpm test:unit` (or `pnpm test:unit -- <path>`) instead of pnpx vitest.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx eslint *)",
"command": "echo 'Use `pnpm lint` or `pnpm lint:fix` instead of npx eslint.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpx eslint *)",
"command": "echo 'Use `pnpm lint` or `pnpm lint:fix` instead of pnpx eslint.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx prettier *)",
"command": "echo 'This project uses oxfmt, not prettier. Use `pnpm format` or `pnpm format:check`.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpx prettier *)",
"command": "echo 'This project uses oxfmt, not prettier. Use `pnpm format` or `pnpm format:check`.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx oxlint *)",
"command": "echo 'Use `pnpm oxlint` instead of npx oxlint.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx stylelint *)",
"command": "echo 'Use `pnpm stylelint` instead of npx stylelint.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(npx knip *)",
"command": "echo 'Use `pnpm knip` instead of npx knip.' >&2 && exit 2"
},
{
"type": "command",
"if": "Bash(pnpx knip *)",
"command": "echo 'Use `pnpm knip` instead of pnpx knip.' >&2 && exit 2"
}
]
}
"permissions": {
"allow": [
"Bash(pnpx vitest run --testPathPattern=\"draftCacheV2.property\")",
"Bash(pnpx vitest run \"draftCacheV2.property\")",
"Bash(node -e \"const fc = require\\(''fast-check''\\); console.log\\(Object.keys\\(fc\\).filter\\(k => k.includes\\(''string''\\)\\).join\\('', ''\\)\\)\")"
]
}
}

View File

@@ -1,695 +0,0 @@
---
name: bug-dump-ingest
description: 'Syncs the #bug-dump Slack channel into Linear as the system of record AND auto-fixes verified real bugs via red-green-fix. Every Linear operation (create, search, link, label) is performed by posting an @Linear mention in the bug-dump thread — no Linear MCP, no API key. Flow: fetch → mandatory dedupe gate (@Linear search + gh PR search) → false-defect verification → post @Linear create in thread (tool call) → parse bot card for FE-NNNN + URL → post :white_check_mark: confirmation reply → if candidate is a verified real bug with no dedupe hit and no open PR, invoke red-green-fix automatically to produce failing test + fix + PR. Respects team emoji scheme (:white_check_mark: ticket created, :pr-open: PR open, :question: needs context, :repeat: duplicate). Use when asked to sync #bug-dump to Linear, triage slack bugs, run a bug-dump sweep, or ingest bug reports. Triggers on: bug-dump, sync bug-dump, ingest bugs, triage slack bugs, bug sweep.'
---
# Bug Dump Ingest
**Primary job: sync `#bug-dump` (Slack: `C0A4XMHANP3`) into Linear as the source of truth, then auto-fix the verified real bugs.** Linear is where status, labels, and follow-up triage happen — this skill gets every bug into Linear with enough context that a downstream agent or human can work from Linear alone. **Every Linear action is performed by mentioning `@Linear` in the bug-dump thread**; there is no Linear MCP and no API key path. When pre-flight verification confirms a candidate is a real bug (not dedupe, not already in a PR, not out of scope), the skill then invokes `red-green-fix` automatically.
```text
fetch → pre-flight dedupe gate (@Linear search + gh) → verify false defects → present approvals
→ POST "@Linear create ..." thread reply via slack_send_message (mandatory tool call)
→ poll slack_read_thread → parse Linear bot card for FE-NNNN + URL
→ POST :white_check_mark: confirmation thread reply via slack_send_message
→ if verification = "real bug" AND no dedupe AND no open PR:
invoke Skill(skill="red-green-fix") → POST :pr-open: thread reply
```
### Non-negotiable rules
1. **Linear actions are Slack tool calls.** The skill MUST drive Linear by calling `mcp__plugin_slack_slack__slack_send_message` with `thread_ts` set and text that mentions `@Linear`. There is no MCP-direct path and no API-key path. Printing `@Linear create ...` into the Claude CLI response is NOT a substitute — the Slack thread reply is what triggers the Linear bot, and its card is the canonical receipt.
2. **Dedupe is a gate, not a suggestion.** No candidate is proposed for creation until `@Linear search` AND `gh pr` search have been run and recorded. A hit short-circuits creation to `L` (link) or `pr-open`.
3. **Auto-fix real bugs.** When the dedupe gate is clean AND false-defect verification is clean AND the candidate isn't on the handoff-exclusion list (see § Handoff conditions), after Linear creation the skill invokes `red-green-fix` via the `Skill` tool — without waiting for an extra human prompt.
### What the skill cannot do
The Slack MCP exposes no `reactions.add` tool, so the skill cannot put a `:white_check_mark:` reaction on the parent message. The thread reply with the leading `:white_check_mark:` emoji is the skill's canonical marker; a human can additionally add the parent reaction for channel visibility (see § Parent reaction — optional visibility nudge). Both are respected by Processed Detection.
## Team emoji scheme
| Emoji | Meaning | Who adds it | Skill behavior |
| -------------------- | ------------------ | ------------------------------------------------------ | ---------------------------------------------- |
| `:white_check_mark:` | Ticket created | Human on parent (after skill files); also in bot reply | Skip in future sweeps |
| `:pr-open:` | PR open | Human | Skip creation; include PR link in approval row |
| `:question:` | Needs more context | Human | Skip creation; agent may ask for clarification |
| `:repeat:` | Duplicate | Human | Skip creation; link existing Linear issue |
## Design Priority
Optimize for **coverage, label quality, and proven fixes** over fix-path cleverness. Linear is the downstream triage surface — once every bug is there with status, labels, and context, agents and humans can work from Linear alone. A Linear ticket with a wrong severity is cheap to fix; a Slack-only bug is invisible to downstream tooling; a "filed but not fixed" real regression wastes a human turn that the skill could have spent on a red-green PR.
## Quick Start
1. **Scope** — default window: messages in the last 48h. Override with `--since YYYY-MM-DD` or a Slack permalink list.
2. **Fetch**`slack_read_channel` for `C0A4XMHANP3`; `slack_read_thread` per message with replies.
3. **Filter** — drop already-processed (see Processed Detection).
4. **Classify** — bug / discussion / meta (see Classification Rules).
5. **Pre-flight dedupe gate (MANDATORY)** — for every bug candidate, run `@Linear search` AND `gh pr` search BEFORE proposing (see § Pre-flight Dedupe Gate). A hit means the candidate goes into the batch as `L` (link) or `pr-open`, not as a new create.
6. **Verify false defects** — per candidate, run quick checks before proposing (see False-Defect Verification).
7. **Extract** — normalize to ticket schema (see Ticket Schema).
8. **Human approval** — batch table, collect Y/N/?/S/L/R per candidate (see Interactive Approval). Default recommendation for clean candidates is `Y` (file + auto-fix).
9. **Post `@Linear create` thread reply — MANDATORY TOOL CALL** — for each approved `Y`/`L` row, call `mcp__plugin_slack_slack__slack_send_message` with `channel_id=C0A4XMHANP3`, `thread_ts=<parent-ts>`, and text starting with `@Linear create` (see § Linear Slack Bot Integration). Do NOT print the command into chat as a substitute.
10. **Capture the Linear bot card** — poll `slack_read_thread` up to 3× with ~3s spacing, parse the first Linear-app reply for the `FE-NNNN` identifier and `https://linear.app/...` URL. No URL = not ingested; never fabricate one.
11. **Post `:white_check_mark:` confirmation reply — MANDATORY TOOL CALL** — call `slack_send_message` again with text starting with `:white_check_mark: Filed to Linear: <URL>` so future sweeps can detect the marker via `has::white_check_mark: from:me`. Record both `ts` values in the session log.
12. **Auto-fix (clean candidates only)** — if dedupe gate is clean AND false-defect verification is clean AND the candidate isn't on the Handoff-Exclusion list, immediately invoke the `red-green-fix` skill via the `Skill` tool. See § Fix Workflow for the exact call contract.
13. **Log** — append to session log; update `processed.json`.
## System Context
| Item | Value |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| Source channel | `#bug-dump` (`C0A4XMHANP3`) |
| Destination | Linear `Frontend Engineering` team, via the Linear Slack app (`@Linear`). Team is named in every `@Linear create` message. |
| Default state | `Triage` — every `@Linear create` message includes `Status: Triage` |
| State dir | `~/temp/bug-dump-ingest/` |
| Processed registry | `~/temp/bug-dump-ingest/processed.json` |
| Session log | `~/temp/bug-dump-ingest/session-YYYY-MM-DD.md` |
| Drafts (failure) | `~/temp/bug-dump-ingest/drafts/*.md` — written only when `@Linear` never replies, so the human can retry manually |
## Label Taxonomy
Every created Linear issue MUST get the following labels, passed as a comma-separated list in the `Labels:` line of the `@Linear create` message. The Linear Slack app creates missing labels on first use:
| Label kind | Values | Source |
| ------------ | ------------------------------------------------------------------------------ | ------------------------- |
| `source:` | `source:bug-dump` | Always (marks Slack sync) |
| `area:` | `area:ui`, `area:node-system`, `area:workflow`, `area:cloud`, `area:templates` | Area Heuristics |
| `env:` | `env:cloud-prod`, `env:cloud-dev`, `env:local`, `env:electron` | Env Heuristics |
| `severity:` | `sev:high`, `sev:medium`, `sev:low` | Severity Heuristics |
| `reporter:` | `reporter:<slack-handle>` (kebab-case) | From message author |
| Status flags | `needs-repro`, `needs-backend`, `regression`, `pr-open` | When applicable |
Label rules:
- Always include `source:bug-dump`, exactly one `area:`, at least one `env:` (or `env:unknown`), exactly one `severity:`, exactly one `reporter:`.
- `needs-repro` — set when repro steps were ambiguous; signals "human should confirm before fix".
- `needs-backend` — set when fix is clearly in ComfyUI backend, not this frontend repo.
- `regression` — set when the bug mentions a version/upgrade correlation.
- `pr-open` — set instead of creating a fresh ticket when a fix PR already exists; the Linear issue becomes a tracker.
Labels are the primary affordance for downstream triage — invest in getting them right, not just in the title.
## Processed Detection
A top-level message is considered already-handled (skip creation) if ANY of:
- Its timestamp appears in `processed.json`.
- It carries a `:white_check_mark:` reaction on the parent — ticket already created.
- It carries a `:pr-open:` reaction — fix PR is open; skill records the PR link in the session log rather than creating a fresh Linear issue.
- It carries a `:repeat:` reaction — duplicate; skill attempts to find the original Linear issue and link it in the session log.
- It carries a `:question:` reaction — needs more context; skill skips creation and records for follow-up.
- Its thread contains a reply with a `https://linear.app/` URL (fetch via `slack_read_thread`).
- Its thread contains a reply starting with `:white_check_mark:` from the skill's bot user.
- It is a system/meta message (`has joined the channel`, bot-only message).
- Its thread already contains resolution confirmation (`"solved"`, `"resolved"`, `:done:` reaction from the reporter) AND has no fix PR referenced — treat as "resolved without ticket, skip".
Never re-ingest a message already marked in any of the above ways.
Filter query for Slack search-based sweeps:
```text
in:<#C0A4XMHANP3> -has::white_check_mark: -has::pr-open: -has::repeat: -has::question: after:YYYY-MM-DD
```
## False-Defect Verification
Before a candidate hits the approval batch, run cheap checks to demote obvious non-bugs. Goal: keep the approval table high-signal. This is not a full repro — just fast heuristics that catch the top false-positive classes.
| Check | Command / Signal | Demote-to |
| ---------------------------------------- | ---------------------------------------------------------------- | ---------- |
| Reporter self-resolved in same msg | "no action needed", "solved", "nvm", "fixed it" | `resolved` |
| Reporter self-resolved in thread | `slack_read_thread` → reporter's last reply contains "solved" | `resolved` |
| Fix PR merged on main | `gh search prs "in:title <keyword>" --state merged --limit 3` | `fixed` |
| Fix PR open (already-filed) | `gh search prs "<keyword>" --state open --limit 3` | `pr-open` |
| Linear issue exists (open) | Linear `searchIssues` on title keywords → any open match | `dedupe` |
| Behavior is documented / intended | grep `docs/` and `src/locales/en/*.json` for the feature | `expected` |
| Not reproducible — feature doesn't exist | grep `src/` for mentioned component/feature → 0 hits | `stale` |
| Env drift only (local setup issue) | Thread contains "my machine", "my setup", "proxy" without others | `env` |
For each demoted candidate, record the demotion reason in the approval table as `Verify: <tag>` so the human can override if they disagree. Never hard-skip based on verification alone — always show the row with the demotion.
### Recommended verify commands
```bash
# 1. Search recent PRs for the feature in question
gh search prs "<keyword>" --repo Comfy-Org/ComfyUI_frontend --limit 5
# 2. Grep for the feature / component mentioned
rg -l "<ComponentOrFeatureName>" src/ apps/
# 3. Check if it's a known i18n / documented setting
rg "<setting-key>" src/locales/en/ docs/
```
Keep verification under ~30s per candidate. If it takes longer, propose a ticket and let the human decide — don't let verification become the bottleneck.
## Classification Rules
For each unprocessed top-level message, decide:
| Class | Signal | Action |
| ----------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------- |
| **bug** | Describes unexpected behavior, visual glitch, error, regression, crash. Usually has repro steps or media. | Propose Linear ticket |
| **discussion** | Design question, rollout thoughts, team chatter, PR planning (e.g. "how about we make a PR to do...") | Skip |
| **question** | User asking if something is expected or known | Skip unless answered = bug |
| **meta** | Channel joins, bot messages, cross-posts without content | Skip |
| **already-filed** | Thread shows PR already open OR existing Linear link | Skip, log with existing link |
When ambiguous, default to **bug** and let the human decide in the approval batch.
## Ticket Schema
Normalize each bug to this shape before presenting:
```json
{
"slack_ts": "1776639963.837519",
"slack_permalink": "https://comfy-organization.slack.com/archives/C0A4XMHANP3/p1776639963837519",
"reporter": "Ali Ranjah (wavey)",
"title": "Unet model dropdown missing selected model",
"description": "Body with repro steps, env, attachments list, thread summary",
"env": ["cloud prod"],
"severity": "low | medium | high",
"area": "ui | node-system | workflow | cloud | templates | unknown",
"attachments": [{ "name": "...", "id": "F...", "type": "image/png" }],
"thread_resolution": "solved | open | none"
}
```
Keep descriptions copy-paste friendly: lead with repro bullets, then env, then "See Slack: <permalink>". Attach thread summary only if it adds context beyond the top-level message.
### Severity Heuristics
- **high** — crash, data loss, blocks a template or core feature, affects paying users broadly (e.g. "job ends in 30m on Pro", "widget values reset").
- **medium** — visible regression, template error, wrong pricing, broken UX on a common path.
- **low** — cosmetic, single-template edge case, minor tooltip/boundary issue.
When unsure, mark `medium` and flag for human in the approval batch.
### Area Heuristics
- `ui` — visual glitches, palette issues, popover clipping, dropdown styling.
- `node-system` — canvas perf, reroute, node drag, widget rendering, undo.
- `workflow` — template failures, save/load, refresh regressions.
- `cloud` — jobs, pricing, assets, auth, queue.
- `templates` — specific template errors.
## Pre-flight Dedupe Gate (MANDATORY)
Before any candidate enters the approval table, run BOTH checks below and record the result in the row's `Dedup` and `PR` columns. This is a hard gate — no candidate may be proposed for creation without a verdict.
### Check 1 — Open Linear issues (via `@Linear search`)
Extract 3-5 keyword terms from the proposed title (strip stopwords). Post a search command to the bug-dump thread — use a scratch thread if no parent `ts` is available yet, but prefer the candidate's own parent thread so the search card becomes part of that thread's audit trail:
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear search <keyword-1> <keyword-2>\nTeam: Frontend Engineering\nStatus: open"
})
```
Poll `slack_read_thread` for up to 10s; parse the Linear app's card reply for `FE-NNNN` identifiers and URLs. Run the search twice with different keyword subsets if the first returns zero hits — reworded titles are the top false-negative class.
If `@Linear search` is not supported by the workspace's Linear app version, fall back to a Slack search for prior `@Linear` card replies in the channel:
```text
mcp__plugin_slack_slack__slack_search_public({
query: "in:<#C0A4XMHANP3> from:@Linear <keyword-1> <keyword-2>"
})
```
This scans past Linear bot replies in the channel — any reply containing a matching `FE-NNNN` URL is a candidate duplicate. Record which dedupe path was used in the session log.
Treat a hit as a duplicate if any of:
- Title overlap ≥ 80% (after lowercasing + stopword removal)
- Same reporter + same component reference in description
- Same stack trace or error code
**Verdict:** set `Dedup: FE-NNNN` and default recommendation to `L` (link, don't create). The human may still override to `Y` to file a separate ticket.
### Check 2 — Open or merged fix PRs on GitHub
```bash
# Open PRs matching title keywords
gh pr list --repo Comfy-Org/ComfyUI_frontend --state open \
--search "<keyword-1> <keyword-2>" --limit 5 \
--json number,title,url,createdAt
# Recent merged fixes (last 30d) — catches "already fixed, waiting to ship"
gh pr list --repo Comfy-Org/ComfyUI_frontend --state merged \
--search "<keyword-1> <keyword-2> merged:>=<YYYY-MM-DD>" --limit 5 \
--json number,title,url,mergedAt
```
Treat a hit as a match if the PR title/body mentions the same component or bug phrase and the PR is unmerged or merged within the window covering the reporter's observation.
**Verdict:**
- Open PR match → set `PR: #NNNN (open)`, recommendation `pr-open` (file Linear with `pr-open` label linking the PR, skip auto-fix).
- Merged PR match → set `PR: #NNNN (merged)`, recommendation `fixed` (demote in verify, usually skip; human can override if the reporter claims the fix didn't land).
### Failure handling
If either check errors (Linear Slack app silent or not in channel, `gh` auth expired), DO NOT proceed to proposal — stop the sweep, report the failure to the user, and let them decide whether to re-run or manually dedupe. A silent skip of dedupe is never acceptable; it's the single biggest source of duplicate tickets.
Log each dedupe query + top hits in `~/temp/bug-dump-ingest/session-YYYY-MM-DD.md` under a per-candidate `Dedup trace:` block so the human can audit.
## Interactive Approval
Present candidates in batches of 5-10. Table format (10 columns):
```text
# | Slack (author, time) | Proposed title | Env | Sev | Area | Dedup | PR | Verify | Rec
----+------------------------+-----------------------------------------+------------+------+------------+------------+---------------+-------------+-----
1 | wavey, 04-20 08:06 | Unet dropdown missing selected model | cloud prod | low | ui | - | - | resolved | N
2 | Denys, 04-18 05:45 | Pro plan jobs end at 30 minutes | cloud prod | high | cloud | - | - | clean | Y
3 | Terry Jia, 04-18 12:52 | Nodes 2.0 canvas lag on large workflows | - | high | node-system| FE-4521 | - | clean | L
4 | Pablo, 04-17 08:52 | Multi-asset delete popup shows hashes | cloud prod | low | ui | - | #11402 (open) | clean | pr-open
```
Each row MUST show: Slack author + date, proposed title, env tags, severity, area, **dedupe status from the Pre-flight Dedupe Gate**, **open/merged PR hit from the Pre-flight Dedupe Gate**, verify tag (from False-Defect Verification), and agent recommendation.
### Default recommendation logic
The skill computes `Rec` deterministically from the gate results:
- `L` — Dedupe hit on open Linear issue.
- `pr-open` — Open GitHub PR hit.
- `fixed` — Merged PR hit within the reporter's observation window.
- `N` — Verify tag is `resolved`, `expected`, `stale`, or `env` only.
- `?` — Repro incomplete or classification ambiguous.
- `Y` — Everything clean AND candidate is not on the § Handoff-Exclusion list. This is the "file + auto-fix" path.
- `Y (file-only)` — Clean but on the handoff-exclusion list (e.g. touches LGraphNode, needs backend). File Linear, skip auto-fix.
### Response format
- `Y` — default path: create Linear ticket, post `:white_check_mark:` thread reply, AND if the candidate is eligible (dedupe clean, verify clean, not on handoff-exclusion list), immediately invoke `red-green-fix` via the `Skill` tool. See § Fix Workflow.
- `S`**skip auto-fix** for this row: create Linear ticket + thread reply only, do NOT run red-green-fix. Use when the human knows a specific person is already investigating or wants to batch fixes.
- `N` — skip entirely (log reason in session file).
- `?` — mark as needs-context; skill posts a thread reply asking for repro details and prompts the human to add `:question:` to the parent.
- `L` — link to existing Linear issue instead of creating (skill asks which one if the Pre-flight Dedupe Gate didn't return an exact match).
- `R` — duplicate of another bug-dump message; skill links the two and prompts the human for `:repeat:` on the parent.
- `E` — edit proposed title/description before creating (skill shows draft for inline tweaks).
- Bulk responses accepted: `1 N, 2 Y, 3 L FE-4521, 4 pr-open #11402, 5 ?` — any row omitted from the response is treated as its computed `Rec` default.
Do not post any `@Linear create` messages until all candidates in the batch have a terminal decision. Auto-fix invocations run sequentially AFTER every `@Linear create` has produced a parsed `FE-NNNN`, so every `red-green-fix` call has a `Fixes FE-NNNN` to put in the PR body.
## Linear Slack Bot Integration (@Linear)
Every Linear action — create, search, link, label, status change — is performed by posting a message to the candidate's thread in `#bug-dump` that mentions `@Linear`. The Linear Slack app parses the mention and responds with a card in the same thread. There is no Linear MCP path and no `LINEAR_API_KEY` path; see `reference/linear-api.md` § "Why no direct API path" for the rationale.
### Prerequisites
- The Comfy Slack workspace already has the Linear Slack app installed (this is how humans add `@Linear` mentions today).
- Channel `C0A4XMHANP3` is connected to the `Frontend Engineering` Linear team.
- No per-machine setup. If a `@Linear` invocation produces no bot reply, the app is not in the channel — surface to the human, do NOT retry silently.
### Create an issue
For each approved `Y` candidate, call:
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear create\nTeam: Frontend Engineering\nTitle: <title>\nStatus: Triage\nLabels: source:bug-dump, area:<area>, env:<env>, sev:<severity>, reporter:<handle>\n\n<description>\n\nSource: <slack-permalink>"
})
```
Rules:
- First line MUST be `@Linear create` — this is the command token.
- `Team: Frontend Engineering` is required on every create — without it the bot falls back to the workspace default, which may route to a different team.
- `Status: Triage` pins the initial state (per § System Context).
- `Labels:` — comma-separated, full `source:bug-dump, area:*, env:*, sev:*, reporter:*` set per § Label Taxonomy. Missing labels are auto-created by the Linear Slack app on first use.
- Description body is markdown — see `reference/linear-api.md` § "Description body template" and `reference/schema.md` for per-field extraction.
- Use real newlines (not literal `\n`) when constructing the text.
After the tool call returns, poll `slack_read_thread` for the Linear app's reply card (up to 3× with ~3s spacing). Parse the card for:
- An `FE-NNNN` identifier
- A `https://linear.app/<org>/issue/FE-NNNN` URL
The URL is the ingested receipt. The skill then posts the `:white_check_mark:` confirmation reply (§ Slack Thread Reply).
### Search (dedupe)
See § Pre-flight Dedupe Gate § Check 1 for the search command shape and handling of the bot's reply. The search is a tool call in the candidate's thread — not a chat aside.
### Link an existing issue (`L` response)
When the human picks `L FE-4521` for a row, do NOT post `@Linear create`. Instead:
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear link FE-4521"
})
```
The bot replies with the linked issue card. Then post the `:white_check_mark:` confirmation reply (adjusted to say `Linked to Linear:` rather than `Filed to Linear:`) so Processed Detection still matches.
### Label / status updates
When a later sweep needs to flip a ticket (e.g. a PR opened after initial ingest, so add `pr-open` and link):
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear FE-4521 add-labels pr-open"
})
```
Status changes are rarely driven by this skill directly — Linear auto-moves issues to `In Review` when a PR with `Fixes FE-NNNN` is opened, and the `red-green-fix` skill handles that PR body.
### Captured fields per create
Every successful create must produce, via the Linear bot's reply card:
- `identifier` — e.g. `FE-4710`, used in `Fixes <LIN-ID>` references and session log
- `url``https://linear.app/.../issue/FE-4710`, included verbatim in the `:white_check_mark:` reply
- `ts` of the Linear bot's card reply — recorded in session log for audit
If the card is missing the URL or identifier, fall through to the failure path below — do NOT fabricate either value.
### Failure path
If the Linear bot does not reply within the poll window, OR replies with a parse error (`couldn't parse`, `no team matched`, `failed`):
1. Write a draft markdown file to `~/temp/bug-dump-ingest/drafts/NN-short-slug.md` containing the full `@Linear create` text that was sent plus any partial bot reply.
2. Post a thread reply that is explicit about the failure — do NOT include `:white_check_mark:` or a fake Linear URL:
```text
:warning: bug-dump-ingest: @Linear did not respond. Drafted at ~/temp/bug-dump-ingest/drafts/<slug>.md — please file manually and reply with the FE-NNNN.
```
3. Skip auto-fix for this candidate (no Linear ID = no `Fixes` reference).
4. Log the failure in the session log.
Never invent a Linear URL. Never post `:white_check_mark: Filed to Linear: ...` without a real URL parsed from a real Linear bot card.
## Slack Thread Reply (Ingested Marker) — MANDATORY TOOL CALL
Every approved candidate produces **two** mandatory `slack_send_message` calls in the parent thread:
1. The `@Linear create` (or `@Linear link`) command — see § Linear Slack Bot Integration.
2. The `:white_check_mark:` confirmation reply described below, posted after a real `FE-NNNN` + URL have been parsed from the Linear bot's card.
The second reply is what future sweeps grep for via `has::white_check_mark: from:me`. Even though the Linear bot's own card already contains the URL, the `:white_check_mark:` prefix is the canonical Processed Detection marker — without it, a future sweep may re-ingest the same bug.
The skill is not done with a candidate until BOTH calls have succeeded. If either fails, do not claim the candidate is ingested.
### Required call shape
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-message-ts>", // dotted form, e.g. "1776714531.990509"
text: ":white_check_mark: Filed to Linear: <LINEAR_URL>\nReporter: <@USER_ID>\nSev: <severity> • Area: <area>"
})
```
Rules:
- `thread_ts` MUST be the parent message ts — never the channel ts, never omitted. An omitted `thread_ts` posts at channel level, which pollutes `#bug-dump` and breaks Processed Detection.
- The text MUST start with `:white_check_mark:` followed by a space and `Filed to Linear:`. This exact prefix is what future sweeps grep for via `has::white_check_mark: from:me`.
- The Linear URL MUST be present. No URL = not ingested; future sweeps will re-file the same bug.
- Plain text only — no markdown tables, no bold, no code fences. Slack renders the emoji shortcode into a real `:white_check_mark:` only when the message is plain text.
- Capture the returned `ts` and record it in the session log for audit.
### NEVER-do list (common failure mode)
- **Do NOT** print `@Linear create ...` or `:white_check_mark: Filed to Linear: <URL>` into the Claude CLI chat response as a substitute for calling `slack_send_message`. The CLI output is not seen by Slack. If you find yourself typing either into a plain assistant message, stop and issue the tool call instead.
- **Do NOT** claim the thread reply was posted until the `slack_send_message` tool call has returned a success with a `ts`. If the tool call errors, surface the error and halt the batch — do not fabricate a reply.
- **Do NOT** use any other tool (e.g. `slack_schedule_message`, `slack_send_message_draft`) as a substitute. Only an immediate `slack_send_message` with `thread_ts` set counts — the Linear Slack app does not trigger on scheduled/draft messages.
- **Do NOT** substitute any direct Linear API call (MCP, GraphQL, curl) for the `@Linear` mention. The Slack thread is intentionally the single audit trail.
### Fix-path reply (after red-green-fix opens a PR)
When `red-green-fix` returns a PR URL for an auto-fixed candidate, the skill MUST post a second thread reply on the same parent — again via `slack_send_message`:
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<same parent ts>",
text: ":pr-open: Fix PR: <PR_URL>\nRed-green verified: <unit|e2e> test proves the regression.\nFixes <LIN-ID>"
})
```
Same "tool call, not chat output" rule applies.
### Parent reaction — optional visibility nudge (not on critical path)
The Slack MCP does not expose `reactions.add`, so the skill cannot set a `:white_check_mark:` reaction on the parent. The thread reply above is sufficient for Processed Detection; the parent reaction is a human-only "visible in channel" nudge. At the end of the run, the skill MAY print a compact list for the human:
```text
Optional: add :white_check_mark: to parent messages for in-channel visibility.
LIN-4710 → <permalink>
LIN-4711 → <permalink>
```
This is a convenience, not a deliverable — a missing parent reaction does not cause re-ingestion.
## Fix Workflow (auto-invoke red-green-fix)
For every `Y` row whose `Rec` resolved to auto-fix (dedupe clean, verify clean, not on handoff-exclusion list), the skill MUST — after Linear creation and the `:white_check_mark:` thread reply — invoke the `red-green-fix` skill via the `Skill` tool. This is a real tool call, not a narrative handoff.
### Required Skill tool call
```text
Skill({
skill: "red-green-fix",
args: "<composed prompt — see below>"
})
```
Compose `args` as a single self-contained prompt so the sub-invocation has everything it needs without re-reading the Linear issue:
```text
Bug: <title>
Linear: <LIN-ID> (<LINEAR_URL>)
Source: Slack <permalink>
Reporter: <display-name>
Env: <env tags>
Area: <area>
Branch: fix/<lin-id-lowercase>-<short-slug>
Repro:
1. <step>
2. <step>
Expected: <expected behavior>
Actual: <actual behavior>
Test layer (inferred from area):
- ui → Vitest colocated + Playwright e2e tagged @regression
- node-system → Playwright e2e primarily
- workflow / templates → Playwright e2e
- cloud → Vitest if client-side; otherwise STOP and label the Linear issue "needs-backend"
Test naming:
- describe('<LIN-ID>: <one-line bug summary>', ...)
- Playwright test title must include the LIN-ID.
PR body must include:
- "Fixes <LIN-ID>"
- "Source: Slack <permalink>"
Follow the red-green-fix two-commit sequence exactly. Do NOT skip the red commit.
```
The skill MUST wait for `red-green-fix` to return before moving to the next candidate. Process one auto-fix at a time so branch state is deterministic.
### Verifying the invocation ran
After the `Skill` call returns, the skill MUST confirm at least one of:
1. A new git branch named `fix/<lin-id>-*` exists (`git branch --list "fix/<lin-id>-*"`).
2. A PR URL is present in `red-green-fix`'s return payload.
If neither is true, the invocation silently no-op'd. Log the failure to the session log as `auto-fix skipped: invocation returned without branch or PR` and continue — do NOT post the `:pr-open:` thread reply.
### Inputs summary
- **Bug description** — the Linear description (includes repro, env, source permalink).
- **Linear ID** — inserted into the PR body as `Fixes <LIN-ID>`.
- **Branch name** — `fix/<lin-id>-<short-slug>` (e.g. `fix/lin-4711-pro-plan-30min-timeout`).
- **Test layer** — inferred from `area`:
- `ui` → unit (Vitest) + e2e (Playwright)
- `node-system` → e2e primarily; unit if isolable
- `workflow` / `templates` → e2e
- `cloud` → unit if client-side logic, otherwise flag "backend — out of scope for this repo"
### Handoff-Exclusion list (do NOT auto-invoke red-green-fix)
These rows still get a Linear ticket + `:white_check_mark:` thread reply, but the skill MUST skip the `Skill(skill="red-green-fix")` call and instead post a thread nudge explaining why:
- Repro steps are incomplete (no clear numbered steps, no env) — reply in thread: "Need clearer repro before I can write a failing test. What's the shortest path to reproduce?"
- Fix requires backend / ComfyUI repo changes (not frontend) — label Linear `needs-backend`.
- Linear ticket was dedupe-linked rather than newly created — existing owner may already be fixing.
- Severity is cosmetic AND reporter hasn't asked for a fix — file ticket only.
- Fix would touch `LGraphNode`, `LGraphCanvas`, `LGraph`, or `Subgraph` god-objects (ADR-0003/0008 — always human decision).
- Pre-flight Dedupe Gate found an open PR (`pr-open`) or a matching merged PR (`fixed`).
When a row is excluded, record the reason in the session log under `auto-fix excluded: <reason>`.
### Test authoring rules
Both tests MUST be written in the "red" commit BEFORE any fix code (per red-green-fix). Rules specific to bug-dump ingestion:
- **Unit test (Vitest)** — colocated next to the implementation, `<file>.test.ts`. Exercise the specific logic path reproduced by the reporter. One `describe` block named after the Linear ID:
```typescript
// src/components/node/UnetDropdown.test.ts
describe('LIN-4710: unet dropdown missing selected model', () => {
it('includes the currently-selected model in the list even when not in available models', () => {
// ...
})
})
```
- **E2E test (Playwright)** — under `browser_tests/tests/`, follow `writing-playwright-tests` skill. Tag with `@regression` and include the Linear ID in the test title:
```typescript
test.describe(
'LIN-4710 unet dropdown regression',
{ tag: ['@regression'] },
() => {
test('keeps selected model visible in the dropdown', async ({
comfyPage
}) => {
// ...
})
}
)
```
- **Mock data types** — follow `docs/guidance/playwright.md`: mock responses typed from `packages/ingest-types`, `packages/registry-types`, `src/schemas/` — never `as any`.
(The Handoff-Exclusion list above governs when `red-green-fix` is NOT invoked.)
### PR body template
The red-green-fix skill's PR template is extended with a `Source` line:
```markdown
## Summary
<Root cause>
- Fixes LIN-NNN
- Source: Slack <permalink>
## Red-Green Verification
| Commit | CI Status | Purpose |
| ------------------------------------------ | -------------------- | ------------------------------- |
| `test: LIN-NNN add failing test for <bug>` | :red_circle: Red | Proves the test catches the bug |
| `fix: <bug summary>` | :green_circle: Green | Proves the fix resolves the bug |
## Test Plan
- [ ] Unit regression test passes locally
- [ ] E2E regression test passes locally (if UI)
- [ ] Manual repro no longer reproduces
- [ ] Linear ticket linked
```
After the PR merges, post the second thread reply on Slack (see Slack Thread Reply § Fix-path reply).
## Emoji Reaction Hints (read-only)
The agent cannot add reactions, but respects human-set reactions when filtering. The canonical team scheme (primary):
| Reaction | Meaning | Action |
| -------------------- | ------------------ | -------------------------------------------------------- |
| `:white_check_mark:` | Ticket created | Skip — already ingested |
| `:pr-open:` | PR open | Skip creation; record PR link in session log |
| `:question:` | Needs more context | Skip creation; agent may post a thread reply asking |
| `:repeat:` | Duplicate | Skip creation; link existing Linear issue in session log |
Incidental reactions observed in the channel — treat as soft hints only, do NOT skip solely on these:
| Reaction | Meaning | Action |
| -------- | ------------------- | -------------------------------------------------- |
| `:eyes:` | Someone is triaging | Still ingestable |
| `:done:` | Reporter resolved | Demote to `resolved` in verify, but still show row |
| `:+1:` | Acknowledged | Ignore |
Approval-table response code `R` (new) corresponds to `:repeat:` — if you pick `R`, the skill treats it as duplicate and asks for the target Linear ID.
## Session Log
Append to `~/temp/bug-dump-ingest/session-YYYY-MM-DD.md`:
```text
Bug Dump Ingest Session -- 2026-04-20 11:40 KST
Window: 2026-04-18 00:00 — 2026-04-20 12:00 KST
Scanned: 28 top-level messages
Skipped (meta/discussion/processed): 14
Proposed: 14
Approved: 11
Created in Linear: 10
Draft-only (creation failed): 1
Linked-only (dedupe): 1
Thread replies posted: 11
Created:
- LIN-4710 Unet model dropdown missing selected model -- wavey -- low/ui
- LIN-4711 Pro plan jobs end at 30 minutes -- Denys -- high/cloud
- ...
Skipped with reason:
- 1776592837.616399 -- design discussion in thread, not a bug
- ...
```
## Gotchas
### Thread summaries, not raw dumps
Pulling the full thread often adds noise. Summarize replies to: (a) confirmed reproductions by other users, (b) env/version details added in replies, (c) links to related PRs/commits. Drop emojis-only replies, joined-channel notifications, and off-topic chatter.
### Cross-posts are not bugs
When the top-level message is just a link to a Slack message in another channel (e.g. "X posting" with a URL and nothing else), follow the link to the original source and ingest from there — do NOT create a ticket from the cross-post itself.
### Resolved-in-thread messages
If the reporter replies `"No action needed, this is solved"` (see wavey 2026-04-20 08:06), mark the ticket for SKIP in the approval table, not auto-skip. The human may still want a regression test ticket.
### Permalinks
Construct Slack permalinks as:
```text
https://comfy-organization.slack.com/archives/{CHANNEL_ID}/p{TS_WITH_DOT_REMOVED}
```
E.g. `1776510375.473579` → `p1776510375473579`.
### Attachment handling
Slack file IDs (e.g. `F0AT...`) are private. Do NOT link them directly in Linear. Instead, list the filename and type in the Linear description and include the Slack permalink — anyone with Slack access can see the attachments from the thread.
### No auto-create without approval
Never create Linear issues without a human `Y`. This is a hard rule — the skill exists to reduce human toil, not to replace triage judgment.
## Reference Files
- `reference/linear-api.md` — `@Linear` Slack bot command reference (create, search, link, labels, status).
- `reference/schema.md` — full ticket schema with field-by-field extraction notes.
- `reference/examples.md` — worked examples drawn from real #bug-dump messages.
- `reference/verify-commands.md` — cookbook of false-defect verification commands per bug class.
## Related Skills
- `red-green-fix` — auto-invoked via the `Skill` tool for every eligible `Y` candidate to produce a failing test + fix + PR with the red-green CI proof.
- `writing-playwright-tests` — used by red-green-fix when an e2e test is needed.
- `hardening-flaky-e2e-tests` — if the e2e test added in the fix PR starts flaking, jump to this skill.

View File

@@ -1,123 +0,0 @@
# Worked Examples
Real #bug-dump messages (2026-04-17 → 2026-04-20) normalized through the skill.
## Example 1 — Clean bug with repro
**Source message** (wavey, 2026-04-20 08:06):
> unet model dropdown doesnt display all available models, think this is part of a larger issue with model dropdowns..
>
> • open flux.2 klein 4b image edit template
> • open unet drop down --> notice selected model isnt present in the list, even though its selected
> • execute (to check if it flags the model as missing) --> notice it still runs
> No action needed, this is solved
**Thread resolution**: "No action needed, this is solved" — reporter resolved it in the same message.
**Classification**: bug, but `thread_resolution = solved`. Flag for human.
**Approval row**:
```text
1 | wavey, 04-20 08:06 | Unet dropdown missing selected model | cloud | low | ui | N | N (reporter marked solved)
```
Default recommendation: `N`. If human overrides to `Y`, file with a "Regression test" label so QA still tracks it.
---
## Example 2 — Clear high-severity cloud bug
**Source message** (Denys Puziak, 2026-04-18 05:45):
> I see two reports about jobs ending in 30 minutes while the user is on the Pro plan
> cc @Hunter
> https://discord.com/channels/.../1494078128971055145
**Classification**: bug, `env: [cloud prod]` (Pro plan = cloud), `severity: high` (paying users), `area: cloud`.
**Proposed title**: `Pro plan jobs end at 30 minutes`
**Description** (excerpt):
```markdown
**Reporter:** Denys Puziak
**Env:** cloud prod
**Severity (proposed):** high
**Area:** cloud
## Repro
1. User on Pro plan submits a job
2. Job ends at 30 minutes instead of the Pro plan limit
## Notes
- Two user reports aggregated by Denys
- cc'd @Hunter
## Source
Slack: <permalink>
Discord thread: https://discord.com/channels/.../1494078128971055145
```
---
## Example 3 — Not a bug (discussion)
**Source message** (Christian Byrne, 2026-04-19 19:00):
> @Glary-Bot okay option A is clearly superior and I feel embarrassed I didn't see that line myself...
**Classification**: discussion (design review chatter). Skip. Log reason in session file.
---
## Example 4 — Meta-action / PR planning
**Source message** (Christian Byrne, 2026-04-19 09:30):
> @Glary-Bot how about we make a PR to do:
>
> 1. Audit the rest of the codebase...
> 2. Create a helper in src/base...
**Classification**: discussion (PR-plan proposal). Skip.
---
## Example 5 — Performance regression
**Source message** (Terry Jia, 2026-04-18 12:52):
> With Nodes 2.0, large workflows (hundreds of nodes) make the canvas extremely laggy and unusable for actual work — switching tabs takes several seconds or more. Switching back to Litegraph, performance is significantly better.
**Classification**: bug, `area: node-system`, `severity: high`.
**Dedupe**: Post `@Linear search nodes 2.0 performance canvas lag` (Team: Frontend Engineering, Status: open) in the candidate's thread. Likely matches exist — flag `Dedup? ?` and ask human which ticket to link to.
---
## Example 6 — Reporter says it's a question, not a report
**Source message** (Luke, 2026-04-17 08:27):
> Is NodeInfo supposed to show information or docs about the node? It just brings up the node sidebar
**Classification**: question → ambiguous. Read thread. If replies confirm "that's unexpected, should show docs", upgrade to bug. If "yes that's intended", skip.
Default recommendation in the approval batch: `?` (needs expansion).
---
## Example 7 — Bug with PR already in flight
**Source message** (Pablo, 2026-04-17 08:52):
> when deleting multiple assets on cloud -> the confirmation popup still has the assets hashes as names instead of the display name
**Reaction**: `pr-open (1)` — someone's opened a PR.
**Classification**: `already-filed` branch. Skip creation; in the session log, note "PR already open". If the human wants a tracking Linear ticket anyway, still fileable with a link to the PR.

View File

@@ -1,160 +0,0 @@
# Linear Slack Bot (@Linear) Reference
The skill drives Linear exclusively through the Linear Slack app (`@Linear`). **There is no Linear MCP, no `LINEAR_API_KEY`, no GraphQL.** Every Linear read/write happens as a Slack message that mentions `@Linear` in the `#bug-dump` thread, and the Linear Slack app performs the action and posts a reply card containing the issue URL.
## Why Slack-only
- The `#bug-dump` thread is already the source of truth; keeping the entire lifecycle (report → ticket → PR → resolution) in one thread means Processed Detection can grep the thread instead of a separate registry.
- No API key rotation, no MCP server install, no OAuth browser flow — works on any machine that already has the Slack MCP configured.
- The Linear Slack app's reply card (with issue URL, title, status, and assignee) IS the canonical receipt; the skill records its `ts` in the session log.
## Prerequisites (one-time, per workspace)
The Comfy Slack workspace must already have the Linear Slack app installed (it is — that's how humans use `@Linear` reactions today) and `#bug-dump` (channel `C0A4XMHANP3`) must have Linear enabled for the `Frontend Engineering` team. Nothing else to configure. If a `@Linear` invocation silently does nothing, the bot isn't present in the channel — surface that to the human rather than re-trying.
## Supported operations
Every operation is a `mcp__plugin_slack_slack__slack_send_message` call with `channel_id=C0A4XMHANP3` and `thread_ts=<parent-ts>`. The `text` is a natural-language instruction to the Linear bot. Keep the text concise — Linear parses the first line as the command intent.
### 1. Create an issue from the thread
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear create\nTeam: Frontend Engineering\nTitle: <title>\nStatus: Triage\nLabels: source:bug-dump, area:<area>, env:<env>, sev:<severity>, reporter:<handle>\n\n<description body>\n\nSource: <slack-permalink>"
})
```
Rules:
- Start with `@Linear create` on its own line — this is the command token the bot keys on.
- Always specify `Team: Frontend Engineering`. Without it, the bot falls back to the Slack workspace's default team, which may not be FE.
- `Status: Triage` pins the initial workflow state.
- `Labels:` — comma-separated. If a label doesn't exist yet in Linear, the bot creates it on first use (verified in Linear workspace settings). Keep the taxonomy exactly as SKILL.md § Label Taxonomy.
- `<description body>` — markdown per `reference/schema.md` Description Template. Use real newlines, not literal `\n`.
- End with `Source: <slack-permalink>` so the Linear issue body links back even if the auto-attachment of the parent message fails.
The Linear bot replies in the same thread with a card that contains:
- The Linear URL (`https://linear.app/comfy-org/issue/FE-NNNN`)
- Status, assignee (initially unassigned), and applied labels
- A "View in Linear" button
Parse the URL out of the bot's reply text (or attachments). If no card reply appears within ~10s of polling `slack_read_thread`, treat it as a creation failure — do NOT proceed to the `:white_check_mark:` confirmation reply.
### 2. Search existing open issues (dedupe)
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear search <keyword-1> <keyword-2>\nTeam: Frontend Engineering\nStatus: open"
})
```
The bot replies with a card listing up to ~5 matching open issues. Parse identifier (`FE-NNNN`) and URL per row. Treat a hit as a duplicate per SKILL.md § Pre-flight Dedupe Gate § Check 1.
If `@Linear search` is not supported in the installed Slack app version, fall back to Slack-native search across the `#bug-dump` thread replies (previous `@Linear` cards contain title + URL — grep those for the same keywords). Record which path was used in the session log so the human can see dedupe coverage.
### 3. Link an existing issue (dedupe: `L` response)
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear link FE-4521"
})
```
The bot replies with the linked issue card. The skill then posts its own `:white_check_mark: Linked to Linear: <URL>` confirmation reply (see SKILL.md § Slack Thread Reply).
### 4. Add labels to an existing issue
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear FE-4521 add-labels pr-open"
})
```
Used when an open PR is discovered after ticket creation and the Linear issue should flip to `pr-open`.
### 5. Change status
```text
mcp__plugin_slack_slack__slack_send_message({
channel_id: "C0A4XMHANP3",
thread_ts: "<parent-ts>",
text: "@Linear FE-4521 status In Progress"
})
```
Rarely used by the skill directly — usually status changes come from the `red-green-fix` PR lifecycle (Linear auto-moves to `In Review` when a PR references `Fixes FE-4521`).
## Description body template
The text that follows the command headers is rendered verbatim as the Linear issue description (markdown). Use this template — see `reference/schema.md` for field-by-field extraction notes:
```markdown
**Reporter:** <slack-display-name>
**Env:** cloud prod / local / electron / ...
**Severity (proposed):** high/medium/low
**Area:** ui / node-system / workflow / cloud / templates
## Repro
1. ...
2. ...
## Expected
...
## Actual
...
## Attachments (in Slack thread)
- image.png (png, 315 KB)
- Screen Recording.mov (mov, 37 MB)
## Source
Slack: <permalink>
Thread summary: <1-3 bullets if thread adds context>
```
The Slack permalink is load-bearing — it's the canonical route to attachments, reporter, and any follow-up discussion. Do NOT embed Slack file IDs (`F0AT...`) directly; they're permissioned.
## Parsing the bot's reply
After each `slack_send_message` that mentions `@Linear`, poll `slack_read_thread` (with `channel_id=C0A4XMHANP3`, `thread_ts=<parent-ts>`) up to 3 times, ~3s apart. Scan replies authored by the Linear Slack app user for:
- Any `https://linear.app/<org>/issue/FE-\d+` URL → capture as the issue URL.
- The `FE-NNNN` identifier pattern → capture as the issue identifier.
- An error phrase (`couldn't`, `failed`, `not found`, `no team matched`) → treat as failure; surface the full bot text to the human.
Record the bot reply's `ts` alongside the captured URL and identifier in the session log.
## Failure modes & handling
| Symptom | Likely cause | Handling |
| ------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| No bot reply within 10s | Linear app not in channel, or bot outage | Halt the batch, surface to human, do NOT fabricate a Linear URL. Remaining approved candidates stay queued for re-run. |
| Bot replies with "no team matched" | Team name typo or Linear workspace drift | Re-send with the exact team name from the Linear workspace (default: `Frontend Engineering`). If it still fails, ask the human to verify. |
| Bot replies with "couldn't parse labels" | One of the labels has syntax the bot rejects | Drop the offending label, re-send; log the partial-label failure so the human can patch after. |
| Bot creates the issue but reply lacks the URL | Rare bot format change | Re-fetch the thread after ~5s; if URL still absent, open Linear search via `@Linear search <title>` and recover the identifier + URL. |
| Multiple `@Linear` replies match (duplicate card) | The skill retried without polling first | Keep the earliest card's URL; log the extras. Never re-issue `@Linear create` for the same candidate without confirming the first card failed. |
Never retry `@Linear create` without first running `@Linear search` for the same title keywords — a duplicate card is worse than an initial failure because the human has to close one of them manually.
## Why no direct API path
- The Linear MCP (official or community) would require either OAuth setup or `LINEAR_API_KEY` in env — both are per-machine hurdles the skill should not depend on.
- Direct GraphQL against `api.linear.app` has the same key-management cost and bypasses the Slack thread as the audit trail.
- Routing every action through `@Linear` in the thread gives humans full visibility in the channel (the bot's card is the receipt) and Processed Detection becomes a simple Slack thread read.
If a future need requires capabilities the `@Linear` Slack app doesn't expose (bulk operations, private field edits, webhooks), stop and surface the limitation to the human rather than quietly adding an API-key path — the "Slack-only" constraint is intentional.

View File

@@ -1,94 +0,0 @@
# Ticket Schema — Extraction Notes
Field-by-field guidance for normalizing a Slack #bug-dump message into a ticket.
## `slack_ts`
The top-level message timestamp from `slack_read_channel` response (`Message TS:` field). Always store the dotted form (`1776510375.473579`). This is the ingestion identity used in `processed.json`.
## `slack_permalink`
Construct:
```text
https://comfy-organization.slack.com/archives/C0A4XMHANP3/p<ts-without-dot>
```
Example: `1776510375.473579``.../p1776510375473579`.
## `reporter`
The display name + parenthetical nickname if present. Examples from the channel:
- `Ali Ranjah (wavey)`
- `Denys Puziak`
- `Christian Byrne`
Do NOT use the Slack user ID (`U087MJCDHHC`) in Linear — names are more readable.
## `title`
Rules:
- Start with a verb or noun phrase describing the observed defect, not the reporter.
- ≤ 80 chars.
- Include env qualifier ("cloud prod", "local dev", "electron") only if ambiguous.
- Strip emoji and reactions from the original message when extracting.
Transformations:
| Slack message (excerpt) | Title |
| ----------------------------------------------------------------------- | --------------------------------------------------- |
| "unet model dropdown doesnt display all available models..." | Unet dropdown missing selected model |
| "Dates are broken on Settings -> Secrets. Cloud Prod" | Settings → Secrets dates broken on cloud prod |
| "LTX-2: Audio to VIdeo template results in the "RuntimeError..." error" | LTX-2 Audio-to-Video template RuntimeError on cloud |
## `description`
Structure — see `linear-api.md` § "Description body template". Key rules:
- Lead with **Repro** numbered list. Extract from the message body; if no steps are given, write "Repro: [Slack message body quoted verbatim]" and flag for human in approval.
- Preserve the reporter's own words in the Repro section when they include "step 1 / step 2" markers.
- Collapse multi-paragraph asides into "Notes" at the end.
## `env`
Detect from message text using these terms:
| Text in message | Tag |
| -------------------------- | ---------------------- |
| `cloud prod`, `prod cloud` | `cloud prod` |
| `cloud dev` | `cloud dev` |
| `cloud` | `cloud` (unqual.) |
| `local`, `localhost` | `local` |
| `electron`, `desktop` | `electron` |
| `nodes 2.0`, `LG` | (feature tag, not env) |
A message can have multiple env tags. If none are detectable, set `env: []` and flag "env unclear" in the approval row.
## `severity`
Heuristics in SKILL.md. When uncertain, mark `medium` and note in approval table: `Sev: medium (flag)`.
## `area`
Single tag. Use the one that best fits; tiebreak toward the more actionable team:
- `cloud` > `workflow` when the reported behavior is specific to cloud-hosted features (billing, queue, jobs)
- `node-system` > `ui` when the defect is canvas interaction, not just visual
- `templates` only when a named template is the subject
## `attachments`
From `slack_read_channel` message `Files:` field. Parse name, ID, type. Never include the Slack file ID in the Linear description — those are permissioned — just the filename and type.
## `thread_resolution`
Fetch via `slack_read_thread`. Scan replies for:
- `solved`, `resolved`, `fixed`, `no action needed``solved`
- A `:done:` reaction from the reporter → `solved`
- A `https://github.com/Comfy-Org/ComfyUI_frontend/pull/` URL in a reply → `pr-open` (keep but note in description)
- Otherwise → `open`
If `solved` and no PR merged, flag in approval table: reporter marked solved — confirm before filing.

View File

@@ -1,99 +0,0 @@
# Verify Commands Cookbook
One-shot commands for each False-Defect Verification class. Keep each under ~30s.
## 1. Check for existing fix PR
```bash
# By keyword in title
gh search prs --repo Comfy-Org/ComfyUI_frontend "<keyword>" --state merged --limit 5
# By keyword in body
gh pr list --repo Comfy-Org/ComfyUI_frontend --search "<keyword>" --state all --limit 5
# Recent closing PRs near the reported date
gh pr list --repo Comfy-Org/ComfyUI_frontend --state merged \
--search "merged:>=<YYYY-MM-DD> <keyword>" --limit 10
```
Verify tag: `fixed` if a merged PR explicitly matches; `pr-open` if an open PR matches.
## 2. Check for existing open Linear issue
```text
# Primary: @Linear search in the candidate's bug-dump thread
# mcp__plugin_slack_slack__slack_send_message({
# channel_id: "C0A4XMHANP3",
# thread_ts: "<parent-ts>",
# text: "@Linear search <keyword-1> <keyword-2>\nTeam: Frontend Engineering\nStatus: open"
# })
# → poll slack_read_thread, parse the Linear app's reply card for FE-NNNN matches.
#
# Fallback: grep past @Linear bot replies in the channel for prior ingested titles
# mcp__plugin_slack_slack__slack_search_public({
# query: "in:<#C0A4XMHANP3> from:@Linear <keyword-1> <keyword-2>"
# })
```
Verify tag: `dedupe` with the `FE-NNNN` identifier in the approval row. See `reference/linear-api.md` § "Search existing open issues (dedupe)" for full handling.
## 3. Feature actually exists in codebase
```bash
# Find the component / feature mentioned
rg -l "<ComponentOrFeatureName>" src/ apps/ --type vue --type ts
# Find a setting key
rg "<setting-key>" src/locales/en/ src/stores/settingStore.ts
# Find a store action
rg "<actionName>" src/stores/ --type ts
```
Verify tag: `stale` if 0 hits AND the feature name is specific (not a generic word).
## 4. Intended behavior check
```bash
# Check docs and release notes
rg -l "<feature keyword>" docs/ CHANGELOG.md
# Check if behavior is asserted in an existing test (green today)
rg "<observed behavior>" src/**/*.test.ts browser_tests/
```
Verify tag: `expected` if docs describe this as the intended behavior, or a test asserts it.
## 5. Reporter self-resolution
Already gathered via `slack_read_thread`. Look for reporter's own replies containing:
- "solved", "resolved", "fixed", "no action needed", "nvm", "my bad"
- A `:done:` reaction from the reporter
- A `:white_check_mark:` reaction
Verify tag: `resolved`.
## 6. Env-specific / local setup
If the message mentions "my machine", "my proxy", "my docker", "my cache" AND no other reporter has confirmed in-thread:
```bash
# Check thread for cross-user confirmations
# slack_read_thread → count distinct users replying with "same", "repro'd", "+1"
```
Verify tag: `env` if only the reporter is affected.
## 7. Cross-post (X posting)
If the top-level message is just a link + "X posting":
```bash
# Follow the link — use slack_search_public to find the original thread
# slack_search_public({ query: "<in:channel from:@reporter> <before:date>" })
```
If the original is already ingestable, ingest from the original's permalink. If it's a GitHub issue, prefer linking that GitHub issue to the Linear ticket instead of creating two entries.
Verify tag: `cross-post` with the resolved source permalink.

View File

@@ -0,0 +1,278 @@
---
name: reproduce-issue
description: 'Reproduce a GitHub issue by researching prerequisites, setting up the environment (custom nodes, workflows, settings), and interactively exploring ComfyUI via playwright-cli until the bug is confirmed. Then records a clean demo video.'
---
# Issue Reproduction Skill
Reproduce a reported GitHub issue against a running ComfyUI instance. This skill uses an interactive, agent-driven approach — not a static script. You will research, explore, retry, and adapt until the bug is reproduced, then record a clean demo.
## Architecture
Two videos are produced:
1. **Research video** — the full exploration session: installing deps, trying things, failing, retrying, figuring out the bug. Valuable for debugging context.
2. **Reproduce video** — a clean, minimal recording of just the reproduction steps. This is the demo you'd attach to the issue.
```
Phase 1: Research → Read issue, understand prerequisites
Phase 2: Environment → Install custom nodes, load workflows, configure settings
Phase 3: Explore → [VIDEO 1: research] Interactively try to reproduce (retries OK)
Phase 4: Record → [VIDEO 2: reproduce] Clean recording of just the minimal repro steps
Phase 5: Report → Generate a structured reproduction report
```
## Prerequisites
- ComfyUI server running (ask user for URL, default: `http://127.0.0.1:8188`)
- `playwright-cli` installed: `npm install -g @playwright/cli@latest`
- `gh` CLI (authenticated, for reading issues)
- ComfyUI backend with Python environment (for installing custom nodes)
## Phase 1: Research the Issue
1. Fetch the issue details:
```bash
gh issue view <number> --repo Comfy-Org/ComfyUI_frontend --json title,body,comments
```
2. Extract from the issue body:
- **Reproduction steps** (the exact sequence)
- **Prerequisites**: specific workflows, custom nodes, settings, models
- **Environment**: OS, browser, ComfyUI version
- **Media**: screenshots or videos showing the bug
3. Search the codebase for related code:
- Find the feature/component mentioned in the issue
- Understand how it works currently
- Identify what state the UI needs to be in
## Phase 2: Environment Setup
Set up everything the issue requires BEFORE attempting reproduction.
### Custom Nodes
If the issue mentions custom nodes:
```bash
# Find the custom node repo
# Clone into ComfyUI's custom_nodes directory
cd <comfyui_path>/custom_nodes
git clone <custom_node_repo_url>
# Install dependencies if needed
cd <custom_node_name>
pip install -r requirements.txt 2>/dev/null || true
# Restart ComfyUI server to load the new nodes
```
### Workflows
If the issue references a specific workflow:
```bash
# Download workflow JSON if a URL is provided
curl -L "<workflow_url>" -o /tmp/test-workflow.json
# Load it via the API
curl -X POST http://127.0.0.1:8188/api/workflow \
-H "Content-Type: application/json" \
-d @/tmp/test-workflow.json
```
Or load via playwright-cli:
```bash
playwright-cli goto "http://127.0.0.1:8188"
# Drag-and-drop or use File > Open to load the workflow
```
### Settings
If the issue requires specific settings:
```bash
# Use playwright-cli to open settings and change them
playwright-cli press "Control+,"
playwright-cli snapshot
# Find and modify the relevant setting
```
## Phase 3: Interactive Exploration — Research Video
Start recording the **research video** (Video 1). This captures the full exploration — mistakes, retries, dead ends — all valuable context.
```bash
# Open browser and start video recording
playwright-cli open "http://127.0.0.1:8188"
playwright-cli video-start
# Take a snapshot to see current state
playwright-cli snapshot
# Interact based on what you see
playwright-cli click <ref>
playwright-cli fill <ref> "text"
playwright-cli press "Control+s"
# Check results
playwright-cli snapshot
playwright-cli screenshot --filename=/tmp/qa/research-step-1.png
```
### Key Principles
- **Observe before acting**: Always `snapshot` before interacting
- **Retry and adapt**: If a step fails, try a different approach
- **Document what works**: Keep notes on which steps trigger the bug
- **Don't give up**: Try multiple approaches if the first doesn't work
- **Establish prerequisites**: Many bugs require specific UI state:
- Save a workflow first (File > Save)
- Make changes to dirty the workflow
- Open multiple tabs
- Add specific node types
- Change settings
- Resize the window
### Common ComfyUI Interactions via playwright-cli
| Action | Command |
| ------------------- | -------------------------------------------------------------- |
| Open hamburger menu | `playwright-cli click` on the C logo button |
| Navigate menu | `playwright-cli hover <ref>` then `playwright-cli click <ref>` |
| Add node | Double-click canvas → type node name → select from results |
| Connect nodes | Drag from output slot to input slot |
| Save workflow | `playwright-cli press "Control+s"` |
| Save As | Menu > File > Save As |
| Select node | Click on the node |
| Delete node | Select → `playwright-cli press "Delete"` |
| Right-click menu | `playwright-cli click <ref> --button right` |
| Keyboard shortcut | `playwright-cli press "Control+z"` |
## Phase 4: Record Clean Demo — Reproduce Video (max 5 minutes)
Once the bug is confirmed, **stop the research video** and **close the research browser**:
```bash
playwright-cli video-stop
playwright-cli close
```
Now start a **fresh browser session** for the clean reproduce video (Video 2).
**IMPORTANT constraints:**
- **Max 5 minutes** — the reproduce video must be short and focused
- **No environment setup** — server, user, custom nodes are already set up from Phase 3. Just log in and go.
- **No exploration** — you already know the exact steps. Execute them quickly and precisely.
- **Start video recording immediately**, execute steps, stop. Don't leave the recording running while thinking.
1. **Open browser and start recording**:
```bash
playwright-cli open "http://127.0.0.1:8188"
playwright-cli video-start
```
2. **Execute only the minimal reproduction steps** — no exploration, no mistakes. Just the clean sequence that demonstrates the bug. You already know exactly what works from Phase 3.
3. **Take key screenshots** at critical moments:
```bash
playwright-cli screenshot --filename=/tmp/qa/before-bug.png
# ... trigger the bug ...
playwright-cli screenshot --filename=/tmp/qa/bug-visible.png
```
4. **Stop recording and close** immediately after the bug is demonstrated:
```bash
playwright-cli video-stop
playwright-cli close
```
## Phase 5: Generate Report
Create a reproduction report at `tmp/qa/reproduce-report.md`:
```markdown
# Issue Reproduction Report
- **Issue**: <issue_url>
- **Title**: <issue_title>
- **Date**: <today>
- **Status**: Reproduced / Not Reproduced / Partially Reproduced
## Environment
- ComfyUI Server: <url>
- OS: <os>
- Custom Nodes Installed: <list or "none">
- Settings Changed: <list or "none">
## Prerequisites
List everything that had to be set up before the bug could be triggered:
1. ...
2. ...
## Reproduction Steps
Minimal steps to reproduce (the clean sequence):
1. ...
2. ...
3. ...
## Expected Behavior
<from the issue>
## Actual Behavior
<what actually happened>
## Evidence
- Research video: `research-video/video.webm` (full exploration session)
- Reproduce video: `reproduce-video/video.webm` (clean minimal repro)
- Screenshots: `before-bug.png`, `bug-visible.png`
## Root Cause Analysis (if identified)
<code pointers, hypothesis about what's going wrong>
## Notes
<any additional observations, workarounds discovered, related issues>
```
## Handling Failures
If the bug **cannot be reproduced**:
1. Document what you tried and why it didn't work
2. Check if the issue was already fixed (search git log for related commits)
3. Check if it's environment-specific (OS, browser, specific version)
4. Set report status to "Not Reproduced" with detailed notes
5. The report is still valuable — it saves others from repeating the same investigation
## CI Integration
In CI, this skill runs as a Claude Code agent with:
- `ANTHROPIC_API_KEY` for Claude
- `GEMINI_API_KEY` for initial issue analysis (optional)
- ComfyUI server pre-started in the container
- `playwright-cli` pre-installed
The CI workflow:
1. Gemini generates a reproduce guide (markdown) from the issue
2. Claude agent receives the guide and runs this skill
3. Claude explores interactively, installs dependencies, retries
4. Claude records a clean demo once reproduced
5. Video and report are uploaded as artifacts

View File

@@ -0,0 +1,133 @@
---
name: comfy-qa
description: 'Comprehensive QA of ComfyUI frontend. Reproduces bugs via E2E tests, records narrated demo videos, deploys reports. Works in CI and locally via `pnpm qa`.'
---
# ComfyUI Frontend QA Skill
Automated quality assurance pipeline that reproduces reported bugs using Playwright E2E tests, records narrated demo videos with demowright, and deploys reports to Cloudflare Pages.
## Quick Start
```bash
# Reproduce an issue
pnpm qa 10253
# Test a PR
pnpm qa 10270
# Test PR base (reproduce bug)
pnpm qa 10270 -t base
# Test both base + head
pnpm qa 10270 -t both
# Test local uncommitted changes
pnpm qa --uncommitted
# Help
pnpm qa -h
```
Auto-loads `.env.local` / `.env` for `GEMINI_API_KEY` and `ANTHROPIC_API_KEY`.
Auto-detects issue vs PR via GitHub API. Auto-starts ComfyUI if not running.
## Architecture
Three-phase pipeline:
1. **RESEARCH** — Claude Sonnet 4.6 (Agent SDK) writes Playwright E2E tests to reproduce bugs
2. **RECORD** — Re-runs test with demowright for narrated demo video (title cards, TTS, subtitles)
3. **REPORT** — Gemini reviews video, deploys to Cloudflare Pages with badge + verdict
Key principle: **Playwright assertions are truth** — no AI hallucination. If the test passes, the bug is proven.
## Pipeline Scripts
All scripts live in `.claude/skills/comfy-qa/scripts/`:
| Script | Role |
| ------------------------- | ----------------------------------------------------- |
| `qa.ts` | CLI entry point (`pnpm qa`) |
| `qa-agent.ts` | Research phase: Claude writes E2E tests via Agent SDK |
| `qa-record.ts` | Orchestrator: 3-phase pipeline |
| `qa-deploy-pages.sh` | Cloudflare Pages deploy + badge generation |
| `qa-report-template.html` | Report site template |
| `qa-video-review.ts` | Gemini video review |
| `qa-analyze-pr.ts` | Deep PR/issue analysis → QA guide |
## Triggering QA
### Via CLI (`pnpm qa`)
```bash
pnpm qa 10253 # issue (auto-detect)
pnpm qa 10270 # PR head
pnpm qa 10270 -t base # PR base
pnpm qa 10270 -t both # both
pnpm qa --uncommitted # local changes
```
### Via GitHub Labels
- **`qa-issue`** — Reproduce a bug from an issue
- **`qa-changes`** — Focused QA on a PR (Linux-only, before/after)
- **`qa-full`** — Full QA (3-OS matrix)
### Via Push to trigger branches
```bash
git push origin sno-skills:sno-qa-10253 --force
```
## Research Phase (`qa-agent.ts`)
Claude receives the issue/PR context + a11y tree snapshot + ComfyPage fixture API docs.
Tools:
- **`inspect(selector?)`** — Read a11y tree
- **`readFixture(path)`** — Read fixture source code
- **`readTest(path)`** — Read existing tests for patterns
- **`writeTest(code)`** — Write a Playwright .spec.ts
- **`runTest()`** — Execute and get pass/fail + errors
- **`done(verdict, summary, evidence, testCode, videoScript?)`** — Finish
When `verdict=REPRODUCED`, Claude also provides a `videoScript` — a separate test file using demowright's `createVideoScript()` for professional narrated demo video with title cards, TTS segments, and outro.
## Video Recording (demowright)
Phase 2 uses the video script to record with:
- `showTitleCard()` / `hideTitleCard()` — covers setup/loading screen
- `createVideoScript().title().segment().outro()` — structured narration
- `pace()` — narration-then-action timing
- TTS audio + subtitles + cursor overlay + key badges
## Report Site
Deployed to `https://sno-qa-{number}.comfy-qa.pages.dev/`
Features:
- Video player (1x default, adjustable speed)
- Research log (verdict, tool calls, timing)
- E2E test code + video script code
- Verdict banner for NOT_REPRODUCIBLE/INCONCLUSIVE with failure reason
- Copy badge button (markdown)
## Prerequisites
- `GEMINI_API_KEY` — video review, TTS
- `ANTHROPIC_API_KEY` — Claude Agent SDK (research phase)
- `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` — report deployment (CI only)
- ComfyUI server running (auto-detected, or auto-started)
## CI Workflow (`.github/workflows/pr-qa.yaml`)
```
resolve-matrix → analyze-pr ──┐
├→ qa-before (main branch)
├→ qa-after (PR branch)
└→ report (video review, deploy, comment)
```

View File

@@ -0,0 +1,708 @@
#!/usr/bin/env tsx
/**
* QA Research Phase — Claude writes & debugs E2E tests to reproduce bugs
*
* Instead of driving a browser interactively, Claude:
* 1. Reads the issue + a11y snapshot of the UI
* 2. Writes a Playwright E2E test (.spec.ts) that reproduces the bug
* 3. Runs the test → reads errors → rewrites → repeats until it works
* 4. Outputs the passing test + verdict
*
* Tools:
* - inspect(selector) — read a11y tree to understand UI state
* - writeTest(code) — write a Playwright test file
* - runTest() — execute the test and get results
* - done(verdict, summary, testCode) — finish with the working test
*/
import type { Page } from '@playwright/test'
/* eslint-disable import-x/no-unresolved */
// @ts-expect-error — claude-agent-sdk has no type declarations for vue-tsc
import { query, tool, createSdkMcpServer } from '@anthropic-ai/claude-agent-sdk'
/* eslint-enable import-x/no-unresolved */
import { z } from 'zod'
import { mkdirSync, readFileSync, writeFileSync } from 'fs'
import { execSync } from 'child_process'
// ── Types ──
interface ResearchOptions {
page: Page
issueContext: string
qaGuide: string
outputDir: string
serverUrl: string
anthropicApiKey?: string
maxTurns?: number
timeBudgetMs?: number
model?: string
}
export type ReproMethod = 'e2e_test' | 'video' | 'both' | 'none'
export interface ResearchResult {
verdict: 'REPRODUCED' | 'NOT_REPRODUCIBLE' | 'INCONCLUSIVE'
reproducedBy: ReproMethod
summary: string
evidence: string
testCode: string
videoScript?: string
log: Array<{
turn: number
timestampMs: number
toolName: string
toolInput: unknown
toolResult: string
}>
}
// ── Main research function ──
export async function runResearchPhase(
opts: ResearchOptions
): Promise<ResearchResult> {
const { page, issueContext, qaGuide, outputDir, serverUrl, anthropicApiKey } =
opts
const maxTurns = opts.maxTurns ?? 50
let agentDone = false
let finalVerdict: ResearchResult['verdict'] = 'INCONCLUSIVE'
let finalReproducedBy: ReproMethod = 'none'
let finalSummary = 'Agent did not complete'
let finalEvidence = ''
let finalTestCode = ''
let finalVideoScript = ''
let turnCount = 0
let lastPassedTurn = -1
const startTime = Date.now()
const researchLog: ResearchResult['log'] = []
const testDir = `${outputDir}/research`
mkdirSync(testDir, { recursive: true })
const testPath = `${testDir}/reproduce.spec.ts`
// Get initial a11y snapshot for context
let initialA11y = ''
try {
initialA11y = await page.locator('body').ariaSnapshot({ timeout: 5000 })
initialA11y = initialA11y.slice(0, 3000)
} catch {
initialA11y = '(could not capture initial a11y snapshot)'
}
// ── Tool: inspect ──
const inspectTool = tool(
'inspect',
'Read the current accessibility tree to understand UI state. Use this to discover element names, roles, and selectors for your test.',
{
selector: z
.string()
.optional()
.describe(
'Optional filter — only show elements matching this name/role. Omit for full tree.'
)
},
async (args: { selector?: string }) => {
let resultText: string
try {
const ariaText = await page
.locator('body')
.ariaSnapshot({ timeout: 5000 })
if (args.selector) {
const lines = ariaText.split('\n')
const matches = lines.filter((l: string) =>
l.toLowerCase().includes(args.selector!.toLowerCase())
)
resultText =
matches.length > 0
? `Found "${args.selector}":\n${matches.slice(0, 15).join('\n')}`
: `"${args.selector}" not found. Full tree:\n${ariaText.slice(0, 2000)}`
} else {
resultText = ariaText.slice(0, 3000)
}
} catch (e) {
resultText = `inspect failed: ${e instanceof Error ? e.message : e}`
}
researchLog.push({
turn: turnCount,
timestampMs: Date.now() - startTime,
toolName: 'inspect',
toolInput: args,
toolResult: resultText.slice(0, 500)
})
return { content: [{ type: 'text' as const, text: resultText }] }
}
)
// ── Tool: readFixture ──
const readFixtureTool = tool(
'readFixture',
'Read a fixture or helper file from browser_tests/fixtures/ to understand the API. Use this to discover available methods on comfyPage helpers before writing your test.',
{
path: z
.string()
.describe(
'Relative path within browser_tests/fixtures/, e.g. "helpers/CanvasHelper.ts" or "components/Topbar.ts" or "ComfyPage.ts"'
)
},
async (args: { path: string }) => {
let resultText: string
try {
const fullPath = `${projectRoot}/browser_tests/fixtures/${args.path}`
const content = readFileSync(fullPath, 'utf-8')
resultText = content.slice(0, 4000)
if (content.length > 4000) {
resultText += `\n\n... (truncated, ${content.length} total chars)`
}
} catch (e) {
resultText = `Could not read fixture: ${e instanceof Error ? e.message : e}`
}
researchLog.push({
turn: turnCount,
timestampMs: Date.now() - startTime,
toolName: 'readFixture',
toolInput: args,
toolResult: resultText.slice(0, 500)
})
return { content: [{ type: 'text' as const, text: resultText }] }
}
)
// ── Tool: readTest ──
const readTestTool = tool(
'readTest',
'Read an existing E2E test file from browser_tests/tests/ to learn patterns and conventions used in this project.',
{
path: z
.string()
.describe(
'Relative path within browser_tests/tests/, e.g. "workflow.spec.ts" or "subgraph.spec.ts"'
)
},
async (args: { path: string }) => {
let resultText: string
try {
const fullPath = `${projectRoot}/browser_tests/tests/${args.path}`
const content = readFileSync(fullPath, 'utf-8')
resultText = content.slice(0, 4000)
if (content.length > 4000) {
resultText += `\n\n... (truncated, ${content.length} total chars)`
}
} catch (e) {
// List available test files if the path doesn't exist
try {
const { readdirSync } = await import('fs')
const files = readdirSync(`${projectRoot}/browser_tests/tests/`)
.filter((f: string) => f.endsWith('.spec.ts'))
.slice(0, 30)
resultText = `File not found: ${args.path}\n\nAvailable test files:\n${files.join('\n')}`
} catch {
resultText = `Could not read test: ${e instanceof Error ? e.message : e}`
}
}
researchLog.push({
turn: turnCount,
timestampMs: Date.now() - startTime,
toolName: 'readTest',
toolInput: args,
toolResult: resultText.slice(0, 500)
})
return { content: [{ type: 'text' as const, text: resultText }] }
}
)
// ── Tool: writeTest ──
const writeTestTool = tool(
'writeTest',
'Write a Playwright E2E test file that reproduces the bug. The test should assert the broken behavior exists.',
{
code: z
.string()
.describe('Complete Playwright test file content (.spec.ts)')
},
async (args: { code: string }) => {
writeFileSync(testPath, args.code)
researchLog.push({
turn: turnCount,
timestampMs: Date.now() - startTime,
toolName: 'writeTest',
toolInput: { path: testPath, codeLength: args.code.length },
toolResult: `Test written to ${testPath} (${args.code.length} chars)`
})
return {
content: [
{
type: 'text' as const,
text: `Test written to ${testPath}. Use runTest() to execute it.`
}
]
}
}
)
// ── Tool: runTest ──
// Place test in browser_tests/ so Playwright config finds fixtures
const projectRoot = process.cwd()
const browserTestPath = `${projectRoot}/browser_tests/tests/qa-reproduce.spec.ts`
const runTestTool = tool(
'runTest',
'Run the Playwright test and get results. Returns stdout/stderr including assertion errors.',
{},
async () => {
turnCount++
// Copy the test to browser_tests/tests/ where Playwright expects it
const { copyFileSync } = await import('fs')
try {
copyFileSync(testPath, browserTestPath)
} catch {
// directory may not exist
mkdirSync(`${projectRoot}/browser_tests/tests`, { recursive: true })
copyFileSync(testPath, browserTestPath)
}
let resultText: string
try {
const output = execSync(
`cd "${projectRoot}" && npx playwright test browser_tests/tests/qa-reproduce.spec.ts --reporter=list --timeout=30000 --retries=0 --workers=1 2>&1`,
{
timeout: 90000,
encoding: 'utf-8',
env: {
...process.env,
COMFYUI_BASE_URL: serverUrl
}
}
)
resultText = `TEST PASSED:\n${output.slice(-1500)}`
} catch (e) {
const err = e as { stdout?: string; stderr?: string; message?: string }
const output = (err.stdout || '') + '\n' + (err.stderr || '')
resultText = `TEST FAILED:\n${output.slice(-2000)}`
}
researchLog.push({
turn: turnCount,
timestampMs: Date.now() - startTime,
toolName: 'runTest',
toolInput: { testPath },
toolResult: resultText.slice(0, 1000)
})
// Auto-save passing test code for fallback completion — but only if
// the test contains a bug-specific assertion (not just a discovery/debug test)
if (resultText.startsWith('TEST PASSED')) {
try {
const code = readFileSync(browserTestPath, 'utf-8')
const hasBugAssertion =
/expect\s*\(/.test(code) &&
!/^\s*expect\([^)]+\)\.toBeDefined\(\)/m.test(code) &&
!/^\s*expect\([^)]+\)\.toBeGreaterThan\(0\)/m.test(code) &&
!/Inspect|Find|Debug|discover/i.test(
code.match(/test\(['"`]([^'"`]+)/)?.[1] ?? ''
)
if (hasBugAssertion) {
finalTestCode = code
lastPassedTurn = turnCount
}
} catch {
// ignore
}
resultText +=
'\n\n⚠ Test PASSED — call done() now with verdict REPRODUCED and the test code. Do NOT write more tests.'
}
return { content: [{ type: 'text' as const, text: resultText }] }
}
)
// ── Tool: done ──
const doneTool = tool(
'done',
'Finish research with verdict and the final test code.',
{
verdict: z.enum(['REPRODUCED', 'NOT_REPRODUCIBLE', 'INCONCLUSIVE']),
reproducedBy: z
.enum(['e2e_test', 'video', 'both', 'none'])
.describe(
'How the bug was proven: e2e_test = Playwright assertion passed, video = visual evidence only, both = both methods, none = not reproduced'
),
summary: z.string().describe('What you found and why'),
evidence: z.string().describe('Test output that proves the verdict'),
testCode: z
.string()
.describe(
'Final Playwright test code. If REPRODUCED, this test asserts the bug exists and passes.'
),
videoScript: z
.string()
.optional()
.describe(
'Demowright video script for Phase 2 demo recording. REQUIRED when verdict is REPRODUCED. A separate test file using createVideoScript with title, segments, and outro. Do NOT include demowright imports in testCode.'
)
},
async (args: {
verdict: ResearchResult['verdict']
reproducedBy: ReproMethod
summary: string
evidence: string
testCode: string
videoScript?: string
}) => {
agentDone = true
finalVerdict = args.verdict
finalReproducedBy = args.reproducedBy
finalSummary = args.summary
finalEvidence = args.evidence
finalTestCode = args.testCode
finalVideoScript = args.videoScript ?? ''
writeFileSync(testPath, args.testCode)
if (args.videoScript) {
writeFileSync(`${outputDir}/video-script.spec.ts`, args.videoScript)
}
return {
content: [
{ type: 'text' as const, text: `Research complete: ${args.verdict}` }
]
}
}
)
// ── MCP Server ──
const server = createSdkMcpServer({
name: 'qa-research',
version: '1.0.0',
tools: [
inspectTool,
readFixtureTool,
readTestTool,
writeTestTool,
runTestTool,
doneTool
]
})
// ── System prompt ──
const systemPrompt = `You are a senior QA engineer who writes Playwright E2E tests to reproduce reported bugs.
## Your tools
- inspect(selector?) — Read the accessibility tree to understand the current UI. Use to discover selectors, element names, and UI state.
- readFixture(path) — Read fixture source code from browser_tests/fixtures/. Use to discover available methods. E.g. "helpers/CanvasHelper.ts", "components/Topbar.ts", "ComfyPage.ts"
- readTest(path) — Read an existing test from browser_tests/tests/ to learn patterns. E.g. "workflow.spec.ts". Pass any name to list available files.
- writeTest(code) — Write a Playwright test file (.spec.ts)
- runTest() — Execute the test and get results (pass/fail + errors)
- done(verdict, summary, evidence, testCode) — Finish with the final test
## Workflow
1. Read the issue description carefully. Think about:
- What PRECONDITIONS are needed? (many nodes on canvas? specific layout? saved workflow? subgraph?)
- What HIDDEN ASSUMPTIONS exist? (e.g. "z-index bug" means nodes must overlap → need a crowded canvas)
- What specific UI STATE triggers the bug? (dirty workflow? collapsed node? specific menu open?)
2. Use readTest() to read 1-2 existing tests similar to the bug:
- For menu/workflow bugs: readTest("workflow.spec.ts") or readTest("topbarMenu.spec.ts")
- For node/canvas bugs: readTest("nodeInteraction.spec.ts") or readTest("copyPaste.spec.ts")
- For settings bugs: readTest("settingDialogSearch.spec.ts")
- For subgraph bugs: readTest("subgraph.spec.ts")
3. Use inspect() to understand the current UI state and discover element selectors
4. If unsure about the fixture API, use readFixture("ComfyPage.ts") or relevant helper
5. Write a Playwright test that:
- FIRST sets up the preconditions (add multiple nodes, create specific layout, save workflow, etc.)
- THEN performs the reproduction steps from the issue
- FINALLY asserts the BROKEN behavior (the bug) — so the test PASSES when the bug exists
- Think like a tester: the bug may only appear under specific conditions that the reporter assumed were obvious
6. Run the test with runTest()
7. If it fails, ANALYZE the error before retrying:
- Is it a selector issue? Use inspect() to find the right element
- Is it a timing issue? The UI may need time to update — use nextFrame() or expect.poll()
- Is the precondition wrong? Maybe the bug only appears with MORE nodes, AFTER a save, etc.
- Try a DIFFERENT approach, not the same code with minor tweaks
8. Call done() with the final verdict and test code
## Test writing guidelines
- Import the project fixture: \`import { comfyPageFixture as test } from '../fixtures/ComfyPage'\`
- Import expect: \`import { expect } from '@playwright/test'\`
- The fixture provides \`comfyPage\` which has all the helpers listed below
- If the bug IS present, the test should PASS. If the bug is fixed, the test would FAIL.
- Keep tests focused and minimal — test ONLY the reported bug
- Write ONE test, not multiple. Focus on the single clearest reproduction.
- The test file will be placed in browser_tests/tests/qa-reproduce.spec.ts
- Use \`comfyPage.nextFrame()\` after interactions that trigger UI updates
- NEVER use \`page.waitForTimeout()\` — use Locator actions and retrying assertions instead
- ALWAYS call done() when finished, even if the test passed — do not keep iterating after a passing test
- CRITICAL: If your test FAILS 3 times in a row with the same or similar error, call done(NOT_REPRODUCIBLE) immediately. Do NOT keep retrying the same approach — try a completely different strategy or give up. Spending 20+ tool calls on failing tests is wasteful.
- Budget your turns: spend at most 3 turns on inspect/readFixture, 2 turns writing the first test, then max 3 fix attempts. If still failing after ~10 tool calls, call done().
- Use \`expect.poll()\` for async assertions: \`await expect.poll(() => comfyPage.nodeOps.getGraphNodesCount()).toBe(8)\`
- CRITICAL: Your assertions must be SPECIFIC TO THE BUG. A test that asserts \`expect(count).toBeGreaterThan(0)\` proves nothing — it would pass even without the bug. Instead assert the exact broken state, e.g. \`expect(clonedWidgets).toHaveLength(0)\` (missing widgets) or \`expect(zIndex).toBeLessThan(parentZIndex)\` (wrong z-order). If a test passes trivially, it's a false positive.
- NEVER write "debug", "discovery", or "inspect node types" tests. These waste turns and produce false REPRODUCED verdicts. If you need to discover node type names, use inspect() or readFixture() — not a passing test.
- If you cannot write a bug-specific assertion, call done() with verdict NOT_REPRODUCIBLE and explain why.
## ComfyPage Fixture API Reference
### Core properties
- \`comfyPage.page\` — raw Playwright Page
- \`comfyPage.canvas\` — Locator for #graph-canvas
- \`comfyPage.queueButton\` — "Queue Prompt" button
- \`comfyPage.runButton\` — "Run" button (new UI)
- \`comfyPage.confirmDialog\` — ConfirmDialog (has .confirm, .delete, .overwrite, .reject locators + .click(name) method)
- \`comfyPage.nextFrame()\` — wait for next requestAnimationFrame
- \`comfyPage.setup()\` — navigate + wait for app ready (called automatically by fixture)
### Menu (comfyPage.menu)
- \`comfyPage.menu.topbar\` — Topbar helper:
- \`.triggerTopbarCommand(['File', 'Save As'])\` — navigate menu hierarchy
- \`.openTopbarMenu()\` / \`.closeTopbarMenu()\` — open/close hamburger
- \`.openSubmenu('File')\` — hover to open submenu, returns submenu Locator
- \`.getTabNames()\` — get all open workflow tab names
- \`.getActiveTabName()\` — get active tab name
- \`.getWorkflowTab(name)\` — get tab Locator
- \`.closeWorkflowTab(name)\` — close a tab
- \`.saveWorkflow(name)\` / \`.saveWorkflowAs(name)\` / \`.exportWorkflow(name)\`
- \`.switchTheme('dark' | 'light')\`
- \`comfyPage.menu.workflowsTab\` — WorkflowsSidebarTab:
- \`.open()\` / \`.close()\` — toggle workflows sidebar
- \`.getTopLevelSavedWorkflowNames()\` — list saved workflow names
- \`comfyPage.menu.nodeLibraryTab\` — NodeLibrarySidebarTab
- \`comfyPage.menu.assetsTab\` — AssetsSidebarTab
### Canvas (comfyPage.canvasOps)
- \`.click({x, y})\` — click at position on canvas
- \`.rightClick(x, y)\` — right-click (opens context menu)
- \`.doubleClick()\` — double-click canvas (opens node search)
- \`.clickEmptySpace()\` — click known empty area
- \`.dragAndDrop(source, target)\` — drag from source to target position
- \`.pan(offset, safeSpot?)\` — pan canvas by offset
- \`.zoom(deltaY, steps?)\` — zoom via scroll wheel
- \`.resetView()\` — reset zoom/pan to default
- \`.getScale()\` / \`.setScale(n)\` — get/set canvas zoom
- \`.getNodeCenterByTitle(title)\` — get screen coords of node center
- \`.disconnectEdge()\` / \`.connectEdge()\` — default graph edge operations
### Node Operations (comfyPage.nodeOps)
- \`.getGraphNodesCount()\` — count all nodes
- \`.getSelectedGraphNodesCount()\` — count selected nodes
- \`.getNodes()\` — get all nodes
- \`.getFirstNodeRef()\` — get NodeReference for first node
- \`.getNodeRefById(id)\` — get NodeReference by ID
- \`.getNodeRefsByType(type)\` — get all nodes of a type
- \`.waitForGraphNodes(count)\` — wait until node count matches
### Settings (comfyPage.settings)
- \`.setSetting(id, value)\` — change a ComfyUI setting
- \`.getSetting(id)\` — read current setting value
- Common setting IDs:
- \`'Comfy.UseNewMenu'\` — 'Top' | 'Bottom' | 'Disabled'
- \`'Comfy.Locale'\` — 'en' | 'zh' | 'ja' | 'ko' | 'ru' | 'fr' | 'es' etc. (change UI language)
- \`'Comfy.NodeBadge.NodeSourceBadgeMode'\` — node badge display
### Keyboard (comfyPage.keyboard)
- \`.undo()\` / \`.redo()\` — Ctrl+Z / Ctrl+Y
- \`.bypass()\` — Ctrl+B
- \`.selectAll()\` — Ctrl+A
- \`.ctrlSend(key)\` — send Ctrl+key
### Workflow (comfyPage.workflow)
- \`.loadWorkflow(name)\` — load from browser_tests/assets/{name}.json
- \`.setupWorkflowsDirectory(structure)\` — setup test directory
- \`.deleteWorkflow(name)\`
- \`.isCurrentWorkflowModified()\` — check dirty state
- Available subgraph assets: loadWorkflow('subgraphs/basic-subgraph'), 'subgraphs/nested-subgraph', 'subgraphs/subgraph-with-promoted-text-widget', etc.
### Context Menu (comfyPage.contextMenu)
- \`.openFor(locator)\` — right-click locator and wait for menu
- \`.clickMenuItem(name)\` — click a menu item by name
- \`.isVisible()\` — check if context menu is showing
- \`.assertHasItems(items)\` — assert menu contains items
### Queue & Assets (comfyPage.assets)
- \`comfyPage.runButton.click()\` — execute current workflow (backend runs with --cpu in CI)
- \`comfyPage.assets.mockOutputHistory(jobs)\` — mock queue history with fake job items
- \`comfyPage.assets.mockEmptyState()\` — clear all mocked state
- Queue overlay: \`page.getByTestId('queue-overlay-toggle')\` to open queue panel
### Subgraph (comfyPage.subgraph)
- \`.isInSubgraph()\` — check if currently viewing a subgraph
- \`.getNodeCount()\` — nodes in current graph view
- \`.getSlotCount('input'|'output')\` — I/O slot count
- \`.connectToInput(sourceNode, slotIdx, inputName)\` — connect to subgraph input
- \`.exitViaBreadcrumb()\` — navigate out of subgraph
- \`.convertDefaultKSamplerToSubgraph()\` — helper: convert default workflow node to subgraph
- NodeReference: \`.convertToSubgraph()\`, \`.navigateIntoSubgraph()\`
### Other helpers
- \`comfyPage.settingDialog\` — SettingDialog component
- \`comfyPage.searchBox\` / \`comfyPage.searchBoxV2\` — node search
- \`comfyPage.toast\` — ToastHelper (\`.visibleToasts\`)
- \`comfyPage.vueNodes\` — VueNodeHelpers (\`.enterSubgraph(nodeId)\`, \`.selectNode(nodeId)\`)
- \`comfyPage.bottomPanel\` — BottomPanel
- \`comfyPage.clipboard\` — ClipboardHelper
- \`comfyPage.dragDrop\` — DragDropHelper
### Available fixture files (use readFixture to explore)
- ComfyPage.ts — main fixture with all helpers
- helpers/CanvasHelper.ts, NodeOperationsHelper.ts, WorkflowHelper.ts
- helpers/KeyboardHelper.ts, SettingsHelper.ts, SubgraphHelper.ts
- components/Topbar.ts, ContextMenu.ts, SettingDialog.ts, SidebarTab.ts
## Video Script (IMPORTANT — provide via done() tool)
When calling done(), provide a \`videoScript\` parameter with a SEPARATE test file that uses demowright's createVideoScript.
Do NOT put demowright imports in testCode — they won't resolve in Phase 1.
The videoScript is a complete, standalone Playwright test file for Phase 2 demo recording:
\`\`\`typescript
import { comfyPageFixture as test } from '../fixtures/ComfyPage'
import { createVideoScript, showTitleCard, hideTitleCard } from 'demowright/video-script'
test('Demo: Bug Title', async ({ comfyPage }) => {
// Show title card IMMEDIATELY — covers the screen while setup runs behind it
await showTitleCard(comfyPage.page, 'Bug Title Here', { subtitle: 'Issue #NNNN' })
// Setup runs while title card is visible
await comfyPage.settings.setSetting('Comfy.UseNewMenu', 'Top')
await comfyPage.workflow.setupWorkflowsDirectory({})
// Remove early title card before script starts (script will show its own)
await hideTitleCard(comfyPage.page)
const script = createVideoScript()
.title('Bug Title Here', { subtitle: 'Issue #NNNN', durationMs: 4000 })
.segment('Step 1: description of what we do', async (pace) => {
await pace() // narration finishes FIRST
await comfyPage.menu.topbar.saveWorkflow('name') // THEN action
await comfyPage.page.waitForTimeout(2000) // pause for viewer
})
.segment('Bug evidence: what we see proves the bug', async (pace) => {
await pace()
await comfyPage.page.waitForTimeout(5000) // hold on evidence
})
.outro({ text: 'Bug Reproduced', subtitle: 'Summary' })
await script.render(comfyPage.page)
})
\`\`\`
Key API:
- \`.title(text, {subtitle?, durationMs?})\` — title card (4s default)
- \`.segment(narrationText, async (pace) => { await pace(); ...actions... })\` — TTS narrated step
- \`.outro({text?, subtitle?, durationMs?})\` — ending card
- \`pace()\` — wait for narration audio to finish
CRITICAL TIMING: Call \`await pace()\` FIRST in each segment, BEFORE the Playwright actions.
This makes the narration play and finish, THEN the actions execute — so viewers hear what's about
to happen before it happens. Pattern:
\`\`\`typescript
.segment('Now we save the workflow as a new name', async (pace) => {
await pace() // narration finishes first
await comfyPage.menu.topbar.saveWorkflowAs('new-name') // then action happens
await comfyPage.page.waitForTimeout(2000) // pause so viewer sees the result
})
\`\`\`
IMPORTANT RULES for videoScript:
1. You MUST provide videoScript when verdict is REPRODUCED — every reproduced bug needs a narrated demo
2. Call showTitleCard() BEFORE setup, run setup behind it, call hideTitleCard() before createVideoScript() — see example
3. Call \`await pace()\` FIRST in each segment callback, BEFORE actions
4. Add \`waitForTimeout(2000)\` after each action so viewers can see the result
5. Final evidence segment: hold for 5+ seconds
6. Reproduce the same steps as testCode but slower with clear narration
## Current UI state (accessibility tree)
${initialA11y}
${qaGuide ? `## QA Analysis Guide\n${qaGuide}\n` : ''}
## Issue to Reproduce
${issueContext}`
// ── Run the agent ──
console.warn('Starting research phase (Claude writes E2E tests)...')
try {
for await (const message of query({
prompt:
'Write a Playwright E2E test that reproduces the reported bug. Use inspect() to discover selectors, readFixture() or readTest() if you need to understand the fixture API or see existing test patterns, writeTest() to write the test, runTest() to execute it. Iterate until it works or you determine the bug cannot be reproduced.',
options: {
model: opts.model ?? 'claude-sonnet-4-6',
systemPrompt,
...(anthropicApiKey ? { apiKey: anthropicApiKey } : {}),
maxTurns,
mcpServers: { 'qa-research': server },
allowedTools: [
'mcp__qa-research__inspect',
'mcp__qa-research__readFixture',
'mcp__qa-research__readTest',
'mcp__qa-research__writeTest',
'mcp__qa-research__runTest',
'mcp__qa-research__done'
]
}
})) {
if (message.type === 'assistant' && message.message?.content) {
for (const block of message.message.content) {
if ('text' in block && block.text) {
console.warn(` Claude: ${block.text.slice(0, 200)}`)
}
if ('name' in block) {
console.warn(
` Tool: ${block.name}(${JSON.stringify(block.input).slice(0, 100)})`
)
}
}
}
if (agentDone) break
}
} catch (e) {
const errMsg = e instanceof Error ? e.message : String(e)
console.warn(`Research error: ${errMsg}`)
// Detect billing/auth errors and surface them clearly
if (
errMsg.includes('Credit balance is too low') ||
errMsg.includes('insufficient_quota') ||
errMsg.includes('rate_limit')
) {
finalSummary = `API error: ${errMsg.slice(0, 200)}`
finalEvidence = 'Agent could not start due to API billing/auth issue'
console.warn(
'::error::Anthropic API credits exhausted — cannot run research phase'
)
}
}
// Auto-complete: if a test passed but done() was never called, use the passing test
if (!agentDone && lastPassedTurn >= 0 && finalTestCode) {
console.warn(
`Auto-completing: test passed at turn ${lastPassedTurn} but done() was not called`
)
finalVerdict = 'REPRODUCED'
finalReproducedBy = 'e2e_test'
finalSummary = `Test passed at turn ${lastPassedTurn} (auto-completed — agent did not call done())`
finalEvidence = `Test passed with exit code 0`
}
const result: ResearchResult = {
verdict: finalVerdict,
reproducedBy: finalReproducedBy,
summary: finalSummary,
evidence: finalEvidence,
testCode: finalTestCode,
videoScript: finalVideoScript || undefined,
log: researchLog
}
writeFileSync(`${testDir}/research-log.json`, JSON.stringify(result, null, 2))
console.warn(
`Research complete: ${finalVerdict} (${researchLog.length} tool calls)`
)
return result
}

View File

@@ -0,0 +1,84 @@
import { describe, expect, it } from 'vitest'
import { extractMediaUrls } from './qa-analyze-pr'
describe('extractMediaUrls', () => {
it('extracts markdown image URLs', () => {
const text = '![screenshot](https://example.com/image.png)'
expect(extractMediaUrls(text)).toEqual(['https://example.com/image.png'])
})
it('extracts multiple markdown images', () => {
const text = [
'![before](https://example.com/before.png)',
'Some text',
'![after](https://example.com/after.jpg)'
].join('\n')
expect(extractMediaUrls(text)).toEqual([
'https://example.com/before.png',
'https://example.com/after.jpg'
])
})
it('extracts raw URLs with media extensions', () => {
const text = 'Check this: https://cdn.example.com/demo.mp4 for details'
expect(extractMediaUrls(text)).toEqual(['https://cdn.example.com/demo.mp4'])
})
it('extracts GitHub user-attachments URLs', () => {
const text =
'https://github.com/user-attachments/assets/abc12345-6789-0def-1234-567890abcdef'
expect(extractMediaUrls(text)).toEqual([
'https://github.com/user-attachments/assets/abc12345-6789-0def-1234-567890abcdef'
])
})
it('extracts private-user-images URLs', () => {
const text =
'https://private-user-images.githubusercontent.com/12345/abcdef-1234?jwt=token123'
expect(extractMediaUrls(text)).toEqual([
'https://private-user-images.githubusercontent.com/12345/abcdef-1234?jwt=token123'
])
})
it('extracts URLs with query parameters', () => {
const text = 'https://example.com/image.png?w=800&h=600'
expect(extractMediaUrls(text)).toEqual([
'https://example.com/image.png?w=800&h=600'
])
})
it('deduplicates URLs', () => {
const text = [
'![img](https://example.com/same.png)',
'![img2](https://example.com/same.png)',
'Also https://example.com/same.png'
].join('\n')
expect(extractMediaUrls(text)).toEqual(['https://example.com/same.png'])
})
it('returns empty array for empty input', () => {
expect(extractMediaUrls('')).toEqual([])
})
it('returns empty array for text with no media URLs', () => {
expect(extractMediaUrls('Just some text without any URLs')).toEqual([])
})
it('handles mixed media types', () => {
const text = [
'![screen](https://example.com/screenshot.png)',
'Video: https://example.com/demo.webm',
'![gif](https://example.com/animation.gif)'
].join('\n')
const urls = extractMediaUrls(text)
expect(urls).toContain('https://example.com/screenshot.png')
expect(urls).toContain('https://example.com/demo.webm')
expect(urls).toContain('https://example.com/animation.gif')
})
it('ignores non-http URLs in markdown', () => {
const text = '![local](./local-image.png)'
expect(extractMediaUrls(text)).toEqual([])
})
})

View File

@@ -0,0 +1,799 @@
#!/usr/bin/env tsx
/**
* QA PR Analysis Script
*
* Deeply analyzes a PR using Gemini Pro to generate targeted QA guides
* for before/after recording sessions. Fetches PR thread, extracts media,
* and produces structured test plans.
*
* Usage:
* pnpm exec tsx scripts/qa-analyze-pr.ts \
* --pr-number 10270 \
* --repo owner/repo \
* --output-dir qa-guides/ \
* [--model gemini-3.1-pro-preview]
*
* Env: GEMINI_API_KEY (required)
*/
import { execSync } from 'node:child_process'
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { GoogleGenerativeAI } from '@google/generative-ai'
// ── Types ──
interface QaGuideStep {
action: string
description: string
expected_before?: string
expected_after?: string
}
interface QaGuide {
summary: string
test_focus: string
prerequisites: string[]
steps: QaGuideStep[]
visual_checks: string[]
}
interface PrThread {
title: string
body: string
labels: string[]
issueComments: string[]
reviewComments: string[]
reviews: string[]
diff: string
}
type TargetType = 'pr' | 'issue'
interface Options {
prNumber: string
repo: string
outputDir: string
model: string
apiKey: string
mediaBudgetBytes: number
maxVideoBytes: number
type: TargetType
}
// ── CLI parsing ──
function parseArgs(): Options {
const args = process.argv.slice(2)
const opts: Partial<Options> = {
model: 'gemini-3.1-pro-preview',
apiKey: process.env.GEMINI_API_KEY || '',
mediaBudgetBytes: 20 * 1024 * 1024,
maxVideoBytes: 10 * 1024 * 1024,
type: 'pr'
}
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '--pr-number':
opts.prNumber = args[++i]
break
case '--repo':
opts.repo = args[++i]
break
case '--output-dir':
opts.outputDir = args[++i]
break
case '--model':
opts.model = args[++i]
break
case '--type':
opts.type = args[++i] as TargetType
break
case '--help':
console.warn(
'Usage: qa-analyze-pr.ts --pr-number <num> --repo <owner/repo> --output-dir <path> [--model <model>] [--type pr|issue]'
)
process.exit(0)
}
}
if (!opts.prNumber || !opts.repo || !opts.outputDir) {
console.error(
'Required: --pr-number <num> --repo <owner/repo> --output-dir <path>'
)
process.exit(1)
}
if (!opts.apiKey) {
console.error('GEMINI_API_KEY environment variable is required')
process.exit(1)
}
return opts as Options
}
// ── PR thread fetching ──
function ghExec(cmd: string): string {
try {
return execSync(cmd, {
encoding: 'utf-8',
timeout: 30_000,
stdio: ['pipe', 'pipe', 'pipe']
}).trim()
} catch (err) {
console.warn(`gh command failed: ${cmd}`)
console.warn((err as Error).message)
return ''
}
}
function fetchPrThread(prNumber: string, repo: string): PrThread {
console.warn('Fetching PR thread...')
const prView = ghExec(
`gh pr view ${prNumber} --repo ${repo} --json title,body,labels`
)
const prData = prView
? JSON.parse(prView)
: { title: '', body: '', labels: [] }
const issueCommentsRaw = ghExec(
`gh api repos/${repo}/issues/${prNumber}/comments --paginate`
)
const issueComments: string[] = issueCommentsRaw
? JSON.parse(issueCommentsRaw).map((c: { body: string }) => c.body)
: []
const reviewCommentsRaw = ghExec(
`gh api repos/${repo}/pulls/${prNumber}/comments --paginate`
)
const reviewComments: string[] = reviewCommentsRaw
? JSON.parse(reviewCommentsRaw).map((c: { body: string }) => c.body)
: []
const reviewsRaw = ghExec(
`gh api repos/${repo}/pulls/${prNumber}/reviews --paginate`
)
const reviews: string[] = reviewsRaw
? JSON.parse(reviewsRaw)
.filter((r: { body: string }) => r.body)
.map((r: { body: string }) => r.body)
: []
const diff = ghExec(`gh pr diff ${prNumber} --repo ${repo}`)
console.warn(
`PR #${prNumber}: "${prData.title}" | ` +
`${issueComments.length} issue comments, ` +
`${reviewComments.length} review comments, ` +
`${reviews.length} reviews, ` +
`diff: ${diff.length} chars`
)
return {
title: prData.title || '',
body: prData.body || '',
labels: (prData.labels || []).map((l: { name: string }) => l.name),
issueComments,
reviewComments,
reviews,
diff
}
}
interface IssueThread {
title: string
body: string
labels: string[]
comments: string[]
}
function fetchIssueThread(issueNumber: string, repo: string): IssueThread {
console.warn('Fetching issue thread...')
const issueView = ghExec(
`gh issue view ${issueNumber} --repo ${repo} --json title,body,labels`
)
const issueData = issueView
? JSON.parse(issueView)
: { title: '', body: '', labels: [] }
const commentsRaw = ghExec(
`gh api repos/${repo}/issues/${issueNumber}/comments --paginate`
)
const comments: string[] = commentsRaw
? JSON.parse(commentsRaw).map((c: { body: string }) => c.body)
: []
console.warn(
`Issue #${issueNumber}: "${issueData.title}" | ` +
`${comments.length} comments`
)
return {
title: issueData.title || '',
body: issueData.body || '',
labels: (issueData.labels || []).map((l: { name: string }) => l.name),
comments
}
}
// ── Media extraction ──
const MEDIA_EXTENSIONS = /\.(png|jpg|jpeg|gif|webp|mp4|webm|mov)$/i
const MEDIA_URL_PATTERNS = [
// Markdown images: ![alt](url)
/!\[[^\]]*\]\(([^)]+)\)/g,
// GitHub user-attachments
/https:\/\/github\.com\/user-attachments\/assets\/[a-f0-9-]+/g,
// Private user images
/https:\/\/private-user-images\.githubusercontent\.com\/[^\s)"]+/g,
// Raw URLs with media extensions (standalone or in text)
/(?<!="|=')https?:\/\/[^\s)<>"]+\.(?:png|jpg|jpeg|gif|webp|mp4|webm|mov)(?:\?[^\s)<>"]*)?/gi
]
export function extractMediaUrls(text: string): string[] {
if (!text) return []
const urls = new Set<string>()
for (const pattern of MEDIA_URL_PATTERNS) {
// Reset lastIndex for global patterns
pattern.lastIndex = 0
let match: RegExpExecArray | null
while ((match = pattern.exec(text)) !== null) {
// For markdown images, the URL is in capture group 1
const url = match[1] || match[0]
// Clean trailing markdown/html artifacts
const cleaned = url.replace(/[)>"'\s]+$/, '')
if (cleaned.startsWith('http')) {
urls.add(cleaned)
}
}
}
return [...urls]
}
// ── Media downloading ──
const ALLOWED_MEDIA_DOMAINS = [
'github.com',
'raw.githubusercontent.com',
'user-images.githubusercontent.com',
'private-user-images.githubusercontent.com',
'objects.githubusercontent.com',
'github.githubassets.com'
]
function isAllowedMediaDomain(url: string): boolean {
try {
const hostname = new URL(url).hostname
return ALLOWED_MEDIA_DOMAINS.some(
(domain) => hostname === domain || hostname.endsWith(`.${domain}`)
)
} catch {
return false
}
}
async function downloadMedia(
urls: string[],
outputDir: string,
budgetBytes: number,
maxVideoBytes: number
): Promise<Array<{ path: string; mimeType: string }>> {
const downloaded: Array<{ path: string; mimeType: string }> = []
let totalBytes = 0
const mediaDir = resolve(outputDir, 'media')
mkdirSync(mediaDir, { recursive: true })
for (const url of urls) {
if (totalBytes >= budgetBytes) {
console.warn(
`Media budget exhausted (${totalBytes} bytes), skipping rest`
)
break
}
if (!isAllowedMediaDomain(url)) {
console.warn(`Skipping non-GitHub URL: ${url.slice(0, 80)}`)
continue
}
try {
const response = await fetch(url, {
signal: AbortSignal.timeout(15_000),
headers: { Accept: 'image/*,video/*' },
redirect: 'follow'
})
if (!response.ok) {
console.warn(`Failed to download ${url}: ${response.status}`)
continue
}
const contentLength = response.headers.get('content-length')
if (contentLength) {
const declaredSize = Number.parseInt(contentLength, 10)
if (declaredSize > budgetBytes - totalBytes) {
console.warn(
`Content-Length ${declaredSize} would exceed budget, skipping ${url}`
)
continue
}
}
const contentType = response.headers.get('content-type') || ''
const buffer = Buffer.from(await response.arrayBuffer())
// Skip oversized videos
const isVideo =
contentType.startsWith('video/') || /\.(mp4|webm|mov)$/i.test(url)
if (isVideo && buffer.length > maxVideoBytes) {
console.warn(
`Skipping large video ${url} (${(buffer.length / 1024 / 1024).toFixed(1)}MB > ${(maxVideoBytes / 1024 / 1024).toFixed(0)}MB cap)`
)
continue
}
if (totalBytes + buffer.length > budgetBytes) {
console.warn(`Would exceed budget, skipping ${url}`)
continue
}
const ext = guessExtension(url, contentType)
const filename = `media-${downloaded.length}${ext}`
const filepath = resolve(mediaDir, filename)
writeFileSync(filepath, buffer)
totalBytes += buffer.length
const mimeType = contentType.split(';')[0].trim() || guessMimeType(ext)
downloaded.push({ path: filepath, mimeType })
console.warn(
`Downloaded: ${url.slice(0, 80)}... (${(buffer.length / 1024).toFixed(0)}KB)`
)
} catch (err) {
console.warn(`Failed to download ${url}: ${(err as Error).message}`)
}
}
console.warn(
`Downloaded ${downloaded.length}/${urls.length} media files ` +
`(${(totalBytes / 1024 / 1024).toFixed(1)}MB)`
)
return downloaded
}
function guessExtension(url: string, contentType: string): string {
const urlMatch = url.match(MEDIA_EXTENSIONS)
if (urlMatch) return urlMatch[0].toLowerCase()
const typeMap: Record<string, string> = {
'image/png': '.png',
'image/jpeg': '.jpg',
'image/gif': '.gif',
'image/webp': '.webp',
'video/mp4': '.mp4',
'video/webm': '.webm'
}
return typeMap[contentType.split(';')[0]] || '.bin'
}
function guessMimeType(ext: string): string {
const map: Record<string, string> = {
'.png': 'image/png',
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.gif': 'image/gif',
'.webp': 'image/webp',
'.mp4': 'video/mp4',
'.webm': 'video/webm',
'.mov': 'video/quicktime'
}
return map[ext] || 'application/octet-stream'
}
// ── Gemini analysis ──
function buildIssueAnalysisPrompt(issue: IssueThread): string {
const allText = [
`# Issue: ${issue.title}`,
'',
'## Description',
issue.body,
'',
issue.comments.length > 0
? `## Comments\n${issue.comments.join('\n\n---\n\n')}`
: ''
]
.filter(Boolean)
.join('\n')
return `You are a senior QA engineer analyzing a bug report for ComfyUI frontend — a node-based visual workflow editor for AI image generation (Vue 3 + TypeScript).
The UI has:
- A large canvas (1280x720 viewport) showing a node graph centered at ~(640, 400)
- Nodes are boxes with input/output slots connected by wires
- A hamburger menu (top-left C logo) with File, Edit, Help submenus
- Sidebars (Workflows, Node Library, Models)
- A topbar with workflow tabs and Queue button
- The default workflow loads with these nodes (approximate center coordinates):
- Load Checkpoint (~150, 300), CLIP Text Encode x2 (~450, 250 and ~450, 450)
- Empty Latent Image (~450, 600), KSampler (~750, 350), VAE Decode (~1000, 350), Save Image (~1200, 350)
- Right-clicking ON a node shows node actions (Clone, Bypass, Convert, etc.)
- Right-clicking on EMPTY canvas shows Add Node menu — different from node context menu
Your task: Generate a DETAILED reproduction guide (8-15 steps) to trigger this bug on main.
${allText}
## Available test actions
Each step must use one of these actions:
### Menu actions
- "openMenu" — clicks the Comfy hamburger menu (top-left C logo)
- "hoverMenuItem" — hovers a top-level menu item to open submenu (label required)
- "clickMenuItem" — clicks an item in the visible submenu (label required)
### Element actions (by visible text)
- "click" — clicks an element by visible text (text required)
- "rightClick" — right-clicks an element to open context menu (text required)
- "doubleClick" — double-clicks an element or coordinates (text or x,y)
- "fillDialog" — fills dialog input and presses Enter (text required)
- "pressKey" — presses a keyboard key (key required: Escape, Tab, Delete, Enter, etc.)
### Canvas actions (by coordinates — viewport is 1280x720)
- "clickCanvas" — click at coordinates (x, y required)
- "rightClickCanvas" — right-click at coordinates (x, y required)
- "doubleClick" — double-click at coordinates to open node search (x, y)
- "dragCanvas" — drag from one point to another (fromX, fromY, toX, toY)
- "scrollCanvas" — scroll wheel for zoom (x, y, deltaY: negative=zoom in, positive=zoom out)
### Utility
- "wait" — waits briefly (ms required, max 3000)
- "screenshot" — takes a screenshot (name required)
## Common ComfyUI interactions
- Right-click a node → context menu with Clone, Bypass, Remove, Colors, etc.
- Double-click empty canvas → opens node search dialog
- Ctrl+C / Ctrl+V → copy/paste selected nodes
- Delete key → remove selected node
- Ctrl+G → group selected nodes
- Drag from output slot to input slot → create connection
- Click a node to select it, Shift+click for multi-select
## Output format
Return a JSON object with exactly one key: "reproduce", containing:
{
"summary": "One sentence: what bug this issue reports",
"test_focus": "Specific behavior to reproduce",
"prerequisites": ["e.g. Load default workflow"],
"steps": [
{
"action": "clickCanvas",
"description": "Click on first node to select it",
"expected_before": "What should happen if the bug is present"
}
],
"visual_checks": ["Specific visual evidence of the bug to look for"]
}
## Rules
- Generate 8-15 DETAILED steps that actually trigger the reported bug.
- Follow the issue's reproduction steps PRECISELY — translate them into available actions.
- Use canvas coordinates for node interactions (nodes are typically in the center area 300-900 x 200-500).
- Take screenshots BEFORE and AFTER critical actions to capture the bug state.
- Do NOT just open a menu and screenshot — actually perform the full reproduction sequence.
- Do NOT include login steps.
- Output ONLY valid JSON, no markdown fences or explanation.`
}
function buildAnalysisPrompt(thread: PrThread): string {
const allText = [
`# PR: ${thread.title}`,
'',
'## Description',
thread.body,
'',
thread.issueComments.length > 0
? `## Issue Comments\n${thread.issueComments.join('\n\n---\n\n')}`
: '',
thread.reviewComments.length > 0
? `## Review Comments\n${thread.reviewComments.join('\n\n---\n\n')}`
: '',
thread.reviews.length > 0
? `## Reviews\n${thread.reviews.join('\n\n---\n\n')}`
: '',
'',
'## Diff (truncated)',
'```',
thread.diff.slice(0, 8000),
'```'
]
.filter(Boolean)
.join('\n')
return `You are a senior QA engineer analyzing a pull request for ComfyUI frontend (a Vue 3 + TypeScript web application for AI image generation workflows).
Your task: Generate TWO targeted QA test guides — one for BEFORE the PR (main branch) and one for AFTER (PR branch).
${allText}
## Available test actions
Each step must use one of these actions:
- "openMenu" — clicks the Comfy hamburger menu (top-left C logo)
- "hoverMenuItem" — hovers a top-level menu item to open submenu (label required)
- "clickMenuItem" — clicks an item in the visible submenu (label required)
- "fillDialog" — fills dialog input and presses Enter (text required)
- "pressKey" — presses a keyboard key (key required)
- "click" — clicks an element by visible text (text required)
- "wait" — waits briefly (ms required, max 3000)
- "screenshot" — takes a screenshot (name required)
## Output format
Return a JSON object with exactly two keys: "before" and "after", each containing:
{
"summary": "One sentence: what this PR changes",
"test_focus": "Specific behaviors to verify in this recording",
"prerequisites": ["e.g. Load default workflow"],
"steps": [
{
"action": "openMenu",
"description": "Open the main menu to check file options",
"expected_before": "Old behavior description (before key only)",
"expected_after": "New behavior description (after key only)"
}
],
"visual_checks": ["Specific visual elements to look for"]
}
## Rules
- BEFORE guide: 2-4 steps, under 15 seconds. Show OLD/missing behavior.
- AFTER guide: 3-6 steps, under 30 seconds. Prove the fix/feature works.
- Focus on the SPECIFIC behavior changed by this PR, not generic testing.
- Use information from PR description, screenshots, and comments to understand intended behavior.
- Include at least one screenshot step in each guide.
- Do NOT include login steps.
- Menu pattern: openMenu -> hoverMenuItem -> clickMenuItem or screenshot.
- Output ONLY valid JSON, no markdown fences or explanation.`
}
async function analyzeWithGemini(
thread: PrThread,
media: Array<{ path: string; mimeType: string }>,
model: string,
apiKey: string
): Promise<{ before: QaGuide; after: QaGuide }> {
const genAI = new GoogleGenerativeAI(apiKey)
const geminiModel = genAI.getGenerativeModel({ model })
const prompt = buildAnalysisPrompt(thread)
const parts: Array<
{ text: string } | { inlineData: { mimeType: string; data: string } }
> = [{ text: prompt }]
// Add media as inline data
for (const item of media) {
try {
const buffer = readFileSync(item.path)
parts.push({
inlineData: {
mimeType: item.mimeType,
data: buffer.toString('base64')
}
})
} catch (err) {
console.warn(
`Failed to read media ${item.path}: ${(err as Error).message}`
)
}
}
console.warn(
`Sending to ${model}: ${prompt.length} chars text, ${media.length} media files`
)
const result = await geminiModel.generateContent({
contents: [{ role: 'user', parts }],
generationConfig: {
temperature: 0.2,
maxOutputTokens: 8192,
responseMimeType: 'application/json'
}
})
let text = result.response.text()
// Strip markdown fences if present
text = text
.replace(/^```(?:json)?\n?/gm, '')
.replace(/```$/gm, '')
.trim()
console.warn('Gemini response received')
console.warn('Raw response (first 500 chars):', text.slice(0, 500))
const parsed = JSON.parse(text)
// Handle different response shapes from Gemini
let before: QaGuide
let after: QaGuide
if (Array.isArray(parsed) && parsed.length >= 2) {
// Array format: [before, after]
before = parsed[0]
after = parsed[1]
} else if (parsed.before && parsed.after) {
// Object format: { before, after }
before = parsed.before
after = parsed.after
} else {
// Try nested wrapper keys
const inner = parsed.qa_guide ?? parsed.guides ?? parsed
if (inner.before && inner.after) {
before = inner.before
after = inner.after
} else {
console.warn(
'Full response:',
JSON.stringify(parsed, null, 2).slice(0, 2000)
)
throw new Error(
`Unexpected response shape. Got keys: ${Object.keys(parsed).join(', ')}`
)
}
}
return { before, after }
}
async function analyzeIssueWithGemini(
issue: IssueThread,
media: Array<{ path: string; mimeType: string }>,
model: string,
apiKey: string
): Promise<QaGuide> {
const genAI = new GoogleGenerativeAI(apiKey)
const geminiModel = genAI.getGenerativeModel({ model })
const prompt = buildIssueAnalysisPrompt(issue)
const parts: Array<
{ text: string } | { inlineData: { mimeType: string; data: string } }
> = [{ text: prompt }]
for (const item of media) {
try {
const buffer = readFileSync(item.path)
parts.push({
inlineData: {
mimeType: item.mimeType,
data: buffer.toString('base64')
}
})
} catch (err) {
console.warn(
`Failed to read media ${item.path}: ${(err as Error).message}`
)
}
}
console.warn(
`Sending to ${model}: ${prompt.length} chars text, ${media.length} media files`
)
const result = await geminiModel.generateContent({
contents: [{ role: 'user', parts }],
generationConfig: {
temperature: 0.2,
maxOutputTokens: 8192,
responseMimeType: 'application/json'
}
})
let text = result.response.text()
text = text
.replace(/^```(?:json)?\n?/gm, '')
.replace(/```$/gm, '')
.trim()
console.warn('Gemini response received')
console.warn('Raw response (first 500 chars):', text.slice(0, 500))
const parsed = JSON.parse(text)
const guide: QaGuide =
parsed.reproduce ?? parsed.qa_guide?.reproduce ?? parsed
return guide
}
// ── Main ──
async function main() {
const opts = parseArgs()
mkdirSync(opts.outputDir, { recursive: true })
if (opts.type === 'issue') {
await analyzeIssue(opts)
} else {
await analyzePr(opts)
}
}
async function analyzeIssue(opts: Options) {
const issue = fetchIssueThread(opts.prNumber, opts.repo)
const allText = [issue.body, ...issue.comments].join('\n')
const mediaUrls = extractMediaUrls(allText)
console.warn(`Found ${mediaUrls.length} media URLs`)
const media = await downloadMedia(
mediaUrls,
opts.outputDir,
opts.mediaBudgetBytes,
opts.maxVideoBytes
)
const guide = await analyzeIssueWithGemini(
issue,
media,
opts.model,
opts.apiKey
)
const beforePath = resolve(opts.outputDir, 'qa-guide-before.json')
writeFileSync(beforePath, JSON.stringify(guide, null, 2))
console.warn(`Wrote QA guide:`)
console.warn(` Reproduce: ${beforePath}`)
}
async function analyzePr(opts: Options) {
const thread = fetchPrThread(opts.prNumber, opts.repo)
const allText = [
thread.body,
...thread.issueComments,
...thread.reviewComments,
...thread.reviews
].join('\n')
const mediaUrls = extractMediaUrls(allText)
console.warn(`Found ${mediaUrls.length} media URLs`)
const media = await downloadMedia(
mediaUrls,
opts.outputDir,
opts.mediaBudgetBytes,
opts.maxVideoBytes
)
const guides = await analyzeWithGemini(thread, media, opts.model, opts.apiKey)
const beforePath = resolve(opts.outputDir, 'qa-guide-before.json')
const afterPath = resolve(opts.outputDir, 'qa-guide-after.json')
writeFileSync(beforePath, JSON.stringify(guides.before, null, 2))
writeFileSync(afterPath, JSON.stringify(guides.after, null, 2))
console.warn(`Wrote QA guides:`)
console.warn(` Before: ${beforePath}`)
console.warn(` After: ${afterPath}`)
}
function isExecutedAsScript(metaUrl: string): boolean {
const modulePath = fileURLToPath(metaUrl)
const scriptPath = process.argv[1] ? resolve(process.argv[1]) : ''
return modulePath === scriptPath
}
if (isExecutedAsScript(import.meta.url)) {
main().catch((err) => {
console.error('PR analysis failed:', err)
process.exit(1)
})
}

View File

@@ -0,0 +1,413 @@
#!/usr/bin/env bash
# Deploy QA report to Cloudflare Pages.
# Expected env vars: CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, RAW_BRANCH,
# BEFORE_SHA, AFTER_SHA, TARGET_NUM, TARGET_TYPE, REPO, RUN_ID
# Writes outputs to GITHUB_OUTPUT: badge_status, url
set -euo pipefail
npm install -g wrangler@4.74.0 >/dev/null 2>&1
DEPLOY_DIR=$(mktemp -d)
mkdir -p "$DEPLOY_DIR"
for os in Linux macOS Windows; do
DIR="qa-artifacts/qa-report-${os}-${RUN_ID}"
for prefix in qa qa-before; do
VID="${DIR}/${prefix}-session.mp4"
if [ -f "$VID" ]; then
DEST="$DEPLOY_DIR/${prefix}-${os}.mp4"
cp "$VID" "$DEST"
echo "Found ${prefix} ${os} video ($(du -h "$VID" | cut -f1))"
fi
done
# Copy multi-pass session videos (qa-session-1, qa-session-2, etc.)
for numbered in "$DIR"/qa-session-[0-9].mp4; do
[ -f "$numbered" ] || continue
NUM=$(basename "$numbered" | sed 's/qa-session-\([0-9]\).mp4/\1/')
DEST="$DEPLOY_DIR/qa-${os}-pass${NUM}.mp4"
cp "$numbered" "$DEST"
echo "Found pass ${NUM} ${os} video ($(du -h "$numbered" | cut -f1))"
done
# Generate GIF thumbnail from after video (or first pass)
THUMB_SRC="$DEPLOY_DIR/qa-${os}.mp4"
[ ! -f "$THUMB_SRC" ] && THUMB_SRC="$DEPLOY_DIR/qa-${os}-pass1.mp4"
if [ -f "$THUMB_SRC" ]; then
ffmpeg -y -ss 10 -i "$THUMB_SRC" -t 8 \
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|| echo "GIF generation failed for ${os} (non-fatal)"
fi
done
# Build video cards and report sections
CARDS=""
# shellcheck disable=SC2034 # accessed via eval
ICONS_Linux="&#x1F427;" ICONS_macOS="&#x1F34E;" ICONS_Windows="&#x1FA9F;"
CARD_COUNT=0
DL_ICON="<svg width=14 height=14 viewBox='0 0 24 24' fill=none stroke=currentColor stroke-width=2><path d='M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4'/><polyline points='7 10 12 15 17 10'/><line x1=12 y1=15 x2=12 y2=3'/></svg>"
for os in Linux macOS Windows; do
eval "ICON=\$ICONS_${os}"
OS_LOWER=$(echo "$os" | tr '[:upper:]' '[:lower:]')
HAS_BEFORE=$([ -f "$DEPLOY_DIR/qa-before-${os}.mp4" ] && echo 1 || echo 0)
HAS_AFTER=$( { [ -f "$DEPLOY_DIR/qa-${os}.mp4" ] || [ -f "$DEPLOY_DIR/qa-${os}-pass1.mp4" ]; } && echo 1 || echo 0)
[ "$HAS_AFTER" = "0" ] && continue
# Collect all reports for this platform (single + multi-pass)
REPORT_FILES=""
REPORT_LINK=""
REPORT_HTML=""
for rpt in "video-reviews/${OS_LOWER}-qa-video-report.md" "video-reviews/${OS_LOWER}-pass"*-qa-video-report.md; do
[ -f "$rpt" ] && REPORT_FILES="${REPORT_FILES} ${rpt}"
done
if [ -n "$REPORT_FILES" ]; then
# Concatenate all reports into one combined report file
COMBINED_MD=""
for rpt in $REPORT_FILES; do
cp "$rpt" "$DEPLOY_DIR/$(basename "$rpt")"
RPT_MD=$(sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g' "$rpt")
[ -n "$COMBINED_MD" ] && COMBINED_MD="${COMBINED_MD}&#10;&#10;---&#10;&#10;"
COMBINED_MD="${COMBINED_MD}${RPT_MD}"
done
FIRST_REPORT=$(echo "$REPORT_FILES" | awk '{print $1}')
FIRST_BASENAME=$(basename "$FIRST_REPORT")
REPORT_LINK="<a class=dl href=${FIRST_BASENAME}><svg width=14 height=14 viewBox='0 0 24 24' fill=none stroke=currentColor stroke-width=2><path d='M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8z'/><polyline points='14 2 14 8 20 8'/><line x1=16 y1=13 x2=8 y2=13/><line x1=16 y1=17 x2=8 y2=17'/></svg>Report</a>"
REPORT_HTML="<details class=report open><summary><svg width=14 height=14 viewBox='0 0 24 24' fill=none stroke=currentColor stroke-width=2><circle cx=12 cy=12 r=10/><line x1=12 y1=16 x2=12 y2=12/><line x1=12 y1=8 x2=12.01 y2=8'/></svg> AI Comparative Review</summary><div class=report-body data-md>${COMBINED_MD}</div></details>"
fi
if [ "$HAS_BEFORE" = "1" ]; then
CARDS="${CARDS}<div class='card reveal' style='--i:${CARD_COUNT}'><div class=card-header><span class=platform><span class=icon>${ICON}</span>${os}</span><span class=links>${REPORT_LINK}</span></div><div class=comparison><div class=comp-panel><div class=comp-label>Before <span class=comp-tag>main</span></div><div class=video-wrap><video controls preload=auto><source src=qa-before-${os}.mp4 type=video/mp4></video></div><div class=comp-dl><a class=dl href=qa-before-${os}.mp4 download>${DL_ICON}Before</a></div></div><div class=comp-panel><div class=comp-label>After <span class=comp-tag>PR</span></div><div class=video-wrap><video controls preload=auto><source src=qa-${os}.mp4 type=video/mp4></video></div><div class=comp-dl><a class=dl href=qa-${os}.mp4 download>${DL_ICON}After</a></div></div></div>${REPORT_HTML}</div>"
elif [ -f "$DEPLOY_DIR/qa-${os}.mp4" ]; then
CARDS="${CARDS}<div class='card reveal' style='--i:${CARD_COUNT}'><div class=video-wrap><video controls preload=auto><source src=qa-${os}.mp4 type=video/mp4></video></div><div class=card-body><span class=platform><span class=icon>${ICON}</span>${os}</span><span class=links><a class=dl href=qa-${os}.mp4 download>${DL_ICON}Download</a>${REPORT_LINK}</span></div>${REPORT_HTML}</div>"
else
PASS_VIDEOS=""
for pass_vid in "$DEPLOY_DIR/qa-${os}-pass"[0-9].mp4; do
[ -f "$pass_vid" ] || continue
PASS_NUM=$(basename "$pass_vid" | sed "s/qa-${os}-pass\([0-9]\).mp4/\1/")
PASS_VIDEOS="${PASS_VIDEOS}<div class=comp-panel><div class=comp-label>Pass ${PASS_NUM}</div><div class=video-wrap><video controls preload=auto><source src=qa-${os}-pass${PASS_NUM}.mp4 type=video/mp4></video></div><div class=comp-dl><a class=dl href=qa-${os}-pass${PASS_NUM}.mp4 download>${DL_ICON}Pass ${PASS_NUM}</a></div></div>"
done
CARDS="${CARDS}<div class='card reveal' style='--i:${CARD_COUNT}'><div class=card-header><span class=platform><span class=icon>${ICON}</span>${os}</span><span class=links>${REPORT_LINK}</span></div><div class=comparison>${PASS_VIDEOS}</div>${REPORT_HTML}</div>"
fi
CARD_COUNT=$((CARD_COUNT + 1))
done
# Build commit info and target link for the report header
COMMIT_HTML=""
REPO_URL="https://github.com/${REPO}"
if [ -n "${TARGET_NUM:-}" ]; then
if [ "$TARGET_TYPE" = "issue" ]; then
COMMIT_HTML="<a href=${REPO_URL}/issues/${TARGET_NUM} class=sha title='Issue'>Issue #${TARGET_NUM}</a>"
else
COMMIT_HTML="<a href=${REPO_URL}/pull/${TARGET_NUM} class=sha title='Pull Request'>PR #${TARGET_NUM}</a>"
fi
fi
if [ -n "${BEFORE_SHA:-}" ]; then
SHORT_BEFORE="${BEFORE_SHA:0:7}"
COMMIT_HTML="${COMMIT_HTML:+${COMMIT_HTML} &middot; }<a href=${REPO_URL}/commit/${BEFORE_SHA} class=sha title='main branch'>main @ ${SHORT_BEFORE}</a>"
fi
if [ -n "${AFTER_SHA:-}" ]; then
SHORT_AFTER="${AFTER_SHA:0:7}"
AFTER_LABEL="PR"
[ -n "${TARGET_NUM:-}" ] && AFTER_LABEL="#${TARGET_NUM}"
COMMIT_HTML="${COMMIT_HTML:+${COMMIT_HTML} &middot; }<a href=${REPO_URL}/commit/${AFTER_SHA} class=sha title='PR head commit'>${AFTER_LABEL} @ ${SHORT_AFTER}</a>"
fi
if [ -n "${PIPELINE_SHA:-}" ]; then
SHORT_PIPE="${PIPELINE_SHA:0:7}"
COMMIT_HTML="${COMMIT_HTML:+${COMMIT_HTML} &middot; }<a href=${REPO_URL}/commit/${PIPELINE_SHA} class=sha title='QA pipeline version'>QA @ ${SHORT_PIPE}</a>"
fi
[ -n "$COMMIT_HTML" ] && COMMIT_HTML=" &middot; ${COMMIT_HTML}"
RUN_LINK=""
if [ -n "${RUN_URL:-}" ]; then
RUN_LINK=" &middot; <a href=\"${RUN_URL}\" class=sha title=\"GitHub Actions run\">CI Job</a>"
fi
# Timing info
DEPLOY_TIME=$(date -u '+%Y-%m-%d %H:%M UTC')
TIMING_HTML=""
if [ -n "${RUN_START_TIME:-}" ]; then
TIMING_HTML=" &middot; <span class=sha title='Pipeline timing'>${RUN_START_TIME} &rarr; ${DEPLOY_TIME}</span>"
fi
# Generate index.html from template
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
TEMPLATE="$SCRIPT_DIR/qa-report-template.html"
# Write dynamic content to temp files for safe substitution
# Cloudflare Pages _headers file — enable range requests for video seeking
cat > "$DEPLOY_DIR/_headers" <<'HEADERSEOF'
/*.mp4
Accept-Ranges: bytes
Cache-Control: public, max-age=86400
HEADERSEOF
# Build purpose description from pr-context.txt
PURPOSE_HTML=""
if [ -f pr-context.txt ]; then
# Extract title line and first paragraph of description
PR_TITLE=$(grep -m1 '^Title:' pr-context.txt 2>/dev/null | sed 's/^Title: //' || true)
if [ "$TARGET_TYPE" = "issue" ]; then
PURPOSE_LABEL="Issue #${TARGET_NUM}"
PURPOSE_VERB="reports"
else
PURPOSE_LABEL="PR #${TARGET_NUM}"
PURPOSE_VERB="aims to"
fi
# Get first ~300 chars of description body (after "Description:" line)
PR_DESC=$(sed -n '/^Description:/,/^###/p' pr-context.txt 2>/dev/null | grep -v '^Description:\|^###' | head -5 | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g' | tr '\n' ' ' | head -c 400 || true)
[ -z "$PR_DESC" ] && PR_DESC=$(sed -n '3,8p' pr-context.txt 2>/dev/null | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g' | tr '\n' ' ' | head -c 400 || true)
# Build requirements from QA guide JSON
REQS_HTML=""
QA_GUIDE=$(ls qa-guides/qa-guide-*.json 2>/dev/null | head -1 || true)
if [ -f "$QA_GUIDE" ]; then
PREREQS=$(python3 -c "
import json, sys, html
try:
g = json.load(open(sys.argv[1]))
prereqs = g.get('prerequisites', [])
steps = g.get('steps', [])
focus = g.get('test_focus', '')
parts = []
if focus:
parts.append('<strong>Test focus:</strong> ' + html.escape(focus))
if prereqs:
parts.append('<strong>Prerequisites:</strong> ' + ', '.join(html.escape(p) for p in prereqs))
if steps:
parts.append('<strong>Steps:</strong> ' + ' → '.join(html.escape(s.get('description', str(s))) for s in steps[:6]))
if len(steps) > 6:
parts[-1] += ' → ...'
print('<br>'.join(parts))
except: pass
" "$QA_GUIDE" 2>/dev/null)
[ -n "$PREREQS" ] && REQS_HTML="<div class=purpose-reqs>${PREREQS}</div>"
fi
PURPOSE_HTML="<div class=purpose><div class=purpose-label>${PURPOSE_LABEL} ${PURPOSE_VERB}</div><strong>${PR_TITLE}</strong><br>${PR_DESC}${REQS_HTML}</div>"
fi
echo -n "$COMMIT_HTML" > "$DEPLOY_DIR/.commit_html"
echo -n "$CARDS" > "$DEPLOY_DIR/.cards_html"
echo -n "$RUN_LINK" > "$DEPLOY_DIR/.run_link"
# Badge HTML with copy button (placeholder URL filled after deploy)
echo -n '<div class="badge-bar"><img src="badge.svg" alt="QA Badge" class="badge-img"/><button class="copy-badge" title="Copy badge markdown" onclick="copyBadge()"><svg width=14 height=14 viewBox="0 0 24 24" fill=none stroke=currentColor stroke-width=2><rect x=9 y=9 width=13 height=13 rx=2/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg></button></div>' > "$DEPLOY_DIR/.badge_html"
echo -n "${TIMING_HTML:-}" > "$DEPLOY_DIR/.timing_html"
echo -n "$PURPOSE_HTML" > "$DEPLOY_DIR/.purpose_html"
python3 -c "
import sys, pathlib
d = pathlib.Path(sys.argv[1])
t = pathlib.Path(sys.argv[2]).read_text()
t = t.replace('{{COMMIT_HTML}}', (d / '.commit_html').read_text())
t = t.replace('{{CARDS}}', (d / '.cards_html').read_text())
t = t.replace('{{RUN_LINK}}', (d / '.run_link').read_text())
t = t.replace('{{BADGE_HTML}}', (d / '.badge_html').read_text())
t = t.replace('{{TIMING_HTML}}', (d / '.timing_html').read_text())
t = t.replace('{{PURPOSE_HTML}}', (d / '.purpose_html').read_text())
sys.stdout.write(t)
" "$DEPLOY_DIR" "$TEMPLATE" > "$DEPLOY_DIR/index.html"
rm -f "$DEPLOY_DIR/.commit_html" "$DEPLOY_DIR/.cards_html" "$DEPLOY_DIR/.run_link" "$DEPLOY_DIR/.badge_html" "$DEPLOY_DIR/.timing_html" "$DEPLOY_DIR/.purpose_html"
cat > "$DEPLOY_DIR/404.html" <<'ERROREOF'
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><title>404</title>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600&display=swap" rel=stylesheet>
<style>:root{--bg:oklch(8% 0.02 265);--fg:oklch(45% 0.01 265);--err:oklch(62% 0.22 25)}*{margin:0;padding:0;box-sizing:border-box}body{background:var(--bg);color:var(--fg);font-family:'Inter',system-ui,sans-serif;display:flex;align-items:center;justify-content:center;min-height:100vh}div{text-align:center}h1{color:var(--err);font-size:clamp(3rem,8vw,5rem);font-weight:700;letter-spacing:-.04em;margin-bottom:.5rem}p{font-size:1rem;max-width:32ch;line-height:1.5}</style>
</head><body><div><h1>404</h1><p>File not found. The QA recording may have failed or been cancelled.</p></div></body></html>
ERROREOF
# Copy research log to deploy dir if it exists
for rlog in qa-artifacts/*/research/research-log.json qa-artifacts/*/*/research/research-log.json qa-artifacts/before/*/research/research-log.json; do
if [ -f "$rlog" ]; then
cp "$rlog" "$DEPLOY_DIR/research-log.json"
echo "Found research log: $rlog"
break
fi
done
# Copy generated test code to deploy dir
for tfile in qa-artifacts/*/research/reproduce.spec.ts qa-artifacts/*/*/research/reproduce.spec.ts qa-artifacts/before/*/research/reproduce.spec.ts; do
if [ -f "$tfile" ]; then
cp "$tfile" "$DEPLOY_DIR/reproduce.spec.ts"
echo "Found test code: $tfile"
break
fi
done
# Copy video script if available
for vsfile in qa-artifacts/*/video-script.spec.ts qa-artifacts/*/*/video-script.spec.ts qa-artifacts/before/*/video-script.spec.ts; do
if [ -f "$vsfile" ]; then
cp "$vsfile" "$DEPLOY_DIR/video-script.spec.ts"
echo "Found video script: $vsfile"
break
fi
done
# Generate badge SVGs into deploy dir
# Priority: research-log.json verdict (a11y-verified) > video review verdict (AI interpretation)
REPRO_COUNT=0 INCONC_COUNT=0 NOT_REPRO_COUNT=0 TOTAL_REPORTS=0
# Try research log first (ground truth from a11y assertions)
RESEARCH_VERDICT=""
REPRO_METHOD=""
if [ -f "$DEPLOY_DIR/research-log.json" ]; then
RESEARCH_VERDICT=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('verdict',''))" "$DEPLOY_DIR/research-log.json" 2>/dev/null || true)
REPRO_METHOD=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('reproducedBy','none'))" "$DEPLOY_DIR/research-log.json" 2>/dev/null || true)
echo "Research verdict (a11y-verified): ${RESEARCH_VERDICT:-none} (by: ${REPRO_METHOD:-none})"
if [ -n "$RESEARCH_VERDICT" ]; then
TOTAL_REPORTS=1
case "$RESEARCH_VERDICT" in
REPRODUCED) REPRO_COUNT=1 ;;
NOT_REPRODUCIBLE) NOT_REPRO_COUNT=1 ;;
INCONCLUSIVE) INCONC_COUNT=1 ;;
esac
fi
fi
# Check video review verdicts (always, not just as fallback)
VIDEO_REPRODUCED=false
if [ -d video-reviews ]; then
for rpt in video-reviews/*-qa-video-report.md; do
[ -f "$rpt" ] || continue
VERDICT_JSON=$(grep -oP '"verdict":\s*"[A-Z_]+' "$rpt" 2>/dev/null | tail -1 | grep -oP '[A-Z_]+$' || true)
if [ -n "$VERDICT_JSON" ]; then
echo "Video review verdict: $VERDICT_JSON ($(basename "$rpt"))"
[ "$VERDICT_JSON" = "REPRODUCED" ] && VIDEO_REPRODUCED=true
# Only count video as separate report if no research log
if [ -z "$RESEARCH_VERDICT" ]; then
TOTAL_REPORTS=$((TOTAL_REPORTS + 1))
case "$VERDICT_JSON" in
REPRODUCED) REPRO_COUNT=$((REPRO_COUNT + 1)) ;;
NOT_REPRODUCIBLE) NOT_REPRO_COUNT=$((NOT_REPRO_COUNT + 1)) ;;
INCONCLUSIVE) INCONC_COUNT=$((INCONC_COUNT + 1)) ;;
esac
fi
elif [ -z "$RESEARCH_VERDICT" ]; then
TOTAL_REPORTS=$((TOTAL_REPORTS + 1))
# Fallback: grep Summary section (for older reports without ## Verdict)
SUMM=$(sed -n '/^## Summary/,/^## /p' "$rpt" 2>/dev/null | head -15)
if echo "$SUMM" | grep -iq 'INCONCLUSIVE'; then
INCONC_COUNT=$((INCONC_COUNT + 1))
elif echo "$SUMM" | grep -iq 'not reproduced\|could not reproduce\|could not be confirmed\|unable to reproduce\|fails\? to reproduce\|fails\? to perform\|was NOT\|NOT visible\|not observed\|fail.* to demonstrate\|does not demonstrate\|steps were not performed\|never.*tested\|never.*accessed\|not.* confirmed'; then
NOT_REPRO_COUNT=$((NOT_REPRO_COUNT + 1))
elif echo "$SUMM" | grep -iq 'reproduc\|confirm'; then
REPRO_COUNT=$((REPRO_COUNT + 1))
VIDEO_REPRODUCED=true
fi
fi
done
fi
# Upgrade reproduction method to "both" when E2E and video agree
if [ "$REPRO_METHOD" = "e2e_test" ] && [ "$VIDEO_REPRODUCED" = "true" ]; then
REPRO_METHOD="both"
echo "Upgraded reproducedBy to 'both' (E2E + video review agree)"
elif [ -z "$RESEARCH_VERDICT" ] && [ "$VIDEO_REPRODUCED" = "true" ]; then
REPRO_METHOD="video"
fi
FAIL_COUNT=$((TOTAL_REPORTS - REPRO_COUNT - NOT_REPRO_COUNT))
[ "$FAIL_COUNT" -lt 0 ] && FAIL_COUNT=0
echo "DEBUG verdict: repro=${REPRO_COUNT} not_repro=${NOT_REPRO_COUNT} inconc=${INCONC_COUNT} fail=${FAIL_COUNT} total=${TOTAL_REPORTS}"
# Warn on verdict mismatch between E2E and video review
if [ -n "$RESEARCH_VERDICT" ]; then
VIDEO_VERDICT=$(grep -oP '"verdict":\s*"[A-Z_]+' video-reviews/*-qa-video-report.md 2>/dev/null | tail -1 | grep -oP '[A-Z_]+$' || true)
if [ -n "$VIDEO_VERDICT" ] && [ "$RESEARCH_VERDICT" != "$VIDEO_VERDICT" ]; then
echo "⚠ Verdict mismatch: E2E=$RESEARCH_VERDICT vs Video=$VIDEO_VERDICT (E2E takes priority)"
fi
fi
echo "Verdict: ${REPRO_COUNT}${NOT_REPRO_COUNT}${FAIL_COUNT}⚠ / ${TOTAL_REPORTS}"
# Badge text:
# Single pass: "REPRODUCED" / "NOT REPRODUCIBLE" / "INCONCLUSIVE"
# Multi pass: "2✓ 0✗ 1⚠ / 3" with color based on dominant result
REPRO_RESULT="" REPRO_COLOR="#9f9f9f"
if [ "$TOTAL_REPORTS" -le 1 ]; then
# Single report — simple label
if [ "$REPRO_COUNT" -gt 0 ]; then
REPRO_RESULT="REPRODUCED" REPRO_COLOR="#2196f3"
elif [ "$NOT_REPRO_COUNT" -gt 0 ]; then
REPRO_RESULT="NOT REPRODUCIBLE" REPRO_COLOR="#9f9f9f"
elif [ "$FAIL_COUNT" -gt 0 ]; then
REPRO_RESULT="INCONCLUSIVE" REPRO_COLOR="#9f9f9f"
fi
else
# Multi pass — show breakdown: X✓ Y✗ Z⚠ / N
PARTS=""
[ "$REPRO_COUNT" -gt 0 ] && PARTS="${REPRO_COUNT}"
[ "$NOT_REPRO_COUNT" -gt 0 ] && PARTS="${PARTS:+${PARTS} }${NOT_REPRO_COUNT}"
[ "$FAIL_COUNT" -gt 0 ] && PARTS="${PARTS:+${PARTS} }${FAIL_COUNT}"
REPRO_RESULT="${PARTS} / ${TOTAL_REPORTS}"
# Color based on best outcome
if [ "$REPRO_COUNT" -gt 0 ]; then
REPRO_COLOR="#2196f3"
elif [ "$NOT_REPRO_COUNT" -gt 0 ]; then
REPRO_COLOR="#9f9f9f"
fi
fi
# Badge label: #NUM QA0327 (with today's date)
QA_DATE=$(date -u '+%m%d')
BADGE_LABEL="QA${QA_DATE}"
[ -n "${TARGET_NUM:-}" ] && BADGE_LABEL="#${TARGET_NUM} QA${QA_DATE}"
# For PRs, also extract fix quality from Overall Risk section
FIX_RESULT="" FIX_COLOR="#4c1"
if [ "$TARGET_TYPE" != "issue" ]; then
# Try structured JSON risk first
ALL_RISKS=$(grep -ohP '"risk":\s*"[a-z]+' video-reviews/*.md 2>/dev/null | grep -oP '[a-z]+$' || true)
if [ -n "$ALL_RISKS" ]; then
# Use worst risk across all reports
if echo "$ALL_RISKS" | grep -q 'high'; then
FIX_RESULT="MAJOR ISSUES" FIX_COLOR="#e05d44"
elif echo "$ALL_RISKS" | grep -q 'medium'; then
FIX_RESULT="MINOR ISSUES" FIX_COLOR="#dfb317"
elif echo "$ALL_RISKS" | grep -q 'low'; then
FIX_RESULT="APPROVED" FIX_COLOR="#4c1"
fi
else
# Fallback: grep Overall Risk section
RISK_TEXT=""
if [ -d video-reviews ]; then
RISK_TEXT=$(sed -n '/^## Overall Risk/,/^## /p' video-reviews/*.md 2>/dev/null | sed 's/\*//g' | head -20 || true)
fi
RISK_FIRST=$(echo "$RISK_TEXT" | grep -oiP '^\s*(high|medium|moderate|low|minimal|critical)' | head -1 | tr '[:upper:]' '[:lower:]' || true)
if [ -n "$RISK_FIRST" ]; then
case "$RISK_FIRST" in
*low*|*minimal*) FIX_RESULT="APPROVED" FIX_COLOR="#4c1" ;;
*medium*|*moderate*) FIX_RESULT="MINOR ISSUES" FIX_COLOR="#dfb317" ;;
*high*|*critical*) FIX_RESULT="MAJOR ISSUES" FIX_COLOR="#e05d44" ;;
esac
elif echo "$RISK_TEXT" | grep -iq 'no.*risk\|approved\|looks good'; then
FIX_RESULT="APPROVED" FIX_COLOR="#4c1"
fi
fi
fi
# Always use vertical box badge
/tmp/gen-badge-box.sh "$DEPLOY_DIR/badge.svg" "$BADGE_LABEL" \
"$REPRO_COUNT" "$NOT_REPRO_COUNT" "$FAIL_COUNT" "$TOTAL_REPORTS" \
"$FIX_RESULT" "$FIX_COLOR" "$REPRO_METHOD"
BADGE_STATUS="${REPRO_RESULT:-UNKNOWN}${FIX_RESULT:+ | Fix: ${FIX_RESULT}}"
echo "badge_status=${BADGE_STATUS:-FINISHED}" >> "$GITHUB_OUTPUT"
# Remove files exceeding Cloudflare Pages 25MB limit to prevent silent deploy failures
MAX_SIZE=$((25 * 1024 * 1024))
find "$DEPLOY_DIR" -type f -size +${MAX_SIZE}c | while read -r big_file; do
SIZE_MB=$(( $(stat -c%s "$big_file") / 1024 / 1024 ))
echo "Removing oversized file: $(basename "$big_file") (${SIZE_MB}MB > 25MB limit)"
rm "$big_file"
done
BRANCH=$(echo "$RAW_BRANCH" | sed 's/[^a-zA-Z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | cut -c1-28)
DEPLOY_OUTPUT=$(wrangler pages deploy "$DEPLOY_DIR" \
--project-name="comfy-qa" \
--branch="$BRANCH" 2>&1) || true
echo "$DEPLOY_OUTPUT" | tail -5
URL=$(echo "$DEPLOY_OUTPUT" | grep -oE 'https://[a-zA-Z0-9.-]+\.pages\.dev\S*' | head -1 || true)
FALLBACK_URL="https://${BRANCH}.comfy-qa.pages.dev"
echo "url=${URL:-$FALLBACK_URL}" >> "$GITHUB_OUTPUT"
echo "Deployed to: ${URL:-$FALLBACK_URL}"

View File

@@ -0,0 +1,208 @@
#!/usr/bin/env tsx
/**
* Generates a Playwright regression test (.spec.ts) from a QA report + PR diff.
* Uses Gemini to produce a test that asserts UIUX behavior verified during QA.
*
* Usage:
* pnpm exec tsx scripts/qa-generate-test.ts \
* --qa-report <path> QA video review report (markdown)
* --pr-diff <path> PR diff file
* --output <path> Output .spec.ts file path
* --model <name> Gemini model (default: gemini-3-flash-preview)
*/
import { readFile, writeFile } from 'node:fs/promises'
import { basename, resolve } from 'node:path'
import { GoogleGenerativeAI } from '@google/generative-ai'
interface CliOptions {
qaReport: string
prDiff: string
output: string
model: string
}
const DEFAULTS: CliOptions = {
qaReport: '',
prDiff: '',
output: '',
model: 'gemini-3-flash-preview'
}
// ── Fixture API reference for the prompt ────────────────────────────
const FIXTURE_API = `
## ComfyUI Playwright Test Fixture API
Import pattern:
\`\`\`typescript
import { expect } from '@playwright/test'
import { comfyPageFixture as test } from '../fixtures/ComfyPage'
\`\`\`
### Available helpers on \`comfyPage\`:
- \`comfyPage.page\` — raw Playwright Page
- \`comfyPage.menu.topbar\` — Topbar helper:
- \`.getTabNames(): Promise<string[]>\` — get all open tab names
- \`.getActiveTabName(): Promise<string>\` — get active tab name
- \`.saveWorkflow(name)\` — Save via File > Save dialog
- \`.saveWorkflowAs(name)\` — Save via File > Save As dialog
- \`.exportWorkflow(name)\` — Export via File > Export dialog
- \`.triggerTopbarCommand(path: string[])\` — e.g. ['File', 'Save As']
- \`.getWorkflowTab(name)\` — get a tab locator by name
- \`.closeWorkflowTab(name)\` — close a tab
- \`.openTopbarMenu()\` — open the hamburger menu
- \`.openSubmenu(label)\` — hover to open a submenu
- \`comfyPage.menu.workflowsTab\` — Workflows sidebar:
- \`.open()\` / \`.close()\` — toggle sidebar
- \`.getTopLevelSavedWorkflowNames()\` — list saved workflows
- \`.getPersistedItem(name)\` — get a workflow item locator
- \`comfyPage.workflow\` — WorkflowHelper:
- \`.loadWorkflow(name)\` — load from browser_tests/assets/{name}.json
- \`.setupWorkflowsDirectory(structure)\` — setup test directory
- \`.deleteWorkflow(name)\` — delete a workflow
- \`.isCurrentWorkflowModified(): Promise<boolean>\` — check dirty state
- \`.getUndoQueueSize()\` / \`.getRedoQueueSize()\`
- \`comfyPage.settings.setSetting(key, value)\` — change settings
- \`comfyPage.keyboard\` — KeyboardHelper:
- \`.undo()\` / \`.redo()\` / \`.bypass()\`
- \`comfyPage.nodeOps\` — NodeOperationsHelper
- \`comfyPage.canvas\` — CanvasHelper
- \`comfyPage.contextMenu\` — ContextMenu
- \`comfyPage.toast\` — ToastHelper
- \`comfyPage.confirmDialog\` — confirmation dialog
- \`comfyPage.nextFrame()\` — wait for Vue re-render
### Test patterns:
- Use \`test.describe('Name', { tag: '@ui' }, () => { ... })\` for UI tests
- Use \`test.beforeEach\` to set up common state (settings, workflow dir)
- Use \`expect(locator).toHaveScreenshot('name.png')\` for visual assertions
- Use \`expect(locator).toBeVisible()\` / \`.toHaveText()\` for behavioral assertions
- Use \`comfyPage.workflow.setupWorkflowsDirectory({})\` to ensure clean state
`
// ── Prompt builder ──────────────────────────────────────────────────
function buildPrompt(qaReport: string, prDiff: string): string {
return `You are a Playwright test generator for the ComfyUI frontend.
Your task: Generate a single .spec.ts regression test file that asserts the UIUX behavior
described in the QA report below. The test must:
1. Use the ComfyUI Playwright fixture API (documented below)
2. Test UIUX behavior ONLY — element visibility, tab names, dialog states, workflow states
3. NOT test code implementation details
4. Be concise — only test the behavior that the PR changed
5. Follow existing test conventions (see API reference)
${FIXTURE_API}
## QA Video Review Report
${qaReport}
## PR Diff (for context on what changed)
${prDiff.slice(0, 8000)}
## Output Requirements
- Output ONLY the .spec.ts file content — no markdown fences, no explanations
- Start with imports, end with closing brace
- Use descriptive test names that explain the expected behavior
- Add screenshot assertions where visual verification matters
- Keep it focused: 2-5 test cases covering the core behavioral change
- Use \`test.beforeEach\` for common setup (settings, workflow directory)
- Tag the describe block with \`{ tag: '@ui' }\` or \`{ tag: '@workflow' }\` as appropriate
`
}
// ── Gemini call ─────────────────────────────────────────────────────
async function generateTest(
qaReport: string,
prDiff: string,
model: string
): Promise<string> {
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) throw new Error('GEMINI_API_KEY env var required')
const genAI = new GoogleGenerativeAI(apiKey)
const genModel = genAI.getGenerativeModel({ model })
const prompt = buildPrompt(qaReport, prDiff)
console.warn(`Sending prompt to ${model} (${prompt.length} chars)...`)
const result = await genModel.generateContent({
contents: [{ role: 'user', parts: [{ text: prompt }] }],
generationConfig: {
temperature: 0.2,
maxOutputTokens: 8192
}
})
const text = result.response.text()
// Strip markdown fences if model wraps output
return text
.replace(/^```(?:typescript|ts)?\n?/, '')
.replace(/\n?```$/, '')
.trim()
}
// ── CLI ─────────────────────────────────────────────────────────────
function parseArgs(): CliOptions {
const args = process.argv.slice(2)
const opts = { ...DEFAULTS }
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '--qa-report':
opts.qaReport = args[++i]
break
case '--pr-diff':
opts.prDiff = args[++i]
break
case '--output':
opts.output = args[++i]
break
case '--model':
opts.model = args[++i]
break
case '--help':
console.warn(`Usage:
pnpm exec tsx scripts/qa-generate-test.ts [options]
Options:
--qa-report <path> QA video review report (markdown) [required]
--pr-diff <path> PR diff file [required]
--output <path> Output .spec.ts path [required]
--model <name> Gemini model (default: gemini-3-flash-preview)`)
process.exit(0)
}
}
if (!opts.qaReport || !opts.prDiff || !opts.output) {
console.error('Missing required args. Run with --help for usage.')
process.exit(1)
}
return opts
}
async function main() {
const opts = parseArgs()
const qaReport = await readFile(resolve(opts.qaReport), 'utf-8')
const prDiff = await readFile(resolve(opts.prDiff), 'utf-8')
console.warn(
`QA report: ${basename(opts.qaReport)} (${qaReport.length} chars)`
)
console.warn(`PR diff: ${basename(opts.prDiff)} (${prDiff.length} chars)`)
const testCode = await generateTest(qaReport, prDiff, opts.model)
const outputPath = resolve(opts.output)
await writeFile(outputPath, testCode + '\n')
console.warn(`Generated test: ${outputPath} (${testCode.length} chars)`)
}
main().catch((err) => {
console.error(err)
process.exit(1)
})

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,164 @@
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><title>QA Session Recordings</title>
<link rel=preconnect href=https://fonts.googleapis.com><link rel=preconnect href=https://fonts.gstatic.com crossorigin><link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel=stylesheet>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<style>
:root{--bg:oklch(97% 0.01 265);--surface:oklch(100% 0 0);--surface-up:oklch(94% 0.01 265);--fg:oklch(15% 0.02 265);--fg-muted:oklch(40% 0.01 265);--fg-dim:oklch(55% 0.01 265);--primary:oklch(50% 0.21 265);--primary-up:oklch(45% 0.21 265);--primary-glow:oklch(55% 0.15 265);--ok:oklch(45% 0.18 155);--err:oklch(50% 0.22 25);--border:oklch(85% 0.01 265);--border-faint:oklch(90% 0.01 265);--r:0.75rem;--r-lg:1rem;--ease-out:cubic-bezier(0.22,1,0.36,1);--dur-base:250ms;--dur-slow:500ms;--font:'Inter',system-ui,sans-serif;--font-mono:'JetBrains Mono',monospace}
@media(prefers-color-scheme:dark){:root{--bg:oklch(8% 0.02 265);--surface:oklch(12% 0.02 265);--surface-up:oklch(16% 0.02 265);--fg:oklch(96% 0.01 95);--fg-muted:oklch(65% 0.01 265);--fg-dim:oklch(45% 0.01 265);--primary:oklch(62% 0.21 265);--primary-up:oklch(68% 0.21 265);--primary-glow:oklch(62% 0.15 265);--ok:oklch(62% 0.18 155);--err:oklch(62% 0.22 25);--border:oklch(22% 0.02 265);--border-faint:oklch(15% 0.01 265)}}
*{margin:0;padding:0;box-sizing:border-box}
body{background:var(--bg);color:var(--fg);font-family:var(--font);min-height:100vh;padding:clamp(1.5rem,4vw,3rem) clamp(1rem,3vw,2rem);position:relative}
@media(prefers-color-scheme:dark){body::after{content:'';position:fixed;inset:0;pointer-events:none;opacity:.03;background:url("data:image/svg+xml,%3Csvg viewBox='0 0 256 256' xmlns='http://www.w3.org/2000/svg'%3E%3Cfilter id='n'%3E%3CfeTurbulence type='fractalNoise' baseFrequency='.85' numOctaves='4' stitchTiles='stitch'/%3E%3C/filter%3E%3Crect width='100%25' height='100%25' filter='url(%23n)'/%3E%3C/svg%3E")}}
.container{max-width:1200px;margin:0 auto}
header{display:flex;align-items:center;gap:1rem;margin-bottom:clamp(1.5rem,4vw,3rem);padding-bottom:1.25rem;border-bottom:1px solid var(--border)}
.header-icon{width:36px;height:36px;display:grid;place-items:center;background:linear-gradient(135deg,oklch(100% 0 0/.06),oklch(100% 0 0/.02));backdrop-filter:blur(12px);border:1px solid oklch(100% 0 0/.1);border-radius:var(--r);flex-shrink:0}
.header-icon svg{color:var(--primary)}
h1{font-size:clamp(1.25rem,2.5vw,1.625rem);font-weight:700;letter-spacing:-.03em;background:linear-gradient(135deg,var(--fg),var(--fg-muted));-webkit-background-clip:text;-webkit-text-fill-color:transparent;background-clip:text}
.meta{color:var(--fg-dim);font-size:.8125rem;margin-top:.15rem;letter-spacing:.01em}
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(min(480px,100%),1fr));gap:1.5rem}
.card{background:var(--surface);border:1px solid var(--border);border-radius:var(--r-lg);overflow:hidden;transition:border-color var(--dur-base) var(--ease-out),box-shadow var(--dur-base) var(--ease-out),transform var(--dur-base) var(--ease-out)}
.card:hover{border-color:var(--primary);box-shadow:0 4px 16px oklch(0% 0 0/.1);transform:translateY(-2px)}
.video-wrap{position:relative;background:var(--surface);border-bottom:1px solid var(--border-faint)}
.video-wrap video{width:100%;display:block;aspect-ratio:16/9;object-fit:contain}
.card-body{padding:.75rem 1rem;display:flex;align-items:center;justify-content:space-between}
.platform{display:flex;align-items:center;gap:.5rem;font-weight:600;font-size:.9375rem;letter-spacing:-.01em}
.icon{font-size:1.125rem}
.links{display:flex;gap:.5rem}
.dl{color:var(--fg-muted);text-decoration:none;font-size:.75rem;font-weight:500;display:inline-flex;align-items:center;gap:.3rem;padding:.25rem .6rem;border-radius:9999px;border:1px solid var(--border);background:oklch(100% 0 0/.03);transition:all var(--dur-base) var(--ease-out)}
.dl:hover{color:var(--primary-up);border-color:var(--primary);background:oklch(62% 0.21 265/.08)}
.badge{font-size:.6875rem;font-weight:600;padding:.2rem .625rem;border-radius:9999px;text-transform:uppercase;letter-spacing:.05em}
.card-header{padding:.75rem 1rem;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid var(--border-faint)}
.comparison{display:grid;grid-template-columns:1fr 1fr;gap:0}
.comp-panel{border-right:1px solid var(--border-faint)}
.comp-panel:last-child{border-right:none}
.comp-label{padding:.4rem .75rem;font-size:.7rem;font-weight:600;text-transform:uppercase;letter-spacing:.05em;color:var(--fg-muted);background:var(--surface);display:flex;align-items:center;gap:.4rem}
.comp-tag{font-size:.6rem;padding:.1rem .4rem;border-radius:9999px;font-weight:600}
.comp-panel:first-child .comp-tag{background:oklch(65% 0.01 265/.15);color:var(--fg-muted);border:1px solid var(--border)}
.comp-panel:last-child .comp-tag{background:oklch(62% 0.18 155/.15);color:var(--ok);border:1px solid oklch(62% 0.18 155/.25)}
.comp-dl{padding:.4rem .75rem;display:flex;justify-content:center}
.report{border-top:1px solid var(--border-faint);padding:.75rem 1rem;font-size:.8125rem}
.report summary{cursor:pointer;color:var(--fg-muted);font-weight:500;display:flex;align-items:center;gap:.4rem;user-select:none;transition:color var(--dur-base) var(--ease-out)}
.report summary:hover{color:var(--fg)}
.report summary svg{flex-shrink:0;opacity:.5}
.report[open] summary{margin-bottom:.75rem;padding-bottom:.5rem;border-bottom:1px solid var(--border-faint)}
.report-body{line-height:1.7;color:oklch(80% 0.01 265);overflow-x:auto}
.report-body h1,.report-body h2{margin:1.25rem 0 .5rem;color:var(--fg);font-size:1rem;font-weight:600;letter-spacing:-.02em;border-bottom:1px solid var(--border-faint);padding-bottom:.4rem}
.report-body h3{margin:.75rem 0 .4rem;color:var(--fg);font-size:.875rem;font-weight:600}
.report-body p{margin:.4rem 0}
.report-body ul,.report-body ol{margin:.4rem 0 .4rem 1.5rem}
.report-body li{margin:.25rem 0}
.report-body code{background:var(--surface-up);padding:.125rem .375rem;border-radius:.25rem;font-size:.7rem;font-family:var(--font-mono);border:1px solid var(--border-faint)}
.report-body h3+p>code:first-child{background:oklch(62% 0.22 25/.15);color:var(--err);border-color:oklch(62% 0.22 25/.25)}
.report-body h3+p>code:nth-child(2){background:oklch(62% 0.21 265/.15);color:var(--primary-up);border-color:oklch(62% 0.21 265/.25)}
.report-body h3+p>code:nth-child(3){background:oklch(65% 0.01 265/.15);color:var(--fg-muted);border-color:var(--border)}
.report-body table{width:100%;border-collapse:collapse;margin:.75rem 0;font-size:.75rem;border:1px solid var(--border);border-radius:var(--r);overflow:hidden}
.report-body th,.report-body td{border:1px solid var(--border-faint);padding:.5rem .75rem;text-align:left;vertical-align:top;word-wrap:break-word}
.report-body th{background:var(--surface-up);color:var(--fg);font-weight:600;font-size:.6875rem;text-transform:uppercase;letter-spacing:.05em;position:sticky;top:0;white-space:nowrap}
.report-body tr:nth-child(even){background:color-mix(in oklch,var(--surface) 50%,transparent)}
.report-body tr:hover{background:color-mix(in oklch,var(--surface-up) 50%,transparent)}
.report-body strong{color:var(--fg)}
.report-body hr{border:none;border-top:1px solid var(--border-faint);margin:1rem 0}
@keyframes fade-up{from{opacity:0;transform:translateY(16px)}to{opacity:1;transform:translateY(0)}}
.reveal{animation:fade-up var(--dur-slow) var(--ease-out) both;animation-delay:calc(var(--i,0) * 120ms)}
@media(prefers-reduced-motion:reduce){.reveal{animation:none}}
@media(max-width:480px){.grid{grid-template-columns:1fr}.card-body{flex-wrap:wrap;gap:.5rem}}
.sha{color:var(--primary);text-decoration:none;font-family:var(--font-mono);font-size:.75rem;font-weight:500;padding:.1rem .4rem;border-radius:.25rem;background:oklch(62% 0.21 265/.08);border:1px solid oklch(62% 0.21 265/.15);transition:all var(--dur-base) var(--ease-out)}
.sha:hover{background:oklch(62% 0.21 265/.15);border-color:var(--primary)}
.badge-bar{display:flex;align-items:center;gap:.5rem;margin-bottom:1rem}
.badge-img{height:20px;display:block}
.copy-badge{background:oklch(100% 0 0/.06);border:1px solid var(--border);color:var(--fg-muted);padding:.3rem .4rem;border-radius:var(--r);cursor:pointer;display:inline-flex;align-items:center;transition:all var(--dur-base) var(--ease-out)}
.copy-badge:hover{color:var(--primary-up);border-color:var(--primary);background:oklch(62% 0.21 265/.1)}
.copy-badge.copied{color:var(--ok);border-color:var(--ok)}
.vseek{width:100%;padding:0 .75rem;background:var(--surface);border-top:1px solid var(--border-faint);position:relative;height:24px;display:flex;align-items:center}
.vseek input[type=range]{-webkit-appearance:none;appearance:none;width:100%;height:4px;background:var(--border);border-radius:2px;outline:none;cursor:pointer;position:relative;z-index:2}
.vseek input[type=range]::-webkit-slider-thumb{-webkit-appearance:none;width:12px;height:12px;border-radius:50%;background:var(--primary);cursor:pointer;border:2px solid var(--bg);box-shadow:0 0 4px oklch(0% 0 0/.3)}
.vseek input[type=range]::-moz-range-thumb{width:12px;height:12px;border-radius:50%;background:var(--primary);cursor:pointer;border:2px solid var(--bg)}
.vseek .vbuf{position:absolute;left:.75rem;right:.75rem;height:4px;border-radius:2px;pointer-events:none;top:50%;transform:translateY(-50%)}
.vseek .vbuf-bar{height:100%;background:oklch(62% 0.21 265/.25);border-radius:2px;transition:width 200ms linear}
.vctrl{display:flex;align-items:center;gap:.375rem;padding:.5rem .75rem;background:var(--surface);border-top:1px solid var(--border-faint);flex-wrap:wrap}
.vctrl button{background:oklch(100% 0 0/.06);border:1px solid var(--border);color:var(--fg-muted);font-size:.6875rem;font-weight:600;font-family:var(--font-mono);padding:.25rem .5rem;border-radius:.25rem;cursor:pointer;transition:all var(--dur-base) var(--ease-out);white-space:nowrap}
.vctrl button:hover{color:var(--primary-up);border-color:var(--primary);background:oklch(62% 0.21 265/.1)}
.vctrl button.active{color:var(--primary);border-color:var(--primary);background:oklch(62% 0.21 265/.15)}
.vctrl .vtime{font-family:var(--font-mono);font-size:.6875rem;color:var(--fg-dim);min-width:10ch;text-align:center}
.vctrl .vsep{width:1px;height:1rem;background:var(--border);flex-shrink:0}
.vctrl .vhint{font-size:.6rem;color:var(--fg-dim);margin-left:auto}
.purpose{background:linear-gradient(135deg,oklch(100% 0 0/.04),oklch(100% 0 0/.02));border:1px solid oklch(100% 0 0/.08);border-radius:var(--r-lg);padding:1rem 1.25rem;margin-bottom:1.5rem;font-size:.85rem;line-height:1.7;color:oklch(80% 0.01 265)}
.purpose strong{color:var(--fg);font-weight:600}
.purpose .purpose-label{font-size:.7rem;font-weight:600;text-transform:uppercase;letter-spacing:.05em;color:var(--fg-muted);margin-bottom:.4rem}
.purpose .purpose-reqs{margin-top:.75rem;padding-top:.75rem;border-top:1px solid oklch(100% 0 0/.06);font-size:.8rem;color:oklch(70% 0.01 265);line-height:1.8}
</style></head><body><div class=container>
<header><div class=header-icon><svg width=20 height=20 viewBox="0 0 24 24" fill=none stroke=currentColor stroke-width=2 stroke-linecap=round stroke-linejoin=round><polygon points="23 7 16 12 23 17 23 7"/><rect x=1 y=5 width=15 height=14 rx=2 ry=2/></svg></div><div><h1>QA Session Recordings</h1><div class=meta>ComfyUI Frontend &middot; Automated QA{{COMMIT_HTML}}{{RUN_LINK}}{{TIMING_HTML}}</div>{{BADGE_HTML}}</div></header>
{{PURPOSE_HTML}}<div class=grid>{{CARDS}}</div>
<div id=research-section style="margin-top:2rem"></div>
</div><script>
// Load research-log.json and reproduce.spec.ts if available
(async()=>{
const sec=document.getElementById('research-section');
try{
const [logRes,testRes,vsRes]=await Promise.allSettled([fetch('research-log.json'),fetch('reproduce.spec.ts'),fetch('video-script.spec.ts')]);
let html='';
if(logRes.status==='fulfilled'&&logRes.value.ok){
const log=await logRes.value.json();
// Show verdict banner for non-reproduced results
if(log.verdict&&log.verdict!=='REPRODUCED'){
const colors={NOT_REPRODUCIBLE:{bg:'oklch(25% 0.08 25)',border:'oklch(40% 0.15 25)',icon:'✗'},INCONCLUSIVE:{bg:'oklch(25% 0.06 80)',border:'oklch(40% 0.12 80)',icon:'⚠'}};
const c=colors[log.verdict]||colors.INCONCLUSIVE;
html+=`<div style="margin-bottom:1.5rem;padding:1.25rem;background:${c.bg};border:1px solid ${c.border};border-radius:var(--r-lg)"><div style="font-size:1.25rem;font-weight:700;margin-bottom:.5rem">${c.icon} ${log.verdict.replace(/_/g,' ')}</div><div style="font-size:.9rem;line-height:1.6;opacity:.9">${(log.summary||'No details available.').replace(/</g,'&lt;')}</div>${log.evidence?`<div style="margin-top:.75rem;padding:.75rem;background:oklch(0% 0 0/.2);border-radius:var(--r);font-family:var(--font-mono);font-size:.8rem;white-space:pre-wrap;max-height:200px;overflow:auto">${log.evidence.replace(/</g,'&lt;')}</div>`:''}</div>`;
}
html+=`<details style="margin-bottom:1.5rem"><summary style="cursor:pointer;font-weight:600;font-size:1rem;padding:.75rem 1rem;background:var(--surface);border:1px solid var(--border);border-radius:var(--r-lg)">Research Log &mdash; ${log.verdict||'?'} (${(log.log||[]).length||'?'} tool calls, ${((log.elapsedMs||0)/1000).toFixed(1)}s)</summary><div style="padding:1rem;background:var(--surface);border:1px solid var(--border);border-top:0;border-radius:0 0 var(--r-lg) var(--r-lg);overflow:auto;max-height:600px"><pre style="font-family:var(--font-mono);font-size:.8rem;line-height:1.6;white-space:pre-wrap">${JSON.stringify(log,null,2)}</pre></div></details>`;
}
if(testRes.status==='fulfilled'&&testRes.value.ok){
const code=await testRes.value.text();
html+=`<details><summary style="cursor:pointer;font-weight:600;font-size:1rem;padding:.75rem 1rem;background:var(--surface);border:1px solid var(--border);border-radius:var(--r-lg)">E2E Test Code (reproduce.spec.ts)</summary><div style="padding:1rem;background:var(--surface);border:1px solid var(--border);border-top:0;border-radius:0 0 var(--r-lg) var(--r-lg);overflow:auto;max-height:600px"><pre style="font-family:var(--font-mono);font-size:.8rem;line-height:1.6;white-space:pre-wrap">${code.replace(/</g,'&lt;').replace(/>/g,'&gt;')}</pre></div></details>`;
}
if(vsRes.status==='fulfilled'&&vsRes.value.ok){
const vsCode=await vsRes.value.text();
html+=`<details style="margin-top:1rem"><summary style="cursor:pointer;font-weight:600;font-size:1rem;padding:.75rem 1rem;background:var(--surface);border:1px solid var(--border);border-radius:var(--r-lg)">Video Script (video-script.spec.ts)</summary><div style="padding:1rem;background:var(--surface);border:1px solid var(--border);border-top:0;border-radius:0 0 var(--r-lg) var(--r-lg);overflow:auto;max-height:600px"><pre style="font-family:var(--font-mono);font-size:.8rem;line-height:1.6;white-space:pre-wrap">${vsCode.replace(/</g,'&lt;').replace(/>/g,'&gt;')}</pre></div></details>`;
}
if(html)sec.innerHTML=html;
}catch(e){console.warn('research load failed',e)}
})();
</script><script>
function copyBadge(){const u=location.href.replace(/\/[^/]*$/,'/');const b=u+'badge.svg';const md='[![QA Badge]('+b+')]('+u+')';navigator.clipboard.writeText(md).then(()=>{const btn=document.querySelector('.copy-badge');btn.classList.add('copied');btn.innerHTML='<svg width=14 height=14 viewBox="0 0 24 24" fill=none stroke=currentColor stroke-width=2><polyline points="20 6 9 17 4 12"/></svg>';setTimeout(()=>{btn.classList.remove('copied');btn.innerHTML='<svg width=14 height=14 viewBox="0 0 24 24" fill=none stroke=currentColor stroke-width=2><rect x=9 y=9 width=13 height=13 rx=2/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg>'},2000)})}
document.querySelectorAll('[data-md]').forEach(el=>{const t=el.textContent;el.removeAttribute('data-md');el.innerHTML=marked.parse(t)});
const FPS=30,FT=1/FPS,SPEEDS=[0.1,0.25,0.5,1,1.5,2];
document.querySelectorAll('.video-wrap video').forEach(v=>{
v.playbackRate=1;
const c=document.createElement('div');c.className='vctrl';
const btn=(label,fn)=>{const b=document.createElement('button');b.textContent=label;b.onclick=fn;c.appendChild(b);return b};
const sep=()=>{const s=document.createElement('div');s.className='vsep';c.appendChild(s)};
const time=document.createElement('span');time.className='vtime';time.textContent='0:00.000';
btn('\u23EE',()=>{v.currentTime=0});
btn('\u25C0\u25C0',()=>{v.currentTime=Math.max(0,v.currentTime-FT*10)});
btn('\u25C0',()=>{v.pause();v.currentTime=Math.max(0,v.currentTime-FT)});
const playBtn=btn('\u25B6',()=>{v.paused?v.play():v.pause()});
btn('\u25B6\u25B6',()=>{v.pause();v.currentTime+=FT});
btn('\u25B6\u25B6\u25B6',()=>{v.currentTime+=FT*10});
sep();
const spdBtns=SPEEDS.map(s=>{const b=btn(s+'x',()=>{v.playbackRate=s;spdBtns.forEach(x=>x.classList.remove('active'));b.classList.add('active')});if(s===1)b.classList.add('active');return b});
sep();c.appendChild(time);
const hint=document.createElement('span');hint.className='vhint';hint.textContent='\u2190\u2192 frame \u2022 space play';c.appendChild(hint);
// Custom seekbar — works even without server range request support
const seekWrap=document.createElement('div');seekWrap.className='vseek';
const seekBar=document.createElement('input');seekBar.type='range';seekBar.min=0;seekBar.max=1000;seekBar.value=0;seekBar.step=1;
const bufWrap=document.createElement('div');bufWrap.className='vbuf';
const bufBar=document.createElement('div');bufBar.className='vbuf-bar';bufBar.style.width='0%';
bufWrap.appendChild(bufBar);seekWrap.appendChild(bufWrap);seekWrap.appendChild(seekBar);
let seeking=false;
seekBar.oninput=()=>{seeking=true;if(v.duration){v.currentTime=v.duration*(seekBar.value/1000)}};
seekBar.onchange=()=>{seeking=false};
v.closest('.video-wrap').after(seekWrap);
seekWrap.after(c);
v.ontimeupdate=()=>{
const m=Math.floor(v.currentTime/60),s=Math.floor(v.currentTime%60),ms=Math.floor((v.currentTime%1)*1000);
time.textContent=m+':'+(s<10?'0':'')+s+'.'+String(ms).padStart(3,'0');
if(!seeking&&v.duration){seekBar.value=Math.round((v.currentTime/v.duration)*1000)}
};
v.onprogress=v.onloadeddata=()=>{if(v.buffered.length&&v.duration){bufBar.style.width=(v.buffered.end(v.buffered.length-1)/v.duration*100)+'%'}};
v.onplay=()=>{playBtn.textContent='\u23F8'};v.onpause=()=>{playBtn.textContent='\u25B6'};
v.parentElement.addEventListener('keydown',e=>{
if(e.key==='ArrowLeft'){e.preventDefault();v.pause();v.currentTime=Math.max(0,v.currentTime-FT)}
if(e.key==='ArrowRight'){e.preventDefault();v.pause();v.currentTime+=FT}
if(e.key===' '){e.preventDefault();v.paused?v.play():v.pause()}
});
v.parentElement.setAttribute('tabindex','0');
});
</script></body></html>

View File

@@ -0,0 +1,253 @@
#!/usr/bin/env tsx
/**
* QA Reproduce Phase — Deterministic replay of research plan with narration
*
* Takes a reproduction plan from the research phase and replays it:
* 1. Execute each action deterministically (no AI decisions)
* 2. Capture a11y snapshot before/after each action
* 3. Gemini describes what visually changed (narration for humans)
* 4. Output: narration-log.json with full evidence chain
*/
import type { Page } from '@playwright/test'
import { GoogleGenerativeAI } from '@google/generative-ai'
import { mkdirSync, writeFileSync } from 'fs'
import type { ActionResult } from './qa-record.js'
// ── Types ──
interface ReproductionStep {
action: Record<string, unknown> & { action: string }
expectedAssertion: string
}
interface NarrationEntry {
step: number
action: string
params: Record<string, unknown>
result: ActionResult
a11yBefore: unknown
a11yAfter: unknown
assertionExpected: string
assertionPassed: boolean
assertionActual: string
geminiNarration: string
timestampMs: number
}
export interface NarrationLog {
entries: NarrationEntry[]
allAssertionsPassed: boolean
}
interface ReproduceOptions {
page: Page
plan: ReproductionStep[]
geminiApiKey: string
outputDir: string
}
// ── A11y helpers ──
interface A11yNode {
role: string
name: string
value?: string
checked?: boolean
disabled?: boolean
expanded?: boolean
children?: A11yNode[]
}
function searchA11y(node: A11yNode | null, selector: string): A11yNode | null {
if (!node) return null
const sel = selector.toLowerCase()
if (
node.name?.toLowerCase().includes(sel) ||
node.role?.toLowerCase().includes(sel)
) {
return node
}
if (node.children) {
for (const child of node.children) {
const found = searchA11y(child, selector)
if (found) return found
}
}
return null
}
function summarizeA11y(node: A11yNode | null): string {
if (!node) return 'null'
const parts = [`role=${node.role}`, `name="${node.name}"`]
if (node.value !== undefined) parts.push(`value="${node.value}"`)
if (node.checked !== undefined) parts.push(`checked=${node.checked}`)
if (node.disabled) parts.push('disabled')
if (node.expanded !== undefined) parts.push(`expanded=${node.expanded}`)
return `{${parts.join(', ')}}`
}
// ── Subtitle overlay ──
async function showSubtitle(page: Page, text: string, step: number) {
const encoded = encodeURIComponent(
text.slice(0, 120).replace(/'/g, "\\'").replace(/\n/g, ' ')
)
await page.addScriptTag({
content: `(function(){
var id='qa-subtitle';
var el=document.getElementById(id);
if(!el){
el=document.createElement('div');
el.id=id;
Object.assign(el.style,{position:'fixed',bottom:'32px',left:'50%',transform:'translateX(-50%)',zIndex:'2147483646',maxWidth:'90%',padding:'6px 14px',borderRadius:'6px',background:'rgba(0,0,0,0.8)',color:'rgba(255,255,255,0.95)',fontSize:'12px',fontFamily:'system-ui,sans-serif',fontWeight:'400',lineHeight:'1.4',pointerEvents:'none',textAlign:'center',whiteSpace:'normal'});
document.body.appendChild(el);
}
el.textContent='['+${step}+'] '+decodeURIComponent('${encoded}');
})()`
})
}
// ── Gemini visual narration ──
async function geminiDescribe(
page: Page,
geminiApiKey: string,
focus: string
): Promise<string> {
try {
const screenshot = await page.screenshot({ type: 'jpeg', quality: 70 })
const genAI = new GoogleGenerativeAI(geminiApiKey)
const model = genAI.getGenerativeModel({ model: 'gemini-3-flash-preview' })
const result = await model.generateContent([
{
text: `Describe in 1-2 sentences what you see on this ComfyUI screen. Focus on: ${focus}. Be factual — only describe what is visible.`
},
{
inlineData: {
mimeType: 'image/jpeg',
data: screenshot.toString('base64')
}
}
])
return result.response.text().trim()
} catch (e) {
return `(Gemini narration failed: ${e instanceof Error ? e.message.slice(0, 50) : e})`
}
}
// ── Main reproduce function ──
export async function runReproducePhase(
opts: ReproduceOptions
): Promise<NarrationLog> {
const { page, plan, geminiApiKey, outputDir } = opts
const { executeAction } = await import('./qa-record.js')
const narrationDir = `${outputDir}/narration`
mkdirSync(narrationDir, { recursive: true })
const entries: NarrationEntry[] = []
const startMs = Date.now()
console.warn(`Reproduce phase: replaying ${plan.length} steps...`)
for (let i = 0; i < plan.length; i++) {
const step = plan[i]
const actionObj = step.action
const elapsed = Date.now() - startMs
// Show subtitle
await showSubtitle(page, `Step ${i + 1}: ${actionObj.action}`, i + 1)
console.warn(` [${i + 1}/${plan.length}] ${actionObj.action}`)
// Capture a11y BEFORE
const a11yBefore = await page
.locator('body')
.ariaSnapshot({ timeout: 3000 })
.catch(() => null)
// Execute action
const result = await executeAction(
page,
actionObj as Parameters<typeof executeAction>[1],
outputDir
)
await new Promise((r) => setTimeout(r, 500))
// Capture a11y AFTER
const a11yAfter = await page
.locator('body')
.ariaSnapshot({ timeout: 3000 })
.catch(() => null)
// Check assertion
let assertionPassed = false
let assertionActual = ''
if (step.expectedAssertion) {
// Parse the expected assertion — e.g. "Settings dialog: visible" or "tab count: 2"
const parts = step.expectedAssertion.split(':').map((s) => s.trim())
const selectorName = parts[0]
const expectedState = parts.slice(1).join(':').trim()
const found = searchA11y(a11yAfter as A11yNode | null, selectorName)
assertionActual = found ? summarizeA11y(found) : 'NOT FOUND'
if (expectedState === 'visible' || expectedState === 'exists') {
assertionPassed = found !== null
} else if (expectedState === 'hidden' || expectedState === 'gone') {
assertionPassed = found === null
} else {
// Generic: check if the actual state contains the expected text
assertionPassed = assertionActual
.toLowerCase()
.includes(expectedState.toLowerCase())
}
console.warn(
` Assertion: "${step.expectedAssertion}" → ${assertionPassed ? '✓ PASS' : '✗ FAIL'} (actual: ${assertionActual})`
)
}
// Gemini narration (visual description for humans)
const geminiNarration = await geminiDescribe(
page,
geminiApiKey,
`What changed after ${actionObj.action}?`
)
entries.push({
step: i + 1,
action: actionObj.action,
params: actionObj,
result,
a11yBefore,
a11yAfter,
assertionExpected: step.expectedAssertion,
assertionPassed,
assertionActual,
geminiNarration,
timestampMs: elapsed
})
}
// Final screenshot
await page.screenshot({ path: `${outputDir}/reproduce-final.png` })
const log: NarrationLog = {
entries,
allAssertionsPassed: entries.every((e) => e.assertionPassed)
}
writeFileSync(
`${narrationDir}/narration-log.json`,
JSON.stringify(log, null, 2)
)
console.warn(
`Reproduce phase complete: ${entries.filter((e) => e.assertionPassed).length}/${entries.length} assertions passed`
)
return log
}

View File

@@ -0,0 +1,150 @@
import { describe, expect, it } from 'vitest'
import {
extractPlatformFromArtifactDirName,
pickLatestVideosByPlatform,
selectVideoCandidateByFile
} from './qa-video-review'
describe('extractPlatformFromArtifactDirName', () => {
it('extracts and normalizes known qa artifact directory names', () => {
expect(
extractPlatformFromArtifactDirName('qa-report-Windows-22818315023')
).toBe('windows')
expect(
extractPlatformFromArtifactDirName('qa-report-macOS-22818315023')
).toBe('macos')
expect(
extractPlatformFromArtifactDirName('qa-report-Linux-22818315023')
).toBe('linux')
})
it('falls back to slugifying unknown directory names', () => {
expect(extractPlatformFromArtifactDirName('custom platform run')).toBe(
'custom-platform-run'
)
})
})
describe('pickLatestVideosByPlatform', () => {
it('keeps only the latest candidate per platform', () => {
const selected = pickLatestVideosByPlatform([
{
platformName: 'windows',
videoPath: '/tmp/windows-old.mp4',
mtimeMs: 100
},
{
platformName: 'windows',
videoPath: '/tmp/windows-new.mp4',
mtimeMs: 200
},
{
platformName: 'linux',
videoPath: '/tmp/linux.mp4',
mtimeMs: 150
}
])
expect(selected).toEqual([
{
platformName: 'linux',
videoPath: '/tmp/linux.mp4',
mtimeMs: 150
},
{
platformName: 'windows',
videoPath: '/tmp/windows-new.mp4',
mtimeMs: 200
}
])
})
})
describe('selectVideoCandidateByFile', () => {
it('selects a single candidate by artifacts-relative path', () => {
const selected = selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
},
{
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-report-Linux-1/qa-session.mp4'
}
)
expect(selected).toEqual({
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
})
})
it('throws when basename matches multiple videos', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
},
{
platformName: 'linux',
videoPath: '/tmp/qa-artifacts/qa-report-Linux-1/qa-session.mp4',
mtimeMs: 200
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-session.mp4'
}
)
).toThrow('matched 2 videos')
})
it('throws when there is no matching video', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: 'qa-report-macOS-1/qa-session.mp4'
}
)
).toThrow('No video matched')
})
it('throws when video file is missing', () => {
expect(() =>
selectVideoCandidateByFile(
[
{
platformName: 'windows',
videoPath: '/tmp/qa-artifacts/qa-report-Windows-1/qa-session.mp4',
mtimeMs: 100
}
],
{
artifactsDir: '/tmp/qa-artifacts',
videoFile: ' '
}
)
).toThrow('--video-file is required')
})
})

View File

@@ -0,0 +1,771 @@
#!/usr/bin/env tsx
import { mkdir, readFile, stat, writeFile } from 'node:fs/promises'
import { basename, dirname, extname, relative, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { GoogleGenerativeAI } from '@google/generative-ai'
import { globSync } from 'glob'
interface CliOptions {
artifactsDir: string
videoFile: string
beforeVideo: string
outputDir: string
model: string
requestTimeoutMs: number
dryRun: boolean
prContext: string
targetUrl: string
passLabel: string
}
interface VideoCandidate {
platformName: string
videoPath: string
mtimeMs: number
}
const DEFAULT_OPTIONS: CliOptions = {
artifactsDir: './tmp/qa-artifacts',
videoFile: '',
beforeVideo: '',
outputDir: './tmp',
model: 'gemini-3-flash-preview',
requestTimeoutMs: 300_000,
dryRun: false,
prContext: '',
targetUrl: '',
passLabel: ''
}
const USAGE = `Usage:
pnpm exec tsx scripts/qa-video-review.ts [options]
Options:
--artifacts-dir <path> Artifacts root directory
(default: ./tmp/qa-artifacts)
--video-file <name-or-path> Video file to analyze (required)
(supports basename or relative/absolute path)
--before-video <path> Before video (main branch) for comparison
When provided, sends both videos to Gemini
for comparative before/after analysis
--output-dir <path> Output directory for markdown reports
(default: ./tmp)
--model <name> Gemini model
(default: gemini-3-flash-preview)
--request-timeout-ms <n> Request timeout in milliseconds
(default: 300000)
--pr-context <file> File with PR context (title, body, diff)
for PR-aware review
--target-url <url> Issue or PR URL to include in the report
--pass-label <label> Label for multi-pass reports (e.g. pass1)
Output becomes {platform}-{label}-qa-video-report.md
--dry-run Discover videos and output targets only
--help Show this help text
Environment:
GEMINI_API_KEY Required unless --dry-run
`
function parsePositiveInteger(rawValue: string, flagName: string): number {
const parsedValue = Number.parseInt(rawValue, 10)
if (!Number.isInteger(parsedValue) || parsedValue <= 0) {
throw new Error(`Invalid value for ${flagName}: "${rawValue}"`)
}
return parsedValue
}
function parseCliOptions(args: string[]): CliOptions {
const options: CliOptions = { ...DEFAULT_OPTIONS }
for (let index = 0; index < args.length; index += 1) {
const argument = args[index]
const nextValue = args[index + 1]
const requireValue = (flagName: string): string => {
if (!nextValue || nextValue.startsWith('--')) {
throw new Error(`Missing value for ${flagName}`)
}
index += 1
return nextValue
}
if (argument === '--help') {
process.stdout.write(USAGE)
process.exit(0)
}
if (argument === '--artifacts-dir') {
options.artifactsDir = requireValue(argument)
continue
}
if (argument === '--video-file') {
options.videoFile = requireValue(argument)
continue
}
if (argument === '--output-dir') {
options.outputDir = requireValue(argument)
continue
}
if (argument === '--model') {
options.model = requireValue(argument)
continue
}
if (argument === '--request-timeout-ms') {
options.requestTimeoutMs = parsePositiveInteger(
requireValue(argument),
argument
)
continue
}
if (argument === '--before-video') {
options.beforeVideo = requireValue(argument)
continue
}
if (argument === '--pr-context') {
options.prContext = requireValue(argument)
continue
}
if (argument === '--target-url') {
options.targetUrl = requireValue(argument)
continue
}
if (argument === '--pass-label') {
options.passLabel = requireValue(argument)
continue
}
if (argument === '--dry-run') {
options.dryRun = true
continue
}
throw new Error(`Unknown argument: ${argument}`)
}
return options
}
function normalizePlatformName(value: string): string {
const slug = value
.trim()
.toLowerCase()
.replace(/[^a-z0-9]+/g, '-')
.replace(/^-+|-+$/g, '')
return slug.length > 0 ? slug : 'unknown-platform'
}
export function extractPlatformFromArtifactDirName(dirName: string): string {
const matchedValue = dirName.match(/^qa-report-(.+?)(?:-\d+)?$/i)?.[1]
return normalizePlatformName(matchedValue ?? dirName)
}
function extractPlatformFromVideoPath(videoPath: string): string {
const artifactDirName = basename(dirname(videoPath))
return extractPlatformFromArtifactDirName(artifactDirName)
}
export function pickLatestVideosByPlatform(
candidates: VideoCandidate[]
): VideoCandidate[] {
const latestByPlatform = new Map<string, VideoCandidate>()
for (const candidate of candidates) {
const current = latestByPlatform.get(candidate.platformName)
if (!current || candidate.mtimeMs > current.mtimeMs) {
latestByPlatform.set(candidate.platformName, candidate)
}
}
return [...latestByPlatform.values()].sort((a, b) =>
a.platformName.localeCompare(b.platformName)
)
}
function toProjectRelativePath(targetPath: string): string {
const relativePath = relative(process.cwd(), targetPath)
if (relativePath.startsWith('.')) {
return relativePath
}
return `./${relativePath}`
}
function errorToString(error: unknown): string {
return error instanceof Error ? error.message : String(error)
}
function normalizePathForMatch(value: string): string {
return value.replaceAll('\\', '/').replace(/^\.\/+/, '')
}
export function selectVideoCandidateByFile(
candidates: VideoCandidate[],
options: { artifactsDir: string; videoFile: string }
): VideoCandidate {
const requestedValue = options.videoFile.trim()
if (requestedValue.length === 0) {
throw new Error('--video-file is required')
}
const artifactsRoot = resolve(options.artifactsDir)
const requestedAbsolutePath = resolve(requestedValue)
const requestedPathKey = normalizePathForMatch(requestedValue)
const matches = candidates.filter((candidate) => {
const candidateAbsolutePath = resolve(candidate.videoPath)
if (candidateAbsolutePath === requestedAbsolutePath) {
return true
}
const candidateBaseName = basename(candidate.videoPath)
if (candidateBaseName === requestedValue) {
return true
}
const relativeToCwd = normalizePathForMatch(
relative(process.cwd(), candidateAbsolutePath)
)
if (relativeToCwd === requestedPathKey) {
return true
}
const relativeToArtifacts = normalizePathForMatch(
relative(artifactsRoot, candidateAbsolutePath)
)
return relativeToArtifacts === requestedPathKey
})
if (matches.length === 1) {
return matches[0]
}
if (matches.length === 0) {
const availableVideos = candidates.map((candidate) =>
toProjectRelativePath(candidate.videoPath)
)
throw new Error(
[
`No video matched --video-file "${options.videoFile}".`,
'Available videos:',
...availableVideos.map((videoPath) => `- ${videoPath}`)
].join('\n')
)
}
throw new Error(
[
`--video-file "${options.videoFile}" matched ${matches.length} videos.`,
'Please pass a more specific path.',
...matches.map((match) => `- ${toProjectRelativePath(match.videoPath)}`)
].join('\n')
)
}
async function collectVideoCandidates(
artifactsDir: string
): Promise<VideoCandidate[]> {
const absoluteArtifactsDir = resolve(artifactsDir)
const videoPaths = globSync('**/qa-session{,-[0-9]}.mp4', {
cwd: absoluteArtifactsDir,
absolute: true,
nodir: true
}).sort()
const candidates = await Promise.all(
videoPaths.map(async (videoPath) => {
const videoStat = await stat(videoPath)
return {
platformName: extractPlatformFromVideoPath(videoPath),
videoPath,
mtimeMs: videoStat.mtimeMs
}
})
)
return candidates
}
function getMimeType(filePath: string): string {
const ext = extname(filePath).toLowerCase()
const mimeMap: Record<string, string> = {
'.mp4': 'video/mp4',
'.webm': 'video/webm',
'.mov': 'video/quicktime',
'.avi': 'video/x-msvideo',
'.mkv': 'video/x-matroska',
'.m4v': 'video/mp4'
}
return mimeMap[ext] || 'video/mp4'
}
function buildReviewPrompt(options: {
platformName: string
videoPath: string
prContext: string
isComparative: boolean
}): string {
const { platformName, videoPath, prContext, isComparative } = options
if (isComparative) {
return buildComparativePrompt(platformName, videoPath, prContext)
}
return buildSingleVideoPrompt(platformName, videoPath, prContext)
}
function buildComparativePrompt(
platformName: string,
videoPath: string,
prContext: string
): string {
const lines = [
'You are a senior QA engineer performing a BEFORE/AFTER comparison review.',
'',
'You are given TWO videos:',
'- **Video 1 (BEFORE)**: The main branch BEFORE the PR. This shows the OLD behavior.',
'- **Video 2 (AFTER)**: The PR branch AFTER the changes. This shows the NEW behavior.',
'',
'Both videos show the same test steps executed on different code versions.',
''
]
if (prContext) {
lines.push('## PR Context', prContext, '')
}
lines.push(
'## Your Task',
`Platform: "${platformName}". After video: ${toProjectRelativePath(videoPath)}.`,
'',
'1. **BEFORE video**: Does it demonstrate the old behavior or bug that the PR aims to fix?',
' Describe what you observe — this establishes the baseline.',
'2. **AFTER video**: Does it prove the PR fix works? Is the intended new behavior visible?',
'3. **Comparison**: What specifically changed between before and after?',
'4. **Regressions**: Did the PR introduce any new problems visible in the AFTER video',
' that were NOT present in the BEFORE video?',
'',
'Note: Brief black frames during page transitions are NORMAL.',
'Note: Small cyan/purple dashed labels prefixed with "QA:" are annotations placed by the automated test script — they are NOT part of the application UI. Do not treat them as bugs or evidence.',
'Report only concrete, visible differences. Avoid speculation.',
'',
'Return markdown with these sections exactly:',
'## Summary',
'(What the PR changes, whether BEFORE confirms the old behavior, whether AFTER proves the fix)',
'',
'## Behavior Changes',
'Summarize ALL behavioral differences as a markdown TABLE:',
'| Behavior | Before (main) | After (PR) | Verdict |',
'',
'- **Behavior**: short name for the behavior (e.g. "Save shortcut label", "Menu hover style")',
'- **Before (main)**: how it works/looks in the BEFORE video',
'- **After (PR)**: how it works/looks in the AFTER video',
'- **Verdict**: `Fixed`, `Improved`, `Changed`, `Regression`, or `No Change`',
'',
'One row per distinct behavior. Include both changed AND unchanged key behaviors',
'that were tested, so reviewers can confirm nothing was missed.',
'',
'## Timeline Comparison',
'Present a chronological frame-by-frame comparison as a markdown TABLE:',
'| Time | Type | Severity | Before (main) | After (PR) |',
'',
'- **Time**: timestamp or range from the videos (e.g. `0:05-0:08`)',
'- **Type**: category such as `Visual`, `Behavior`, `Layout`, `Text`, `Animation`, `Menu`, `State`',
'- **Severity**: `None` (neutral change), `Fixed` (bug resolved), `Regression`, `Minor`, `Major`',
'- **Before (main)**: what is observed in the BEFORE video at that time',
'- **After (PR)**: what is observed in the AFTER video at that time',
'',
'Include one row per distinct observable difference. If behavior is identical at a timestamp,',
'omit that row. Focus on meaningful differences, not narrating every frame.',
'',
'## Confirmed Issues',
'For each issue, use this exact format:',
'',
'### [Short issue title]',
'`SEVERITY` `TIMESTAMP` `Confidence: LEVEL`',
'',
'[Description — specify whether it appears in BEFORE, AFTER, or both]',
'',
'**Evidence:** [What you observed at the given timestamp in which video]',
'',
'**Suggested Fix:** [Actionable recommendation]',
'',
'---',
'',
'## Possible Issues (Needs Human Verification)',
'## Overall Risk',
'(Assess whether the PR achieves its goal based on the before/after comparison)',
'',
'## Verdict',
'End your report with this EXACT JSON block (no markdown fence):',
'{"verdict": "REPRODUCED" | "NOT_REPRODUCIBLE" | "INCONCLUSIVE", "risk": "low" | "medium" | "high", "confidence": "high" | "medium" | "low"}',
'- REPRODUCED: the before video confirms the old behavior and the after video shows the fix working',
'- NOT_REPRODUCIBLE: the before video does not show the reported bug',
'- INCONCLUSIVE: the videos do not adequately demonstrate the behavior change'
)
return lines.filter(Boolean).join('\n')
}
function buildSingleVideoPrompt(
platformName: string,
videoPath: string,
prContext: string
): string {
const lines = [
'You are a senior QA engineer reviewing a UI test session recording.',
'',
'## ANTI-HALLUCINATION RULES (READ FIRST)',
'- Describe ONLY what you can directly observe in the video frames',
'- NEVER infer or assume what "must have happened" between frames',
'- If a step is not visible in the video, say "NOT SHOWN" — do not guess',
'- Your job is to be a CAMERA — report facts, not interpretations',
''
]
const isIssueContext =
prContext &&
/^### Issue #|^Title:.*\bbug\b|^This video attempts to reproduce/im.test(
prContext
)
if (prContext) {
lines.push(
'## Phase 1: Blind Observation (describe what you SEE and HEAR)',
'First, describe every UI interaction chronologically WITHOUT knowing the expected outcome:',
'- What elements does the user click/hover/type?',
'- What dialogs/menus open and close?',
'- What keyboard indicators appear? (look for subtitle overlays)',
'- What is the BEFORE state and AFTER state of each action?',
'- **Audio**: Does the video have a TTS narration audio track? If yes, transcribe what the voice says. This narration describes the bug being reproduced.',
'',
'## Phase 2: Compare against expected behavior',
'Now compare your observations against the context below.',
'Only claim a match if your Phase 1 observations EXPLICITLY support it.',
''
)
if (isIssueContext) {
lines.push(
'## Issue Context',
prContext,
'',
'## Comparison Questions',
'1. Did the video perform the reproduction steps described in the issue?',
'2. Did your Phase 1 observations show the reported bug behavior?',
'3. If the steps were not performed or the bug was not visible, say INCONCLUSIVE.',
''
)
} else {
lines.push(
'## PR Context',
prContext,
'',
'## Comparison Questions',
'1. Did the video test the specific behavior the PR changes?',
'2. Did your Phase 1 observations show the expected before/after difference?',
'3. If the test was incomplete or inconclusive, say so honestly.',
''
)
}
}
lines.push(
`Review this QA session video for platform "${platformName}".`,
`Source video: ${toProjectRelativePath(videoPath)}.`,
'The video shows the full test session — analyze it chronologically.',
'Focus on UI regressions, broken states, visual glitches, unreadable text, missing labels/i18n, and clear workflow failures.',
'Note: Brief black frames during page transitions are NORMAL and should NOT be reported as issues.',
'Note: Small cyan/purple dashed labels prefixed with "QA:" are annotations placed by the automated test script — they are NOT part of the application UI. Do not treat them as bugs or evidence.',
'Report only concrete, visible problems and avoid speculation.',
'If confidence is low, mark it explicitly.',
'',
'Return markdown with these sections exactly:',
'## Summary',
isIssueContext
? '(Explain what bug was reported and whether the video confirms it is reproducible)'
: prContext
? '(Explain what the PR intended and whether the video confirms it works)'
: '',
'## Confirmed Issues',
'For each confirmed issue, use this exact format (one block per issue):',
'',
'### [Short issue title]',
'`HIGH` `01:03` `Confidence: High`',
'',
'[Description of the issue — what went wrong and what was expected]',
'',
'**Evidence:** [What you observed in the video at the given timestamp]',
'',
'**Suggested Fix:** [Actionable recommendation]',
'',
'---',
'',
'The first line after the heading MUST be exactly three backtick-wrapped labels:',
'`SEVERITY` `TIMESTAMP` `Confidence: LEVEL`',
'Do NOT use a table for issues — use the block format above.',
'## Possible Issues (Needs Human Verification)',
'## Overall Risk',
'',
'## Narration',
'If the video contains a TTS audio narration track, transcribe it here.',
'If there is no audio or the video is silent, write "No narration detected."',
'',
'## Verdict',
'End your report with this EXACT JSON block (no markdown fence):',
'{"verdict": "REPRODUCED" | "NOT_REPRODUCIBLE" | "INCONCLUSIVE", "risk": "low" | "medium" | "high" | null, "confidence": "high" | "medium" | "low", "narrationDetected": true | false}',
'- REPRODUCED: the bug/behavior is clearly visible in the video',
'- NOT_REPRODUCIBLE: the steps were performed correctly but the bug was not observed',
'- INCONCLUSIVE: the reproduction steps were not performed or the video is insufficient',
'- narrationDetected: true if you heard TTS voice narration in the video, false if silent'
)
return lines.filter(Boolean).join('\n')
}
const MAX_VIDEO_BYTES = 100 * 1024 * 1024
async function readVideoFile(videoPath: string): Promise<Buffer> {
const fileStat = await stat(videoPath)
if (fileStat.size > MAX_VIDEO_BYTES) {
throw new Error(
`Video ${basename(videoPath)} is ${formatBytes(fileStat.size)}, exceeds ${formatBytes(MAX_VIDEO_BYTES)} limit`
)
}
return readFile(videoPath)
}
async function requestGeminiReview(options: {
apiKey: string
model: string
platformName: string
videoPath: string
beforeVideoPath: string
timeoutMs: number
prContext: string
}): Promise<string> {
const genAI = new GoogleGenerativeAI(options.apiKey)
const model = genAI.getGenerativeModel({ model: options.model })
const isComparative = options.beforeVideoPath.length > 0
const prompt = buildReviewPrompt({
platformName: options.platformName,
videoPath: options.videoPath,
prContext: options.prContext,
isComparative
})
const parts: Array<
{ text: string } | { inlineData: { mimeType: string; data: string } }
> = [{ text: prompt }]
if (isComparative) {
const beforeBuffer = await readVideoFile(options.beforeVideoPath)
parts.push(
{ text: 'Video 1 — BEFORE (main branch):' },
{
inlineData: {
mimeType: getMimeType(options.beforeVideoPath),
data: beforeBuffer.toString('base64')
}
}
)
}
const afterBuffer = await readVideoFile(options.videoPath)
if (isComparative) {
parts.push({ text: 'Video 2 — AFTER (PR branch):' })
}
parts.push({
inlineData: {
mimeType: getMimeType(options.videoPath),
data: afterBuffer.toString('base64')
}
})
const result = await model.generateContent(parts, {
timeout: options.timeoutMs
})
const response = result.response
const text = response.text()
if (!text || text.trim().length === 0) {
throw new Error('Gemini API returned no output text')
}
return text.trim()
}
function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
}
function buildReportMarkdown(input: {
platformName: string
model: string
videoPath: string
videoSizeBytes: number
beforeVideoPath?: string
beforeVideoSizeBytes?: number
reviewText: string
targetUrl?: string
}): string {
const headerLines = [
`# ${input.platformName} QA Video Report`,
'',
`- Generated at: ${new Date().toISOString()}`,
`- Model: \`${input.model}\``
]
if (input.targetUrl) {
headerLines.push(`- Target: ${input.targetUrl}`)
}
if (input.beforeVideoPath) {
headerLines.push(
`- Before video: \`${toProjectRelativePath(input.beforeVideoPath)}\` (${formatBytes(input.beforeVideoSizeBytes ?? 0)})`,
`- After video: \`${toProjectRelativePath(input.videoPath)}\` (${formatBytes(input.videoSizeBytes)})`,
'- Mode: **Comparative (before/after)**'
)
} else {
headerLines.push(
`- Source video: \`${toProjectRelativePath(input.videoPath)}\``,
`- Video size: ${formatBytes(input.videoSizeBytes)}`
)
}
headerLines.push('', '## AI Review', '')
return `${headerLines.join('\n')}${input.reviewText.trim()}\n`
}
async function reviewVideo(
video: VideoCandidate,
options: CliOptions,
apiKey: string
): Promise<void> {
let prContext = ''
if (options.prContext) {
try {
prContext = await readFile(options.prContext, 'utf-8')
process.stdout.write(
`[${video.platformName}] Loaded PR context from ${options.prContext}\n`
)
} catch {
process.stdout.write(
`[${video.platformName}] Warning: Could not read PR context file ${options.prContext}\n`
)
}
}
const beforeVideoPath = options.beforeVideo
? resolve(options.beforeVideo)
: ''
if (beforeVideoPath) {
const beforeStat = await stat(beforeVideoPath)
process.stdout.write(
`[${video.platformName}] Before video: ${toProjectRelativePath(beforeVideoPath)} (${formatBytes(beforeStat.size)})\n`
)
}
process.stdout.write(
`[${video.platformName}] Sending ${beforeVideoPath ? '2 videos (comparative)' : 'video'} to ${options.model}\n`
)
const reviewText = await requestGeminiReview({
apiKey,
model: options.model,
platformName: video.platformName,
videoPath: video.videoPath,
beforeVideoPath,
timeoutMs: options.requestTimeoutMs,
prContext
})
const videoStat = await stat(video.videoPath)
const passSegment = options.passLabel ? `-${options.passLabel}` : ''
const outputPath = resolve(
options.outputDir,
`${video.platformName}${passSegment}-qa-video-report.md`
)
const reportInput: Parameters<typeof buildReportMarkdown>[0] = {
platformName: video.platformName,
model: options.model,
videoPath: video.videoPath,
videoSizeBytes: videoStat.size,
reviewText,
targetUrl: options.targetUrl || undefined
}
if (beforeVideoPath) {
const beforeStat = await stat(beforeVideoPath)
reportInput.beforeVideoPath = beforeVideoPath
reportInput.beforeVideoSizeBytes = beforeStat.size
}
const reportMarkdown = buildReportMarkdown(reportInput)
await mkdir(dirname(outputPath), { recursive: true })
await writeFile(outputPath, reportMarkdown, 'utf-8')
process.stdout.write(
`[${video.platformName}] Wrote ${toProjectRelativePath(outputPath)}\n`
)
}
function isExecutedAsScript(metaUrl: string): boolean {
const modulePath = fileURLToPath(metaUrl)
const scriptPath = process.argv[1] ? resolve(process.argv[1]) : ''
return modulePath === scriptPath
}
async function main(): Promise<void> {
const options = parseCliOptions(process.argv.slice(2))
const candidates = await collectVideoCandidates(options.artifactsDir)
if (candidates.length === 0) {
process.stdout.write(
`No qa-session.mp4 files found under ${toProjectRelativePath(resolve(options.artifactsDir))}\n`
)
return
}
const selectedVideo = selectVideoCandidateByFile(candidates, {
artifactsDir: options.artifactsDir,
videoFile: options.videoFile
})
process.stdout.write(
`Selected ${selectedVideo.platformName}: ${toProjectRelativePath(selectedVideo.videoPath)}\n`
)
if (options.dryRun) {
process.stdout.write('\nDry run mode enabled, no API calls were made.\n')
return
}
const apiKey = process.env.GEMINI_API_KEY
if (!apiKey) {
throw new Error('GEMINI_API_KEY is required unless --dry-run is set')
}
await reviewVideo(selectedVideo, options, apiKey)
}
if (isExecutedAsScript(import.meta.url)) {
void main().catch((error: unknown) => {
const message = errorToString(error)
process.stderr.write(`qa-video-review failed: ${message}\n`)
process.exit(1)
})
}

View File

@@ -0,0 +1,523 @@
#!/usr/bin/env tsx
/**
* QA CLI — simplified entry point for local & CI QA runs
*
* Usage:
* pnpm qa 10253 # auto-detects issue vs PR
* pnpm qa https://github.com/.../pull/10270
* pnpm qa 10270 -t base # test PR base (reproduce bug)
* pnpm qa 10270 -t both # test base + head
* pnpm qa --uncommitted # test local uncommitted changes
*
* Automatically loads .env.local / .env for GEMINI_API_KEY, ANTHROPIC_API_KEY.
* Results are written to .comfy-qa/<number>/ by default.
*/
import { parseArgs } from 'node:util'
import { config } from 'dotenv'
import { existsSync, mkdirSync, writeFileSync } from 'fs'
import { dirname, resolve } from 'path'
import { execSync, spawn, spawnSync } from 'child_process'
import { fileURLToPath } from 'url'
// ── Constants ──
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url))
const RECORD_SCRIPT = resolve(SCRIPT_DIR, 'qa-record.ts')
const DEFAULT_REPO = 'Comfy-Org/ComfyUI_frontend'
const VALID_TARGETS = ['head', 'base', 'both'] as const
const CLOUD_FALLBACK_URL = 'https://testcloud.comfy.org/'
type PrTarget = (typeof VALID_TARGETS)[number]
type TargetType = 'issue' | 'pr'
// ── Load .env.local / .env ──
for (const f of ['.env.local', '.env']) {
if (existsSync(f)) {
config({ path: f })
break
}
}
// ── Parse CLI ──
const { values, positionals } = tryParseArgs()
if (values.help) {
printUsage()
process.exit(0)
}
const serverUrl =
values.url || process.env.DEV_SERVER_COMFYUI_URL || 'http://127.0.0.1:8188'
const prTarget = values.target as PrTarget
if (!VALID_TARGETS.includes(prTarget)) {
console.error(
`Invalid --target "${prTarget}". Must be one of: ${VALID_TARGETS.join(', ')}`
)
process.exit(1)
}
// ── Ensure server is reachable (may fall back to cloud) ──
const resolvedServerUrl = await ensureServer(serverUrl)
// ── Dispatch by mode ──
if (values.uncommitted) {
runUncommitted()
} else {
const input = positionals[0]
if (!input) {
printUsage()
process.exit(1)
}
runTarget(input)
}
// ── Mode: uncommitted changes ──
function runUncommitted(): never {
const diff = shell('git diff && git diff --staged')
if (!diff.trim()) {
console.error('No uncommitted changes found')
process.exit(1)
}
const outputDir = resolveOutputDir('.comfy-qa/local')
const diffFile = writeTmpFile(outputDir, 'uncommitted.diff', diff)
logHeader({ label: 'uncommitted changes', outputDir })
const code = runQaRecord('after', diffFile, outputDir)
exit(code, outputDir)
}
// ── Mode: issue or PR by number/URL ──
function runTarget(input: string): never {
const { targetType, number, repo } = resolveTarget(input)
const outputDir = resolveOutputDir(`.comfy-qa/${number}`)
logHeader({
label: `${targetType} #${number} (${repo})`,
outputDir,
extra: targetType === 'pr' ? `Target: ${prTarget}` : undefined
})
const diffFile =
targetType === 'issue'
? fetchIssue(number, repo, outputDir)
: fetchPR(number, repo, outputDir)
let exitCode: number
if (targetType === 'issue') {
exitCode = runQaRecord('reproduce', diffFile, outputDir)
} else if (prTarget === 'both') {
exitCode = runPrBoth(diffFile, outputDir)
} else if (prTarget === 'base') {
exitCode = runQaRecord('before', diffFile, outputDir)
} else {
exitCode = runQaRecord('after', diffFile, outputDir)
}
exit(exitCode, outputDir)
}
// ── PR both phases ──
function runPrBoth(diffFile: string, outputDir: string): number {
console.warn('\n=== Phase 1: Reproduce bug on base ===')
const baseDir = resolve(outputDir, 'base')
mkdirSync(baseDir, { recursive: true })
const baseCode = runQaRecord('before', diffFile, baseDir)
if (baseCode !== 0) {
console.warn('Base phase failed, continuing to head...')
}
console.warn('\n=== Phase 2: Demonstrate fix on head ===')
const headDir = resolve(outputDir, 'head')
mkdirSync(headDir, { recursive: true })
return runQaRecord('after', diffFile, headDir)
}
// ── Target resolution ──
function resolveTarget(input: string): {
targetType: TargetType
number: string
repo: string
} {
const urlMatch = input.match(
/github\.com\/([^/]+\/[^/]+)\/(issues|pull)\/(\d+)/
)
if (urlMatch) {
return {
repo: urlMatch[1],
targetType: urlMatch[2] === 'pull' ? 'pr' : 'issue',
number: urlMatch[3]
}
}
if (/^\d+$/.test(input)) {
return {
repo: DEFAULT_REPO,
targetType: detectType(input, DEFAULT_REPO),
number: input
}
}
console.error(`Cannot parse target: ${input}`)
console.error('Expected a GitHub URL or issue/PR number')
printUsage()
process.exit(1)
}
function detectType(number: string, repo: string): TargetType {
try {
const result = execSync(
`gh api repos/${repo}/issues/${number} --jq 'has("pull_request")'`,
{ encoding: 'utf-8', timeout: 15000, stdio: ['pipe', 'pipe', 'pipe'] }
)
return result.trim() === 'true' ? 'pr' : 'issue'
} catch {
return 'issue'
}
}
// ── Data fetching ──
function fetchIssue(number: string, repo: string, outputDir: string): string {
console.warn(`Fetching issue #${number}...`)
const body = shell(
`gh issue view ${number} --repo ${repo} --json title,body,labels --jq '"Title: " + .title + "\\n\\nLabels: " + ([.labels[].name] | join(", ")) + "\\n\\n" + .body'`
)
// Append relevant comments for reproduction context
let comments = ''
try {
comments = shell(
`gh issue view ${number} --repo ${repo} --comments --json comments --jq '[.comments[] | select(.body | test("repro|step|how to|workaround"; "i")) | .body] | first(5; .[]) // empty'`
)
} catch {
// comments fetch failed, not critical
}
const content = comments ? `${body}\n\n--- Comments ---\n\n${comments}` : body
return writeTmpFile(outputDir, `issue-${number}.txt`, content)
}
function fetchPR(number: string, repo: string, outputDir: string): string {
console.warn(`Fetching PR #${number}...`)
const prJson = shell(
`gh pr view ${number} --repo ${repo} --json title,body,baseRefName,headRefName,baseRefOid,headRefOid`
)
const pr = JSON.parse(prJson) as {
title: string
body: string
baseRefName: string
headRefName: string
baseRefOid: string
headRefOid: string
}
console.warn(` Base: ${pr.baseRefName} (${pr.baseRefOid.slice(0, 8)})`)
console.warn(` Head: ${pr.headRefName} (${pr.headRefOid.slice(0, 8)})`)
let diff = ''
try {
diff = shell(`gh pr diff ${number} --repo ${repo}`)
} catch {
console.warn('Could not fetch PR diff')
}
writeTmpFile(
outputDir,
'refs.json',
JSON.stringify(
{
base: { ref: pr.baseRefName, sha: pr.baseRefOid },
head: { ref: pr.headRefName, sha: pr.headRefOid }
},
null,
2
)
)
return writeTmpFile(
outputDir,
`pr-${number}.txt`,
`Title: ${pr.title}\n\n${pr.body}\n\n--- DIFF ---\n\n${diff}`
)
}
// ── QA record runner ──
function runQaRecord(
mode: string,
diffFile: string,
outputDir: string
): number {
console.warn(`\nStarting QA ${mode} mode...\n`)
const r = spawnSync(
'pnpm',
[
'exec',
'tsx',
RECORD_SCRIPT,
'--mode',
mode,
'--diff',
diffFile,
'--output-dir',
outputDir,
'--url',
resolvedServerUrl
],
{ stdio: 'inherit', env: process.env }
)
return r.status ?? 1
}
// ── Server management ──
async function ensureServer(url: string): Promise<string> {
if (await isReachable(url)) {
console.warn(`Server OK: ${url}`)
return url
}
console.warn(`Server not reachable at ${url}, attempting auto-start...`)
const port = new URL(url).port || '8188'
// Strategy 1: comfy-cli (pip install comfy-cli)
try {
execSync('which comfy', { stdio: 'pipe' })
console.warn('Starting ComfyUI via comfy-cli...')
const proc = spawn(
'comfy',
['launch', '--background', '--', '--cpu', '--port', port],
{
stdio: 'ignore',
detached: true
}
)
proc.unref()
await waitForServer(url, 120000)
return url
} catch {
// comfy-cli not available
}
// Strategy 2: python main.py from TEST_COMFYUI_DIR or .comfy-qa/ComfyUI
const comfyDir = findComfyUIDir()
if (comfyDir) {
console.warn(`Starting ComfyUI from ${comfyDir}...`)
const proc = spawn('python', ['main.py', '--cpu', '--port', port], {
cwd: comfyDir,
stdio: 'ignore',
detached: true
})
proc.unref()
await waitForServer(url, 120000)
return url
}
// Strategy 3: clone ComfyUI and start
const cloneDir = resolve('.comfy-qa/ComfyUI')
if (!existsSync(resolve(cloneDir, 'main.py'))) {
console.warn('No ComfyUI installation found, cloning...')
try {
execSync(
`git clone --depth 1 https://github.com/comfyanonymous/ComfyUI.git "${cloneDir}"`,
{ stdio: 'inherit', timeout: 120000 }
)
console.warn('Installing ComfyUI dependencies...')
execSync('pip install -r requirements.txt', {
cwd: cloneDir,
stdio: 'inherit',
timeout: 300000
})
} catch (err) {
console.warn(
`Clone/install failed: ${err instanceof Error ? err.message : err}`
)
}
}
if (existsSync(resolve(cloneDir, 'main.py'))) {
console.warn(`Starting ComfyUI from ${cloneDir}...`)
const proc = spawn('python', ['main.py', '--cpu', '--port', port], {
cwd: cloneDir,
stdio: 'ignore',
detached: true
})
proc.unref()
await waitForServer(url, 120000)
return url
}
// Strategy 4: fallback to testcloud
console.warn(`Local server failed. Falling back to ${CLOUD_FALLBACK_URL}`)
if (await isReachable(CLOUD_FALLBACK_URL)) {
console.warn(`Cloud server OK: ${CLOUD_FALLBACK_URL}`)
return CLOUD_FALLBACK_URL
}
console.error(`
No ComfyUI server available. Tried:
1. ${url} (not reachable)
2. comfy-cli (not installed)
3. Local ComfyUI installation (not found)
4. ${CLOUD_FALLBACK_URL} (not reachable)
Install: pip install comfy-cli && comfy install && comfy launch --cpu
`)
process.exit(1)
}
function findComfyUIDir(): string | undefined {
const candidates = [
process.env.TEST_COMFYUI_DIR,
resolve('.comfy-qa/ComfyUI'),
'/home/ComfyUI'
]
return candidates.find((d) => d && existsSync(resolve(d, 'main.py')))
}
async function isReachable(url: string): Promise<boolean> {
try {
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 5000)
const res = await fetch(url, { signal: controller.signal })
clearTimeout(timeout)
return res.ok || res.status === 200 || res.status === 304
} catch {
return false
}
}
async function waitForServer(url: string, timeoutMs: number): Promise<void> {
const start = Date.now()
while (Date.now() - start < timeoutMs) {
if (await isReachable(url)) {
console.warn('Server is ready')
return
}
await new Promise((r) => setTimeout(r, 2000))
}
console.error(`Server did not start within ${timeoutMs / 1000}s`)
process.exit(1)
}
// ── Utilities ──
function shell(cmd: string): string {
return execSync(cmd, { encoding: 'utf-8', timeout: 30000 })
}
function writeTmpFile(
outputDir: string,
filename: string,
content: string
): string {
const tmpDir = resolve(outputDir, '.tmp')
mkdirSync(tmpDir, { recursive: true })
const filePath = resolve(tmpDir, filename)
writeFileSync(filePath, content)
return filePath
}
function resolveOutputDir(defaultPath: string): string {
const dir = values.output ? resolve(values.output) : resolve(defaultPath)
mkdirSync(dir, { recursive: true })
return dir
}
function logHeader(opts: { label: string; outputDir: string; extra?: string }) {
console.warn(`QA target: ${opts.label}`)
console.warn(`Output: ${opts.outputDir}`)
console.warn(`Server: ${resolvedServerUrl}`)
if (values.ref) console.warn(`Ref: ${values.ref}`)
if (opts.extra) console.warn(opts.extra)
}
function exit(code: number, outputDir: string): never {
console.warn('\n=== QA Complete ===')
console.warn(`Results: ${outputDir}`)
try {
console.warn(shell(`ls -la "${outputDir}"`))
} catch {
// not critical
}
process.exit(code)
}
function tryParseArgs() {
try {
const parsed = parseArgs({
args: process.argv.slice(2),
options: {
target: { type: 'string', short: 't', default: 'head' },
uncommitted: { type: 'boolean', default: false },
url: { type: 'string', default: '' },
ref: { type: 'string', default: '' },
output: { type: 'string', short: 'o', default: '' },
help: { type: 'boolean', short: 'h', default: false }
},
allowPositionals: true,
strict: true
})
return {
values: parsed.values as {
target: string
uncommitted: boolean
url: string
ref: string
output: string
help: boolean
},
positionals: parsed.positionals
}
} catch (err) {
console.error(`Error: ${err instanceof Error ? err.message : err}\n`)
printUsage()
process.exit(1)
}
}
function printUsage() {
console.warn(`
QA CLI — Reproduce issues & test PRs for ComfyUI frontend
Usage:
pnpm qa <number|url> [options]
pnpm qa --uncommitted
Targets:
10253 Number (auto-detects issue vs PR via gh CLI)
https://github.com/Comfy-Org/ComfyUI_frontend/issues/10253
https://github.com/Comfy-Org/ComfyUI_frontend/pull/10270
Options:
-t, --target <head|base|both>
For PRs: which ref to test (default: head)
head — test the fix (PR head)
base — reproduce the bug (PR base)
both — base then head
--uncommitted Test local uncommitted changes
--url <url> ComfyUI server URL (default: from .env or http://127.0.0.1:8188)
--ref <ref> Git ref to test against
-o, --output <dir> Override output directory (default: .comfy-qa/<number>)
-h, --help Show this help
Environment (auto-loaded from .env.local or .env):
GEMINI_API_KEY Required — used for PR analysis, video review, TTS
ANTHROPIC_API_KEY Optional locally — Claude Agent SDK auto-detects Claude Code session
Examples:
pnpm qa 10253 # reproduce an issue
pnpm qa 10270 # test PR head (the fix)
pnpm qa 10270 -t base # reproduce bug on PR base
pnpm qa 10270 -t both # test base + head
pnpm qa --uncommitted # test local changes
`)
}

View File

@@ -0,0 +1,361 @@
---
name: ticket-intake
description: 'Parse ticket URL (Notion or GitHub), extract all data, initialize pipeline run. Use when starting work on a new ticket or when asked to pick up a ticket.'
---
# Ticket Intake
Parses a ticket URL from supported sources (Notion or GitHub), extracts all relevant information, and creates a ticket in the pipeline API.
> **🚨 CRITICAL REQUIREMENT**: This skill MUST register the ticket in the Pipeline API and update the source (Notion/GitHub). If these steps are skipped, the entire pipeline breaks. See [Mandatory API Calls](#mandatory-api-calls-execute-all-three) below.
## Supported Sources
| Source | URL Pattern | Provider File |
| ------ | --------------------------------------------------- | --------------------- |
| Notion | `https://notion.so/...` `https://www.notion.so/...` | `providers/notion.md` |
| GitHub | `https://github.com/{owner}/{repo}/issues/{n}` | `providers/github.md` |
## Quick Start
When given a ticket URL:
1. **Detect source type** from URL pattern
2. **Load provider-specific logic** from `providers/` directory
3. Fetch ticket content via appropriate API
4. Extract and normalize properties to common schema
5. **Register ticket in pipeline API** ← MANDATORY
6. **Update source** (Notion status / GitHub comment) ← MANDATORY
7. **Run verification script** to confirm API registration
8. Output summary and handoff to `research-orchestrator`
## Configuration
Uses the **production API** by default. No configuration needed for read operations.
**Defaults (no setup required):**
- API URL: `https://api-gateway-856475788601.us-central1.run.app`
- Read-only endpoints at `/public/*` require no authentication
**For write operations** (transitions, creating tickets), set:
```bash
export PIPELINE_API_KEY="..." # Get from GCP Secret Manager or ask admin
```
**Optional (for local working artifacts):**
```bash
PIPELINE_DIR="${PIPELINE_DIR:-$HOME/repos/ticket-to-pr-pipeline}"
```
## Mandatory API Calls (Execute ALL Three)
**⚠️ These three API calls are the ENTIRE POINT of this skill. Without them, the ticket is invisible to the pipeline, downstream skills will fail, and Notion status won't update.**
**You MUST make these HTTP requests.** Use `curl` from bash — do not just read this as documentation.
### Call 1: Create Ticket
```bash
API_URL="${PIPELINE_API_URL:-https://api-gateway-856475788601.us-central1.run.app}"
API_KEY="${PIPELINE_API_KEY}"
curl -s -X POST "${API_URL}/v1/tickets" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
-H "X-Agent-ID: ${AGENT_ID:-amp-agent}" \
-d '{
"notion_page_id": "NOTION_PAGE_UUID_HERE",
"title": "TICKET_TITLE_HERE",
"source": "notion",
"metadata": {
"description": "DESCRIPTION_HERE",
"priority": "High",
"labels": [],
"acceptanceCriteria": []
}
}'
```
Save the returned `id` — you need it for the next two calls.
### Call 2: Transition to RESEARCH
```bash
TICKET_ID="id-from-step-1"
curl -s -X POST "${API_URL}/v1/tickets/${TICKET_ID}/transition" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
-H "X-Agent-ID: ${AGENT_ID:-amp-agent}" \
-d '{
"to_state": "RESEARCH",
"reason": "Intake complete, starting research"
}'
```
### Call 3: Queue Source Update
```bash
curl -s -X POST "${API_URL}/v1/sync/queue" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Content-Type: application/json" \
-H "X-Agent-ID: ${AGENT_ID:-amp-agent}" \
-d '{
"ticket_id": "TICKET_ID_HERE",
"action": "update_status",
"payload": { "status": "In Progress" },
"priority": "normal"
}'
```
> **Note:** The action MUST be `"update_status"` (not `"UPDATE_NOTION_STATUS"`). Valid actions: `update_status`, `update_pr_url`, `mark_done`.
### TypeScript Equivalent (if using pipeline client)
```typescript
import { PipelineClient } from '@pipeline/client'
const client = new PipelineClient({
apiUrl:
process.env.PIPELINE_API_URL ||
'https://api-gateway-856475788601.us-central1.run.app',
agentId: process.env.AGENT_ID!
})
const ticket = await client.createTicket({
notion_page_id: pageId,
title: ticketTitle,
source: 'notion',
metadata: { description, priority, labels, acceptanceCriteria }
})
await client.transitionState(
ticket.id,
'RESEARCH',
'Intake complete, starting research'
)
await client.queueSync(ticket.id, 'update_status', { status: 'In Progress' })
```
## Workflow
### Step 1: Detect Source Type
Parse the URL to determine source:
```javascript
if (url.includes('notion.so')) {
source = 'notion'
// Load providers/notion.md
} else if (url.match(/github\.com\/[^\/]+\/[^\/]+\/issues\/\d+/)) {
source = 'github'
// Load providers/github.md
} else {
// Error: Unsupported URL format
}
```
### Step 2: Load Provider and Fetch Data
Read the appropriate provider file for source-specific instructions:
- **Notion**: `providers/notion.md` - Uses Notion MCP, handles Slack links
- **GitHub**: `providers/github.md` - Uses `gh` CLI, handles Dosu comments
Follow the provider's instructions for:
- Fetching content
- Extracting properties
- **Updating the source** (Notion status → "In Progress", Assignee → pipeline owner)
### Step 3: Normalize to Common Schema
All providers must extract normalized ticket data following `schema.md`:
```json
{
"id": "abc12345",
"url": "https://...",
"source": "notion | github",
"title": "Ticket title",
"description": "Full description",
"status": "Not Started",
"assignee": "username",
"priority": "High",
"area": "UI",
"labels": ["bug", "frontend"],
"acceptanceCriteria": ["Criterion 1", "Criterion 2"],
"fetchedAt": "2024-01-15T10:30:00Z"
}
```
### Step 4: Register Ticket in Pipeline API (MANDATORY — DO NOT SKIP)
**Execute all three API calls from [Mandatory API Calls](#mandatory-api-calls-execute-all-three) above.**
This is not optional. This is not documentation. You MUST make these HTTP requests right now.
1. `createTicket()` → save the returned ticket ID
2. `transitionState(id, 'RESEARCH')` → confirm state changed
3. `queueSync(id, 'update_status', { status: 'In Progress' })` → confirm queued
**If any call fails**, retry once. If it still fails, report the error prominently — do NOT silently continue.
### Step 5: Run Verification Script
After making the API calls, run the verification script to confirm everything worked:
```bash
bash scripts/verify-intake.sh TICKET_ID_OR_NOTION_PAGE_ID
```
**If the script is not available locally**, verify manually via the public API:
```bash
curl -s "${API_URL}/public/tickets/${TICKET_ID}" | jq '{id, state, title, notion_page_id}'
```
Expected output:
```json
{
"id": "...",
"state": "RESEARCH",
"title": "...",
"notion_page_id": "..."
}
```
**If `state` is not `RESEARCH`, go back to Step 4 and complete the missing calls.**
### Step 6: Output Summary and Handoff
Print a clear summary:
```markdown
## Ticket Intake Complete
**Source:** Notion | GitHub
**Title:** [Ticket title]
**ID:** abc12345
**Status:** In Progress (queued)
**Priority:** High
**Area:** UI
### Description
[Brief description or first 200 chars]
### Acceptance Criteria
- [ ] Criterion 1
- [ ] Criterion 2
### Links
- **Ticket:** [Original URL]
- **Slack:** [Slack thread content fetched via slackdump] (Notion only)
### Pipeline
- **API Ticket ID:** abc12345
- **State:** RESEARCH
- **Verified:** ✅ (via verify-intake.sh or public API)
```
**After printing the summary, immediately handoff** to continue the pipeline. Use the `handoff` tool with all necessary context (ticket ID, source, title, description, slack context if any):
> **Handoff goal:** "Continue pipeline for ticket {ID} ({title}). Ticket is in RESEARCH state. Load skill: `research-orchestrator` to begin research phase. Ticket data: source={source}, notion_page_id={pageId}, priority={priority}. {slack context summary if available}"
**Do NOT wait for human approval to proceed.** The intake phase is complete — handoff immediately.
## Error Handling
### Unsupported URL
```
❌ Unsupported ticket URL format.
Supported formats:
- Notion: https://notion.so/... or https://www.notion.so/...
- GitHub: https://github.com/{owner}/{repo}/issues/{number}
Received: [provided URL]
```
### Provider-Specific Errors
See individual provider files for source-specific error handling:
- `providers/notion.md` - Authentication, page not found
- `providers/github.md` - Auth, rate limits, issue not found
### Missing Properties
Continue with available data and note what's missing:
```
⚠️ Some properties unavailable:
- Priority: not found (using default: Medium)
- Area: not found
Proceeding with available data...
```
### API Call Failures
```
❌ Pipeline API call failed: {method} {endpoint}
Status: {status}
Error: {message}
Retrying once...
❌ Retry also failed. INTAKE IS INCOMPLETE.
The ticket was NOT registered in the pipeline.
Downstream skills will not work until this is fixed.
```
## Notes
- This skill focuses ONLY on intake — it does not do research
- Slack thread content is fetched automatically via the `slackdump` skill — no manual copy-paste needed
- ALL API calls (createTicket, transitionState, queueSync) are MANDATORY — never skip them
- The `queueSync` action must be `"update_status"`, NOT `"UPDATE_NOTION_STATUS"`
- Pipeline state is tracked via the API, not local files
- Working artifacts (research-report.md, plan.md) can be saved locally to `$PIPELINE_DIR/runs/{ticket-id}/`
- The `source` field in the ticket determines which research strategies to use
## API Client Reference
### Available Methods
| Method | Description |
| ----------------------------------------------------------- | ------------------------------------------------------------------- |
| `createTicket({ notion_page_id, title, source, metadata })` | Create a new ticket in the API |
| `getTicket(id)` | Retrieve a ticket by ID |
| `findByNotionId(notionPageId)` | Look up a ticket by its Notion page ID |
| `listTickets({ state, agent_id, limit, offset })` | List tickets with optional filters |
| `transitionState(id, state, reason)` | Move ticket to a new state (e.g., `'RESEARCH'`) |
| `setPRCreated(id, prUrl)` | Mark ticket as having a PR created |
| `queueSync(id, action, payload)` | Queue a sync action (`update_status`, `update_pr_url`, `mark_done`) |
| `registerBranch(id, branch, repo)` | Register working branch for automatic PR detection |
### Error Handling
```typescript
import { PipelineClient, PipelineAPIError } from '@pipeline/client';
try {
await client.createTicket({ ... });
} catch (error) {
if (error instanceof PipelineAPIError) {
console.error(`API Error ${error.status}: ${error.message}`);
}
throw error;
}
```

View File

@@ -0,0 +1,194 @@
# GitHub Provider - Ticket Intake
Provider-specific logic for ingesting tickets from GitHub Issues.
## URL Pattern
```
https://github.com/{owner}/{repo}/issues/{number}
https://www.github.com/{owner}/{repo}/issues/{number}
```
Extract: `owner`, `repo`, `issue_number` from URL.
## Prerequisites
- `gh` CLI authenticated (`gh auth status`)
- Access to the repository
## Fetch Issue Content
Use `gh` CLI to fetch issue details:
```bash
# Get issue details in JSON
gh issue view {number} --repo {owner}/{repo} --json title,body,state,labels,assignees,milestone,author,createdAt,comments,linkedPRs
# Get comments separately if needed
gh issue view {number} --repo {owner}/{repo} --comments
```
## Extract Ticket Data
Map GitHub issue fields to normalized ticket data (stored via API):
| GitHub Field | ticket.json Field | Notes |
| ------------ | ----------------- | -------------------------- |
| title | title | Direct mapping |
| body | description | Issue body/description |
| state | status | Map: open → "Not Started" |
| labels | labels | Array of label names |
| assignees | assignee | First assignee login |
| author | author | Issue author login |
| milestone | milestone | Milestone title if present |
| comments | comments | Array of comment objects |
| linkedPRs | linkedPRs | PRs linked to this issue |
### Priority Mapping
Infer priority from labels:
- `priority:critical`, `P0` → "Critical"
- `priority:high`, `P1` → "High"
- `priority:medium`, `P2` → "Medium"
- `priority:low`, `P3` → "Low"
- No priority label → "Medium" (default)
### Area Mapping
Infer area from labels:
- `area:ui`, `frontend`, `component:*` → "UI"
- `area:api`, `backend` → "API"
- `area:docs`, `documentation` → "Docs"
- `bug`, `fix` → "Bug"
- `enhancement`, `feature` → "Feature"
## Update Source
**For GitHub issues, update is optional but recommended.**
Add a comment to indicate work has started:
```bash
gh issue comment {number} --repo {owner}/{repo} --body "🤖 Pipeline started processing this issue."
```
Optionally assign to self:
```bash
gh issue edit {number} --repo {owner}/{repo} --add-assignee @me
```
Log any updates via the Pipeline API:
```typescript
await client.updateTicket(ticketId, {
metadata: {
...ticket.metadata,
githubWrites: [
...(ticket.metadata?.githubWrites || []),
{
action: 'comment',
issueNumber: 123,
at: new Date().toISOString(),
skill: 'ticket-intake',
success: true
}
]
}
})
```
## GitHub-Specific Ticket Fields
Store via API using `client.createTicket()`:
```json
{
"source": "github",
"githubOwner": "Comfy-Org",
"githubRepo": "ComfyUI_frontend",
"githubIssueNumber": 123,
"githubIssueUrl": "https://github.com/Comfy-Org/ComfyUI_frontend/issues/123",
"labels": ["bug", "area:ui", "priority:high"],
"linkedPRs": [456, 789],
"dosuComment": "..." // Extracted Dosu bot analysis if present
}
```
## Dosu Bot Detection
Many repositories use Dosu bot for automated issue analysis. Check comments for Dosu:
```bash
gh issue view {number} --repo {owner}/{repo} --comments | grep -A 100 "dosu"
```
Look for comments from:
- `dosu[bot]`
- `dosu-bot`
Extract Dosu analysis which typically includes:
- Root cause analysis
- Suggested files to modify
- Related issues/PRs
- Potential solutions
Store in ticket data via API:
```json
{
"dosuComment": {
"found": true,
"analysis": "...",
"suggestedFiles": ["src/file1.ts", "src/file2.ts"],
"relatedIssues": [100, 101]
}
}
```
## Extract Linked Issues/PRs
Parse issue body and comments for references:
- `#123` → Issue or PR reference
- `fixes #123`, `closes #123` → Linked issue
- `https://github.com/.../issues/123` → Full URL reference
Store in ticket data via API for research phase:
```json
{
"referencedIssues": [100, 101, 102],
"referencedPRs": [200, 201]
}
```
## Error Handling
### Authentication Error
```
⚠️ GitHub CLI not authenticated.
Run: gh auth login
```
### Issue Not Found
```
❌ GitHub issue not found or inaccessible.
- Check the URL is correct
- Ensure you have access to this repository
- Run: gh auth status
```
### Rate Limiting
```
⚠️ GitHub API rate limited.
Wait a few minutes and try again.
Check status: gh api rate_limit
```

View File

@@ -0,0 +1,202 @@
# Notion Provider - Ticket Intake
Provider-specific logic for ingesting tickets from Notion.
## URL Pattern
```
https://www.notion.so/workspace/Page-Title-abc123def456...
https://notion.so/Page-Title-abc123def456...
https://www.notion.so/abc123def456...
```
Page ID is the 32-character hex string (with or without hyphens).
## Prerequisites
- Notion MCP connected and authenticated
- If not setup: `claude mcp add --transport http notion https://mcp.notion.com/mcp`
- Authenticate via `/mcp` command if prompted
## Fetch Ticket Content
Use `Notion:notion-fetch` with the page URL or ID:
```
Fetch the full page content including all properties
```
## Extract Ticket Data
Extract these properties (names may vary):
| Property | Expected Name | Type |
| ------------- | ------------------------- | ------------ |
| Title | Name / Title | Title |
| Status | Status | Select |
| Assignee | Assignee / Assigned To | Person |
| Description | - | Page content |
| Slack Link | Slack Link / Slack Thread | URL |
| GitHub PR | GitHub PR / PR Link | URL |
| Priority | Priority | Select |
| Area | Area / Category | Select |
| Related Tasks | Related Tasks | Relation |
**If properties are missing**: Note what's unavailable and continue with available data.
## Update Source (REQUIRED)
**⚠️ DO NOT SKIP THIS STEP. This is a required action, not optional.**
**⚠️ Notion Write Safety rules apply (see `$PIPELINE_DIR/docs/notion-write-safety.md` for full reference):**
- **Whitelist**: Only `Status`, `GitHub PR`, and `Assignee` fields may be written
- **Valid transitions**: Not Started → In Progress, In Progress → In Review, In Review → Done
- **Logging**: Every write attempt MUST be logged with timestamp, field, value, previous value, skill name, and success status
Use `Notion:notion-update-page` to update the ticket:
1. **Status**: Set to "In Progress" (only valid from "Not Started")
2. **Assignee**: Assign to pipeline owner (Notion ID: `175d872b-594c-81d4-ba5a-0002911c5966`)
```json
{
"page_id": "{page_id_from_ticket}",
"command": "update_properties",
"properties": {
"Status": "In Progress",
"Assignee": "175d872b-594c-81d4-ba5a-0002911c5966"
}
}
```
**After the update succeeds**, log the write via the Pipeline API:
```typescript
await client.updateTicket(ticketId, {
metadata: {
...ticket.metadata,
notionWrites: [
...(ticket.metadata?.notionWrites || []),
{
field: 'Status',
value: 'In Progress',
previousValue: 'Not Started',
at: new Date().toISOString(),
skill: 'ticket-intake',
success: true
}
]
}
})
```
If update fails, log with `success: false` and continue.
## Notion-Specific Ticket Fields
Store via API using `client.createTicket()`:
```json
{
"source": "notion",
"notionPageId": "abc123def456...",
"slackLink": "https://slack.com/...",
"relatedTasks": ["page-id-1", "page-id-2"]
}
```
## Slack Thread Handling
If a Slack link exists, use the `slackdump` skill to fetch the thread content programmatically.
### Slack URL Conversion
Notion stores Slack links in `slackMessage://` format:
```
slackMessage://comfy-organization.slack.com/CHANNEL_ID/THREAD_TS/MESSAGE_TS
```
Convert to browser-clickable format:
```
https://comfy-organization.slack.com/archives/CHANNEL_ID/pMESSAGE_TS_NO_DOT
```
**Example:**
- Input: `slackMessage://comfy-organization.slack.com/C075ANWQ8KS/1766022478.450909/1764772881.854829`
- Output: `https://comfy-organization.slack.com/archives/C075ANWQ8KS/p1764772881854829`
(Remove the dot from the last timestamp and prefix with `p`)
### Fetching Thread Content
Load the `slackdump` skill and use the **export-thread** workflow:
```bash
# Export thread by URL
slackdump dump "https://comfy-organization.slack.com/archives/CHANNEL_ID/pMESSAGE_TS"
# Or by colon notation (channel_id:thread_ts)
slackdump dump CHANNEL_ID:THREAD_TS
```
Save the thread content to `$RUN_DIR/slack-context.md` and include it in the ticket metadata.
> **No manual action required.** The slackdump CLI handles authentication via stored credentials at `~/.cache/slackdump/comfy-organization.bin`.
## Database Reference: Comfy Tasks
The "Comfy Tasks" database has these properties (verify via `notion-search`):
- **Status values**: Not Started, In Progress, In Review, Done
- **Team assignment**: "Frontend Team" for unassigned tickets
- **Filtering note**: Team filtering in Notion may have quirks - handle gracefully
### Pipeline Owner Details
When assigning tickets, use these identifiers:
| Platform | Identifier |
| --------------- | -------------------------------------- |
| Notion User ID | `175d872b-594c-81d4-ba5a-0002911c5966` |
| Notion Name | Christian Byrne |
| Notion Email | cbyrne@comfy.org |
| Slack User ID | U087MJCDHHC |
| GitHub Username | christian-byrne |
**To update Assignee**, use the Notion User ID (not name):
```
properties: {"Assignee": "175d872b-594c-81d4-ba5a-0002911c5966"}
```
### Finding Active Tickets
To list your active tickets:
```
Use Notion:notion-search for "Comfy Tasks"
Filter by Assignee = current user OR Team = "Frontend Team"
```
## Error Handling
### Authentication Error
```
⚠️ Notion authentication required.
Run: claude mcp add --transport http notion https://mcp.notion.com/mcp
Then authenticate via /mcp command.
```
### Page Not Found
```
❌ Notion page not found or inaccessible.
- Check the URL is correct
- Ensure you have access to this page
- Try re-authenticating via /mcp
```

View File

@@ -0,0 +1,81 @@
# Ticket Schema
Common schema for normalized ticket data across all sources. This data is stored and retrieved via the Pipeline API, not local files.
## Ticket Data Schema
```json
{
// Required fields (all sources)
"id": "string", // Unique identifier (short form)
"url": "string", // Original URL
"source": "notion | github", // Source type
"title": "string", // Ticket title
"description": "string", // Full description/body
"fetchedAt": "ISO8601", // When ticket was fetched
// Common optional fields
"status": "string", // Current status
"assignee": "string", // Assigned user
"priority": "string", // Priority level
"area": "string", // Category/area
"labels": ["string"], // Tags/labels
"acceptanceCriteria": ["string"] // List of AC items
// Source-specific fields (see providers)
// Notion: notionPageId, slackLink, relatedTasks, notionWrites
// GitHub: githubOwner, githubRepo, githubIssueNumber, linkedPRs, dosuComment, referencedIssues
}
```
## Ticket State Schema (via API)
State is managed via the Pipeline API using `client.transitionState()`:
```json
{
"ticketId": "string",
"state": "intake | research | planning | implementation | pr_created | done | failed",
"stateChangedAt": "ISO8601",
// Timestamps tracked by API
"createdAt": "ISO8601",
"updatedAt": "ISO8601"
}
```
## Priority Normalization
All sources should normalize to these values:
| Normalized | Description |
| ---------- | ------------------------- |
| Critical | Production down, security |
| High | Blocking work, urgent |
| Medium | Normal priority (default) |
| Low | Nice to have, backlog |
## Status Normalization
Pipeline tracks these statuses internally:
| Status | Description |
| -------------- | ---------------------------- |
| research | Gathering context |
| planning | Creating implementation plan |
| implementation | Writing code |
| review | Code review in progress |
| qa | Quality assurance |
| done | PR merged or completed |
## ID Generation
IDs are generated by the API when creating tickets. For reference:
- **Notion**: First 8 characters of page ID
- **GitHub**: `gh-{owner}-{repo}-{issue_number}` (sanitized)
Examples:
- Notion: `abc12345`
- GitHub: `gh-comfy-org-frontend-123`

View File

@@ -1,12 +0,0 @@
root = true
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[*.md]
trim_trailing_whitespace = false

View File

@@ -47,8 +47,10 @@ ALGOLIA_API_KEY=684d998c36b67a9a9fce8fc2d8860579
# SENTRY_PROJECT=cloud-frontend-staging
# SENTRY_PROJECT_PROD= # prod project slug for sourcemap uploads
# Ashby (apps/website careers page build).
# Server-only; read inside the Astro build context. Do NOT prefix with PUBLIC_.
# When unset, the committed snapshot at apps/website/src/data/ashby-roles.snapshot.json is used.
# WEBSITE_ASHBY_API_KEY=
# WEBSITE_ASHBY_JOB_BOARD_NAME=comfy-org
# ── QA Skill (scripts/qa-record.ts) ──
# Required for automated bug reproduction via `pnpm exec tsx scripts/qa-record.ts`
# GEMINI_API_KEY is required — used for PR analysis, video review, and TTS narration
GEMINI_API_KEY=
# ANTHROPIC_API_KEY is optional locally — Claude Agent SDK auto-detects Claude Code session
# Required in CI (set as GitHub Actions secret)
# ANTHROPIC_API_KEY=

View File

@@ -1,65 +0,0 @@
name: Find Workflow Run
description: Finds a workflow run for a given commit SHA and outputs its status and run ID.
inputs:
workflow-id:
description: The workflow filename (e.g., 'ci-size-data.yaml')
required: true
head-sha:
description: The commit SHA to find runs for
required: true
not-found-status:
description: Status to output when no run exists
required: false
default: pending
token:
description: GitHub token for API access
required: true
outputs:
status:
description: One of 'ready', 'pending', 'failed', or the not-found-status value
value: ${{ steps.find.outputs.status }}
run-id:
description: The workflow run ID (only set when status is 'ready')
value: ${{ steps.find.outputs.run-id }}
runs:
using: composite
steps:
- name: Find workflow run
id: find
uses: actions/github-script@v8
env:
WORKFLOW_ID: ${{ inputs.workflow-id }}
HEAD_SHA: ${{ inputs.head-sha }}
NOT_FOUND_STATUS: ${{ inputs.not-found-status }}
with:
github-token: ${{ inputs.token }}
script: |
const { data: runs } = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: process.env.WORKFLOW_ID,
head_sha: process.env.HEAD_SHA,
per_page: 1,
});
const run = runs.workflow_runs[0];
if (!run) {
core.setOutput('status', process.env.NOT_FOUND_STATUS);
return;
}
if (run.status !== 'completed') {
core.setOutput('status', 'pending');
return;
}
if (run.conclusion !== 'success') {
core.setOutput('status', 'failed');
return;
}
core.setOutput('status', 'ready');
core.setOutput('run-id', String(run.id));

View File

@@ -1,31 +0,0 @@
name: 'Lint and format verify'
description: >
Runs the lint/format/knip verification suite plus a conditional
browser-tests typecheck. Shared by ci-lint-format.yaml (PR) and
ci-lint-format-queue.yaml (merge queue) so both paths run the exact
same checks. The caller is responsible for checkout and frontend setup
before invoking this action.
runs:
using: composite
steps:
- name: Detect browser_tests changes
id: changed-paths
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
with:
filters: |
browser_tests:
- 'browser_tests/**'
- name: Verify lint and format
shell: bash
run: |
pnpm lint
pnpm stylelint
pnpm format:check
pnpm knip
- name: Typecheck browser tests
if: steps.changed-paths.outputs.browser_tests == 'true'
shell: bash
run: pnpm typecheck:browser

View File

@@ -1,88 +0,0 @@
name: Resolve PR from workflow_run
description: >
Resolves the PR number from a workflow_run event using pull_requests[0]
with a listPullRequestsAssociatedWithCommit fallback.
Skips closed/merged PRs and stale runs (head SHA mismatch).
inputs:
token:
description: GitHub token for API calls
required: false
default: ${{ github.token }}
outputs:
skip:
description: "'true' when no open PR was found or the run is stale"
value: ${{ steps.resolve.outputs.skip }}
number:
description: The PR number (empty when skip is true)
value: ${{ steps.resolve.outputs.number }}
base:
description: The PR base branch (empty when skip is true)
value: ${{ steps.resolve.outputs.base }}
head-sha:
description: The PR head SHA (empty when skip is true)
value: ${{ steps.resolve.outputs.head-sha }}
runs:
using: composite
steps:
- name: Resolve PR
id: resolve
uses: actions/github-script@v8
with:
github-token: ${{ inputs.token }}
script: |
let pr = context.payload.workflow_run.pull_requests?.[0];
if (!pr) {
const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
owner: context.repo.owner,
repo: context.repo.repo,
commit_sha: context.payload.workflow_run.head_sha,
});
pr = prs.find(p => p.state === 'open');
}
// Fork PRs: pull_requests is empty and commit SHA may not be in
// the base repo graph. Fall back to pulls.list with head filter.
if (!pr && context.payload.workflow_run.head_repository?.owner?.login) {
const { data: prs } = await github.rest.pulls.list({
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
head: `${context.payload.workflow_run.head_repository.owner.login}:${context.payload.workflow_run.head_branch}`,
per_page: 1,
});
pr = prs.find(p => p.head.sha === context.payload.workflow_run.head_sha);
}
if (!pr) {
core.info('No open PR found for this workflow run — skipping.');
core.setOutput('skip', 'true');
return;
}
const { data: livePr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pr.number,
});
if (livePr.state !== 'open') {
core.info(`PR #${pr.number} is ${livePr.state} — skipping.`);
core.setOutput('skip', 'true');
return;
}
if (livePr.head.sha !== context.payload.workflow_run.head_sha) {
core.info(
`Stale run: workflow SHA ${context.payload.workflow_run.head_sha} != PR head ${livePr.head.sha}`
);
core.setOutput('skip', 'true');
return;
}
core.setOutput('base', livePr.base.ref);
core.setOutput('head-sha', livePr.head.sha);
core.setOutput('skip', 'false');
core.setOutput('number', String(pr.number));

View File

@@ -1,79 +0,0 @@
name: Upsert Comment Section
description: >
Manage a consolidated PR comment with independently-updatable sections.
All website CI workflows share the marker <!-- WEBSITE_CI_REPORT -->.
Valid section names: "e2e", "preview", "screenshot-update".
inputs:
pr-number:
description: PR number to comment on
required: true
section-name:
description: 'Section identifier: "e2e", "preview", or "screenshot-update"'
required: true
section-content:
description: Markdown content for this section
required: true
comment-marker:
description: Top-level HTML comment marker (must be <!-- WEBSITE_CI_REPORT --> for all callers)
required: true
token:
description: GitHub token with pull-requests write permission
required: true
runs:
using: composite
steps:
- uses: actions/github-script@v8
env:
INPUT_PR_NUMBER: ${{ inputs.pr-number }}
INPUT_SECTION_NAME: ${{ inputs.section-name }}
INPUT_SECTION_CONTENT: ${{ inputs.section-content }}
INPUT_COMMENT_MARKER: ${{ inputs.comment-marker }}
with:
github-token: ${{ inputs.token }}
script: |
const prNumber = Number(process.env.INPUT_PR_NUMBER)
const sectionName = process.env.INPUT_SECTION_NAME
const sectionContent = process.env.INPUT_SECTION_CONTENT
const commentMarker = process.env.INPUT_COMMENT_MARKER
const sectionStart = `<!-- section:${sectionName}:start -->`
const sectionEnd = `<!-- section:${sectionName}:end -->`
const sectionBlock = `${sectionStart}\n${sectionContent}\n${sectionEnd}`
// Escape special regex characters in delimiter strings
const escapeRegex = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
const comments = await github.paginate(
github.rest.issues.listComments,
{ ...context.repo, issue_number: prNumber }
)
const existing = comments.find(
(c) =>
c.user?.login === 'github-actions[bot]' &&
c.body?.includes(commentMarker)
)
if (!existing) {
return github.rest.issues.createComment({
...context.repo,
issue_number: prNumber,
body: `${commentMarker}\n${sectionBlock}`
})
}
const body = existing.body ?? ''
const sectionRegex = new RegExp(
`${escapeRegex(sectionStart)}[\\s\\S]*?${escapeRegex(sectionEnd)}`
)
const updated = sectionRegex.test(body)
? body.replace(sectionRegex, sectionBlock)
: body.trimEnd() + '\n\n' + sectionBlock
return github.rest.issues.updateComment({
...context.repo,
comment_id: existing.id,
body: updated
})

View File

@@ -1,29 +0,0 @@
# Description: Lint and format verification for GitHub merge queue runs.
# Paired with ci-lint-format.yaml — workflow name and job name must match
# so branch protection resolves a single required check in both the
# pull_request and merge_group contexts. This file runs verify-only steps
# with a read-only token; auto-fix and PR comments live in the PR workflow.
name: 'CI: Lint Format'
on:
merge_group:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
lint-and-format:
runs-on: ubuntu-latest
steps:
- name: Checkout merge group ref
uses: actions/checkout@v6
- name: Setup frontend
uses: ./.github/actions/setup-frontend
- name: Verify lint and format
uses: ./.github/actions/lint-format-verify

View File

@@ -1,7 +1,4 @@
# Description: Linting and code formatting validation for pull requests.
# Paired with ci-lint-format-queue.yaml - workflow name and job name must
# match so branch protection resolves a single required check in both the
# pull_request and merge_group contexts.
# Description: Linting and code formatting validation for pull requests
name: 'CI: Lint Format'
on:
@@ -29,6 +26,14 @@ jobs:
- name: Setup frontend
uses: ./.github/actions/setup-frontend
- name: Detect browser_tests changes
id: changed-paths
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
with:
filters: |
browser_tests:
- 'browser_tests/**'
- name: Run ESLint with auto-fix
run: pnpm lint:fix
@@ -72,8 +77,16 @@ jobs:
echo "See CONTRIBUTING.md for more details."
exit 1
- name: Verify lint and format
uses: ./.github/actions/lint-format-verify
- name: Final validation
run: |
pnpm lint
pnpm stylelint
pnpm format:check
pnpm knip
- name: Typecheck browser tests
if: steps.changed-paths.outputs.browser_tests == 'true'
run: pnpm typecheck:browser
- name: Comment on PR about auto-fix
if: steps.verify-changed-files.outputs.changed == 'true' && github.event.pull_request.head.repo.full_name == github.repository

View File

@@ -58,6 +58,21 @@ jobs:
retention-days: 30
if-no-files-found: warn
- name: Save PR metadata
if: github.event_name == 'pull_request'
run: |
mkdir -p temp/perf-meta
echo "${{ github.event.number }}" > temp/perf-meta/number.txt
echo "${{ github.event.pull_request.base.ref }}" > temp/perf-meta/base.txt
echo "${{ github.event.pull_request.head.sha }}" > temp/perf-meta/head-sha.txt
- name: Upload PR metadata
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v6
with:
name: perf-meta
path: temp/perf-meta/
- name: Save perf baseline to perf-data branch
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.perf.outcome == 'success'
continue-on-error: true

View File

@@ -32,6 +32,13 @@ jobs:
- name: Collect size data
run: node scripts/size-collect.js
- name: Save PR metadata
if: ${{ github.event_name == 'pull_request' }}
run: |
echo ${{ github.event.number }} > ./temp/size/number.txt
echo ${{ github.base_ref }} > ./temp/size/base.txt
echo ${{ github.event.pull_request.head.sha }} > ./temp/size/head-sha.txt
- name: Upload size data
uses: actions/upload-artifact@v6
with:

View File

@@ -1,147 +0,0 @@
name: 'CI: E2E Coverage'
on:
workflow_run:
workflows: ['CI: Tests E2E']
types:
- completed
concurrency:
group: e2e-coverage-${{ github.event.workflow_run.head_sha }}
cancel-in-progress: true
permissions:
contents: read
jobs:
merge:
if: >
github.repository == 'Comfy-Org/ComfyUI_frontend' &&
github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup frontend
uses: ./.github/actions/setup-frontend
- name: Download all shard coverage data
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
run_id: ${{ github.event.workflow_run.id }}
name: e2e-coverage-shard-.*
name_is_regexp: true
path: temp/coverage-shards
if_no_artifact_found: warn
- name: Install lcov
run: sudo apt-get install -y -qq lcov
- name: Merge shard coverage into single LCOV
run: |
mkdir -p coverage/playwright
LCOV_FILES=$(find temp/coverage-shards -name 'coverage.lcov' -type f)
if [ -z "$LCOV_FILES" ]; then
echo "No coverage.lcov files found"
touch coverage/playwright/coverage.lcov
exit 0
fi
ADD_ARGS=""
for f in $LCOV_FILES; do ADD_ARGS="$ADD_ARGS -a $f"; done
lcov $ADD_ARGS -o coverage/playwright/coverage.lcov
wc -l coverage/playwright/coverage.lcov
- name: Validate merged coverage
run: |
SHARD_COUNT=$(find temp/coverage-shards -name 'coverage.lcov' -type f | wc -l | tr -d ' ')
if [ "$SHARD_COUNT" -eq 0 ]; then
echo "::notice::No shard coverage files; upstream E2E was likely skipped."
exit 0
fi
MERGED_SF=$(grep -c '^SF:' coverage/playwright/coverage.lcov || echo 0)
MERGED_LH=$(awk -F: '/^LH:/{s+=$2}END{print s+0}' coverage/playwright/coverage.lcov)
MERGED_LF=$(awk -F: '/^LF:/{s+=$2}END{print s+0}' coverage/playwright/coverage.lcov)
echo "### Merged coverage" >> "$GITHUB_STEP_SUMMARY"
echo "- **$MERGED_SF** source files" >> "$GITHUB_STEP_SUMMARY"
echo "- **$MERGED_LH / $MERGED_LF** lines hit" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "| Shard | Files | Lines Hit |" >> "$GITHUB_STEP_SUMMARY"
echo "|-------|-------|-----------|" >> "$GITHUB_STEP_SUMMARY"
for f in $(find temp/coverage-shards -name 'coverage.lcov' -type f | sort); do
SHARD=$(basename "$(dirname "$f")")
SHARD_SF=$(grep -c '^SF:' "$f" || echo 0)
SHARD_LH=$(awk -F: '/^LH:/{s+=$2}END{print s+0}' "$f")
echo "| $SHARD | $SHARD_SF | $SHARD_LH |" >> "$GITHUB_STEP_SUMMARY"
if [ "$MERGED_LH" -lt "$SHARD_LH" ]; then
echo "::error::Merged LH ($MERGED_LH) < shard LH ($SHARD_LH) in $SHARD — possible data loss"
fi
done
- name: Upload merged coverage data
if: always()
uses: actions/upload-artifact@v6
with:
name: e2e-coverage
path: coverage/playwright/
retention-days: 30
if-no-files-found: warn
- name: Upload E2E coverage to Codecov
if: always()
uses: codecov/codecov-action@1af58845a975a7985b0beb0cbe6fbbb71a41dbad # v5.5.3
with:
files: coverage/playwright/coverage.lcov
flags: e2e
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false
- name: Generate HTML coverage report
run: |
if [ ! -s coverage/playwright/coverage.lcov ]; then
echo "No coverage data; generating placeholder report."
mkdir -p coverage/html
echo '<html><body><h1>No E2E coverage data available for this run.</h1></body></html>' > coverage/html/index.html
exit 0
fi
genhtml coverage/playwright/coverage.lcov \
-o coverage/html \
--title "ComfyUI E2E Coverage" \
--no-function-coverage \
--precision 1
- name: Upload HTML report artifact
uses: actions/upload-artifact@v6
with:
name: e2e-coverage-html
path: coverage/html/
retention-days: 30
deploy:
needs: merge
if: github.event.workflow_run.head_branch == 'main'
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Download HTML report
uses: actions/download-artifact@v7
with:
name: e2e-coverage-html
path: coverage/html
- name: Upload to GitHub Pages
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3.0.1
with:
path: coverage/html
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4.0.5

View File

@@ -6,10 +6,6 @@ on:
workflows: ['CI: Tests E2E']
types: [requested, completed]
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true
jobs:
deploy-and-comment-forked-pr:
runs-on: ubuntu-latest
@@ -34,33 +30,49 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v6
- name: Resolve PR from workflow_run context
- name: Get PR Number
id: pr
uses: ./.github/actions/resolve-pr-from-workflow-run
uses: actions/github-script@v8
with:
script: |
const { data: prs } = await github.rest.pulls.list({
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
});
const pr = prs.find(p => p.head.sha === context.payload.workflow_run.head_sha);
if (!pr) {
console.log('No PR found for SHA:', context.payload.workflow_run.head_sha);
return null;
}
console.log(`Found PR #${pr.number} from fork: ${context.payload.workflow_run.head_repository.full_name}`);
return pr.number;
- name: Handle Test Start
if: steps.pr.outputs.skip != 'true' && github.event.action == 'requested'
if: steps.pr.outputs.result != 'null' && github.event.action == 'requested'
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
chmod +x scripts/cicd/pr-playwright-deploy-and-comment.sh
./scripts/cicd/pr-playwright-deploy-and-comment.sh \
"${{ steps.pr.outputs.number }}" \
"${{ steps.pr.outputs.result }}" \
"${{ github.event.workflow_run.head_branch }}" \
"starting"
- name: Download and Deploy Reports
if: steps.pr.outputs.skip != 'true' && github.event.action == 'completed'
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
if: steps.pr.outputs.result != 'null' && github.event.action == 'completed'
uses: actions/download-artifact@v7
with:
run_id: ${{ github.event.workflow_run.id }}
name: playwright-report-.*
name_is_regexp: true
github-token: ${{ secrets.GITHUB_TOKEN }}
run-id: ${{ github.event.workflow_run.id }}
pattern: playwright-report-*
path: reports
if_no_artifact_found: warn
- name: Handle Test Completion
if: steps.pr.outputs.skip != 'true' && github.event.action == 'completed' && hashFiles('reports/**') != ''
if: steps.pr.outputs.result != 'null' && github.event.action == 'completed'
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
@@ -73,6 +85,6 @@ jobs:
chmod +x scripts/cicd/pr-playwright-deploy-and-comment.sh
./scripts/cicd/pr-playwright-deploy-and-comment.sh \
"${{ steps.pr.outputs.number }}" \
"${{ steps.pr.outputs.result }}" \
"${{ github.event.workflow_run.head_branch }}" \
"completed"

View File

@@ -7,7 +7,7 @@ on:
paths-ignore: ['**/*.md']
pull_request:
branches-ignore: [wip/*, draft/*, temp/*]
merge_group:
paths-ignore: ['**/*.md']
workflow_dispatch:
concurrency:
@@ -15,36 +15,7 @@ concurrency:
cancel-in-progress: true
jobs:
# Detect whether e2e-relevant files changed. Required checks see "skipped"
# (which counts as passing) when only docs/apps/storybook files are touched,
# avoiding the stall that paths-ignore would cause.
changes:
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
should_run: ${{ github.event_name != 'pull_request' || steps.filter.outputs.e2e }}
steps:
- name: Checkout repository
if: ${{ github.event_name == 'pull_request' }}
uses: actions/checkout@v6
- name: Check for e2e-relevant changes
if: ${{ github.event_name == 'pull_request' }}
id: filter
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
with:
predicate-quantifier: 'every'
filters: |
e2e:
- '**'
- '!apps/**'
- '!docs/**'
- '!.storybook/**'
- '!**/*.md'
setup:
needs: changes
if: ${{ needs.changes.outputs.should_run == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -115,7 +86,6 @@ jobs:
run: pnpm exec playwright test --project=chromium --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob
env:
PLAYWRIGHT_BLOB_OUTPUT_DIR: ./blob-report
COLLECT_COVERAGE: 'true'
- name: Upload blob report
uses: actions/upload-artifact@v6
@@ -125,15 +95,6 @@ jobs:
path: blob-report/
retention-days: 1
- name: Upload shard coverage data
if: always()
uses: actions/upload-artifact@v6
with:
name: e2e-coverage-shard-${{ matrix.shardIndex }}
path: coverage/playwright/
retention-days: 1
if-no-files-found: warn
playwright-tests:
# Ideally, each shard runs test in 6 minutes, but allow up to 15 minutes
timeout-minutes: 15
@@ -192,9 +153,9 @@ jobs:
# Merge sharded test reports (no container needed - only runs CLI)
merge-reports:
needs: [changes, playwright-tests-chromium-sharded]
needs: [playwright-tests-chromium-sharded]
runs-on: ubuntu-latest
if: ${{ !cancelled() && needs.changes.outputs.should_run == 'true' }}
if: ${{ !cancelled() }}
steps:
- name: Install pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
@@ -223,38 +184,14 @@ jobs:
path: ./playwright-report/
retention-days: 30
# Gate job — single required check that passes whether the matrix ran or was
# skipped. Branch rulesets require this instead of the individual matrix-
# expanded check names so PRs with no e2e-relevant changes aren't stuck.
e2e-status:
if: ${{ always() }}
needs: [changes, playwright-tests-chromium-sharded, playwright-tests]
runs-on: ubuntu-latest
steps:
- name: Check E2E results
env:
SHOULD_RUN: ${{ needs.changes.outputs.should_run }}
SHARDED: ${{ needs.playwright-tests-chromium-sharded.result }}
BROWSERS: ${{ needs.playwright-tests.result }}
run: |
[[ "$SHOULD_RUN" != "true" ]] && echo "E2E skipped" && exit 0
[[ "$SHARDED" != "success" || "$BROWSERS" != "success" ]] && echo "E2E failed" && exit 1
echo "E2E passed"
#### BEGIN Deployment and commenting (non-forked PRs only)
# when using pull_request event, we have permission to comment directly
# if its a forked repo, we need to use workflow_run event in a separate workflow (pr-playwright-deploy.yaml)
# Post starting comment for non-forked PRs
comment-on-pr-start:
needs: changes
runs-on: ubuntu-latest
if: >-
${{
needs.changes.outputs.should_run == 'true' &&
github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.fork == false
}}
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
permissions:
pull-requests: write
steps:
@@ -273,15 +210,9 @@ jobs:
# Deploy and comment for non-forked PRs only
deploy-and-comment:
needs: [changes, playwright-tests, merge-reports]
needs: [playwright-tests, merge-reports]
runs-on: ubuntu-latest
if: >-
${{
always() &&
needs.changes.outputs.should_run == 'true' &&
github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.fork == false
}}
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
permissions:
pull-requests: write
contents: read

View File

@@ -6,10 +6,6 @@ on:
workflows: ['CI: Tests Storybook']
types: [requested, completed]
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true
jobs:
deploy-and-comment-forked-pr:
runs-on: ubuntu-latest
@@ -34,23 +30,40 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v6
- name: Resolve PR from workflow_run context
- name: Get PR Number
id: pr
uses: ./.github/actions/resolve-pr-from-workflow-run
uses: actions/github-script@v8
with:
script: |
const { data: prs } = await github.rest.pulls.list({
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
});
const pr = prs.find(p => p.head.sha === context.payload.workflow_run.head_sha);
if (!pr) {
console.log('No PR found for SHA:', context.payload.workflow_run.head_sha);
return null;
}
console.log(`Found PR #${pr.number} from fork: ${context.payload.workflow_run.head_repository.full_name}`);
return pr.number;
- name: Handle Storybook Start
if: steps.pr.outputs.skip != 'true' && github.event.action == 'requested'
if: steps.pr.outputs.result != 'null' && github.event.action == 'requested'
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
chmod +x scripts/cicd/pr-storybook-deploy-and-comment.sh
./scripts/cicd/pr-storybook-deploy-and-comment.sh \
"${{ steps.pr.outputs.number }}" \
"${{ steps.pr.outputs.result }}" \
"${{ github.event.workflow_run.head_branch }}" \
"starting"
- name: Download and Deploy Storybook
if: steps.pr.outputs.skip != 'true' && github.event.action == 'completed' && github.event.workflow_run.conclusion == 'success'
if: steps.pr.outputs.result != 'null' && github.event.action == 'completed' && github.event.workflow_run.conclusion == 'success'
uses: actions/download-artifact@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -59,7 +72,7 @@ jobs:
path: storybook-static
- name: Handle Storybook Completion
if: steps.pr.outputs.skip != 'true' && github.event.action == 'completed'
if: steps.pr.outputs.result != 'null' && github.event.action == 'completed'
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
@@ -69,6 +82,6 @@ jobs:
run: |
chmod +x scripts/cicd/pr-storybook-deploy-and-comment.sh
./scripts/cicd/pr-storybook-deploy-and-comment.sh \
"${{ steps.pr.outputs.number }}" \
"${{ steps.pr.outputs.result }}" \
"${{ github.event.workflow_run.head_branch }}" \
"completed"

View File

@@ -8,7 +8,6 @@ on:
pull_request:
branches-ignore: [wip/*, draft/*, temp/*]
paths-ignore: ['**/*.md']
merge_group:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -27,20 +26,10 @@ jobs:
- name: Run Vitest tests with coverage
run: pnpm test:coverage
- name: Upload unit coverage artifact
if: always() && github.event_name == 'push'
uses: actions/upload-artifact@v6
with:
name: unit-coverage
path: coverage/lcov.info
retention-days: 30
if-no-files-found: warn
- name: Upload coverage to Codecov
if: always()
uses: codecov/codecov-action@1af58845a975a7985b0beb0cbe6fbbb71a41dbad # v5.5.3
with:
files: coverage/lcov.info
flags: unit
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false

View File

@@ -1,158 +0,0 @@
---
name: 'CI: Vercel Website Preview'
on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'apps/website/**'
- 'packages/design-system/**'
- 'packages/tailwind-utils/**'
push:
branches: [main]
paths:
- 'apps/website/**'
- 'packages/design-system/**'
- 'packages/tailwind-utils/**'
env:
VERCEL_ORG_ID: ${{ secrets.VERCEL_WEBSITE_ORG_ID }}
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_WEBSITE_PROJECT_ID }}
VERCEL_TOKEN: ${{ secrets.VERCEL_WEBSITE_TOKEN }}
VERCEL_SCOPE: comfyui
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
deploy-preview:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
permissions:
contents: read
env:
ALIAS_HOST: comfy-website-preview-pr-${{ github.event.pull_request.number }}.vercel.app
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
- name: Install Vercel CLI
run: npm install --global vercel@latest
- name: Pull Vercel environment information
run: vercel pull --yes --environment=preview
- name: Build project artifacts
run: vercel build
- name: Fetch head commit metadata
id: head-commit
uses: actions/github-script@v8
with:
script: |
const { data } = await github.rest.repos.getCommit({
owner: context.repo.owner,
repo: context.repo.repo,
ref: context.payload.pull_request.head.sha,
})
const author = data.author?.login || data.commit.author?.name || ''
const message = (data.commit.message || '').split('\n', 1)[0]
core.setOutput('author', author)
core.setOutput('message', message)
- name: Deploy project artifacts to Vercel
id: deploy
env:
GIT_COMMIT_REF: ${{ github.event.pull_request.head.ref }}
GIT_COMMIT_SHA: ${{ github.event.pull_request.head.sha }}
GIT_AUTHOR_LOGIN: ${{ steps.head-commit.outputs.author }}
GIT_COMMIT_MESSAGE: ${{ steps.head-commit.outputs.message }}
GIT_PR_ID: ${{ github.event.pull_request.number }}
GIT_REPO: ${{ github.repository }}
run: |
URL=$(vercel deploy --prebuilt \
--meta githubCommitRef="$GIT_COMMIT_REF" \
--meta githubCommitSha="$GIT_COMMIT_SHA" \
--meta githubCommitAuthorLogin="$GIT_AUTHOR_LOGIN" \
--meta githubCommitMessage="$GIT_COMMIT_MESSAGE" \
--meta githubPrId="$GIT_PR_ID" \
--meta githubRepo="$GIT_REPO")
echo "url=$URL" >> "$GITHUB_OUTPUT"
- name: Alias deployment to stable PR hostname
id: alias-set
continue-on-error: true
env:
DEPLOY_URL: ${{ steps.deploy.outputs.url }}
run: |
vercel alias set "$DEPLOY_URL" "$ALIAS_HOST" --scope="$VERCEL_SCOPE"
- name: Publish preview outputs
env:
DEPLOY_URL: ${{ steps.deploy.outputs.url }}
ALIAS_OK: ${{ steps.alias-set.outcome == 'success' }}
run: |
if [[ "$ALIAS_OK" == "true" ]]; then
STABLE_URL="https://$ALIAS_HOST"
else
STABLE_URL="$DEPLOY_URL"
fi
mkdir -p temp/vercel-preview
echo "$DEPLOY_URL" > temp/vercel-preview/url.txt
echo "$STABLE_URL" > temp/vercel-preview/stable-url.txt
{
echo "**Preview:** $STABLE_URL"
if [[ "$ALIAS_OK" == "true" ]]; then
echo "**This commit:** $DEPLOY_URL"
else
echo "_Stable alias update failed — URL reflects this commit only._"
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload preview metadata
uses: actions/upload-artifact@v6
with:
name: vercel-preview
path: temp/vercel-preview
deploy-production:
if: github.event_name == 'push'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
- name: Install Vercel CLI
run: npm install --global vercel@latest
- name: Pull Vercel environment information
run: vercel pull --yes --environment=production
- name: Build project artifacts
run: vercel build --prod
- name: Deploy project artifacts to Vercel
id: deploy
run: |
URL=$(vercel deploy --prebuilt --prod)
echo "url=$URL" >> "$GITHUB_OUTPUT"
- name: Add deployment URL to summary
run: echo "**Production:** ${{ steps.deploy.outputs.url }}" >> "$GITHUB_STEP_SUMMARY"

View File

@@ -1,260 +0,0 @@
name: 'CI: Website E2E'
on:
push:
branches: [main, website/*]
paths:
- 'apps/website/**'
- 'packages/design-system/**'
- 'packages/tailwind-utils/**'
- 'pnpm-lock.yaml'
pull_request:
branches-ignore: [wip/*, draft/*, temp/*]
paths:
- 'apps/website/**'
- 'packages/design-system/**'
- 'packages/tailwind-utils/**'
- 'pnpm-lock.yaml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
website-e2e:
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.58.1-noble
timeout-minutes: 15
permissions:
contents: read
outputs:
test-outcome: ${{ steps.tests.outcome }}
report-url: ${{ steps.deploy.outputs.url }}
screenshot-failures: ${{ steps.failures.outputs.screenshot }}
other-failures: ${{ steps.failures.outputs.other }}
# Evaluated at job level (not from a step) — static expression.
is-pr: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false }}
steps:
- uses: actions/checkout@v6
- name: Install pnpm
run: corepack enable && corepack prepare
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Build website
run: pnpm --filter @comfyorg/website build
- name: Run Playwright tests
id: tests
run: pnpm --filter @comfyorg/website test:e2e
- name: Upload test report
uses: actions/upload-artifact@v6
if: ${{ !cancelled() }}
with:
name: website-playwright-report
path: apps/website/playwright-report/
retention-days: 30
- name: Deploy report to Cloudflare
id: deploy
if: always() && !cancelled()
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
HEAD_REF: ${{ github.head_ref || github.ref_name }}
run: |
BRANCH=$(echo "$HEAD_REF" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9-]/-/g;s/--*/-/g;s/^-\|-$//g')
DEPLOY_OK=false
for i in 1 2 3; do
echo "Deployment attempt $i of 3..."
OUTPUT=$(npx wrangler@^4.0.0 pages deploy apps/website/playwright-report \
--project-name=comfyui-website-e2e \
--branch="$BRANCH" 2>&1) && { DEPLOY_OK=true; break; } || echo "$OUTPUT"
[ $i -lt 3 ] && sleep 10
done
echo "$OUTPUT"
if [ "$DEPLOY_OK" != "true" ]; then
echo "::error::All 3 deployment attempts failed"
exit 1
fi
URL=$(echo "$OUTPUT" | grep -oE 'https://[a-zA-Z0-9.-]+\.pages\.dev\S*' | head -1)
echo "url=${URL}" >> $GITHUB_OUTPUT
- name: Categorize failures
id: failures
if: always() && !cancelled() && steps.tests.outcome != 'success'
uses: actions/github-script@v8
with:
script: |
const fs = require('fs')
const report = JSON.parse(fs.readFileSync('apps/website/results.json', 'utf8'))
function isFailed(t) { return t.status === 'unexpected' || t.status === 'flaky' }
function isVisual(spec) {
return spec.file?.includes('visual') ||
spec.tests?.some(t => t.results?.some(r => r.error?.message?.includes('toHaveScreenshot')))
}
function specsOf(suite) {
return [
...(suite.specs || []),
...(suite.suites || []).flatMap(specsOf)
]
}
// True: Visual
// False: Other
const failed = specsOf(report)
.flatMap(spec => (spec.tests || [])
.filter(isFailed)
.map(() => isVisual(spec)))
const screenshotFailures = failed.filter(Boolean).length
core.setOutput('screenshot', screenshotFailures)
core.setOutput('other', failed.length - screenshotFailures)
- name: Write job summary
if: always() && !cancelled()
uses: actions/github-script@v8
env:
TEST_OUTCOME: ${{ steps.tests.outcome }}
REPORT_URL: ${{ steps.deploy.outputs.url }}
SCREENSHOT_FAILURES: ${{ steps.failures.outputs.screenshot }}
OTHER_FAILURES: ${{ steps.failures.outputs.other }}
with:
script: |
const passed = process.env.TEST_OUTCOME === 'success'
const reportUrl = process.env.REPORT_URL
const screenshotFailures = parseInt(process.env.SCREENSHOT_FAILURES) || 0
const otherFailures = parseInt(process.env.OTHER_FAILURES) || 0
const lines = ['## 🌐 Website E2E', '']
if (passed) {
lines.push('> [!TIP]', '> All tests passed.')
} else {
lines.push('> [!CAUTION]', '> Some tests failed.')
}
const rows = [
['Status', passed ? '✅ Passed' : '❌ Failed'],
['Report', reportUrl ? `[View Report](${reportUrl})` : '_unavailable_']
]
if (!passed) {
rows.push(
['Screenshot diffs', String(screenshotFailures)],
['Other failures', String(otherFailures)]
)
}
lines.push(
'',
'| | |',
'|---|---|',
...rows.map(([k, v]) => `| **${k}** | ${v} |`)
)
await core.summary.addRaw(lines.join('\n')).write()
post-starting-comment:
# Safe to comment from pull_request trigger: fork PRs are excluded by the guard below.
# This avoids a ci-*/pr-* workflow_run split for a comment that must appear immediately.
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false
runs-on: ubuntu-latest
permissions:
pull-requests: write
contents: read
concurrency:
group: website-pr-comment-${{ github.event.pull_request.number }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/upsert-comment-section
with:
pr-number: ${{ github.event.pull_request.number }}
section-name: e2e
comment-marker: '<!-- WEBSITE_CI_REPORT -->'
token: ${{ secrets.GITHUB_TOKEN }}
section-content: |-
## 🌐 Website E2E
<!-- WEBSITE_E2E_STATUS -->
> [!NOTE]
> Tests are running… [View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
post-result-comment:
needs: website-e2e
if: always() && !cancelled() && needs.website-e2e.outputs.is-pr == 'true'
runs-on: ubuntu-latest
permissions:
pull-requests: write
contents: read
concurrency:
group: website-pr-comment-${{ github.event.pull_request.number }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Build e2e section content
id: content
uses: actions/github-script@v8
env:
TEST_OUTCOME: ${{ needs.website-e2e.outputs.test-outcome }}
REPORT_URL: ${{ needs.website-e2e.outputs.report-url }}
SCREENSHOT_FAILURES: ${{ needs.website-e2e.outputs.screenshot-failures }}
OTHER_FAILURES: ${{ needs.website-e2e.outputs.other-failures }}
with:
script: |
const passed = process.env.TEST_OUTCOME === 'success'
const reportUrl = process.env.REPORT_URL
const screenshotFailures = parseInt(process.env.SCREENSHOT_FAILURES) || 0
const otherFailures = parseInt(process.env.OTHER_FAILURES) || 0
const lines = ['## 🌐 Website E2E', '<!-- WEBSITE_E2E_STATUS -->', '']
if (passed) {
lines.push('> [!TIP]', '> All tests passed.')
} else {
lines.push('> [!CAUTION]', '> Some tests failed.')
}
const rows = [
['Status', passed ? '✅ Passed' : '❌ Failed'],
['Report', reportUrl ? `[View Report](${reportUrl})` : '_unavailable_']
]
if (!passed) {
rows.push(
['Screenshot diffs', String(screenshotFailures)],
['Other failures', String(otherFailures)]
)
}
lines.push(
'',
'| | |',
'|---|---|',
...rows.map(([k, v]) => `| **${k}** | ${v} |`)
)
if (screenshotFailures > 0) {
const s = screenshotFailures === 1 ? '' : 's'
lines.push('', `- [ ] Update website screenshots (${screenshotFailures} screenshot diff${s})`)
}
if (otherFailures > 0) {
lines.push(
'',
'> [!WARNING]',
`> ${otherFailures} non-screenshot failure${otherFailures === 1 ? '' : 's'} — these require manual review.`
)
}
core.setOutput('section-content', lines.join('\n'))
- uses: ./.github/actions/upsert-comment-section
with:
pr-number: ${{ github.event.pull_request.number }}
section-name: e2e
comment-marker: '<!-- WEBSITE_CI_REPORT -->'
token: ${{ secrets.GITHUB_TOKEN }}
section-content: ${{ steps.content.outputs.section-content }}

View File

@@ -1,149 +0,0 @@
name: 'Coverage: Slack Notification'
on:
workflow_run:
workflows: ['CI: Tests Unit']
branches: [main]
types:
- completed
permissions:
contents: read
actions: read
pull-requests: read
jobs:
notify:
if: >
github.repository == 'Comfy-Org/ComfyUI_frontend' &&
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Setup frontend
uses: ./.github/actions/setup-frontend
- name: Download current unit coverage
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
run_id: ${{ github.event.workflow_run.id }}
name: unit-coverage
path: coverage
- name: Download previous unit coverage baseline
continue-on-error: true
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
branch: main
workflow: coverage-slack-notify.yaml
name: unit-coverage-baseline
path: temp/coverage-baseline
if_no_artifact_found: warn
- name: Download latest E2E coverage
continue-on-error: true
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
branch: main
workflow: ci-tests-e2e-coverage.yaml
name: e2e-coverage
path: temp/e2e-coverage
if_no_artifact_found: warn
- name: Download previous E2E coverage baseline
continue-on-error: true
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
branch: main
workflow: coverage-slack-notify.yaml
name: e2e-coverage-baseline
path: temp/e2e-coverage-baseline
if_no_artifact_found: warn
- name: Resolve merged PR metadata
id: pr-meta
uses: actions/github-script@v8
with:
script: |
const sha = context.payload.workflow_run.head_sha;
const { data: commit } = await github.rest.repos.getCommit({
owner: context.repo.owner,
repo: context.repo.repo,
ref: sha,
});
const message = commit.commit.message ?? '';
const firstLine = message.split('\n')[0];
const match = firstLine.match(/\(#(\d+)\)\s*$/);
if (!match) {
core.setOutput('skip', 'true');
core.info('No PR number found in commit message — skipping.');
return;
}
const prNumber = match[1];
const { data: pr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: Number(prNumber),
});
core.setOutput('skip', 'false');
core.setOutput('number', prNumber);
core.setOutput('url', pr.html_url);
core.setOutput('author', pr.user.login);
- name: Generate Slack notification
if: steps.pr-meta.outputs.skip != 'true'
id: slack-payload
env:
PR_URL: ${{ steps.pr-meta.outputs.url }}
PR_NUMBER: ${{ steps.pr-meta.outputs.number }}
PR_AUTHOR: ${{ steps.pr-meta.outputs.author }}
run: |
PAYLOAD=$(pnpm exec tsx scripts/coverage-slack-notify.ts \
--pr-url="$PR_URL" \
--pr-number="$PR_NUMBER" \
--author="$PR_AUTHOR")
if [ -n "$PAYLOAD" ]; then
echo "has_payload=true" >> "$GITHUB_OUTPUT"
DELIM="SLACK_PAYLOAD_$(date +%s)"
echo "payload<<$DELIM" >> "$GITHUB_OUTPUT"
printf '%s\n' "$PAYLOAD" >> "$GITHUB_OUTPUT"
echo "$DELIM" >> "$GITHUB_OUTPUT"
else
echo "has_payload=false" >> "$GITHUB_OUTPUT"
fi
- name: Post to Slack
if: steps.slack-payload.outputs.has_payload == 'true'
continue-on-error: true
env:
SLACK_PAYLOAD: ${{ steps.slack-payload.outputs.payload }}
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
run: |
# Channel: #p-deprecated-frontend-automated-testing
BODY=$(echo "$SLACK_PAYLOAD" | jq --arg ch "C0AP09LKRDZ" '. + {channel: $ch}')
curl -sf -X POST \
-H "Authorization: Bearer $SLACK_BOT_TOKEN" \
-H "Content-Type: application/json" \
-d "$BODY" \
-o /dev/null \
https://slack.com/api/chat.postMessage
- name: Save unit coverage baseline
if: always() && hashFiles('coverage/lcov.info') != ''
uses: actions/upload-artifact@v6
with:
name: unit-coverage-baseline
path: coverage/lcov.info
retention-days: 90
if-no-files-found: warn
- name: Save E2E coverage baseline
if: always() && hashFiles('temp/e2e-coverage/coverage.lcov') != ''
uses: actions/upload-artifact@v6
with:
name: e2e-coverage-baseline
path: temp/e2e-coverage/coverage.lcov
retention-days: 90
if-no-files-found: warn

1100
.github/workflows/pr-qa.yaml vendored Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -2,7 +2,7 @@ name: 'PR: Unified Report'
on:
workflow_run:
workflows: ['CI: Size Data', 'CI: Performance Report', 'CI: E2E Coverage']
workflows: ['CI: Size Data', 'CI: Performance Report']
types:
- completed
@@ -30,25 +30,110 @@ jobs:
- name: Resolve PR from workflow_run context
id: pr-meta
uses: ./.github/actions/resolve-pr-from-workflow-run
uses: actions/github-script@v8
with:
script: |
let pr = context.payload.workflow_run.pull_requests?.[0];
if (!pr) {
const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
owner: context.repo.owner,
repo: context.repo.repo,
commit_sha: context.payload.workflow_run.head_sha,
});
pr = prs.find(p => p.state === 'open');
}
- name: Find size workflow run
if (!pr) {
core.info('No open PR found for this workflow run — skipping.');
core.setOutput('skip', 'true');
return;
}
// Verify the workflow_run head SHA matches the current PR head
const { data: livePr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pr.number,
});
if (livePr.head.sha !== context.payload.workflow_run.head_sha) {
core.info(`Stale run: workflow SHA ${context.payload.workflow_run.head_sha} != PR head ${livePr.head.sha}`);
core.setOutput('skip', 'true');
return;
}
core.setOutput('skip', 'false');
core.setOutput('number', String(pr.number));
core.setOutput('base', livePr.base.ref);
core.setOutput('head-sha', livePr.head.sha);
- name: Find size workflow run for this commit
if: steps.pr-meta.outputs.skip != 'true'
id: find-size
uses: ./.github/actions/find-workflow-run
uses: actions/github-script@v8
with:
workflow-id: ci-size-data.yaml
head-sha: ${{ steps.pr-meta.outputs.head-sha }}
token: ${{ secrets.GITHUB_TOKEN }}
script: |
const headSha = '${{ steps.pr-meta.outputs.head-sha }}';
const { data: runs } = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: 'ci-size-data.yaml',
head_sha: headSha,
per_page: 1,
});
- name: Find perf workflow run
const run = runs.workflow_runs[0];
if (!run) {
core.setOutput('status', 'pending');
return;
}
if (run.status !== 'completed') {
core.setOutput('status', 'pending');
return;
}
if (run.conclusion !== 'success') {
core.setOutput('status', 'failed');
return;
}
core.setOutput('status', 'ready');
core.setOutput('run-id', String(run.id));
- name: Find perf workflow run for this commit
if: steps.pr-meta.outputs.skip != 'true'
id: find-perf
uses: ./.github/actions/find-workflow-run
uses: actions/github-script@v8
with:
workflow-id: ci-perf-report.yaml
head-sha: ${{ steps.pr-meta.outputs.head-sha }}
token: ${{ secrets.GITHUB_TOKEN }}
script: |
const headSha = '${{ steps.pr-meta.outputs.head-sha }}';
const { data: runs } = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: 'ci-perf-report.yaml',
head_sha: headSha,
per_page: 1,
});
const run = runs.workflow_runs[0];
if (!run) {
core.setOutput('status', 'pending');
return;
}
if (run.status !== 'completed') {
core.setOutput('status', 'pending');
return;
}
if (run.conclusion !== 'success') {
core.setOutput('status', 'failed');
return;
}
core.setOutput('status', 'ready');
core.setOutput('run-id', String(run.id));
- name: Download size data (current)
if: steps.pr-meta.outputs.skip != 'true' && steps.find-size.outputs.status == 'ready'
@@ -69,25 +154,6 @@ jobs:
path: temp/size-prev
if_no_artifact_found: warn
- name: Find coverage workflow run
if: steps.pr-meta.outputs.skip != 'true'
id: find-coverage
uses: ./.github/actions/find-workflow-run
with:
workflow-id: ci-tests-e2e-coverage.yaml
head-sha: ${{ steps.pr-meta.outputs.head-sha }}
not-found-status: skip
token: ${{ secrets.GITHUB_TOKEN }}
- name: Download coverage data
if: steps.pr-meta.outputs.skip != 'true' && steps.find-coverage.outputs.status == 'ready'
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
name: e2e-coverage
run_id: ${{ steps.find-coverage.outputs.run-id }}
path: temp/coverage
if_no_artifact_found: warn
- name: Download perf metrics (current)
if: steps.pr-meta.outputs.skip != 'true' && steps.find-perf.outputs.status == 'ready'
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
@@ -123,10 +189,9 @@ jobs:
- name: Generate unified report
if: steps.pr-meta.outputs.skip != 'true'
run: >
pnpm exec tsx scripts/unified-report.ts
node scripts/unified-report.js
--size-status=${{ steps.find-size.outputs.status }}
--perf-status=${{ steps.find-perf.outputs.status }}
--coverage-status=${{ steps.find-coverage.outputs.status }}
> pr-report.md
- name: Remove legacy separate comments

View File

@@ -1,239 +0,0 @@
name: 'PR: Update Website Screenshots'
on:
pull_request:
types: [labeled]
issue_comment:
types: [created, edited]
concurrency:
group: ${{ github.workflow }}-${{ github.event.issue.number || github.event.pull_request.number || github.ref }}
cancel-in-progress: false
jobs:
update-screenshots:
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.58.1-noble
timeout-minutes: 15
permissions:
contents: write
pull-requests: read
# Trigger: (1) label, (2) /slash-command, or (3) checkbox in E2E status comment
# ⚠️ This condition is duplicated on `post-starting-comment` — keep them in sync.
if: >
( github.event_name == 'pull_request' &&
github.event.label.name == 'Update Website Screenshots' ) ||
( github.event.issue.pull_request &&
github.event_name == 'issue_comment' &&
(
github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR'
) &&
startsWith(github.event.comment.body, '/update-website-screenshots') ) ||
( github.event.issue.pull_request &&
github.event_name == 'issue_comment' &&
github.event.comment.user.login == 'github-actions[bot]' &&
github.actor != 'github-actions[bot]' &&
contains(github.event.comment.body, '<!-- WEBSITE_E2E_STATUS -->') &&
contains(github.event.comment.body, '- [x] Update website screenshots') )
outputs:
pr-number: ${{ steps.pr-info.outputs.pr-number }}
update-outcome: ${{ steps.update-screenshots.outcome }}
has-changes: ${{ steps.commit.outputs.has-changes }}
changed-count: ${{ steps.commit.outputs.changed-count }}
steps:
- name: Verify sender permissions
if: >
github.event_name == 'issue_comment' &&
contains(github.event.comment.body, '<!-- WEBSITE_E2E_STATUS -->')
uses: actions/github-script@v8
with:
script: |
const { data } = await github.rest.repos.getCollaboratorPermissionLevel({
...context.repo,
username: context.actor
})
if (!['admin', 'write'].includes(data.permission)) {
core.setFailed(`User ${context.actor} does not have write access`)
}
- name: Get PR info
id: pr-info
uses: actions/github-script@v8
env:
PR_NUMBER: ${{ github.event.number || github.event.issue.number }}
with:
script: |
const prNumber = Number(process.env.PR_NUMBER)
const { data: pr } = await github.rest.pulls.get({
...context.repo,
pull_number: prNumber
})
core.setOutput('pr-number', prNumber)
core.setOutput('branch', pr.head.ref)
- uses: actions/checkout@v6
with:
ref: ${{ steps.pr-info.outputs.branch }}
token: ${{ secrets.PR_GH_TOKEN }}
- name: Install pnpm
run: corepack enable && corepack prepare
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Build website
run: pnpm --filter @comfyorg/website build
- name: Update screenshots
id: update-screenshots
run: pnpm --filter @comfyorg/website test:visual:update
continue-on-error: true
- name: Commit updated screenshots
id: commit
if: steps.update-screenshots.outcome == 'success'
run: |
git config --global --add safe.directory "$(pwd)"
git config --global user.name 'github-actions'
git config --global user.email 'github-actions@github.com'
CHANGED=$(git status --porcelain=v1 --untracked-files=all -- apps/website/e2e/ | wc -l)
echo "changed-count=${CHANGED}" >> $GITHUB_OUTPUT
if [ "$CHANGED" -eq 0 ]; then
echo "No screenshot changes to commit"
echo "has-changes=false" >> $GITHUB_OUTPUT
exit 0
fi
echo "has-changes=true" >> $GITHUB_OUTPUT
git add apps/website/e2e/
git commit -m "[automated] Update website screenshot expectations"
git push origin ${{ steps.pr-info.outputs.branch }}
- name: Upload test report
uses: actions/upload-artifact@v6
if: ${{ !cancelled() }}
with:
name: website-screenshot-update-report
path: apps/website/playwright-report/
retention-days: 14
- name: Remove label
if: always() && github.event_name == 'pull_request'
uses: actions/github-script@v8
env:
PR_NUMBER: ${{ steps.pr-info.outputs.pr-number }}
with:
script: |
try {
await github.rest.issues.removeLabel({
...context.repo,
issue_number: Number(process.env.PR_NUMBER),
name: 'Update Website Screenshots'
})
} catch (e) {
// Label may already be removed
}
post-starting-comment:
# Runs in parallel with update-screenshots to show "in progress" immediately.
# ⚠️ This condition is duplicated from `update-screenshots` — keep them in sync.
if: >
( github.event_name == 'pull_request' &&
github.event.label.name == 'Update Website Screenshots' ) ||
( github.event.issue.pull_request &&
github.event_name == 'issue_comment' &&
(
github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR'
) &&
startsWith(github.event.comment.body, '/update-website-screenshots') ) ||
( github.event.issue.pull_request &&
github.event_name == 'issue_comment' &&
github.event.comment.user.login == 'github-actions[bot]' &&
github.actor != 'github-actions[bot]' &&
contains(github.event.comment.body, '<!-- WEBSITE_E2E_STATUS -->') &&
contains(github.event.comment.body, '- [x] Update website screenshots') )
runs-on: ubuntu-latest
permissions:
pull-requests: write
contents: read
concurrency:
group: website-pr-comment-${{ github.event.number || github.event.issue.number }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/upsert-comment-section
with:
pr-number: ${{ github.event.number || github.event.issue.number }}
section-name: screenshot-update
comment-marker: '<!-- WEBSITE_CI_REPORT -->'
token: ${{ secrets.GITHUB_TOKEN }}
section-content: |-
## 📸 Screenshot Update
> [!NOTE]
> Updating screenshots… [View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
post-result-comment:
needs: update-screenshots
if: always() && !cancelled() && needs.update-screenshots.result != 'skipped'
runs-on: ubuntu-latest
permissions:
pull-requests: write
contents: read
concurrency:
group: website-pr-comment-${{ needs.update-screenshots.outputs.pr-number }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- name: Build screenshot-update section content
id: content
uses: actions/github-script@v8
env:
UPDATE_OUTCOME: ${{ needs.update-screenshots.outputs.update-outcome }}
HAS_CHANGES: ${{ needs.update-screenshots.outputs.has-changes }}
CHANGED_COUNT: ${{ needs.update-screenshots.outputs.changed-count }}
with:
script: |
const outcome = process.env.UPDATE_OUTCOME
const hasChanges = process.env.HAS_CHANGES === 'true'
const changedCount = parseInt(process.env.CHANGED_COUNT) || 0
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`
const lines = ['## 📸 Screenshot Update', '']
if (outcome !== 'success') {
lines.push(
'> [!CAUTION]',
`> Screenshot update failed. [View workflow run](${runUrl})`
)
} else if (!hasChanges) {
lines.push(
'> [!TIP]',
'> All screenshots are already up to date.'
)
} else {
const s = changedCount === 1 ? '' : 's'
lines.push(
'> [!TIP]',
`> Updated ${changedCount} screenshot${s} and pushed to the branch.`
)
}
core.setOutput('section-content', lines.join('\n'))
- uses: ./.github/actions/upsert-comment-section
with:
pr-number: ${{ needs.update-screenshots.outputs.pr-number }}
section-name: screenshot-update
comment-marker: '<!-- WEBSITE_CI_REPORT -->'
token: ${{ secrets.GITHUB_TOKEN }}
section-content: ${{ steps.content.outputs.section-content }}

View File

@@ -1,70 +0,0 @@
---
name: 'PR: Vercel Website Preview'
on:
workflow_run:
workflows: ['CI: Vercel Website Preview']
types:
- completed
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true
jobs:
comment:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
actions: read
# Uses head_branch as proxy for PR number (unavailable at job-level in workflow_run).
# Preview and E2E comment writes are NOT mutually serialized — the race window is
# small and self-healing on next push.
concurrency:
group: website-pr-comment-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: false
if: >
github.repository == 'Comfy-Org/ComfyUI_frontend' &&
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.conclusion == 'success'
steps:
- uses: actions/checkout@v6
- name: Download preview metadata
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
with:
name: vercel-preview
run_id: ${{ github.event.workflow_run.id }}
path: temp/vercel-preview
- name: Resolve PR number from workflow_run context
id: pr-meta
uses: ./.github/actions/resolve-pr-from-workflow-run
- name: Read preview URLs
if: steps.pr-meta.outputs.skip != 'true'
id: urls
run: |
echo "stable-url=$(cat temp/vercel-preview/stable-url.txt)" >> "$GITHUB_OUTPUT"
echo "unique-url=$(cat temp/vercel-preview/url.txt)" >> "$GITHUB_OUTPUT"
echo "short-sha=${HEAD_SHA:0:7}" >> "$GITHUB_OUTPUT"
env:
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
- name: Post preview comment
if: steps.pr-meta.outputs.skip != 'true'
uses: ./.github/actions/upsert-comment-section
with:
pr-number: ${{ steps.pr-meta.outputs.number }}
section-name: preview
comment-marker: '<!-- WEBSITE_CI_REPORT -->'
token: ${{ secrets.GITHUB_TOKEN }}
section-content: |-
## 🔗 Website Preview
**Website Preview:** ${{ steps.urls.outputs.stable-url }}
<sub>This commit: ${{ steps.urls.outputs.unique-url }}</sub>
<sub>Last updated: ${{ github.event.workflow_run.updated_at }} for `${{ steps.urls.outputs.short-sha }}`</sub>

View File

@@ -1,23 +1,14 @@
# Release workflow for ComfyUI frontend: version bump → PyPI publish → ComfyUI PR.
# Runs on a bi-weekly schedule for minor releases, or manually for patch/hotfix releases.
name: 'Release: ComfyUI'
# Automated bi-weekly workflow to bump ComfyUI frontend RC releases
name: 'Release: Bi-weekly ComfyUI'
on:
# Bi-weekly schedule: Monday at 20:00 UTC
# Schedule for Monday at 12:00 PM PST (20:00 UTC)
schedule:
- cron: '0 20 * * 1'
# Manual trigger for both on-demand minor and patch/hotfix releases
# Allow manual triggering (bypasses bi-weekly check)
workflow_dispatch:
inputs:
release_type:
description: 'minor = next minor version (bi-weekly cadence), patch = hotfix for current production version'
required: true
default: 'minor'
type: choice
options:
- minor
- patch
comfyui_fork:
description: 'ComfyUI fork to use for PR (e.g., Comfy-Org/ComfyUI)'
required: false
@@ -50,11 +41,10 @@ jobs:
- name: Summary
run: |
echo "## Release Check" >> $GITHUB_STEP_SUMMARY
echo "## Bi-weekly Check" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "- Is release week: ${{ steps.check.outputs.is_release_week }}" >> $GITHUB_STEP_SUMMARY
echo "- Manual trigger: ${{ github.event_name == 'workflow_dispatch' }}" >> $GITHUB_STEP_SUMMARY
echo "- Release type: ${{ inputs.release_type || 'minor (scheduled)' }}" >> $GITHUB_STEP_SUMMARY
resolve-version:
needs: check-release-week
@@ -86,8 +76,6 @@ jobs:
- name: Install pnpm
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
with:
package_json_file: frontend/package.json
- name: Setup Node.js
uses: actions/setup-node@v6
@@ -101,8 +89,6 @@ jobs:
- name: Resolve release information
id: resolve
working-directory: frontend
env:
RELEASE_TYPE: ${{ inputs.release_type || 'minor' }}
run: |
set -euo pipefail

3
.gitignore vendored
View File

@@ -67,6 +67,9 @@ dist.zip
/temp/
/tmp/
# QA local output
/.comfy-qa/
# Generated JSON Schemas
/schemas/

View File

@@ -84,7 +84,6 @@
"typescript/no-unsafe-declaration-merging": "off",
"typescript/no-unused-vars": "off",
"unicorn/no-empty-file": "off",
"vitest/require-mock-type-parameters": "off",
"unicorn/no-new-array": "off",
"unicorn/no-single-promise-in-promise-methods": "off",
"unicorn/no-useless-fallback-in-spread": "off",
@@ -117,60 +116,13 @@
},
{
"files": ["browser_tests/**/*.ts"],
"jsPlugins": ["eslint-plugin-playwright"],
"rules": {
"typescript/no-explicit-any": "error",
"no-async-promise-executor": "error",
"no-control-regex": "error",
"no-useless-rename": "error",
"no-unused-private-class-members": "error",
"unicorn/no-empty-file": "error",
"playwright/consistent-spacing-between-blocks": "error",
"playwright/expect-expect": [
"error",
{
"assertFunctionNames": [
"recordMeasurement",
"logMeasurement",
"builderSaveAs"
],
"assertFunctionPatterns": [
"^expect",
"^assert",
"^verify",
"^searchAndExpect",
"waitForOpen",
"waitForClosed",
"waitForRequest"
]
}
],
"playwright/max-nested-describe": "error",
"playwright/no-duplicate-hooks": "error",
"playwright/no-element-handle": "error",
"playwright/no-eval": "error",
"playwright/no-focused-test": "error",
"playwright/no-force-option": "error",
"playwright/no-networkidle": "error",
"playwright/no-page-pause": "error",
"playwright/no-skipped-test": "error",
"playwright/no-unsafe-references": "error",
"playwright/no-unused-locators": "error",
"playwright/no-useless-await": "error",
"playwright/no-useless-not": "error",
"playwright/no-wait-for-navigation": "error",
"playwright/no-wait-for-selector": "error",
"playwright/no-wait-for-timeout": "error",
"playwright/prefer-hooks-on-top": "error",
"playwright/prefer-locator": "error",
"playwright/prefer-to-have-count": "error",
"playwright/prefer-to-have-length": "error",
"playwright/prefer-web-first-assertions": "error",
"playwright/prefer-native-locators": "error",
"playwright/require-to-pass-timeout": "error",
"playwright/valid-expect": "error",
"playwright/valid-expect-in-promise": "error",
"playwright/valid-title": "error"
"unicorn/no-empty-file": "error"
}
}
]

View File

@@ -67,7 +67,6 @@
"ignoreFiles": [
"node_modules/**",
"dist/**",
"**/dist/**",
"playwright-report/**",
"public/**",
"src/lib/litegraph/**"

View File

@@ -1,12 +1,16 @@
{
"recommendations": [
"antfu.vite",
"austenc.tailwind-docs",
"bradlc.vscode-tailwindcss",
"davidanson.vscode-markdownlint",
"dbaeumer.vscode-eslint",
"donjayamanne.githistory",
"eamodio.gitlens",
"github.vscode-github-actions",
"github.vscode-pull-request-github",
"hbenl.vscode-test-explorer",
"kisstkondoros.vscode-codemetrics",
"lokalise.i18n-ally",
"ms-playwright.playwright",
"oxc.oxc-vscode",

View File

@@ -44,7 +44,6 @@ This project uses **pnpm**. Always prefer scripts defined in `package.json` (e.g
## Build, Test, and Development Commands
- `pnpm dev`: Start Vite dev server.
- `pnpm dev:cloud`: Dev server connected to cloud backend (testcloud.comfy.org)
- `pnpm dev:electron`: Dev server with Electron API mocks
- `pnpm build`: Type-check then production build to `dist/`
- `pnpm preview`: Preview the production build locally
@@ -312,16 +311,13 @@ When referencing Comfy-Org repos:
- Instead use a semantic value from the `style.css` theme
- e.g. `bg-node-component-surface`
- NEVER use `:class="[]"` to merge class names
- Always use `import { cn } from '@comfyorg/tailwind-utils'`
- Always use `import { cn } from '@/utils/tailwindUtil'`
- e.g. `<div :class="cn('text-node-component-header-icon', hasError && 'text-danger')" />`
- Use `cn()` inline in the template when feasible instead of creating a `computed` to hold the value
- NEVER use `!important` or the `!` important prefix for tailwind classes
- Find existing `!important` classes that are interfering with the styling and propose corrections of those instead.
- NEVER use arbitrary percentage values like `w-[80%]` when a Tailwind fraction utility exists
- Use `w-4/5` instead of `w-[80%]`, `w-1/2` instead of `w-[50%]`, etc.
- NEVER use font-size classes (`text-xs`, `text-sm`, etc.) to size `icon-[...]` (iconify) icons
- Iconify icons size via `width`/`height: 1.2em`, so font-size produces unpredictable results
- Use `size-*` classes for explicit sizing, or set font-size on the **parent** container and let `1.2em` scale naturally
## Agent-only rules

View File

@@ -5,14 +5,12 @@
"scripts": {
"lint": "nx run @comfyorg/desktop-ui:lint",
"typecheck": "nx run @comfyorg/desktop-ui:typecheck",
"test:unit": "vitest run --config vitest.config.mts",
"storybook": "storybook dev -p 6007",
"build-storybook": "storybook build -o dist/storybook"
},
"dependencies": {
"@comfyorg/comfyui-electron-types": "catalog:",
"@comfyorg/shared-frontend-utils": "workspace:*",
"@comfyorg/tailwind-utils": "workspace:*",
"@primevue/core": "catalog:",
"@primevue/themes": "catalog:",
"@vueuse/core": "catalog:",

View File

@@ -32,7 +32,7 @@ import { useI18n } from 'vue-i18n'
import { useTerminal } from '@/composables/bottomPanelTabs/useTerminal'
import { electronAPI, isElectron } from '@/utils/envUtil'
import { cn } from '@comfyorg/tailwind-utils'
import { cn } from '@/utils/tailwindUtil'
const { t } = useI18n()

View File

@@ -1,97 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it } from 'vitest'
import { createI18n } from 'vue-i18n'
import StartupDisplay from '@/components/common/StartupDisplay.vue'
const i18n = createI18n({
legacy: false,
locale: 'en',
messages: { en: { g: { logoAlt: 'ComfyUI' } } }
})
const ProgressBarStub = {
props: ['mode', 'value', 'showValue'],
template:
'<div data-testid="progress-bar" :data-mode="mode" :data-value="value" />'
}
function renderDisplay(
props: {
progressPercentage?: number
title?: string
statusText?: string
hideProgress?: boolean
fullScreen?: boolean
} = {}
) {
return render(StartupDisplay, {
props,
global: {
plugins: [[PrimeVue, { unstyled: true }], i18n],
stubs: { ProgressBar: ProgressBarStub }
}
})
}
describe('StartupDisplay', () => {
describe('progressMode', () => {
it('renders indeterminate mode when progressPercentage is undefined', () => {
renderDisplay()
expect(screen.getByTestId('progress-bar').dataset.mode).toBe(
'indeterminate'
)
})
it('renders determinate mode when progressPercentage is provided', () => {
renderDisplay({ progressPercentage: 50 })
expect(screen.getByTestId('progress-bar').dataset.mode).toBe(
'determinate'
)
})
it('passes progressPercentage as value to the progress bar', () => {
renderDisplay({ progressPercentage: 75 })
expect(screen.getByTestId('progress-bar').dataset.value).toBe('75')
})
})
describe('hideProgress', () => {
it('hides the progress bar when hideProgress is true', () => {
renderDisplay({ hideProgress: true })
expect(screen.queryByTestId('progress-bar')).toBeNull()
})
it('shows the progress bar by default', () => {
renderDisplay()
expect(screen.getByTestId('progress-bar')).toBeDefined()
})
})
describe('title', () => {
it('renders the title text when provided', () => {
renderDisplay({ title: 'Loading...' })
expect(screen.getByText('Loading...')).toBeDefined()
})
it('does not render h1 when title is not provided', () => {
renderDisplay()
expect(screen.queryByRole('heading', { level: 1 })).toBeNull()
})
})
describe('statusText', () => {
it('renders statusText with data-testid attribute', () => {
renderDisplay({ statusText: 'Starting server' })
expect(screen.getByTestId('startup-status-text').textContent).toContain(
'Starting server'
)
})
it('does not render statusText element when not provided', () => {
renderDisplay()
expect(screen.queryByTestId('startup-status-text')).toBeNull()
})
})
})

View File

@@ -1,208 +0,0 @@
import { render, screen, waitFor } from '@testing-library/vue'
import userEvent from '@testing-library/user-event'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi, beforeEach } from 'vitest'
vi.mock('@comfyorg/shared-frontend-utils/networkUtil', () => ({
checkUrlReachable: vi.fn()
}))
import { checkUrlReachable } from '@comfyorg/shared-frontend-utils/networkUtil'
import UrlInput from '@/components/common/UrlInput.vue'
import { ValidationState } from '@/utils/validationUtil'
const InputTextStub = {
props: ['modelValue', 'invalid'],
emits: ['update:modelValue', 'blur'],
template: `<input
data-testid="url-input"
:value="modelValue"
:data-invalid="invalid"
@input="$emit('update:modelValue', $event.target.value)"
@blur="$emit('blur')"
/>`
}
const InputIconStub = {
template: '<span data-testid="input-icon" />'
}
const IconFieldStub = {
template: '<div><slot /></div>'
}
function renderUrlInput(
modelValue = '',
validateUrlFn?: (url: string) => Promise<boolean>
) {
return render(UrlInput, {
props: { modelValue, ...(validateUrlFn ? { validateUrlFn } : {}) },
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
InputText: InputTextStub,
InputIcon: InputIconStub,
IconField: IconFieldStub
}
}
})
}
describe('UrlInput', () => {
beforeEach(() => {
vi.resetAllMocks()
})
describe('initial validation on mount', () => {
it('stays IDLE when modelValue is empty on mount', async () => {
renderUrlInput('')
await waitFor(() => {
expect(screen.getByTestId('url-input').dataset.invalid).toBe('false')
})
})
it('sets VALID state when modelValue is a reachable URL on mount', async () => {
vi.mocked(checkUrlReachable).mockResolvedValue(true)
renderUrlInput('https://example.com')
await waitFor(() => {
expect(screen.getByTestId('url-input').dataset.invalid).toBe('false')
})
})
it('sets INVALID state when URL is not reachable on mount', async () => {
vi.mocked(checkUrlReachable).mockResolvedValue(false)
renderUrlInput('https://unreachable.example')
await waitFor(() => {
expect(screen.getByTestId('url-input').dataset.invalid).toBe('true')
})
})
})
describe('input handling', () => {
it('resets validation state to IDLE on user input', async () => {
vi.mocked(checkUrlReachable).mockResolvedValue(false)
renderUrlInput('https://bad.example')
await waitFor(() => {
expect(screen.getByTestId('url-input').dataset.invalid).toBe('true')
})
const user = userEvent.setup()
await user.type(screen.getByTestId('url-input'), 'x')
expect(screen.getByTestId('url-input').dataset.invalid).toBe('false')
})
it('strips whitespace from typed input', async () => {
const onUpdate = vi.fn()
render(UrlInput, {
props: {
modelValue: '',
'onUpdate:modelValue': onUpdate
},
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
InputText: InputTextStub,
InputIcon: InputIconStub,
IconField: IconFieldStub
}
}
})
const user = userEvent.setup()
const input = screen.getByTestId('url-input')
await user.type(input, 'htt ps')
expect((input as HTMLInputElement).value).not.toContain(' ')
})
})
describe('blur handling', () => {
it('emits update:modelValue on blur', async () => {
const onUpdate = vi.fn()
render(UrlInput, {
props: {
modelValue: 'https://example.com',
'onUpdate:modelValue': onUpdate
},
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
InputText: InputTextStub,
InputIcon: InputIconStub,
IconField: IconFieldStub
}
}
})
const user = userEvent.setup()
await user.click(screen.getByTestId('url-input'))
await user.tab()
expect(onUpdate).toHaveBeenCalled()
})
it('normalizes URL on blur', async () => {
const onUpdate = vi.fn()
render(UrlInput, {
props: {
modelValue: 'https://example.com',
'onUpdate:modelValue': onUpdate
},
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
InputText: InputTextStub,
InputIcon: InputIconStub,
IconField: IconFieldStub
}
}
})
const user = userEvent.setup()
await user.click(screen.getByTestId('url-input'))
await user.tab()
const emittedUrl = onUpdate.mock.calls[0]?.[0]
expect(emittedUrl).toBe('https://example.com/')
})
})
describe('custom validateUrlFn', () => {
it('uses custom validateUrlFn when provided', async () => {
const customValidator = vi.fn().mockResolvedValue(true)
renderUrlInput('https://custom.example', customValidator)
await waitFor(() => {
expect(customValidator).toHaveBeenCalledWith('https://custom.example')
})
expect(checkUrlReachable).not.toHaveBeenCalled()
})
})
describe('state-change emission', () => {
it('emits state-change when validation state changes', async () => {
const onStateChange = vi.fn()
vi.mocked(checkUrlReachable).mockResolvedValue(true)
render(UrlInput, {
props: {
modelValue: 'https://example.com',
'onState-change': onStateChange
},
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
InputText: InputTextStub,
InputIcon: InputIconStub,
IconField: IconFieldStub
}
}
})
await waitFor(() => {
expect(onStateChange).toHaveBeenCalledWith(ValidationState.VALID)
})
})
})
})

View File

@@ -1,112 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi } from 'vitest'
import { createI18n } from 'vue-i18n'
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => ({
getPlatform: vi.fn().mockReturnValue('win32')
}))
}))
vi.mock('@/i18n', () => ({
t: (key: string) => key,
te: () => false,
st: (_key: string, fallback: string) => fallback
}))
import type { TorchDeviceType } from '@comfyorg/comfyui-electron-types'
import GpuPicker from '@/components/install/GpuPicker.vue'
const i18n = createI18n({
legacy: false,
locale: 'en',
missingWarn: false,
fallbackWarn: false,
messages: { en: {} }
})
const HardwareOptionStub = {
props: ['imagePath', 'placeholderText', 'subtitle', 'selected'],
emits: ['click'],
template:
'<button :data-testid="placeholderText" :data-selected="selected" @click="$emit(\'click\')" >{{ placeholderText }}</button>'
}
function renderPicker(device: TorchDeviceType | null = null) {
return render(GpuPicker, {
props: { device },
global: {
plugins: [[PrimeVue, { unstyled: true }], i18n],
stubs: {
HardwareOption: HardwareOptionStub,
Tag: {
props: ['value'],
template: '<span data-testid="recommended-tag">{{ value }}</span>'
}
}
}
})
}
describe('GpuPicker', () => {
describe('recommended badge', () => {
it('shows recommended badge for nvidia', () => {
renderPicker('nvidia')
expect(screen.getByTestId('recommended-tag')).toBeVisible()
})
it('shows recommended badge for amd', () => {
renderPicker('amd')
expect(screen.getByTestId('recommended-tag')).toBeVisible()
})
it('does not show recommended badge for cpu', () => {
renderPicker('cpu')
expect(screen.getByTestId('recommended-tag')).not.toBeVisible()
})
it('does not show recommended badge for unsupported', () => {
renderPicker('unsupported')
expect(screen.getByTestId('recommended-tag')).not.toBeVisible()
})
it('does not show recommended badge when no device is selected', () => {
renderPicker(null)
expect(screen.getByTestId('recommended-tag')).not.toBeVisible()
})
})
describe('selection state', () => {
it('marks nvidia as selected when device is nvidia', () => {
renderPicker('nvidia')
expect(screen.getByTestId('NVIDIA').dataset.selected).toBe('true')
})
it('marks cpu as selected when device is cpu', () => {
renderPicker('cpu')
expect(screen.getByTestId('CPU').dataset.selected).toBe('true')
})
it('marks unsupported as selected when device is unsupported', () => {
renderPicker('unsupported')
expect(screen.getByTestId('Manual Install').dataset.selected).toBe('true')
})
it('no option is selected when device is null', () => {
renderPicker(null)
expect(screen.getByTestId('CPU').dataset.selected).toBe('false')
expect(screen.getByTestId('NVIDIA').dataset.selected).toBe('false')
})
})
describe('gpu options on non-darwin platform', () => {
it('shows NVIDIA, AMD, CPU, and Manual Install options', () => {
renderPicker(null)
expect(screen.getByTestId('NVIDIA')).toBeDefined()
expect(screen.getByTestId('AMD')).toBeDefined()
expect(screen.getByTestId('CPU')).toBeDefined()
expect(screen.getByTestId('Manual Install')).toBeDefined()
})
})
})

View File

@@ -4,7 +4,7 @@
<button
:class="
cn(
'hardware-option flex h-[190px] w-[170px] flex-col items-center rounded-3xl border-4 bg-neutral-900/70 p-5 transition-all duration-200',
'hardware-option w-[170px] h-[190px] p-5 flex flex-col items-center rounded-3xl transition-all duration-200 bg-neutral-900/70 border-4',
selected ? 'border-solid border-brand-yellow' : 'border-transparent'
)
"
@@ -12,13 +12,13 @@
>
<!-- Icon/Logo Area - Rounded square container -->
<div
class="icon-container flex h-[110px] w-[110px] shrink-0 items-center justify-center overflow-hidden rounded-2xl bg-neutral-800"
class="icon-container w-[110px] h-[110px] shrink-0 rounded-2xl bg-neutral-800 flex items-center justify-center overflow-hidden"
>
<img
v-if="imagePath"
:src="imagePath"
:alt="placeholderText"
class="size-full object-cover"
class="w-full h-full object-cover"
style="object-position: 57% center"
draggable="false"
/>
@@ -28,7 +28,7 @@
</div>
<!-- Text Content -->
<div v-if="subtitle" class="mt-4 text-center">
<div v-if="subtitle" class="text-center mt-4">
<div class="text-sm text-neutral-500">{{ subtitle }}</div>
</div>
</button>
@@ -36,7 +36,7 @@
</template>
<script setup lang="ts">
import { cn } from '@comfyorg/tailwind-utils'
import { cn } from '@/utils/tailwindUtil'
interface Props {
imagePath?: string

View File

@@ -1,223 +0,0 @@
import { render, screen, waitFor } from '@testing-library/vue'
import userEvent from '@testing-library/user-event'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi, beforeEach } from 'vitest'
import { createI18n } from 'vue-i18n'
const mockValidateComfyUISource = vi.fn()
const mockShowDirectoryPicker = vi.fn()
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => ({
validateComfyUISource: mockValidateComfyUISource,
showDirectoryPicker: mockShowDirectoryPicker
}))
}))
import MigrationPicker from '@/components/install/MigrationPicker.vue'
const i18n = createI18n({
legacy: false,
locale: 'en',
messages: {
en: {
install: {
migrationSourcePathDescription: 'Source path description',
migrationOptional: 'Migration is optional',
selectItemsToMigrate: 'Select items to migrate',
pathValidationFailed: 'Validation failed',
failedToSelectDirectory: 'Failed to select directory',
locationPicker: {
migrationPathPlaceholder: 'Enter path'
}
}
}
}
})
const InputTextStub = {
props: ['modelValue', 'invalid'],
emits: ['update:modelValue'],
template: `<input
data-testid="source-input"
:value="modelValue"
@input="$emit('update:modelValue', $event.target.value)"
/>`
}
const CheckboxStub = {
props: ['modelValue', 'inputId', 'binary'],
emits: ['update:modelValue', 'click'],
template: `<input
type="checkbox"
:data-testid="'checkbox-' + inputId"
:checked="modelValue"
@change="$emit('update:modelValue', $event.target.checked)"
@click.stop="$emit('click')"
/>`
}
function renderPicker(sourcePath = '', migrationItemIds: string[] = []) {
return render(MigrationPicker, {
props: { sourcePath, migrationItemIds },
global: {
plugins: [[PrimeVue, { unstyled: true }], i18n],
stubs: {
InputText: InputTextStub,
Checkbox: CheckboxStub,
Button: { template: '<button data-testid="browse-btn" />' },
Message: {
props: ['severity'],
template: '<div data-testid="error-msg"><slot /></div>'
}
}
}
})
}
describe('MigrationPicker', () => {
beforeEach(() => {
vi.resetAllMocks()
})
describe('isValidSource', () => {
it('hides migration options when source path is empty', () => {
renderPicker('')
expect(screen.queryByText('Select items to migrate')).toBeNull()
})
it('shows migration options when source path is valid', async () => {
mockValidateComfyUISource.mockResolvedValue({ isValid: true })
const { rerender } = renderPicker('')
await rerender({ sourcePath: '/valid/path' })
await waitFor(() => {
expect(screen.getByText('Select items to migrate')).toBeDefined()
})
})
it('shows optional message when no valid source', () => {
renderPicker('')
expect(screen.getByText('Migration is optional')).toBeDefined()
})
})
describe('validateSource', () => {
it('clears error when source path becomes empty', async () => {
mockValidateComfyUISource.mockResolvedValue({
isValid: false,
error: 'Not found'
})
const user = userEvent.setup()
renderPicker()
await user.type(screen.getByTestId('source-input'), '/bad/path')
await waitFor(() => {
expect(screen.getByTestId('error-msg')).toBeDefined()
})
await user.clear(screen.getByTestId('source-input'))
await waitFor(() => {
expect(screen.queryByTestId('error-msg')).toBeNull()
})
})
it('shows error message when validation fails', async () => {
mockValidateComfyUISource.mockResolvedValue({
isValid: false,
error: 'Path not found'
})
const user = userEvent.setup()
renderPicker()
await user.type(screen.getByTestId('source-input'), '/bad/path')
await waitFor(() => {
expect(screen.getByTestId('error-msg')).toBeDefined()
})
})
it('shows no error when validation passes', async () => {
mockValidateComfyUISource.mockResolvedValue({ isValid: true })
const user = userEvent.setup()
renderPicker()
await user.type(screen.getByTestId('source-input'), '/valid/path')
await waitFor(() => {
expect(screen.queryByTestId('error-msg')).toBeNull()
})
})
})
describe('migrationItemIds watchEffect', () => {
it('emits all item IDs by default (all items start selected)', async () => {
const onUpdate = vi.fn()
render(MigrationPicker, {
props: {
sourcePath: '',
migrationItemIds: [],
'onUpdate:migrationItemIds': onUpdate
},
global: {
plugins: [[PrimeVue, { unstyled: true }], i18n],
stubs: {
InputText: InputTextStub,
Checkbox: CheckboxStub,
Button: { template: '<button />' },
Message: { template: '<div />' }
}
}
})
await waitFor(() => {
expect(onUpdate).toHaveBeenCalled()
const emittedIds = onUpdate.mock.calls[0][0]
expect(Array.isArray(emittedIds)).toBe(true)
expect(emittedIds.length).toBeGreaterThan(0)
})
})
})
describe('browse path', () => {
it('opens directory picker on browse click', async () => {
mockShowDirectoryPicker.mockResolvedValue(null)
renderPicker()
const user = userEvent.setup()
await user.click(screen.getByTestId('browse-btn'))
expect(mockShowDirectoryPicker).toHaveBeenCalledOnce()
})
it('updates source path when directory is selected', async () => {
mockShowDirectoryPicker.mockResolvedValue('/selected/path')
mockValidateComfyUISource.mockResolvedValue({ isValid: true })
const onUpdate = vi.fn()
render(MigrationPicker, {
props: {
sourcePath: '',
'onUpdate:sourcePath': onUpdate
},
global: {
plugins: [[PrimeVue, { unstyled: true }], i18n],
stubs: {
InputText: InputTextStub,
Checkbox: CheckboxStub,
Button: { template: '<button data-testid="browse-btn" />' },
Message: { template: '<div />' }
}
}
})
const user = userEvent.setup()
await user.click(screen.getByTestId('browse-btn'))
await waitFor(() => {
expect(onUpdate).toHaveBeenCalledWith('/selected/path')
})
})
})
})

View File

@@ -1,85 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi } from 'vitest'
import { defineComponent } from 'vue'
vi.mock('@/i18n', () => ({
t: (key: string) => key
}))
import StatusTag from '@/components/maintenance/StatusTag.vue'
const TagStub = defineComponent({
name: 'Tag',
props: {
icon: String,
severity: String,
value: String
},
template: `<span data-testid="tag" :data-icon="icon" :data-severity="severity" :data-value="value">{{ value }}</span>`
})
function renderStatusTag(props: { error: boolean; refreshing?: boolean }) {
return render(StatusTag, {
props,
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: { Tag: TagStub }
}
})
}
describe('StatusTag', () => {
describe('refreshing state', () => {
it('shows info severity when refreshing', () => {
renderStatusTag({ error: false, refreshing: true })
expect(screen.getByTestId('tag').dataset.severity).toBe('info')
})
it('shows refreshing translation key when refreshing', () => {
renderStatusTag({ error: false, refreshing: true })
expect(screen.getByTestId('tag').dataset.value).toBe(
'maintenance.refreshing'
)
})
it('shows question icon when refreshing', () => {
renderStatusTag({ error: false, refreshing: true })
expect(screen.getByTestId('tag').dataset.icon).toBeDefined()
})
})
describe('error state', () => {
it('shows danger severity when error is true', () => {
renderStatusTag({ error: true })
expect(screen.getByTestId('tag').dataset.severity).toBe('danger')
})
it('shows error translation key when error is true', () => {
renderStatusTag({ error: true })
expect(screen.getByTestId('tag').dataset.value).toBe('g.error')
})
})
describe('OK state', () => {
it('shows success severity when not refreshing and not error', () => {
renderStatusTag({ error: false })
expect(screen.getByTestId('tag').dataset.severity).toBe('success')
})
it('shows OK translation key when not refreshing and not error', () => {
renderStatusTag({ error: false })
expect(screen.getByTestId('tag').dataset.value).toBe('maintenance.OK')
})
})
describe('precedence', () => {
it('shows refreshing state when both refreshing and error are true', () => {
renderStatusTag({ error: true, refreshing: true })
expect(screen.getByTestId('tag').dataset.severity).toBe('info')
expect(screen.getByTestId('tag').dataset.value).toBe(
'maintenance.refreshing'
)
})
})
})

View File

@@ -1,89 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi } from 'vitest'
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => ({
Validation: { validateInstallation: vi.fn() }
}))
}))
vi.mock('@/constants/desktopMaintenanceTasks', () => ({
DESKTOP_MAINTENANCE_TASKS: []
}))
vi.mock('@/utils/refUtil', () => ({
useMinLoadingDurationRef: (source: { value: boolean }) => source
}))
const mockGetRunner = vi.fn()
vi.mock('@/stores/maintenanceTaskStore', () => ({
useMaintenanceTaskStore: vi.fn(() => ({
getRunner: mockGetRunner
}))
}))
import type { MaintenanceTask } from '@/types/desktop/maintenanceTypes'
import TaskCard from '@/components/maintenance/TaskCard.vue'
const baseTask: MaintenanceTask = {
id: 'testTask',
name: 'Test Task',
shortDescription: 'Short description',
errorDescription: 'Error occurred',
execute: vi.fn().mockResolvedValue(true)
}
const cardStubs = {
Card: {
template: '<div data-testid="card"><slot name="content"></slot></div>'
},
Button: { template: '<button />' }
}
function renderCard(
state: 'OK' | 'error' | 'warning' | 'skipped',
task: MaintenanceTask = baseTask
) {
mockGetRunner.mockReturnValue({
state,
executing: false,
refreshing: false,
resolved: false
})
return render(TaskCard, {
props: { task },
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: cardStubs
}
})
}
describe('TaskCard', () => {
describe('description computed', () => {
it('shows errorDescription when task state is error', () => {
renderCard('error')
expect(screen.getByText('Error occurred')).toBeDefined()
})
it('shows shortDescription when task state is OK', () => {
renderCard('OK')
expect(screen.getByText('Short description')).toBeDefined()
})
it('shows shortDescription when task state is warning', () => {
renderCard('warning')
expect(screen.getByText('Short description')).toBeDefined()
})
it('falls back to shortDescription when errorDescription is absent and state is error', () => {
const taskWithoutErrorDesc: MaintenanceTask = {
...baseTask,
errorDescription: undefined
}
renderCard('error', taskWithoutErrorDesc)
expect(screen.getByText('Short description')).toBeDefined()
})
})
})

View File

@@ -64,7 +64,7 @@ import { computed } from 'vue'
import { useMaintenanceTaskStore } from '@/stores/maintenanceTaskStore'
import type { MaintenanceTask } from '@/types/desktop/maintenanceTypes'
import { cn } from '@comfyorg/tailwind-utils'
import { cn } from '@/utils/tailwindUtil'
import { useMinLoadingDurationRef } from '@/utils/refUtil'
const taskStore = useMaintenanceTaskStore()

View File

@@ -1,97 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi } from 'vitest'
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => ({
Validation: { validateInstallation: vi.fn() }
}))
}))
vi.mock('@/constants/desktopMaintenanceTasks', () => ({
DESKTOP_MAINTENANCE_TASKS: []
}))
vi.mock('@/utils/refUtil', () => ({
useMinLoadingDurationRef: (source: { value: boolean }) => source
}))
vi.mock('@/i18n', () => ({
t: (key: string) => key
}))
const mockGetRunner = vi.fn()
vi.mock('@/stores/maintenanceTaskStore', () => ({
useMaintenanceTaskStore: vi.fn(() => ({
getRunner: mockGetRunner
}))
}))
import type { MaintenanceTask } from '@/types/desktop/maintenanceTypes'
import TaskListItem from '@/components/maintenance/TaskListItem.vue'
const baseTask: MaintenanceTask = {
id: 'testTask',
name: 'Test Task',
button: { text: 'Fix', icon: 'pi pi-check' },
execute: vi.fn().mockResolvedValue(true)
}
const ButtonStub = {
props: ['severity', 'label', 'icon', 'loading'],
template:
'<button :data-severity="severity" :data-label="label" :data-testid="label ? \'action-button\' : \'icon-button\'" />'
}
function renderItem(state: 'OK' | 'error' | 'warning' | 'skipped') {
mockGetRunner.mockReturnValue({
state,
executing: false,
refreshing: false,
resolved: false
})
return render(TaskListItem, {
props: { task: baseTask },
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
Button: ButtonStub,
Popover: { template: '<div />' },
TaskListStatusIcon: { template: '<span />' }
}
}
})
}
describe('TaskListItem', () => {
describe('severity computed', () => {
it('uses primary severity for error state', () => {
renderItem('error')
expect(screen.getByTestId('action-button').dataset.severity).toBe(
'primary'
)
})
it('uses primary severity for warning state', () => {
renderItem('warning')
expect(screen.getByTestId('action-button').dataset.severity).toBe(
'primary'
)
})
it('uses secondary severity for OK state', () => {
renderItem('OK')
expect(screen.getByTestId('action-button').dataset.severity).toBe(
'secondary'
)
})
it('uses secondary severity for skipped state', () => {
renderItem('skipped')
expect(screen.getByTestId('action-button').dataset.severity).toBe(
'secondary'
)
})
})
})

View File

@@ -1,44 +0,0 @@
import { render, screen } from '@testing-library/vue'
import PrimeVue from 'primevue/config'
import { describe, expect, it, vi } from 'vitest'
vi.mock('@/i18n', () => ({
t: (key: string) => key
}))
import TaskListStatusIcon from '@/components/maintenance/TaskListStatusIcon.vue'
type TaskState = 'warning' | 'error' | 'resolved' | 'OK' | 'skipped' | undefined
function renderIcon(state: TaskState, loading?: boolean) {
return render(TaskListStatusIcon, {
props: { state, loading },
global: {
plugins: [[PrimeVue, { unstyled: true }]],
stubs: {
ProgressSpinner: {
template: '<div data-testid="spinner" />'
}
}
}
})
}
describe('TaskListStatusIcon', () => {
describe('loading / no state', () => {
it('renders spinner when state is undefined', () => {
renderIcon(undefined)
expect(screen.getByTestId('spinner')).toBeDefined()
})
it('renders spinner when loading is true', () => {
renderIcon('OK', true)
expect(screen.getByTestId('spinner')).toBeDefined()
})
it('hides spinner when state is defined and not loading', () => {
renderIcon('OK', false)
expect(screen.queryByTestId('spinner')).toBeNull()
})
})
})

View File

@@ -1,124 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { ref } from 'vue'
const { mockTerminal, MockTerminal, mockFitAddon, MockFitAddon } = vi.hoisted(
() => {
const mockTerminal = {
loadAddon: vi.fn(),
attachCustomKeyEventHandler: vi.fn(),
open: vi.fn(),
dispose: vi.fn(),
hasSelection: vi.fn<[], boolean>(),
resize: vi.fn(),
cols: 80,
rows: 24
}
const MockTerminal = vi.fn(function () {
return mockTerminal
})
const mockFitAddon = {
proposeDimensions: vi.fn().mockReturnValue({ cols: 80, rows: 24 })
}
const MockFitAddon = vi.fn(function () {
return mockFitAddon
})
return { mockTerminal, MockTerminal, mockFitAddon, MockFitAddon }
}
)
vi.mock('@xterm/xterm', () => ({ Terminal: MockTerminal }))
vi.mock('@xterm/addon-fit', () => ({ FitAddon: MockFitAddon }))
vi.mock('@xterm/xterm/css/xterm.css', () => ({}))
import { withSetup } from '@/test/withSetup'
import { useTerminal } from '@/composables/bottomPanelTabs/useTerminal'
function getKeyHandler(): (event: KeyboardEvent) => boolean {
return mockTerminal.attachCustomKeyEventHandler.mock.calls[0][0]
}
describe('useTerminal key event handler', () => {
beforeEach(() => {
vi.clearAllMocks()
mockTerminal.hasSelection.mockReturnValue(false)
const element = ref<HTMLElement | undefined>(undefined)
withSetup(() => useTerminal(element))
})
it('allows browser to handle copy when text is selected (Ctrl+C)', () => {
mockTerminal.hasSelection.mockReturnValue(true)
const event = {
type: 'keydown',
ctrlKey: true,
metaKey: false,
key: 'c'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(false)
})
it('allows browser to handle copy when text is selected (Meta+C)', () => {
mockTerminal.hasSelection.mockReturnValue(true)
const event = {
type: 'keydown',
ctrlKey: false,
metaKey: true,
key: 'c'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(false)
})
it('does not pass copy to browser when no text is selected', () => {
mockTerminal.hasSelection.mockReturnValue(false)
const event = {
type: 'keydown',
ctrlKey: true,
metaKey: false,
key: 'c'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(true)
})
it('allows browser to handle paste (Ctrl+V)', () => {
const event = {
type: 'keydown',
ctrlKey: true,
metaKey: false,
key: 'v'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(false)
})
it('allows browser to handle paste (Meta+V)', () => {
const event = {
type: 'keydown',
ctrlKey: false,
metaKey: true,
key: 'v'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(false)
})
it('does not intercept non-keydown events', () => {
mockTerminal.hasSelection.mockReturnValue(true)
const event = {
type: 'keyup',
ctrlKey: true,
metaKey: false,
key: 'c'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(true)
})
it('passes through unrelated key combinations', () => {
const event = {
type: 'keydown',
ctrlKey: false,
metaKey: false,
key: 'Enter'
} as KeyboardEvent
expect(getKeyHandler()(event)).toBe(true)
})
})

View File

@@ -1,48 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockSerialize, MockSerializeAddon } = vi.hoisted(() => {
const mockSerialize = vi.fn<[], string>()
const MockSerializeAddon = vi.fn(function () {
return { serialize: mockSerialize }
})
return { mockSerialize, MockSerializeAddon }
})
vi.mock('@xterm/xterm', () => ({
Terminal: vi.fn(function () {
return { loadAddon: vi.fn(), dispose: vi.fn(), write: vi.fn() }
})
}))
vi.mock('@xterm/addon-serialize', () => ({
SerializeAddon: MockSerializeAddon
}))
import type { Terminal } from '@xterm/xterm'
import { withSetup } from '@/test/withSetup'
import { useTerminalBuffer } from '@/composables/bottomPanelTabs/useTerminalBuffer'
describe('useTerminalBuffer', () => {
beforeEach(() => {
vi.clearAllMocks()
mockSerialize.mockReturnValue('')
})
describe('copyTo', () => {
it('writes serialized buffer content to the destination terminal', () => {
mockSerialize.mockReturnValue('hello world')
const { copyTo } = withSetup(() => useTerminalBuffer())
const mockWrite = vi.fn()
copyTo({ write: mockWrite } as Pick<Terminal, 'write'>)
expect(mockWrite).toHaveBeenCalledWith('hello world')
})
it('writes empty string when buffer is empty', () => {
mockSerialize.mockReturnValue('')
const { copyTo } = withSetup(() => useTerminalBuffer())
const mockWrite = vi.fn()
copyTo({ write: mockWrite } as Pick<Terminal, 'write'>)
expect(mockWrite).toHaveBeenCalledWith('')
})
})
})

View File

@@ -1,50 +0,0 @@
import { describe, expect, it } from 'vitest'
import { DESKTOP_DIALOGS, getDialog } from '@/constants/desktopDialogs'
describe('getDialog', () => {
it('returns the matching dialog for a valid ID', () => {
const result = getDialog('reinstallVenv')
expect(result.id).toBe('reinstallVenv')
expect(result.title).toBe(DESKTOP_DIALOGS.reinstallVenv.title)
expect(result.message).toBe(DESKTOP_DIALOGS.reinstallVenv.message)
})
it('returns invalidDialog for an unknown string ID', () => {
const result = getDialog('unknownDialog')
expect(result.id).toBe('invalidDialog')
})
it('returns invalidDialog when given an array of strings', () => {
const result = getDialog(['reinstallVenv', 'other'])
expect(result.id).toBe('invalidDialog')
})
it('returns invalidDialog for empty string', () => {
const result = getDialog('')
expect(result.id).toBe('invalidDialog')
})
it('returns a deep clone — mutations do not affect the original', () => {
const result = getDialog('reinstallVenv')
const originalFirstLabel = DESKTOP_DIALOGS.reinstallVenv.buttons[0].label
result.buttons[0].label = 'Mutated'
expect(DESKTOP_DIALOGS.reinstallVenv.buttons[0].label).toBe(
originalFirstLabel
)
})
it('every button has a returnValue', () => {
for (const id of Object.keys(DESKTOP_DIALOGS)) {
const result = getDialog(id)
for (const button of result.buttons) {
expect(button.returnValue).toBeDefined()
}
}
})
it('invalidDialog has a close/cancel button', () => {
const result = getDialog('invalidDialog')
expect(result.buttons.some((b) => b.action === 'cancel')).toBe(true)
})
})

View File

@@ -1,75 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockElectron } = vi.hoisted(() => ({
mockElectron: {
setBasePath: vi.fn(),
reinstall: vi.fn<[], Promise<void>>().mockResolvedValue(undefined),
uv: {
installRequirements: vi.fn<[], Promise<void>>(),
clearCache: vi.fn<[], Promise<void>>().mockResolvedValue(undefined),
resetVenv: vi.fn<[], Promise<void>>().mockResolvedValue(undefined)
}
}
}))
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => mockElectron)
}))
import { DESKTOP_MAINTENANCE_TASKS } from '@/constants/desktopMaintenanceTasks'
function findTask(id: string) {
const task = DESKTOP_MAINTENANCE_TASKS.find((t) => t.id === id)
if (!task) throw new Error(`Task not found: ${id}`)
return task
}
describe('desktopMaintenanceTasks', () => {
beforeEach(() => {
vi.resetAllMocks()
vi.spyOn(window, 'open').mockReturnValue(null)
mockElectron.reinstall.mockResolvedValue(undefined)
mockElectron.uv.clearCache.mockResolvedValue(undefined)
mockElectron.uv.resetVenv.mockResolvedValue(undefined)
})
describe('pythonPackages', () => {
it('returns true when installation succeeds', async () => {
mockElectron.uv.installRequirements.mockResolvedValue(undefined)
expect(await findTask('pythonPackages').execute()).toBe(true)
})
it('returns false when installation throws', async () => {
mockElectron.uv.installRequirements.mockRejectedValue(
new Error('install failed')
)
expect(await findTask('pythonPackages').execute()).toBe(false)
})
})
describe('URL-opening tasks', () => {
it('git execute opens the git download page', () => {
findTask('git').execute()
expect(window.open).toHaveBeenCalledWith(
'https://git-scm.com/downloads/',
'_blank'
)
})
it('uv execute opens the uv installation page', () => {
findTask('uv').execute()
expect(window.open).toHaveBeenCalledWith(
'https://docs.astral.sh/uv/getting-started/installation/',
'_blank'
)
})
it('vcRedist execute opens the VC++ redistributable download', () => {
findTask('vcRedist').execute()
expect(window.open).toHaveBeenCalledWith(
'https://aka.ms/vs/17/release/vc_redist.x64.exe',
'_blank'
)
})
})
})

View File

@@ -1,288 +0,0 @@
import { createTestingPinia } from '@pinia/testing'
import type { InstallValidation } from '@comfyorg/comfyui-electron-types'
import { setActivePinia } from 'pinia'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockElectron, testTasks } = vi.hoisted(() => {
const terminalTaskExecute = vi.fn().mockResolvedValue(true)
const basicTaskExecute = vi.fn().mockResolvedValue(true)
return {
mockElectron: {
Validation: {
validateInstallation: vi.fn()
}
},
testTasks: [
{
id: 'basicTask',
name: 'Basic Task',
execute: basicTaskExecute
},
{
id: 'terminalTask',
name: 'Terminal Task',
execute: terminalTaskExecute,
usesTerminal: true,
isInstallationFix: true
}
]
}
})
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => mockElectron)
}))
vi.mock('@/constants/desktopMaintenanceTasks', () => ({
DESKTOP_MAINTENANCE_TASKS: testTasks
}))
import { useMaintenanceTaskStore } from '@/stores/maintenanceTaskStore'
import type { MaintenanceTask } from '@/types/desktop/maintenanceTypes'
type PartialInstallValidation = Partial<InstallValidation> &
Record<string, unknown>
function makeUpdate(
overrides: PartialInstallValidation = {}
): InstallValidation {
return {
inProgress: false,
installState: 'installed',
...overrides
} as InstallValidation
}
function createStore() {
setActivePinia(createTestingPinia({ stubActions: false }))
return useMaintenanceTaskStore()
}
describe('useMaintenanceTaskStore', () => {
let store: ReturnType<typeof useMaintenanceTaskStore>
const [basicTask, terminalTask] = testTasks as MaintenanceTask[]
beforeEach(() => {
vi.resetAllMocks()
store = createStore()
})
describe('processUpdate', () => {
it('sets isRefreshing to true during in-progress update', () => {
store.processUpdate(makeUpdate({ inProgress: true }))
expect(store.isRefreshing).toBe(true)
})
it('sets isRefreshing to false when update is complete', () => {
store.processUpdate(makeUpdate({ inProgress: false, basicTask: 'OK' }))
expect(store.isRefreshing).toBe(false)
})
it('updates runner state for tasks present in the final update', () => {
store.processUpdate(makeUpdate({ basicTask: 'error' }))
expect(store.getRunner(basicTask).state).toBe('error')
})
it('sets task state to warning from update', () => {
store.processUpdate(makeUpdate({ basicTask: 'warning' }))
expect(store.getRunner(basicTask).state).toBe('warning')
})
it('marks runners as refreshing when task id is absent from in-progress update', () => {
store.processUpdate(makeUpdate({ inProgress: true }))
expect(store.getRunner(basicTask).refreshing).toBe(true)
})
it('marks task as skipped when absent from final update', () => {
store.processUpdate(makeUpdate({ inProgress: false }))
expect(store.getRunner(basicTask).state).toBe('skipped')
})
it('clears refreshing flag after final update', () => {
store.processUpdate(makeUpdate({ inProgress: true }))
store.processUpdate(makeUpdate({ inProgress: false }))
expect(store.getRunner(basicTask).refreshing).toBe(false)
})
it('stores lastUpdate and exposes unsafeBasePath', () => {
store.processUpdate(makeUpdate({ unsafeBasePath: true }))
expect(store.unsafeBasePath).toBe(true)
})
it('exposes unsafeBasePathReason from the update', () => {
store.processUpdate(
makeUpdate({ unsafeBasePath: true, unsafeBasePathReason: 'oneDrive' })
)
expect(store.unsafeBasePathReason).toBe('oneDrive')
})
})
describe('anyErrors', () => {
it('returns true when any task has error state', () => {
store.processUpdate(makeUpdate({ basicTask: 'error' }))
expect(store.anyErrors).toBe(true)
})
it('returns false when all tasks are OK', () => {
store.processUpdate(makeUpdate({ basicTask: 'OK', terminalTask: 'OK' }))
expect(store.anyErrors).toBe(false)
})
it('returns false when all tasks are warning', () => {
store.processUpdate(
makeUpdate({ basicTask: 'warning', terminalTask: 'warning' })
)
expect(store.anyErrors).toBe(false)
})
})
describe('runner state transitions', () => {
it('marks runner as resolved when transitioning from error to OK', () => {
store.processUpdate(makeUpdate({ basicTask: 'error' }))
store.processUpdate(makeUpdate({ basicTask: 'OK' }))
expect(store.getRunner(basicTask).resolved).toBe(true)
})
it('does not mark resolved for warning to OK transition', () => {
store.processUpdate(makeUpdate({ basicTask: 'warning' }))
store.processUpdate(makeUpdate({ basicTask: 'OK' }))
expect(store.getRunner(basicTask).resolved).toBeFalsy()
})
it('clears resolved flag when task returns to error', () => {
store.processUpdate(makeUpdate({ basicTask: 'error' }))
store.processUpdate(makeUpdate({ basicTask: 'OK' }))
store.processUpdate(makeUpdate({ basicTask: 'error' }))
expect(store.getRunner(basicTask).resolved).toBeFalsy()
})
})
describe('clearResolved', () => {
it('clears resolved flags on all runners', () => {
store.processUpdate(makeUpdate({ basicTask: 'error' }))
store.processUpdate(makeUpdate({ basicTask: 'OK' }))
expect(store.getRunner(basicTask).resolved).toBe(true)
store.clearResolved()
expect(store.getRunner(basicTask).resolved).toBeFalsy()
})
})
describe('execute', () => {
it('returns true when task execution succeeds', async () => {
vi.mocked(basicTask.execute).mockResolvedValue(true)
const result = await store.execute(basicTask)
expect(result).toBe(true)
})
it('returns false when task execution fails', async () => {
vi.mocked(basicTask.execute).mockResolvedValue(false)
const result = await store.execute(basicTask)
expect(result).toBe(false)
})
it('calls refreshDesktopTasks after successful installation-fix task', async () => {
vi.mocked(terminalTask.execute).mockResolvedValue(true)
await store.execute(terminalTask)
expect(
mockElectron.Validation.validateInstallation
).toHaveBeenCalledOnce()
})
it('does not call refreshDesktopTasks when task is not an installation fix', async () => {
vi.mocked(basicTask.execute).mockResolvedValue(true)
await store.execute(basicTask)
expect(
mockElectron.Validation.validateInstallation
).not.toHaveBeenCalled()
})
it('does not call refreshDesktopTasks when installation-fix task fails', async () => {
vi.mocked(terminalTask.execute).mockResolvedValue(false)
await store.execute(terminalTask)
expect(
mockElectron.Validation.validateInstallation
).not.toHaveBeenCalled()
})
it('sets runner executing to true during task execution', async () => {
let resolveTask!: (value: boolean) => void
vi.mocked(basicTask.execute).mockReturnValue(
new Promise<boolean>((resolve) => {
resolveTask = resolve
})
)
const executePromise = store.execute(basicTask)
expect(store.getRunner(basicTask).executing).toBe(true)
resolveTask(true)
await executePromise
expect(store.getRunner(basicTask).executing).toBe(false)
})
it('clears executing flag when task throws', async () => {
vi.mocked(basicTask.execute).mockRejectedValue(new Error('fail'))
await expect(store.execute(basicTask)).rejects.toThrow('fail')
expect(store.getRunner(basicTask).executing).toBe(false)
})
it('sets runner error message when task throws', async () => {
vi.mocked(basicTask.execute).mockRejectedValue(
new Error('something broke')
)
await expect(store.execute(basicTask)).rejects.toThrow()
expect(store.getRunner(basicTask).error).toBe('something broke')
})
it('clears runner error on successful execution', async () => {
vi.mocked(basicTask.execute).mockRejectedValue(new Error('fail'))
await expect(store.execute(basicTask)).rejects.toThrow()
vi.mocked(basicTask.execute).mockResolvedValue(true)
await store.execute(basicTask)
expect(store.getRunner(basicTask).error).toBeUndefined()
})
})
describe('isRunningTerminalCommand', () => {
it('returns true while a terminal task is executing', async () => {
let resolveTask!: (value: boolean) => void
vi.mocked(terminalTask.execute).mockReturnValue(
new Promise<boolean>((resolve) => {
resolveTask = resolve
})
)
const executePromise = store.execute(terminalTask)
expect(store.isRunningTerminalCommand).toBe(true)
resolveTask(true)
await executePromise
expect(store.isRunningTerminalCommand).toBe(false)
})
it('returns false when no terminal tasks are executing', () => {
expect(store.isRunningTerminalCommand).toBe(false)
})
})
describe('isRunningInstallationFix', () => {
it('returns true while an installation-fix task is executing', async () => {
let resolveTask!: (value: boolean) => void
vi.mocked(terminalTask.execute).mockReturnValue(
new Promise<boolean>((resolve) => {
resolveTask = resolve
})
)
const executePromise = store.execute(terminalTask)
expect(store.isRunningInstallationFix).toBe(true)
resolveTask(true)
await executePromise
expect(store.isRunningInstallationFix).toBe(false)
})
})
})

View File

@@ -1 +0,0 @@
import '@testing-library/jest-dom/vitest'

View File

@@ -1,16 +0,0 @@
import { render } from '@testing-library/vue'
import { defineComponent } from 'vue'
export function withSetup<T>(composable: () => T): T {
let result!: T
render(
defineComponent({
setup() {
result = composable()
return {}
},
template: '<div />'
})
)
return result
}

View File

@@ -1,52 +0,0 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockElectron } = vi.hoisted(() => ({
mockElectron: {
NetWork: {
canAccessUrl: vi.fn<[url: string], Promise<boolean>>()
}
}
}))
vi.mock('@/utils/envUtil', () => ({
electronAPI: vi.fn(() => mockElectron)
}))
import { checkMirrorReachable } from '@/utils/electronMirrorCheck'
describe('checkMirrorReachable', () => {
beforeEach(() => {
vi.clearAllMocks()
})
it('returns false for an invalid URL without calling canAccessUrl', async () => {
const result = await checkMirrorReachable('not-a-url')
expect(result).toBe(false)
expect(mockElectron.NetWork.canAccessUrl).not.toHaveBeenCalled()
})
it('returns false when canAccessUrl returns false', async () => {
mockElectron.NetWork.canAccessUrl.mockResolvedValue(false)
const result = await checkMirrorReachable('https://example.com')
expect(result).toBe(false)
})
it('returns true when URL is valid and canAccessUrl returns true', async () => {
mockElectron.NetWork.canAccessUrl.mockResolvedValue(true)
const result = await checkMirrorReachable('https://example.com')
expect(result).toBe(true)
})
it('passes the mirror URL to canAccessUrl', async () => {
const url = 'https://pypi.org/simple/'
mockElectron.NetWork.canAccessUrl.mockResolvedValue(true)
await checkMirrorReachable(url)
expect(mockElectron.NetWork.canAccessUrl).toHaveBeenCalledWith(url)
})
it('returns false for empty string', async () => {
const result = await checkMirrorReachable('')
expect(result).toBe(false)
expect(mockElectron.NetWork.canAccessUrl).not.toHaveBeenCalled()
})
})

View File

@@ -1,72 +0,0 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { isElectron, isNativeWindow } from '@/utils/envUtil'
describe('isElectron', () => {
afterEach(() => {
vi.unstubAllGlobals()
})
it('returns true when window.electronAPI is an object', () => {
vi.stubGlobal('window', { ...window, electronAPI: {} })
expect(isElectron()).toBe(true)
})
it('returns false when window.electronAPI is undefined', () => {
vi.stubGlobal('window', { ...window, electronAPI: undefined })
expect(isElectron()).toBe(false)
})
it('returns false when window.electronAPI is absent', () => {
const copy = { ...window } as Record<string, unknown>
delete copy['electronAPI']
vi.stubGlobal('window', copy)
expect(isElectron()).toBe(false)
})
})
describe('isNativeWindow', () => {
afterEach(() => {
vi.unstubAllGlobals()
})
it('returns true when Electron and windowControlsOverlay.visible is true', () => {
vi.stubGlobal('window', {
...window,
electronAPI: {},
navigator: {
...window.navigator,
windowControlsOverlay: { visible: true }
}
})
expect(isNativeWindow()).toBe(true)
})
it('returns false when not in Electron', () => {
const copy = { ...window } as Record<string, unknown>
delete copy['electronAPI']
vi.stubGlobal('window', copy)
expect(isNativeWindow()).toBe(false)
})
it('returns false when windowControlsOverlay.visible is false', () => {
vi.stubGlobal('window', {
...window,
electronAPI: {},
navigator: {
...window.navigator,
windowControlsOverlay: { visible: false }
}
})
expect(isNativeWindow()).toBe(false)
})
it('returns false when windowControlsOverlay is absent', () => {
vi.stubGlobal('window', {
...window,
electronAPI: {},
navigator: { ...window.navigator, windowControlsOverlay: undefined }
})
expect(isNativeWindow()).toBe(false)
})
})

View File

@@ -1,102 +0,0 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { computed, nextTick, ref } from 'vue'
import { withSetup } from '@/test/withSetup'
import { useMinLoadingDurationRef } from '@/utils/refUtil'
describe('useMinLoadingDurationRef', () => {
beforeEach(() => {
vi.useFakeTimers()
})
afterEach(() => {
vi.useRealTimers()
})
it('reflects false when source is initially false', () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source))
expect(result.value).toBe(false)
})
it('reflects true when source is initially true', () => {
const source = ref(true)
const result = withSetup(() => useMinLoadingDurationRef(source))
expect(result.value).toBe(true)
})
it('becomes true immediately when source transitions to true', async () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source))
source.value = true
await nextTick()
expect(result.value).toBe(true)
})
it('stays true within minDuration after source returns to false', async () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source, 250))
source.value = true
await nextTick()
source.value = false
await nextTick()
vi.advanceTimersByTime(100)
await nextTick()
expect(result.value).toBe(true)
})
it('becomes false after minDuration has elapsed', async () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source, 250))
source.value = true
await nextTick()
source.value = false
await nextTick()
vi.advanceTimersByTime(250)
await nextTick()
expect(result.value).toBe(false)
})
it('remains true while source is true even after minDuration elapses', async () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source, 250))
source.value = true
await nextTick()
vi.advanceTimersByTime(500)
await nextTick()
expect(result.value).toBe(true)
})
it('works with a computed ref as input', async () => {
const raw = ref(false)
const source = computed(() => raw.value)
const result = withSetup(() => useMinLoadingDurationRef(source))
raw.value = true
await nextTick()
expect(result.value).toBe(true)
})
it('uses 250ms as default minDuration', async () => {
const source = ref(false)
const result = withSetup(() => useMinLoadingDurationRef(source))
source.value = true
await nextTick()
source.value = false
await nextTick()
vi.advanceTimersByTime(249)
await nextTick()
expect(result.value).toBe(true)
vi.advanceTimersByTime(1)
await nextTick()
expect(result.value).toBe(false)
})
})

View File

@@ -0,0 +1 @@
export { cn } from '@comfyorg/tailwind-utils'

View File

@@ -13,8 +13,7 @@
"src/**/*.ts",
"src/**/*.vue",
"src/**/*.d.ts",
"vite.config.mts",
"vitest.config.mts"
"vite.config.mts"
],
"references": []
}

View File

@@ -1,22 +0,0 @@
import vue from '@vitejs/plugin-vue'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { defineConfig } from 'vitest/config'
const projectRoot = fileURLToPath(new URL('.', import.meta.url))
export default defineConfig({
plugins: [vue()],
resolve: {
alias: {
'@': path.resolve(projectRoot, 'src'),
'@frontend-locales': path.resolve(projectRoot, '../../src/locales')
}
},
test: {
globals: true,
environment: 'happy-dom',
include: ['src/**/*.{test,spec}.{ts,tsx}'],
setupFiles: ['./src/test/setup.ts']
}
})

View File

@@ -1,8 +1,2 @@
dist/
.astro/
test-results/
playwright-report/
results.json
# Platform-specific Playwright snapshots (CI runs Linux)
*-win32.png

View File

@@ -1,123 +0,0 @@
# @comfyorg/website
Marketing/brand website built with Astro + Vue.
## Ashby careers integration
`/careers` and `/zh-CN/careers` are rendered from Ashby's public job board
API at build time. Data flow:
1. `src/pages/careers.astro` awaits `fetchRolesForBuild()` during the
Astro build.
2. `src/utils/ashby.ts` calls
`GET https://api.ashbyhq.com/posting-api/job-board/{board}?includeCompensation=false`,
validates the envelope and each posting with Zod
(`src/utils/ashby.schema.ts`), and maps to the domain type in
`src/data/roles.ts`.
3. On any failure (network, HTTP 4xx/5xx, envelope schema drift),
the fetcher falls back to the committed JSON snapshot at
`src/data/ashby-roles.snapshot.json`.
4. `src/utils/ashby.ci.ts` emits GitHub Actions annotations and a
`$GITHUB_STEP_SUMMARY` block so stale fetches are visible on green
builds.
### Required environment variables
Both are build-time only. Never prefix with `PUBLIC_` (Astro would
inline that into the client bundle).
| Name | Purpose | Default (when unset) |
| ------------------------------ | --------------------------- | --------------------------------- |
| `WEBSITE_ASHBY_API_KEY` | Ashby API key (Basic auth) | Build uses the committed snapshot |
| `WEBSITE_ASHBY_JOB_BOARD_NAME` | Ashby public job board slug | Build uses the committed snapshot |
### CI wiring (manual step — required)
This repo's `.github/workflows/*.yaml` changes cannot be pushed by a
GitHub App. A maintainer must apply the following edits **once**:
**`.github/workflows/ci-website-build.yaml`** — pass the env into the
build step and run the unit tests before it:
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Setup frontend
uses: ./.github/actions/setup-frontend
- name: Run website unit tests
run: pnpm --filter @comfyorg/website test:unit
- name: Build website
env:
WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
WEBSITE_ASHBY_JOB_BOARD_NAME: ${{ vars.WEBSITE_ASHBY_JOB_BOARD_NAME || 'comfy-org' }}
run: pnpm --filter @comfyorg/website build
- name: Verify API key is not leaked into build output
env:
WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
run: |
set +x
if [ -z "${WEBSITE_ASHBY_API_KEY:-}" ]; then
echo "Secret not available in this run; skipping leak check."
exit 0
fi
# grep -rlF prints only file paths (never match content).
MATCHES=$(grep -rlF --exclude-dir=node_modules --null \
-e "$WEBSITE_ASHBY_API_KEY" apps/website/dist/ 2>/dev/null \
| tr '\0' '\n' || true)
if [ -n "$MATCHES" ]; then
echo "::error title=Ashby API key leaked into build output::$MATCHES"
exit 1
fi
```
**`.github/workflows/ci-vercel-website-preview.yaml`** — add the
two env vars to the top-level `env:` block so `vercel build` (both
`deploy-preview` and `deploy-production` jobs) sees them:
```yaml
env:
VERCEL_ORG_ID: ${{ secrets.VERCEL_WEBSITE_ORG_ID }}
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_WEBSITE_PROJECT_ID }}
VERCEL_TOKEN: ${{ secrets.VERCEL_WEBSITE_TOKEN }}
VERCEL_SCOPE: comfyui
WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
WEBSITE_ASHBY_JOB_BOARD_NAME: ${{ vars.WEBSITE_ASHBY_JOB_BOARD_NAME || 'comfy-org' }}
```
The secret must also be added to the Vercel project environment
(`vercel env add WEBSITE_ASHBY_API_KEY …` or via the Vercel UI) so
that `vercel build` in the preview job has access to it.
Fork PRs do not exercise this path: `ci-vercel-website-preview.yaml`
receives an empty `VERCEL_TOKEN` for forks and fails at `vercel pull`
before the build runs. Fork-safe PR interactions (the preview-URL
comment) are handled by `pr-vercel-website-preview.yaml`.
### Refreshing the snapshot
When a maintainer wants to update the committed snapshot (e.g. after
onboarding/offboarding roles):
```bash
WEBSITE_ASHBY_API_KEY=WEBSITE_ASHBY_JOB_BOARD_NAME=comfy-org \
pnpm --filter @comfyorg/website ashby:refresh-snapshot
git commit apps/website/src/data/ashby-roles.snapshot.json
```
The script exits non-zero on any non-fresh outcome so stale/empty
snapshots can't be accidentally committed.
## Scripts
- `pnpm dev` — Astro dev server
- `pnpm build` — production build to `dist/`
- `pnpm typecheck``astro check`
- `pnpm test:unit` — Vitest unit tests
- `pnpm test:e2e` — Playwright E2E tests (requires `pnpm build` first)
- `pnpm ashby:refresh-snapshot` — refresh the committed careers snapshot

View File

@@ -6,25 +6,14 @@ import tailwindcss from '@tailwindcss/vite'
export default defineConfig({
site: 'https://comfy.org',
output: 'static',
prefetch: { prefetchAll: true },
redirects: {
'/cloud/enterprise-case-studies/comfyui-at-architectural-scale-how-moment-factory-reimagined-3d-projection-mapping':
'/customers/moment-factory/',
'/cloud/enterprise-case-studies/how-series-entertainment-rebuilt-game-and-video-production-with-comfyui':
'/customers/series-entertainment/'
},
build: {
assets: '_website'
},
devToolbar: { enabled: !process.env.NO_TOOLBAR },
integrations: [vue(), sitemap()],
vite: {
plugins: [tailwindcss()],
server: {
watch: {
ignored: ['**/playwright-report/**']
}
}
plugins: [tailwindcss()]
},
build: {
assetsPrefix: process.env.VERCEL_URL
? `https://${process.env.VERCEL_URL}`
: undefined
},
i18n: {
locales: ['en', 'zh-CN'],

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 B

View File

@@ -1,57 +0,0 @@
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
test.describe('Careers page @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/careers')
})
test('has correct title', async ({ page }) => {
await expect(page).toHaveTitle('Careers — Comfy')
})
test('Roles section heading is visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: 'Roles', level: 2 })
).toBeVisible()
})
test('renders at least one role from the snapshot', async ({ page }) => {
const roles = page.getByTestId('careers-role-link')
await expect(roles.first()).toBeVisible()
expect(await roles.count()).toBeGreaterThan(0)
})
test('each role links to jobs.ashbyhq.com', async ({ page }) => {
const roles = page.getByTestId('careers-role-link')
const count = await roles.count()
for (let i = 0; i < count; i++) {
const href = await roles.nth(i).getAttribute('href')
expect(href).toMatch(/^https:\/\/jobs\.ashbyhq\.com\//)
}
})
test('ENGINEERING category filter narrows the role list', async ({
page
}) => {
const allCount = await page.getByTestId('careers-role-link').count()
await page.getByRole('button', { name: 'ENGINEERING', exact: true }).click()
const engineeringLocator = page.getByTestId('careers-role-link')
await expect(engineeringLocator.first()).toBeVisible()
const engineeringCount = await engineeringLocator.count()
expect(engineeringCount).toBeLessThanOrEqual(allCount)
expect(engineeringCount).toBeGreaterThan(0)
})
})
test.describe('Careers page (zh-CN) @smoke', () => {
test('renders localized heading and roles', async ({ page }) => {
await page.goto('/zh-CN/careers')
await expect(page).toHaveTitle('招聘 — Comfy')
await expect(
page.getByRole('heading', { name: '职位', level: 2 })
).toBeVisible()
await expect(page.getByTestId('careers-role-link').first()).toBeVisible()
})
})

View File

@@ -1,143 +0,0 @@
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
test.describe('Cloud page @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/cloud')
})
test('has correct title', async ({ page }) => {
await expect(page).toHaveTitle('Comfy Cloud — AI in the Cloud')
})
test('HeroSection heading and subtitle are visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /The full power of/i, level: 1 })
).toBeVisible()
await expect(
page.getByText(/The easiest way to start with ComfyUI/)
).toBeVisible()
})
test('HeroSection has CTA button linking to cloud', async ({ page }) => {
const cta = page.getByRole('link', { name: /TRY COMFY CLOUD FOR FREE/i })
await expect(cta).toBeVisible()
})
test('ReasonSection heading and reasons are visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /Why.*professionals.*choose/i })
).toBeVisible()
for (const title of [
'Powerful GPUs',
'All models',
'More control',
'Community workflows'
]) {
await expect(page.getByText(title).first()).toBeVisible()
}
})
test('AIModelsSection heading and 5 model cards are visible', async ({
page
}) => {
const heading = page.getByRole('heading', { name: /leading AI models/i })
await expect(heading).toBeVisible()
const section = heading.locator('xpath=ancestor::section')
const grid = section.locator('.grid')
const modelCards = grid.locator('a[href="https://comfy.org/workflows"]')
await expect(modelCards).toHaveCount(5)
})
test('AIModelsSection CTA links to workflows', async ({ page }) => {
const cta = page.getByRole('link', {
name: /EXPLORE WORKFLOWS/i
})
await expect(cta.first()).toBeVisible()
await expect(cta.first()).toHaveAttribute(
'href',
'https://comfy.org/workflows'
)
})
test('AudienceSection heading and cards are visible', async ({ page }) => {
await expect(page.getByText(/creators/i).first()).toBeVisible()
for (const label of ['CREATORS', 'TEAMS & STUDIOS']) {
await expect(page.getByText(label).first()).toBeVisible()
}
})
test('PricingSection heading and CTA are visible', async ({ page }) => {
await expect(page.getByText(/Simple, credit-based pricing/)).toBeVisible()
const cta = page.getByRole('link', { name: /SEE PRICING PLANS/i })
await expect(cta).toBeVisible()
await expect(cta).toHaveAttribute('href', '/cloud/pricing')
})
test('ProductCardsSection has 3 product cards', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
const cards = section.locator('a[href]')
await expect(cards).toHaveCount(3)
})
test('FAQSection heading is visible with 15 items', async ({ page }) => {
await expect(page.getByRole('heading', { name: /FAQ/i })).toBeVisible()
const faqButtons = page.locator('button[aria-controls^="faq-panel-"]')
await expect(faqButtons).toHaveCount(15)
})
})
test.describe('Cloud FAQ accordion @interaction', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/cloud')
})
test('all FAQs are collapsed by default', async ({ page }) => {
await expect(
page.getByText(/Comfy Cloud is a version of ComfyUI/i)
).toBeHidden()
})
test('clicking a collapsed FAQ expands it', async ({ page }) => {
const firstQuestion = page.getByRole('button', {
name: /What is Comfy Cloud/i
})
await firstQuestion.scrollIntoViewIfNeeded()
// Gate: wait for Vue hydration to bind aria-expanded
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await firstQuestion.click()
await expect(
page.getByText(/Comfy Cloud is a version of ComfyUI/i)
).toBeVisible()
})
test('clicking an expanded FAQ collapses it again', async ({ page }) => {
const firstQuestion = page.getByRole('button', {
name: /What is Comfy Cloud/i
})
await firstQuestion.scrollIntoViewIfNeeded()
// Gate: wait for Vue hydration to bind aria-expanded
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await firstQuestion.click()
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'true')
await expect(
page.getByText(/Comfy Cloud is a version of ComfyUI/i)
).toBeVisible()
await firstQuestion.click()
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await expect(
page.getByText(/Comfy Cloud is a version of ComfyUI/i)
).toBeHidden()
})
})

View File

@@ -1,196 +0,0 @@
import { devices, expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
const WINDOWS_UA =
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
test.describe('Download page @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/download')
})
test('has correct title', async ({ page }) => {
await expect(page).toHaveTitle('Download Comfy — Run AI Locally')
})
test('CloudBannerSection is visible with cloud link', async ({ page }) => {
const link = page.getByRole('link', { name: /TRY COMFY CLOUD/i })
await expect(link).toBeVisible()
await expect(link).toHaveAttribute('href', 'https://cloud.comfy.org')
})
test('HeroSection heading and subtitle are visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /Run on your hardware/i, level: 1 })
).toBeVisible()
await expect(page.getByText(/The full ComfyUI engine/)).toBeVisible()
})
test('HeroSection has download and GitHub buttons', async ({ browser }) => {
const context = await browser.newContext({ userAgent: WINDOWS_UA })
const page = await context.newPage()
await page.goto('/download')
const hero = page.locator('section', {
has: page.getByRole('heading', {
name: /Run on your hardware/i,
level: 1
})
})
const downloadBtn = hero.getByRole('link', { name: /DOWNLOAD LOCAL/i })
await expect(downloadBtn).toBeVisible()
await expect(downloadBtn).toHaveAttribute('target', '_blank')
const githubBtn = hero.getByRole('link', { name: /INSTALL FROM GITHUB/i })
await expect(githubBtn).toBeVisible()
await expect(githubBtn).toHaveAttribute(
'href',
'https://github.com/Comfy-Org/ComfyUI'
)
await context.close()
})
test('ReasonSection heading and reasons are visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /Why.*professionals.*choose/i })
).toBeVisible()
for (const title of [
'Unlimited',
'Any model',
'Your machine',
'Free. Open Source'
]) {
await expect(page.getByText(title).first()).toBeVisible()
}
})
test('EcoSystemSection heading is visible', async ({ page }) => {
await expect(page.getByText(/An ecosystem that moves faster/)).toBeVisible()
})
test('ProductCardsSection has 3 product cards', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
const cards = section.locator('a[href]')
await expect(cards).toHaveCount(3)
})
test('ProductCardsSection links to cloud, api, enterprise', async ({
page
}) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
for (const href of ['/cloud', '/api', '/cloud/enterprise']) {
await expect(section.locator(`a[href="${href}"]`)).toBeVisible()
}
})
test('FAQSection heading is visible with 8 items', async ({ page }) => {
await expect(page.getByRole('heading', { name: /FAQ/i })).toBeVisible()
const faqButtons = page.locator('button[aria-controls^="faq-panel-"]')
await expect(faqButtons).toHaveCount(8)
})
})
test.describe('FAQ accordion @interaction', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/download')
})
test('all FAQs are collapsed by default', async ({ page }) => {
await expect(
page.getByText(/A dedicated GPU is strongly recommended/i)
).toBeHidden()
await expect(page.getByText(/ComfyUI is lightweight/i)).toBeHidden()
})
test('clicking a collapsed FAQ expands it', async ({ page }) => {
const firstQuestion = page.getByRole('button', {
name: /Do I need a GPU/i
})
await firstQuestion.scrollIntoViewIfNeeded()
// Gate: wait for Vue hydration to bind aria-expanded
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await firstQuestion.click()
await expect(
page.getByText(/A dedicated GPU is strongly recommended/i)
).toBeVisible()
})
test('clicking an expanded FAQ collapses it again', async ({ page }) => {
const firstQuestion = page.getByRole('button', {
name: /Do I need a GPU/i
})
await firstQuestion.scrollIntoViewIfNeeded()
// Gate: wait for Vue hydration to bind aria-expanded
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await firstQuestion.click()
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'true')
await expect(
page.getByText(/A dedicated GPU is strongly recommended/i)
).toBeVisible()
await firstQuestion.click()
await expect(firstQuestion).toHaveAttribute('aria-expanded', 'false')
await expect(
page.getByText(/A dedicated GPU is strongly recommended/i)
).toBeHidden()
})
})
test.describe('Download page mobile @mobile', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/download')
})
test('CloudBannerSection is visible', async ({ page }) => {
await expect(page.getByText(/Need more power/)).toBeVisible()
})
test('HeroSection heading is visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /Run on your hardware/i, level: 1 })
).toBeVisible()
})
test('download buttons are stacked vertically', async ({ browser }) => {
const context = await browser.newContext({
...devices['Pixel 5'],
userAgent: WINDOWS_UA
})
const page = await context.newPage()
await page.goto('/download')
const hero = page.locator('section', {
has: page.getByRole('heading', {
name: /Run on your hardware/i,
level: 1
})
})
const downloadBtn = hero.getByRole('link', { name: /DOWNLOAD LOCAL/i })
const githubBtn = hero.getByRole('link', { name: /INSTALL FROM GITHUB/i })
await expect(downloadBtn).toBeVisible()
await expect(githubBtn).toBeVisible()
await expect
.poll(async () => {
const downloadBox = await downloadBtn.boundingBox()
const githubBox = await githubBtn.boundingBox()
if (!downloadBox || !githubBox) return false
return githubBox.y > downloadBox.y
})
.toBe(true)
await context.close()
})
})

View File

@@ -1,47 +0,0 @@
import { fileURLToPath } from 'node:url'
import type { Route } from '@playwright/test'
import { test as base } from '@playwright/test'
function assetPath(relativePath: string) {
return fileURLToPath(new URL(relativePath, import.meta.url))
}
const IMAGE_PLACEHOLDER = assetPath('../assets/placeholder-1x1.webp')
const VIDEO_PLACEHOLDER = assetPath('../assets/placeholder.webm')
const ANALYTICS_PATTERN = '**/va.vercel-scripts.com/**' as const
const MEDIA_PATTERN =
/^https:\/\/media\.comfy\.org\/.*\.(webp|webm|mp4|png|jpg|jpeg|vtt)(\?.*)?$/i
const VIDEO_PATTERN = /\.(webm|mp4)(\?|$)/i
const SUBTITLE_PATTERN = /\.vtt(\?|$)/i
function blockAnalytics(route: Route) {
return route.abort('blockedbyclient')
}
async function fulfillMedia(route: Route) {
const url = route.request().url()
if (VIDEO_PATTERN.test(url))
return route.fulfill({ path: VIDEO_PLACEHOLDER, status: 200 })
if (SUBTITLE_PATTERN.test(url))
return route.fulfill({
status: 200,
contentType: 'text/vtt',
body: 'WEBVTT\n'
})
await route.fulfill({ path: IMAGE_PLACEHOLDER, status: 200 })
}
export const test = base.extend<{ blockExternalMedia: void }>({
blockExternalMedia: [
async ({ page }, use) => {
await page.route(ANALYTICS_PATTERN, blockAnalytics)
await page.route(MEDIA_PATTERN, fulfillMedia)
await use()
},
{ auto: true }
]
})

View File

@@ -1,180 +0,0 @@
import { fileURLToPath } from 'node:url'
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
const caseStudyVideoPath = fileURLToPath(
new URL(
'../../../public/assets/images/cloud-subscription.webm',
import.meta.url
)
)
test.describe('Homepage @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('has correct title', async ({ page }) => {
await expect(page).toHaveTitle('Comfy — Professional Control of Visual AI')
})
test('HeroSection heading is visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: /Professional Control/i, level: 1 })
).toBeVisible()
})
test('SocialProofBar logos are visible', async ({ page }) => {
await expect(
page.locator('img[src*="/icons/clients/"]').first()
).toBeVisible()
})
test('ProductShowcase section is visible', async ({ page }) => {
await expect(page.getByText('HOW', { exact: true }).first()).toBeVisible()
await expect(
page.getByText(/Connect models, processing steps, and outputs/)
).toBeVisible()
})
test('UseCaseSection is visible', async ({ page }) => {
await expect(
page.getByText('Industries that create with ComfyUI')
).toBeVisible()
})
test('GetStartedSection with heading is visible', async ({ page }) => {
await expect(
page.getByRole('heading', { name: 'Get started in minutes' })
).toBeVisible()
})
test('ProductCardsSection has 4 product cards', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
const cards = section.locator('a[href]')
await expect(cards).toHaveCount(4)
})
test('CaseStudySpotlight section is visible', async ({ page }) => {
const section = page.locator('section', {
has: page.getByText('Customer Stories')
})
await expect(section).toBeVisible()
await expect(
section.getByRole('heading', { name: /See Comfy/i })
).toBeVisible()
})
test('BuildWhatSection is visible', async ({ page }) => {
// "DOESN'T EXIST" is the actual badge text rendered in the Build What section
await expect(page.getByText("DOESN'T EXIST")).toBeVisible()
})
})
test.describe('Product showcase accordion @interaction', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('first feature is active by default', async ({ page }) => {
await expect(
page.getByText(/Build powerful AI pipelines by connecting nodes/).first()
).toBeVisible()
})
test('clicking inactive feature expands it and collapses previous', async ({
page
}) => {
const secondFeature = page
.getByRole('button', { name: /App mode/i })
.first()
await secondFeature.scrollIntoViewIfNeeded()
await secondFeature.click()
await expect(
secondFeature.getByText(/If you are new to ComfyUI/)
).toBeVisible()
const firstFeature = page
.getByRole('button', { name: /Full Control with Nodes/i })
.first()
await expect(firstFeature).not.toHaveClass(/bg-primary-comfy-yellow/)
await expect(secondFeature).toHaveClass(/bg-primary-comfy-yellow/)
})
})
test.describe('Video player @interaction', () => {
test.beforeEach(async ({ page }) => {
await page.route(
'https://media.comfy.org/website/customers/blackmath/video.webm',
(route) =>
route.fulfill({
contentType: 'video/webm',
path: caseStudyVideoPath
})
)
await page.goto('/')
})
test('clicking play advances playback', async ({ page }) => {
const section = page.locator('section', {
has: page.getByText('Customer Stories')
})
const video = section.locator('video')
await expect
.poll(
async () =>
video.evaluate((element: HTMLVideoElement) => element.duration),
{ timeout: 15_000 }
)
.toBeGreaterThan(0)
await section.getByRole('button', { name: 'Play' }).click()
await expect
.poll(async () =>
video.evaluate((element: HTMLVideoElement) => element.currentTime)
)
.toBeGreaterThan(0)
})
})
test.describe('Product cards links @smoke', () => {
test('cards have correct hrefs', async ({ page }) => {
await page.goto('/')
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
for (const href of ['/download', '/cloud', '/api', '/cloud/enterprise']) {
await expect(section.locator(`a[href="${href}"]`)).toBeVisible()
}
})
})
test.describe('Get started section links @smoke', () => {
test('has download and cloud links', async ({ page }) => {
await page.goto('/')
const section = page.locator('section', {
has: page.getByRole('heading', { name: 'Get started in minutes' })
})
const downloadLink = section.getByRole('link', { name: 'Download Local' })
await expect(downloadLink).toBeVisible()
await expect(downloadLink).toHaveAttribute('href', '/download')
const cloudLink = section.getByRole('link', { name: 'Launch Cloud' })
await expect(cloudLink).toBeVisible()
await expect(cloudLink).toHaveAttribute('href', 'https://cloud.comfy.org')
})
})

View File

@@ -1,163 +0,0 @@
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
test.describe('Desktop navigation @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('logo links to homepage', async ({ page }) => {
const logo = page.getByRole('link', { name: 'Comfy home' })
await expect(logo).toBeVisible()
await expect(logo).toHaveAttribute('href', '/')
})
test('has all top-level nav items', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopLinks = nav.getByTestId('desktop-nav-links')
for (const label of [
'PRODUCTS',
'PRICING',
'COMMUNITY',
'RESOURCES',
'COMPANY'
]) {
await expect(desktopLinks.getByText(label).first()).toBeVisible()
}
})
test('CTA buttons are visible', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopCTA = nav.getByTestId('desktop-nav-cta')
await expect(
desktopCTA.getByRole('link', { name: 'DOWNLOAD LOCAL' })
).toBeVisible()
await expect(
desktopCTA.getByRole('link', { name: 'LAUNCH CLOUD' })
).toBeVisible()
})
})
test.describe('Desktop dropdown @interaction', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('hovering PRODUCTS shows dropdown items', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopLinks = nav.getByTestId('desktop-nav-links')
const productsButton = desktopLinks.getByRole('button', {
name: /PRODUCTS/i
})
await productsButton.hover()
const dropdown = productsButton.locator('..').getByTestId('nav-dropdown')
for (const item of [
'Comfy Local',
'Comfy Cloud',
'Comfy API',
'Comfy Enterprise'
]) {
await expect(dropdown.getByText(item)).toBeVisible()
}
})
test('moving mouse away closes dropdown', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopLinks = nav.getByTestId('desktop-nav-links')
await desktopLinks.getByRole('button', { name: /PRODUCTS/i }).hover()
const comfyLocal = nav.getByRole('link', { name: 'Comfy Local' }).first()
await expect(comfyLocal).toBeVisible()
await page.locator('main').hover()
await expect(comfyLocal).toBeHidden()
})
test('Escape key closes dropdown', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopLinks = nav.getByTestId('desktop-nav-links')
await desktopLinks.getByRole('button', { name: /PRODUCTS/i }).hover()
const comfyLocal = nav.getByRole('link', { name: 'Comfy Local' }).first()
await expect(comfyLocal).toBeVisible()
await page.keyboard.press('Escape')
await expect(comfyLocal).toBeHidden()
})
})
test.describe('Mobile menu @mobile', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('hamburger button is visible', async ({ page }) => {
await expect(
page.getByRole('button', { name: 'Toggle menu' })
).toBeVisible()
})
test('clicking hamburger opens mobile menu with nav items', async ({
page
}) => {
await page.getByRole('button', { name: 'Toggle menu' }).click()
const menu = page.locator('#site-mobile-menu')
await expect(menu).toBeVisible()
for (const label of ['PRODUCTS', 'PRICING', 'COMMUNITY']) {
await expect(menu.getByText(label).first()).toBeVisible()
}
})
test('clicking section with subitems drills down and back works', async ({
page
}) => {
await page.getByRole('button', { name: 'Toggle menu' }).click()
const menu = page.locator('#site-mobile-menu')
await menu.getByText('PRODUCTS').first().click()
await expect(menu.getByText('Comfy Local')).toBeVisible()
await expect(menu.getByText('Comfy Cloud')).toBeVisible()
await menu.getByRole('button', { name: /BACK/i }).click()
await expect(menu.getByText('PRODUCTS').first()).toBeVisible()
})
test('CTA buttons visible in mobile menu', async ({ page }) => {
await page.getByRole('button', { name: 'Toggle menu' }).click()
const menu = page.locator('#site-mobile-menu')
await expect(
menu.getByRole('link', { name: 'DOWNLOAD LOCAL' })
).toBeVisible()
await expect(menu.getByRole('link', { name: 'LAUNCH CLOUD' })).toBeVisible()
})
})
test.describe('Footer @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('footer is visible with link sections', async ({ page }) => {
const footer = page.locator('footer')
await expect(footer).toBeVisible()
for (const heading of ['Products', 'Resources', 'Company']) {
await expect(
footer.getByRole('heading', { name: heading }).first()
).toBeVisible()
}
})
test('copyright text is visible', async ({ page }) => {
await expect(
page.locator('footer').getByText(/© \d{4} Comfy Org/)
).toBeVisible()
})
})

View File

@@ -1,50 +0,0 @@
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
test.describe('Desktop layout @smoke', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('navigation links visible and hamburger hidden', async ({ page }) => {
const nav = page.getByRole('navigation', { name: 'Main navigation' })
const desktopLinks = nav.getByTestId('desktop-nav-links')
await expect(desktopLinks.getByText('PRODUCTS').first()).toBeVisible()
await expect(desktopLinks.getByText('PRICING').first()).toBeVisible()
await expect(page.getByRole('button', { name: 'Toggle menu' })).toBeHidden()
})
test('product cards in grid layout', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/ })
})
const cards = section.locator('a[href]')
await expect(cards).toHaveCount(4)
const firstBox = await cards.nth(0).boundingBox()
const secondBox = await cards.nth(1).boundingBox()
expect(firstBox, 'first card bounding box').not.toBeNull()
expect(secondBox, 'second card bounding box').not.toBeNull()
expect(firstBox!.y).toBeCloseTo(secondBox!.y, 0)
})
})
test.describe('Mobile layout @mobile', () => {
test.beforeEach(async ({ page }) => {
await page.goto('/')
})
test('hamburger visible and desktop nav hidden', async ({ page }) => {
await expect(
page.getByRole('button', { name: 'Toggle menu' })
).toBeVisible()
})
test('SocialProofBar shows two marquee rows on mobile', async ({ page }) => {
const mobileContainer = page.getByTestId('social-proof-mobile')
await expect(mobileContainer).toBeVisible()
})
})

View File

@@ -1,6 +0,0 @@
export const VIEWPORTS = [
{ name: '1-sm', width: 393, height: 851 },
{ name: '2-md', width: 768, height: 1024 },
{ name: '3-lg', width: 1280, height: 800 },
{ name: '4-xl', width: 1536, height: 864 }
] as const

View File

@@ -1,145 +0,0 @@
import type { Page } from '@playwright/test'
import { expect } from '@playwright/test'
import { test } from './fixtures/blockExternalMedia'
import { VIEWPORTS } from './viewports'
test.describe.configure({ timeout: 60_000 })
const SMALL_VIEWPORTS = VIEWPORTS.filter(
(v) => v.name === '1-sm' || v.name === '2-md'
)
async function assertNoOverflow(page: Page) {
await expect
.poll(
() =>
page.evaluate(
() =>
document.documentElement.scrollWidth >
document.documentElement.clientWidth
),
{ message: 'page has horizontal overflow', timeout: 5_000 }
)
.toBe(false)
}
async function navigateAndSettle(page: Page, url: string) {
await page.goto(url)
await page.waitForLoadState('networkidle')
}
test.describe('Home', { tag: '@visual' }, () => {
for (const vp of VIEWPORTS) {
test.describe(vp.name, () => {
test.beforeEach(async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await navigateAndSettle(page, '/')
})
test('product-cards screenshot', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /The AI creation/i })
})
await expect(section).toBeVisible()
await section.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`home-product-cards-${vp.name}.png`)
})
test('get-started screenshot', async ({ page }) => {
const section = page.locator('section', {
has: page.getByRole('heading', { name: /Get started/i })
})
await expect(section).toBeVisible()
await section.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`home-get-started-${vp.name}.png`)
})
})
}
})
test.describe('Pricing', { tag: '@visual' }, () => {
for (const vp of VIEWPORTS) {
test(`pricing-tiers-${vp.name}`, async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await navigateAndSettle(page, '/cloud/pricing')
await assertNoOverflow(page)
const section = page.locator('section', {
has: page.getByRole('heading', { name: /Pricing/i })
})
await expect(section).toBeVisible()
await section.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`pricing-tiers-${vp.name}.png`)
})
}
})
test.describe('Contact', { tag: '@visual' }, () => {
for (const vp of SMALL_VIEWPORTS) {
test(`form-${vp.name}`, async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await navigateAndSettle(page, '/contact')
const section = page.locator('section', {
has: page.getByRole('heading', { name: /Create powerful workflows/i })
})
await expect(section).toBeVisible()
await section.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`contact-form-${vp.name}.png`)
})
}
})
test.describe('Gallery', { tag: '@visual' }, () => {
for (const vp of SMALL_VIEWPORTS) {
test(`gallery-grid-${vp.name}`, async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await navigateAndSettle(page, '/gallery')
const section = page.getByTestId('gallery-grid')
await expect(section).toBeVisible()
await section.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`gallery-grid-${vp.name}.png`)
})
}
})
test.describe('About', { tag: '@visual' }, () => {
for (const vp of SMALL_VIEWPORTS) {
test(`hero-${vp.name}`, async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await navigateAndSettle(page, '/about')
const hero = page.locator('section', {
has: page.getByRole('heading', { name: /Build the tools/i })
})
await expect(hero).toBeVisible()
await hero.scrollIntoViewIfNeeded()
await expect(page).toHaveScreenshot(`about-hero-${vp.name}.png`)
})
}
})
test.describe('Overflow guards', { tag: '@visual' }, () => {
const pages = [
'/',
'/cloud',
'/cloud/pricing',
'/contact',
'/download',
'/gallery',
'/about',
'/careers'
]
for (const url of pages) {
for (const vp of VIEWPORTS) {
test(`${url} ${vp.name} no overflow`, async ({ page }) => {
await page.setViewportSize({ width: vp.width, height: vp.height })
await page.goto(url)
await assertNoOverflow(page)
})
}
}
})

Binary file not shown.

Before

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

Some files were not shown because too many files have changed in this diff Show More