Compare commits
89 Commits
perf/minim
...
copilot/su
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6b90510068 | ||
|
|
8ced2683b5 | ||
|
|
b7f3fdb6d2 | ||
|
|
b0ad8d3ea1 | ||
|
|
03a6279ada | ||
|
|
7323fe84c0 | ||
|
|
e115619da7 | ||
|
|
8a61c9232c | ||
|
|
5b27cceef3 | ||
|
|
d5ed5f1b6e | ||
|
|
85b51bd49b | ||
|
|
07777923bf | ||
|
|
c42719b803 | ||
|
|
b9ced27b38 | ||
|
|
5e3328f351 | ||
|
|
8e55276571 | ||
|
|
6d3717e14a | ||
|
|
afbf089580 | ||
|
|
170a07f327 | ||
|
|
712b394d83 | ||
|
|
0210d41229 | ||
|
|
f3df5e8948 | ||
|
|
50a187506b | ||
|
|
1a06b530f4 | ||
|
|
11d438b2e3 | ||
|
|
a19d768bf4 | ||
|
|
42bb381b91 | ||
|
|
65e999c921 | ||
|
|
c739ce5bd7 | ||
|
|
60e2dc6469 | ||
|
|
9f29252378 | ||
|
|
4bd579f34f | ||
|
|
bea707005e | ||
|
|
0defd1393d | ||
|
|
40bf25198a | ||
|
|
bfea5a7ec7 | ||
|
|
1acc0f8894 | ||
|
|
6be36f4f52 | ||
|
|
4f808b50f8 | ||
|
|
7b1eea5f96 | ||
|
|
7d73327f0c | ||
|
|
0260647d03 | ||
|
|
60eb18b97b | ||
|
|
3109e68877 | ||
|
|
eed68652ac | ||
|
|
4115d5b3b6 | ||
|
|
c2806b2922 | ||
|
|
5c4296940e | ||
|
|
6760013d2d | ||
|
|
ee5a207407 | ||
|
|
1e1fdaa253 | ||
|
|
1285049d03 | ||
|
|
684fbb473a | ||
|
|
2c671fe2fd | ||
|
|
ae55b95f5d | ||
|
|
122fb850fc | ||
|
|
6bb19a4af6 | ||
|
|
bf23384de0 | ||
|
|
eaf1f9ac77 | ||
|
|
b92f7f19d0 | ||
|
|
144cb0f821 | ||
|
|
0c44129bb4 | ||
|
|
513510104c | ||
|
|
f6bc0ff1ee | ||
|
|
8851ab1821 | ||
|
|
06f7e13957 | ||
|
|
e31d98b743 | ||
|
|
736f0fa416 | ||
|
|
d8d8aeb99e | ||
|
|
f4868894fa | ||
|
|
a96c61d2c2 | ||
|
|
c420cff4f0 | ||
|
|
8ccfe852b4 | ||
|
|
e2ef041170 | ||
|
|
a4952e9c45 | ||
|
|
64c852bf82 | ||
|
|
697087fdca | ||
|
|
a7218b2922 | ||
|
|
39a774bc15 | ||
|
|
74dedc314d | ||
|
|
5ae747e7c1 | ||
|
|
21069d54e7 | ||
|
|
8a456043e8 | ||
|
|
585e6f87fa | ||
|
|
4781775a78 | ||
|
|
74a48ab2aa | ||
|
|
0875e2f50f | ||
|
|
63442d2fb0 | ||
|
|
bb6f00dc68 |
358
.claude/skills/comfy-qa/SKILL.md
Normal file
@@ -0,0 +1,358 @@
|
||||
---
|
||||
name: comfy-qa
|
||||
description: 'Comprehensive QA of ComfyUI frontend. Navigates all routes, tests all interactive features using Playwright MCP, generates a report, and submits a draft PR. Works in CI and local environments, cross-platform.'
|
||||
---
|
||||
|
||||
# ComfyUI Frontend QA Skill
|
||||
|
||||
Perform comprehensive quality assurance of the ComfyUI frontend application by navigating all routes, clicking interactive elements, and testing features. Generate a structured report and submit it as a draft PR.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18+
|
||||
- `pnpm` package manager
|
||||
- `gh` CLI (authenticated)
|
||||
- Playwright MCP server (browser automation)
|
||||
|
||||
## Step 1: Environment Detection & Setup
|
||||
|
||||
Detect the runtime environment and ensure the app is accessible.
|
||||
|
||||
### CI Environment
|
||||
|
||||
If `CI=true` is set:
|
||||
|
||||
1. The ComfyUI backend is pre-configured in the CI container (`ghcr.io/comfy-org/comfyui-ci-container`)
|
||||
2. Frontend dist is already built and served by the backend
|
||||
3. Server runs at `http://127.0.0.1:8188`
|
||||
4. Skip user prompts — run fully automated
|
||||
|
||||
### Local Environment
|
||||
|
||||
If `CI` is not set:
|
||||
|
||||
1. **Ask the user**: "Is a ComfyUI server already running? If so, what URL? (default: http://127.0.0.1:8188)"
|
||||
- If yes: use the provided URL
|
||||
- If no: offer to start one:
|
||||
|
||||
```bash
|
||||
# Option A: Use existing ComfyUI installation
|
||||
# Ask for the path to ComfyUI, then:
|
||||
cd <comfyui_path>
|
||||
python main.py --cpu --multi-user --front-end-root <frontend_dist_path> &
|
||||
|
||||
# Option B: Build frontend and use preview server (no backend features)
|
||||
pnpm build && pnpm preview &
|
||||
```
|
||||
|
||||
2. Wait for server readiness by polling the URL (retry with 2s intervals, 60s timeout)
|
||||
|
||||
### Browser Automation Setup
|
||||
|
||||
Use **Playwright MCP** for browser interaction. The agent should use whichever Playwright MCP is available in its tool set:
|
||||
|
||||
- **Claude Code**: Uses `remote-chrome` MCP or built-in browser tools
|
||||
- **Interactive mode**: User can opt for `npx @anthropic-ai/mcp-server-playwright` or `npx @playwright/mcp@latest --port 3100` for headless
|
||||
- **User's real browser**: `npx @playwright/mcp@latest --extension` — lets QA happen in the user's actual browser session
|
||||
|
||||
Determine the server URL. For local dev servers behind proxies, adjust the URL accordingly (e.g., `https://[port].stukivx.xyz` pattern if configured).
|
||||
|
||||
## Step 2: QA Test Plan
|
||||
|
||||
Navigate to the application URL and systematically test each area below. For each test, record:
|
||||
|
||||
- **Status**: pass / fail / skip (with reason)
|
||||
- **Notes**: any issues, unexpected behavior, or visual glitches
|
||||
- **Screenshots**: take screenshots of failures or notable states
|
||||
|
||||
### 2.1 Application Load & Routes
|
||||
|
||||
| Test | Steps |
|
||||
| ----------------- | ------------------------------------------------------------ |
|
||||
| Root route loads | Navigate to `/` — GraphView should render with canvas |
|
||||
| User select route | Navigate to `/user-select` — user selection UI should appear |
|
||||
| Default redirect | If multi-user mode, `/` redirects to `/user-select` first |
|
||||
| 404 handling | Navigate to `/nonexistent` — should handle gracefully |
|
||||
|
||||
### 2.2 Canvas & Graph View
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------------- | -------------------------------------------------------------- |
|
||||
| Canvas renders | The LiteGraph canvas is visible and interactive |
|
||||
| Pan canvas | Click and drag on empty canvas area |
|
||||
| Zoom in/out | Use scroll wheel or Alt+=/Alt+- |
|
||||
| Fit view | Press `.` key — canvas fits to content |
|
||||
| Add node via double-click | Double-click canvas to open search, type "KSampler", select it |
|
||||
| Add node via search | Open search box, find and add a node |
|
||||
| Delete node | Select a node, press Delete key |
|
||||
| Connect nodes | Drag from output slot to input slot |
|
||||
| Disconnect nodes | Right-click a link and remove, or drag from connected slot |
|
||||
| Multi-select | Shift+click or drag-select multiple nodes |
|
||||
| Copy/Paste | Select nodes, Ctrl+C then Ctrl+V |
|
||||
| Undo/Redo | Make changes, Ctrl+Z to undo, Ctrl+Y to redo |
|
||||
| Node context menu | Right-click a node — menu appears with all expected options |
|
||||
| Canvas context menu | Right-click empty canvas — menu appears |
|
||||
|
||||
### 2.3 Node Operations
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | ---------------------------------------------------------- |
|
||||
| Bypass node | Select node, Ctrl+B — node shows bypass state |
|
||||
| Mute node | Select node, Ctrl+M — node shows muted state |
|
||||
| Collapse node | Select node, Alt+C — node collapses |
|
||||
| Pin node | Select node, press P — node becomes pinned |
|
||||
| Rename node | Double-click node title — edit mode activates |
|
||||
| Node color | Right-click > Color — color picker works |
|
||||
| Group nodes | Select multiple nodes, Ctrl+G — group created |
|
||||
| Ungroup | Right-click group > Ungroup |
|
||||
| Widget interactions | Toggle checkboxes, adjust sliders, type in text fields |
|
||||
| Combo widget | Click dropdown widgets — options appear and are selectable |
|
||||
|
||||
### 2.4 Sidebar Tabs
|
||||
|
||||
| Test | Steps |
|
||||
| ---------------------- | ------------------------------------------------------ |
|
||||
| Workflows tab | Press W — workflows sidebar opens with saved workflows |
|
||||
| Node Library tab | Press N — node library opens with categories |
|
||||
| Model Library tab | Press M — model library opens |
|
||||
| Assets tab | Press A — assets browser opens |
|
||||
| Tab toggle | Press same key again — sidebar closes |
|
||||
| Search in sidebar | Type in search box — results filter |
|
||||
| Drag node from library | Drag a node from library onto canvas |
|
||||
|
||||
### 2.5 Topbar & Workflow Tabs
|
||||
|
||||
| Test | Steps |
|
||||
| -------------------- | ------------------------------------------------------ |
|
||||
| Workflow tab display | Current workflow name shown in tab bar |
|
||||
| New workflow | Ctrl+N — new blank workflow created |
|
||||
| Rename workflow | Double-click workflow tab |
|
||||
| Tab context menu | Right-click workflow tab — menu with Close/Rename/etc. |
|
||||
| Multiple tabs | Open multiple workflows, switch between them |
|
||||
| Queue button | Click Queue/Run button — prompt queues |
|
||||
| Batch count | Click batch count editor, change value |
|
||||
| Menu hamburger | Click hamburger menu — options appear |
|
||||
|
||||
### 2.6 Settings Dialog
|
||||
|
||||
| Test | Steps |
|
||||
| ---------------- | ---------------------------------------------------- |
|
||||
| Open settings | Press Ctrl+, or click settings button |
|
||||
| Settings tabs | Navigate through all setting categories |
|
||||
| Change a setting | Toggle a boolean setting — it persists after closing |
|
||||
| Search settings | Type in settings search box — results filter |
|
||||
| Keybindings tab | Navigate to keybindings panel |
|
||||
| About tab | Navigate to about panel — version info shown |
|
||||
| Close settings | Press Escape or click close button |
|
||||
|
||||
### 2.7 Bottom Panel
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | -------------------------------------- |
|
||||
| Toggle panel | Press Ctrl+` — bottom panel opens |
|
||||
| Logs tab | Logs/terminal tab shows server output |
|
||||
| Shortcuts tab | Shortcuts reference is displayed |
|
||||
| Keybindings display | Press Ctrl+Shift+K — keybindings panel |
|
||||
|
||||
### 2.8 Execution & Queue
|
||||
|
||||
| Test | Steps |
|
||||
| -------------- | ----------------------------------------------------- |
|
||||
| Queue prompt | Load default workflow, click Queue — execution starts |
|
||||
| Queue progress | Progress indicator shows during execution |
|
||||
| Interrupt | Press Ctrl+Alt+Enter during execution — interrupts |
|
||||
| Job history | Open job history sidebar — past executions listed |
|
||||
| Clear history | Clear execution history via menu |
|
||||
|
||||
### 2.9 Workflow File Operations
|
||||
|
||||
| Test | Steps |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| Save workflow | Ctrl+S — workflow saves (check for prompt if new) |
|
||||
| Open workflow | Ctrl+O — file picker or workflow browser opens |
|
||||
| Export JSON | Menu > Export — workflow JSON downloads |
|
||||
| Import workflow | Drag a .json workflow file onto canvas |
|
||||
| Load default | Menu > Load Default — default workflow loads |
|
||||
| Clear workflow | Menu > Clear — canvas clears (after confirmation) |
|
||||
|
||||
### 2.10 Advanced Features
|
||||
|
||||
| Test | Steps |
|
||||
| --------------- | ------------------------------------------------- |
|
||||
| Minimap | Alt+M — minimap toggle |
|
||||
| Focus mode | Toggle focus mode |
|
||||
| Canvas lock | Press H to lock, V to unlock |
|
||||
| Link visibility | Ctrl+Shift+L — toggle links |
|
||||
| Subgraph | Select nodes > Ctrl+Shift+E — convert to subgraph |
|
||||
|
||||
### 2.11 Error Handling
|
||||
|
||||
| Test | Steps |
|
||||
| --------------------- | -------------------------------------------- |
|
||||
| Missing nodes dialog | Load workflow with non-existent node types |
|
||||
| Missing models dialog | Trigger missing model warning |
|
||||
| Network error | Disconnect backend, verify graceful handling |
|
||||
| Invalid workflow | Try loading malformed JSON |
|
||||
|
||||
### 2.12 Responsive & Accessibility
|
||||
|
||||
| Test | Steps |
|
||||
| ------------------- | ------------------------------------- |
|
||||
| Window resize | Resize browser window — layout adapts |
|
||||
| Keyboard navigation | Tab through interactive elements |
|
||||
| Sidebar resize | Drag sidebar edge to resize |
|
||||
|
||||
## Step 3: Generate Report
|
||||
|
||||
After completing all tests, generate a markdown report file.
|
||||
|
||||
### Report Location
|
||||
|
||||
```
|
||||
docs/qa/YYYY-MM-DD-NNN-report.md
|
||||
```
|
||||
|
||||
Where:
|
||||
|
||||
- `YYYY-MM-DD` is today's date
|
||||
- `NNN` is a zero-padded increment index (001, 002, etc.)
|
||||
|
||||
To determine the increment, check existing files:
|
||||
|
||||
```bash
|
||||
ls docs/qa/ | grep "$(date +%Y-%m-%d)" | wc -l
|
||||
```
|
||||
|
||||
### Report Template
|
||||
|
||||
```markdown
|
||||
# QA Report: ComfyUI Frontend
|
||||
|
||||
**Date**: YYYY-MM-DD
|
||||
**Environment**: CI / Local (OS, Browser)
|
||||
**Frontend Version**: (git sha or version)
|
||||
**Agent**: Claude / Codex / Other
|
||||
**Server URL**: http://...
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Pass | Fail | Skip | Total |
|
||||
| --------------- | ---- | ---- | ---- | ----- |
|
||||
| Routes & Load | | | | |
|
||||
| Canvas | | | | |
|
||||
| Node Operations | | | | |
|
||||
| Sidebar | | | | |
|
||||
| Topbar | | | | |
|
||||
| Settings | | | | |
|
||||
| Bottom Panel | | | | |
|
||||
| Execution | | | | |
|
||||
| File Operations | | | | |
|
||||
| Advanced | | | | |
|
||||
| Error Handling | | | | |
|
||||
| Responsive | | | | |
|
||||
| **Total** | | | | |
|
||||
|
||||
## Results
|
||||
|
||||
### Routes & Load
|
||||
|
||||
- [x] Root route loads — pass
|
||||
- [ ] ...
|
||||
|
||||
### Canvas & Graph View
|
||||
|
||||
- [x] Canvas renders — pass
|
||||
- [ ] ...
|
||||
|
||||
(repeat for each category)
|
||||
|
||||
## Issues Found
|
||||
|
||||
### Issue 1: [Title]
|
||||
|
||||
- **Severity**: critical / major / minor / cosmetic
|
||||
- **Steps to reproduce**: ...
|
||||
- **Expected**: ...
|
||||
- **Actual**: ...
|
||||
- **Screenshot**: (if available)
|
||||
|
||||
## Notes
|
||||
|
||||
Any additional observations, performance notes, or suggestions.
|
||||
```
|
||||
|
||||
## Step 4: Commit and Push Report
|
||||
|
||||
### In CI (when `CI=true`)
|
||||
|
||||
Commit and push the report directly to the current branch. Do **not** create
|
||||
a new PR — the CI workflow handles posting results as a PR comment.
|
||||
|
||||
```bash
|
||||
git add docs/qa/
|
||||
git commit -m "docs: add QA report $(date +%Y-%m-%d)"
|
||||
git push
|
||||
```
|
||||
|
||||
### Local / interactive use
|
||||
|
||||
When running locally, create a draft PR after committing:
|
||||
|
||||
```bash
|
||||
# Ensure on a feature branch
|
||||
BRANCH_NAME="qa/$(date +%Y-%m-%d)-$(git rev-parse --short HEAD)"
|
||||
git checkout -b "$BRANCH_NAME" 2>/dev/null || git checkout "$BRANCH_NAME"
|
||||
|
||||
git add docs/qa/
|
||||
git commit -m "docs: add QA report $(date +%Y-%m-%d)
|
||||
|
||||
Automated QA report covering all frontend routes and features."
|
||||
git push -u origin "$BRANCH_NAME"
|
||||
|
||||
# Create draft PR assigned to comfy-pr-bot
|
||||
gh pr create \
|
||||
--draft \
|
||||
--title "QA Report: $(date +%Y-%m-%d)" \
|
||||
--body "## QA Report
|
||||
|
||||
Automated frontend QA run covering all routes and interactive features.
|
||||
|
||||
See \`docs/qa/\` for the full report.
|
||||
|
||||
/cc @comfy-pr-bot" \
|
||||
--assignee comfy-pr-bot
|
||||
```
|
||||
|
||||
## Execution Notes
|
||||
|
||||
### Cross-Platform Considerations
|
||||
|
||||
- **Windows**: Use `pwsh` or `cmd` equivalents for shell commands. `gh` CLI works on all platforms.
|
||||
- **macOS**: Keyboard shortcuts use Cmd instead of Ctrl in the actual app, but Playwright sends OS-appropriate keys.
|
||||
- **Linux**: Primary CI platform. Screenshot baselines are Linux-only.
|
||||
|
||||
### Agent Compatibility
|
||||
|
||||
This skill is written to be agent-agnostic:
|
||||
|
||||
- **Claude Code**: Has native Playwright MCP via `remote-chrome` or configured MCP servers
|
||||
- **Codex**: Can use `npx @playwright/mcp@latest` for browser automation
|
||||
- **Other agents**: Any agent with browser automation tools can follow these instructions
|
||||
|
||||
The key requirement is the ability to:
|
||||
|
||||
1. Navigate to URLs
|
||||
2. Take snapshots/screenshots
|
||||
3. Click elements
|
||||
4. Type text
|
||||
5. Press keyboard shortcuts
|
||||
6. Read page content
|
||||
|
||||
### Tips for Reliable QA
|
||||
|
||||
1. **Wait for page stability** before interacting — check that elements are visible and enabled
|
||||
2. **Take a snapshot after each major navigation** to verify state
|
||||
3. **Don't use fixed timeouts** — poll for expected conditions
|
||||
4. **Record the full page snapshot** at the start for baseline comparison
|
||||
5. **If a test fails**, document it and continue — don't abort the entire QA run
|
||||
6. **Group related tests** — complete one category before moving to the next
|
||||
221
.claude/skills/red-green-fix/SKILL.md
Normal file
@@ -0,0 +1,221 @@
|
||||
---
|
||||
name: red-green-fix
|
||||
description: 'Bug fix workflow that proves test validity with a red-then-green CI sequence. Commits a failing test first (CI red), then the minimal fix (CI green). Use when fixing a bug, writing a regression test, or when asked to prove a fix works.'
|
||||
---
|
||||
|
||||
# Red-Green Fix
|
||||
|
||||
Fixes bugs as two commits so CI automatically proves the test catches the bug.
|
||||
|
||||
## Why Two Commits
|
||||
|
||||
If you commit the test and fix together, the test always passes — reviewers cannot tell whether the test actually detects the bug or is a no-op. Splitting into two commits creates a verifiable CI trail:
|
||||
|
||||
1. **Commit 1 (test-only)** — adds a test that exercises the bug. CI runs it → test fails → red X.
|
||||
2. **Commit 2 (fix)** — adds the minimal fix. CI runs the same test → test passes → green check.
|
||||
|
||||
The red-then-green sequence in the commit history proves the test is valid.
|
||||
|
||||
## Input
|
||||
|
||||
The user provides a bug description as an argument. If no description is given, ask the user to describe the bug before proceeding.
|
||||
|
||||
Bug description: $ARGUMENTS
|
||||
|
||||
## Step 0 — Setup
|
||||
|
||||
Create an isolated branch from main:
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git checkout -b fix/<bug-name> origin/main
|
||||
```
|
||||
|
||||
## Step 1 — Red: Failing Test Only
|
||||
|
||||
Write a test that reproduces the bug. **Do NOT write any fix code.**
|
||||
|
||||
### Choosing the Test Framework
|
||||
|
||||
| Bug type | Framework | File location |
|
||||
| --------------------------------- | ---------- | ------------------------------- |
|
||||
| Logic, utils, stores, composables | Vitest | `src/**/*.test.ts` (colocated) |
|
||||
| UI interaction, canvas, workflows | Playwright | `browser_tests/tests/*.spec.ts` |
|
||||
|
||||
For Playwright tests, follow the `/writing-playwright-tests` skill for patterns, fixtures, and tags.
|
||||
|
||||
### Rules
|
||||
|
||||
- The test MUST fail against the current codebase (this is the whole point)
|
||||
- Do NOT modify any source code outside of test files
|
||||
- Do NOT include any fix, workaround, or behavioral change
|
||||
- Do NOT add unrelated tests or refactor existing tests
|
||||
- Keep the test minimal — only what is needed to reproduce the bug
|
||||
- Avoid common anti-patterns — see `reference/testing-anti-patterns.md`
|
||||
|
||||
### Vitest Example
|
||||
|
||||
```typescript
|
||||
// src/utils/pathUtil.test.ts
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { resolveModelPath } from './pathUtil'
|
||||
|
||||
describe('resolveModelPath', () => {
|
||||
it('handles absolute paths from folder_paths API', () => {
|
||||
const result = resolveModelPath(
|
||||
'/absolute/models',
|
||||
'/absolute/models/checkpoints'
|
||||
)
|
||||
expect(result).toBe('/absolute/models/checkpoints')
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
### Playwright Example
|
||||
|
||||
```typescript
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
|
||||
test.describe('Model Download', { tag: ['@smoke'] }, () => {
|
||||
test('downloads model when path is absolute', async ({ comfyPage }) => {
|
||||
await comfyPage.workflow.loadWorkflow('missing-model')
|
||||
const downloadBtn = comfyPage.page.getByTestId('download-model-button')
|
||||
await downloadBtn.click()
|
||||
await expect(comfyPage.page.getByText('Download complete')).toBeVisible()
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
### Verify Locally First
|
||||
|
||||
Run the test locally before pushing to confirm it fails for the right reason:
|
||||
|
||||
```bash
|
||||
# Vitest
|
||||
pnpm test:unit -- <test-file>
|
||||
|
||||
# Playwright
|
||||
pnpm test:browser:local -- --grep "<test name>"
|
||||
```
|
||||
|
||||
If the test passes locally, it does not reproduce the bug — revisit your test before pushing.
|
||||
|
||||
### Quality Checks and Commit
|
||||
|
||||
```bash
|
||||
pnpm typecheck
|
||||
pnpm lint
|
||||
pnpm format:check
|
||||
|
||||
git add <test-files-only>
|
||||
git commit -m "test: add failing test for <concise bug description>"
|
||||
git push -u origin HEAD
|
||||
```
|
||||
|
||||
### Verify CI Failure
|
||||
|
||||
```bash
|
||||
gh run list --branch $(git branch --show-current) --limit 1
|
||||
```
|
||||
|
||||
**STOP HERE.** Inform the user of the CI status and wait for confirmation before proceeding to Step 2.
|
||||
|
||||
- If CI passes: the test does not catch the bug. Revisit the test.
|
||||
- If CI fails for unrelated reasons: investigate and fix the test setup, not the bug.
|
||||
- If CI fails because the test correctly catches the bug: proceed to Step 2.
|
||||
|
||||
## Step 2 — Green: Minimal Fix
|
||||
|
||||
Write the minimum code change needed to make the failing test pass.
|
||||
|
||||
### Rules
|
||||
|
||||
- Do NOT modify, weaken, or delete the test from Step 1 — it is immutable. If the test needs changes, restart from Step 1 and re-prove the red.
|
||||
- Do NOT add new tests (tests were finalized in Step 1)
|
||||
- Do NOT refactor, clean up, or make "drive-by" improvements
|
||||
- Do NOT modify code unrelated to the bug
|
||||
- The fix should be the smallest correct change
|
||||
|
||||
### Quality Checks and Commit
|
||||
|
||||
```bash
|
||||
pnpm typecheck
|
||||
pnpm lint
|
||||
pnpm format
|
||||
|
||||
git add <fix-files-only>
|
||||
git commit -m "fix: <concise bug description>"
|
||||
git push
|
||||
```
|
||||
|
||||
### Verify CI Pass
|
||||
|
||||
```bash
|
||||
gh run list --branch $(git branch --show-current) --limit 1
|
||||
```
|
||||
|
||||
- If CI passes: the fix is verified. Proceed to PR creation.
|
||||
- If CI fails: investigate and fix. Do NOT change the test from Step 1.
|
||||
|
||||
## Step 3 — Open Pull Request
|
||||
|
||||
```bash
|
||||
gh pr create --title "fix: <description>" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
|
||||
<Brief explanation of the bug and root cause>
|
||||
|
||||
- Fixes #<issue-number>
|
||||
|
||||
## Red-Green Verification
|
||||
|
||||
| Commit | CI Status | Purpose |
|
||||
|--------|-----------|---------|
|
||||
| `test: ...` | :red_circle: Red | Proves the test catches the bug |
|
||||
| `fix: ...` | :green_circle: Green | Proves the fix resolves the bug |
|
||||
|
||||
## Test Plan
|
||||
|
||||
- [ ] CI red on test-only commit
|
||||
- [ ] CI green on fix commit
|
||||
- [ ] Added/updated E2E regression under `browser_tests/` or explained why not applicable
|
||||
- [ ] Manual verification (if applicable)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
### CI fails on test commit for unrelated reasons
|
||||
|
||||
Lint, typecheck, or other tests may fail — not just your new test. Check the CI logs carefully. If the failure is unrelated, fix it in a separate commit before the `test:` commit so the red X is clearly attributable to your test.
|
||||
|
||||
### Test passes when it should fail
|
||||
|
||||
The bug may only manifest under specific conditions (e.g., Windows paths, external model directories, certain workflow structures). Make sure your test setup matches the actual bug scenario. Check that you're not accidentally testing the happy path.
|
||||
|
||||
### Flaky Playwright tests
|
||||
|
||||
If your e2e test is intermittent, it doesn't prove anything. Use retrying assertions (`toBeVisible`, `toHaveText`) instead of `waitForTimeout`. See the `/writing-playwright-tests` skill for anti-patterns.
|
||||
|
||||
### Pre-existing CI failures on main
|
||||
|
||||
If main itself is red, branch from the last green commit or fix the pre-existing failure first. A red-green proof is meaningless if the baseline is already red.
|
||||
|
||||
## Reference
|
||||
|
||||
| Resource | Path |
|
||||
| --------------------- | -------------------------------------------------- |
|
||||
| Unit test framework | Vitest (`src/**/*.test.ts`) |
|
||||
| E2E test framework | Playwright (`browser_tests/tests/*.spec.ts`) |
|
||||
| E2E fixtures | `browser_tests/fixtures/` |
|
||||
| E2E assets | `browser_tests/assets/` |
|
||||
| Playwright skill | `.claude/skills/writing-playwright-tests/SKILL.md` |
|
||||
| Unit CI | `.github/workflows/ci-tests-unit.yaml` |
|
||||
| E2E CI | `.github/workflows/ci-tests-e2e.yaml` |
|
||||
| Lint CI | `.github/workflows/ci-lint-format.yaml` |
|
||||
| Testing anti-patterns | `reference/testing-anti-patterns.md` |
|
||||
| Related skill | `.claude/skills/perf-fix-with-proof/SKILL.md` |
|
||||
214
.claude/skills/red-green-fix/reference/testing-anti-patterns.md
Normal file
@@ -0,0 +1,214 @@
|
||||
# Testing Anti-Patterns for Red-Green Fixes
|
||||
|
||||
Common mistakes that undermine the red-green proof. Avoid these when writing the test commit (Step 1).
|
||||
|
||||
## Testing Implementation Details
|
||||
|
||||
Test observable behavior, not internal state.
|
||||
|
||||
**Bad** — coupling to internals:
|
||||
|
||||
```typescript
|
||||
it('uses cache internally', () => {
|
||||
const service = new UserService()
|
||||
service.getUser(1)
|
||||
expect(service._cache.has(1)).toBe(true) // Implementation detail
|
||||
})
|
||||
```
|
||||
|
||||
**Good** — testing through the public interface:
|
||||
|
||||
```typescript
|
||||
it('returns same user on repeated calls', async () => {
|
||||
const service = new UserService()
|
||||
const user1 = await service.getUser(1)
|
||||
const user2 = await service.getUser(1)
|
||||
expect(user1).toBe(user2) // Behavior, not implementation
|
||||
})
|
||||
```
|
||||
|
||||
Why this matters for red-green: if your test is coupled to internals, a valid fix that changes the implementation may break the test — even though the bug is fixed. The green commit should only require changing source code, not rewriting the test.
|
||||
|
||||
## Assertion-Free Tests
|
||||
|
||||
Every test must assert something meaningful. A test without assertions always passes — it cannot produce the red X needed in Step 1.
|
||||
|
||||
**Bad**:
|
||||
|
||||
```typescript
|
||||
it('processes the download', () => {
|
||||
processDownload('/models/checkpoints', 'model.safetensors')
|
||||
// No expect()!
|
||||
})
|
||||
```
|
||||
|
||||
**Good**:
|
||||
|
||||
```typescript
|
||||
it('processes the download to correct path', () => {
|
||||
const result = processDownload('/models/checkpoints', 'model.safetensors')
|
||||
expect(result.savePath).toBe('/models/checkpoints/model.safetensors')
|
||||
})
|
||||
```
|
||||
|
||||
## Over-Mocking
|
||||
|
||||
Mock only system boundaries (network, filesystem, Electron APIs). If you mock the module under test, you are testing your mocks — the test will not detect the real bug.
|
||||
|
||||
**Bad** — mocking everything:
|
||||
|
||||
```typescript
|
||||
vi.mock('./pathResolver')
|
||||
vi.mock('./validator')
|
||||
vi.mock('./downloader')
|
||||
|
||||
it('downloads model', () => {
|
||||
// This only tests that mocks were called, not that the bug exists
|
||||
})
|
||||
```
|
||||
|
||||
**Good** — mock only the boundary:
|
||||
|
||||
```typescript
|
||||
vi.mock('./electronAPI') // Boundary: Electron IPC
|
||||
|
||||
it('resolves absolute path correctly', () => {
|
||||
const result = resolveModelPath('/root/models', '/root/models/checkpoints')
|
||||
expect(result).toBe('/root/models/checkpoints')
|
||||
})
|
||||
```
|
||||
|
||||
See also: [Don't Mock What You Don't Own](https://hynek.me/articles/what-to-mock-in-5-mins/)
|
||||
|
||||
## Giant Tests
|
||||
|
||||
A test that covers the entire flow makes it hard to pinpoint which part catches the bug. Keep it focused — one concept per test.
|
||||
|
||||
**Bad**:
|
||||
|
||||
```typescript
|
||||
it('full model download flow', async () => {
|
||||
// 80 lines: load workflow, open dialog, select model,
|
||||
// click download, verify path, check progress, confirm completion
|
||||
})
|
||||
```
|
||||
|
||||
**Good**:
|
||||
|
||||
```typescript
|
||||
it('resolves absolute savePath without nesting under modelsDirectory', () => {
|
||||
const result = getLocalSavePath(
|
||||
'/models',
|
||||
'/models/checkpoints',
|
||||
'file.safetensors'
|
||||
)
|
||||
expect(result).toBe('/models/checkpoints/file.safetensors')
|
||||
})
|
||||
```
|
||||
|
||||
If the bug is in path resolution, test path resolution — not the entire download flow.
|
||||
|
||||
## Test Duplication
|
||||
|
||||
Duplicated test code hides what actually differs between cases. Use parameterized tests.
|
||||
|
||||
**Bad**:
|
||||
|
||||
```typescript
|
||||
it('resolves checkpoints path', () => {
|
||||
expect(resolve('/models', '/models/checkpoints', 'a.safetensors')).toBe(
|
||||
'/models/checkpoints/a.safetensors'
|
||||
)
|
||||
})
|
||||
it('resolves loras path', () => {
|
||||
expect(resolve('/models', '/models/loras', 'b.safetensors')).toBe(
|
||||
'/models/loras/b.safetensors'
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
**Good**:
|
||||
|
||||
```typescript
|
||||
it.each([
|
||||
['/models/checkpoints', 'a.safetensors', '/models/checkpoints/a.safetensors'],
|
||||
['/models/loras', 'b.safetensors', '/models/loras/b.safetensors']
|
||||
])('resolves %s/%s to %s', (dir, file, expected) => {
|
||||
expect(resolve('/models', dir, file)).toBe(expected)
|
||||
})
|
||||
```
|
||||
|
||||
## Flaky Tests
|
||||
|
||||
A flaky test cannot prove anything — it may show red for reasons unrelated to the bug, or green despite the bug still existing.
|
||||
|
||||
**Common causes in this codebase:**
|
||||
|
||||
| Cause | Fix |
|
||||
| -------------------------------------- | --------------------------------------- |
|
||||
| Missing `nextFrame()` after canvas ops | Add `await comfyPage.nextFrame()` |
|
||||
| `waitForTimeout` instead of assertions | Use `toBeVisible()`, `toHaveText()` |
|
||||
| Shared state between tests | Isolate with `afterEach` / `beforeEach` |
|
||||
| Timing-dependent logic | Use `expect.poll()` or `toPass()` |
|
||||
|
||||
## Gaming the Red-Green Process
|
||||
|
||||
These are ways the red-green proof gets invalidated during Step 2 (the fix commit). The test from Step 1 is immutable — if any of these happen, restart from Step 1.
|
||||
|
||||
**Weakening the assertion to make it pass:**
|
||||
|
||||
```typescript
|
||||
// Step 1 (red) — strict assertion
|
||||
expect(result).toBe('/external/drive/models/checkpoints/file.safetensors')
|
||||
|
||||
// Step 2 (green) — weakened to pass without a real fix
|
||||
expect(result).toBeDefined() // This proves nothing
|
||||
```
|
||||
|
||||
**Updating snapshots to bless the bug:**
|
||||
|
||||
```bash
|
||||
# Instead of fixing the code, just updating the snapshot to match buggy output
|
||||
pnpm test:unit -- --update
|
||||
```
|
||||
|
||||
If a snapshot needs updating, the fix should change the code behavior, not the expected output.
|
||||
|
||||
**Adding mocks in Step 2 that hide the failure:**
|
||||
|
||||
```typescript
|
||||
// Step 2 adds a mock that didn't exist in Step 1
|
||||
vi.mock('./pathResolver', () => ({
|
||||
resolve: () => '/expected/path' // Hardcoded to pass
|
||||
}))
|
||||
```
|
||||
|
||||
Step 2 should only change source code — not test infrastructure.
|
||||
|
||||
## Testing the Happy Path Only
|
||||
|
||||
The red-green pattern specifically requires the test to exercise the **broken path**. If you only test the case that already works, the test will pass (green) on Step 1 — defeating the purpose.
|
||||
|
||||
**Bad** — testing the default case that works:
|
||||
|
||||
```typescript
|
||||
it('downloads to default models directory', () => {
|
||||
// This already works — it won't produce a red X
|
||||
const result = resolve('/models', 'checkpoints', 'file.safetensors')
|
||||
expect(result).toBe('/models/checkpoints/file.safetensors')
|
||||
})
|
||||
```
|
||||
|
||||
**Good** — testing the case that is actually broken:
|
||||
|
||||
```typescript
|
||||
it('downloads to external models directory configured via extra_model_paths', () => {
|
||||
// This is the broken case — absolute path from folder_paths API
|
||||
const result = resolve(
|
||||
'/models',
|
||||
'/external/drive/models/checkpoints',
|
||||
'file.safetensors'
|
||||
)
|
||||
expect(result).toBe('/external/drive/models/checkpoints/file.safetensors')
|
||||
})
|
||||
```
|
||||
@@ -3,6 +3,7 @@ issue_enrichment:
|
||||
enabled: true
|
||||
reviews:
|
||||
high_level_summary: false
|
||||
request_changes_workflow: true
|
||||
auto_review:
|
||||
drafts: true
|
||||
ignore_title_keywords:
|
||||
@@ -13,10 +14,14 @@ reviews:
|
||||
- github-actions
|
||||
- github-actions[bot]
|
||||
pre_merge_checks:
|
||||
override_requested_reviewers_only: true
|
||||
custom_checks:
|
||||
- name: End-to-end regression coverage for fixes
|
||||
mode: warning
|
||||
mode: error
|
||||
instructions: |
|
||||
Use only PR metadata already available in the review context: the PR title, commit subjects in this PR, the files changed in this PR relative to the PR base (equivalent to `base...head`), and the PR description.
|
||||
Do not rely on shell commands. Do not inspect reverse diffs, files changed only on the base branch, or files outside this PR. If the changed-file list or commit subjects are unavailable, mark the check inconclusive instead of guessing.
|
||||
|
||||
Pass if at least one of the following is true:
|
||||
1. Neither the PR title nor any commit subject in the PR uses bug-fix language such as `fix`, `fixed`, `fixes`, `fixing`, `bugfix`, or `hotfix`.
|
||||
2. The PR changes at least one file under `browser_tests/`.
|
||||
|
||||
2
.gitattributes
vendored
@@ -3,4 +3,6 @@
|
||||
|
||||
# Generated files
|
||||
packages/registry-types/src/comfyRegistryTypes.ts linguist-generated=true
|
||||
packages/ingest-types/src/types.gen.ts linguist-generated=true
|
||||
packages/ingest-types/src/zod.gen.ts linguist-generated=true
|
||||
src/workbench/extensions/manager/types/generatedManagerTypes.ts linguist-generated=true
|
||||
|
||||
@@ -44,7 +44,6 @@ runs:
|
||||
python -m pip install --upgrade pip
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install -r requirements.txt
|
||||
pip install wait-for-it
|
||||
|
||||
- name: Start ComfyUI server
|
||||
if: ${{ inputs.launch_server == 'true' }}
|
||||
@@ -52,4 +51,8 @@ runs:
|
||||
working-directory: ComfyUI
|
||||
run: |
|
||||
python main.py --cpu --multi-user --front-end-root ../dist ${{ inputs.extra_server_params }} &
|
||||
wait-for-it --service 127.0.0.1:8188 -t 600
|
||||
for i in $(seq 1 300); do
|
||||
curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1 && echo "Server ready" && exit 0
|
||||
sleep 2
|
||||
done
|
||||
echo "::error::ComfyUI server did not start within 600s" && exit 1
|
||||
|
||||
48
.github/workflows/ci-perf-report.yaml
vendored
@@ -10,7 +10,7 @@ on:
|
||||
|
||||
concurrency:
|
||||
group: perf-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -26,12 +26,15 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
permissions:
|
||||
contents: read
|
||||
contents: write
|
||||
packages: read
|
||||
actions: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Setup frontend
|
||||
uses: ./.github/actions/setup-frontend
|
||||
@@ -68,3 +71,44 @@ jobs:
|
||||
with:
|
||||
name: perf-meta
|
||||
path: temp/perf-meta/
|
||||
|
||||
- name: Save perf baseline to perf-data branch
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.perf.outcome == 'success'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git config url."https://x-access-token:${GH_TOKEN}@github.com/".insteadOf "https://github.com/"
|
||||
|
||||
cp test-results/perf-metrics.json /tmp/perf-metrics.json
|
||||
|
||||
git fetch origin perf-data || {
|
||||
echo "Creating perf-data branch"
|
||||
git checkout --orphan perf-data
|
||||
git rm -rf . 2>/dev/null || true
|
||||
echo "# Performance Baselines" > README.md
|
||||
mkdir -p baselines
|
||||
git add README.md baselines
|
||||
git commit -m "Initialize perf-data branch"
|
||||
git push origin perf-data
|
||||
git fetch origin perf-data
|
||||
}
|
||||
|
||||
git worktree add /tmp/perf-data origin/perf-data
|
||||
|
||||
TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
SHA=$(echo "${{ github.sha }}" | cut -c1-8)
|
||||
mkdir -p /tmp/perf-data/baselines
|
||||
cp /tmp/perf-metrics.json "/tmp/perf-data/baselines/perf-${TIMESTAMP}-${SHA}.json"
|
||||
|
||||
# Keep only last 20 baselines
|
||||
cd /tmp/perf-data
|
||||
ls -t baselines/perf-*.json 2>/dev/null | tail -n +21 | xargs -r rm
|
||||
|
||||
git -C /tmp/perf-data add baselines/
|
||||
git -C /tmp/perf-data commit -m "perf: add baseline for ${SHA}" || echo "No changes to commit"
|
||||
git -C /tmp/perf-data push origin HEAD:perf-data
|
||||
|
||||
git worktree remove /tmp/perf-data --force 2>/dev/null || true
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
46
.github/workflows/pr-perf-report.yaml
vendored
@@ -10,6 +10,7 @@ permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
actions: read
|
||||
|
||||
jobs:
|
||||
comment:
|
||||
@@ -73,7 +74,28 @@ jobs:
|
||||
core.setOutput('number', String(pr.number));
|
||||
core.setOutput('base', trustedBase);
|
||||
|
||||
- name: Check if results are still current
|
||||
id: sha-check
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const prNumber = Number('${{ steps.pr-meta.outputs.number }}');
|
||||
const { data: pr } = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
const runSha = context.payload.workflow_run.head_sha;
|
||||
const currentSha = pr.head.sha;
|
||||
if (runSha !== currentSha) {
|
||||
core.info(`Skipping stale report: run SHA ${runSha} != current PR SHA ${currentSha}`);
|
||||
core.setOutput('stale', 'true');
|
||||
} else {
|
||||
core.setOutput('stale', 'false');
|
||||
}
|
||||
|
||||
- name: Download PR perf metrics
|
||||
if: steps.sha-check.outputs.stale != 'true'
|
||||
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
|
||||
with:
|
||||
name: perf-metrics
|
||||
@@ -81,6 +103,7 @@ jobs:
|
||||
path: test-results/
|
||||
|
||||
- name: Download baseline perf metrics
|
||||
if: steps.sha-check.outputs.stale != 'true'
|
||||
uses: dawidd6/action-download-artifact@0bd50d53a6d7fb5cb921e607957e9cc12b4ce392 # v12
|
||||
with:
|
||||
branch: ${{ steps.pr-meta.outputs.base }}
|
||||
@@ -90,10 +113,33 @@ jobs:
|
||||
path: temp/perf-baseline/
|
||||
if_no_artifact_found: warn
|
||||
|
||||
- name: Load historical baselines from perf-data branch
|
||||
if: steps.sha-check.outputs.stale != 'true'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
mkdir -p temp/perf-history
|
||||
|
||||
git fetch origin perf-data 2>/dev/null || {
|
||||
echo "perf-data branch not found, skipping historical data"
|
||||
exit 0
|
||||
}
|
||||
|
||||
INDEX=0
|
||||
for file in $(git ls-tree --name-only origin/perf-data baselines/ 2>/dev/null | sort -r | head -5); do
|
||||
DIR="temp/perf-history/$INDEX"
|
||||
mkdir -p "$DIR"
|
||||
git show "origin/perf-data:${file}" > "$DIR/perf-metrics.json" 2>/dev/null || true
|
||||
INDEX=$((INDEX + 1))
|
||||
done
|
||||
|
||||
echo "Loaded $INDEX historical baselines"
|
||||
|
||||
- name: Generate perf report
|
||||
if: steps.sha-check.outputs.stale != 'true'
|
||||
run: npx --yes tsx scripts/perf-report.ts > perf-report.md
|
||||
|
||||
- name: Post PR comment
|
||||
if: steps.sha-check.outputs.stale != 'true'
|
||||
uses: ./.github/actions/post-pr-report-comment
|
||||
with:
|
||||
pr-number: ${{ steps.pr-meta.outputs.number }}
|
||||
|
||||
564
.github/workflows/pr-qa.yaml
vendored
Normal file
@@ -0,0 +1,564 @@
|
||||
# Automated QA of ComfyUI frontend using Claude CLI + Playwright MCP.
|
||||
# Two modes:
|
||||
# Focused (qa-changes label): Linux-only, tests areas affected by PR changes
|
||||
# Full (qa-full label): 3-OS matrix, full test plan
|
||||
name: 'PR: QA'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
mode:
|
||||
description: 'QA mode'
|
||||
type: choice
|
||||
options: [focused, full]
|
||||
default: focused
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
resolve-matrix:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
os: ${{ steps.set.outputs.os }}
|
||||
mode: ${{ steps.set.outputs.mode }}
|
||||
skip: ${{ steps.set.outputs.skip }}
|
||||
steps:
|
||||
- name: Determine QA mode
|
||||
id: set
|
||||
env:
|
||||
LABEL: ${{ github.event.label.name }}
|
||||
EVENT_ACTION: ${{ github.event.action }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
INPUT_MODE: ${{ inputs.mode }}
|
||||
run: |
|
||||
FULL=false
|
||||
|
||||
# Only run on label events if it's one of our labels
|
||||
if [ "$EVENT_ACTION" = "labeled" ] && \
|
||||
[ "$LABEL" != "qa-changes" ] && [ "$LABEL" != "qa-full" ]; then
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# Full QA triggers
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] && \
|
||||
[ "$INPUT_MODE" = "full" ]; then
|
||||
FULL=true
|
||||
fi
|
||||
if [ "$LABEL" = "qa-full" ]; then
|
||||
FULL=true
|
||||
fi
|
||||
|
||||
if [ "$FULL" = "true" ]; then
|
||||
echo 'os=["ubuntu-latest","macos-latest","windows-latest"]' >> "$GITHUB_OUTPUT"
|
||||
echo "mode=full" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo 'os=["ubuntu-latest"]' >> "$GITHUB_OUTPUT"
|
||||
echo "mode=focused" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
echo "Mode: $([ "$FULL" = "true" ] && echo full || echo focused)"
|
||||
|
||||
qa:
|
||||
needs: resolve-matrix
|
||||
if: needs.resolve-matrix.outputs.skip != 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: ${{ fromJson(needs.resolve-matrix.outputs.os) }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 60
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
env:
|
||||
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
|
||||
steps:
|
||||
- name: Set QA artifacts path
|
||||
shell: bash
|
||||
run: echo "QA_ARTIFACTS=$RUNNER_TEMP/qa-artifacts" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.head_ref || github.ref }}
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Setup frontend
|
||||
uses: ./.github/actions/setup-frontend
|
||||
with:
|
||||
include_build_step: true
|
||||
|
||||
- name: Setup and start ComfyUI server
|
||||
uses: ./.github/actions/setup-comfyui-server
|
||||
with:
|
||||
launch_server: 'true'
|
||||
|
||||
- name: Wait for ComfyUI server
|
||||
shell: bash
|
||||
run: |
|
||||
echo "Waiting for ComfyUI server..."
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1; then
|
||||
echo "Server ready"; exit 0
|
||||
fi; sleep 2
|
||||
done
|
||||
echo "::error::Server timeout"; exit 1
|
||||
|
||||
- name: Install Playwright for MCP
|
||||
shell: bash
|
||||
run: pnpm exec playwright install chromium --with-deps
|
||||
|
||||
- name: Install Claude Code
|
||||
shell: bash
|
||||
run: npm install -g @anthropic-ai/claude-code@2.1.71
|
||||
|
||||
# --- Virtual display + recording: Linux ---
|
||||
- name: Setup Xvfb (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq xvfb ffmpeg x11-xserver-utils >/dev/null 2>&1
|
||||
Xvfb :99 -screen 0 1280x720x24 -ac -nolisten tcp &
|
||||
echo $! > "${{ runner.temp }}/xvfb.pid"
|
||||
sleep 2
|
||||
echo "DISPLAY=:99" >> "$GITHUB_ENV"
|
||||
# Set black background and normal cursor (removes the default X cursor)
|
||||
DISPLAY=:99 xsetroot -solid black -cursor_name left_ptr 2>/dev/null || true
|
||||
|
||||
- name: Start recording (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -y -f x11grab -video_size 1280x720 -framerate 10 \
|
||||
-i :99.0 -c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 1
|
||||
|
||||
# --- Recording: macOS (avfoundation) ---
|
||||
- name: Install ffmpeg (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
run: brew install ffmpeg
|
||||
|
||||
- name: Grant screen recording permission (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
run: |
|
||||
# Grant kTCCServiceScreenCapture to ffmpeg and bash so avfoundation
|
||||
# recording works without the blocking permission dialog.
|
||||
FFMPEG_PATH=$(which ffmpeg)
|
||||
echo "Granting screen recording permission to: $FFMPEG_PATH"
|
||||
# macOS 14+ (Sonoma/Sequoia) TCC.db schema has 17 columns
|
||||
for BIN in "$FFMPEG_PATH" "/bin/bash"; do
|
||||
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
|
||||
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,0,'UNUSED',NULL,0,$(date +%s),NULL,NULL,NULL,NULL);" 2>/dev/null \
|
||||
|| sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
|
||||
"INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','${BIN}',1,2,0,1,NULL,NULL,NULL,'UNUSED',NULL,0,$(date +%s));" 2>/dev/null \
|
||||
|| echo "Warning: Could not grant TCC permission to ${BIN}"
|
||||
done
|
||||
|
||||
- name: Start recording (macOS)
|
||||
if: runner.os == 'macOS'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -f avfoundation -list_devices true -i "" 2>&1 || true
|
||||
ffmpeg -y -f avfoundation -framerate 10 -capture_cursor 1 \
|
||||
-i "Capture screen 0:none" -c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 2
|
||||
|
||||
# --- Recording: Windows (gdigrab) ---
|
||||
- name: Install ffmpeg (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
shell: pwsh
|
||||
run: choco install ffmpeg -y --no-progress
|
||||
|
||||
- name: Start recording (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
ffmpeg -y -f gdigrab -framerate 10 -i desktop \
|
||||
-c:v libx264 -preset ultrafast -crf 28 \
|
||||
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
|
||||
echo $! > "${{ runner.temp }}/ffmpeg.pid"
|
||||
sleep 1
|
||||
|
||||
- name: Create MCP config
|
||||
shell: bash
|
||||
run: |
|
||||
cat > "${{ runner.temp }}/mcp-config.json" <<EOF
|
||||
{"mcpServers":{"playwright":{"command":"npx","args":["@playwright/mcp@0.0.68"]}}}
|
||||
EOF
|
||||
|
||||
- name: Get PR diff for focused QA
|
||||
if: needs.resolve-matrix.outputs.mode == 'focused'
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
gh pr diff ${{ github.event.pull_request.number || '' }} \
|
||||
--repo ${{ github.repository }} > "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || \
|
||||
git diff origin/main...HEAD > "${{ runner.temp }}/pr-diff.txt"
|
||||
|
||||
# Summarize changed files for the prompt
|
||||
echo "Changed files:"
|
||||
grep '^diff --git' "${{ runner.temp }}/pr-diff.txt" | \
|
||||
sed 's|diff --git a/||;s| b/.*||' | sort -u | tee "${{ runner.temp }}/changed-files.txt"
|
||||
|
||||
- name: Write QA prompt
|
||||
shell: bash
|
||||
env:
|
||||
BRANCH: ${{ github.head_ref || github.ref_name }}
|
||||
PR_NUM: ${{ github.event.pull_request.number || 'N/A' }}
|
||||
SHA: ${{ github.sha }}
|
||||
run: |
|
||||
OS_LOWER=$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')
|
||||
|
||||
if [ "$QA_MODE" = "full" ]; then
|
||||
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
|
||||
You are running a FULL automated QA pass on the ComfyUI frontend.
|
||||
Read the file .claude/skills/comfy-qa/SKILL.md and follow the FULL QA test plan.
|
||||
|
||||
Environment: CI=true, OS=${{ runner.os }}
|
||||
Server URL: http://127.0.0.1:8188
|
||||
Branch: ${BRANCH}
|
||||
PR: #${PR_NUM}
|
||||
Commit: ${SHA}
|
||||
|
||||
1. Use playwright MCP tools to navigate http://127.0.0.1:8188
|
||||
2. Run the FULL QA test plan from the skill file
|
||||
3. Take screenshots of failures or notable states
|
||||
4. Save report to docs/qa/ as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
|
||||
5. Commit and push the report to this branch
|
||||
|
||||
Do NOT create a new PR. Do NOT post PR comments.
|
||||
Skip tests not available in CI (file dialogs, GPU execution).
|
||||
PROMPT
|
||||
else
|
||||
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
|
||||
You are running a FOCUSED QA pass on a pull request to the ComfyUI frontend.
|
||||
Your goal is to verify that the changes in this PR work correctly and don't break related functionality.
|
||||
|
||||
Environment: CI=true, OS=${{ runner.os }}
|
||||
Server URL: http://127.0.0.1:8188
|
||||
Branch: ${BRANCH}
|
||||
PR: #${PR_NUM}
|
||||
Commit: ${SHA}
|
||||
|
||||
CHANGED FILES:
|
||||
$(cat "${{ runner.temp }}/changed-files.txt" 2>/dev/null || echo "Unknown")
|
||||
|
||||
DIFF (truncated to 500 lines):
|
||||
$(head -500 "${{ runner.temp }}/pr-diff.txt" 2>/dev/null || echo "No diff available")
|
||||
|
||||
Instructions:
|
||||
1. Read the diff above to understand what changed in this PR
|
||||
2. Use playwright MCP tools to navigate http://127.0.0.1:8188
|
||||
3. Test the specific UI areas affected by these changes
|
||||
4. Also do a quick smoke test of core functionality (app loads, canvas renders, sidebar works)
|
||||
5. Take screenshots of any failures or the areas you tested
|
||||
6. Save a concise report to docs/qa/ as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
|
||||
7. Commit and push the report to this branch
|
||||
|
||||
Focus on:
|
||||
- Does the changed functionality work as expected?
|
||||
- Are there visual regressions in affected areas?
|
||||
- Do related features still work?
|
||||
|
||||
Do NOT run the full QA test plan. Do NOT create a new PR. Do NOT post PR comments.
|
||||
Skip tests not available in CI (file dialogs, GPU execution).
|
||||
PROMPT
|
||||
fi
|
||||
|
||||
- name: Run Claude QA
|
||||
shell: bash
|
||||
env:
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
CI: 'true'
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
MAX_TURNS=128
|
||||
if [ "$QA_MODE" = "focused" ]; then MAX_TURNS=30; fi
|
||||
|
||||
cat "${{ runner.temp }}/qa-prompt.txt" | claude --print --verbose \
|
||||
--max-turns "$MAX_TURNS" \
|
||||
--mcp-config "${{ runner.temp }}/mcp-config.json" \
|
||||
--allowedTools "mcp__playwright__browser_navigate,mcp__playwright__browser_snapshot,mcp__playwright__browser_click,mcp__playwright__browser_type,mcp__playwright__browser_press_key,mcp__playwright__browser_take_screenshot,mcp__playwright__browser_hover,mcp__playwright__browser_drag,mcp__playwright__browser_select_option,mcp__playwright__browser_handle_dialog,mcp__playwright__browser_tab_list,mcp__playwright__browser_tab_new,mcp__playwright__browser_tab_select,mcp__playwright__browser_tab_close,mcp__playwright__browser_console_messages,mcp__playwright__browser_resize,mcp__playwright__browser_wait_for,Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git status:*),Bash(git log:*),Bash(git diff:*),Bash(date:*),Bash(ls:*),Bash(mkdir:*),Read,Write,Edit,Glob,Grep"
|
||||
|
||||
- name: Stop recording
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
PID_FILE="${{ runner.temp }}/ffmpeg.pid"
|
||||
if [ -f "$PID_FILE" ]; then
|
||||
if [ "$RUNNER_OS" = "Windows" ]; then
|
||||
# Graceful stop: taskkill without /F sends WM_CLOSE so ffmpeg
|
||||
# can finalize the mp4 (write moov atom). Force-kill as fallback.
|
||||
taskkill //PID $(cat "$PID_FILE") 2>/dev/null || true
|
||||
sleep 5
|
||||
taskkill //F //PID $(cat "$PID_FILE") 2>/dev/null || true
|
||||
else
|
||||
kill -INT $(cat "$PID_FILE") 2>/dev/null || true
|
||||
sleep 3; kill $(cat "$PID_FILE") 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
[ -f "$QA_ARTIFACTS/qa-session.mp4" ] && \
|
||||
echo "Video: $(du -h "$QA_ARTIFACTS/qa-session.mp4" | cut -f1)" || \
|
||||
echo "No video (non-fatal)"
|
||||
|
||||
- name: Collect artifacts
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
mkdir -p "$QA_ARTIFACTS"
|
||||
cp -r docs/qa/* "$QA_ARTIFACTS/" 2>/dev/null || true
|
||||
ls -la "$QA_ARTIFACTS/" || true
|
||||
|
||||
- name: Upload QA artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v6.2.0
|
||||
with:
|
||||
name: qa-report-${{ runner.os }}-${{ github.run_id }}
|
||||
path: ${{ env.QA_ARTIFACTS }}/
|
||||
retention-days: 14
|
||||
|
||||
- name: Cleanup (Linux)
|
||||
if: always() && runner.os == 'Linux'
|
||||
run: kill $(cat "${{ runner.temp }}/xvfb.pid") 2>/dev/null || true
|
||||
|
||||
report:
|
||||
needs: [resolve-matrix, qa]
|
||||
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Download QA artifacts
|
||||
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
|
||||
with:
|
||||
path: qa-artifacts
|
||||
pattern: qa-report-*
|
||||
|
||||
- name: Install ffmpeg
|
||||
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
|
||||
|
||||
- name: Deploy videos to Cloudflare Pages
|
||||
id: deploy-videos
|
||||
env:
|
||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
||||
RAW_BRANCH: ${{ github.head_ref || github.ref_name }}
|
||||
run: |
|
||||
npm install -g wrangler@^4.0.0 >/dev/null 2>&1
|
||||
|
||||
DEPLOY_DIR=$(mktemp -d)
|
||||
mkdir -p "$DEPLOY_DIR"
|
||||
|
||||
for os in Linux macOS Windows; do
|
||||
VID="qa-artifacts/qa-report-${os}-${{ github.run_id }}/qa-session.mp4"
|
||||
if [ -f "$VID" ]; then
|
||||
cp "$VID" "$DEPLOY_DIR/qa-${os}.mp4"
|
||||
echo "Found ${os} video ($(du -h "$VID" | cut -f1))"
|
||||
|
||||
# Generate GIF thumbnail: 8s starting at 10s, 480px wide, 8fps
|
||||
ffmpeg -y -ss 10 -i "$VID" -t 8 \
|
||||
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
|
||||
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|
||||
|| ffmpeg -y -i "$VID" -t 8 \
|
||||
-vf "fps=8,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen=max_colors=64[p];[s1][p]paletteuse=dither=bayer" \
|
||||
-loop 0 "$DEPLOY_DIR/qa-${os}-thumb.gif" 2>/dev/null \
|
||||
|| echo "GIF generation failed for ${os} (non-fatal)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Build video cards HTML
|
||||
CARDS=""
|
||||
ICONS_Linux="🐧" ICONS_macOS="🍎" ICONS_Windows="🪟"
|
||||
for os in Linux macOS Windows; do
|
||||
eval "ICON=\$ICONS_${os}"
|
||||
if [ -f "$DEPLOY_DIR/qa-${os}.mp4" ]; then
|
||||
CARDS="${CARDS}<div class=card><video controls autoplay muted loop preload=metadata><source src=qa-${os}.mp4 type=video/mp4></video><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><a class=download href=qa-${os}.mp4 download>Download</a></div></div>"
|
||||
else
|
||||
CARDS="${CARDS}<div class=card><div class=empty-card>No recording available</div><div class=card-body><span class=platform><span class=icon>${ICON}</span> ${os}</span><span class='badge missing'>Missing</span></div></div>"
|
||||
fi
|
||||
done
|
||||
|
||||
cat > "$DEPLOY_DIR/index.html" <<INDEXEOF
|
||||
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><title>QA Session Recordings</title>
|
||||
<style>
|
||||
*{margin:0;padding:0;box-sizing:border-box}body{background:#0d1117;color:#e6edf3;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;min-height:100vh;padding:2rem 1rem}.container{max-width:1200px;margin:0 auto}header{display:flex;align-items:center;gap:.75rem;margin-bottom:2rem;padding-bottom:1rem;border-bottom:1px solid #30363d}h1{font-size:1.5rem;font-weight:600}.meta{color:#8b949e;font-size:.875rem;margin-top:.25rem}.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(340px,1fr));gap:1.25rem}.card{background:#161b22;border:1px solid #30363d;border-radius:.5rem;overflow:hidden;transition:border-color .15s}.card:hover{border-color:#58a6ff}.card video{width:100%;display:block;background:#010409;aspect-ratio:16/9;object-fit:contain}.card-body{padding:.75rem 1rem;display:flex;align-items:center;justify-content:space-between}.platform{display:flex;align-items:center;gap:.5rem;font-weight:500}.icon{font-size:1.25rem}.badge{font-size:.75rem;padding:.125rem .5rem;border-radius:999px;background:#1f6feb33;color:#58a6ff;border:1px solid #1f6feb55}.badge.missing{background:#da363333;color:#f85149;border-color:#da363355}.empty-card{display:flex;align-items:center;justify-content:center;min-height:200px;color:#484f58;font-size:.875rem}a.download{color:#58a6ff;text-decoration:none;font-size:.8125rem}a.download:hover{text-decoration:underline}
|
||||
</style></head><body><div class=container>
|
||||
<header><svg width=28 height=28 viewBox="0 0 24 24" fill=none stroke=#58a6ff stroke-width=2 stroke-linecap=round stroke-linejoin=round><polygon points="23 7 16 12 23 17 23 7"/><rect x=1 y=5 width=15 height=14 rx=2 ry=2/></svg><div><h1>QA Session Recordings</h1><div class=meta>ComfyUI Frontend · Automated QA</div></div></header>
|
||||
<div class=grid>${CARDS}</div>
|
||||
</div></body></html>
|
||||
INDEXEOF
|
||||
|
||||
# 404 page so Cloudflare Pages returns proper 404 for missing files
|
||||
# (instead of SPA fallback serving index.html)
|
||||
cat > "$DEPLOY_DIR/404.html" <<'ERROREOF'
|
||||
<!DOCTYPE html><html><head><meta charset=utf-8><title>404</title>
|
||||
<style>body{background:#0d1117;color:#8b949e;font-family:sans-serif;display:flex;align-items:center;justify-content:center;min-height:100vh;margin:0}div{text-align:center}h1{color:#f85149;font-size:3rem;margin-bottom:.5rem}p{font-size:1rem}</style>
|
||||
</head><body><div><h1>404</h1><p>File not found. The QA recording may have failed or been cancelled.</p></div></body></html>
|
||||
ERROREOF
|
||||
|
||||
# Sanitize branch name for Cloudflare Pages URL (same rules CF uses)
|
||||
BRANCH=$(echo "$RAW_BRANCH" | sed 's/[^a-zA-Z0-9-]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | cut -c1-28)
|
||||
URL=$(wrangler pages deploy "$DEPLOY_DIR" \
|
||||
--project-name="comfyui-qa-videos" \
|
||||
--branch="$BRANCH" 2>&1 \
|
||||
| grep -oE 'https://[a-zA-Z0-9.-]+\.pages\.dev\S*' | head -1)
|
||||
|
||||
echo "url=${URL:-https://${BRANCH}.comfyui-qa-videos.pages.dev}" >> "$GITHUB_OUTPUT"
|
||||
echo "Deployed to: ${URL}"
|
||||
|
||||
- name: Post QA comment on PR
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
VIDEO_BASE: ${{ steps.deploy-videos.outputs.url }}
|
||||
QA_MODE: ${{ needs.resolve-matrix.outputs.mode }}
|
||||
run: |
|
||||
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
COMMENT_MARKER="<!-- QA_REPORT_COMMENT -->"
|
||||
|
||||
MODE_BADGE="🔍 Focused"
|
||||
if [ "$QA_MODE" = "full" ]; then MODE_BADGE="🔬 Full (3-OS)"; fi
|
||||
|
||||
# Build video section with GIF thumbnails linking to full videos
|
||||
VIDEO_SECTION=""
|
||||
for os in Linux macOS Windows; do
|
||||
GIF_URL="${VIDEO_BASE}/qa-${os}-thumb.gif"
|
||||
VID_URL="${VIDEO_BASE}/qa-${os}.mp4"
|
||||
if curl -sf --head "$VID_URL" >/dev/null 2>&1; then
|
||||
if curl -sf --head "$GIF_URL" >/dev/null 2>&1; then
|
||||
VIDEO_SECTION="${VIDEO_SECTION}[](${VID_URL})"$'\n'
|
||||
else
|
||||
VIDEO_SECTION="${VIDEO_SECTION}[${os} video](${VID_URL})"$'\n'
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
BODY=$(cat <<EOF
|
||||
${COMMENT_MARKER}
|
||||
## QA ${MODE_BADGE}
|
||||
|
||||
${VIDEO_SECTION}
|
||||
**Run**: [${RUN}](${RUN}) · [Download artifacts](${RUN}#artifacts) · [All videos](${VIDEO_BASE})
|
||||
EOF
|
||||
)
|
||||
|
||||
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
|
||||
--jq ".[] | select(.body | contains(\"${COMMENT_MARKER}\")) | .id" | head -1)
|
||||
|
||||
if [ -n "$EXISTING" ]; then
|
||||
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
|
||||
--field body="$BODY"
|
||||
else
|
||||
gh pr comment ${{ github.event.pull_request.number }} \
|
||||
--repo ${{ github.repository }} --body "$BODY"
|
||||
fi
|
||||
|
||||
- name: Remove QA label
|
||||
if: >-
|
||||
github.event.label.name == 'qa-changes' ||
|
||||
github.event.label.name == 'qa-full'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
LABEL: ${{ github.event.label.name }}
|
||||
run: |
|
||||
gh pr edit ${{ github.event.pull_request.number }} \
|
||||
--repo ${{ github.repository }} --remove-label "$LABEL"
|
||||
|
||||
video-review:
|
||||
needs: [qa, report]
|
||||
if: always() && github.event.pull_request.number && github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
|
||||
with:
|
||||
node-version: 24
|
||||
|
||||
- name: Install pnpm
|
||||
uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0
|
||||
with:
|
||||
version: 10
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install ffmpeg
|
||||
run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg >/dev/null 2>&1
|
||||
|
||||
- name: Download QA artifacts
|
||||
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
|
||||
with:
|
||||
path: qa-artifacts
|
||||
pattern: qa-report-*
|
||||
|
||||
- name: Run video review
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
run: |
|
||||
mkdir -p video-reviews
|
||||
for vid in qa-artifacts/qa-report-*/qa-session.mp4; do
|
||||
[ -f "$vid" ] || continue
|
||||
echo "::group::Reviewing $vid"
|
||||
pnpm exec tsx scripts/qa-video-review.ts \
|
||||
--artifacts-dir qa-artifacts \
|
||||
--output-dir video-reviews \
|
||||
--video-file "$vid" \
|
||||
--model gpt-4o || true
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
- name: Post video review comment
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
MARKER="<!-- QA_VIDEO_REVIEW_COMMENT -->"
|
||||
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
|
||||
# Combine per-platform reports
|
||||
REPORT=""
|
||||
for f in video-reviews/*-qa-video-report.md; do
|
||||
[ -f "$f" ] || continue
|
||||
[ -n "$REPORT" ] && REPORT="${REPORT}
|
||||
---
|
||||
"
|
||||
REPORT="${REPORT}$(cat "$f")"
|
||||
done
|
||||
|
||||
[ -z "$REPORT" ] && REPORT="No video reports were generated."
|
||||
|
||||
BODY="${MARKER}
|
||||
## QA Video Report
|
||||
|
||||
${REPORT}
|
||||
|
||||
---
|
||||
**Run**: [${RUN}](${RUN})"
|
||||
|
||||
EXISTING=$(gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
|
||||
--jq ".[] | select(.body | contains(\"${MARKER}\")) | .id" | head -1)
|
||||
|
||||
if [ -n "$EXISTING" ]; then
|
||||
gh api --method PATCH "repos/${{ github.repository }}/issues/comments/${EXISTING}" \
|
||||
--field body="$BODY"
|
||||
else
|
||||
gh pr comment ${{ github.event.pull_request.number }} \
|
||||
--repo ${{ github.repository }} --body "$BODY"
|
||||
fi
|
||||
4876
browser_tests/assets/large-graph-workflow.json
Normal file
@@ -8,7 +8,7 @@
|
||||
"id": 11,
|
||||
"type": "422723e8-4bf6-438c-823f-881ca81acead",
|
||||
"pos": [791.59912109375, 386.13336181640625],
|
||||
"size": [140, 26],
|
||||
"size": [400, 200],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
|
||||
@@ -25,9 +25,11 @@ import {
|
||||
import { Topbar } from './components/Topbar'
|
||||
import { CanvasHelper } from './helpers/CanvasHelper'
|
||||
import { PerformanceHelper } from './helpers/PerformanceHelper'
|
||||
import { QueueHelper } from './helpers/QueueHelper'
|
||||
import { ClipboardHelper } from './helpers/ClipboardHelper'
|
||||
import { CommandHelper } from './helpers/CommandHelper'
|
||||
import { DragDropHelper } from './helpers/DragDropHelper'
|
||||
import { FeatureFlagHelper } from './helpers/FeatureFlagHelper'
|
||||
import { KeyboardHelper } from './helpers/KeyboardHelper'
|
||||
import { NodeOperationsHelper } from './helpers/NodeOperationsHelper'
|
||||
import { SettingsHelper } from './helpers/SettingsHelper'
|
||||
@@ -184,9 +186,11 @@ export class ComfyPage {
|
||||
public readonly contextMenu: ContextMenu
|
||||
public readonly toast: ToastHelper
|
||||
public readonly dragDrop: DragDropHelper
|
||||
public readonly featureFlags: FeatureFlagHelper
|
||||
public readonly command: CommandHelper
|
||||
public readonly bottomPanel: BottomPanel
|
||||
public readonly perf: PerformanceHelper
|
||||
public readonly queue: QueueHelper
|
||||
|
||||
/** Worker index to test user ID */
|
||||
public readonly userIds: string[] = []
|
||||
@@ -227,9 +231,11 @@ export class ComfyPage {
|
||||
this.contextMenu = new ContextMenu(page)
|
||||
this.toast = new ToastHelper(page)
|
||||
this.dragDrop = new DragDropHelper(page, this.assetPath.bind(this))
|
||||
this.featureFlags = new FeatureFlagHelper(page)
|
||||
this.command = new CommandHelper(page)
|
||||
this.bottomPanel = new BottomPanel(page)
|
||||
this.perf = new PerformanceHelper(page)
|
||||
this.queue = new QueueHelper(page)
|
||||
}
|
||||
|
||||
get visibleToasts() {
|
||||
@@ -431,9 +437,9 @@ export const comfyPageFixture = base.extend<{
|
||||
// Disable toast warning about version compatibility, as they may or
|
||||
// may not appear - depending on upstream ComfyUI dependencies
|
||||
'Comfy.VersionCompatibility.DisableWarnings': true,
|
||||
// Browser tests should opt into missing-model warnings explicitly so
|
||||
// workflows do not render differently based on models present on disk.
|
||||
'Comfy.Workflow.ShowMissingModelsWarning': false
|
||||
// Disable errors tab to prevent missing model detection from
|
||||
// rendering error indicators on nodes during unrelated tests.
|
||||
'Comfy.RightSidePanel.ShowErrorsTab': false
|
||||
})
|
||||
} catch (e) {
|
||||
console.error(e)
|
||||
|
||||
73
browser_tests/fixtures/helpers/FeatureFlagHelper.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
import type { Page, Route } from '@playwright/test'
|
||||
|
||||
export class FeatureFlagHelper {
|
||||
private featuresRouteHandler: ((route: Route) => void) | null = null
|
||||
|
||||
constructor(private readonly page: Page) {}
|
||||
|
||||
/**
|
||||
* Seed feature flags via `addInitScript` so they are available in
|
||||
* localStorage before the app JS executes on first load.
|
||||
* Must be called before `comfyPage.setup()` / `page.goto()`.
|
||||
*
|
||||
* Note: Playwright init scripts persist for the page lifetime and
|
||||
* cannot be removed. Call this once per test, before navigation.
|
||||
*/
|
||||
async seedFlags(flags: Record<string, unknown>): Promise<void> {
|
||||
await this.page.addInitScript((flagMap: Record<string, unknown>) => {
|
||||
for (const [key, value] of Object.entries(flagMap)) {
|
||||
localStorage.setItem(`ff:${key}`, JSON.stringify(value))
|
||||
}
|
||||
}, flags)
|
||||
}
|
||||
|
||||
/**
|
||||
* Set feature flags at runtime via localStorage. Uses the `ff:` prefix
|
||||
* that devFeatureFlagOverride.ts reads in dev mode.
|
||||
* For flags needed before page init, use `seedFlags()` instead.
|
||||
*/
|
||||
async setFlags(flags: Record<string, unknown>): Promise<void> {
|
||||
await this.page.evaluate((flagMap: Record<string, unknown>) => {
|
||||
for (const [key, value] of Object.entries(flagMap)) {
|
||||
localStorage.setItem(`ff:${key}`, JSON.stringify(value))
|
||||
}
|
||||
}, flags)
|
||||
}
|
||||
|
||||
async setFlag(name: string, value: unknown): Promise<void> {
|
||||
await this.setFlags({ [name]: value })
|
||||
}
|
||||
|
||||
async clearFlags(): Promise<void> {
|
||||
await this.page.evaluate(() => {
|
||||
const keysToRemove: string[] = []
|
||||
for (let i = 0; i < localStorage.length; i++) {
|
||||
const key = localStorage.key(i)
|
||||
if (key?.startsWith('ff:')) keysToRemove.push(key)
|
||||
}
|
||||
keysToRemove.forEach((k) => {
|
||||
localStorage.removeItem(k)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Mock server feature flags via route interception on /api/features.
|
||||
*/
|
||||
async mockServerFeatures(features: Record<string, unknown>): Promise<void> {
|
||||
this.featuresRouteHandler = (route: Route) =>
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify(features)
|
||||
})
|
||||
await this.page.route('**/api/features', this.featuresRouteHandler)
|
||||
}
|
||||
|
||||
async clearMocks(): Promise<void> {
|
||||
if (this.featuresRouteHandler) {
|
||||
await this.page.unroute('**/api/features', this.featuresRouteHandler)
|
||||
this.featuresRouteHandler = null
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,10 @@ interface PerfSnapshot {
|
||||
TaskDuration: number
|
||||
JSHeapUsedSize: number
|
||||
Timestamp: number
|
||||
Nodes: number
|
||||
JSHeapTotalSize: number
|
||||
ScriptDuration: number
|
||||
JSEventListeners: number
|
||||
}
|
||||
|
||||
export interface PerfMeasurement {
|
||||
@@ -19,6 +23,12 @@ export interface PerfMeasurement {
|
||||
layoutDurationMs: number
|
||||
taskDurationMs: number
|
||||
heapDeltaBytes: number
|
||||
domNodes: number
|
||||
jsHeapTotalBytes: number
|
||||
scriptDurationMs: number
|
||||
eventListeners: number
|
||||
totalBlockingTimeMs: number
|
||||
frameDurationMs: number
|
||||
}
|
||||
|
||||
export class PerformanceHelper {
|
||||
@@ -59,16 +69,100 @@ export class PerformanceHelper {
|
||||
LayoutDuration: get('LayoutDuration'),
|
||||
TaskDuration: get('TaskDuration'),
|
||||
JSHeapUsedSize: get('JSHeapUsedSize'),
|
||||
Timestamp: get('Timestamp')
|
||||
Timestamp: get('Timestamp'),
|
||||
Nodes: get('Nodes'),
|
||||
JSHeapTotalSize: get('JSHeapTotalSize'),
|
||||
ScriptDuration: get('ScriptDuration'),
|
||||
JSEventListeners: get('JSEventListeners')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Collect longtask entries from PerformanceObserver and compute TBT.
|
||||
* TBT = sum of (duration - 50ms) for every task longer than 50ms.
|
||||
*/
|
||||
private async collectTBT(): Promise<number> {
|
||||
return this.page.evaluate(() => {
|
||||
const state = (window as unknown as Record<string, unknown>)
|
||||
.__perfLongtaskState as
|
||||
| { observer: PerformanceObserver; tbtMs: number }
|
||||
| undefined
|
||||
if (!state) return 0
|
||||
|
||||
// Flush any queued-but-undelivered entries into our accumulator
|
||||
for (const entry of state.observer.takeRecords()) {
|
||||
if (entry.duration > 50) state.tbtMs += entry.duration - 50
|
||||
}
|
||||
const result = state.tbtMs
|
||||
state.tbtMs = 0
|
||||
return result
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Measure average frame duration via rAF timing over a sample window.
|
||||
* Returns average ms per frame (lower = better, 16.67 = 60fps).
|
||||
*/
|
||||
private async measureFrameDuration(sampleFrames = 10): Promise<number> {
|
||||
return this.page.evaluate((frames) => {
|
||||
return new Promise<number>((resolve) => {
|
||||
const timeout = setTimeout(() => resolve(0), 5000)
|
||||
const timestamps: number[] = []
|
||||
let count = 0
|
||||
function tick(ts: number) {
|
||||
timestamps.push(ts)
|
||||
count++
|
||||
if (count <= frames) {
|
||||
requestAnimationFrame(tick)
|
||||
} else {
|
||||
clearTimeout(timeout)
|
||||
if (timestamps.length < 2) {
|
||||
resolve(0)
|
||||
return
|
||||
}
|
||||
const total = timestamps[timestamps.length - 1] - timestamps[0]
|
||||
resolve(total / (timestamps.length - 1))
|
||||
}
|
||||
}
|
||||
requestAnimationFrame(tick)
|
||||
})
|
||||
}, sampleFrames)
|
||||
}
|
||||
|
||||
async startMeasuring(): Promise<void> {
|
||||
if (this.snapshot) {
|
||||
throw new Error(
|
||||
'Measurement already in progress — call stopMeasuring() first'
|
||||
)
|
||||
}
|
||||
// Install longtask observer if not already present, then reset the
|
||||
// accumulator so old longtasks don't bleed into the new measurement window.
|
||||
await this.page.evaluate(() => {
|
||||
const win = window as unknown as Record<string, unknown>
|
||||
if (!win.__perfLongtaskState) {
|
||||
const state: { observer: PerformanceObserver; tbtMs: number } = {
|
||||
observer: new PerformanceObserver((list) => {
|
||||
const self = (window as unknown as Record<string, unknown>)
|
||||
.__perfLongtaskState as {
|
||||
observer: PerformanceObserver
|
||||
tbtMs: number
|
||||
}
|
||||
for (const entry of list.getEntries()) {
|
||||
if (entry.duration > 50) self.tbtMs += entry.duration - 50
|
||||
}
|
||||
}),
|
||||
tbtMs: 0
|
||||
}
|
||||
state.observer.observe({ type: 'longtask', buffered: true })
|
||||
win.__perfLongtaskState = state
|
||||
}
|
||||
const state = win.__perfLongtaskState as {
|
||||
observer: PerformanceObserver
|
||||
tbtMs: number
|
||||
}
|
||||
state.tbtMs = 0
|
||||
state.observer.takeRecords()
|
||||
})
|
||||
this.snapshot = await this.getSnapshot()
|
||||
}
|
||||
|
||||
@@ -82,6 +176,11 @@ export class PerformanceHelper {
|
||||
return after[key] - before[key]
|
||||
}
|
||||
|
||||
const [totalBlockingTimeMs, frameDurationMs] = await Promise.all([
|
||||
this.collectTBT(),
|
||||
this.measureFrameDuration()
|
||||
])
|
||||
|
||||
return {
|
||||
name,
|
||||
durationMs: delta('Timestamp') * 1000,
|
||||
@@ -90,7 +189,13 @@ export class PerformanceHelper {
|
||||
layouts: delta('LayoutCount'),
|
||||
layoutDurationMs: delta('LayoutDuration') * 1000,
|
||||
taskDurationMs: delta('TaskDuration') * 1000,
|
||||
heapDeltaBytes: delta('JSHeapUsedSize')
|
||||
heapDeltaBytes: delta('JSHeapUsedSize'),
|
||||
domNodes: delta('Nodes'),
|
||||
jsHeapTotalBytes: delta('JSHeapTotalSize'),
|
||||
scriptDurationMs: delta('ScriptDuration') * 1000,
|
||||
eventListeners: delta('JSEventListeners'),
|
||||
totalBlockingTimeMs,
|
||||
frameDurationMs
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
79
browser_tests/fixtures/helpers/QueueHelper.ts
Normal file
@@ -0,0 +1,79 @@
|
||||
import type { Page, Route } from '@playwright/test'
|
||||
|
||||
export class QueueHelper {
|
||||
private queueRouteHandler: ((route: Route) => void) | null = null
|
||||
private historyRouteHandler: ((route: Route) => void) | null = null
|
||||
|
||||
constructor(private readonly page: Page) {}
|
||||
|
||||
/**
|
||||
* Mock the /api/queue endpoint to return specific queue state.
|
||||
*/
|
||||
async mockQueueState(
|
||||
running: number = 0,
|
||||
pending: number = 0
|
||||
): Promise<void> {
|
||||
this.queueRouteHandler = (route: Route) =>
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({
|
||||
queue_running: Array.from({ length: running }, (_, i) => [
|
||||
i,
|
||||
`running-${i}`,
|
||||
{},
|
||||
{},
|
||||
[]
|
||||
]),
|
||||
queue_pending: Array.from({ length: pending }, (_, i) => [
|
||||
i,
|
||||
`pending-${i}`,
|
||||
{},
|
||||
{},
|
||||
[]
|
||||
])
|
||||
})
|
||||
})
|
||||
await this.page.route('**/api/queue', this.queueRouteHandler)
|
||||
}
|
||||
|
||||
/**
|
||||
* Mock the /api/history endpoint with completed/failed job entries.
|
||||
*/
|
||||
async mockHistory(
|
||||
jobs: Array<{ promptId: string; status: 'success' | 'error' }>
|
||||
): Promise<void> {
|
||||
const history: Record<string, unknown> = {}
|
||||
for (const job of jobs) {
|
||||
history[job.promptId] = {
|
||||
prompt: [0, job.promptId, {}, {}, []],
|
||||
outputs: {},
|
||||
status: {
|
||||
status_str: job.status === 'success' ? 'success' : 'error',
|
||||
completed: true
|
||||
}
|
||||
}
|
||||
}
|
||||
this.historyRouteHandler = (route: Route) =>
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify(history)
|
||||
})
|
||||
await this.page.route('**/api/history**', this.historyRouteHandler)
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear all route mocks set by this helper.
|
||||
*/
|
||||
async clearMocks(): Promise<void> {
|
||||
if (this.queueRouteHandler) {
|
||||
await this.page.unroute('**/api/queue', this.queueRouteHandler)
|
||||
this.queueRouteHandler = null
|
||||
}
|
||||
if (this.historyRouteHandler) {
|
||||
await this.page.unroute('**/api/history**', this.historyRouteHandler)
|
||||
this.historyRouteHandler = null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
Before Width: | Height: | Size: 138 KiB After Width: | Height: | Size: 130 KiB |
|
Before Width: | Height: | Size: 132 KiB After Width: | Height: | Size: 126 KiB |
|
Before Width: | Height: | Size: 99 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 133 KiB After Width: | Height: | Size: 126 KiB |
|
Before Width: | Height: | Size: 138 KiB After Width: | Height: | Size: 130 KiB |
|
Before Width: | Height: | Size: 158 KiB After Width: | Height: | Size: 154 KiB |
|
Before Width: | Height: | Size: 151 KiB After Width: | Height: | Size: 142 KiB |
|
Before Width: | Height: | Size: 150 KiB After Width: | Height: | Size: 141 KiB |
|
Before Width: | Height: | Size: 147 KiB After Width: | Height: | Size: 140 KiB |
|
Before Width: | Height: | Size: 133 KiB After Width: | Height: | Size: 126 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 115 KiB After Width: | Height: | Size: 107 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 104 KiB After Width: | Height: | Size: 97 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 100 KiB After Width: | Height: | Size: 100 KiB |
@@ -1,4 +1,3 @@
|
||||
import type { Locator } from '@playwright/test'
|
||||
import { expect } from '@playwright/test'
|
||||
|
||||
import type { Keybinding } from '../../src/platform/keybindings/types'
|
||||
@@ -72,6 +71,10 @@ test('Does not report warning on undo/redo', async ({ comfyPage }) => {
|
||||
test.describe('Execution error', () => {
|
||||
test.beforeEach(async ({ comfyPage }) => {
|
||||
await comfyPage.settings.setSetting('Comfy.UseNewMenu', 'Top')
|
||||
await comfyPage.settings.setSetting(
|
||||
'Comfy.RightSidePanel.ShowErrorsTab',
|
||||
true
|
||||
)
|
||||
await comfyPage.setup()
|
||||
})
|
||||
|
||||
@@ -88,117 +91,58 @@ test.describe('Execution error', () => {
|
||||
})
|
||||
})
|
||||
|
||||
test.describe('Missing models warning', () => {
|
||||
test('Should be disabled by default in browser tests', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
await comfyPage.workflow.loadWorkflow('missing/missing_models')
|
||||
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).not.toBeVisible()
|
||||
})
|
||||
|
||||
test.describe('Missing models in Error Tab', () => {
|
||||
test.beforeEach(async ({ comfyPage }) => {
|
||||
await comfyPage.settings.setSetting('Comfy.UseNewMenu', 'Top')
|
||||
await comfyPage.settings.setSetting(
|
||||
'Comfy.Workflow.ShowMissingModelsWarning',
|
||||
'Comfy.RightSidePanel.ShowErrorsTab',
|
||||
true
|
||||
)
|
||||
await comfyPage.page.evaluate((url: string) => {
|
||||
return fetch(`${url}/api/devtools/cleanup_fake_model`)
|
||||
const cleanupOk = await comfyPage.page.evaluate(async (url: string) => {
|
||||
const response = await fetch(`${url}/api/devtools/cleanup_fake_model`)
|
||||
return response.ok
|
||||
}, comfyPage.url)
|
||||
expect(cleanupOk).toBeTruthy()
|
||||
})
|
||||
|
||||
test('Should display a warning when missing models are found', async ({
|
||||
test('Should show error overlay with missing models when workflow has missing models', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
await comfyPage.workflow.loadWorkflow('missing/missing_models')
|
||||
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).toBeVisible()
|
||||
const errorOverlay = comfyPage.page.locator('[data-testid="error-overlay"]')
|
||||
await expect(errorOverlay).toBeVisible()
|
||||
|
||||
const downloadAllButton = comfyPage.page.getByText('Download all')
|
||||
await expect(downloadAllButton).toBeVisible()
|
||||
const missingModelsTitle = comfyPage.page.getByText(/Missing Models/)
|
||||
await expect(missingModelsTitle).toBeVisible()
|
||||
})
|
||||
|
||||
test('Should display a warning when missing models are found in node properties', async ({
|
||||
test('Should show missing models from node properties', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
// Load workflow that has a node with models metadata at the node level
|
||||
await comfyPage.workflow.loadWorkflow(
|
||||
'missing/missing_models_from_node_properties'
|
||||
)
|
||||
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).toBeVisible()
|
||||
const errorOverlay = comfyPage.page.locator('[data-testid="error-overlay"]')
|
||||
await expect(errorOverlay).toBeVisible()
|
||||
|
||||
const downloadAllButton = comfyPage.page.getByText('Download all')
|
||||
await expect(downloadAllButton).toBeVisible()
|
||||
const missingModelsTitle = comfyPage.page.getByText(/Missing Models/)
|
||||
await expect(missingModelsTitle).toBeVisible()
|
||||
})
|
||||
|
||||
test('Should not display a warning when no missing models are found', async ({
|
||||
test('Should not show missing models when widget values have changed', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
const modelFoldersRes = {
|
||||
status: 200,
|
||||
body: JSON.stringify([
|
||||
{
|
||||
name: 'text_encoders',
|
||||
folders: ['ComfyUI/models/text_encoders']
|
||||
}
|
||||
])
|
||||
}
|
||||
await comfyPage.page.route(
|
||||
'**/api/experiment/models',
|
||||
(route) => route.fulfill(modelFoldersRes),
|
||||
{ times: 1 }
|
||||
)
|
||||
|
||||
// Reload page to trigger indexing of model folders
|
||||
await comfyPage.setup()
|
||||
|
||||
const clipModelsRes = {
|
||||
status: 200,
|
||||
body: JSON.stringify([
|
||||
{
|
||||
name: 'fake_model.safetensors',
|
||||
pathIndex: 0
|
||||
}
|
||||
])
|
||||
}
|
||||
await comfyPage.page.route(
|
||||
'**/api/experiment/models/text_encoders',
|
||||
(route) => route.fulfill(clipModelsRes),
|
||||
{ times: 1 }
|
||||
)
|
||||
|
||||
await comfyPage.workflow.loadWorkflow('missing/missing_models')
|
||||
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).not.toBeVisible()
|
||||
})
|
||||
|
||||
test('Should not display warning when model metadata exists but widget values have changed', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
// This tests the scenario where outdated model metadata exists in the workflow
|
||||
// but the actual selected models (widget values) have changed
|
||||
await comfyPage.workflow.loadWorkflow(
|
||||
'missing/model_metadata_widget_mismatch'
|
||||
)
|
||||
|
||||
// The missing models warning should NOT appear
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).not.toBeVisible()
|
||||
const missingModelsTitle = comfyPage.page.getByText(/Missing Models/)
|
||||
await expect(missingModelsTitle).not.toBeVisible()
|
||||
|
||||
const errorOverlay = comfyPage.page.locator('[data-testid="error-overlay"]')
|
||||
await expect(errorOverlay).not.toBeVisible()
|
||||
})
|
||||
|
||||
// Flaky test after parallelization
|
||||
@@ -206,14 +150,10 @@ test.describe('Missing models warning', () => {
|
||||
test.skip('Should download missing model when clicking download button', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
// The fake_model.safetensors is served by
|
||||
// https://github.com/Comfy-Org/ComfyUI_devtools/blob/main/__init__.py
|
||||
await comfyPage.workflow.loadWorkflow('missing/missing_models')
|
||||
|
||||
const dialogTitle = comfyPage.page.getByText(
|
||||
'This workflow is missing models'
|
||||
)
|
||||
await expect(dialogTitle).toBeVisible()
|
||||
const errorOverlay = comfyPage.page.locator('[data-testid="error-overlay"]')
|
||||
await expect(errorOverlay).toBeVisible()
|
||||
|
||||
const downloadAllButton = comfyPage.page.getByText('Download all')
|
||||
await expect(downloadAllButton).toBeVisible()
|
||||
@@ -223,50 +163,6 @@ test.describe('Missing models warning', () => {
|
||||
const download = await downloadPromise
|
||||
expect(download.suggestedFilename()).toBe('fake_model.safetensors')
|
||||
})
|
||||
|
||||
test.describe('Do not show again checkbox', () => {
|
||||
let checkbox: Locator
|
||||
let closeButton: Locator
|
||||
|
||||
test.beforeEach(async ({ comfyPage }) => {
|
||||
await comfyPage.settings.setSetting(
|
||||
'Comfy.Workflow.ShowMissingModelsWarning',
|
||||
true
|
||||
)
|
||||
await comfyPage.workflow.loadWorkflow('missing/missing_models')
|
||||
|
||||
checkbox = comfyPage.page.getByLabel("Don't show this again")
|
||||
closeButton = comfyPage.page.getByLabel('Close')
|
||||
})
|
||||
|
||||
test('Should disable warning dialog when checkbox is checked', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
const changeSettingPromise = comfyPage.page.waitForRequest(
|
||||
'**/api/settings/Comfy.Workflow.ShowMissingModelsWarning'
|
||||
)
|
||||
await checkbox.click()
|
||||
await changeSettingPromise
|
||||
|
||||
await closeButton.click()
|
||||
|
||||
const settingValue = await comfyPage.settings.getSetting(
|
||||
'Comfy.Workflow.ShowMissingModelsWarning'
|
||||
)
|
||||
expect(settingValue).toBe(false)
|
||||
})
|
||||
|
||||
test('Should keep warning dialog enabled when checkbox is unchecked', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
await closeButton.click()
|
||||
|
||||
const settingValue = await comfyPage.settings.getSetting(
|
||||
'Comfy.Workflow.ShowMissingModelsWarning'
|
||||
)
|
||||
expect(settingValue).toBe(true)
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
test.describe('Settings', () => {
|
||||
|
||||
@@ -9,6 +9,10 @@ test.beforeEach(async ({ comfyPage }) => {
|
||||
test.describe('Execution', { tag: ['@smoke', '@workflow'] }, () => {
|
||||
test.beforeEach(async ({ comfyPage }) => {
|
||||
await comfyPage.settings.setSetting('Comfy.UseNewMenu', 'Top')
|
||||
await comfyPage.settings.setSetting(
|
||||
'Comfy.RightSidePanel.ShowErrorsTab',
|
||||
true
|
||||
)
|
||||
await comfyPage.setup()
|
||||
})
|
||||
|
||||
|
||||
@@ -60,6 +60,15 @@ test.describe('Graph Canvas Menu', { tag: ['@screenshot', '@canvas'] }, () => {
|
||||
await comfyPage.nextFrame()
|
||||
})
|
||||
|
||||
test('Fit view button is present and clickable', async ({ comfyPage }) => {
|
||||
const fitViewButton = comfyPage.page
|
||||
.locator('button')
|
||||
.filter({ has: comfyPage.page.locator('.icon-\\[lucide--focus\\]') })
|
||||
await expect(fitViewButton).toBeVisible()
|
||||
await fitViewButton.click()
|
||||
await comfyPage.nextFrame()
|
||||
})
|
||||
|
||||
test('Zoom controls popup opens and closes', async ({ comfyPage }) => {
|
||||
// Find the zoom button by its percentage text content
|
||||
const zoomButton = comfyPage.page.locator('button').filter({
|
||||
|
||||
|
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 80 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 95 KiB |
44
browser_tests/tests/groupCopyPaste.spec.ts
Normal file
@@ -0,0 +1,44 @@
|
||||
import { expect } from '@playwright/test'
|
||||
|
||||
import { comfyPageFixture as test } from '../fixtures/ComfyPage'
|
||||
|
||||
test.describe('Group Copy Paste', { tag: ['@canvas'] }, () => {
|
||||
test.afterEach(async ({ comfyPage }) => {
|
||||
await comfyPage.canvasOps.resetView()
|
||||
})
|
||||
|
||||
test('Pasted group is offset from original position', async ({
|
||||
comfyPage
|
||||
}) => {
|
||||
await comfyPage.workflow.loadWorkflow('groups/single_group_only')
|
||||
|
||||
const titlePos = await comfyPage.page.evaluate(() => {
|
||||
const app = window.app!
|
||||
const group = app.graph.groups[0]
|
||||
const clientPos = app.canvasPosToClientPos([
|
||||
group.pos[0] + 50,
|
||||
group.pos[1] + 15
|
||||
])
|
||||
return { x: clientPos[0], y: clientPos[1] }
|
||||
})
|
||||
await comfyPage.canvas.click({ position: titlePos })
|
||||
await comfyPage.nextFrame()
|
||||
|
||||
await comfyPage.clipboard.copy()
|
||||
await comfyPage.clipboard.paste()
|
||||
await comfyPage.nextFrame()
|
||||
|
||||
const positions = await comfyPage.page.evaluate(() =>
|
||||
window.app!.graph.groups.map((g: { pos: number[] }) => ({
|
||||
x: g.pos[0],
|
||||
y: g.pos[1]
|
||||
}))
|
||||
)
|
||||
|
||||
expect(positions).toHaveLength(2)
|
||||
const dx = Math.abs(positions[0].x - positions[1].x)
|
||||
const dy = Math.abs(positions[0].y - positions[1].y)
|
||||
expect(dx).toBeCloseTo(50, 0)
|
||||
expect(dy).toBeCloseTo(15, 0)
|
||||
})
|
||||
})
|
||||
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 91 KiB After Width: | Height: | Size: 85 KiB |
|
Before Width: | Height: | Size: 75 KiB After Width: | Height: | Size: 70 KiB |
|
Before Width: | Height: | Size: 75 KiB After Width: | Height: | Size: 70 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 93 KiB After Width: | Height: | Size: 87 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 92 KiB After Width: | Height: | Size: 87 KiB |
|
Before Width: | Height: | Size: 107 KiB After Width: | Height: | Size: 101 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 113 KiB After Width: | Height: | Size: 106 KiB |
|
Before Width: | Height: | Size: 103 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 102 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 102 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 72 KiB After Width: | Height: | Size: 69 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 100 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 102 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 98 KiB After Width: | Height: | Size: 92 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 102 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 85 KiB |
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 100 KiB After Width: | Height: | Size: 94 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 66 KiB After Width: | Height: | Size: 62 KiB |
|
Before Width: | Height: | Size: 99 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 68 KiB After Width: | Height: | Size: 65 KiB |
|
Before Width: | Height: | Size: 95 KiB After Width: | Height: | Size: 90 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 34 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 97 KiB After Width: | Height: | Size: 91 KiB |
|
Before Width: | Height: | Size: 35 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 99 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 104 KiB After Width: | Height: | Size: 99 KiB |
|
Before Width: | Height: | Size: 99 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 99 KiB After Width: | Height: | Size: 93 KiB |
|
Before Width: | Height: | Size: 53 KiB After Width: | Height: | Size: 49 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 30 KiB |
|
Before Width: | Height: | Size: 18 KiB After Width: | Height: | Size: 18 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 95 KiB |
|
Before Width: | Height: | Size: 103 KiB After Width: | Height: | Size: 96 KiB |