mirror of
https://github.com/Comfy-Org/ComfyUI_frontend.git
synced 2026-03-29 16:57:30 +00:00
Compare commits
16 Commits
main
...
playwright
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
589eb850e0 | ||
|
|
530efba5bb | ||
|
|
2e96ddc823 | ||
|
|
89fc98fd25 | ||
|
|
fe531c40b5 | ||
|
|
4d84c0a1c1 | ||
|
|
71996019a0 | ||
|
|
cff127db6b | ||
|
|
f80f6b299e | ||
|
|
6153717781 | ||
|
|
dd7bed936c | ||
|
|
ac77f65720 | ||
|
|
49fa1a3bc4 | ||
|
|
5e43d55bc7 | ||
|
|
1244a9c204 | ||
|
|
054e4cff39 |
125
.claude/agents/playwright-test-generator.md
Normal file
125
.claude/agents/playwright-test-generator.md
Normal file
@@ -0,0 +1,125 @@
|
||||
---
|
||||
name: playwright-test-generator
|
||||
description: 'Use this agent when you need to create automated browser tests using Playwright Examples: <example>Context: User wants to generate a test for the test plan item. <test-suite><!-- Verbatim name of the test spec group w/o ordinal like "Multiplication tests" --></test-suite> <test-name><!-- Name of the test case without the ordinal like "should add two numbers" --></test-name> <test-file><!-- Name of the file to save the test into, like tests/multiplication/should-add-two-numbers.spec.ts --></test-file> <seed-file><!-- Seed file path from test plan --></seed-file> <body><!-- Test case content including steps and expectations --></body></example>'
|
||||
tools: Glob, Grep, Read, LS, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__generator_read_log, mcp__playwright-test__generator_setup_page, mcp__playwright-test__generator_write_test
|
||||
model: sonnet
|
||||
color: blue
|
||||
---
|
||||
|
||||
You are a Playwright Test Generator, an expert in browser automation and end-to-end testing.
|
||||
Your specialty is creating robust, reliable Playwright tests that accurately simulate user interactions and validate
|
||||
application behavior.
|
||||
|
||||
# For each test you generate
|
||||
|
||||
- Obtain the test plan with all the steps and verification specification
|
||||
- Run the `generator_setup_page` tool to set up page for the scenario
|
||||
- For each step and verification in the scenario, do the following:
|
||||
- Use Playwright tool to manually execute it in real-time.
|
||||
- Use the step description as the intent for each Playwright tool call.
|
||||
- Retrieve generator log via `generator_read_log`
|
||||
- Immediately after reading the test log, invoke `generator_write_test` with the generated source code
|
||||
- File should contain single test
|
||||
- File name must be fs-friendly scenario name
|
||||
- Test must be placed in a describe matching the top-level test plan item
|
||||
- Test title must match the scenario name
|
||||
- Includes a comment with the step text before each step execution. Do not duplicate comments if step requires
|
||||
multiple actions.
|
||||
- Always use best practices from the log when generating tests.
|
||||
|
||||
<example-generation>
|
||||
For following plan:
|
||||
|
||||
```markdown file=specs/plan.md
|
||||
### 1. Adding New Todos
|
||||
|
||||
**Seed:** `tests/seed.spec.ts`
|
||||
|
||||
#### 1.1 Add Valid Todo
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Click in the "What needs to be done?" input field
|
||||
|
||||
#### 1.2 Add Multiple Todos
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
Following file is generated:
|
||||
|
||||
```ts file=add-valid-todo.spec.ts
|
||||
// spec: specs/plan.md
|
||||
// seed: tests/seed.spec.ts
|
||||
|
||||
test.describe('Adding New Todos', () => {
|
||||
test('Add Valid Todo', async { page } => {
|
||||
// 1. Click in the "What needs to be done?" input field
|
||||
await page.click(...);
|
||||
|
||||
...
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
</example-generation>
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Required Import Pattern
|
||||
|
||||
Generated tests MUST use ComfyUI fixtures, not generic `@playwright/test`:
|
||||
|
||||
```typescript
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
```
|
||||
|
||||
### Fixture Object
|
||||
|
||||
Tests receive `comfyPage` (not `page`) as their fixture:
|
||||
|
||||
```typescript
|
||||
test('my test', async ({ comfyPage }) => {
|
||||
// Access raw page via comfyPage.page if needed
|
||||
})
|
||||
```
|
||||
|
||||
### Key APIs
|
||||
|
||||
| Need | Use | Notes |
|
||||
| ---------------- | ---------------------------------------------------- | --------------------------------- |
|
||||
| Canvas element | `comfyPage.canvas` | Pre-configured Locator |
|
||||
| Wait for render | `comfyPage.nextFrame()` | After canvas mutations |
|
||||
| Load workflow | `comfyPage.workflow.loadWorkflow('name')` | Assets in `browser_tests/assets/` |
|
||||
| Get node by type | `comfyPage.nodeOps.getNodeRefsByType('KSampler')` | Returns NodeReference[] |
|
||||
| Search box | `comfyPage.searchBox.fillAndSelectFirstNode('name')` | Opens on canvas dblclick |
|
||||
| Settings | `comfyPage.settings.setSetting(key, value)` | Clean up in afterEach |
|
||||
| Keyboard | `comfyPage.keyboard.press('Delete')` | Focus canvas first |
|
||||
| Context menu | `comfyPage.contextMenu` | Right-click interactions |
|
||||
|
||||
### Mandatory Test Structure
|
||||
|
||||
Every generated test must:
|
||||
|
||||
1. Be wrapped in `test.describe('Name', { tag: ['@canvas'] }, () => { ... })`
|
||||
2. Include `test.afterEach(async ({ comfyPage }) => { await comfyPage.canvasOps.resetView() })`
|
||||
3. Use descriptive test names (not "test" or "test1")
|
||||
|
||||
### Anti-Patterns — NEVER Use
|
||||
|
||||
- ❌ `page.goto()` — fixture handles navigation
|
||||
- ❌ `page.waitForTimeout()` — use `comfyPage.nextFrame()` or retrying assertions
|
||||
- ❌ `import from '@playwright/test'` — use `from '../fixtures/ComfyPage'`
|
||||
- ❌ Bare `page.` references — use `comfyPage.page.` if you need raw page access
|
||||
|
||||
### Reference
|
||||
|
||||
Read the fixture code for full API surface:
|
||||
|
||||
- `browser_tests/fixtures/ComfyPage.ts` — main fixture
|
||||
- `browser_tests/fixtures/helpers/` — helper classes
|
||||
- `browser_tests/fixtures/components/` — page object components
|
||||
- See also: `.claude/skills/codegen-transform/SKILL.md` for transform rules
|
||||
91
.claude/agents/playwright-test-healer.md
Normal file
91
.claude/agents/playwright-test-healer.md
Normal file
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: playwright-test-healer
|
||||
description: Use this agent when you need to debug and fix failing Playwright tests
|
||||
tools: Glob, Grep, Read, LS, Edit, MultiEdit, Write, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_generate_locator, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_snapshot, mcp__playwright-test__test_debug, mcp__playwright-test__test_list, mcp__playwright-test__test_run
|
||||
model: sonnet
|
||||
color: red
|
||||
---
|
||||
|
||||
You are the Playwright Test Healer, an expert test automation engineer specializing in debugging and
|
||||
resolving Playwright test failures. Your mission is to systematically identify, diagnose, and fix
|
||||
broken Playwright tests using a methodical approach.
|
||||
|
||||
Your workflow:
|
||||
|
||||
1. **Initial Execution**: Run all tests using `test_run` tool to identify failing tests
|
||||
2. **Debug failed tests**: For each failing test run `test_debug`.
|
||||
3. **Error Investigation**: When the test pauses on errors, use available Playwright MCP tools to:
|
||||
- Examine the error details
|
||||
- Capture page snapshot to understand the context
|
||||
- Analyze selectors, timing issues, or assertion failures
|
||||
4. **Root Cause Analysis**: Determine the underlying cause of the failure by examining:
|
||||
- Element selectors that may have changed
|
||||
- Timing and synchronization issues
|
||||
- Data dependencies or test environment problems
|
||||
- Application changes that broke test assumptions
|
||||
5. **Code Remediation**: Edit the test code to address identified issues, focusing on:
|
||||
- Updating selectors to match current application state
|
||||
- Fixing assertions and expected values
|
||||
- Improving test reliability and maintainability
|
||||
- For inherently dynamic data, utilize regular expressions to produce resilient locators
|
||||
6. **Verification**: Restart the test after each fix to validate the changes
|
||||
7. **Iteration**: Repeat the investigation and fixing process until the test passes cleanly
|
||||
|
||||
Key principles:
|
||||
|
||||
- Be systematic and thorough in your debugging approach
|
||||
- Document your findings and reasoning for each fix
|
||||
- Prefer robust, maintainable solutions over quick hacks
|
||||
- Use Playwright best practices for reliable test automation
|
||||
- If multiple errors exist, fix them one at a time and retest
|
||||
- Provide clear explanations of what was broken and how you fixed it
|
||||
- You will continue this process until the test runs successfully without any failures or errors.
|
||||
- If the error persists and you have high confidence the test is correct, do not auto-skip by default.
|
||||
- Summarize root-cause evidence and escalate as a likely app regression.
|
||||
- Use `test.fixme()` only when a known issue is documented and referenced, and include a short rationale comment.
|
||||
Auto-skipping can mask real regressions — require explicit justification.
|
||||
- Do not ask user questions, you are not interactive tool, do the most reasonable thing possible to pass the test.
|
||||
- Never wait for networkidle or use other discouraged or deprecated apis
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Custom Fixtures
|
||||
|
||||
Tests in this project use `comfyPage` fixture, not bare `page`. When healing:
|
||||
|
||||
- Replace any `page.` references with `comfyPage.page.` if adding new code
|
||||
- Use `comfyPage.nextFrame()` instead of adding `waitForTimeout()`
|
||||
- Use fixture helpers (`comfyPage.nodeOps`, `comfyPage.canvas`, etc.) over raw locators
|
||||
|
||||
### Common Failure Causes in ComfyUI Tests
|
||||
|
||||
1. **Missing `nextFrame()`**: Canvas operations need `await comfyPage.nextFrame()` after mutations. This is the #1 cause of "works locally, fails in CI" issues.
|
||||
|
||||
2. **Canvas focus required**: Keyboard shortcuts won't work unless `await comfyPage.canvas.click()` is called first.
|
||||
|
||||
3. **Node position drift**: Pixel coordinates can shift between environments. When possible, replace with node references:
|
||||
|
||||
```typescript
|
||||
// Instead of: canvas.click({ position: { x: 423, y: 267 } })
|
||||
const node = (await comfyPage.nodeOps.getNodeRefsByType('KSampler'))[0]
|
||||
await node.click('title')
|
||||
```
|
||||
|
||||
4. **Settings pollution**: Settings persist across tests on the backend. Always reset changed settings in `afterEach`.
|
||||
|
||||
5. **Drag animation timing**: Use `{ steps: 10 }` option for drag operations, not `{ steps: 1 }`.
|
||||
|
||||
### Healing Safety Rules
|
||||
|
||||
- ❌ NEVER add `waitForTimeout()` — always use retrying assertions or `nextFrame()`
|
||||
- ❌ NEVER "fix" a test by weakening assertions (e.g., removing an assertion that fails)
|
||||
- ❌ NEVER modify the application code — only modify test code
|
||||
- ⚠️ If a test fails because expected UI elements are missing, the app may have a regression — mark as `test.fixme()` with explanation, don't "heal" the assertion away
|
||||
- ⚠️ If a test fails only in CI but passes locally, likely missing `nextFrame()` — don't mask with timeouts
|
||||
|
||||
### Reference
|
||||
|
||||
- `browser_tests/fixtures/ComfyPage.ts` — full fixture API
|
||||
- `browser_tests/fixtures/helpers/` — available helper classes
|
||||
- `.claude/skills/writing-playwright-tests/SKILL.md` — testing conventions
|
||||
- `.claude/skills/codegen-transform/SKILL.md` — transform rules
|
||||
89
.claude/agents/playwright-test-planner.md
Normal file
89
.claude/agents/playwright-test-planner.md
Normal file
@@ -0,0 +1,89 @@
|
||||
---
|
||||
name: playwright-test-planner
|
||||
description: Use this agent when you need to create comprehensive test plan for a web application or website
|
||||
tools: Glob, Grep, Read, LS, mcp__playwright-test__browser_click, mcp__playwright-test__browser_close, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_navigate_back, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_run_code, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_take_screenshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_wait_for, mcp__playwright-test__planner_setup_page, mcp__playwright-test__planner_save_plan
|
||||
model: sonnet
|
||||
color: green
|
||||
---
|
||||
|
||||
You are an expert web test planner with extensive experience in quality assurance, user experience testing, and test
|
||||
scenario design. Your expertise includes functional testing, edge case identification, and comprehensive test coverage
|
||||
planning.
|
||||
|
||||
You will:
|
||||
|
||||
1. **Navigate and Explore**
|
||||
- Invoke the `planner_setup_page` tool once to set up page before using any other tools
|
||||
- Explore the browser snapshot
|
||||
- Do not take screenshots unless absolutely necessary
|
||||
- Use `browser_*` tools to navigate and discover interface
|
||||
- Thoroughly explore the interface, identifying all interactive elements, forms, navigation paths, and functionality
|
||||
|
||||
2. **Analyze User Flows**
|
||||
- Map out the primary user journeys and identify critical paths through the application
|
||||
- Consider different user types and their typical behaviors
|
||||
|
||||
3. **Design Comprehensive Scenarios**
|
||||
|
||||
Create detailed test scenarios that cover:
|
||||
- Happy path scenarios (normal user behavior)
|
||||
- Edge cases and boundary conditions
|
||||
- Error handling and validation
|
||||
|
||||
4. **Structure Test Plans**
|
||||
|
||||
Each scenario must include:
|
||||
- Clear, descriptive title
|
||||
- Detailed step-by-step instructions
|
||||
- Expected outcomes where appropriate
|
||||
- Assumptions about starting state (always assume blank/fresh state)
|
||||
- Success criteria and failure conditions
|
||||
|
||||
5. **Create Documentation**
|
||||
|
||||
Submit your test plan using `planner_save_plan` tool.
|
||||
|
||||
**Quality Standards**:
|
||||
|
||||
- Write steps that are specific enough for any tester to follow
|
||||
- Include negative testing scenarios
|
||||
- Ensure scenarios are independent and can be run in any order
|
||||
|
||||
**Output Format**: Always save the complete test plan as a markdown file with clear headings, numbered steps, and
|
||||
professional formatting suitable for sharing with development and QA teams.
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Application Overview
|
||||
|
||||
ComfyUI is a **canvas-based node graph editor** for AI image generation. It is a complex SPA with:
|
||||
|
||||
- A **LiteGraph canvas** where users create workflows by connecting nodes
|
||||
- A **Vue 3 sidebar** with node library, workflows panel, and settings
|
||||
- A **topbar** with queue/run buttons and workspace controls
|
||||
- A **search box** for finding and adding nodes (opens on double-click)
|
||||
- WebSocket-based real-time communication with a Python backend
|
||||
|
||||
### Exploration Tips
|
||||
|
||||
- Start by loading a workflow: the app is most useful with nodes on the canvas
|
||||
- Key UI areas to explore: canvas interactions, sidebar panels, topbar buttons, search box, context menus, settings dialog
|
||||
- Double-click the canvas to open the node search box
|
||||
- Right-click nodes/canvas for context menus
|
||||
- The bottom panel shows job queue and execution logs
|
||||
|
||||
### Test Environment
|
||||
|
||||
- The seed test uses `comfyPageFixture` which provides a `comfyPage` object with extensive helpers
|
||||
- Workflows (JSON files) are loaded via `comfyPage.workflow.loadWorkflow('name')`
|
||||
- Available workflow assets are in `browser_tests/assets/`
|
||||
- The backend MUST be running with `--multi-user` flag
|
||||
- A Vite dev server runs on `:5173`
|
||||
|
||||
### When Creating Test Plans
|
||||
|
||||
- Reference specific workflow assets when a scenario needs a starting state
|
||||
- Note that canvas interactions use pixel coordinates — these may vary across environments
|
||||
- Distinguish between "canvas tests" (LiteGraph) and "UI tests" (Vue components)
|
||||
- Include tags in your plans: `@canvas`, `@widget`, `@sidebar`, `@smoke`, `@screenshot`
|
||||
- Reference `browser_tests/fixtures/ComfyPage.ts` for available test helpers
|
||||
174
.claude/skills/codegen-transform/SKILL.md
Normal file
174
.claude/skills/codegen-transform/SKILL.md
Normal file
@@ -0,0 +1,174 @@
|
||||
---
|
||||
name: codegen-transform
|
||||
description: 'Transforms raw Playwright codegen output into ComfyUI convention-compliant tests. Use when: user pastes raw codegen, asks to convert raw Playwright code, refactor recorded tests, or rewrite to project conventions. Triggers on: transform codegen, convert raw test, rewrite to conventions, codegen output, raw playwright.'
|
||||
---
|
||||
|
||||
# Codegen → Convention Transform
|
||||
|
||||
Transform raw Playwright codegen output into tests that follow ComfyUI conventions.
|
||||
|
||||
## When to Use
|
||||
|
||||
- QA tester recorded a test with `pnpm comfy-test record` and wants refinement
|
||||
- Developer pasted raw `npx playwright codegen` output
|
||||
- Agent needs to post-process Playwright test agent output
|
||||
- Reviewing a test that uses raw `page.*` calls instead of fixture helpers
|
||||
|
||||
## Transform Rules
|
||||
|
||||
Apply these replacements in order:
|
||||
|
||||
| Raw codegen | Convention replacement | Why |
|
||||
| ------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------- |
|
||||
| `import { test, expect } from '@playwright/test'` | `import { comfyPageFixture as test, comfyExpect as expect } from '../fixtures/ComfyPage'` | Use custom fixtures with ComfyUI helpers |
|
||||
| `test('test', async ({ page }) =>` | `test('descriptive-name', async ({ comfyPage }) =>` | Use comfyPage fixture, descriptive names |
|
||||
| `await page.goto('http://...')` | **Remove entirely** | Fixture handles navigation automatically |
|
||||
| `page.locator('canvas')` | `comfyPage.canvas` | Pre-configured canvas locator |
|
||||
| `page.waitForTimeout(N)` | `comfyPage.nextFrame()` | Never use arbitrary waits |
|
||||
| `page.getByPlaceholder('Search Nodes...')` | `comfyPage.searchBox.input` | Use search box page object |
|
||||
| `page` (bare reference) | `comfyPage.page` | Access raw page through fixture |
|
||||
| Bare `test(...)` | `test.describe('Feature', { tag: ['@canvas'] }, () => { test(...) })` | All tests need describe + tags |
|
||||
| No cleanup | Add `test.afterEach(async ({ comfyPage }) => { await comfyPage.canvasOps.resetView() })` | Canvas tests need cleanup |
|
||||
|
||||
## Fixture API Quick Reference
|
||||
|
||||
| Need | Use | Notes |
|
||||
| ----------------- | ------------------------------------------------- | ------------------------------------------------------------- |
|
||||
| Canvas element | `comfyPage.canvas` | Pre-configured Locator |
|
||||
| Wait for render | `comfyPage.nextFrame()` | After canvas mutations. NOT needed after `loadWorkflow()` |
|
||||
| Load workflow | `comfyPage.workflow.loadWorkflow('name')` | Assets in `browser_tests/assets/` |
|
||||
| Get node by type | `comfyPage.nodeOps.getNodeRefsByType('KSampler')` | Returns array of NodeReference |
|
||||
| Get node by title | `comfyPage.nodeOps.getNodeRefsByTitle('My Node')` | Returns array of NodeReference |
|
||||
| Search box | `comfyPage.searchBox` | Has `.input`, `.fillAndSelectFirstNode()` |
|
||||
| Settings | `comfyPage.settings.setSetting(key, value)` | Persistent — clean up in afterEach |
|
||||
| Keyboard | `comfyPage.keyboard.press('Delete')` | Focus canvas first |
|
||||
| Drag & drop | `comfyPage.dragDrop.*` | Use `{ steps: 10 }` for reliability |
|
||||
| Context menu | `comfyPage.contextMenu.*` | Right-click interactions |
|
||||
| Toast messages | `comfyPage.toast.*` | Notification assertions |
|
||||
| Subgraph | `comfyPage.subgraph.*` | Subgraph/group node operations |
|
||||
| Vue Nodes | `comfyPage.vueNodes.*` | Requires opt-in: `setSetting('Comfy.VueNodes.Enabled', true)` |
|
||||
| Mouse ops | `comfyPage.page` + `ComfyMouse` | For precise canvas mouse interactions |
|
||||
| Bottom panel | `comfyPage.bottomPanel.*` | Job queue, logs panel |
|
||||
| Commands | `comfyPage.command.*` | Command palette interactions |
|
||||
| Clipboard | `comfyPage.clipboard.*` | Copy/paste operations |
|
||||
|
||||
## Canvas Coordinates → Node References
|
||||
|
||||
Raw codegen records fragile pixel coordinates. Replace with node references when possible:
|
||||
|
||||
```typescript
|
||||
// ❌ Raw codegen — fragile pixel coordinates
|
||||
await page.locator('canvas').click({ position: { x: 423, y: 267 } })
|
||||
|
||||
// ✅ If clicking a specific node
|
||||
const node = (await comfyPage.nodeOps.getNodeRefsByType('KSampler'))[0]
|
||||
await node.click('title')
|
||||
|
||||
// ✅ If double-clicking canvas to open search
|
||||
await comfyPage.canvas.dblclick({ position: { x: 500, y: 400 } })
|
||||
await comfyPage.searchBox.fillAndSelectFirstNode('KSampler')
|
||||
```
|
||||
|
||||
**When to keep coordinates**: Canvas background clicks (pan, zoom), empty area clicks to deselect. These are inherently position-based.
|
||||
|
||||
## Complete Before/After Example
|
||||
|
||||
### Raw codegen output
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test'
|
||||
|
||||
test('test', async ({ page }) => {
|
||||
await page.goto('http://localhost:5173/')
|
||||
await page.locator('canvas').dblclick({ position: { x: 500, y: 400 } })
|
||||
await page.getByPlaceholder('Search Nodes...').fill('KSampler')
|
||||
await page.getByPlaceholder('Search Nodes...').press('Enter')
|
||||
await page.locator('canvas').click({ position: { x: 600, y: 300 } })
|
||||
await page.waitForTimeout(1000)
|
||||
await page.getByRole('button', { name: 'Queue' }).click()
|
||||
})
|
||||
```
|
||||
|
||||
### Convention-compliant output
|
||||
|
||||
```typescript
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
|
||||
test.describe('Queue workflow with KSampler', { tag: ['@canvas'] }, () => {
|
||||
test.afterEach(async ({ comfyPage }) => {
|
||||
await comfyPage.canvasOps.resetView()
|
||||
})
|
||||
|
||||
test('should add KSampler node and queue', async ({ comfyPage }) => {
|
||||
// Open search and add KSampler
|
||||
await comfyPage.canvas.dblclick({ position: { x: 500, y: 400 } })
|
||||
await comfyPage.searchBox.fillAndSelectFirstNode('KSampler')
|
||||
await comfyPage.nextFrame()
|
||||
|
||||
// Queue the workflow
|
||||
await comfyPage.menu.topbar.runButton.click()
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
### What changed and why
|
||||
|
||||
1. **Imports**: `@playwright/test` → `../fixtures/ComfyPage` (custom fixtures)
|
||||
2. **Fixture**: `{ page }` → `{ comfyPage }` (access all helpers)
|
||||
3. **goto removed**: Fixture navigates automatically
|
||||
4. **Search box**: Raw locator → `comfyPage.searchBox.fillAndSelectFirstNode()`
|
||||
5. **waitForTimeout**: Replaced with `comfyPage.nextFrame()`
|
||||
6. **Queue button**: Used `comfyPage.menu.topbar.runButton` page object
|
||||
7. **Structure**: Wrapped in `describe` with `@canvas` tag and `afterEach` cleanup
|
||||
8. **Test name**: Generic "test" → descriptive name
|
||||
|
||||
## Decision Guide
|
||||
|
||||
| Question | Answer |
|
||||
| -------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| Canvas or DOM interaction? | Canvas: `comfyPage.nodeOps.*`. DOM: `comfyPage.vueNodes.*` (needs opt-in) |
|
||||
| Need `nextFrame()`? | Yes after canvas mutations. No after `loadWorkflow()`, no after DOM clicks |
|
||||
| Which tag? | `@canvas` for canvas tests, `@widget` for widget tests, `@screenshot` for visual regression |
|
||||
| Need cleanup? | Yes for canvas tests (`resetView`), yes if changing settings (`setSetting` back) |
|
||||
| Keep pixel coords? | Only for empty canvas clicks. Replace with node refs for node interactions |
|
||||
| Use `page` directly? | Only via `comfyPage.page` for Playwright APIs not wrapped by fixtures |
|
||||
|
||||
## Tags Reference
|
||||
|
||||
| Tag | When to use |
|
||||
| ------------- | ------------------------------------ |
|
||||
| `@canvas` | Any test interacting with the canvas |
|
||||
| `@widget` | Testing widget inputs |
|
||||
| `@smoke` | Quick essential tests |
|
||||
| `@screenshot` | Visual regression (Linux CI only) |
|
||||
| `@mobile` | Mobile viewport (runs on Pixel 5) |
|
||||
| `@2x` | HiDPI tests (2x scale) |
|
||||
| `@0.5x` | Low-DPI tests (0.5x scale) |
|
||||
| `@slow` | Tests taking >10 seconds |
|
||||
| `@perf` | Performance measurement tests |
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
1. **Never use `waitForTimeout`** → use `nextFrame()` or retrying assertions
|
||||
2. **Never use `page.goto`** → fixture handles navigation
|
||||
3. **Never import from `@playwright/test`** → use `../fixtures/ComfyPage`
|
||||
4. **Never use bare CSS selectors** → use test IDs or semantic locators
|
||||
5. **Never share state between tests** → each test is independent
|
||||
6. **Never commit local screenshots** → Linux CI generates baselines
|
||||
|
||||
## For Deeper Reference
|
||||
|
||||
Read fixture code directly — it's the source of truth:
|
||||
|
||||
| Purpose | Path |
|
||||
| ----------------- | ------------------------------------------ |
|
||||
| Main fixture | `browser_tests/fixtures/ComfyPage.ts` |
|
||||
| Helper classes | `browser_tests/fixtures/helpers/` |
|
||||
| Component objects | `browser_tests/fixtures/components/` |
|
||||
| Test selectors | `browser_tests/fixtures/selectors.ts` |
|
||||
| Vue Node helpers | `browser_tests/fixtures/VueNodeHelpers.ts` |
|
||||
| Existing tests | `browser_tests/tests/` |
|
||||
| Test assets | `browser_tests/assets/` |
|
||||
8
.mcp.json
Normal file
8
.mcp.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"playwright-test": {
|
||||
"command": "pnpm",
|
||||
"args": ["dlx", "@playwright/mcp@latest"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,8 @@
|
||||
"dist/*",
|
||||
"packages/registry-types/src/comfyRegistryTypes.ts",
|
||||
"playwright-report/*",
|
||||
"scripts/patch-playwright-agents.js",
|
||||
"tools/test-recorder/*",
|
||||
"src/extensions/core/*",
|
||||
"src/scripts/*",
|
||||
"src/types/generatedManagerTypes.ts",
|
||||
|
||||
15
AGENTS.md
15
AGENTS.md
@@ -49,6 +49,21 @@ This project uses **pnpm**. Always prefer scripts defined in `package.json` (e.g
|
||||
- `pnpm preview`: Preview the production build locally
|
||||
- `pnpm test:unit`: Run Vitest unit tests
|
||||
- `pnpm test:browser:local`: Run Playwright E2E tests (`browser_tests/`)
|
||||
- `pnpm comfy-test record`: Interactive test recorder (guided setup for non-devs)
|
||||
- `pnpm comfy-test transform <file>`: Transform raw codegen to conventions
|
||||
- `pnpm comfy-test check`: Check environment prerequisites
|
||||
- `pnpm comfy-test list`: List available test workflows
|
||||
|
||||
### Playwright Test Agents (`.claude/agents/`)
|
||||
|
||||
| Agent | Responsibility |
|
||||
| ------------------------------ | ---------------------------------------------------------------------------------- |
|
||||
| `playwright-test-planner.md` | Explores the app, identifies testable scenarios, creates structured test plans |
|
||||
| `playwright-test-generator.md` | Generates Playwright test code from plans using ComfyUI fixtures and conventions |
|
||||
| `playwright-test-healer.md` | Diagnoses and fixes failing tests; escalates regressions rather than auto-skipping |
|
||||
|
||||
Guardrails: agents must use `comfyPage` fixture (not bare `page`), never add `waitForTimeout()`, never weaken assertions, and reference `.claude/skills/codegen-transform/SKILL.md` for transform rules.
|
||||
|
||||
- `pnpm lint` / `pnpm lint:fix`: Lint (ESLint)
|
||||
- `pnpm format` / `pnpm format:check`: oxfmt
|
||||
- `pnpm typecheck`: Vue TSC type checking
|
||||
|
||||
@@ -67,3 +67,51 @@ Reserve `toPass()` for blocks with multiple assertions or complex async logic th
|
||||
A Playwright test-writing skill exists at `.claude/skills/writing-playwright-tests/SKILL.md`.
|
||||
|
||||
The skill documents **meta-level guidance only** (gotchas, anti-patterns, decision guides). It does **not** duplicate fixture APIs - agents should read the fixture code directly in `browser_tests/fixtures/`.
|
||||
|
||||
## AI-Assisted Test Creation
|
||||
|
||||
Three systems work together for test authoring:
|
||||
|
||||
### 1. Interactive Recorder CLI (`comfy-test`)
|
||||
|
||||
For QA testers and non-developers. Guides through the full flow:
|
||||
|
||||
```bash
|
||||
pnpm comfy-test record # Interactive 7-step recording flow
|
||||
pnpm comfy-test transform # Transform raw codegen to conventions
|
||||
pnpm comfy-test check # Verify environment prerequisites
|
||||
pnpm comfy-test list # List available workflow assets
|
||||
```
|
||||
|
||||
Source: `tools/test-recorder/`
|
||||
|
||||
### 2. Codegen Transform Skill
|
||||
|
||||
For AI agents transforming raw Playwright codegen output. See `.claude/skills/codegen-transform/SKILL.md`.
|
||||
|
||||
Key transforms:
|
||||
|
||||
- `@playwright/test` → `../fixtures/ComfyPage` imports
|
||||
- `page` destructure → `comfyPage` fixture
|
||||
- `page.goto()` → removed (fixture handles navigation)
|
||||
- `page.locator('canvas')` → `comfyPage.canvas`
|
||||
- `waitForTimeout()` → `comfyPage.nextFrame()`
|
||||
- Wraps in `test.describe` with tags and `afterEach` cleanup
|
||||
|
||||
### 3. Playwright AI Agents
|
||||
|
||||
Three agents in `.claude/agents/` are patched with ComfyUI context:
|
||||
|
||||
- **planner** — explores the app and creates test plans in `browser_tests/specs/`
|
||||
- **generator** — converts test plans into executable `.spec.ts` files
|
||||
- **healer** — debugs and fixes failing tests
|
||||
|
||||
To regenerate after Playwright updates: `bash scripts/update-playwright-agents.sh`
|
||||
|
||||
### MCP Server
|
||||
|
||||
The `.mcp.json` configures `playwright-test` MCP server for agent browser interaction:
|
||||
|
||||
```bash
|
||||
pnpm exec playwright run-test-mcp-server
|
||||
```
|
||||
|
||||
@@ -70,6 +70,32 @@ await comfyPage.setup({ mockReleases: false })
|
||||
|
||||
For tests that specifically need to test release functionality, see the example in `tests/releaseNotifications.spec.ts`.
|
||||
|
||||
## Recording Tests (For Non-Developers)
|
||||
|
||||
If you're a QA tester or non-developer, use the interactive recorder:
|
||||
|
||||
```bash
|
||||
pnpm comfy-test record
|
||||
```
|
||||
|
||||
This guides you through a 7-step flow:
|
||||
|
||||
1. **Environment check** — verifies all tools are installed (with install instructions if not)
|
||||
2. **Project setup** — installs dependencies
|
||||
3. **Backend check** — ensures ComfyUI is running
|
||||
4. **Configure** — set test name, tags, and starting workflow
|
||||
5. **Record** — opens browser with Playwright Inspector for recording
|
||||
6. **Transform** — paste recorded code, auto-transforms to project conventions
|
||||
7. **PR creation** — creates a PR via `gh` CLI or gives manual instructions
|
||||
|
||||
Other commands:
|
||||
|
||||
```bash
|
||||
pnpm comfy-test check # Just run environment checks
|
||||
pnpm comfy-test transform <file> # Transform a raw codegen file
|
||||
pnpm comfy-test list # List available workflow assets
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
**Always use UI mode for development:**
|
||||
|
||||
3
browser_tests/specs/README.md
Normal file
3
browser_tests/specs/README.md
Normal file
@@ -0,0 +1,3 @@
|
||||
# Specs
|
||||
|
||||
This is a directory for test plans.
|
||||
26
browser_tests/tests/seed.spec.ts
Normal file
26
browser_tests/tests/seed.spec.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* Seed test for Playwright AI agents.
|
||||
*
|
||||
* This test bootstraps the ComfyUI environment for agent exploration.
|
||||
* When agents (Planner, Generator, Healer) run, they execute this test
|
||||
* first to set up the browser state, then use it as a template for
|
||||
* generated tests.
|
||||
*
|
||||
* Usage:
|
||||
* - Planner: Runs this to explore the app, then generates a test plan
|
||||
* - Generator: Uses this as an import/fixture template
|
||||
* - Healer: Runs this to establish baseline state
|
||||
*/
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
|
||||
test('seed', async ({ comfyPage }) => {
|
||||
// Load the default workflow — gives agents a realistic starting state
|
||||
await comfyPage.workflow.loadWorkflow('default')
|
||||
await comfyPage.nextFrame()
|
||||
|
||||
// Verify the app is ready
|
||||
await expect(comfyPage.canvas).toBeVisible()
|
||||
})
|
||||
@@ -38,6 +38,9 @@ const config: KnipConfig = {
|
||||
],
|
||||
project: ['src/**/*.{astro,vue,ts}', '*.{js,ts,mjs}'],
|
||||
ignoreDependencies: ['@comfyorg/design-system', '@vercel/analytics']
|
||||
},
|
||||
'tools/test-recorder': {
|
||||
project: ['src/**/*.ts']
|
||||
}
|
||||
},
|
||||
ignoreBinaries: ['python3'],
|
||||
|
||||
@@ -42,6 +42,7 @@
|
||||
"storybook:desktop": "nx run @comfyorg/desktop-ui:storybook",
|
||||
"stylelint:fix": "stylelint --cache --fix '{apps,packages,src}/**/*.{css,vue}'",
|
||||
"stylelint": "stylelint --cache '{apps,packages,src}/**/*.{css,vue}'",
|
||||
"comfy-test": "tsx tools/test-recorder/src/index.ts",
|
||||
"test:browser": "pnpm exec nx e2e",
|
||||
"test:browser:local": "cross-env PLAYWRIGHT_LOCAL=1 PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm test:browser",
|
||||
"test:unit": "nx run test",
|
||||
|
||||
33
pnpm-lock.yaml
generated
33
pnpm-lock.yaml
generated
@@ -984,6 +984,22 @@ importers:
|
||||
specifier: 'catalog:'
|
||||
version: 5.9.3
|
||||
|
||||
tools/test-recorder:
|
||||
dependencies:
|
||||
'@clack/prompts':
|
||||
specifier: ^0.9.1
|
||||
version: 0.9.1
|
||||
picocolors:
|
||||
specifier: 'catalog:'
|
||||
version: 1.1.1
|
||||
devDependencies:
|
||||
'@types/node':
|
||||
specifier: 'catalog:'
|
||||
version: 24.10.4
|
||||
typescript:
|
||||
specifier: 'catalog:'
|
||||
version: 5.9.3
|
||||
|
||||
packages:
|
||||
|
||||
'@acemir/cssom@0.9.30':
|
||||
@@ -1654,6 +1670,12 @@ packages:
|
||||
resolution: {integrity: sha512-VERIM64vtTP1C4mxQ5thVT9fK0apjPFobqybMtA1UdUujWka24ERHbRHFGmpbbhp73MhV+KSsHQH9C6uOTdEQA==}
|
||||
engines: {node: '>=18'}
|
||||
|
||||
'@clack/core@0.4.1':
|
||||
resolution: {integrity: sha512-Pxhij4UXg8KSr7rPek6Zowm+5M22rbd2g1nfojHJkxp5YkFqiZ2+YLEM/XGVIzvGOcM0nqjIFxrpDwWRZYWYjA==}
|
||||
|
||||
'@clack/prompts@0.9.1':
|
||||
resolution: {integrity: sha512-JIpyaboYZeWYlyP0H+OoPPxd6nqueG/CmN6ixBiNFsIDHREevjIf0n0Ohh5gr5C8pEDknzgvz+pIJ8dMhzWIeg==}
|
||||
|
||||
'@comfyorg/comfyui-electron-types@0.6.2':
|
||||
resolution: {integrity: sha512-r3By5Wbizq8jagUrhtcym79HYUTinsvoBnYkFFWbUmrURBWIaC0HduFVkRkI1PNdI76piW+JSOJJnw00YCVXeg==}
|
||||
|
||||
@@ -10853,6 +10875,17 @@ snapshots:
|
||||
dependencies:
|
||||
fontkitten: 1.0.3
|
||||
|
||||
'@clack/core@0.4.1':
|
||||
dependencies:
|
||||
picocolors: 1.1.1
|
||||
sisteransi: 1.0.5
|
||||
|
||||
'@clack/prompts@0.9.1':
|
||||
dependencies:
|
||||
'@clack/core': 0.4.1
|
||||
picocolors: 1.1.1
|
||||
sisteransi: 1.0.5
|
||||
|
||||
'@comfyorg/comfyui-electron-types@0.6.2': {}
|
||||
|
||||
'@csstools/color-helpers@5.1.0': {}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
packages:
|
||||
- apps/**
|
||||
- packages/**
|
||||
- tools/**
|
||||
|
||||
catalog:
|
||||
'@alloc/quick-lru': ^5.2.0
|
||||
|
||||
167
scripts/patch-playwright-agents.js
Normal file
167
scripts/patch-playwright-agents.js
Normal file
@@ -0,0 +1,167 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
|
||||
const agentsDir = join(process.cwd(), '.claude', 'agents')
|
||||
|
||||
const patches = {
|
||||
'playwright-test-planner.md': `
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Application Overview
|
||||
ComfyUI is a **canvas-based node graph editor** for AI image generation. It is a complex SPA with:
|
||||
- A **LiteGraph canvas** where users create workflows by connecting nodes
|
||||
- A **Vue 3 sidebar** with node library, workflows panel, and settings
|
||||
- A **topbar** with queue/run buttons and workspace controls
|
||||
- A **search box** for finding and adding nodes (opens on double-click)
|
||||
- WebSocket-based real-time communication with a Python backend
|
||||
|
||||
### Exploration Tips
|
||||
- Start by loading a workflow: the app is most useful with nodes on the canvas
|
||||
- Key UI areas to explore: canvas interactions, sidebar panels, topbar buttons, search box, context menus, settings dialog
|
||||
- Double-click the canvas to open the node search box
|
||||
- Right-click nodes/canvas for context menus
|
||||
- The bottom panel shows job queue and execution logs
|
||||
|
||||
### Test Environment
|
||||
- The seed test uses \`comfyPageFixture\` which provides a \`comfyPage\` object with extensive helpers
|
||||
- Workflows (JSON files) are loaded via \`comfyPage.workflow.loadWorkflow('name')\`
|
||||
- Available workflow assets are in \`browser_tests/assets/\`
|
||||
- The backend MUST be running with \`--multi-user\` flag
|
||||
- A Vite dev server runs on \`:5173\`
|
||||
|
||||
### When Creating Test Plans
|
||||
- Reference specific workflow assets when a scenario needs a starting state
|
||||
- Note that canvas interactions use pixel coordinates — these may vary across environments
|
||||
- Distinguish between "canvas tests" (LiteGraph) and "UI tests" (Vue components)
|
||||
- Include tags in your plans: \`@canvas\`, \`@widget\`, \`@sidebar\`, \`@smoke\`, \`@screenshot\`
|
||||
- Reference \`browser_tests/fixtures/ComfyPage.ts\` for available test helpers`,
|
||||
|
||||
'playwright-test-generator.md': `
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Required Import Pattern
|
||||
Generated tests MUST use ComfyUI fixtures, not generic \`@playwright/test\`:
|
||||
|
||||
\`\`\`typescript
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
\`\`\`
|
||||
|
||||
### Fixture Object
|
||||
Tests receive \`comfyPage\` (not \`page\`) as their fixture:
|
||||
|
||||
\`\`\`typescript
|
||||
test('my test', async ({ comfyPage }) => {
|
||||
// Access raw page via comfyPage.page if needed
|
||||
})
|
||||
\`\`\`
|
||||
|
||||
### Key APIs
|
||||
| Need | Use | Notes |
|
||||
|------|-----|-------|
|
||||
| Canvas element | \`comfyPage.canvas\` | Pre-configured Locator |
|
||||
| Wait for render | \`comfyPage.nextFrame()\` | After canvas mutations |
|
||||
| Load workflow | \`comfyPage.workflow.loadWorkflow('name')\` | Assets in \`browser_tests/assets/\` |
|
||||
| Get node by type | \`comfyPage.nodeOps.getNodeRefsByType('KSampler')\` | Returns NodeReference[] |
|
||||
| Search box | \`comfyPage.searchBox.fillAndSelectFirstNode('name')\` | Opens on canvas dblclick |
|
||||
| Settings | \`comfyPage.settings.setSetting(key, value)\` | Clean up in afterEach |
|
||||
| Keyboard | \`comfyPage.keyboard.press('Delete')\` | Focus canvas first |
|
||||
| Context menu | \`comfyPage.contextMenu\` | Right-click interactions |
|
||||
|
||||
### Mandatory Test Structure
|
||||
Every generated test must:
|
||||
1. Be wrapped in \`test.describe('Name', { tag: ['@canvas'] }, () => { ... })\`
|
||||
2. Include \`test.afterEach(async ({ comfyPage }) => { await comfyPage.canvasOps.resetView() })\`
|
||||
3. Use descriptive test names (not "test" or "test1")
|
||||
|
||||
### Anti-Patterns — NEVER Use
|
||||
- ❌ \`page.goto()\` — fixture handles navigation
|
||||
- ❌ \`page.waitForTimeout()\` — use \`comfyPage.nextFrame()\` or retrying assertions
|
||||
- ❌ \`import from '@playwright/test'\` — use \`from '../fixtures/ComfyPage'\`
|
||||
- ❌ Bare \`page.\` references — use \`comfyPage.page.\` if you need raw page access
|
||||
|
||||
### Reference
|
||||
Read the fixture code for full API surface:
|
||||
- \`browser_tests/fixtures/ComfyPage.ts\` — main fixture
|
||||
- \`browser_tests/fixtures/helpers/\` — helper classes
|
||||
- \`browser_tests/fixtures/components/\` — page object components
|
||||
- See also: \`.claude/skills/codegen-transform/SKILL.md\` for transform rules`,
|
||||
|
||||
'playwright-test-healer.md': `
|
||||
|
||||
## ComfyUI Project Context
|
||||
|
||||
### Custom Fixtures
|
||||
Tests in this project use \`comfyPage\` fixture, not bare \`page\`. When healing:
|
||||
- Replace any \`page.\` references with \`comfyPage.page.\` if adding new code
|
||||
- Use \`comfyPage.nextFrame()\` instead of adding \`waitForTimeout()\`
|
||||
- Use fixture helpers (\`comfyPage.nodeOps\`, \`comfyPage.canvas\`, etc.) over raw locators
|
||||
|
||||
### Common Failure Causes in ComfyUI Tests
|
||||
|
||||
1. **Missing \`nextFrame()\`**: Canvas operations need \`await comfyPage.nextFrame()\` after mutations. This is the #1 cause of "works locally, fails in CI" issues.
|
||||
|
||||
2. **Canvas focus required**: Keyboard shortcuts won't work unless \`await comfyPage.canvas.click()\` is called first.
|
||||
|
||||
3. **Node position drift**: Pixel coordinates can shift between environments. When possible, replace with node references:
|
||||
\`\`\`typescript
|
||||
// Instead of: canvas.click({ position: { x: 423, y: 267 } })
|
||||
const node = (await comfyPage.nodeOps.getNodeRefsByType('KSampler'))[0]
|
||||
await node.click('title')
|
||||
\`\`\`
|
||||
|
||||
4. **Settings pollution**: Settings persist across tests on the backend. Always reset changed settings in \`afterEach\`.
|
||||
|
||||
5. **Drag animation timing**: Use \`{ steps: 10 }\` option for drag operations, not \`{ steps: 1 }\`.
|
||||
|
||||
### Healing Safety Rules
|
||||
- ❌ NEVER add \`waitForTimeout()\` — always use retrying assertions or \`nextFrame()\`
|
||||
- ❌ NEVER "fix" a test by weakening assertions (e.g., removing an assertion that fails)
|
||||
- ❌ NEVER modify the application code — only modify test code
|
||||
- ⚠️ If a test fails because expected UI elements are missing, the app may have a regression — mark as \`test.fixme()\` with explanation, don't "heal" the assertion away
|
||||
- ⚠️ If a test fails only in CI but passes locally, likely missing \`nextFrame()\` — don't mask with timeouts
|
||||
|
||||
### Reference
|
||||
- \`browser_tests/fixtures/ComfyPage.ts\` — full fixture API
|
||||
- \`browser_tests/fixtures/helpers/\` — available helper classes
|
||||
- \`.claude/skills/writing-playwright-tests/SKILL.md\` — testing conventions
|
||||
- \`.claude/skills/codegen-transform/SKILL.md\` — transform rules`
|
||||
}
|
||||
|
||||
const CONTEXT_HEADING = '## ComfyUI Project Context'
|
||||
|
||||
const missingFiles = []
|
||||
|
||||
for (const [filename, patch] of Object.entries(patches)) {
|
||||
const filePath = join(agentsDir, filename)
|
||||
if (!existsSync(filePath)) {
|
||||
missingFiles.push(filename)
|
||||
console.error(` ❌ ${filename}: not found (run init-agents first)`)
|
||||
continue
|
||||
}
|
||||
let content = readFileSync(filePath, 'utf-8')
|
||||
|
||||
// Strip existing ComfyUI context section (heading to EOF)
|
||||
const idx = content.indexOf(CONTEXT_HEADING)
|
||||
if (idx !== -1) {
|
||||
// Trim trailing whitespace before the heading too
|
||||
content = content.substring(0, idx).trimEnd()
|
||||
console.log(` ♻️ ${filename}: stripped existing ComfyUI context`)
|
||||
}
|
||||
|
||||
content += patch
|
||||
writeFileSync(filePath, content, 'utf-8')
|
||||
console.log(` ✅ ${filename}: patched`)
|
||||
}
|
||||
|
||||
if (missingFiles.length > 0) {
|
||||
throw new Error(
|
||||
`Missing expected Playwright agent files: ${missingFiles.join(', ')}`
|
||||
)
|
||||
}
|
||||
28
scripts/update-playwright-agents.sh
Executable file
28
scripts/update-playwright-agents.sh
Executable file
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Navigate to repo root (script location relative)
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
echo "🎭 Updating Playwright agent definitions..."
|
||||
echo ""
|
||||
|
||||
# Step 1: Regenerate base agent files
|
||||
echo "Step 1: Running init-agents..."
|
||||
cd "$REPO_ROOT"
|
||||
pnpm exec playwright init-agents --loop=claude
|
||||
|
||||
echo ""
|
||||
echo "Step 2: Applying ComfyUI patches..."
|
||||
node "$SCRIPT_DIR/patch-playwright-agents.js"
|
||||
|
||||
echo ""
|
||||
echo "✅ Agent files updated and patched."
|
||||
echo ""
|
||||
echo "Files modified:"
|
||||
echo " .claude/agents/playwright-test-planner.md"
|
||||
echo " .claude/agents/playwright-test-generator.md"
|
||||
echo " .claude/agents/playwright-test-healer.md"
|
||||
echo ""
|
||||
echo "Review changes with: git diff .claude/agents/"
|
||||
26
tools/test-recorder/README.md
Normal file
26
tools/test-recorder/README.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# @comfyorg/test-recorder
|
||||
|
||||
Interactive CLI for recording and transforming Playwright browser tests for ComfyUI.
|
||||
|
||||
## Usage
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
pnpm comfy-test record # Record a new test
|
||||
pnpm comfy-test transform <file> # Transform raw codegen to conventions
|
||||
pnpm comfy-test check # Check environment prerequisites
|
||||
pnpm comfy-test list # List available workflows
|
||||
```
|
||||
|
||||
## For QA Testers
|
||||
|
||||
See the [Browser Tests README](../../browser_tests/README.md) for full setup instructions.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
cd tools/test-recorder
|
||||
pnpm build # Compile TypeScript
|
||||
pnpm dev # Watch mode
|
||||
```
|
||||
24
tools/test-recorder/package.json
Normal file
24
tools/test-recorder/package.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"name": "@comfyorg/test-recorder",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Interactive CLI for recording and transforming Playwright browser tests",
|
||||
"bin": {
|
||||
"comfy-test": "./dist/index.js"
|
||||
},
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"dev": "tsc --watch",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@clack/prompts": "^0.9.1",
|
||||
"picocolors": "catalog:"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "catalog:",
|
||||
"typescript": "catalog:"
|
||||
},
|
||||
"packageManager": "pnpm@10.17.1"
|
||||
}
|
||||
33
tools/test-recorder/src/checks/backend.ts
Normal file
33
tools/test-recorder/src/checks/backend.ts
Normal file
@@ -0,0 +1,33 @@
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkBackend(port = 8188): Promise<CheckResult> {
|
||||
const url = `http://localhost:${port}`
|
||||
try {
|
||||
const res = await fetch(`${url}/system_stats`, {
|
||||
signal: AbortSignal.timeout(3000)
|
||||
})
|
||||
if (res.ok) {
|
||||
pass('ComfyUI backend', url)
|
||||
return { name: 'ComfyUI backend', ok: true, version: url }
|
||||
}
|
||||
throw new Error(`Status ${res.status}`)
|
||||
} catch {
|
||||
fail('ComfyUI backend', `not running on :${port}`)
|
||||
const instructions = [
|
||||
'ComfyUI backend must be running for browser tests.',
|
||||
'',
|
||||
'In a separate terminal, navigate to your ComfyUI folder and run:',
|
||||
'',
|
||||
' python main.py --multi-user',
|
||||
'',
|
||||
'⚠️ The --multi-user flag is REQUIRED for parallel test support.'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'ComfyUI backend',
|
||||
ok: false,
|
||||
installInstructions: instructions
|
||||
}
|
||||
}
|
||||
}
|
||||
25
tools/test-recorder/src/checks/devServer.ts
Normal file
25
tools/test-recorder/src/checks/devServer.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkDevServer(port = 5173): Promise<CheckResult> {
|
||||
const url = `http://localhost:${port}`
|
||||
try {
|
||||
const res = await fetch(url, { signal: AbortSignal.timeout(3000) })
|
||||
if (res.ok || res.status === 304) {
|
||||
pass('Dev server', url)
|
||||
return { name: 'Dev server', ok: true, version: url }
|
||||
}
|
||||
throw new Error(`Status ${res.status}`)
|
||||
} catch {
|
||||
fail('Dev server', `not running on :${port}`)
|
||||
const instructions = [
|
||||
'Start the Vite dev server in another terminal:',
|
||||
'',
|
||||
' pnpm dev',
|
||||
'',
|
||||
`Then wait for it to show "Local: http://localhost:${port}"`
|
||||
]
|
||||
info(instructions)
|
||||
return { name: 'Dev server', ok: false, installInstructions: instructions }
|
||||
}
|
||||
}
|
||||
55
tools/test-recorder/src/checks/gh.ts
Normal file
55
tools/test-recorder/src/checks/gh.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { detectPlatform } from './platform'
|
||||
import { pass, warn, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkGh(): Promise<CheckResult> {
|
||||
try {
|
||||
const version = execSync('gh --version', { encoding: 'utf-8' })
|
||||
.split('\n')[0]
|
||||
.trim()
|
||||
// Check if authenticated
|
||||
try {
|
||||
execSync('gh auth status', { encoding: 'utf-8', stdio: 'pipe' })
|
||||
pass('GitHub CLI (gh)', version.replace('gh version ', ''))
|
||||
return { name: 'GitHub CLI', ok: true, optional: true, version }
|
||||
} catch {
|
||||
warn('GitHub CLI (gh)', 'installed but not authenticated')
|
||||
info([
|
||||
'Run `gh auth login` to authenticate.',
|
||||
"Without auth, we'll help you create PRs manually."
|
||||
])
|
||||
return { name: 'GitHub CLI', ok: false, optional: true }
|
||||
}
|
||||
} catch {
|
||||
warn('GitHub CLI (gh)', 'not installed (optional)')
|
||||
const platform = detectPlatform()
|
||||
const instructions =
|
||||
platform === 'macos'
|
||||
? [
|
||||
"gh CLI lets us auto-create PRs. Without it, we'll help",
|
||||
'you create the PR manually on github.com.',
|
||||
'',
|
||||
'To install (optional):',
|
||||
' brew install gh',
|
||||
' gh auth login'
|
||||
]
|
||||
: platform === 'windows'
|
||||
? [
|
||||
'gh CLI is optional. Install from: https://cli.github.com/',
|
||||
"Without it, we'll help you create PRs via github.com."
|
||||
]
|
||||
: [
|
||||
'gh CLI is optional. Install:',
|
||||
' sudo apt install gh # Debian/Ubuntu',
|
||||
' gh auth login'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'GitHub CLI',
|
||||
ok: false,
|
||||
optional: true,
|
||||
installInstructions: instructions
|
||||
}
|
||||
}
|
||||
}
|
||||
37
tools/test-recorder/src/checks/git.ts
Normal file
37
tools/test-recorder/src/checks/git.ts
Normal file
@@ -0,0 +1,37 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { detectPlatform } from './platform'
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkGit(): Promise<CheckResult> {
|
||||
try {
|
||||
const version = execSync('git --version', { encoding: 'utf-8' }).trim()
|
||||
const v = version.replace('git version ', '')
|
||||
pass('Git', v)
|
||||
return { name: 'Git', ok: true, version: v }
|
||||
} catch {
|
||||
fail('Git', 'not installed')
|
||||
const platform = detectPlatform()
|
||||
const instructions =
|
||||
platform === 'macos'
|
||||
? [
|
||||
'Git is included with Xcode CLI Tools. Install them first:',
|
||||
'',
|
||||
' xcode-select --install'
|
||||
]
|
||||
: platform === 'windows'
|
||||
? [
|
||||
'Download Git from: https://git-scm.com/download/win',
|
||||
'Run the installer with default settings.'
|
||||
]
|
||||
: [
|
||||
'Install git using your package manager:',
|
||||
'',
|
||||
' sudo apt install git # Debian/Ubuntu',
|
||||
' sudo dnf install git # Fedora',
|
||||
' sudo pacman -S git # Arch'
|
||||
]
|
||||
info(instructions)
|
||||
return { name: 'Git', ok: false, installInstructions: instructions }
|
||||
}
|
||||
}
|
||||
59
tools/test-recorder/src/checks/node.ts
Normal file
59
tools/test-recorder/src/checks/node.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { detectPlatform } from './platform'
|
||||
import { pass, fail, warn, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkNode(): Promise<CheckResult> {
|
||||
try {
|
||||
const version = execSync('node --version', { encoding: 'utf-8' }).trim()
|
||||
const major = parseInt(version.replace('v', '').split('.')[0])
|
||||
if (major < 20) {
|
||||
warn('Node.js', `${version} (need v20+)`)
|
||||
const instructions = [
|
||||
`Node.js ${version} is too old. You need v20 or later.`,
|
||||
'',
|
||||
'Update via nvm:',
|
||||
' nvm install 20',
|
||||
' nvm use 20'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'Node.js',
|
||||
ok: false,
|
||||
version,
|
||||
installInstructions: instructions
|
||||
}
|
||||
}
|
||||
pass('Node.js', version)
|
||||
return { name: 'Node.js', ok: true, version }
|
||||
} catch {
|
||||
fail('Node.js', 'not installed')
|
||||
const platform = detectPlatform()
|
||||
const instructions =
|
||||
platform === 'macos'
|
||||
? [
|
||||
'Install Node.js via nvm (recommended):',
|
||||
'',
|
||||
' curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash',
|
||||
'',
|
||||
'Close and reopen Terminal, then run:',
|
||||
'',
|
||||
' nvm install 20'
|
||||
]
|
||||
: platform === 'windows'
|
||||
? [
|
||||
'Download Node.js from: https://nodejs.org/',
|
||||
'Choose the LTS version (v20+).',
|
||||
'Run the installer with default settings.'
|
||||
]
|
||||
: [
|
||||
'Install Node.js via nvm:',
|
||||
'',
|
||||
' curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash',
|
||||
' source ~/.bashrc',
|
||||
' nvm install 20'
|
||||
]
|
||||
info(instructions)
|
||||
return { name: 'Node.js', ok: false, installInstructions: instructions }
|
||||
}
|
||||
}
|
||||
24
tools/test-recorder/src/checks/platform.ts
Normal file
24
tools/test-recorder/src/checks/platform.ts
Normal file
@@ -0,0 +1,24 @@
|
||||
import os from 'node:os'
|
||||
import { pass } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
type Platform = 'macos' | 'windows' | 'linux'
|
||||
|
||||
export function detectPlatform(): Platform {
|
||||
switch (os.platform()) {
|
||||
case 'darwin':
|
||||
return 'macos'
|
||||
case 'win32':
|
||||
return 'windows'
|
||||
default:
|
||||
return 'linux'
|
||||
}
|
||||
}
|
||||
|
||||
export function checkPlatform(): CheckResult {
|
||||
const p = os.platform()
|
||||
const arch = os.arch()
|
||||
const release = os.release()
|
||||
pass('Operating System', `${p} ${release} (${arch})`)
|
||||
return { name: 'Operating System', ok: true, version: `${p} ${arch}` }
|
||||
}
|
||||
51
tools/test-recorder/src/checks/playwright.ts
Normal file
51
tools/test-recorder/src/checks/playwright.ts
Normal file
@@ -0,0 +1,51 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { pass, fail, warn, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkPlaywright(): Promise<CheckResult> {
|
||||
try {
|
||||
// Check if chromium browser is installed
|
||||
execSync('pnpm exec playwright install --dry-run chromium', {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
})
|
||||
pass('Playwright browsers', 'chromium installed')
|
||||
return { name: 'Playwright browsers', ok: true, version: 'chromium' }
|
||||
} catch {
|
||||
// Browser might not be installed, try to check another way
|
||||
try {
|
||||
const result = execSync('pnpm exec playwright --version', {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
}).trim()
|
||||
warn('Playwright', `${result} (browsers may need installing)`)
|
||||
const instructions = [
|
||||
'Playwright browsers need to be installed:',
|
||||
'',
|
||||
' pnpm exec playwright install chromium --with-deps',
|
||||
'',
|
||||
'This downloads ~200MB. Please wait...'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'Playwright browsers',
|
||||
ok: false,
|
||||
installInstructions: instructions
|
||||
}
|
||||
} catch {
|
||||
fail('Playwright', 'not installed')
|
||||
const instructions = [
|
||||
'Playwright is a project dependency. Run:',
|
||||
'',
|
||||
' pnpm install',
|
||||
' pnpm exec playwright install chromium --with-deps'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'Playwright',
|
||||
ok: false,
|
||||
installInstructions: instructions
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
25
tools/test-recorder/src/checks/pnpm.ts
Normal file
25
tools/test-recorder/src/checks/pnpm.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkPnpm(): Promise<CheckResult> {
|
||||
try {
|
||||
const version = execSync('pnpm --version', { encoding: 'utf-8' }).trim()
|
||||
pass('pnpm', version)
|
||||
return { name: 'pnpm', ok: true, version }
|
||||
} catch {
|
||||
fail('pnpm', 'not installed')
|
||||
const instructions = [
|
||||
'Install pnpm via corepack (comes with Node.js):',
|
||||
'',
|
||||
' corepack enable',
|
||||
' corepack prepare pnpm@10.17.1 --activate',
|
||||
'',
|
||||
'Or install directly:',
|
||||
'',
|
||||
' npm install -g pnpm'
|
||||
]
|
||||
info(instructions)
|
||||
return { name: 'pnpm', ok: false, installInstructions: instructions }
|
||||
}
|
||||
}
|
||||
39
tools/test-recorder/src/checks/python.ts
Normal file
39
tools/test-recorder/src/checks/python.ts
Normal file
@@ -0,0 +1,39 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { detectPlatform } from './platform'
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkPython(): Promise<CheckResult> {
|
||||
for (const cmd of ['python3', 'python']) {
|
||||
try {
|
||||
const version = execSync(`${cmd} --version`, { encoding: 'utf-8' }).trim()
|
||||
pass('Python', version)
|
||||
return { name: 'Python', ok: true, version }
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
fail('Python 3', 'not installed')
|
||||
const platform = detectPlatform()
|
||||
const instructions =
|
||||
platform === 'macos'
|
||||
? [
|
||||
'Python 3 is needed for the ComfyUI backend.',
|
||||
'',
|
||||
' brew install python3',
|
||||
'',
|
||||
'Or download from: https://www.python.org/downloads/'
|
||||
]
|
||||
: platform === 'windows'
|
||||
? [
|
||||
'Download Python from: https://www.python.org/downloads/',
|
||||
'Check "Add Python to PATH" during install.'
|
||||
]
|
||||
: [
|
||||
' sudo apt install python3 # Debian/Ubuntu',
|
||||
' sudo dnf install python3 # Fedora'
|
||||
]
|
||||
info(instructions)
|
||||
return { name: 'Python 3', ok: false, installInstructions: instructions }
|
||||
}
|
||||
7
tools/test-recorder/src/checks/types.ts
Normal file
7
tools/test-recorder/src/checks/types.ts
Normal file
@@ -0,0 +1,7 @@
|
||||
export interface CheckResult {
|
||||
name: string
|
||||
ok: boolean
|
||||
version?: string
|
||||
optional?: boolean
|
||||
installInstructions?: string[]
|
||||
}
|
||||
34
tools/test-recorder/src/checks/xcode.ts
Normal file
34
tools/test-recorder/src/checks/xcode.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
import { execSync } from 'node:child_process'
|
||||
import { detectPlatform } from './platform'
|
||||
import { pass, fail, info } from '../ui/logger'
|
||||
import type { CheckResult } from './types'
|
||||
|
||||
export async function checkXcode(): Promise<CheckResult> {
|
||||
if (detectPlatform() !== 'macos') {
|
||||
return { name: 'Xcode CLI Tools', ok: true, version: 'n/a (not macOS)' }
|
||||
}
|
||||
|
||||
try {
|
||||
const path = execSync('xcode-select -p', { encoding: 'utf-8' }).trim()
|
||||
pass('Xcode CLI Tools', path)
|
||||
return { name: 'Xcode CLI Tools', ok: true, version: path }
|
||||
} catch {
|
||||
fail('Xcode CLI Tools', 'not installed')
|
||||
const instructions = [
|
||||
'Xcode Command Line Tools are required for git and build tools.',
|
||||
'',
|
||||
'To install, run this in Terminal:',
|
||||
'',
|
||||
' xcode-select --install',
|
||||
'',
|
||||
'A popup will appear — click "Install" and wait (~5 min).',
|
||||
'When done, come back here and press Enter.'
|
||||
]
|
||||
info(instructions)
|
||||
return {
|
||||
name: 'Xcode CLI Tools',
|
||||
ok: false,
|
||||
installInstructions: instructions
|
||||
}
|
||||
}
|
||||
}
|
||||
39
tools/test-recorder/src/commands/check.ts
Normal file
39
tools/test-recorder/src/commands/check.ts
Normal file
@@ -0,0 +1,39 @@
|
||||
import { checkPlatform } from '../checks/platform'
|
||||
import { checkXcode } from '../checks/xcode'
|
||||
import { checkGit } from '../checks/git'
|
||||
import { checkNode } from '../checks/node'
|
||||
import { checkPnpm } from '../checks/pnpm'
|
||||
import { checkPython } from '../checks/python'
|
||||
import { checkPlaywright } from '../checks/playwright'
|
||||
import { checkGh } from '../checks/gh'
|
||||
import { checkDevServer } from '../checks/devServer'
|
||||
import { checkBackend } from '../checks/backend'
|
||||
import { header } from '../ui/logger'
|
||||
import type { CheckResult } from '../checks/types'
|
||||
|
||||
export async function runChecks(): Promise<{
|
||||
results: CheckResult[]
|
||||
allPassed: boolean
|
||||
}> {
|
||||
header('Environment Check')
|
||||
|
||||
const results: CheckResult[] = []
|
||||
|
||||
// System checks (sequential — each depends on prior)
|
||||
results.push(checkPlatform())
|
||||
results.push(await checkXcode())
|
||||
results.push(await checkGit())
|
||||
results.push(await checkNode())
|
||||
results.push(await checkPnpm())
|
||||
results.push(await checkPython())
|
||||
results.push(await checkPlaywright())
|
||||
results.push(await checkGh())
|
||||
|
||||
header('Services Check')
|
||||
|
||||
results.push(await checkBackend())
|
||||
results.push(await checkDevServer())
|
||||
|
||||
const requiredFailed = results.filter((r) => !r.ok && !r.optional)
|
||||
return { results, allPassed: requiredFailed.length === 0 }
|
||||
}
|
||||
38
tools/test-recorder/src/commands/list.ts
Normal file
38
tools/test-recorder/src/commands/list.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
import pc from 'picocolors'
|
||||
import { listWorkflows, findProjectRoot } from '../recorder/runner'
|
||||
import { header } from '../ui/logger'
|
||||
|
||||
export async function runList(): Promise<void> {
|
||||
header('Available Workflows')
|
||||
|
||||
const projectRoot = findProjectRoot()
|
||||
const workflows = listWorkflows(projectRoot)
|
||||
|
||||
if (workflows.length === 0) {
|
||||
console.log(pc.dim(' No workflow assets found in browser_tests/assets/'))
|
||||
return
|
||||
}
|
||||
|
||||
console.log(
|
||||
pc.dim(` Found ${workflows.length} workflows in browser_tests/assets/:\n`)
|
||||
)
|
||||
|
||||
let currentDir = ''
|
||||
for (const wf of workflows) {
|
||||
const parts = wf.split('/')
|
||||
if (parts.length > 1) {
|
||||
const dir = parts.slice(0, -1).join('/')
|
||||
if (dir !== currentDir) {
|
||||
currentDir = dir
|
||||
console.log(pc.bold(` ${dir}/`))
|
||||
}
|
||||
console.log(` ${pc.cyan(parts[parts.length - 1])}`)
|
||||
} else {
|
||||
if (currentDir !== '') {
|
||||
currentDir = ''
|
||||
console.log()
|
||||
}
|
||||
console.log(` ${pc.cyan(wf)}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
257
tools/test-recorder/src/commands/record.ts
Normal file
257
tools/test-recorder/src/commands/record.ts
Normal file
@@ -0,0 +1,257 @@
|
||||
import { createInterface } from 'node:readline'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { writeFileSync, mkdirSync, readFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import {
|
||||
text,
|
||||
select,
|
||||
multiselect,
|
||||
confirm,
|
||||
isCancel,
|
||||
cancel,
|
||||
spinner
|
||||
} from '@clack/prompts'
|
||||
import pc from 'picocolors'
|
||||
import { runChecks } from './check'
|
||||
import {
|
||||
runRecording,
|
||||
findProjectRoot,
|
||||
listWorkflows
|
||||
} from '../recorder/runner'
|
||||
import { transform, formatTransformSummary } from '../transform/engine'
|
||||
import { stepHeader } from '../ui/steps'
|
||||
import { pass, fail, info, blank, box } from '../ui/logger'
|
||||
import { checkGhAvailable, createPr } from '../pr/gh'
|
||||
import { printManualInstructions } from '../pr/manual'
|
||||
import { copyToClipboard } from '../pr/clipboard'
|
||||
|
||||
function toSlug(description: string): string {
|
||||
return description
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
}
|
||||
|
||||
function readMultiline(): Promise<string> {
|
||||
return new Promise((resolve) => {
|
||||
const lines: string[] = []
|
||||
const rl = createInterface({ input: process.stdin })
|
||||
rl.on('line', (line) => lines.push(line))
|
||||
rl.on('close', () => resolve(lines.join('\n')))
|
||||
})
|
||||
}
|
||||
|
||||
export async function runRecord(): Promise<void> {
|
||||
// ── Step 1: Environment Check ──────────────────────────────
|
||||
stepHeader(1, 7, 'Environment Check')
|
||||
const { allPassed } = await runChecks()
|
||||
if (!allPassed) {
|
||||
blank()
|
||||
fail('Some required checks failed. Fix the issues above.')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// ── Step 2: Project Setup ──────────────────────────────────
|
||||
stepHeader(2, 7, 'Project Setup')
|
||||
|
||||
let projectRoot: string
|
||||
try {
|
||||
projectRoot = findProjectRoot()
|
||||
} catch (err) {
|
||||
fail(err instanceof Error ? err.message : 'Could not find project root')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
const s = spinner()
|
||||
s.start('Installing dependencies...')
|
||||
const install = spawnSync('pnpm', ['install'], {
|
||||
cwd: projectRoot,
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (install.status !== 0) {
|
||||
s.stop('Dependency installation failed')
|
||||
fail('pnpm install failed', install.stderr?.toString() ?? '')
|
||||
process.exit(1)
|
||||
}
|
||||
s.stop('Dependencies installed')
|
||||
pass('Project ready', projectRoot)
|
||||
|
||||
// ── Step 3: Backend & Dev Server ───────────────────────────
|
||||
stepHeader(3, 7, 'Backend & Dev Server')
|
||||
pass('Services checked in Step 1')
|
||||
|
||||
// ── Step 4: Configure Your Test ────────────────────────────
|
||||
stepHeader(4, 7, 'Configure Your Test')
|
||||
|
||||
const description = await text({
|
||||
message: 'What are you testing?',
|
||||
placeholder: 'e.g., adding a KSampler node and queuing'
|
||||
})
|
||||
if (isCancel(description)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
let slug = toSlug(description)
|
||||
|
||||
const filenameOk = await confirm({
|
||||
message: `Generated filename: ${slug}.spec.ts — looks good?`
|
||||
})
|
||||
if (isCancel(filenameOk)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
if (!filenameOk) {
|
||||
const customName = await text({
|
||||
message: 'Enter a custom filename (without .spec.ts):',
|
||||
placeholder: slug
|
||||
})
|
||||
if (isCancel(customName)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
slug = toSlug(customName)
|
||||
}
|
||||
|
||||
const selectedTags = await multiselect({
|
||||
message: 'Select tags for this test:',
|
||||
options: [
|
||||
{ value: '@canvas', label: '@canvas' },
|
||||
{ value: '@widget', label: '@widget' },
|
||||
{ value: '@sidebar', label: '@sidebar' },
|
||||
{ value: '@smoke', label: '@smoke' },
|
||||
{ value: '@mobile', label: '@mobile' },
|
||||
{ value: '@screenshot', label: '@screenshot' }
|
||||
],
|
||||
initialValues: ['@canvas']
|
||||
})
|
||||
if (isCancel(selectedTags)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
const workflows = listWorkflows(projectRoot)
|
||||
const workflowOptions: {
|
||||
value: string
|
||||
label: string
|
||||
hint?: string
|
||||
}[] = [
|
||||
{ value: '', label: '(empty canvas)', hint: 'start fresh' },
|
||||
...workflows.map((wf) => ({ value: wf, label: wf }))
|
||||
]
|
||||
|
||||
const selectedWorkflow = await select({
|
||||
message: 'Start with a pre-loaded workflow?',
|
||||
options: workflowOptions
|
||||
})
|
||||
if (isCancel(selectedWorkflow)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
// ── Step 5: Record ─────────────────────────────────────────
|
||||
stepHeader(5, 7, 'Record')
|
||||
|
||||
const result = await runRecording({
|
||||
testName: slug,
|
||||
workflow: selectedWorkflow || undefined,
|
||||
projectRoot
|
||||
})
|
||||
if (!result.success) {
|
||||
fail('Recording failed', result.error)
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
// ── Step 6: Paste & Transform ──────────────────────────────
|
||||
stepHeader(6, 7, 'Paste & Transform')
|
||||
|
||||
info([
|
||||
'Copy the generated code from the Playwright Inspector.',
|
||||
'',
|
||||
'Paste your code below, then press ' +
|
||||
pc.bold('Ctrl+D') +
|
||||
' (Mac/Linux) or ' +
|
||||
pc.bold('Ctrl+Z') +
|
||||
' (Windows) when done:'
|
||||
])
|
||||
blank()
|
||||
|
||||
const pastedCode = await readMultiline()
|
||||
|
||||
if (!pastedCode.trim()) {
|
||||
blank()
|
||||
info([
|
||||
'No code pasted. You can transform later with:',
|
||||
pc.cyan(` comfy-test transform <file>`)
|
||||
])
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
const transformResult = transform(pastedCode, {
|
||||
testName: slug,
|
||||
tags: selectedTags as string[]
|
||||
})
|
||||
|
||||
blank()
|
||||
const summary = formatTransformSummary(transformResult)
|
||||
for (const line of summary) {
|
||||
console.log(` ${line}`)
|
||||
}
|
||||
|
||||
const testsDir = join(projectRoot, 'browser_tests', 'tests')
|
||||
mkdirSync(testsDir, { recursive: true })
|
||||
const outputPath = join(testsDir, `${slug}.spec.ts`)
|
||||
writeFileSync(outputPath, transformResult.code)
|
||||
|
||||
blank()
|
||||
pass('Test saved', outputPath)
|
||||
|
||||
// ── Step 7: Finalize ───────────────────────────────────────
|
||||
stepHeader(7, 7, 'Finalize')
|
||||
|
||||
box([
|
||||
'Run your test:',
|
||||
pc.cyan(` pnpm exec playwright test ${slug} --headed`),
|
||||
'',
|
||||
'Review in UI mode:',
|
||||
pc.cyan(' pnpm exec playwright test --ui')
|
||||
])
|
||||
blank()
|
||||
|
||||
const wantPr = await confirm({
|
||||
message: 'Create a Pull Request now?'
|
||||
})
|
||||
if (isCancel(wantPr)) {
|
||||
cancel('Operation cancelled')
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
if (wantPr) {
|
||||
const gh = await checkGhAvailable()
|
||||
if (gh.available && gh.authenticated) {
|
||||
await createPr({
|
||||
testFilePath: outputPath,
|
||||
testName: slug,
|
||||
description: description as string
|
||||
})
|
||||
} else {
|
||||
const fileContents = readFileSync(outputPath, 'utf-8')
|
||||
const copied = await copyToClipboard(fileContents)
|
||||
if (!copied.ok) {
|
||||
info([
|
||||
'Could not copy to clipboard. File contents are at:',
|
||||
pc.cyan(outputPath)
|
||||
])
|
||||
}
|
||||
const relativePath = `browser_tests/tests/${slug}.spec.ts`
|
||||
printManualInstructions({
|
||||
testFilePath: outputPath,
|
||||
testName: slug,
|
||||
relativePath
|
||||
})
|
||||
}
|
||||
} else {
|
||||
blank()
|
||||
info(['You can create a PR later.', pc.dim(`Test file: ${outputPath}`)])
|
||||
}
|
||||
}
|
||||
55
tools/test-recorder/src/commands/transform.ts
Normal file
55
tools/test-recorder/src/commands/transform.ts
Normal file
@@ -0,0 +1,55 @@
|
||||
import { readFileSync, writeFileSync } from 'node:fs'
|
||||
import { basename } from 'node:path'
|
||||
import pc from 'picocolors'
|
||||
import { transform, formatTransformSummary } from '../transform/engine'
|
||||
import { header } from '../ui/logger'
|
||||
|
||||
export async function runTransform(
|
||||
filePath: string,
|
||||
options: {
|
||||
testName?: string
|
||||
tags?: string[]
|
||||
output?: string
|
||||
} = {}
|
||||
): Promise<void> {
|
||||
header('Transform Codegen → Conventions')
|
||||
|
||||
// Read input
|
||||
let rawCode: string
|
||||
try {
|
||||
rawCode = readFileSync(filePath, 'utf-8')
|
||||
} catch {
|
||||
console.log(pc.red(` Could not read file: ${filePath}`))
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
console.log(pc.dim(` Input: ${filePath}`))
|
||||
console.log()
|
||||
|
||||
// Transform
|
||||
const inferredName = basename(filePath).replace(/\.raw\.spec\.ts$/, '')
|
||||
const testName = options.testName ?? (inferredName || 'test')
|
||||
const tags = options.tags ?? ['@canvas']
|
||||
const result = transform(rawCode, { testName, tags })
|
||||
|
||||
// Print summary
|
||||
console.log(' Transforms applied:')
|
||||
for (const line of formatTransformSummary(result)) {
|
||||
console.log(` ${line}`)
|
||||
}
|
||||
console.log()
|
||||
|
||||
// Write output
|
||||
const outputPath =
|
||||
options.output ?? filePath.replace(/\.raw\.spec\.ts$/, '.spec.ts')
|
||||
if (!options.output && outputPath === filePath) {
|
||||
console.log(
|
||||
pc.red(
|
||||
' Refusing to overwrite input file. Pass --output or use a *.raw.spec.ts input.'
|
||||
)
|
||||
)
|
||||
process.exit(1)
|
||||
}
|
||||
writeFileSync(outputPath, result.code)
|
||||
console.log(pc.green(` ✅ Saved: ${outputPath}`))
|
||||
}
|
||||
65
tools/test-recorder/src/index.ts
Normal file
65
tools/test-recorder/src/index.ts
Normal file
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { intro, outro } from '@clack/prompts'
|
||||
import pc from 'picocolors'
|
||||
|
||||
const args = process.argv.slice(2)
|
||||
const command = args[0]
|
||||
|
||||
intro(pc.bgCyan(pc.black(' 🎭 ComfyUI Test Recorder ')))
|
||||
|
||||
switch (command) {
|
||||
case 'record': {
|
||||
const { runRecord } = await import('./commands/record')
|
||||
await runRecord()
|
||||
break
|
||||
}
|
||||
case 'transform': {
|
||||
const filePath = args[1]
|
||||
if (!filePath) {
|
||||
console.log(pc.red(' Usage: comfy-test transform <file>'))
|
||||
process.exit(1)
|
||||
}
|
||||
const { runTransform } = await import('./commands/transform')
|
||||
await runTransform(filePath, {
|
||||
testName: args[2],
|
||||
tags: args.slice(3)
|
||||
})
|
||||
break
|
||||
}
|
||||
case 'check': {
|
||||
const { runChecks } = await import('./commands/check')
|
||||
const { allPassed } = await runChecks()
|
||||
if (!allPassed) {
|
||||
console.log()
|
||||
console.log(
|
||||
pc.red(
|
||||
' Some required checks failed. Fix the issues above and try again.'
|
||||
)
|
||||
)
|
||||
process.exitCode = 1
|
||||
}
|
||||
break
|
||||
}
|
||||
case 'list': {
|
||||
const { runList } = await import('./commands/list')
|
||||
await runList()
|
||||
break
|
||||
}
|
||||
default:
|
||||
console.log(`
|
||||
Usage: comfy-test <command>
|
||||
|
||||
Commands:
|
||||
record Record a new browser test interactively
|
||||
transform Transform raw codegen output to conventions
|
||||
check Check environment prerequisites
|
||||
list List available test workflows
|
||||
|
||||
Options:
|
||||
--help Show help
|
||||
`)
|
||||
break
|
||||
}
|
||||
|
||||
outro(pc.dim('https://github.com/Comfy-Org/ComfyUI_frontend'))
|
||||
44
tools/test-recorder/src/pr/clipboard.ts
Normal file
44
tools/test-recorder/src/pr/clipboard.ts
Normal file
@@ -0,0 +1,44 @@
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { detectPlatform } from '../checks/platform'
|
||||
|
||||
type ClipboardResult = { ok: true } | { ok: false; reason: string }
|
||||
|
||||
export async function copyToClipboard(
|
||||
content: string
|
||||
): Promise<ClipboardResult> {
|
||||
const platform = detectPlatform()
|
||||
|
||||
let cmd: string
|
||||
let args: string[]
|
||||
|
||||
if (platform === 'macos') {
|
||||
cmd = 'pbcopy'
|
||||
args = []
|
||||
} else if (platform === 'windows') {
|
||||
cmd = 'clip.exe'
|
||||
args = []
|
||||
} else {
|
||||
// Linux: try xclip first, fall back to xsel
|
||||
const xclip = spawnSync('xclip', ['-selection', 'clipboard'], {
|
||||
input: content,
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (xclip.status === 0) return { ok: true }
|
||||
|
||||
const xsel = spawnSync('xsel', ['--clipboard', '--input'], {
|
||||
input: content,
|
||||
stdio: 'pipe'
|
||||
})
|
||||
return xsel.status === 0
|
||||
? { ok: true }
|
||||
: { ok: false, reason: 'xclip/xsel unavailable or failed' }
|
||||
}
|
||||
|
||||
const result = spawnSync(cmd, args, {
|
||||
input: content,
|
||||
stdio: 'pipe'
|
||||
})
|
||||
return result.status === 0
|
||||
? { ok: true }
|
||||
: { ok: false, reason: `${cmd} failed` }
|
||||
}
|
||||
100
tools/test-recorder/src/pr/gh.ts
Normal file
100
tools/test-recorder/src/pr/gh.ts
Normal file
@@ -0,0 +1,100 @@
|
||||
import { execSync, spawnSync } from 'node:child_process'
|
||||
import { pass, fail } from '../ui/logger'
|
||||
|
||||
interface PrOptions {
|
||||
testFilePath: string
|
||||
testName: string
|
||||
description: string
|
||||
branchName?: string
|
||||
}
|
||||
|
||||
interface PrResult {
|
||||
success: boolean
|
||||
url?: string
|
||||
error?: string
|
||||
}
|
||||
|
||||
export async function checkGhAvailable(): Promise<{
|
||||
available: boolean
|
||||
authenticated: boolean
|
||||
}> {
|
||||
try {
|
||||
execSync('gh --version', { stdio: 'pipe' })
|
||||
} catch {
|
||||
return { available: false, authenticated: false }
|
||||
}
|
||||
|
||||
try {
|
||||
execSync('gh auth status', {
|
||||
stdio: 'pipe',
|
||||
encoding: 'utf-8'
|
||||
})
|
||||
return { available: true, authenticated: true }
|
||||
} catch {
|
||||
return { available: true, authenticated: false }
|
||||
}
|
||||
}
|
||||
|
||||
export async function createPr(options: PrOptions): Promise<PrResult> {
|
||||
const branchName = options.branchName ?? `test/${options.testName}`
|
||||
const commitMsg = `test: add ${options.testName} e2e test\n\n${options.description}`
|
||||
const prTitle = `test: add ${options.testName} e2e test`
|
||||
const prBody =
|
||||
`${options.description}\n\n---\n\n` + 'Recorded with `comfy-test record`'
|
||||
|
||||
const checkout = spawnSync('git', ['checkout', '-b', branchName], {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (checkout.status !== 0) {
|
||||
fail('Branch creation failed', checkout.stderr.trim())
|
||||
return {
|
||||
success: false,
|
||||
error: checkout.stderr.trim()
|
||||
}
|
||||
}
|
||||
pass('Created branch', branchName)
|
||||
|
||||
const add = spawnSync('git', ['add', options.testFilePath], {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (add.status !== 0) {
|
||||
fail('Git add failed', add.stderr.trim())
|
||||
return { success: false, error: add.stderr.trim() }
|
||||
}
|
||||
|
||||
const commit = spawnSync('git', ['commit', '-m', commitMsg], {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (commit.status !== 0) {
|
||||
fail('Git commit failed', commit.stderr.trim())
|
||||
return { success: false, error: commit.stderr.trim() }
|
||||
}
|
||||
pass('Committed test file')
|
||||
|
||||
const push = spawnSync('git', ['push', '-u', 'origin', branchName], {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe'
|
||||
})
|
||||
if (push.status !== 0) {
|
||||
fail('Git push failed', push.stderr.trim())
|
||||
return { success: false, error: push.stderr.trim() }
|
||||
}
|
||||
pass('Pushed branch', branchName)
|
||||
|
||||
const pr = spawnSync(
|
||||
'gh',
|
||||
['pr', 'create', '--title', prTitle, '--body', prBody, '--fill'],
|
||||
{ encoding: 'utf-8', stdio: 'pipe' }
|
||||
)
|
||||
if (pr.status !== 0) {
|
||||
fail('PR creation failed', pr.stderr.trim())
|
||||
return { success: false, error: pr.stderr.trim() }
|
||||
}
|
||||
|
||||
const url = pr.stdout.trim()
|
||||
pass('Pull request created', url)
|
||||
return { success: true, url }
|
||||
}
|
||||
29
tools/test-recorder/src/pr/manual.ts
Normal file
29
tools/test-recorder/src/pr/manual.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
import { info } from '../ui/logger'
|
||||
|
||||
interface ManualPrOptions {
|
||||
testFilePath: string
|
||||
testName: string
|
||||
relativePath: string
|
||||
repoUrl?: string
|
||||
}
|
||||
|
||||
const DEFAULT_REPO = 'https://github.com/Comfy-Org/ComfyUI_frontend'
|
||||
|
||||
export function printManualInstructions(options: ManualPrOptions): void {
|
||||
const repo = options.repoUrl ?? DEFAULT_REPO
|
||||
|
||||
info([
|
||||
'Create a PR manually:',
|
||||
'',
|
||||
` 1. Go to: ${repo}`,
|
||||
' 2. Click "Add file" → "Create new file"',
|
||||
` 3. Set path to: ${options.relativePath}`,
|
||||
' 4. Paste the contents (copied to your clipboard ✅)',
|
||||
` 5. Write commit message: "test: add ${options.testName} e2e test"`,
|
||||
' 6. Select "Create a new branch" → click "Propose new file"',
|
||||
' 7. Click "Create pull request"',
|
||||
'',
|
||||
'Or, ask an AI agent: "Create a PR with this test file"',
|
||||
`(The file is at ${options.testFilePath})`
|
||||
])
|
||||
}
|
||||
160
tools/test-recorder/src/recorder/runner.ts
Normal file
160
tools/test-recorder/src/recorder/runner.ts
Normal file
@@ -0,0 +1,160 @@
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { existsSync, readdirSync, statSync } from 'node:fs'
|
||||
import { dirname, join, parse } from 'node:path'
|
||||
import pc from 'picocolors'
|
||||
import { generateRecordingTemplate, cleanupRecordingTemplate } from './template'
|
||||
import { box } from '../ui/logger'
|
||||
|
||||
interface RunnerOptions {
|
||||
testName: string
|
||||
workflow?: string
|
||||
projectRoot: string
|
||||
}
|
||||
|
||||
interface RecordingResult {
|
||||
success: boolean
|
||||
rawOutputPath?: string
|
||||
error?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the project root by looking for playwright.config.ts
|
||||
*/
|
||||
export function findProjectRoot(): string {
|
||||
let dir = process.cwd()
|
||||
const { root } = parse(dir)
|
||||
while (true) {
|
||||
if (existsSync(join(dir, 'playwright.config.ts'))) {
|
||||
return dir
|
||||
}
|
||||
if (dir === root) break
|
||||
dir = dirname(dir)
|
||||
}
|
||||
throw new Error(
|
||||
'Could not find project root (no playwright.config.ts found). ' +
|
||||
'Run this command from the ComfyUI_frontend directory.'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* List available workflow assets.
|
||||
*/
|
||||
export function listWorkflows(projectRoot: string): string[] {
|
||||
const assetsDir = join(projectRoot, 'browser_tests', 'assets')
|
||||
const results: string[] = []
|
||||
|
||||
function walk(dir: string, prefix: string) {
|
||||
try {
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const fullPath = join(dir, entry)
|
||||
const relPath = prefix ? `${prefix}/${entry}` : entry
|
||||
const stat = statSync(fullPath)
|
||||
if (stat.isDirectory()) {
|
||||
walk(fullPath, relPath)
|
||||
} else if (entry.endsWith('.json')) {
|
||||
results.push(relPath.replace(/\.json$/, ''))
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Directory might not exist
|
||||
}
|
||||
}
|
||||
|
||||
walk(assetsDir, '')
|
||||
return results.sort()
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the recording session.
|
||||
*
|
||||
* 1. Generate the temporary test file with page.pause()
|
||||
* 2. Run it in headed mode — this opens the Playwright Inspector
|
||||
* 3. User records their actions
|
||||
* 4. User closes the browser
|
||||
* 5. We save the output as *.raw.spec.ts
|
||||
*/
|
||||
export async function runRecording(
|
||||
options: RunnerOptions
|
||||
): Promise<RecordingResult> {
|
||||
const browserTestsDir = join(options.projectRoot, 'browser_tests')
|
||||
|
||||
// Generate the recording template
|
||||
const tempFile = generateRecordingTemplate(
|
||||
{ testName: options.testName, workflow: options.workflow },
|
||||
browserTestsDir
|
||||
)
|
||||
|
||||
console.log()
|
||||
box([
|
||||
'A browser window will open.',
|
||||
'',
|
||||
'👉 Perform your test actions:',
|
||||
' • Click, type, drag — everything is recorded',
|
||||
' • Use toolbar buttons to add assertions',
|
||||
' • When done, close the browser window',
|
||||
'',
|
||||
'The Playwright Inspector shows generated code.'
|
||||
])
|
||||
console.log()
|
||||
|
||||
try {
|
||||
// Run the test in headed mode with PWDEBUG to force inspector
|
||||
const result = spawnSync(
|
||||
'pnpm',
|
||||
[
|
||||
'exec',
|
||||
'playwright',
|
||||
'test',
|
||||
'_recording-session',
|
||||
'--headed',
|
||||
'--project=chromium',
|
||||
'--timeout=0'
|
||||
],
|
||||
{
|
||||
cwd: options.projectRoot,
|
||||
stdio: 'inherit',
|
||||
env: {
|
||||
...process.env,
|
||||
PWDEBUG: '1',
|
||||
PLAYWRIGHT_LOCAL: '1'
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
if (result.error) {
|
||||
return {
|
||||
success: false,
|
||||
error: `Failed to spawn pnpm: ${result.error.message}`
|
||||
}
|
||||
}
|
||||
|
||||
if (result.status !== 0) {
|
||||
return {
|
||||
success: false,
|
||||
error: `Playwright exited with status ${result.status}`
|
||||
}
|
||||
}
|
||||
|
||||
console.log()
|
||||
console.log(pc.green(' ✅ Recording session complete.'))
|
||||
console.log()
|
||||
|
||||
// Save a placeholder raw file — the user pastes codegen output here
|
||||
// or the record command handles prompting for it
|
||||
const rawOutputPath = join(
|
||||
browserTestsDir,
|
||||
'tests',
|
||||
`${options.testName}.raw.spec.ts`
|
||||
)
|
||||
|
||||
return { success: true, rawOutputPath }
|
||||
} catch (err) {
|
||||
return {
|
||||
success: false,
|
||||
error: `Recording failed: ${err instanceof Error ? err.message : String(err)}`
|
||||
}
|
||||
} finally {
|
||||
// Always clean up the temp recording file
|
||||
cleanupRecordingTemplate(browserTestsDir)
|
||||
}
|
||||
}
|
||||
70
tools/test-recorder/src/recorder/template.ts
Normal file
70
tools/test-recorder/src/recorder/template.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
import { writeFileSync, mkdirSync, unlinkSync } from 'node:fs'
|
||||
import { join, dirname } from 'node:path'
|
||||
|
||||
interface TemplateOptions {
|
||||
workflow?: string
|
||||
testName: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a temporary test file that uses page.pause() to open
|
||||
* the Playwright Inspector with codegen controls.
|
||||
*
|
||||
* The test file:
|
||||
* 1. Uses comfyPageFixture to get full fixture context
|
||||
* 2. Optionally loads a workflow
|
||||
* 3. Calls page.pause() to open the Inspector
|
||||
*/
|
||||
export function generateRecordingTemplate(
|
||||
options: TemplateOptions,
|
||||
browserTestsDir: string
|
||||
): string {
|
||||
const filePath = join(browserTestsDir, 'tests', `_recording-session.spec.ts`)
|
||||
|
||||
const workflowLine = options.workflow
|
||||
? ` // Load the selected workflow\n await comfyPage.workflow.loadWorkflow('${options.workflow.replace(/'/g, "\\'")}')\n await comfyPage.nextFrame()\n`
|
||||
: ''
|
||||
|
||||
const code = `/**
|
||||
* Auto-generated recording session.
|
||||
* This file is temporary — it will be deleted after recording.
|
||||
*
|
||||
* DO NOT COMMIT THIS FILE.
|
||||
*/
|
||||
import {
|
||||
comfyPageFixture as test,
|
||||
comfyExpect as expect
|
||||
} from '../fixtures/ComfyPage'
|
||||
|
||||
test('recording: ${options.testName}', async ({ comfyPage }) => {
|
||||
${workflowLine}
|
||||
// ┌────────────────────────────────────────────────────────┐
|
||||
// │ The Playwright Inspector will open. │
|
||||
// │ │
|
||||
// │ 1. Click the Record button (red circle) to start │
|
||||
// │ 2. Perform your test actions in the browser │
|
||||
// │ 3. Use toolbar buttons to add assertions │
|
||||
// │ 4. Click Stop when done │
|
||||
// │ 5. Copy the generated code from the Inspector │
|
||||
// │ 6. Close the browser window │
|
||||
// └────────────────────────────────────────────────────────┘
|
||||
await comfyPage.page.pause()
|
||||
})
|
||||
`
|
||||
|
||||
mkdirSync(dirname(filePath), { recursive: true })
|
||||
writeFileSync(filePath, code)
|
||||
return filePath
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up the temporary recording file.
|
||||
*/
|
||||
export function cleanupRecordingTemplate(browserTestsDir: string): void {
|
||||
const filePath = join(browserTestsDir, 'tests', '_recording-session.spec.ts')
|
||||
try {
|
||||
unlinkSync(filePath)
|
||||
} catch {
|
||||
// File might already be deleted
|
||||
}
|
||||
}
|
||||
89
tools/test-recorder/src/transform/engine.ts
Normal file
89
tools/test-recorder/src/transform/engine.ts
Normal file
@@ -0,0 +1,89 @@
|
||||
import { transformRules, structuralTransforms } from './rules'
|
||||
|
||||
interface TransformResult {
|
||||
code: string
|
||||
appliedRules: { name: string; description: string }[]
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Transform raw Playwright codegen output into ComfyUI conventions.
|
||||
*/
|
||||
export function transform(
|
||||
rawCode: string,
|
||||
options: {
|
||||
testName?: string
|
||||
tags?: string[]
|
||||
} = {}
|
||||
): TransformResult {
|
||||
const testName = options.testName ?? 'unnamed-test'
|
||||
const tags = options.tags ?? ['@canvas']
|
||||
|
||||
let code = rawCode
|
||||
const appliedRules: { name: string; description: string }[] = []
|
||||
const warnings: string[] = []
|
||||
|
||||
// Phase 1: Apply regex-based rules
|
||||
for (const rule of transformRules) {
|
||||
const before = code
|
||||
if (typeof rule.replacement === 'string') {
|
||||
code = code.replace(rule.pattern, rule.replacement)
|
||||
} else {
|
||||
code = code.replace(
|
||||
rule.pattern,
|
||||
rule.replacement as (...args: string[]) => string
|
||||
)
|
||||
}
|
||||
if (code !== before) {
|
||||
appliedRules.push({ name: rule.name, description: rule.description })
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 2: Clean up empty lines from removed statements
|
||||
code = code.replace(/\n{3,}/g, '\n\n')
|
||||
|
||||
// Phase 3: Apply structural transforms
|
||||
for (const transform of structuralTransforms) {
|
||||
const before = code
|
||||
code = transform.apply(code, testName, tags)
|
||||
if (code !== before) {
|
||||
appliedRules.push({
|
||||
name: transform.name,
|
||||
description: transform.description
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 4: Check for remaining issues and warn
|
||||
if (code.includes('waitForTimeout')) {
|
||||
warnings.push(
|
||||
'Still contains waitForTimeout — replace with comfyPage.nextFrame() or retrying assertions'
|
||||
)
|
||||
}
|
||||
if (code.includes("from '@playwright/test'")) {
|
||||
warnings.push(
|
||||
'Still imports from @playwright/test — should use ../fixtures/ComfyPage'
|
||||
)
|
||||
}
|
||||
if (/position:\s*\{\s*x:\s*\d+,\s*y:\s*\d+/.test(code)) {
|
||||
warnings.push(
|
||||
'Contains pixel coordinates — consider replacing with node references (comfyPage.nodeOps.*) where possible'
|
||||
)
|
||||
}
|
||||
|
||||
return { code: code.trim() + '\n', appliedRules, warnings }
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a human-readable summary of what was transformed.
|
||||
*/
|
||||
export function formatTransformSummary(result: TransformResult): string[] {
|
||||
const lines: string[] = []
|
||||
for (const rule of result.appliedRules) {
|
||||
lines.push(`✅ ${rule.description}`)
|
||||
}
|
||||
for (const warning of result.warnings) {
|
||||
lines.push(`⚠️ ${warning}`)
|
||||
}
|
||||
return lines
|
||||
}
|
||||
125
tools/test-recorder/src/transform/rules.ts
Normal file
125
tools/test-recorder/src/transform/rules.ts
Normal file
@@ -0,0 +1,125 @@
|
||||
interface TransformRule {
|
||||
name: string
|
||||
description: string
|
||||
pattern: RegExp
|
||||
replacement: string | ((match: string, ...groups: string[]) => string)
|
||||
category: 'import' | 'fixture' | 'locator' | 'wait' | 'structure' | 'cleanup'
|
||||
}
|
||||
|
||||
export const transformRules: TransformRule[] = [
|
||||
// === Import transforms ===
|
||||
{
|
||||
name: 'replace-test-import',
|
||||
description: 'Use comfyPageFixture instead of @playwright/test',
|
||||
pattern:
|
||||
/import\s*\{\s*test\s*,?\s*expect\s*\}\s*from\s*['"]@playwright\/test['"]/,
|
||||
replacement: `import {\n comfyPageFixture as test,\n comfyExpect as expect\n} from '../fixtures/ComfyPage'`,
|
||||
category: 'import'
|
||||
},
|
||||
{
|
||||
name: 'replace-test-only-import',
|
||||
description: 'Use comfyPageFixture when only test is imported',
|
||||
pattern: /import\s*\{\s*test\s*\}\s*from\s*['"]@playwright\/test['"]/,
|
||||
replacement: `import { comfyPageFixture as test } from '../fixtures/ComfyPage'`,
|
||||
category: 'import'
|
||||
},
|
||||
{
|
||||
name: 'replace-expect-only-import',
|
||||
description: 'Use comfyExpect when only expect is imported',
|
||||
pattern: /import\s*\{\s*expect\s*\}\s*from\s*['"]@playwright\/test['"]/,
|
||||
replacement: `import { comfyExpect as expect } from '../fixtures/ComfyPage'`,
|
||||
category: 'import'
|
||||
},
|
||||
|
||||
// === Fixture transforms ===
|
||||
{
|
||||
name: 'replace-page-destructure',
|
||||
description: 'Use comfyPage fixture instead of page',
|
||||
pattern: /async\s*\(\s*\{\s*page\s*\}\s*\)/g,
|
||||
replacement: 'async ({ comfyPage })',
|
||||
category: 'fixture'
|
||||
},
|
||||
|
||||
// === Remove page.goto ===
|
||||
{
|
||||
name: 'remove-goto',
|
||||
description: 'Remove page.goto — fixture handles navigation',
|
||||
pattern: /^\s*await\s+page\.goto\s*\([^)]*\)\s*;?\s*$/gm,
|
||||
replacement: '',
|
||||
category: 'locator'
|
||||
},
|
||||
|
||||
// === Locator transforms ===
|
||||
{
|
||||
name: 'replace-canvas-locator',
|
||||
description: 'Use comfyPage.canvas instead of page.locator("canvas")',
|
||||
pattern: /page\.locator\(\s*['"]canvas['"]\s*\)/g,
|
||||
replacement: 'comfyPage.canvas',
|
||||
category: 'locator'
|
||||
},
|
||||
{
|
||||
name: 'replace-search-placeholder',
|
||||
description: 'Use comfyPage.searchBox for search input',
|
||||
pattern: /page\.getByPlaceholder\(\s*['"]Search\s+Nodes\.{0,3}['"]\s*\)/g,
|
||||
replacement: 'comfyPage.searchBox.input',
|
||||
category: 'locator'
|
||||
},
|
||||
{
|
||||
name: 'replace-bare-page',
|
||||
description: 'Replace bare page references with comfyPage.page',
|
||||
pattern: /(?<!\w)page\./g,
|
||||
replacement: 'comfyPage.page.',
|
||||
category: 'locator'
|
||||
},
|
||||
|
||||
// === Wait transforms ===
|
||||
{
|
||||
name: 'replace-waitForTimeout',
|
||||
description: 'Use comfyPage.nextFrame() instead of arbitrary waits',
|
||||
pattern:
|
||||
/await\s+(?:comfyPage\.)?page\.waitForTimeout\s*\(\s*\d+\s*\)\s*;?/g,
|
||||
replacement: 'await comfyPage.nextFrame()',
|
||||
category: 'wait'
|
||||
}
|
||||
]
|
||||
|
||||
/**
|
||||
* Rules that need structural changes (not just regex replacement).
|
||||
* These are applied by the engine after regex rules.
|
||||
*/
|
||||
interface StructuralTransform {
|
||||
name: string
|
||||
description: string
|
||||
apply: (code: string, testName: string, tags: string[]) => string
|
||||
}
|
||||
|
||||
export const structuralTransforms: StructuralTransform[] = [
|
||||
{
|
||||
name: 'wrap-in-describe',
|
||||
description: 'Wrap test in test.describe with tags and afterEach',
|
||||
apply: (code: string, testName: string, tags: string[]) => {
|
||||
// If already has test.describe, skip
|
||||
if (code.includes('test.describe')) return code
|
||||
|
||||
const tagStr = tags.map((t) => JSON.stringify(t)).join(', ')
|
||||
const descName = JSON.stringify(
|
||||
testName.replace(/[-_]/g, ' ').replace(/\.spec\.ts$/, '')
|
||||
)
|
||||
|
||||
// Find the test() call and wrap it
|
||||
const testMatch = code.match(/^(import[\s\S]*?\n\n?)(test\s*\([\s\S]*)$/m)
|
||||
if (!testMatch) return code
|
||||
|
||||
const imports = testMatch[1]
|
||||
const testBody = testMatch[2]
|
||||
|
||||
return `${imports}test.describe(${descName}, { tag: [${tagStr}] }, () => {
|
||||
test.afterEach(async ({ comfyPage }) => {
|
||||
await comfyPage.canvasOps.resetView()
|
||||
})
|
||||
|
||||
${testBody.replace(/^/gm, ' ').trimStart()}
|
||||
})\n`
|
||||
}
|
||||
}
|
||||
]
|
||||
46
tools/test-recorder/src/ui/logger.ts
Normal file
46
tools/test-recorder/src/ui/logger.ts
Normal file
@@ -0,0 +1,46 @@
|
||||
import pc from 'picocolors'
|
||||
|
||||
export function pass(label: string, detail?: string) {
|
||||
const d = detail ? pc.dim(` ${detail}`) : ''
|
||||
console.log(` ${pc.green('✅')} ${label}${d}`)
|
||||
}
|
||||
|
||||
export function fail(label: string, detail?: string) {
|
||||
const d = detail ? pc.dim(` ${detail}`) : ''
|
||||
console.log(` ${pc.red('❌')} ${label}${d}`)
|
||||
}
|
||||
|
||||
export function warn(label: string, detail?: string) {
|
||||
const d = detail ? pc.dim(` ${detail}`) : ''
|
||||
console.log(` ${pc.yellow('⚠️')} ${label}${d}`)
|
||||
}
|
||||
|
||||
function wait(label: string) {
|
||||
console.log(` ${pc.blue('⏳')} ${label}`)
|
||||
}
|
||||
|
||||
export function info(lines: string[]) {
|
||||
for (const line of lines) {
|
||||
console.log(` ${pc.dim('┃')} ${line}`)
|
||||
}
|
||||
}
|
||||
|
||||
export function blank() {
|
||||
console.log()
|
||||
}
|
||||
|
||||
export function header(text: string) {
|
||||
console.log()
|
||||
console.log(pc.bold(` ━━━ ${text} ━━━`))
|
||||
console.log()
|
||||
}
|
||||
|
||||
export function box(lines: string[]) {
|
||||
const maxLen = Math.max(...lines.map((l) => l.length))
|
||||
const border = '─'.repeat(maxLen + 4)
|
||||
console.log(` ┌${border}┐`)
|
||||
for (const line of lines) {
|
||||
console.log(` │ ${line.padEnd(maxLen + 2)}│`)
|
||||
}
|
||||
console.log(` └${border}┘`)
|
||||
}
|
||||
7
tools/test-recorder/src/ui/steps.ts
Normal file
7
tools/test-recorder/src/ui/steps.ts
Normal file
@@ -0,0 +1,7 @@
|
||||
import pc from 'picocolors'
|
||||
|
||||
export function stepHeader(current: number, total: number, label: string) {
|
||||
console.log()
|
||||
console.log(pc.bold(` ━━━ Step ${current} of ${total}: ${label} ━━━`))
|
||||
console.log()
|
||||
}
|
||||
15
tools/test-recorder/tsconfig.json
Normal file
15
tools/test-recorder/tsconfig.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"declaration": true,
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
Reference in New Issue
Block a user