mirror of
https://github.com/Comfy-Org/ComfyUI_frontend.git
synced 2026-04-19 22:09:37 +00:00
feat: expand QA action set and improve issue reproduction depth
- Add new canvas actions: rightClick, doubleClick, clickCanvas, rightClickCanvas, dragCanvas, scrollCanvas for node graph interactions - Increase reproduce mode step limit from 3-6 to 8-15 steps - Add ComfyUI UI context to prompts (canvas layout, node interactions) - Add anti-hallucination instructions to video review for issue mode - Improve issue analysis prompt with detailed action descriptions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
snomiao
@@ -416,23 +416,55 @@ function buildIssueAnalysisPrompt(issue: IssueThread): string {
|
||||
.filter(Boolean)
|
||||
.join('\n')
|
||||
|
||||
return `You are a senior QA engineer analyzing a bug report for ComfyUI frontend (a Vue 3 + TypeScript web application for AI image generation workflows).
|
||||
return `You are a senior QA engineer analyzing a bug report for ComfyUI frontend — a node-based visual workflow editor for AI image generation (Vue 3 + TypeScript).
|
||||
|
||||
Your task: Generate a single targeted QA reproduction guide to verify this bug on the current main branch.
|
||||
The UI has:
|
||||
- A large canvas (1280x720 viewport) showing a node graph centered at ~(640, 400)
|
||||
- Nodes are boxes with input/output slots connected by wires
|
||||
- A hamburger menu (top-left C logo) with File, Edit, Help submenus
|
||||
- Sidebars (Workflows, Node Library, Models)
|
||||
- A topbar with workflow tabs and Queue button
|
||||
- The default workflow loads with ~5 nodes already visible on canvas
|
||||
|
||||
Your task: Generate a DETAILED reproduction guide (8-15 steps) to trigger this bug on main.
|
||||
|
||||
${allText}
|
||||
|
||||
## Available test actions
|
||||
Each step must use one of these actions:
|
||||
|
||||
### Menu actions
|
||||
- "openMenu" — clicks the Comfy hamburger menu (top-left C logo)
|
||||
- "hoverMenuItem" — hovers a top-level menu item to open submenu (label required)
|
||||
- "clickMenuItem" — clicks an item in the visible submenu (label required)
|
||||
- "fillDialog" — fills dialog input and presses Enter (text required)
|
||||
- "pressKey" — presses a keyboard key (key required)
|
||||
|
||||
### Element actions (by visible text)
|
||||
- "click" — clicks an element by visible text (text required)
|
||||
- "rightClick" — right-clicks an element to open context menu (text required)
|
||||
- "doubleClick" — double-clicks an element or coordinates (text or x,y)
|
||||
- "fillDialog" — fills dialog input and presses Enter (text required)
|
||||
- "pressKey" — presses a keyboard key (key required: Escape, Tab, Delete, Enter, etc.)
|
||||
|
||||
### Canvas actions (by coordinates — viewport is 1280x720)
|
||||
- "clickCanvas" — click at coordinates (x, y required)
|
||||
- "rightClickCanvas" — right-click at coordinates (x, y required)
|
||||
- "doubleClick" — double-click at coordinates to open node search (x, y)
|
||||
- "dragCanvas" — drag from one point to another (fromX, fromY, toX, toY)
|
||||
- "scrollCanvas" — scroll wheel for zoom (x, y, deltaY: negative=zoom in, positive=zoom out)
|
||||
|
||||
### Utility
|
||||
- "wait" — waits briefly (ms required, max 3000)
|
||||
- "screenshot" — takes a screenshot (name required)
|
||||
|
||||
## Common ComfyUI interactions
|
||||
- Right-click a node → context menu with Clone, Bypass, Remove, Colors, etc.
|
||||
- Double-click empty canvas → opens node search dialog
|
||||
- Ctrl+C / Ctrl+V → copy/paste selected nodes
|
||||
- Delete key → remove selected node
|
||||
- Ctrl+G → group selected nodes
|
||||
- Drag from output slot to input slot → create connection
|
||||
- Click a node to select it, Shift+click for multi-select
|
||||
|
||||
## Output format
|
||||
Return a JSON object with exactly one key: "reproduce", containing:
|
||||
{
|
||||
@@ -441,8 +473,8 @@ Return a JSON object with exactly one key: "reproduce", containing:
|
||||
"prerequisites": ["e.g. Load default workflow"],
|
||||
"steps": [
|
||||
{
|
||||
"action": "openMenu",
|
||||
"description": "Open the main menu to trigger the reported bug",
|
||||
"action": "clickCanvas",
|
||||
"description": "Click on first node to select it",
|
||||
"expected_before": "What should happen if the bug is present"
|
||||
}
|
||||
],
|
||||
@@ -450,12 +482,12 @@ Return a JSON object with exactly one key: "reproduce", containing:
|
||||
}
|
||||
|
||||
## Rules
|
||||
- REPRODUCE guide: 3-6 steps, under 30 seconds. Follow the issue's reproduction steps closely.
|
||||
- Focus on triggering and demonstrating the SPECIFIC bug reported.
|
||||
- Use information from the issue description and comments to understand the bug.
|
||||
- Include at least one screenshot step to capture the bug state.
|
||||
- Generate 8-15 DETAILED steps that actually trigger the reported bug.
|
||||
- Follow the issue's reproduction steps PRECISELY — translate them into available actions.
|
||||
- Use canvas coordinates for node interactions (nodes are typically in the center area 300-900 x 200-500).
|
||||
- Take screenshots BEFORE and AFTER critical actions to capture the bug state.
|
||||
- Do NOT just open a menu and screenshot — actually perform the full reproduction sequence.
|
||||
- Do NOT include login steps.
|
||||
- Menu pattern: openMenu -> hoverMenuItem -> clickMenuItem or screenshot.
|
||||
- Output ONLY valid JSON, no markdown fences or explanation.`
|
||||
}
|
||||
|
||||
|
||||
@@ -30,6 +30,18 @@ type TestAction =
|
||||
| { action: 'fillDialog'; text: string }
|
||||
| { action: 'pressKey'; key: string }
|
||||
| { action: 'click'; text: string }
|
||||
| { action: 'rightClick'; text: string }
|
||||
| { action: 'doubleClick'; text?: string; x?: number; y?: number }
|
||||
| { action: 'clickCanvas'; x: number; y: number }
|
||||
| { action: 'rightClickCanvas'; x: number; y: number }
|
||||
| {
|
||||
action: 'dragCanvas'
|
||||
fromX: number
|
||||
fromY: number
|
||||
toX: number
|
||||
toY: number
|
||||
}
|
||||
| { action: 'scrollCanvas'; x: number; y: number; deltaY: number }
|
||||
| { action: 'wait'; ms: number }
|
||||
| { action: 'screenshot'; name: string }
|
||||
|
||||
@@ -127,7 +139,7 @@ function buildPrompt(
|
||||
after:
|
||||
'AFTER (PR branch). Prove the changes work — 3-6 targeted steps, under 30 seconds.',
|
||||
reproduce:
|
||||
'REPRODUCE a reported issue on main branch. Follow the reproduction steps to trigger and demonstrate the reported bug — 3-6 steps, under 30 seconds.'
|
||||
'REPRODUCE a reported issue on main branch. Follow the reproduction steps PRECISELY to trigger and demonstrate the reported bug. Use 8-15 detailed steps, up to 60 seconds. You must actually perform the actions that trigger the bug — not just open a menu and take a screenshot.'
|
||||
}
|
||||
const modeDesc = modeDescriptions[mode] ?? modeDescriptions.before
|
||||
|
||||
@@ -154,16 +166,39 @@ ${testPlan.slice(0, 4000)}
|
||||
|
||||
return `You are generating test steps for a ComfyUI frontend QA recording.
|
||||
|
||||
ComfyUI is a node-based visual workflow editor for AI image generation. The UI has:
|
||||
- A large **canvas** (1280x720 viewport) showing a node graph, centered roughly at (640, 400)
|
||||
- Nodes are boxes with input/output slots that can be connected with wires
|
||||
- A **hamburger menu** (top-left C logo) with File, Edit, Help submenus
|
||||
- A **sidebar** on the left (Workflows, Node Library, Models)
|
||||
- A **topbar** with workflow tabs and Queue button
|
||||
- The **default workflow** loads with ~5 nodes already on canvas
|
||||
|
||||
MODE: ${modeDesc}
|
||||
|
||||
## Available actions (JSON array)
|
||||
Each step is an object with an "action" field:
|
||||
|
||||
### Menu actions
|
||||
- { "action": "openMenu" } — clicks the Comfy hamburger menu (top-left C logo)
|
||||
- { "action": "hoverMenuItem", "label": "File" } — hovers a top-level menu item to open submenu
|
||||
- { "action": "clickMenuItem", "label": "Save As" } — clicks an item in the visible submenu
|
||||
- { "action": "fillDialog", "text": "test-name" } — fills the dialog input and presses Enter
|
||||
- { "action": "pressKey", "key": "Escape" } — presses a keyboard key
|
||||
|
||||
### Element actions (by visible text)
|
||||
- { "action": "click", "text": "Button Text" } — clicks an element by visible text
|
||||
- { "action": "rightClick", "text": "NodeTitle" } — right-clicks an element (opens context menu)
|
||||
- { "action": "doubleClick", "text": "NodeTitle" } — double-clicks an element
|
||||
- { "action": "fillDialog", "text": "test-name" } — fills dialog input and presses Enter
|
||||
- { "action": "pressKey", "key": "Escape" } — presses a keyboard key (Escape, Tab, Delete, Enter, etc.)
|
||||
|
||||
### Canvas actions (by coordinates — viewport is 1280x720)
|
||||
- { "action": "clickCanvas", "x": 640, "y": 400 } — click at coordinates on canvas
|
||||
- { "action": "rightClickCanvas", "x": 500, "y": 350 } — right-click on canvas (node/canvas context menu)
|
||||
- { "action": "doubleClick", "x": 640, "y": 400 } — double-click on canvas (opens node search)
|
||||
- { "action": "dragCanvas", "fromX": 400, "fromY": 300, "toX": 600, "toY": 300 } — drag (move nodes, pan canvas)
|
||||
- { "action": "scrollCanvas", "x": 640, "y": 400, "deltaY": -300 } — scroll wheel (zoom in: negative, zoom out: positive)
|
||||
|
||||
### Utility actions
|
||||
- { "action": "wait", "ms": 1000 } — waits (use sparingly, max 3000ms)
|
||||
- { "action": "screenshot", "name": "step-name" } — takes a screenshot
|
||||
${qaGuideSection}${testPlanSection}
|
||||
@@ -171,22 +206,28 @@ ${diff ? `## PR Diff\n\`\`\`\n${diff.slice(0, 3000)}\n\`\`\`` : ''}
|
||||
|
||||
## Rules
|
||||
- Output ONLY a valid JSON array of actions, no markdown fences or explanation
|
||||
- ${mode === 'reproduce' ? 'Follow the reproduction steps to trigger and demonstrate the reported bug' : mode === 'before' ? 'Keep it minimal — just show the old/missing behavior' : 'Test the specific behavior that changed in the PR'}
|
||||
- ${mode === 'reproduce' ? 'You MUST follow the reproduction steps from the issue closely. Generate 8-15 steps that actually trigger the bug. Do NOT just open a menu and take a screenshot — perform the FULL reproduction sequence including node interactions, context menus, keyboard shortcuts, and canvas operations' : mode === 'before' ? 'Keep it minimal — just show the old/missing behavior' : 'Test the specific behavior that changed in the PR'}
|
||||
- Always include at least one screenshot
|
||||
- Do NOT include login steps (handled automatically)
|
||||
- The default workflow is already loaded when your steps start
|
||||
- Menu navigation pattern: openMenu → hoverMenuItem → clickMenuItem (or screenshot)
|
||||
- For node interactions: right-click a node to get its context menu, double-click canvas to add nodes
|
||||
- Take screenshots BEFORE and AFTER critical actions to capture the bug state
|
||||
${qaGuide ? '- Follow the QA Analysis Guide steps closely — they are well-researched and specific' : diff ? '- Pick test steps from the QA test plan categories that are most relevant to the diff' : '- Pick test steps from the QA test plan categories most likely to reveal bugs'}
|
||||
${mode === 'reproduce' ? '- Common ComfyUI actions: right-click node for context menu (Clone, Bypass, etc.), Ctrl+C/Ctrl+V to copy/paste, Delete key to remove, double-click canvas to add node via search, drag from output to input to connect' : ''}
|
||||
|
||||
## Example output
|
||||
[
|
||||
{"action":"openMenu"},
|
||||
{"action":"hoverMenuItem","label":"File"},
|
||||
{"action":"screenshot","name":"file-menu"},
|
||||
{"action":"clickMenuItem","label":"Save As"},
|
||||
{"action":"wait","ms":800},
|
||||
{"action":"fillDialog","text":"test-save"},
|
||||
{"action":"wait","ms":2000},
|
||||
{"action":"screenshot","name":"after-save"}
|
||||
{"action":"clickCanvas","x":450,"y":350},
|
||||
{"action":"screenshot","name":"node-selected"},
|
||||
{"action":"rightClickCanvas","x":450,"y":350},
|
||||
{"action":"wait","ms":500},
|
||||
{"action":"screenshot","name":"context-menu"},
|
||||
{"action":"clickMenuItem","label":"Clone"},
|
||||
{"action":"wait","ms":500},
|
||||
{"action":"screenshot","name":"after-clone"},
|
||||
{"action":"pressKey","key":"Delete"},
|
||||
{"action":"screenshot","name":"after-delete"}
|
||||
]`
|
||||
}
|
||||
|
||||
@@ -256,7 +297,21 @@ const FALLBACK_AFTER: TestAction[] = [
|
||||
{ action: 'screenshot', name: 'editor-after' }
|
||||
]
|
||||
|
||||
const FALLBACK_REPRODUCE: TestAction[] = FALLBACK_BEFORE
|
||||
const FALLBACK_REPRODUCE: TestAction[] = [
|
||||
{ action: 'screenshot', name: 'initial-state' },
|
||||
{ action: 'clickCanvas', x: 450, y: 350 },
|
||||
{ action: 'wait', ms: 300 },
|
||||
{ action: 'screenshot', name: 'node-selected' },
|
||||
{ action: 'rightClickCanvas', x: 450, y: 350 },
|
||||
{ action: 'wait', ms: 500 },
|
||||
{ action: 'screenshot', name: 'context-menu' },
|
||||
{ action: 'pressKey', key: 'Escape' },
|
||||
{ action: 'openMenu' },
|
||||
{ action: 'hoverMenuItem', label: 'File' },
|
||||
{ action: 'screenshot', name: 'file-menu' },
|
||||
{ action: 'pressKey', key: 'Escape' },
|
||||
{ action: 'screenshot', name: 'final-state' }
|
||||
]
|
||||
|
||||
const FALLBACK_STEPS: Record<RecordMode, TestAction[]> = {
|
||||
before: FALLBACK_BEFORE,
|
||||
@@ -364,7 +419,7 @@ async function executeSteps(
|
||||
) {
|
||||
for (const step of steps) {
|
||||
console.warn(
|
||||
` → ${step.action}${('label' in step && `: ${step.label}`) || ('text' in step && `: ${step.text}`) || ('name' in step && `: ${step.name}`) || ''}`
|
||||
` → ${step.action}${('label' in step && `: ${step.label}`) || ('text' in step && `: ${step.text}`) || ('name' in step && `: ${step.name}`) || ('x' in step && `: (${step.x},${step.y})`) || ''}`
|
||||
)
|
||||
switch (step.action) {
|
||||
case 'openMenu':
|
||||
@@ -392,6 +447,65 @@ async function executeSteps(
|
||||
case 'click':
|
||||
await clickByText(page, step.text)
|
||||
break
|
||||
case 'rightClick': {
|
||||
const rcEl = page.locator(`text=${step.text}`).first()
|
||||
if (await rcEl.isVisible().catch(() => false)) {
|
||||
await rcEl.click({ button: 'right' })
|
||||
await sleep(500)
|
||||
} else {
|
||||
console.warn(
|
||||
`Element with text "${step.text}" not found for rightClick`
|
||||
)
|
||||
}
|
||||
break
|
||||
}
|
||||
case 'doubleClick': {
|
||||
if (step.x !== undefined && step.y !== undefined) {
|
||||
await page.mouse.dblclick(step.x, step.y)
|
||||
} else if (step.text) {
|
||||
const dcEl = page.locator(`text=${step.text}`).first()
|
||||
if (await dcEl.isVisible().catch(() => false)) {
|
||||
await dcEl.dblclick()
|
||||
} else {
|
||||
console.warn(
|
||||
`Element with text "${step.text}" not found for doubleClick`
|
||||
)
|
||||
}
|
||||
} else {
|
||||
// Double-click center of canvas as fallback
|
||||
await page.mouse.dblclick(640, 400)
|
||||
}
|
||||
await sleep(500)
|
||||
break
|
||||
}
|
||||
case 'clickCanvas':
|
||||
await page.mouse.click(step.x, step.y)
|
||||
await sleep(300)
|
||||
break
|
||||
case 'rightClickCanvas':
|
||||
await page.mouse.click(step.x, step.y, { button: 'right' })
|
||||
await sleep(500)
|
||||
break
|
||||
case 'dragCanvas': {
|
||||
await page.mouse.move(step.fromX, step.fromY)
|
||||
await page.mouse.down()
|
||||
await sleep(100)
|
||||
const dragSteps = 5
|
||||
for (let i = 1; i <= dragSteps; i++) {
|
||||
const x = step.fromX + ((step.toX - step.fromX) * i) / dragSteps
|
||||
const y = step.fromY + ((step.toY - step.fromY) * i) / dragSteps
|
||||
await page.mouse.move(x, y)
|
||||
await sleep(50)
|
||||
}
|
||||
await page.mouse.up()
|
||||
await sleep(300)
|
||||
break
|
||||
}
|
||||
case 'scrollCanvas':
|
||||
await page.mouse.move(step.x, step.y)
|
||||
await page.mouse.wheel(0, step.deltaY)
|
||||
await sleep(500)
|
||||
break
|
||||
case 'wait':
|
||||
await sleep(Math.min(step.ms, 5000))
|
||||
break
|
||||
|
||||
@@ -418,6 +418,12 @@ function buildSingleVideoPrompt(
|
||||
'1. Does the video demonstrate the reported bug occurring?',
|
||||
'2. Is the bug clearly visible and reproducible from the steps shown?',
|
||||
'3. Are there any other issues visible during the reproduction attempt?',
|
||||
'',
|
||||
'## CRITICAL: Honesty Requirements',
|
||||
'- If the video only shows login, idle canvas, or trivial menu interactions WITHOUT actually performing the reproduction steps, say "INCONCLUSIVE — reproduction steps were not performed".',
|
||||
'- Do NOT claim a bug is "confirmed" unless you can clearly see the bug behavior described in the issue.',
|
||||
'- Do NOT hallucinate findings. If the video does not show meaningful interaction, say so clearly.',
|
||||
'- Rate confidence as "Low" if the video does not actually demonstrate the bug scenario.',
|
||||
''
|
||||
)
|
||||
} else {
|
||||
|
||||
Reference in New Issue
Block a user