Compare commits

...

2 Commits

Author SHA1 Message Date
snomiao
fb5813dddb feat: richer fixture API hints — i18n, queue mocks, subgraph helpers
- Add Comfy.Locale setting ID for i18n language switching tests
- Add queue/assets mock API (mockOutputHistory, runButton)
- Add subgraph workflow asset paths for loadWorkflow
- Add SubgraphHelper API docs (slot ops, navigation, conversion)
- Add VueNodeHelpers enterSubgraph/selectNode
2026-04-15 16:40:27 +00:00
snomiao
e153e58c91 feat: smarter agent — precondition reasoning + issue comments
- Prompt agent to reason about hidden preconditions before writing tests
  (e.g. z-index bugs need crowded canvas, not empty default workflow)
- Fetch issue comments with reproduction hints (repro/step/workaround)
- Better error analysis: different strategy on retry, not same code
- Both CI workflow and pnpm qa CLI fetch comments
2026-04-15 16:31:20 +00:00
3 changed files with 52 additions and 8 deletions

View File

@@ -402,8 +402,11 @@ export async function runResearchPhase(
- done(verdict, summary, evidence, testCode) — Finish with the final test
## Workflow
1. Read the issue description carefully
2. FIRST: Use readTest() to read 1-2 existing tests similar to the bug you're reproducing:
1. Read the issue description carefully. Think about:
- What PRECONDITIONS are needed? (many nodes on canvas? specific layout? saved workflow? subgraph?)
- What HIDDEN ASSUMPTIONS exist? (e.g. "z-index bug" means nodes must overlap → need a crowded canvas)
- What specific UI STATE triggers the bug? (dirty workflow? collapsed node? specific menu open?)
2. Use readTest() to read 1-2 existing tests similar to the bug:
- For menu/workflow bugs: readTest("workflow.spec.ts") or readTest("topbarMenu.spec.ts")
- For node/canvas bugs: readTest("nodeInteraction.spec.ts") or readTest("copyPaste.spec.ts")
- For settings bugs: readTest("settingDialogSearch.spec.ts")
@@ -411,10 +414,16 @@ export async function runResearchPhase(
3. Use inspect() to understand the current UI state and discover element selectors
4. If unsure about the fixture API, use readFixture("ComfyPage.ts") or relevant helper
5. Write a Playwright test that:
- Performs the exact reproduction steps from the issue
- Asserts the BROKEN behavior (the bug) — so the test PASSES when the bug exists
- FIRST sets up the preconditions (add multiple nodes, create specific layout, save workflow, etc.)
- THEN performs the reproduction steps from the issue
- FINALLY asserts the BROKEN behavior (the bug) — so the test PASSES when the bug exists
- Think like a tester: the bug may only appear under specific conditions that the reporter assumed were obvious
6. Run the test with runTest()
7. If it fails: read the error, fix the test, run again (max 5 attempts)
7. If it fails, ANALYZE the error before retrying:
- Is it a selector issue? Use inspect() to find the right element
- Is it a timing issue? The UI may need time to update — use nextFrame() or expect.poll()
- Is the precondition wrong? Maybe the bug only appears with MORE nodes, AFTER a save, etc.
- Try a DIFFERENT approach, not the same code with minor tweaks
8. Call done() with the final verdict and test code
## Test writing guidelines
@@ -488,6 +497,10 @@ export async function runResearchPhase(
### Settings (comfyPage.settings)
- \`.setSetting(id, value)\` — change a ComfyUI setting
- \`.getSetting(id)\` — read current setting value
- Common setting IDs:
- \`'Comfy.UseNewMenu'\` — 'Top' | 'Bottom' | 'Disabled'
- \`'Comfy.Locale'\` — 'en' | 'zh' | 'ja' | 'ko' | 'ru' | 'fr' | 'es' etc. (change UI language)
- \`'Comfy.NodeBadge.NodeSourceBadgeMode'\` — node badge display
### Keyboard (comfyPage.keyboard)
- \`.undo()\` / \`.redo()\` — Ctrl+Z / Ctrl+Y
@@ -500,6 +513,7 @@ export async function runResearchPhase(
- \`.setupWorkflowsDirectory(structure)\` — setup test directory
- \`.deleteWorkflow(name)\`
- \`.isCurrentWorkflowModified()\` — check dirty state
- Available subgraph assets: loadWorkflow('subgraphs/basic-subgraph'), 'subgraphs/nested-subgraph', 'subgraphs/subgraph-with-promoted-text-widget', etc.
### Context Menu (comfyPage.contextMenu)
- \`.openFor(locator)\` — right-click locator and wait for menu
@@ -507,12 +521,26 @@ export async function runResearchPhase(
- \`.isVisible()\` — check if context menu is showing
- \`.assertHasItems(items)\` — assert menu contains items
### Queue & Assets (comfyPage.assets)
- \`comfyPage.runButton.click()\` — execute current workflow (backend runs with --cpu in CI)
- \`comfyPage.assets.mockOutputHistory(jobs)\` — mock queue history with fake job items
- \`comfyPage.assets.mockEmptyState()\` — clear all mocked state
- Queue overlay: \`page.getByTestId('queue-overlay-toggle')\` to open queue panel
### Subgraph (comfyPage.subgraph)
- \`.isInSubgraph()\` — check if currently viewing a subgraph
- \`.getNodeCount()\` — nodes in current graph view
- \`.getSlotCount('input'|'output')\` — I/O slot count
- \`.connectToInput(sourceNode, slotIdx, inputName)\` — connect to subgraph input
- \`.exitViaBreadcrumb()\` — navigate out of subgraph
- \`.convertDefaultKSamplerToSubgraph()\` — helper: convert default workflow node to subgraph
- NodeReference: \`.convertToSubgraph()\`, \`.navigateIntoSubgraph()\`
### Other helpers
- \`comfyPage.settingDialog\` — SettingDialog component
- \`comfyPage.searchBox\` / \`comfyPage.searchBoxV2\` — node search
- \`comfyPage.toast\` — ToastHelper (\`.visibleToasts\`)
- \`comfyPage.subgraph\`SubgraphHelper
- \`comfyPage.vueNodes\` — VueNodeHelpers
- \`comfyPage.vueNodes\`VueNodeHelpers (\`.enterSubgraph(nodeId)\`, \`.selectNode(nodeId)\`)
- \`comfyPage.bottomPanel\` — BottomPanel
- \`comfyPage.clipboard\` — ClipboardHelper
- \`comfyPage.dragDrop\` — DragDropHelper

View File

@@ -193,7 +193,17 @@ function fetchIssue(number: string, repo: string, outputDir: string): string {
const body = shell(
`gh issue view ${number} --repo ${repo} --json title,body,labels --jq '"Title: " + .title + "\\n\\nLabels: " + ([.labels[].name] | join(", ")) + "\\n\\n" + .body'`
)
return writeTmpFile(outputDir, `issue-${number}.txt`, body)
// Append relevant comments for reproduction context
let comments = ''
try {
comments = shell(
`gh issue view ${number} --repo ${repo} --comments --json comments --jq '[.comments[] | select(.body | test("repro|step|how to|workaround"; "i")) | .body] | first(5; .[]) // empty'`
)
} catch {
// comments fetch failed, not critical
}
const content = comments ? `${body}\n\n--- Comments ---\n\n${comments}` : body
return writeTmpFile(outputDir, `issue-${number}.txt`, content)
}
function fetchPR(number: string, repo: string, outputDir: string): string {

View File

@@ -272,6 +272,12 @@ jobs:
--repo ${{ github.repository }} \
--json title,body,labels --jq '"Labels: \([.labels[].name] | join(", "))\nTitle: \(.title)\n\n\(.body)"' \
> "${{ runner.temp }}/issue-body.txt"
# Append top comments for reproduction context
gh issue view ${{ needs.resolve-matrix.outputs.number }} \
--repo ${{ github.repository }} \
--comments --json comments \
--jq '[.comments[] | select(.authorAssociation != "NONE" or (.body | test("repro|step|how to|workaround"; "i"))) | .body] | first(5; .[]) // empty' \
>> "${{ runner.temp }}/issue-body.txt" 2>/dev/null || true
echo "Issue body saved ($(wc -c < "${{ runner.temp }}/issue-body.txt") bytes)"
- name: Download QA guide