[PR #189] feat: QA Bot — automated visual QA & video evidence system #189

New issue

Open

opened 2026-10-27 21:10:26 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:10:26 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/Comfy-PR/pull/189
Author: @snomiao
Created: 4/4/2026
Status: 🔄 Open

Base: main ← Head: sno-qabot

📝 Commits (1)

17eecf8 feat: add QA bot system — orchestrator, browser automation, video recording, CLI

📊 Changes

25 files changed (+3370 additions, -1 deletions)

View changed files

➕ TODO.md (+90 -0)
📝 bot/cli.ts (+170 -0)
➕ bot/qa/bootstrap/comfyui-frontend.ts (+65 -0)
➕ bot/qa/bootstrap/generic-nextjs.ts (+54 -0)
➕ bot/qa/bootstrap/generic-vite.ts (+54 -0)
➕ bot/qa/bootstrap/index.ts (+29 -0)
➕ bot/qa/bootstrap/types.ts (+27 -0)
➕ bot/qa/browser/controller.ts (+124 -0)
➕ bot/qa/browser/recorder.ts (+74 -0)
➕ bot/qa/cli.ts (+165 -0)
➕ bot/qa/config.ts (+40 -0)
➕ bot/qa/orchestrator.ts (+143 -0)
➕ bot/qa/qa-agent.ts (+325 -0)
➕ bot/qa/types.ts (+74 -0)
📝 bun.lock (+10 -1)
➕ docs/qabot/ARCHITECTURE.md (+205 -0)
➕ docs/qabot/BROWSER-AUTOMATION.md (+242 -0)
➕ docs/qabot/CLI-SPEC.md (+227 -0)
➕ docs/qabot/README.md (+83 -0)
➕ docs/qabot/REPO-SUPPORT.md (+225 -0)

...and 5 more files

📄 Description

Summary

Adds a centralized QA Bot sub-agent to ComfyPR-Bot that automatically reproduces bugs, verifies PRs, and records HD video evidence — then posts results back to GitHub issues/PRs and Slack.

Unlike the prototype in ComfyUI_frontend PR #9430 (single-repo CI), this lives in Comfy-PR as a centralized service targeting any frontend repo.

What's Included

Implementation (`bot/qa/` — 14 files)

Module	Purpose
`types.ts`, `config.ts`	Shared types, default config (resolution, timeouts, agent model)
`orchestrator.ts`	Full pipeline: clone → detect → boot → agent → report
`bootstrap/` (5 files)	Auto-detect & start ComfyUI-frontend, Vite, Next.js apps
`browser/` (2 files)	Playwright launch, cursor overlay injection, video recording, screenshots
`qa-agent.ts`	AI-driven browser automation loop (Claude via AI SDK)
`report/` (2 files)	Markdown report generation, GitHub comment posting, label management
`cli.ts`	CLI handlers wired into `bot/cli.ts`

CLI Commands

prbot qa reproduce -i Comfy-Org/ComfyUI_frontend#10688
prbot qa verify    -p Comfy-Org/ComfyUI_frontend#9500
prbot qa smoke     -r Comfy-Org/ComfyUI_frontend
prbot qa demo      -r Comfy-Org/ComfyUI_frontend --prompt 'Demo the template browser'
prbot qa run       -r Comfy-Org/ComfyUI_frontend --prompt 'Check drag-and-drop on canvas'

Design Docs (`docs/qabot/` — 9 files)

README, Architecture, Triggers, Video Recording, Browser Automation, Multi-Repo Support, Reporting, CLI Spec, Roadmap

Roadmap (in `TODO.md`)

Phase 0 ✅ Foundation (this PR)
Phase 1 Video delivery (ffmpeg post-processing, GCS upload, Slack posting)
Phase 2 PR before/after verification
Phase 3 Multi-repo triggers (GitHub webhooks, Slack mentions, /qa comments)
Phase 4–7 Smoke scheduling, high-quality video production, Electron support, advanced features

Dependencies

Adds playwright (browser automation + video recording)
Uses existing ai, @ai-sdk/anthropic (already in deps)
Leverages existing lib/slack/, lib/video/, src/cli/ infra

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/Comfy-PR/pull/189 **Author:** [@snomiao](https://github.com/snomiao) **Created:** 4/4/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `sno-qabot` --- ### 📝 Commits (1) - [`17eecf8`](https://github.com/Comfy-Org/Comfy-PR/commit/17eecf84c16f2af492eea49687094203d3a30a63) feat: add QA bot system — orchestrator, browser automation, video recording, CLI ### 📊 Changes **25 files changed** (+3370 additions, -1 deletions) <details> <summary>View changed files</summary> ➕ `TODO.md` (+90 -0) 📝 `bot/cli.ts` (+170 -0) ➕ `bot/qa/bootstrap/comfyui-frontend.ts` (+65 -0) ➕ `bot/qa/bootstrap/generic-nextjs.ts` (+54 -0) ➕ `bot/qa/bootstrap/generic-vite.ts` (+54 -0) ➕ `bot/qa/bootstrap/index.ts` (+29 -0) ➕ `bot/qa/bootstrap/types.ts` (+27 -0) ➕ `bot/qa/browser/controller.ts` (+124 -0) ➕ `bot/qa/browser/recorder.ts` (+74 -0) ➕ `bot/qa/cli.ts` (+165 -0) ➕ `bot/qa/config.ts` (+40 -0) ➕ `bot/qa/orchestrator.ts` (+143 -0) ➕ `bot/qa/qa-agent.ts` (+325 -0) ➕ `bot/qa/types.ts` (+74 -0) 📝 `bun.lock` (+10 -1) ➕ `docs/qabot/ARCHITECTURE.md` (+205 -0) ➕ `docs/qabot/BROWSER-AUTOMATION.md` (+242 -0) ➕ `docs/qabot/CLI-SPEC.md` (+227 -0) ➕ `docs/qabot/README.md` (+83 -0) ➕ `docs/qabot/REPO-SUPPORT.md` (+225 -0) _...and 5 more files_ </details> ### 📄 Description ## Summary Adds a centralized QA Bot sub-agent to ComfyPR-Bot that **automatically reproduces bugs, verifies PRs, and records HD video evidence** — then posts results back to GitHub issues/PRs and Slack. Unlike the prototype in [ComfyUI_frontend PR #9430](https://github.com/Comfy-Org/ComfyUI_frontend/pull/9430) (single-repo CI), this lives in Comfy-PR as a **centralized service** targeting any frontend repo. ## What's Included ### Implementation (`bot/qa/` — 14 files) | Module | Purpose | |---|---| | `types.ts`, `config.ts` | Shared types, default config (resolution, timeouts, agent model) | | `orchestrator.ts` | Full pipeline: clone → detect → boot → agent → report | | `bootstrap/` (5 files) | Auto-detect & start ComfyUI-frontend, Vite, Next.js apps | | `browser/` (2 files) | Playwright launch, cursor overlay injection, video recording, screenshots | | `qa-agent.ts` | AI-driven browser automation loop (Claude via AI SDK) | | `report/` (2 files) | Markdown report generation, GitHub comment posting, label management | | `cli.ts` | CLI handlers wired into `bot/cli.ts` | ### CLI Commands ```bash prbot qa reproduce -i Comfy-Org/ComfyUI_frontend#10688 prbot qa verify -p Comfy-Org/ComfyUI_frontend#9500 prbot qa smoke -r Comfy-Org/ComfyUI_frontend prbot qa demo -r Comfy-Org/ComfyUI_frontend --prompt 'Demo the template browser' prbot qa run -r Comfy-Org/ComfyUI_frontend --prompt 'Check drag-and-drop on canvas' ``` ### Design Docs (`docs/qabot/` — 9 files) README, Architecture, Triggers, Video Recording, Browser Automation, Multi-Repo Support, Reporting, CLI Spec, Roadmap ### Roadmap (in `TODO.md`) - **Phase 0** ✅ Foundation (this PR) - **Phase 1** Video delivery (ffmpeg post-processing, GCS upload, Slack posting) - **Phase 2** PR before/after verification - **Phase 3** Multi-repo triggers (GitHub webhooks, Slack mentions, `/qa` comments) - **Phase 4–7** Smoke scheduling, high-quality video production, Electron support, advanced features ## Dependencies - Adds `playwright` (browser automation + video recording) - Uses existing `ai`, `@ai-sdk/anthropic` (already in deps) - Leverages existing `lib/slack/`, `lib/video/`, `src/cli/` infra --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:10:26 +00:00