Comfy-Org/ComfyUI_frontend
1
0
Fork 0
mirror of https://github.com/Comfy-Org/ComfyUI_frontend.git synced 2026-04-20 06:20:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add multi-OS support to QA workflow (linux/macos/windows)

Browse Source 
- Matrix strategy: ubuntu-latest, macos-latest, windows-latest
- workflow_dispatch input to select specific OS or run all
- Per-platform video recording:
  - Linux: Xvfb + ffmpeg x11grab (headed browser)
  - macOS: ffmpeg avfoundation
  - Windows: ffmpeg gdigrab
- Cross-platform: bash shell, runner.temp paths, curl wait loop
- Separate report job posts combined artifact links on PR
- OS name included in report filenames and artifact names

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
snomiao
2026-03-07 04:09:17 +00:00
parent 1e1fdaa253
commit ee5a207407
1 changed files with 137 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

227
.github/workflows/qa-claude.yaml vendored
View File

@@ -1,6 +1,5 @@
# Description: Automated QA of ComfyUI frontend using Claude CLI + Playwright MCP
# Records a video of the QA session and uploads artifacts
# Triggers on PRs from sno-skills or qa-* branches, or via 'qa-run' label
# Records video and uploads artifacts. Runs on Linux, macOS, and Windows.
name: 'QA: Claude Frontend QA'
on:
@@ -8,6 +7,11 @@ on:
types: [opened, synchronize, labeled]
branches: [main]
workflow_dispatch:
inputs:
os:
description: 'OS to run on (all, ubuntu, macos, windows)'
required: false
default: 'all'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -20,13 +24,26 @@ jobs:
github.event.label.name == 'qa-run' ||
startsWith(github.head_ref, 'sno-skills') ||
startsWith(github.head_ref, 'qa-')
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
exclude:
- os: ${{ github.event.inputs.os == 'ubuntu' && 'macos-latest' || 'NONE' }}
- os: ${{ github.event.inputs.os == 'ubuntu' && 'windows-latest' || 'NONE' }}
- os: ${{ github.event.inputs.os == 'macos' && 'ubuntu-latest' || 'NONE' }}
- os: ${{ github.event.inputs.os == 'macos' && 'windows-latest' || 'NONE' }}
- os: ${{ github.event.inputs.os == 'windows' && 'ubuntu-latest' || 'NONE' }}
- os: ${{ github.event.inputs.os == 'windows' && 'macos-latest' || 'NONE' }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
permissions:
contents: write
pull-requests: write
issues: write
packages: read
env:
QA_ARTIFACTS: ${{ runner.temp }}/qa-artifacts
steps:
- name: Checkout repository
uses: actions/checkout@v6
@@ -46,153 +63,183 @@ jobs:
launch_server: 'true'
- name: Wait for ComfyUI server
shell: bash
run: |
echo "Waiting for ComfyUI server on port 8188..."
timeout 120 bash -c 'until curl -sf http://127.0.0.1:8188/api/system_stats > /dev/null 2>&1; do sleep 2; done'
echo "ComfyUI server is ready"
echo "Waiting for ComfyUI server..."
for i in $(seq 1 60); do
if curl -sf http://127.0.0.1:8188/api/system_stats >/dev/null 2>&1; then
echo "Server ready"; exit 0
fi; sleep 2
done
echo "::error::Server timeout"; exit 1
- name: Install Playwright for MCP
shell: bash
run: pnpm exec playwright install chromium --with-deps
- name: Install Claude Code
shell: bash
run: npm install -g @anthropic-ai/claude-code
- name: Setup virtual display and screen recording
# --- Virtual display + recording: Linux ---
- name: Setup Xvfb (Linux)
if: runner.os == 'Linux'
run: |
sudo apt-get update -qq && sudo apt-get install -y -qq xvfb ffmpeg > /dev/null 2>&1
# Start Xvfb virtual display at 1280x720
sudo apt-get update -qq && sudo apt-get install -y -qq xvfb ffmpeg >/dev/null 2>&1
Xvfb :99 -screen 0 1280x720x24 -ac &
echo $! > /tmp/xvfb.pid
echo $! > "${{ runner.temp }}/xvfb.pid"
sleep 2
echo "DISPLAY=:99" >> "$GITHUB_ENV"
- name: Start screen recording
- name: Start recording (Linux)
if: runner.os == 'Linux'
run: |
mkdir -p /tmp/qa-artifacts
# Record the virtual display at 10fps with fast encoding
mkdir -p "$QA_ARTIFACTS"
ffmpeg -y -f x11grab -video_size 1280x720 -framerate 10 \
-i :99.0 -c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p /tmp/qa-artifacts/qa-session.mp4 &
echo $! > /tmp/ffmpeg.pid
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 1
# --- Recording: macOS (avfoundation) ---
- name: Start recording (macOS)
if: runner.os == 'macOS'
run: |
mkdir -p "$QA_ARTIFACTS"
ffmpeg -y -f avfoundation -framerate 10 -capture_cursor 1 \
-i "1:none" -c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" 2>/dev/null &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 1
# --- Recording: Windows (gdigrab) ---
- name: Start recording (Windows)
if: runner.os == 'Windows'
shell: bash
run: |
mkdir -p "$QA_ARTIFACTS"
ffmpeg -y -f gdigrab -framerate 10 -i desktop \
-c:v libx264 -preset ultrafast -crf 28 \
-pix_fmt yuv420p "$QA_ARTIFACTS/qa-session.mp4" 2>/dev/null &
echo $! > "${{ runner.temp }}/ffmpeg.pid"
sleep 1
echo "Screen recording started (PID: $(cat /tmp/ffmpeg.pid))"
- name: Create MCP config
shell: bash
run: |
cat > /tmp/mcp-config.json <<'MCPEOF'
{
"mcpServers": {
"playwright": {
"command": "npx",
"args": ["@playwright/mcp@latest"]
}
}
}
MCPEOF
if [ "$RUNNER_OS" = "Linux" ]; then
ARGS='["@playwright/mcp@latest"]'
else
ARGS='["@playwright/mcp@latest", "--headless"]'
fi
cat > "${{ runner.temp }}/mcp-config.json" <<EOF
{"mcpServers":{"playwright":{"command":"npx","args":${ARGS}}}}
EOF
- name: Write QA prompt
shell: bash
run: |
cat > /tmp/qa-prompt.txt <<PROMPT
OS_LOWER=$(echo "$RUNNER_OS" | tr '[:upper:]' '[:lower:]')
cat > "${{ runner.temp }}/qa-prompt.txt" <<PROMPT
You are running an automated QA pass on the ComfyUI frontend.
Read the file skills/comfy-qa/SKILL.md and follow the QA test plan.
Environment details:
- CI=true
- Server URL: http://127.0.0.1:8188
- Branch: ${{ github.head_ref || github.ref_name }}
- PR: #${{ github.event.pull_request.number }}
- Commit: ${{ github.sha }}
- Video recording: active (headed browser on virtual display)
Environment: CI=true, OS=${{ runner.os }}
Server URL: http://127.0.0.1:8188
Branch: ${{ github.head_ref || github.ref_name }}
PR: #${{ github.event.pull_request.number }}
Commit: ${{ github.sha }}
Instructions:
1. Use the playwright MCP tools to navigate http://127.0.0.1:8188
2. Run through the FULL QA test plan from the skill file
3. Take screenshots of any failures or notable states
4. Generate a QA report following the template in the skill file
5. Save the report to docs/qa/ with today's date
6. Commit and push the report to this branch
1. Use playwright MCP tools to navigate http://127.0.0.1:8188
2. Run the FULL QA test plan from the skill file
3. Take screenshots of failures or notable states
4. Save report to docs/qa/ as YYYY-MM-DD-NNN-${OS_LOWER}-report.md
5. Commit and push the report to this branch
Do NOT create a new PR. Commit directly to this branch.
Do NOT post a PR comment — the workflow handles that after artifacts upload.
Be thorough but efficient — skip tests that require features not
available in CI (e.g., file upload dialogs, real GPU execution).
Do NOT create a new PR. Do NOT post PR comments.
Skip tests not available in CI (file dialogs, GPU execution).
PROMPT
- name: Run Claude QA
shell: bash
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
CI: 'true'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DISPLAY: ':99'
run: |
cat /tmp/qa-prompt.txt | claude --print --verbose \
cat "${{ runner.temp }}/qa-prompt.txt" | claude --print --verbose \
--max-turns 128 \
--mcp-config /tmp/mcp-config.json \
--mcp-config "${{ runner.temp }}/mcp-config.json" \
--allowedTools "mcp__playwright__browser_navigate,mcp__playwright__browser_snapshot,mcp__playwright__browser_click,mcp__playwright__browser_type,mcp__playwright__browser_press_key,mcp__playwright__browser_take_screenshot,mcp__playwright__browser_hover,mcp__playwright__browser_drag,mcp__playwright__browser_select_option,mcp__playwright__browser_handle_dialog,mcp__playwright__browser_tab_list,mcp__playwright__browser_tab_new,mcp__playwright__browser_tab_select,mcp__playwright__browser_tab_close,mcp__playwright__browser_console_messages,mcp__playwright__browser_resize,mcp__playwright__browser_wait_for,Bash(git add:*),Bash(git commit:*),Bash(git push:*),Bash(git status:*),Bash(git log:*),Bash(git diff:*),Bash(date:*),Bash(ls:*),Bash(mkdir:*),Read,Write,Edit,Glob,Grep"
- name: Stop screen recording
- name: Stop recording
if: always()
shell: bash
run: |
if [ -f /tmp/ffmpeg.pid ]; then
# Send SIGINT for graceful mp4 finalization
kill -INT $(cat /tmp/ffmpeg.pid) 2>/dev/null || true
sleep 3
kill $(cat /tmp/ffmpeg.pid) 2>/dev/null || true
fi
if [ -f /tmp/qa-artifacts/qa-session.mp4 ]; then
SIZE=$(du -h /tmp/qa-artifacts/qa-session.mp4 | cut -f1)
echo "Video recorded: ${SIZE}"
else
echo "No video file found"
PID_FILE="${{ runner.temp }}/ffmpeg.pid"
if [ -f "$PID_FILE" ]; then
if [ "$RUNNER_OS" = "Windows" ]; then
taskkill //F //PID $(cat "$PID_FILE") 2>/dev/null || true
else
kill -INT $(cat "$PID_FILE") 2>/dev/null || true
sleep 3; kill $(cat "$PID_FILE") 2>/dev/null || true
fi
fi
[ -f "$QA_ARTIFACTS/qa-session.mp4" ] && \
echo "Video: $(du -h "$QA_ARTIFACTS/qa-session.mp4" | cut -f1)" || \
echo "No video (non-fatal)"
- name: Collect QA artifacts
- name: Collect artifacts
if: always()
shell: bash
run: |
# Copy screenshots and report into artifacts dir
cp -r docs/qa/* /tmp/qa-artifacts/ 2>/dev/null || true
ls -la /tmp/qa-artifacts/
mkdir -p "$QA_ARTIFACTS"
cp -r docs/qa/* "$QA_ARTIFACTS/" 2>/dev/null || true
ls -la "$QA_ARTIFACTS/" || true
- name: Upload QA artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: qa-report-${{ github.run_id }}
path: /tmp/qa-artifacts/
name: qa-report-${{ runner.os }}-${{ github.run_id }}
path: ${{ runner.temp }}/qa-artifacts/
retention-days: 14
- name: Post artifact link on PR
if: always() && github.event.pull_request.number
- name: Cleanup (Linux)
if: always() && runner.os == 'Linux'
run: kill $(cat "${{ runner.temp }}/xvfb.pid") 2>/dev/null || true
report:
needs: qa
if: always() && github.event.pull_request.number
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Post artifact links on PR
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
ARTIFACTS_URL="${RUN_URL}#artifacts"
RUN="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
gh pr comment ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} \
--body "## QA Artifacts — Multi-OS
gh pr comment ${{ github.event.pull_request.number }} --body "$(cat <<EOF
## QA Artifacts
**Run**: ${RUN}
**Run**: ${RUN_URL}
| OS | Artifact | Recording |
|----|----------|-----------|
| Linux | qa-report-Linux-${{ github.run_id }} | Xvfb + x11grab |
| macOS | qa-report-macOS-${{ github.run_id }} | avfoundation |
| Windows | qa-report-Windows-${{ github.run_id }} | gdigrab |
| Artifact | Details |
|----------|---------|
| Video | Screen recording of full QA session |
| Screenshots | Individual screenshots of each test area |
| Report | Full QA report markdown |
[Download all artifacts](${ARTIFACTS_URL}) (available for 14 days)
EOF
)"
[Download artifacts](${RUN}#artifacts) (14 days)"
- name: Remove qa-run label
if: always() && github.event.label.name == 'qa-run'
run: gh pr edit ${{ github.event.pull_request.number }} --remove-label "qa-run"
if: github.event.label.name == 'qa-run'
run: |
gh pr edit ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} --remove-label "qa-run"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Cleanup
if: always()
run: |
kill $(cat /tmp/xvfb.pid) 2>/dev/null || true