### Description
Improve Playwright PR comment format
### Problem
The current Playwright PR comment format is verbose and doesn't provide
easy access to failing test details.
Developers need to navigate multiple levels deep to:
Find which tests failed
Access test source code
View trace files for debugging
This makes debugging test failures tedious and time-consuming.
### Solution
Improved the Playwright PR comment format to be concise and actionable
by:
Modified extract-playwright-counts.ts to extract detailed failure
information from Playwright JSON reports including test names, file
paths, and trace URLs
Updated pr-playwright-deploy-and-comment.sh to generate concise comments
with failed tests listed upfront
Modified ci-tests-e2e.yaml to pass GITHUB_SHA for source code links
Modified ci-tests-e2e-forks.yaml to pass GITHUB_SHA for forked PR
workflow
**Before:**
Large multi-section layout with emoji-heavy headers
Summary section listing all counts vertically
Browser results displayed prominently with detailed counts
Failed test details only accessible through report links
No direct links to test source code or traces
**After:**
Concise single-line header with status
Single-line summary: "X passed, Y failed, Z flaky, W skipped (Total: N)"
Failed tests section (only shown when tests fail) with:
Direct links to test source code on GitHub
Direct links to trace viewer for each failure
Browser details collapsed in details section
Overall roughly half size reduction in visible text
### Testing
Verified TypeScript extraction logic for parsing Playwright JSON reports
Validated shell script syntax
Confirmed GitHub workflow changes are properly formatted
Will be fully tested on next PR with actual test failures
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-7882-feat-improved-playwright-comment-format-2e16d73d365081609078e34773063511)
by [Unito](https://www.unito.io)
* feat: add test count display to Playwright PR comments
- Add extract-playwright-counts.mjs script to parse test results from Playwright reports
- Update pr-playwright-deploy-and-comment.sh to extract and display test counts
- Show overall summary with passed/failed/flaky/skipped counts
- Display per-browser test counts inline with report links
- Use dynamic status icons based on test results (✅/❌/⚠️)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: include skipped test count in per-browser display
- Add skipped test extraction for individual browser reports
- Update per-browser display format to show all four counts:
(✅ passed / ❌ failed / ⚠️ flaky / ⏭️ skipped)
- Provides complete test result visibility at a glance
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: improve test count extraction reliability in CI
- Use absolute paths for script and report directories
- Add debug logging to help diagnose extraction issues
- Move counts display after View Report link as requested
- Format: [View Report](url) • ✅ passed / ❌ failed / ⚠️ flaky / ⏭️ skipped
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: generate JSON reports alongside HTML for test count extraction
- Add JSON reporter to Playwright test runs
- Generate report.json alongside HTML reports
- Store JSON report in playwright-report directory
- This enables accurate test count extraction from CI artifacts
The HTML reports alone don't contain easily extractable test statistics
as they use a React app with dynamically loaded data. JSON reports
provide direct access to test counts.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: correct JSON reporter syntax for Playwright tests
- Use proper syntax for JSON reporter with outputFile option
- Run separate commands for HTML and JSON report merging
- Specify output path directly in reporter configuration
- Ensures report.json is created in playwright-report directory
This fixes the "No such file or directory" error when trying to move
report.json file, as it wasn't being created in the first place.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Revert "fix: correct JSON reporter syntax for Playwright tests"
This reverts commit 605d7cc1e2.
* fix: use correct Playwright reporter syntax with comma-separated list
- Use --reporter=html,json syntax (comma-separated, not space)
- Move test-results.json to playwright-report/report.json after generation
- Remove incorrect PLAYWRIGHT_JSON_OUTPUT_NAME env variable
- Add || true to prevent failure if JSON file doesn't exist
The JSON reporter outputs to test-results.json by default when using
the comma-separated reporter list syntax.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: improve test count extraction reliability in CI
- Use separate --reporter flags for list, html, and json
- Set PLAYWRIGHT_JSON_OUTPUT_NAME env var to specify JSON output path
- Run HTML and JSON report generation separately for merged reports
- Ensures report.json is created in playwright-report directory
The combined reporter syntax wasn't creating the JSON file properly.
Using separate reporter flags with env var ensures JSON is generated.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update scripts/cicd/pr-playwright-deploy-and-comment.sh
Co-authored-by: Alexander Brown <drjkl@comfy.org>
* refactor: convert extraction script to TypeScript and use tsx
- Convert extract-playwright-counts.mjs to TypeScript (.ts)
- Add proper TypeScript types for better type safety
- Use tsx for execution instead of node
- Auto-install tsx in CI if not available
- Better alignment with the TypeScript codebase
This provides better type safety and consistency with the rest of
the codebase while maintaining the same functionality.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* chore(pr-playwright-deploy-and-comment.sh): move tsx installation check to the beginning of the script for better organization and efficiency
* [auto-fix] Apply ESLint and Prettier fixes
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
Co-authored-by: GitHub Action <action@github.com>
