### Description
Improve Playwright PR comment format
### Problem
The current Playwright PR comment format is verbose and doesn't provide
easy access to failing test details.
Developers need to navigate multiple levels deep to:
Find which tests failed
Access test source code
View trace files for debugging
This makes debugging test failures tedious and time-consuming.
### Solution
Improved the Playwright PR comment format to be concise and actionable
by:
Modified extract-playwright-counts.ts to extract detailed failure
information from Playwright JSON reports including test names, file
paths, and trace URLs
Updated pr-playwright-deploy-and-comment.sh to generate concise comments
with failed tests listed upfront
Modified ci-tests-e2e.yaml to pass GITHUB_SHA for source code links
Modified ci-tests-e2e-forks.yaml to pass GITHUB_SHA for forked PR
workflow
**Before:**
Large multi-section layout with emoji-heavy headers
Summary section listing all counts vertically
Browser results displayed prominently with detailed counts
Failed test details only accessible through report links
No direct links to test source code or traces
**After:**
Concise single-line header with status
Single-line summary: "X passed, Y failed, Z flaky, W skipped (Total: N)"
Failed tests section (only shown when tests fail) with:
Direct links to test source code on GitHub
Direct links to trace viewer for each failure
Browser details collapsed in details section
Overall roughly half size reduction in visible text
### Testing
Verified TypeScript extraction logic for parsing Playwright JSON reports
Validated shell script syntax
Confirmed GitHub workflow changes are properly formatted
Will be fully tested on next PR with actual test failures
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-7882-feat-improved-playwright-comment-format-2e16d73d365081609078e34773063511)
by [Unito](https://www.unito.io)
## Summary
- When the next minor branch (e.g., `core/1.35`) doesn't exist yet, fall
back to current minor (`core/1.34`) for patch releases instead of
failing
- Fixes the weekly release workflow failure when ComfyUI is on version
1.33.x and `core/1.34` doesn't exist yet
## Test plan
- [x] Tested locally with version where next minor exists (1.33.5 →
finds core/1.34)
- [x] Tested fallback when next minor doesn't exist (1.34.7 → falls back
to core/1.34)
- [x] Tested error case when neither branch exists
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-7286-fix-Fall-back-to-current-minor-when-next-minor-branch-doesn-t-exist-2c46d73d365081009762c732954e148d)
by [Unito](https://www.unito.io)
Adds scheduled workflow to bump ComfyUI frontend RC releases every
Monday at noon PST.
## Implementation
- **Resolver script** (`scripts/cicd/resolve-comfyui-release.ts`):
Checks ComfyUI `requirements.txt` and determines next minor version,
compares branch commits to latest patch tag
- **Workflow** (`.github/workflows/weekly-comfyui-release.yaml`):
- Scheduled for Monday 20:00 UTC (noon PST)
- Manual dispatch supported for testing/off-cycle runs
- Three jobs: resolve version → trigger release if needed → create
ComfyUI PR
- Reuses existing `release-version-bump.yaml` workflow
- Creates draft PR from fork to `comfyanonymous/ComfyUI` with updated
`requirements.txt`
- Includes error handling and validation for all steps
- Force pushes to same branch weekly to maintain single open PR
## Testing
- Resolver script tested locally: correctly identified v1.28.8 → v1.29.4
with release needed
- yamllint passes
- knip passes with `gh` binary added to ignore list
## Follow-up
Can test via workflow_dispatch once merged, or in this PR if we enable
Actions on branch.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-6877-feat-Add-weekly-ComfyUI-release-automation-2b46d73d36508154aa05c783c6942d9a)
by [Unito](https://www.unito.io)
## Summary
- Improves Storybook deployment and PR comment workflow similar to the
Playwright improvements in #5425
- Creates unified deployment and commenting system for better
maintainability
- Adds Cloudflare Pages deployment for Storybook previews
## Deployment Cases Matrix
| Case | PR Type | Branch | Deployment | Features |
|------|---------|--------|------------|----------|
| **1** | Non-forked PR | `version-bump-*` | ✅ Chromatic | • Visual diff
testing<br>• Chromatic build URL<br>• Chromatic Storybook URL<br>• Shows
visual changes |
| **2** | Non-forked PR | All branches | ✅ Cloudflare Pages | • Live
Storybook preview<br>• pages.dev URL<br>• No visual diff |
| **3** | Forked PR | Any branch | ✅ Cloudflare Pages | • Live Storybook
preview<br>• pages.dev URL<br>• No visual diff<br>• Runs via separate
workflow to avoid permission problems |
### Key Points:
- **Chromatic** (paid service): Only for `version-bump-*` branches to
track visual changes between releases
- **Cloudflare Pages** (free): For all other PRs to provide Storybook
preview without visual diff
- **Security**: Forked PRs use a separate workflow with limited
permissions
## Changes
### New Features
- 🚀 **Cloudflare Pages Deployment**: Storybook builds are now deployed
to Cloudflare Pages for easy preview
- 🔄 **Unified Script**: Single reusable shell script handles both
deployment and PR comments
- 🔒 **Better Security**: Separate workflows for fork vs non-fork PRs
### Improvements
- ♻️ **Retry Logic**: Automatic retry (3 attempts) for failed
deployments
- 📝 **Better Comments**: Clearer PR comments with deployment links and
status
- 🎯 **Simplified Logic**: Workflow logic moved to reusable script for
easier maintenance
- ⚡ **Better Error Handling**: Proper handling of different workflow
conclusions
- 🐛 **Fixed Comment Output**: Deployment logs now properly redirected to
stderr
### Files Changed
- `scripts/cicd/pr-storybook-deploy-and-comment.sh` - New unified
deployment script
- `.github/workflows/chromatic.yaml` - Updated to use new script and add
deployment
- `.github/workflows/pr-storybook-deploy.yaml` - New workflow for forked
PRs
- `.github/workflows/pr-storybook-comment.yaml` - Removed (replaced by
new system)
## ⚠️ Required Setup
The Cloudflare Pages project `comfyui-storybook` needs to be created
under the organization's Cloudflare account:
```bash
# Using the account ID from GitHub secrets
export CLOUDFLARE_ACCOUNT_ID=5ae914d9b87bcf6bbe1ada5798f92a5f
export CLOUDFLARE_API_TOKEN=<org-token>
wrangler pages project create comfyui-storybook --production-branch main
```
**Note**: The project must be created under the same Cloudflare account
that's configured in the GitHub secrets.
## Test Plan
- [x] Create Cloudflare Pages project `comfyui-storybook`
- [x] Workflow runs successfully on all PRs
- [x] PR comments are posted correctly at start and completion
- [x] Storybook deploys to Cloudflare Pages with correct URL
- [ ] Fork PRs are handled by separate workflow
- [ ] Non-fork PRs get inline deployment
- [ ] version-bump-* branches show Chromatic info
## References
- Similar improvements for Playwright: #5459
- Based on pattern from sno-fix-playwright-comment-2 branch
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Added GitHub Actions workflow and shell script to validate JSON syntax
in repository files, as in the past we have committed locales files with
invalid JSON.
## Changes
- **What**: Added CI workflow for JSON validation with `jq` syntax
checking
- **Dependencies**: CI workflow requires `jq` (pre-installed on
ubuntu-latest runners)
## Review Focus
Script exclusion patterns for TSConfig files and environment variable
override mechanism (`JSON_LINT_EXCLUDES`).
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-5837-ci-add-JSON-validation-CI-workflow-27c6d73d365081a199b1d9ae9e4edfa4)
by [Unito](https://www.unito.io)
* feat: add test count display to Playwright PR comments
- Add extract-playwright-counts.mjs script to parse test results from Playwright reports
- Update pr-playwright-deploy-and-comment.sh to extract and display test counts
- Show overall summary with passed/failed/flaky/skipped counts
- Display per-browser test counts inline with report links
- Use dynamic status icons based on test results (✅/❌/⚠️)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: include skipped test count in per-browser display
- Add skipped test extraction for individual browser reports
- Update per-browser display format to show all four counts:
(✅ passed / ❌ failed / ⚠️ flaky / ⏭️ skipped)
- Provides complete test result visibility at a glance
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: improve test count extraction reliability in CI
- Use absolute paths for script and report directories
- Add debug logging to help diagnose extraction issues
- Move counts display after View Report link as requested
- Format: [View Report](url) • ✅ passed / ❌ failed / ⚠️ flaky / ⏭️ skipped
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: generate JSON reports alongside HTML for test count extraction
- Add JSON reporter to Playwright test runs
- Generate report.json alongside HTML reports
- Store JSON report in playwright-report directory
- This enables accurate test count extraction from CI artifacts
The HTML reports alone don't contain easily extractable test statistics
as they use a React app with dynamically loaded data. JSON reports
provide direct access to test counts.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: correct JSON reporter syntax for Playwright tests
- Use proper syntax for JSON reporter with outputFile option
- Run separate commands for HTML and JSON report merging
- Specify output path directly in reporter configuration
- Ensures report.json is created in playwright-report directory
This fixes the "No such file or directory" error when trying to move
report.json file, as it wasn't being created in the first place.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Revert "fix: correct JSON reporter syntax for Playwright tests"
This reverts commit 605d7cc1e2.
* fix: use correct Playwright reporter syntax with comma-separated list
- Use --reporter=html,json syntax (comma-separated, not space)
- Move test-results.json to playwright-report/report.json after generation
- Remove incorrect PLAYWRIGHT_JSON_OUTPUT_NAME env variable
- Add || true to prevent failure if JSON file doesn't exist
The JSON reporter outputs to test-results.json by default when using
the comma-separated reporter list syntax.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: improve test count extraction reliability in CI
- Use separate --reporter flags for list, html, and json
- Set PLAYWRIGHT_JSON_OUTPUT_NAME env var to specify JSON output path
- Run HTML and JSON report generation separately for merged reports
- Ensures report.json is created in playwright-report directory
The combined reporter syntax wasn't creating the JSON file properly.
Using separate reporter flags with env var ensures JSON is generated.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update scripts/cicd/pr-playwright-deploy-and-comment.sh
Co-authored-by: Alexander Brown <drjkl@comfy.org>
* refactor: convert extraction script to TypeScript and use tsx
- Convert extract-playwright-counts.mjs to TypeScript (.ts)
- Add proper TypeScript types for better type safety
- Use tsx for execution instead of node
- Auto-install tsx in CI if not available
- Better alignment with the TypeScript codebase
This provides better type safety and consistency with the rest of
the codebase while maintaining the same functionality.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* chore(pr-playwright-deploy-and-comment.sh): move tsx installation check to the beginning of the script for better organization and efficiency
* [auto-fix] Apply ESLint and Prettier fixes
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
Co-authored-by: GitHub Action <action@github.com>
* [fix] Consolidate Playwright workflow jobs to fix missing deployment links
The issue in PR #5298 was caused by missing deployment-info artifact
creation. The deploy-reports job was deploying to Cloudflare but wasn't
creating the deployment-info-* artifacts that comment-tests-completed
job expected to download.
This change consolidates the deployment and commenting into a single job,
eliminating the artifact dependency and ensuring links are always available.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: Split PR deployment workflow for forked vs non-forked repos
- Extract deployment logic to reusable script (scripts/cicd/pr-playwright-deploy-and-comment.sh)
- Non-forked PRs: Use direct pull_request event in test-ui.yaml (faster)
- Forked PRs: Use workflow_run in pr-playwright-deploy.yaml (handles permissions)
- Add starting comment for both forked and non-forked PRs
- Make Cloudflare tokens optional for starting status comments
* refactor: Simplify PR deployment workflow and script
- Consolidate workflow into single job with clearer structure
- Reduce script from 200+ to ~140 lines
- Simplify deployment retry logic and comment generation
- Remove redundant checks and unnecessary complexity
* fix: Add debugging and wrangler installation to deployment script
- Add debug output to identify missing reports
- Install wrangler if not available
- Show deployment attempts and failures
- Log available reports before deployment
* chore: Trigger CI to test deployment workflow
* fix: Fix browser artifact name mismatch in deployment script
- Use dot notation (0.5x) for artifact names as Playwright creates them
- Convert to dash notation (0-5x) for Cloudflare project names
- Properly handle browser name display in comments
* refactor: Convert deployment script to POSIX sh for better compatibility
- Replace bash arrays with space-separated strings
- Use while loops instead of bash-specific for syntax
- Remove bash-specific string manipulation features
- Replace local variables (not required in functions)
- Ensure compatibility with standard /bin/sh
* fix: Fix deployment script output to properly capture URLs
- Redirect debug messages to stderr
- Only output URL to stdout for proper capture
- This fixes the missing deployment links in PR comments
* fix: Add input validation to prevent command injection
- Validate PR number is numeric only
- Sanitize branch name at script start
- Validate status parameter values
- Use pre-sanitized branch throughout script
- Addresses high-severity security issue from PR review
* fix: Add null checks and logging to workflow condition
- Add explicit null checks for head_repository and repository
- Add debug logging to help diagnose workflow trigger issues
- Prevents potential failures from undefined repository objects
- Addresses medium-severity issue from PR review
* fix: Pin wrangler to major version 4 with error handling
- Pin wrangler to major version 4 (^4.0.0) for stability
- Add error handling if wrangler installation fails
- Return 'failed' status if installation fails
- Addresses dependency management issue from PR review
* perf: Implement parallel deployments to reduce CI time
- Deploy all browser reports in parallel using background processes
- Use temporary directory to collect deployment results
- Wait for all deployments to complete before generating comment
- Maintains result order for consistent output
- Significantly reduces deployment time from sequential to parallel execution
* fix: Use specific comment ID for updates instead of edit-last
- Use GitHub API to find exact comment ID
- Update specific comment by ID to avoid editing wrong comment
- Prevents race conditions if user posts between finding and editing
- More reliable comment updates
* fix(workflows/test-ui.yaml): change condition to always run deploy job for pull requests to ensure deployment consistency
* fix(workflows/test-ui.yaml): change condition to always run deploy job for pull requests to ensure deployment consistency
* fix(pr-playwright-deploy-and-comment.sh): remove npx prefix from wrangler command for consistency and simplicity
* fix(pr-playwright-deploy-and-comment.sh): remove npx prefix from wrangler command for consistency and simplicity
* Update scripts/cicd/pr-playwright-deploy-and-comment.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix(pr-playwright-deploy-and-comment.sh): improve regex for URL extraction to include valid characters and ensure correct URL format
* chore(pr-playwright-deploy-and-comment.sh): move wrangler installation to the beginning of the script to avoid redundancy and improve efficiency
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
