Commit Graph

5 Commits

Author SHA1 Message Date
Christian Byrne
5e4965d131 ci: add yamllint (#6682)
adds yaml linting to CI and applies rules to existing yaml files.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-6682-ci-add-yamllint-2aa6d73d365081b4b67ae9d9cc86760f)
by [Unito](https://www.unito.io)
2025-11-13 11:10:48 -07:00
snomiao
af10082134 [ci] Improve Storybook deployment and comment workflow (#5426)
## Summary
- Improves Storybook deployment and PR comment workflow similar to the
Playwright improvements in #5425
- Creates unified deployment and commenting system for better
maintainability
- Adds Cloudflare Pages deployment for Storybook previews

## Deployment Cases Matrix

| Case | PR Type | Branch | Deployment | Features |
|------|---------|--------|------------|----------|
| **1** | Non-forked PR | `version-bump-*` |  Chromatic | • Visual diff
testing<br>• Chromatic build URL<br>• Chromatic Storybook URL<br>• Shows
visual changes |
| **2** | Non-forked PR | All branches |  Cloudflare Pages | • Live
Storybook preview<br>• pages.dev URL<br>• No visual diff |
| **3** | Forked PR | Any branch |  Cloudflare Pages | • Live Storybook
preview<br>• pages.dev URL<br>• No visual diff<br>• Runs via separate
workflow to avoid permission problems |

### Key Points:
- **Chromatic** (paid service): Only for `version-bump-*` branches to
track visual changes between releases
- **Cloudflare Pages** (free): For all other PRs to provide Storybook
preview without visual diff
- **Security**: Forked PRs use a separate workflow with limited
permissions

## Changes

### New Features
- 🚀 **Cloudflare Pages Deployment**: Storybook builds are now deployed
to Cloudflare Pages for easy preview
- 🔄 **Unified Script**: Single reusable shell script handles both
deployment and PR comments
- 🔒 **Better Security**: Separate workflows for fork vs non-fork PRs

### Improvements
- ♻️ **Retry Logic**: Automatic retry (3 attempts) for failed
deployments
- 📝 **Better Comments**: Clearer PR comments with deployment links and
status
- 🎯 **Simplified Logic**: Workflow logic moved to reusable script for
easier maintenance
-  **Better Error Handling**: Proper handling of different workflow
conclusions
- 🐛 **Fixed Comment Output**: Deployment logs now properly redirected to
stderr

### Files Changed
- `scripts/cicd/pr-storybook-deploy-and-comment.sh` - New unified
deployment script
- `.github/workflows/chromatic.yaml` - Updated to use new script and add
deployment
- `.github/workflows/pr-storybook-deploy.yaml` - New workflow for forked
PRs
- `.github/workflows/pr-storybook-comment.yaml` - Removed (replaced by
new system)

## ⚠️ Required Setup

The Cloudflare Pages project `comfyui-storybook` needs to be created
under the organization's Cloudflare account:

```bash
# Using the account ID from GitHub secrets
export CLOUDFLARE_ACCOUNT_ID=5ae914d9b87bcf6bbe1ada5798f92a5f
export CLOUDFLARE_API_TOKEN=<org-token>
wrangler pages project create comfyui-storybook --production-branch main
```

**Note**: The project must be created under the same Cloudflare account
that's configured in the GitHub secrets.

## Test Plan
- [x] Create Cloudflare Pages project `comfyui-storybook`
- [x] Workflow runs successfully on all PRs
- [x] PR comments are posted correctly at start and completion
- [x] Storybook deploys to Cloudflare Pages with correct URL
- [ ] Fork PRs are handled by separate workflow
- [ ] Non-fork PRs get inline deployment
- [ ] version-bump-* branches show Chromatic info

## References
- Similar improvements for Playwright: #5459
- Based on pattern from sno-fix-playwright-comment-2 branch

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-09-29 21:42:00 -07:00
Christian Byrne
5d62e01c11 [ci] add JSON validation CI workflow (#5837)
## Summary

Added GitHub Actions workflow and shell script to validate JSON syntax
in repository files, as in the past we have committed locales files with
invalid JSON.

## Changes

- **What**: Added CI workflow for JSON validation with `jq` syntax
checking
- **Dependencies**: CI workflow requires `jq` (pre-installed on
ubuntu-latest runners)

## Review Focus

Script exclusion patterns for TSConfig files and environment variable
override mechanism (`JSON_LINT_EXCLUDES`).

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-5837-ci-add-JSON-validation-CI-workflow-27c6d73d365081a199b1d9ae9e4edfa4)
by [Unito](https://www.unito.io)
2025-09-28 15:30:01 -07:00
snomiao
48d01745cd feat: add test count display to Playwright PR comments (#5458)
* feat: add test count display to Playwright PR comments

- Add extract-playwright-counts.mjs script to parse test results from Playwright reports
- Update pr-playwright-deploy-and-comment.sh to extract and display test counts
- Show overall summary with passed/failed/flaky/skipped counts
- Display per-browser test counts inline with report links
- Use dynamic status icons based on test results (//⚠️)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: include skipped test count in per-browser display

- Add skipped test extraction for individual browser reports
- Update per-browser display format to show all four counts:
  ( passed /  failed / ⚠️ flaky / ⏭️ skipped)
- Provides complete test result visibility at a glance

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: improve test count extraction reliability in CI

- Use absolute paths for script and report directories
- Add debug logging to help diagnose extraction issues
- Move counts display after View Report link as requested
- Format: [View Report](url) •  passed /  failed / ⚠️ flaky / ⏭️ skipped

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: generate JSON reports alongside HTML for test count extraction

- Add JSON reporter to Playwright test runs
- Generate report.json alongside HTML reports
- Store JSON report in playwright-report directory
- This enables accurate test count extraction from CI artifacts

The HTML reports alone don't contain easily extractable test statistics
as they use a React app with dynamically loaded data. JSON reports
provide direct access to test counts.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: correct JSON reporter syntax for Playwright tests

- Use proper syntax for JSON reporter with outputFile option
- Run separate commands for HTML and JSON report merging
- Specify output path directly in reporter configuration
- Ensures report.json is created in playwright-report directory

This fixes the "No such file or directory" error when trying to move
report.json file, as it wasn't being created in the first place.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Revert "fix: correct JSON reporter syntax for Playwright tests"

This reverts commit 605d7cc1e2.

* fix: use correct Playwright reporter syntax with comma-separated list

- Use --reporter=html,json syntax (comma-separated, not space)
- Move test-results.json to playwright-report/report.json after generation
- Remove incorrect PLAYWRIGHT_JSON_OUTPUT_NAME env variable
- Add || true to prevent failure if JSON file doesn't exist

The JSON reporter outputs to test-results.json by default when using
the comma-separated reporter list syntax.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: improve test count extraction reliability in CI

- Use separate --reporter flags for list, html, and json
- Set PLAYWRIGHT_JSON_OUTPUT_NAME env var to specify JSON output path
- Run HTML and JSON report generation separately for merged reports
- Ensures report.json is created in playwright-report directory

The combined reporter syntax wasn't creating the JSON file properly.
Using separate reporter flags with env var ensures JSON is generated.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update scripts/cicd/pr-playwright-deploy-and-comment.sh

Co-authored-by: Alexander Brown <drjkl@comfy.org>

* refactor: convert extraction script to TypeScript and use tsx

- Convert extract-playwright-counts.mjs to TypeScript (.ts)
- Add proper TypeScript types for better type safety
- Use tsx for execution instead of node
- Auto-install tsx in CI if not available
- Better alignment with the TypeScript codebase

This provides better type safety and consistency with the rest of
the codebase while maintaining the same functionality.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(pr-playwright-deploy-and-comment.sh): move tsx installation check to the beginning of the script for better organization and efficiency

* [auto-fix] Apply ESLint and Prettier fixes

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
Co-authored-by: GitHub Action <action@github.com>
2025-09-13 01:18:18 -07:00
snomiao
6fbd692370 fix: Split PR deployment workflow for forked vs non-forked repos (#5425)
* [fix] Consolidate Playwright workflow jobs to fix missing deployment links

The issue in PR #5298 was caused by missing deployment-info artifact
creation. The deploy-reports job was deploying to Cloudflare but wasn't
creating the deployment-info-* artifacts that comment-tests-completed
job expected to download.

This change consolidates the deployment and commenting into a single job,
eliminating the artifact dependency and ensuring links are always available.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Split PR deployment workflow for forked vs non-forked repos

- Extract deployment logic to reusable script (scripts/cicd/pr-playwright-deploy-and-comment.sh)
- Non-forked PRs: Use direct pull_request event in test-ui.yaml (faster)
- Forked PRs: Use workflow_run in pr-playwright-deploy.yaml (handles permissions)
- Add starting comment for both forked and non-forked PRs
- Make Cloudflare tokens optional for starting status comments

* refactor: Simplify PR deployment workflow and script

- Consolidate workflow into single job with clearer structure
- Reduce script from 200+ to ~140 lines
- Simplify deployment retry logic and comment generation
- Remove redundant checks and unnecessary complexity

* fix: Add debugging and wrangler installation to deployment script

- Add debug output to identify missing reports
- Install wrangler if not available
- Show deployment attempts and failures
- Log available reports before deployment

* chore: Trigger CI to test deployment workflow

* fix: Fix browser artifact name mismatch in deployment script

- Use dot notation (0.5x) for artifact names as Playwright creates them
- Convert to dash notation (0-5x) for Cloudflare project names
- Properly handle browser name display in comments

* refactor: Convert deployment script to POSIX sh for better compatibility

- Replace bash arrays with space-separated strings
- Use while loops instead of bash-specific for syntax
- Remove bash-specific string manipulation features
- Replace local variables (not required in functions)
- Ensure compatibility with standard /bin/sh

* fix: Fix deployment script output to properly capture URLs

- Redirect debug messages to stderr
- Only output URL to stdout for proper capture
- This fixes the missing deployment links in PR comments

* fix: Add input validation to prevent command injection

- Validate PR number is numeric only
- Sanitize branch name at script start
- Validate status parameter values
- Use pre-sanitized branch throughout script
- Addresses high-severity security issue from PR review

* fix: Add null checks and logging to workflow condition

- Add explicit null checks for head_repository and repository
- Add debug logging to help diagnose workflow trigger issues
- Prevents potential failures from undefined repository objects
- Addresses medium-severity issue from PR review

* fix: Pin wrangler to major version 4 with error handling

- Pin wrangler to major version 4 (^4.0.0) for stability
- Add error handling if wrangler installation fails
- Return 'failed' status if installation fails
- Addresses dependency management issue from PR review

* perf: Implement parallel deployments to reduce CI time

- Deploy all browser reports in parallel using background processes
- Use temporary directory to collect deployment results
- Wait for all deployments to complete before generating comment
- Maintains result order for consistent output
- Significantly reduces deployment time from sequential to parallel execution

* fix: Use specific comment ID for updates instead of edit-last

- Use GitHub API to find exact comment ID
- Update specific comment by ID to avoid editing wrong comment
- Prevents race conditions if user posts between finding and editing
- More reliable comment updates

* fix(workflows/test-ui.yaml): change condition to always run deploy job for pull requests to ensure deployment consistency

* fix(workflows/test-ui.yaml): change condition to always run deploy job for pull requests to ensure deployment consistency

* fix(pr-playwright-deploy-and-comment.sh): remove npx prefix from wrangler command for consistency and simplicity

* fix(pr-playwright-deploy-and-comment.sh): remove npx prefix from wrangler command for consistency and simplicity

* Update scripts/cicd/pr-playwright-deploy-and-comment.sh

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(pr-playwright-deploy-and-comment.sh): improve regex for URL extraction to include valid characters and ensure correct URL format

* chore(pr-playwright-deploy-and-comment.sh): move wrangler installation to the beginning of the script to avoid redundancy and improve efficiency

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-09-08 23:11:26 -07:00