- Renamed parameter from skip_build to include_build_step
- Changed default from 'false' to 'false' (building now opt-in)
- Updated all workflow usages to explicitly set include_build_step: 'true' where building is needed
- Removed explicit parameter for workflows that don't need building
As suggested by @DrJKL in PR review
The PLAYWRIGHT_BLOB_OUTPUT_DIR was set to ../blob-report which was correct
when working-directory was ComfyUI_frontend, but now that we removed the
working-directory, the path should be ./blob-report to output in the repo root.
This refactoring improves the CI/CD workflow structure by:
- Split the monolithic setup-frontend action into two focused actions:
- setup-frontend: Now only handles frontend dependency installation and building
- setup-comfyui-server: New action for ComfyUI server setup and launch
- Simplified workflow structure with better separation of concerns:
- Frontend and server setup are now independent and reusable
- Each action has clearer responsibilities and inputs
- Removed duplicate setup code across workflows
- Improved tests-ci.yaml workflow:
- Uses cache/save and cache/restore for the entire workspace
- Test jobs now restore cached build instead of rebuilding
- Reduced redundant setup steps in each test shard
- Better parallelization with faster test execution
- Updated all locale update workflows to use new action structure
- Made setup-playwright a standalone reusable action
Benefits:
- Faster CI runs by reducing redundant builds
- More maintainable with DRY principle
- Easier to debug individual components
- Better action reusability across workflows
## Problem
Our CI tests were experiencing non-reproducible results where:
- Tests would pass on a PR initially
- The same PR would fail later when main HEAD changed
- Screenshot comparisons showed excessive differences between expected
vs actual
- Blake identified: *"tests are not reproducible inside a branch - they
change every time main HEAD changes"*
## Root Cause
The issue was caused by **explicit `repository` parameters** in our
`actions/checkout` steps:
```yaml
- name: Checkout ComfyUI_frontend
uses: actions/checkout@v5
with:
repository: 'Comfy-Org/ComfyUI_frontend' # ← This was the problem!
path: 'ComfyUI_frontend'
```
According to GitHub Actions documentation:
> **When checking out the repository that triggered a workflow, `ref`
defaults to the reference or SHA for that event. Otherwise, uses the
default branch.**
When you specify an explicit `repository` parameter (even if it's the
same repo), GitHub Actions treats it as "otherwise" and defaults to the
**main branch** instead of using the **PR context**.
## The Fix
Remove the explicit `repository` parameter when checking out the same
repository:
```yaml
- name: Checkout ComfyUI_frontend
uses: actions/checkout@v5
with:
path: 'ComfyUI_frontend' # No repository parameter = uses PR context
```
## Changes Made
- ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from setup job
checkout
- ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from
merge-reports job checkout
- ✅ Updated setup-frontend action to use `actions/checkout@v5` for
consistency
- ✅ Simplified workflow by removing unnecessary `ref` and `fetch-depth`
parameters
## How This Fixes the Problem
**Before:**
- Setup job checked out main branch (due to explicit repository)
- Tests ran PR code against main branch snapshots
- Results varied based on what was in main at the time
**After:**
- Setup job checks out PR merge commit (natural PR context)
- Tests run PR code against PR snapshots
- Results are consistent and reproducible
## Why It Worked Before (Sometimes)
The explicit `repository` parameter has been there for a long time, but
the issue became more apparent recently due to:
1. GitHub Actions behavior changes over time
2. Increased frequency of main branch updates
3. More sensitive screenshot comparison tests
4. Complex cache/restore workflow where timing mattered
The fix ensures deterministic behavior regardless of GitHub's internal
changes.
## Testing
This change makes the CI behavior explicit and predictable:
- ✅ PR tests will always use PR context
- ✅ Push tests will always use pushed commit
- ✅ No dependency on GitHub's default behavior interpretation
- ✅ Simplified workflow with fewer moving parts
Resolves the issues described in `.github/workflows/problem.log`.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-5950-Fix-CI-Remove-explicit-repository-parameter-causing-non-reproducible-test-results-2846d73d36508159a848c4a2e14a0fb1)
by [Unito](https://www.unito.io)
Co-authored-by: Alexander Brown <drjkl@comfy.org>
This PR introduces a reusable composite action for Playwright setup to
reduce duplication across workflows.
## Changes
- Created `.github/actions/setup-playwright/action.yml` composite action
that:
- Detects or uses provided Playwright version
- Caches Playwright browsers with intelligent cache keys
- Installs browsers only when cache miss occurs
- Installs OS dependencies when cache hit occurs
## Technical Details
- **Important:** The composite action requires `shell: bash` for all
`run` steps as per [GitHub Actions requirements for composite
actions](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action#creating-an-action-metadata-file).
This is a mandatory field for composite actions, unlike regular workflow
steps.
- Updated workflow paths to account for repository checkout locations
(some workflows checkout to subdirectories like `ComfyUI_frontend/`)
- Uses conditional caching to avoid redundant browser installations
## Benefits
- Reduces code duplication across 6 workflow files
- Centralizes Playwright caching logic
- Consistent browser setup across all workflows
- Easier maintenance and updates
- Faster CI runs through intelligent caching
## Affected Workflows
- `.github/workflows/test-ui.yaml` (2 uses)
- `.github/workflows/i18n-custom-nodes.yaml`
- `.github/workflows/i18n-node-defs.yaml`
- `.github/workflows/i18n.yaml`
- `.github/workflows/test-browser-exp.yaml`
---------
Co-authored-by: GitHub Action <action@github.com>
