From 87f5480462e5f5933847f4cf067ed1b7ade38cd4 Mon Sep 17 00:00:00 2001
From: snomiao <snomiao@gmail.com>
Date: Thu, 9 Oct 2025 05:11:53 +0900
Subject: [PATCH] Fix CI: Remove explicit repository parameter causing
 non-reproducible test results (#5950)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Problem

Our CI tests were experiencing non-reproducible results where:
- Tests would pass on a PR initially
- The same PR would fail later when main HEAD changed
- Screenshot comparisons showed excessive differences between expected
vs actual
- Blake identified: *"tests are not reproducible inside a branch - they
change every time main HEAD changes"*

## Root Cause

The issue was caused by **explicit `repository` parameters** in our
`actions/checkout` steps:

```yaml
- name: Checkout ComfyUI_frontend
  uses: actions/checkout@v5
  with:
    repository: 'Comfy-Org/ComfyUI_frontend'  # ← This was the problem!
    path: 'ComfyUI_frontend'
```

According to GitHub Actions documentation:

> **When checking out the repository that triggered a workflow, `ref`
defaults to the reference or SHA for that event. Otherwise, uses the
default branch.**

When you specify an explicit `repository` parameter (even if it's the
same repo), GitHub Actions treats it as "otherwise" and defaults to the
**main branch** instead of using the **PR context**.

## The Fix

Remove the explicit `repository` parameter when checking out the same
repository:

```yaml
- name: Checkout ComfyUI_frontend
  uses: actions/checkout@v5
  with:
    path: 'ComfyUI_frontend'  # No repository parameter = uses PR context
```

## Changes Made

- ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from setup job
checkout
- ✅ Removed `repository: 'Comfy-Org/ComfyUI_frontend'` from
merge-reports job checkout
- ✅ Updated setup-frontend action to use `actions/checkout@v5` for
consistency
- ✅ Simplified workflow by removing unnecessary `ref` and `fetch-depth`
parameters

## How This Fixes the Problem

**Before:**
- Setup job checked out main branch (due to explicit repository)
- Tests ran PR code against main branch snapshots
- Results varied based on what was in main at the time

**After:**
- Setup job checks out PR merge commit (natural PR context)
- Tests run PR code against PR snapshots
- Results are consistent and reproducible

## Why It Worked Before (Sometimes)

The explicit `repository` parameter has been there for a long time, but
the issue became more apparent recently due to:
1. GitHub Actions behavior changes over time
2. Increased frequency of main branch updates
3. More sensitive screenshot comparison tests
4. Complex cache/restore workflow where timing mattered

The fix ensures deterministic behavior regardless of GitHub's internal
changes.

## Testing

This change makes the CI behavior explicit and predictable:
- ✅ PR tests will always use PR context
- ✅ Push tests will always use pushed commit
- ✅ No dependency on GitHub's default behavior interpretation
- ✅ Simplified workflow with fewer moving parts

Resolves the issues described in `.github/workflows/problem.log`.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-5950-Fix-CI-Remove-explicit-repository-parameter-causing-non-reproducible-test-results-2846d73d36508159a848c4a2e14a0fb1)
by [Unito](https://www.unito.io)

Co-authored-by: Alexander Brown <drjkl@comfy.org>
---
 .github/actions/setup-frontend/action.yml | 5 ++---
 .github/workflows/tests-ci.yaml           | 2 --
 2 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/.github/actions/setup-frontend/action.yml b/.github/actions/setup-frontend/action.yml
index 3ebc12eb3..125eaa92f 100644
--- a/.github/actions/setup-frontend/action.yml
+++ b/.github/actions/setup-frontend/action.yml
@@ -9,15 +9,14 @@ runs:
   using: 'composite'
   steps:
     - name: Checkout ComfyUI
-      uses: actions/checkout@v4
+      uses: actions/checkout@v5
       with:
         repository: 'comfyanonymous/ComfyUI'
         path: 'ComfyUI'
 
     - name: Checkout ComfyUI_frontend
-      uses: actions/checkout@v4
+      uses: actions/checkout@v5
       with:
-        repository: 'Comfy-Org/ComfyUI_frontend'
         path: 'ComfyUI_frontend'
 
     - name: Copy ComfyUI_devtools from frontend repo
diff --git a/.github/workflows/tests-ci.yaml b/.github/workflows/tests-ci.yaml
index 1e069ea11..1b9a06db7 100644
--- a/.github/workflows/tests-ci.yaml
+++ b/.github/workflows/tests-ci.yaml
@@ -23,7 +23,6 @@ jobs:
       - name: Checkout ComfyUI_frontend
         uses: actions/checkout@v5
         with:
-          repository: 'Comfy-Org/ComfyUI_frontend'
           path: 'ComfyUI_frontend'
 
       - name: Copy ComfyUI_devtools from frontend repo
@@ -217,7 +216,6 @@ jobs:
       - name: Checkout ComfyUI_frontend
         uses: actions/checkout@v5
         with:
-          repository: 'Comfy-Org/ComfyUI_frontend'
           path: 'ComfyUI_frontend'
 
       - name: Install pnpm