Compare commits

...

12 Commits

Author SHA1 Message Date
GitHub Action
62460d10b6 [automated] Apply ESLint and Oxfmt fixes 2026-03-28 23:52:26 +00:00
bymyself
c42fa2a861 fix(test): rewrite cloud tests for unauthenticated CI environment
Cloud build has an auth guard that redirects to /cloud/login when
no Firebase auth is configured (CI has no auth). Previous tests
used comfyPageFixture which waits for the graph editor to load,
causing timeouts since the auth guard redirects before that.

New tests verify:
1. Cloud build redirects to /cloud/login (route only exists in
   cloud distribution, tree-shaken in OSS)
2. Cloud login page renders sign-in options

Uses raw Playwright test fixture instead of comfyPageFixture since
we don't need the full graph editor setup.
2026-03-28 16:49:31 -07:00
bymyself
dd8595e01a fix(ci): skip Nx cache for cloud build to avoid serving OSS dist
Nx cached the first OSS build and returned it for the cloud build
since env vars aren't part of the cache key. This caused cloud
tests to run against the OSS build where __DISTRIBUTION__ is
'localhost' instead of 'cloud', making cloud-only UI elements
(subscribe button, hidden bottom panel) not render correctly.
2026-03-28 16:39:06 -07:00
GitHub Action
f466574505 [automated] Apply ESLint and Oxfmt fixes 2026-03-28 23:12:01 +00:00
bymyself
703283eb58 ci: add cloud project to e2e test matrix
Addresses review feedback:
https://github.com/Comfy-Org/ComfyUI_frontend/pull/10546#discussion_r3003539443
2026-03-28 16:08:30 -07:00
bymyself
3f0be1b9c2 fix: use tag annotations and document free-tier precondition
Addresses review feedback:
https://github.com/Comfy-Org/ComfyUI_frontend/pull/10546#discussion_r3003514189
https://github.com/Comfy-Org/ComfyUI_frontend/pull/10546#discussion_r2999007426
2026-03-28 16:06:40 -07:00
GitHub Action
3a57c4d394 [automated] Apply ESLint and Oxfmt fixes 2026-03-28 16:06:40 -07:00
bymyself
9c6fe99f0b fix: align cloud test name with toBeAttached assertion 2026-03-28 16:06:40 -07:00
bymyself
8152ef2baa fix: address CodeRabbit review — remove .or(body) fallback, use TestIds, stricter assertions 2026-03-28 16:06:40 -07:00
GitHub Action
409e9cdd8c [automated] Apply ESLint and Oxfmt fixes 2026-03-28 16:06:40 -07:00
bymyself
7f3c15ba24 test(infra): add cloud Playwright project with @cloud/@oss tagging
- Add 'cloud' Playwright project for testing DISTRIBUTION=cloud builds
- Add build:cloud npm script (DISTRIBUTION=cloud vite build)
- Cloud project uses grep: /@cloud/ to run cloud-only tests
- Default chromium project uses grepInvert to exclude @cloud tests
- Add 2 example cloud-only tests (subscribe button, bottom panel toggle)
- Runtime toggle investigation: NOT feasible (breaks tree-shaking)
  → separate build approach chosen

Convention:
  test('feature works @cloud', ...) — cloud-only
  test('feature works @oss', ...) — OSS-only
  test('feature works', ...) — runs in both

Part of: Test Coverage Q2 Overhaul (UTIL-07)
2026-03-28 16:06:40 -07:00
Christian Byrne
8da4640a76 docs: add assertion best practices to Playwright guide (#10663)
## Summary

Document custom expect messages and soft assertions as Playwright best
practices.

## Changes

- **What**: Added "Assertion Best Practices" section to
`docs/guidance/playwright.md` covering custom messages, `expect.soft()`,
and guidelines for when to use each.

## Review Focus

Documentation-only change — no code impact.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10663-docs-add-assertion-best-practices-to-Playwright-guide-3316d73d365081309d83f95bb9b86fe1)
by [Unito](https://www.unito.io)

---------

Co-authored-by: GitHub Action <action@github.com>
2026-03-28 16:00:07 -07:00
6 changed files with 81 additions and 4 deletions

View File

@@ -33,6 +33,20 @@ jobs:
path: dist/
retention-days: 1
# Build cloud distribution for @cloud tagged tests
# NX_SKIP_NX_CACHE=true is required because `nx build` was already run
# for the OSS distribution above. Without skipping cache, Nx returns
# the cached OSS build since env vars aren't part of the cache key.
- name: Build cloud frontend
run: NX_SKIP_NX_CACHE=true pnpm build:cloud
- name: Upload cloud frontend
uses: actions/upload-artifact@v6
with:
name: frontend-dist-cloud
path: dist/
retention-days: 1
# Sharded chromium tests
playwright-tests-chromium-sharded:
needs: setup
@@ -97,14 +111,14 @@ jobs:
strategy:
fail-fast: false
matrix:
browser: [chromium-2x, chromium-0.5x, mobile-chrome]
browser: [chromium-2x, chromium-0.5x, mobile-chrome, cloud]
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Download built frontend
uses: actions/download-artifact@v7
with:
name: frontend-dist
name: ${{ matrix.browser == 'cloud' && 'frontend-dist-cloud' || 'frontend-dist' }}
path: dist/
- name: Start ComfyUI server

View File

@@ -51,7 +51,8 @@ export const TestIds = {
topbar: {
queueButton: 'queue-button',
queueModeMenuTrigger: 'queue-mode-menu-trigger',
saveButton: 'save-workflow-button'
saveButton: 'save-workflow-button',
subscribeButton: 'topbar-subscribe-button'
},
nodeLibrary: {
bookmarksSection: 'node-library-bookmarks-section'

View File

@@ -0,0 +1,29 @@
import { expect, test } from '@playwright/test'
/**
* Cloud distribution E2E tests.
*
* These tests run against the cloud build (DISTRIBUTION=cloud) and verify
* that cloud-specific behavior is present. In CI, no Firebase auth is
* configured, so the auth guard redirects to /cloud/login. The tests
* verify the cloud build loaded correctly by checking for cloud-only
* routes and elements.
*/
test.describe('Cloud distribution UI', { tag: '@cloud' }, () => {
test('cloud build redirects unauthenticated users to login', async ({
page
}) => {
await page.goto('http://localhost:8188')
// Cloud build has an auth guard that redirects to /cloud/login.
// This route only exists in the cloud distribution — it's tree-shaken
// in the OSS build. Its presence confirms the cloud build is active.
await expect(page).toHaveURL(/\/cloud\/login/, { timeout: 10_000 })
})
test('cloud login page renders sign-in options', async ({ page }) => {
await page.goto('http://localhost:8188')
await expect(page).toHaveURL(/\/cloud\/login/, { timeout: 10_000 })
// Verify cloud-specific login UI is rendered
await expect(page.getByRole('button', { name: /google/i })).toBeVisible()
})
})

View File

@@ -75,6 +75,30 @@ await page.evaluate(() => {
// Keep app.extensionManager typed as ExtensionManager, not WorkspaceStore
```
## Assertion Best Practices
When a test depends on an invariant unrelated to what it's actually testing (e.g. asserting a node has 4 widgets before testing node movement), always assert that invariant explicitly — don't leave it unchecked. Use a custom message or `expect.soft()` rather than a bare `expect`, so failures point to the broken assumption instead of producing a confusing error downstream.
```typescript
// ✅ Custom message on an unrelated precondition — clear signal when the invariant breaks
expect(node.widgets, 'Widget count changed — update test fixture').toHaveLength(
4
)
await node.move(100, 200)
// ✅ Soft assertion — verifies multiple invariants without stopping the test early
expect.soft(menuItem1).toBeVisible()
expect.soft(menuItem2).toBeVisible()
expect.soft(menuItem3).toBeVisible()
// ❌ Bare expect on a precondition — no context when it fails
expect(node.widgets).toHaveLength(4)
```
- Use custom messages (`expect(x, 'reason')`) for precondition checks unrelated to the test's purpose
- Use `expect.soft()` when you want to verify multiple invariants without aborting on the first failure
- Prefer Playwright's built-in message parameter over custom error classes
## Test Tags
Tags are respected by config:

View File

@@ -8,6 +8,7 @@
"repository": "https://github.com/Comfy-Org/ComfyUI_frontend",
"type": "module",
"scripts": {
"build:cloud": "cross-env DISTRIBUTION=cloud NODE_OPTIONS='--max-old-space-size=8192' nx build",
"build:desktop": "nx build @comfyorg/desktop-ui",
"build-storybook": "storybook build",
"build:types": "nx build --config vite.types.config.mts && node scripts/prepare-types.js",

View File

@@ -36,7 +36,7 @@ export default defineConfig({
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
timeout: 15000,
grepInvert: /@mobile|@perf|@audit/ // Run all tests except those tagged with @mobile, @perf, or @audit
grepInvert: /@mobile|@perf|@audit|@cloud/ // Run all tests except those tagged with @mobile, @perf, @audit, or @cloud
},
{
@@ -85,6 +85,14 @@ export default defineConfig({
// use: { ...devices['Desktop Safari'] },
// },
{
name: 'cloud',
use: { ...devices['Desktop Chrome'] },
timeout: 15000,
grep: /@cloud/, // Run only tests tagged with @cloud
grepInvert: /@oss/ // Exclude tests tagged with @oss
},
/* Test against mobile viewports. */
{
name: 'mobile-chrome',