Replace claude --print with codex exec for cheaper QA runs.
Uses codex-mini-latest model ($1.50/$6 vs Sonnet $3/$15).
Uses existing OPENAI_API_KEY secret (no new secrets needed).
- Replace saveVideo config (didn't produce video) with explicit
playwright-cli video-start/video-stop commands in QA prompt
- Remove apt-get install ffmpeg step (pre-installed on GH runners)
- Switch video review model from gpt-4o to gpt-4.1-mini
- Enable saveVideo in playwright-cli config for real video recording
- Replace screenshot stitching with webm→mp4 conversion
- Move video review step before deploy so reports are included
- Add GPT video review reports inline on the Cloudflare Pages site
- Each video card now has expandable "GPT Video Review" section
- Set .playwright/cli.config.json with outputDir pointing to screenshots/
- This way bare 'playwright-cli screenshot' auto-saves to the right place
- Create screenshot directory before Claude runs (don't rely on Claude)
- Collect step now searches working directory for stray PNGs
- Simplified prompt: no --filename needed, just 'playwright-cli screenshot'
Screenshots were saved to artifact root but stitch looked in frames/.
Now: prompt tells Claude to save to screenshots/ dir with numbered names,
collect step consolidates PNGs there, stitch step globs from screenshots/.
Removed video-start/video-stop (Claude doesn't use them).
- Add playwright-cli config with outputDir and saveVideo
- Use video-start/video-stop instead of relying on screenshot frames
- Add fallback artifact collection from .playwright-cli/ default dir
- Simplify prompts to focus on video recording workflow
The escaped \$QA_ARTIFACTS in the heredoc produced literal text
'$QA_ARTIFACTS' in the prompt. Claude's Bash tool didn't reliably
expand this env var, so no screenshots or reports were saved.
Remove the escapes so the heredoc expands the variable to the actual
path (e.g. /home/runner/work/_temp/qa-artifacts).
Backtick-wrapped playwright-cli examples in the unquoted heredoc were
being interpreted as bash command substitution, producing empty prompts.
Replace backtick syntax with plain "Run:" prefixed commands.
- Remove all Xvfb/ffmpeg screen recording infrastructure from qa job
(captured blank display since playwright-cli runs headless)
- Add screenshot instructions to QA prompts: Claude saves sequential
frames to $QA_ARTIFACTS/frames/ after every interaction
- Stitch screenshots into video via ffmpeg in report job (2fps)
- Merge video-review job into report job (4 jobs → 3 jobs)
- Unified PR comment with video links + video review in <details> collapse
- Clean up stale QA_VIDEO_REVIEW_COMMENT markers from prior runs
Move extra_server_params input to env var to prevent shell injection
from untrusted input. Replace wait-for-it pip dependency with a
cross-platform curl polling loop.
Add Claude Code skills and a label-triggered QA workflow:
- .claude/skills/comfy-qa/SKILL.md: 12-category QA test plan using
playwright-cli for browser automation
- .github/workflows/pr-qa.yaml: CI workflow triggered by qa-changes
(focused, Linux) or qa-full (3-OS matrix) labels. Records screen via
ffmpeg, runs Claude CLI with playwright-cli, deploys video gallery to
Cloudflare Pages, posts PR comment with GIF thumbnails, and runs
OpenAI vision-based video review
- scripts/qa-video-review.ts: frame extraction + GPT-4o analysis
- scripts/qa-video-review.test.ts: unit tests for video review
- knip.config.ts: resolve knip errors for ingest-types package
*PR Created by the Glary-Bot Agent*
---
## Summary
- Replace all `as unknown as Type` assertions in 59 unit test files with
type-safe `@total-typescript/shoehorn` functions
- Use `fromPartial<Type>()` for partial mock objects where deep-partial
type-checks (21 files)
- Use `fromAny<Type>()` for fundamentally incompatible types: null,
undefined, primitives, variables, class expressions, and mocks with
test-specific extra properties that `PartialDeepObject` rejects
(remaining files)
- All explicit type parameters preserved so TypeScript return types are
correct
- Browser test `.spec.ts` files excluded (shoehorn unavailable in
`page.evaluate` browser context)
## Verification
- `pnpm typecheck` ✅
- `pnpm lint` ✅
- `pnpm format` ✅
- Pre-commit hooks passed (format + oxlint + eslint + typecheck)
- Migrated test files verified passing (ran representative subset)
- No test behavior changes — only type assertion syntax changed
- No UI changes — screenshots not applicable
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10761-test-migrate-as-unknown-as-to-total-typescript-shoehorn-3336d73d365081f6b8adc44db5dcc380)
by [Unito](https://www.unito.io)
---------
Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
Co-authored-by: Amp <amp@ampcode.com>
*PR Created by the Glary-Bot Agent*
---
## Summary
Fixes the `Bulk context menu shows when multiple assets selected` test
that is failing on main.
**Root cause — two issues:**
1. `click({ modifiers: ['ControlOrMeta'] })` does not fire `keydown`
events that VueUse's `useKeyModifier('Control')` tracks (used in
`useAssetSelection.ts`). Multi-select silently fails because the
composable never sees the Control key pressed. Fix: use
`keyboard.down('Control')` / `keyboard.up('Control')` around the click.
2. `click({ button: 'right' })` can be intercepted by canvas overlays
(documented gotcha in `browser_tests/AGENTS.md`). Fix: use
`dispatchEvent('contextmenu', { bubbles: true, cancelable: true })`
which bypasses overlay interception.
Also removed the `toPass()` retry wrapper since the root causes are now
addressed directly.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10762-fix-test-fix-bulk-context-menu-test-using-correct-Playwright-patterns-3346d73d3650811c843ee4a39d3ab305)
by [Unito](https://www.unito.io)
---------
Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
## What changed
Added a runtime-safe `#e2e/*` alias for `browser_tests`, updated the
browser test docs, and migrated a representative fixture/spec import
path to the new convention.
## Why
`@/*` only covers `src/`, so browser test imports were falling back to
deep relative paths. `#e2e/*` resolves in both Node/Playwright runtime
and TypeScript.
## Validation
- `pnpm format`
- `pnpm typecheck:browser`
- `pnpm exec playwright test browser_tests/tests/actionbar.spec.ts
--list`
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10735-test-add-runtime-safe-browser_tests-alias-3336d73d36508122b253cb36a4ead1c1)
by [Unito](https://www.unito.io)
---------
Co-authored-by: Alexander Brown <drjkl@comfy.org>
## Summary
- Closing an inactive workflow tab and clicking "Save" overwrites that
workflow with the **active** tab's content, causing permanent data loss
- `saveWorkflow()` and `saveWorkflowAs()` call `checkState()` which
serializes `app.rootGraph` (the active canvas) into the inactive
workflow's `changeTracker.activeState`
- Guard `checkState()` to only run when the workflow being saved is the
active one — in both `saveWorkflow` and `saveWorkflowAs`
## Linked Issues
- Fixes https://github.com/Comfy-Org/ComfyUI/issues/13230
## Root Cause
PR #9137 (commit `9fb93a5b0`, v1.41.7) added
`workflow.changeTracker?.checkState()` inside `saveWorkflow()` and
`saveWorkflowAs()`. `checkState()` always serializes `app.rootGraph` —
the graph on the canvas. When called on an inactive tab's change
tracker, it captures the active tab's data instead.
## Test plan
- [x] E2E: "Closing an inactive tab with save preserves its own content"
— persisted workflow B with added node, close while A is active, re-open
and verify
- [x] E2E: "Closing an inactive unsaved tab with save preserves its own
content" — temporary workflow B with added node, close while A is
active, save-as with filename, re-open and verify
- [x] Manual: open A and B, edit B, switch to A, close B tab, click
Save, re-open B — content should be B's not A's
## What changed
Removed stale `tests-ui` configuration and documentation references from
the repo.
## Why
`tests-ui/` no longer exists, but the repo still carried:
- a dead `@tests-ui/*` tsconfig path
- stale `tests-ui/**/*` include
- a Vite watch ignore for a missing directory
- documentation examples that still referenced the old path
## Validation
- `pnpm format:check`
- `pnpm typecheck`
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10736-chore-remove-stale-tests-ui-config-3336d73d3650814a98bedfc113b6eb9b)
by [Unito](https://www.unito.io)
## Summary
replacement for https://github.com/Comfy-Org/ComfyUI_frontend/pull/9201
the first commit squashed
https://github.com/Comfy-Org/ComfyUI_frontend/pull/9201 and fixed
conflict.
the second commit change needed by:
- Enable GLSL live preview on SubgraphNodes by detecting the inner
GLSLShader and rendering its preview directly on the parent SubgraphNode
- Previously, SubgraphNodes containing a GLSLShader showed no live
preview at all To achieve this:
- Read shader source, uniform values, and renderer config from the inner
GLSLShader's widgets
- Trace IMAGE inputs through the subgraph boundary so the inner shader
can use images connected to the SubgraphNode's outer inputs
- Set preview output using the inner node's locator ID so the promoted
preview system picks it up on the SubgraphNode
- Extract setNodePreviewsByLocatorId from nodeOutputStore to support
setting previews by locator ID directly
- Fix graphId to use rootGraph.id for widget store lookups (was using
graph.id, which broke lookups for nodes inside subgraphs)
- Read uniform values from connected upstream nodes, not just local
widgets
- Fix blob URL lifecycle: use the store's
createSharedObjectUrl/releaseSharedObjectUrl reference-counting system
instead of manual revoke, preventing leaks on composable re-creation
## Screenshot
https://github.com/user-attachments/assets/9623fa32-de39-4a3a-b8b3-28688851390b
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10349-Feat-glsl-live-preview-3296d73d3650814b83aef52ab1962a77)
by [Unito](https://www.unito.io)
## Summary
Extract auth-routing logic (`getAuthHeaderOrThrow`,
`getFirebaseAuthHeaderOrThrow`) from `workspaceApi.ts` into
`authStore.ts`, eliminating a layering violation where the workspace API
re-implemented auth header resolution.
## Changes
- **What**: Moved `getAuthHeaderOrThrow` and
`getFirebaseAuthHeaderOrThrow` from `workspaceApi.ts` to `authStore.ts`.
`workspaceApi.ts` now calls through `useAuthStore()` instead of
re-implementing token resolution. Added tests for the new methods in
`authStore.test.ts`. Updated `authStoreMock.ts` with the new methods.
- **Files**: 4 files changed
## Review Focus
- The `getAuthHeaderOrThrow` / `getFirebaseAuthHeaderOrThrow` methods
throw `AuthStoreError` (auth domain error) — callers in workspace can
catch and re-wrap if needed
- `workspaceApi.ts` is simplified by ~19 lines
## Stack
PR 2/5: #10483 → **→ This PR** → #10485 → #10486 → #10487
## What
- Add `include: ['src/**/*.{ts,vue}']` to vitest coverage config so ALL
source files appear in reports (previously only imported files showed
up)
- Add `lcov` reporter for CI integration and VS Code coverage gutter
- Add `exclude` patterns for test files, locales, litegraph, assets,
declarations, stories
- Add `test:coverage` npm script
## Why
Coverage reports currently only show files that are imported during test
runs. Adding the `include` pattern reveals the true gap — files with
zero coverage that were previously invisible. The lcov reporter enables
IDE integration and future CI coverage comments (Codecov/Coveralls).
## Testing
`npx tsc --noEmit` passes. No behavioral changes — this only affects
coverage reporting configuration.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10575-config-add-vitest-coverage-include-pattern-lcov-reporter-32f6d73d365081c8b59ad2316dd2b198)
by [Unito](https://www.unito.io)
## Summary
Extract `makeMatcher` and `comfyExpect` from `ComfyPage.ts` into the
standalone `browser_tests/fixtures/utils/customMatchers.ts` module,
reducing the page-object file by ~50 lines.
## Changes
- **What**: Removed duplicate `makeMatcher`/`comfyExpect` definitions
from `ComfyPage.ts`; the canonical implementation now lives in
`customMatchers.ts`. A backward-compatible re-export keeps all existing
imports working.
## Review Focus
- The re-export ensures `import { comfyExpect } from
'../fixtures/ComfyPage'` continues to resolve for all ~25 spec files.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10652-refactor-extract-comfyExpect-and-makeMatcher-from-ComfyPage-3316d73d365081bf8e7cd7fa324bf9a6)
by [Unito](https://www.unito.io)
---------
Co-authored-by: Alexander Brown <drjkl@comfy.org>
Co-authored-by: GitHub Action <action@github.com>
## Summary
Adds Playwright E2E tests for the QueueClearHistoryDialog component.
## Tests added
- Dialog opens from queue panel history actions menu
- Dialog shows confirmation message with title, description, and assets
note
- Cancel button closes dialog without clearing history
- Close (X) button closes dialog without clearing history
- Confirm clear action triggers queue history clear API call
- Dialog state resets properly after close/reopen
## Task
Part of Test Coverage Q2 Overhaul (DLG-02).
## Conventions
- Uses Vue nodes with new menu enabled (`Comfy.UseNewMenu: 'Top'`)
- Tests read as user stories
- No full-page screenshots
- Proper waits, no sleeps
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10586-test-add-QueueClearHistoryDialog-E2E-tests-DLG-02-3306d73d36508174a07bd9782340a0f7)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Summary
Comprehensive Playwright E2E tests for the properties panel (right
sidebar).
Part of the **Test Coverage Q2 Overhaul** initiative (Phase 2: PNL-01).
## What's included
- **PropertiesPanelHelper** page object in `browser_tests/helpers/` —
locators + action methods for all panel elements
- **35 test cases** covering:
- Open/close via actionbar toggle
- Workflow Overview (no selection): tabs, title, nodes list, global
settings
- Single node selection: title, parameters, info tab, widgets display
- Multi-node selection: item count, node listing, hidden Info tab
- Title editing: pencil icon, edit mode, rename, visibility rules
- Search filtering: query, clear, empty state
- Settings tab: Normal/Bypass/Mute state, color swatches, pinned toggle
- Selection transitions: no-selection ↔ single ↔ multi
- Nodes tab: list all, search filter
- Tab label changes based on selection count
- **Errors tab scaffold** (for @jaeone94 ADD-03)
## Testing
- All tests use Vue nodes with new menu enabled
- Zero flaky tests (proper waits, no sleeps)
- Screenshots scoped to panel elements
## Unblocks
- **ADD-03** (error systems by @jaeone94) — errors tab scaffold ready to
extend
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10548-test-comprehensive-properties-panel-E2E-tests-PNL-01-32f6d73d36508199a216fd8d953d8e18)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## What
12 regression tests covering 10 workflow persistence bug gaps, including
the **critical data corruption fix in PR #9531** (pythongosssss) which
previously had ZERO test coverage.
## Why
Deep scan of 37 workflow persistence bugs found 12 E2E-testable gaps
with no regression tests. Workflow persistence is a core reliability
concern — data corruption bugs are the highest risk category.
## Tests
### 🔴 Critical
| Bug | PR | Tests | Description |
|-----|----|-------|-------------|
| Data corruption | #9531 | 2 | checkState during graph loading corrupts
workflow data |
| State desync | #9533 | 2 | Rapid tab switching desyncs workflow/graph
state |
### 🟡 Medium
| Bug | PR/Commit | Tests | Description |
|-----|-----------|-------|-------------|
| Lost previews | #9380 | 1 | Node output previews lost on tab switch |
| Stale canvas | 44bb6f13 | 1 | Canvas not cleared before loading new
workflow |
| Widget loss | #7648 | 1 | Widget values lost on graph change |
| API format | #9694 | 1 | API format workflows fail with missing nodes
|
| Paste duplication | #8259 | 1 | Middle-click paste duplicates workflow
|
| Blob URLs | #8715 | 1 | Transient blob: URLs in serialization |
### 🟢 Low
| Bug | PR/Commit | Tests | Description |
|-----|-----------|-------|-------------|
| Locale break | #8963 | 1 | Locale change breaks workflows |
| Panel drift | — | 1 | Splitter panel size drift |
## Conventions
- All tests use Vue nodes + new menu enabled
- Each test documents which PR/commit it regresses
- Proper waits (no sleeps)
- Screenshots scoped to relevant elements
- Tests read like user stories
## 🎉 Shoutout
PR #9531 by @pythongosssss was a critical data corruption fix that now
has regression test coverage for the first time.
Part of: Test Coverage Q2 Overhaul (REG-01)
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10547-test-12-workflow-persistence-regression-tests-incl-critical-PR-9531-32f6d73d3650818796c6c5950c77f6d1)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Summary
Add deterministic mock data fixtures for browser tests so they can use
`page.route()` to intercept API calls without depending on a live
backend.
## Changes
- **`browser_tests/fixtures/data/nodeDefinitions.ts`** — Mock
`ComfyNodeDef` objects for KSampler, CheckpointLoaderSimple, and
CLIPTextEncode
- **`browser_tests/fixtures/data/systemStats.ts`** — Mock `SystemStats`
with realistic RTX 4090 GPU info
- **`browser_tests/fixtures/data/README.md`** — Usage guide for
`page.route()` interception
All fixtures are typed against the Zod schemas in `src/schemas/` and
pass `pnpm typecheck:browser`.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10662-test-add-mock-data-fixtures-for-backend-API-responses-3316d73d3650813ea5c8c1faa215db63)
by [Unito](https://www.unito.io)
---------
Co-authored-by: dante01yoon <bunggl@naver.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: GitHub Action <action@github.com>
## Summary
Adds Playwright E2E tests for the SignIn dialog component and its
sub-forms.
## Tests added
- Dialog opens from login button in topbar
- Sign In form is the default view with email/password fields
- Toggle between Sign In and Sign Up forms
- API Key form navigation (forward and back)
- Terms of Service and Privacy Policy links present
- Form field presence verification
- Dialog close behavior (close button and Escape key)
- Forgot password link presence
- 'Or continue with' divider and API key button
## Notes
- Tests focus on UI navigation and element presence (no real Firebase
auth in test env)
- Dialog opened via `extensionManager.dialog.showSignInDialog()` API
- All selectors use stable IDs from the component source
(`#comfy-org-sign-in-email`, etc.)
## Task
Part of Test Coverage Q2 Overhaul (DLG-04).
## Conventions
- Uses Vue nodes with new menu enabled (`Comfy.UseNewMenu: 'Top'`)
- Tests read as user stories
- No full-page screenshots
- Proper waits, no sleeps
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10587-test-add-SignIn-dialog-E2E-tests-DLG-04-3306d73d3650815db171f8c5228e2cf3)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Summary
Expose `renderMarkdownToHtml()` on the `ExtensionManager` interface so
custom node extensions can render markdown to sanitized HTML without
bundling their own copies of `marked`/`DOMPurify`.
## Motivation
Multiple custom node packs (KJNodes, comfy_mtb, rgthree-comfy) bundle
their own markdown rendering libraries to implement help popups on
nodes. This causes:
- **Cloud breakage**: KJNodes uses a `kjweb_async` pattern (custom
aiohttp static route) to lazily load `marked.min.js` and
`purify.min.js`. This 404s on Cloud because the custom route is not
registered.
- **Redundant bundling**: Both `marked` (^15.0.11) and `dompurify`
(^3.2.5) are already direct dependencies of the frontend, used
internally by `markdownRendererUtil.ts`, `NodePreview.vue`,
`WhatsNewPopup.vue`, etc.
- **XSS risk**: Custom nodes using raw `marked` without `DOMPurify`
could introduce XSS vulnerabilities.
By exposing the existing `renderMarkdownToHtml()` through the official
`ExtensionManager` API, custom nodes can:
```js
const html = app.extensionManager.renderMarkdownToHtml(nodeData.description)
```
...instead of bundling and loading their own copies.
## Changes
- **`src/types/extensionTypes.ts`**: Add `renderMarkdownToHtml(markdown:
string, baseUrl?: string): string` to the `ExtensionManager` interface
with JSDoc.
- **`src/stores/workspaceStore.ts`**: Import and re-export
`renderMarkdownToHtml` from `@/utils/markdownRendererUtil`.
## Impact
- **Zero bundle size increase** — the function and its dependencies are
already bundled in the `vendor-markdown` chunk.
- **No breaking changes** — purely additive to the `ExtensionManager`
interface.
- **Follows existing pattern** — same approach as `toast`, `dialog`,
`command`, `setting` on `ExtensionManager`.
Related: #TBD (long-term plan for custom node extension library
dependencies)
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10700-feat-expose-renderMarkdownToHtml-on-ExtensionManager-3326d73d36508149bc1dc6bb45e7c077)
by [Unito](https://www.unito.io)
## Summary
Extract `assetPath` from a `ComfyPage` method to a standalone pure
function, removing unnecessary coupling to the page object.
## Changes
- **What**: Moved `assetPath` to
`browser_tests/fixtures/utils/paths.ts`. `DragDropHelper` and
`WorkflowHelper` import it directly instead of receiving it via
`ComfyPage`. `ComfyPage.assetPath` kept as thin delegate for backward
compat.
## Review Focus
Structural-only refactor — no behavioral changes. The function was
already pure (no `this`/`page` usage).
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10651-refactor-extract-assetPath-as-standalone-pure-function-3316d73d365081c0b0e0ce6dde57ef8e)
by [Unito](https://www.unito.io)
## Summary
Add guidance to `docs/guidance/playwright.md` that new node-specific
assertions should be methods on page objects/helpers rather than new
`comfyExpect` custom matchers.
## Changes
- **What**: New "Custom Assertions" section in Playwright guidance
documenting that existing `comfyExpect` matchers are fine to use, but
new assertions should go on the page object for IntelliSense
discoverability.
## Review Focus
Documentation-only change. No code refactoring — this is a convention
for new code only.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10660-docs-add-convention-for-new-assertions-prefer-page-objects-over-custom-matchers-3316d73d3650816d97a8fbbdc33f6b75)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Summary
Remove the exclusion filter that prevented backend-mirrored endpoint
types from being generated in `@comfyorg/ingest-types`.
## Changes
- **What**: The `openapi-ts.config.ts` excluded all endpoints shared
with the ComfyUI Python backend (system_stats, object_info, prompt,
queue, history, settings, userdata, etc.). Since the cloud ingest API
mirrors the backend, these types should be generated from the OpenAPI
spec as the canonical source. This adds ~250 new types and Zod schemas
covering previously excluded endpoints.
- **Breaking**: None. This only adds new exported types — no existing
types or imports are changed.
## Review Focus
- The cloud ingest API is designed to mirror the ComfyUI Python backend.
The original exclusion filter was added to avoid duplication with
`src/schemas/apiSchema.ts`, but the generated types should be the
canonical source since they are auto-generated from the OpenAPI spec.
- A follow-up PR will migrate imports in `src/` from `apiSchema.ts` to
`@comfyorg/ingest-types` where applicable.
- Webhooks and internal analytics endpoints remain excluded
(server-to-server, not frontend-relevant).
Related: #10662
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10697-refactor-include-backend-mirrored-endpoints-in-ingest-types-codegen-3326d73d365081569614f743ab6f074d)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## Motivation
Browser tests mock API responses with `route.fulfill()` using untyped
inline JSON. When the OpenAPI spec changes, these mocks silently drift —
mismatches aren't caught at compile time and only surface as test
failures at runtime.
We already have auto-generated types from OpenAPI and manual Zod
schemas. This PR makes those types the source of truth for test mock
data.
From Mar 27 PR review session action item: "instruct agents to use
schemas and types when writing browser tests."
## Type packages and their API coverage
The frontend has two OpenAPI-generated type packages, each targeting a
different backend API with a different code generation tool:
| Package | Target API | Generator | TS types | Zod schemas |
|---------|-----------|-----------|----------|-------------|
| `@comfyorg/registry-types` | Registry API (node packages, releases,
subscriptions, customers) | `openapi-typescript` | Yes | **No** |
| `@comfyorg/ingest-types` | Ingest API (hub workflows, asset uploads,
workspaces) | `@hey-api/openapi-ts` | Yes | Yes |
Additionally, Python backend endpoints (`/api/queue`, `/api/features`,
`/api/settings`, etc.) are typed via manual Zod schemas in
`src/schemas/apiSchema.ts`.
This PR applies **compile-time type checking** using these existing
types. Runtime validation via Zod `.parse()` is not yet possible for all
endpoints because `registry-types` does not generate Zod schemas — this
requires a separate migration of `registry-types` to
`@hey-api/openapi-ts` (#10674).
## Summary
- Add "Typed API Mocks" guideline to `docs/guidance/playwright.md` with
a sources-of-truth table mapping endpoint categories to their type
packages
- Add rule to `AGENTS.md` Playwright section requiring typed mock data
- Refactor `releaseNotifications.spec.ts` to use `ReleaseNote` type
(from `registry-types`) via `createMockRelease()` factory
- Annotate template mock in `templates.spec.ts` with
`WorkflowTemplates[]` type
Refs #10656
## Example workflow: writing a new typed E2E test mock
When adding a new `route.fulfill()` mock, follow these steps:
### 1. Identify the type source
Check which API the endpoint belongs to:
| Endpoint category | Type source | Zod available |
|---|---|---|
| Ingest API (hub, billing, workflows) | `@comfyorg/ingest-types` | Yes
— use `.parse()` |
| Registry API (releases, nodes, publishers) |
`@comfyorg/registry-types` | Not yet (#10674) — TS type only |
| Python backend (queue, history, settings) | `src/schemas/apiSchema.ts`
| Yes — use `z.infer` |
| Templates | `src/platform/workflow/templates/types/template.ts` | No —
TS type only |
### 2. Create a typed factory (with Zod when available)
**Ingest API endpoints** — Zod schemas exist, use `.parse()` for runtime
validation:
```typescript
import { zBillingStatusResponse } from '@comfyorg/ingest-types/zod'
import type { BillingStatusResponse } from '@comfyorg/ingest-types'
function createMockBillingStatus(
overrides?: Partial<BillingStatusResponse>
): BillingStatusResponse {
return zBillingStatusResponse.parse({
plan: 'free',
credits_remaining: 100,
renewal_date: '2026-04-28T00:00:00Z',
...overrides
})
}
```
**Registry API endpoints** — TS type only (Zod not yet generated):
```typescript
import type { ReleaseNote } from '../../src/platform/updates/common/releaseService'
function createMockRelease(
overrides?: Partial<ReleaseNote>
): ReleaseNote {
return {
id: 1,
project: 'comfyui',
version: 'v0.3.44',
attention: 'medium',
content: '## New Features',
published_at: new Date().toISOString(),
...overrides
}
}
```
### 3. Use in test
```typescript
test('should show upgrade banner for free plan', async ({ comfyPage }) => {
await comfyPage.page.route('**/billing/status', async (route) => {
await route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify(createMockBillingStatus({ plan: 'free' }))
})
})
await comfyPage.setup()
await expect(comfyPage.page.getByText('Upgrade')).toBeVisible()
})
```
The factory pattern keeps test bodies focused on **what varies** (the
override) rather than the full response shape.
## Scope decisions
| File | Decision | Reason |
|------|----------|--------|
| `releaseNotifications.spec.ts` | Typed | `ReleaseNote` type available
from `registry-types` |
| `templates.spec.ts` | Typed | `WorkflowTemplates` type available in
`src/platform/workflow/templates/types/` |
| `QueueHelper.ts` | Skipped | Dead code — instantiated but never called
in any test |
| `FeatureFlagHelper.ts` | Skipped | Response type is inherently
`Record<string, unknown>`, no stronger type exists |
| Fixture factories | Deferred | Coordinate with Ben's fixture
restructuring work to avoid duplication |
## Follow-up work
Sub-issues of #10656:
- #10670 — Clean up dead `QueueHelper` or rewrite against `/api/jobs`
endpoint
- #10671 — Expand typed factory pattern to more endpoints
- #10672 — Evaluate OpenAPI generation for excluded Python backend
endpoints
- #10674 — Migrate `registry-types` from `openapi-typescript` to
`@hey-api/openapi-ts` to enable Zod schema generation
## Test plan
- [x] `pnpm typecheck:browser` passes
- [x] `pnpm lint` passes
- [ ] Existing `releaseNotifications` and `templates` tests pass in CI
## Summary
Document the agreed-upon architectural separation for browser test
fixtures:
- `fixtures/data/` — Static test data (mock API responses, workflow
JSONs, node definitions)
- `fixtures/components/` — Page object components (locators, user
interactions)
- `fixtures/helpers/` — Focused helper classes (domain-specific actions)
- `fixtures/utils/` — Pure utility functions (no page dependency)
## Changes
- **`browser_tests/AGENTS.md`** — Added architectural separation section
with clear rules for each directory
- **`browser_tests/fixtures/data/README.md`** (new) — Explains the data
directory purpose and what belongs here vs `assets/`
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10645-docs-document-fixture-page-object-separation-in-browser-tests-3316d73d365081febf52d165282c68f6)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
## What
Adds a `cloud` Playwright project so E2E tests can run against
`DISTRIBUTION=cloud` builds, with `@cloud` / `@oss` test tagging.
## Why
100+ usages of `isCloud` / `DISTRIBUTION` across 9 categories (API
routing, UI visibility, settings, auth). Zero cloud test infrastructure
existed — cloud-specific UI components (LoginButton, SubscribeButton,
etc.) had no E2E coverage path.
## Investigation: Runtime Toggle
Investigated whether `isCloud` could be made runtime-toggleable in
dev/test mode (via `window.__FORCE_CLOUD__`). **Not feasible** —
`__DISTRIBUTION__` is a Vite `define` compile-time constant used for
dead-code elimination. Runtime override would break tree-shaking in
production.
Full investigation:
`research/architecture/cloud-runtime-toggle-investigation.md`
## What's included
### Playwright Config
- New `cloud` project alongside existing `chromium`
- Cloud project: `grep: /@cloud/` — only runs `@cloud` tagged tests
- Chromium project: `grepInvert: /@cloud/` — excludes cloud tests
### Build Script
- `npm run build:cloud` → `DISTRIBUTION=cloud vite build`
### Test Tagging Convention
```typescript
test('works in both', async () => { ... });
test('subscription button visible @cloud', async () => { ... });
test('install manager prompt @oss', async () => { ... });
```
### Example Tests
- 2 cloud-only tests validating cloud UI visibility
## NOT included (future work)
- CI workflow job for cloud tests (separate PR)
- Cloud project is opt-in — not run by default locally
## Unblocks
- Cloud-specific E2E tests for entire team
- TB-03 LoginButton, TB-04 SubscribeButton (@Kaili Yang)
- DLG-04 SignIn, DLG-06 CancelSubscription
Part of: Test Coverage Q2 Overhaul
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10546-test-infra-cloud-Playwright-project-with-cloud-oss-tagging-32f6d73d3650810ebb59dea8ce4891e9)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
## Summary
Document the recommended pattern for adding new domain-specific test
helpers as Playwright fixtures via `base.extend()` instead of attaching
them to `ComfyPage`.
## Changes
- **What**: Added "Creating New Test Helpers" section to
`docs/guidance/playwright.md` with fixture extension example and rules
## Review Focus
Documentation-only change. Verify the example code matches the existing
pattern in `browser_tests/fixtures/ComfyPage.ts`.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10653-docs-document-Playwright-fixture-injection-pattern-for-new-helpers-3316d73d36508145b402cf02a5c2c696)
by [Unito](https://www.unito.io)
---------
Co-authored-by: Alexander Brown <drjkl@comfy.org>
## Summary
Document the arrange/act/assert pattern for Playwright browser tests to
keep mock setup out of test bodies.
## Changes
- **What**: Added "Test Structure: Arrange/Act/Assert" section to
`docs/guidance/playwright.md` documenting that mock setup belongs in
`beforeEach`/fixtures, test bodies should only act and assert, and
`clearAllMocks` should never be called mid-test. Includes good/bad
examples.
## Review Focus
Docs-only change — no code impact.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10657-docs-add-arrange-act-assert-pattern-guidance-for-browser-tests-3316d73d365081aa92c0fb6442084484)
by [Unito](https://www.unito.io)
---------
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
## Summary
From the primordial entropy of 17 scattered spec files — a formless
sprawl of mixed concerns and inconsistent naming — emerges a clean,
domain-organized hierarchy. Order triumphs over chaos.
## Changes
- **What**: Reorganize all subgraph E2E tests from 17 flat files in
`browser_tests/tests/` into 10 domain-grouped files under
`browser_tests/tests/subgraph/`.
| File | Tests | Domain |
|------|-------|--------|
| `subgraphSlots` | 16 | I/O slot CRUD, rename, alignment, promoted slot
position |
| `subgraphPromotion` | 22 | Auto-promote, visibility, reactivity,
context menu, cleanup |
| `subgraphSerialization` | 16 | Hydration, round-trip, legacy formats,
ID remapping |
| `subgraphNavigation` | 10 | Breadcrumb, viewport, hotkeys, progress
state |
| `subgraphNested` | 9 | Configure order, duplicate names, pack values,
stale proxies |
| `subgraphLifecycle` | 7 | Source removal cleanup, pseudo-preview
lifecycle |
| `subgraphPromotionDom` | 6 | DOM widget persistence, cleanup,
positioning |
| `subgraphCrud` | 5 | Create, delete, copy, unpack |
| `subgraphSearch` | 3 | Search aliases, description, persistence |
| `subgraphOperations` | 2 | Copy/paste inside, undo/redo inside |
Where once the monolith `subgraph.spec.ts` (856 lines) mixed slot CRUD
with hotkeys, DOM widgets with navigation, and copy/paste with undo/redo
— now each behavioral domain has its sovereign territory.
Where once `subgraph-rename-dialog.spec.ts`,
`subgraphInputSlotRename.spec.ts`, and
`subgraph-promoted-slot-position.spec.ts` scattered rename concerns
across three kingdoms — now they answer to one crown:
`subgraphSlots.spec.ts`.
Where once `kebab-case` and `camelCase` warred for dominion — now a
single convention reigns.
All 96 test cases preserved. Zero test logic changes. Purely structural.
## Review Focus
- Verify no tests were lost in the consolidation
- Confirm import paths all resolve correctly at the new depth
(`../../fixtures/`)
- The `import.meta.dirname` asset path in `subgraphSlots.spec.ts` (slot
alignment test) updated for new directory depth
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10695-test-reorganize-subgraph-E2E-tests-into-domain-organized-directory-3326d73d36508197939be8825b69ea88)
by [Unito](https://www.unito.io)
Co-authored-by: Amp <amp@ampcode.com>
## Summary
Add a "Fixture Data & Schemas" section to `docs/guidance/playwright.md`
so agents reference existing Zod schemas and TypeScript types when
creating test fixture data.
## Changes
- **What**: New section listing key schema/type locations (`apiSchema`,
`nodeDefSchema`, `jobTypes`, `workflowSchema`, etc.) to keep test
fixtures in sync with production types.
## Review Focus
Documentation-only change; no runtime impact.
┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10642-docs-add-Fixture-Data-Schemas-section-to-Playwright-test-guidance-3316d73d365081f5a234e4672b3dc4b9)
by [Unito](https://www.unito.io)
