Commit Graph

11 Commits

Author SHA1 Message Date
Christian Byrne
963a7bf178 refactor: consolidate browser_tests/helpers/ into fixtures/ (#11411)
*PR Created by the Glary-Bot Agent*

---

## Summary

- Eliminates the confusing dual-helpers structure where
`browser_tests/helpers/` and `browser_tests/fixtures/helpers/` coexisted
one tier apart with overlapping purposes
- Routes each file to its natural home based on what it actually *is*:
page objects → `components/`, standalone utils → `utils/`, domain helper
classes stay in `helpers/`
- Adds an ESLint guard (`no-restricted-imports`) to prevent re-creating
`browser_tests/helpers/`

## File Moves

| File | From | To | Reason |
|---|---|---|---|
| `actionbar.ts` | `helpers/` | `fixtures/components/Actionbar.ts` |
Page object class imported by ComfyPage |
| `templates.ts` | `helpers/` | `fixtures/components/Templates.ts` |
Page object class imported by ComfyPage |
| `boundsUtils.ts` | `fixtures/helpers/` | `fixtures/utils/` | Pure
function, not a helper class |
| `mimeTypeUtil.ts` | `fixtures/helpers/` | `fixtures/utils/` | Pure
function, not a helper class |
| `builderTestUtils.ts` | `helpers/` | `fixtures/utils/` | Shared test
setup functions |
| `clipboardSpy.ts` | `helpers/` | `fixtures/utils/` | Page injection
utility |
| `fitToView.ts` | `helpers/` | `fixtures/utils/` | Canvas utility
function |
| `manageGroupNode.ts` | `helpers/` | `fixtures/utils/` | Litegraph
interaction helper |
| `painter.ts` | `helpers/` | `fixtures/utils/` | Test helper functions
|
| `perfReporter.ts` | `helpers/` | `fixtures/utils/` | Test
infrastructure |
| `promotedWidgets.ts` | `helpers/` | `fixtures/utils/` | Query helpers
for specs |

## What Changed Beyond File Moves

- **28 import statements** updated across test specs, fixtures, and
infra files
- **AGENTS.md** — directory tree diagram and architectural separation
descriptions updated
- **README.md** — "Leverage Existing Fixtures and Helpers" section
updated
- **`.claude/skills/perf-fix-with-proof/SKILL.md`** — perfReporter path
reference updated
- **`eslint.config.ts`** — added `@e2e/helpers/*` restricted import
pattern to both spec and non-spec browser_tests rules

## Verification

- `pnpm typecheck` — clean
- `pnpm typecheck:browser` — clean
- `pnpm lint` — 0 errors, 0 warnings
- `pnpm format:check` — all files formatted
- `pnpm knip` — clean
- Pre-commit hooks passed full pipeline (oxfmt, oxlint, eslint,
typecheck, typecheck:browser)

## Config Audit

No changes needed to: `tsconfig.json` (`@e2e/*` alias covers all
subdirs), `playwright.config.ts`, `vite.config.mts`, `knip.config.ts`,
`.oxlintrc.json`, `nx.json`

## Manual Verification Note

This is a pure structural refactoring (file moves + import updates) with
zero behavioral or visual changes. The typecheck and lint passes confirm
all imports resolve correctly.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-11411-refactor-consolidate-browser_tests-helpers-into-fixtures-3476d73d3650816cb671ef7fa8433f66)
by [Unito](https://www.unito.io)

---------

Co-authored-by: glary-bot <glary-bot@comfy.org>
Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
Co-authored-by: DrJKL <DrJKL0424@gmail.com>
Co-authored-by: Amp <amp@ampcode.com>
2026-04-28 02:18:31 +00:00
Alexander Brown
8c9328c1b2 feat: add eslint-plugin-playwright via oxlint JS plugins (#11136)
## Summary

Add eslint-plugin-playwright as an oxlint JS plugin scoped to
browser_tests/, enforcing Playwright best practices at lint time.

## Changes

- **What**: Configure eslint-plugin-playwright@2.10.1 via oxlint's alpha
`jsPlugins` field (`.oxlintrc.json` override scoped to
`browser_tests/**/*.ts`). 18 recommended rules +
`prefer-native-locators` + `require-to-pass-timeout` at error severity.
All 173 initial violations resolved (config, auto-fix, manual fixes).
`no-force-option` set to off — 28 violations need triage (canvas overlay
workarounds vs unnecessary force) in a dedicated PR.
- **Dependencies**: `eslint-plugin-playwright@^2.10.1` (devDependency,
required by oxlint jsPlugins at runtime)

## Review Focus

- `.oxlintrc.json` override structure — this is the first use of
oxlint's JS plugins alpha feature in this repo
- Manual fixes in spec files: `waitForSelector` → `locator.waitFor`,
deprecated page methods → locator equivalents, `toPass()` timeout
additions
- Compound CSS selectors replaced with `.and()` (Playwright native
locator composition) to avoid `prefer-native-locators` suppressions
- Lint script changes in `package.json` to include `browser_tests/` in
oxlint targets

---------

Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: GitHub Action <action@github.com>
2026-04-11 01:25:14 +00:00
Alexander Brown
0132c77c7d test: harden 82 Playwright specs for deterministic CI runs (#10967)
## Summary

Harden 98 E2E spec files and 8 fixtures/helpers for deterministic CI
runs by replacing race-prone patterns with retry-safe alternatives.

No source code changes -- only `browser_tests/` is touched.

## Changes

- **E2E spec hardening** (98 spec files, 6 fixtures, 2 helpers):

  | Fix class | Sites | Examples |
  |-----------|-------|---------:|
| `expect(await ...)` -> `expect.poll()` | ~153 | interaction,
defaultKeybindings, workflows, featureFlags |
| `const x = await loc.count(); expect(x)` -> `toHaveCount()` | ~19 |
menu, linkInteraction, assets, bottomPanelShortcuts |
| `nextFrame()` -> `waitForHidden()` after menu clicks | ~22 |
contextMenu, rightClickMenu, subgraphHelper |
| Redundant `nextFrame()` removed | many | defaultKeybindings, minimap,
builderSaveFlow |
| `expect(async () => { ... }).toPass()` retry blocks | 5 | interaction
(graphdialog dismiss guard) |
| `force:true` removed from `BaseDialog.close()` | 1 | BaseDialog
fixture |
| ContextMenu `waitForHidden` simplified (check-then-act race removed) |
1 | ContextMenu fixture |
| Non-deterministic node order -> proximity-based selection | 1 |
interaction (toggle dom widget) |
  | Tight poll timeout (250ms) -> >=2000ms | 2 | templates |

- **Helper improvements**: Exposed locator getters on
`ComfyPage.domWidgets`, `ToastHelper.toastErrors`, and
`WorkflowsSidebarTab.activeWorkflowLabel` so callers can use retrying
assertions (`toHaveCount()`, `toHaveText()`) directly.

- **Flake pattern catalog**: Added section 7 table to
`browser_tests/FLAKE_PREVENTION_RULES.md` documenting 8 pattern classes
for reviewers and future authors.

- **Docs**: Fixed bad examples in `browser_tests/README.md` to use
`expect.poll()`.

- **Breaking**: None
- **Dependencies**: None

## Review Focus

- All fixes follow the rules in
`browser_tests/FLAKE_PREVENTION_RULES.md`
- No behavioral changes to tests -- only timing/retry strategy is
updated
- The `ContextMenu.waitForHidden` simplification removes a
swallowed-error anti-pattern; both locators now use direct `waitFor({
state: 'hidden' })`

---------

Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: github-actions <github-actions@github.com>
2026-04-09 20:50:56 -07:00
Alexander Brown
3b78dfbe1c test: migrate browser_tests/ to @e2e/ path alias and add lint rule (#10958)
## Summary

Complete the @e2e/ path alias migration started in #10735 by converting
all 354 remaining relative imports and adding a lint rule to prevent
backsliding.

## Changes

- **What**: Migrate all relative imports in browser_tests/ to use
`@e2e/` (intra-directory) and `@/` (src/ imports) path aliases. Add
`no-restricted-imports` ESLint rule banning `./` and `../` imports in
`browser_tests/**/*.ts`. Suppress pre-existing oxlint `no-eval` and
`no-console` warnings exposed by touching those files.

## Review Focus

- ESLint flat-config merging: the `@playwright/test` ban and
relative-import ban are in two separate blocks to avoid last-match-wins
collision with the `useI18n`/`useVirtualList` blocks higher in the
config.
- The `['./**', '../**']` glob patterns (not `['./*', '../*']`) are
needed to catch multi-level relative paths like `../../../src/foo`.

Follows up on #10735

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10958-test-migrate-browser_tests-to-e2e-path-alias-and-add-lint-rule-33c6d73d365081649d1be771eac986fd)
by [Unito](https://www.unito.io)

Co-authored-by: Amp <amp@ampcode.com>
2026-04-08 11:28:59 -07:00
Christian Byrne
b2f848893a test: add perf test for viewport pan sweep GC churn (#10479)
## Summary

Adds a `@perf` test to establish a baseline for viewport panning GC
churn on large graphs.

## Changes

- **What**: New `large graph viewport pan sweep` perf test that pans
aggressively back and forth across a 245-node graph, forcing many nodes
to cross the viewport boundary. Measures style recalcs, forced layouts,
task duration, heap delta, and DOM node count.

## Review Focus

This is **PR 1 of 2** (perf-fix-with-proof pattern). The fix (viewport
culling) will follow in a separate PR once this baseline is established
on main. CI will then show the delta proving the improvement.

The test uses 120 steps out + 120 steps back at 8px/step = ~960px total
displacement, enough to sweep across a significant portion of the large
graph layout.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10479-test-add-perf-test-for-viewport-pan-sweep-GC-churn-32d6d73d365081cc9f15fe3d5890675d)
by [Unito](https://www.unito.io)
2026-03-28 13:54:26 -07:00
Christian Byrne
d860d54366 test: add large graph zoom perf test for ResizeObserver baseline (#10478)
## Summary

Adds a `@perf` test that establishes a baseline for ResizeObserver
layout cost during zoom on a large graph (245 nodes).

## Changes

- **What**: New `large graph zoom interaction` perf test that zooms
in/out 30 steps on `large-graph-workflow`, measuring `layouts`,
`layoutDurationMs`, `frameDurationMs`, and `TBT`. Each zoom step
triggers ResizeObserver for all node elements due to CSS scale changes.

## Review Focus

This is **PR 1 of 2** for throttling the ResizeObserver during zoom/pan.
Once this merges and establishes a baseline on main, the fix PR (#10473)
will show a CI-proven delta demonstrating the improvement.

The test follows the same patterns as `large graph pan interaction` and
`canvas zoom sweep`.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10478-test-add-large-graph-zoom-perf-test-for-ResizeObserver-baseline-32d6d73d365081169537e557c14d8c51)
by [Unito](https://www.unito.io)
2026-03-25 21:05:36 -07:00
Christian Byrne
c8a03b8cf1 test: add vue renderer perf tests (idle, pan, zoom culling) (#10001)
Add 3 perf tests exercising Vue renderer (Nodes 2.0) with the 245-node
large-graph-workflow:

- **vue renderer large graph idle** — enables Vue nodes, idles 120
frames, measures style recalcs/layouts/DOM nodes. Catches containment
regressions.
- **vue renderer large graph pan** — middle-click pans 60 frames,
measures frameDurationMs/TBT. Catches TransformPane falling back to
reactive style bindings.
- **vue renderer zoom out culling** — zooms out 20 steps (triggers
size-based culling at <4px), idles 60 frames, zooms back in. Catches
viewport culling regressions.

Complements existing LiteGraph renderer tests for direct A/B comparison
in the perf report. Uses existing `large-graph-workflow.json` asset from
#9940.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-10001-test-add-vue-renderer-perf-tests-idle-pan-zoom-culling-3256d73d365081b4b714d8f3b396604c)
by [Unito](https://www.unito.io)
2026-03-17 05:44:49 -07:00
Christian Byrne
c420cff4f0 feat: add TBT/frameDuration metrics and new perf test scenarios (#9910)
## Summary

Adds Total Blocking Time (TBT) and frame duration metrics to the
performance testing infrastructure, plus three new test scenarios
covering zoom, pan, and many-nodes-idle.

## Changes

### New Metrics
- **`totalBlockingTimeMs`** — Computed from PerformanceObserver
`longtask` entries: `sum(duration - 50ms)` for tasks >50ms. Measures
main thread blocking.
- **`frameDurationMs`** — Average frame duration via rAF timing (16.67ms
= 60fps target). Measures rendering smoothness.

### New Test Scenarios
| Scenario | Description |
|---|---|
| `canvas-zoom-sweep` | 10 zoom-in + 10 zoom-out cycles on default
workflow |
| `canvas-pan-many-nodes` | 10 pan sweeps over 100-node workflow |
| `canvas-many-nodes-idle` | 2-second idle measurement with 100 nodes
rendered |

### Infrastructure
- `PerformanceHelper.ts`: Installs PerformanceObserver for longtask,
collects TBT, measures frame duration via rAF
- `perf-report.ts`: Reports TBT and frame duration in PR comment tables
- `browser_tests/assets/perf/many_nodes_100.json`: 100-node (10×10 grid)
test fixture

## Review Focus
- TBT collection clears entries at `startMeasuring()` and reads at
`stopMeasuring()` — ensure no race with observer buffering
- Frame duration sampling uses 10 frames — enough for signal without
slowing tests

Depends on: #9886, #9887

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9910-feat-add-TBT-frameDuration-metrics-and-new-perf-test-scenarios-3236d73d365081488ae3c594a8bf7cff)
by [Unito](https://www.unito.io)
2026-03-15 08:54:00 -07:00
Christian Byrne
64c852bf82 test: add large-graph perf test with 245-node workflow (backlog N5) (#9940)
## What

Adds a 245-node workflow asset and two `@perf` tests to establish a
baseline for large-graph performance regressions (Tier 6 in the
performance backlog).

## Why

Backlog item N5: we need CI regression detection for compositor layer
management, GPU texture count, and transform pane cost at 245+ nodes.
This is PR1 of 2 — establishes baseline metrics on main. Future
optimization PRs will show improvement deltas against this baseline.

## Tests Added

- **`large graph idle rendering`** — 120 frames idle with 245 nodes,
measures style recalcs, layouts, task duration, heap delta
- **`large graph pan interaction`** — middle-click pan across 245 nodes,
stresses compositor layer management and transform recalculation

## Workflow Asset

`browser_tests/assets/large-graph-workflow.json` — 245 nodes (49
pipelines of CheckpointLoader → 2× CLIPTextEncode → KSampler +
EmptyLatentImage), 294 links. Minimal structure focused on node count.

## Verification

- [x] `pnpm typecheck:browser` passes
- [x] `pnpm lint` passes (eslint on changed file)
- [x] All link references in JSON validated programmatically

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9940-test-add-large-graph-perf-test-with-245-node-workflow-backlog-N5-3246d73d365081f6b5d8ddb9a85e6ad0)
by [Unito](https://www.unito.io)

---------

Co-authored-by: GitHub Action <action@github.com>
2026-03-15 03:04:39 -07:00
Christian Byrne
69076f35f8 test: add subgraph workflow round to performance tests (#9306)
## Summary

Add subgraph workflow performance tests to track style recalculations
and layout thrashing for nested subgraph workflows.

## Changes

- **What**: Add 3 new perf test cases (`subgraph-idle`,
`subgraph-mouse-sweep`, `subgraph-dom-widget-clipping`) that mirror the
existing default workflow tests but load the `subgraphs/nested-subgraph`
workflow. The existing perfReporter pipeline automatically picks up the
new measurements.

## Review Focus

The new tests are structurally identical to the existing 3 default
workflow tests — only the workflow loaded and measurement names differ.
No CI or config changes needed.

Fixes
https://www.notion.so/comfy-org/Implement-Add-subgraph-workflow-round-in-performance-testing-process-3156d73d365081d094efdee58215e15b

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9306-test-add-subgraph-workflow-round-to-performance-tests-3156d73d36508133b85cc53a748bc75f)
by [Unito](https://www.unito.io)
2026-03-04 13:38:48 -08:00
Christian Byrne
8e215b3174 feat: add performance testing infrastructure with CDP metrics (#9170)
## Summary

Add a permanent, non-failing performance regression detection system
using Chrome DevTools Protocol metrics, with automatic PR commenting.

## Changes

- **What**: Performance testing infrastructure — `PerformanceHelper`
fixture class using CDP `Performance.getMetrics` to collect
`RecalcStyleCount`, `LayoutCount`, `LayoutDuration`, `TaskDuration`,
`JSHeapUsedSize`. Adds `@perf` Playwright project (Chromium-only,
single-threaded, 60s timeout), 4 baseline perf tests, CI workflow with
sticky PR comment reporting, and `perf-report.js` script for generating
markdown comparison tables.

## Review Focus

- `PerformanceHelper` uses `page.context().newCDPSession(page)` — CDP is
Chromium-only, so perf metrics are not collected on Firefox. This is
intentional since CDP gives us browser-level style recalc/layout counts
that `performance.mark/measure` cannot capture.
- The CI workflow uses `continue-on-error: true` so perf tests never
block merging.
- Baseline comparison uses `dawidd6/action-download-artifact` to
download metrics from the target branch, following the same pattern as
`pr-size-report.yaml`.

## Stack

This is the foundation PR for the Firefox performance fix stack:
1. **→ This PR: perf testing infrastructure**
2. `perf/fix-cursor-cache` — cursor style caching (depends on this)
3. `perf/fix-subgraph-svg` — SVG pre-rasterization (depends on this)
4. `perf/fix-clippath-raf` — RAF batching for clip-path (depends on
this)

PRs 2-4 are independent of each other.

┆Issue is synchronized with this [Notion
page](https://www.notion.so/PR-9170-feat-add-performance-testing-infrastructure-with-CDP-metrics-3116d73d3650817cb43def6f8e9917f8)
by [Unito](https://www.unito.io)

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Brown <drjkl@comfy.org>
2026-02-25 20:09:57 -08:00