Compare commits

...

36 Commits

Author SHA1 Message Date
Nathaniel Parson Koroso
b99100d0b4 test: un-export AutoRunClass (knip: consumers use AutoRunVerdict) 2026-07-06 17:12:24 -07:00
Nathaniel Parson Koroso
6d5bcb9e04 test: suppress deferred-await false positives in turnstile timer tests
vitest/valid-expect flags assertions stored before advancing fake
timers and awaited after - awaiting at creation would deadlock the
timer advance. The file is byte-identical to main; the warnings appear
because the type-aware lint toolchain moved. Suppressed per line with
the reason; also drops a scratch await added while chasing this.
2026-07-06 17:05:39 -07:00
Nathaniel Parson Koroso
0723702791 test: serialize CI workers, prune comment noise, clearer naming
The auto-run tier needs exclusive backend-queue access, so the CI job
now runs with workers=1 - parallel workers were interrupting each
other's executions and cross-attributing errors. Await the async
toHaveLength assertion the type-aware lint flagged, drop the
calibration measurement harness (one-shot scaffolding; the measured
constant keeps a provenance note), rename NO_SINK to
NO_OBSERVABLE_OUTPUT, and cut comments down to load-bearing WHYs.
2026-07-06 17:03:21 -07:00
Nathaniel Parson Koroso
9825047176 ci: pin every manifest pack to its verified commit
An unpinned pack means any upstream push can red the gating check for
every PR in the repo. Each row now pins the exact SHA the suite was
verified against locally (all tiers green, both environments); bumps
are deliberate, re-verified changes. Also records why the job is not
sharded yet: per-shard setup (~4.5 min of pack installs and backend
boot) dominates the ~5.5 min suite, so a prebuilt image comes first.
2026-07-06 16:33:53 -07:00
Nathaniel Parson Koroso
7adfaa9079 test: stabilize every-node auto-run across environments; document the tiers
The cannotRunAlone baseline (per pack, in the manifest) records nodes
that cannot execute standalone on a bare backend, asserted both ways so
entries cannot rot: an unlisted failure is a regression, a listed node
that runs clean must be removed. queuePrompt rejection is retried once
before classifying VALIDATION_FAIL - pack JS hooking the queue path can
refuse transiently, and the backend log proved several apparent rejects
never reached the server. Nodes whose execution depends on their own
pack JS preprocessing widget values (rgthree Power widgets, KJ editors)
are excluded unconditionally with the mechanism recorded, since whether
a page applies pack JS varies by serving setup; ML-session initializers
and unstable executed-set reporters join them. ADDING_PACKS and the
README document the every-node tiers and all five exception ledgers.

Verified 65/65 in both documented environments: dev server and
dist-serving CI parity, twice consecutively on the latter.
2026-07-06 16:16:51 -07:00
Nathaniel Parson Koroso
1e36107109 test: every-node coverage - mount, save/reload, connect, and auto-run for all pack nodes
All-nodes tiers discover each pack's full node list from the live
backend: chunked mount checks in both renderers (batch size 24, chosen
by the committed calibration tool), chunked save/reload round-trips,
a connectivity corpus widened from the curated sentinels to every
registered node, and an auto-run tier that classifies every node
(AUTO_RUNNABLE / NEEDS_WIRES / NEEDS_MODELS / NO_SINK) and executes the
runnable ones in batches with per-node bisection on failure.

Hard-won harness rules baked in: queuePrompt rejects classify instantly
as VALIDATION_FAIL instead of burning the timeout; a timed-out batch
interrupts and verifies the queue drained so one hung node cannot jam
every later run; a pre-flight queue check fails fast with the real
cause; and three reviewable exception ledgers carry reasons inline
(AUTO_RUN_EXCLUDE for runtime-downloaders like RemBGSession+, a scoped
console-noise allowlist for KJNodes' undefined-filename previews,
connectivity CONNECT_REJECTED/ROUNDTRIP_LOST entries for pack JS that
vetoes or drops links).
2026-07-06 16:16:23 -07:00
Nathaniel Parson Koroso
1d5514c90e docs: rename ADDING_PACKS to ADDING_CUSTOM_NODES
The doc onboards custom nodes; name it what it is.
2026-07-06 16:16:09 -07:00
Nathaniel Parson Koroso
61d1cbfdb0 docs: fix ADDING_PACKS extensions probe and misattributed CI triage
grep -c on the single-line /extensions JSON could only say 0 or 1; count
entries properly with the same python one-liner style the doc already
uses. The Step 7 triage claimed our CI failure was upstream drift; it was
the dev-server blindspot - reorder the advice to reproduce under 6b
before diagnosing.
2026-07-02 18:49:20 -07:00
Nathaniel Parson Koroso
14666b09c4 docs: fold 5-pack onboarding lessons into ADDING_PACKS
Detect frontend-JS packs at install time, split local verification into
the fast dev-server loop and the CI-parity dist run (required when the
pack ships frontend JS), spell out that workflow media paths resolve
against the backend's working directory, and add upstream-drift triage
for unpinned packs. Checklist updated to match.
2026-07-02 18:46:05 -07:00
Nathaniel Parson Koroso
efb0365bc3 test: make connectivity instance-aware; slot drags survive pack page chrome
Two CI failures with the 7-pack backend, both from pack frontend JS that
never loads under the Vite dev server (its /extensions list is core-only):

- rgthree's Seed rebuilds its declared seed input as a widget-only
  control, so the planned BatchCount+.INT -> Seed.seed pair has no socket
  on the instance. The sweep now classifies that as
  WIDGET_ONLY_ON_INSTANCE, logged and excluded like wildcards; a name
  missing from both slots and widgets still fails hard, and the drag test
  picks the first in-pack pair that materializes on real instances.

- rgthree's progress bar shifts the canvas element 16px down, and
  NodeSlotReference.getPosition returned canvas-relative coordinates, so
  every slot drag grabbed the node title instead of the slot dot. Slot
  positions now include the canvas element's page offset (a no-op when
  the canvas sits at 0,0).

Documents the dev-server blindspot and the CI-parity loop (build dist,
--front-end-root) in the suite README and ADDING_PACKS. Verified 36/36
green against both the dev server and a dist-serving 7-pack backend.
2026-07-02 18:42:48 -07:00
Nathaniel Parson Koroso
065bc0c336 test: fail manifest load when a run tier has no workflow
A run row with an empty workflow would skip locally and rely on CI's
skip gate to notice the lost coverage; enforce the documented contract
at load time instead.
2026-07-02 18:13:26 -07:00
Nathaniel Parson Koroso
1248c4628a test: onboard 5 packs (rgthree, essentials, KJNodes, Custom-Scripts, WAS) with vueNodesCompatible flag and ADDING_PACKS guide
Five new manifest rows, each covering load, connectivity, and run tiers
with hand-authored model-free workflows verified against a live backend.
New optional vueNodesCompatible manifest field: a pack proven unable to
mount under Vue Nodes 2.0 runs its LiteGraph assertions only - never a
test.skip, so the zero-skip CI gate stays honest. All five packs mount
under Vue Nodes 2.0 empirically, so no row sets the flag; the decision
helper is unit-tested instead. ADDING_PACKS.md is the authoritative
step-by-step onboarding process, validated against live /object_info.
Manifest rows now also fail fast on an empty repo field.
2026-07-02 18:05:20 -07:00
Nathaniel Parson Koroso
ee83d67834 test: single tier source of truth; fix skip diagnostic to find nested skips
Derive CustomNodeTier from the VALID_TIERS array (as const) so adding a tier
is one edit and the type/runtime lists can't drift. The forbid-skips
diagnostic now recurses the report and prints only specs that actually
skipped - the old dump printed every title and a single-level filter would
miss specs nested under describe() blocks (which the regression spec uses).
2026-07-02 16:07:04 -07:00
Nathaniel Parson Koroso
f63b7d866e ci: gate custom-node job with changes-filter, not trigger paths
A required check gated by a trigger-level paths filter never creates a check
run on a PR that touches none of those paths, leaving branch protection stuck
Pending. Move the gating to a job-level if via the changes-filter action (a
skipped job counts as passing), mirroring ci-tests-unit.yaml, so this can be
marked required without stalling docs-only PRs. Keeps the same-repo fork guard
in the same if.
2026-07-02 16:03:37 -07:00
Nathaniel Parson Koroso
068191ea47 test: harden custom-node CI and manifest per review
Security: the pack-install job now runs only for same-repo PRs and pushes, so
a fork PR can't point the manifest's repo URLs at attacker-controlled code
that the job would clone and pip-install. Fork PRs keep the env-agnostic
coverage via the main e2e shards.

Stability: pack requirements install under a pip constraint pinning the CPU
torch stack, so no pack can swap torch for a GPU/incompatible build on the
--cpu runner.

Correctness: manifest validation rejects unknown tier values (a 'connectivty'
typo would otherwise silently drop that tier's coverage). Connectivity's
'pack installed' predicate is extracted to one isEntryInstalled helper used by
both the breadth and drag tests.
2026-07-02 15:56:17 -07:00
Nathaniel Parson Koroso
07c4b230b2 ci: make the custom-node job gating - fail on pack-install error or any skip
A regression gate that lets a broken pack through as a skip is theater. Pack
clone/dependency failures now fail the job (array+loop instead of a
failure-swallowing jq|while pipe), and a post-run check fails the job if any
test was skipped - on this backend every tier is meant to run, so a skip
means a pack or devtools did not load. Drops the informational framing;
mark custom-nodes-e2e required in branch protection to block merges.
2026-07-02 15:36:19 -07:00
Nathaniel Parson Koroso
9ed51f1e4b ci: run the custom-node suite against a backend with the packs installed
Phase 5. A new informational (non-gating) workflow that reuses the repo's
setup-frontend/setup-playwright/setup-comfyui-server actions, then installs
every pack the manifest declares (jq loop over customNodeManifest.json, so a
new pack row installs itself with no workflow change) and boots ComfyUI with
--multi-user --cache-none before running browser_tests/tests/customNodes.

This makes the load and run tiers actually execute in CI instead of skipping
for want of the packs - the whole point of the suite. A pack whose deps fail
degrades to an honest skip rather than reddening the job.
2026-07-02 15:25:31 -07:00
Nathaniel Parson Koroso
4a91fa4849 test: name connectivity tests in plain language
T-conn was planning-doc shorthand for the connectivity tier; test titles and
logs now say connectivity outright so CI output reads without tribal
knowledge.
2026-07-02 14:11:56 -07:00
Nathaniel Parson Koroso
0991905a89 test: exclude COMBO literals from connectivity auto-pairing
CI caught what a pack-rich local backend masked: isValidConnection compares
only the string COMBO while every combo slot carries its own option set, so
the planner would wire a checkpoint dropdown into a scheduler dropdown and
call it proof, and combo outputs declare a non-string output_name whose
instance slot name never matches (DevToolsNodeWithOutputCombo failed 5
pairs on CI as SLOT_CONTRACT_MISMATCH). Combo slots are now recorded and
counted like wildcards instead of paired, the normalizer coerces slot names
to strings, and a pure spec locks both behaviors. Targeted fixtures remain
the way to cover combo semantics.
2026-07-02 14:10:37 -07:00
Nathaniel Parson Koroso
df6764762b test: make the custom-node suite work on multi-user backends
The Comfy.userId=default settings override broke every test on multi-user
backends (the repo's stated browser-test prerequisite): devtools
set_settings wrote to a user no session reads, so Comfy.TutorialCompleted
never landed, the templates dialog never opened, and the beforeEach wait
timed out - CI sessions even inherited leftover settings (a zh locale) from
earlier tests on the same worker user. Dropping the override lets the
fixture target the real per-worker user everywhere; the harness backend now
runs --multi-user like CI. Connectivity's per-pack guards and drag
derivation apply only to installed packs, so a backend without the manifest
packs reports the absence instead of hard-failing while the core sweep,
native drag, and self-checks still run.
2026-07-02 13:45:34 -07:00
Nathaniel Parson Koroso
2d2b318450 test: drop exports from internal-only custom-node types
knip flags exported types with no external consumers; CustomNodeTier,
ObjectInfoNode, NormalizedSlot, and SlotRef are referenced only within
their own modules.
2026-07-02 13:15:58 -07:00
GitHub Action
0f94da8746 [automated] Apply ESLint and Oxfmt fixes 2026-07-02 19:32:18 +00:00
Nathaniel Parson Koroso
d80427d014 test: assert breadth-sweep console errors and tighten manifest shape checks
The breadth sweep now fails on any console error captured during the
connect/serialize/prompt loop, matching the fidelity test. The wildcard
predicate is exported from typePairing and reused instead of re-derived.
assertEntry validates real shapes (non-empty pack/expectedNodes/tiers,
arrays, boolean requiresGpu, finite positive timeoutMs); workflow stays
allowed as an empty string until a pack gains a run-tier fixture.
2026-07-02 12:28:23 -07:00
Nathaniel Parson Koroso
d02e665290 test: address review feedback on the custom-node suite
Resolve the manifest path from import.meta.url so tests are cwd-independent,
and validate requiresGpu at manifest load. Reuse the centralized TestIds for
the error overlay, error dialog, and templates dialog selectors. Extract the
shared suite settings and templates-dialog dismissal into
fixtures/utils/customNodeSuite so the three specs cannot drift. Rename
spikeDesktop.spec.ts to coreSmoke.spec.ts to match its maintained purpose,
document the full manifest schema in the README, and describe the gate
outcome without a hardcoded test count.
2026-07-02 12:24:33 -07:00
Nathaniel Parson Koroso
dc83cc4df6 test: prove the connectivity executor can reject, and drag every pack
A permanent self-check feeds the shared pair executor a type-incompatible
pair and a fabricated slot name and requires CONNECT_REJECTED and
SLOT_CONTRACT_MISMATCH back, so a green sweep can never come from a
classifier that lost the ability to fail. The breadth test asserts every
connectivity-tier pack contributes pairs, guarding pack attribution. The
drag tier's widget-primitive exclusion is removed: widget-backed inputs
render real slot dots under Vue Nodes (verified empirically), so every pack
now gets an in-pack drag in both renderers, asserted present.
2026-07-02 12:19:40 -07:00
Nathaniel Parson Koroso
8b81a4f359 test: add connectivity tier proving the slot/type contract
A type-pairing generator indexes /object_info producers and consumers and
plans one representative typed edge per slot, excluding wildcard slots
(isValidConnection short-circuits on * before the real type compare, so a
wildcard link proves reachability, not interop). The breadth sweep connects
every planned edge through the real validator in-page and requires each link
to survive serialize/configure and appear in graphToPrompt output; verified
up front that graphToPrompt emits links even when other required inputs
dangle. A curated subset is dragged slot-dot to slot-dot under both
renderers, addressed by data-slot-key so shared labels cannot misfire.
Orphan types are reported, never failed; connect vetoes must match a
committed allow-list. Manifest packs opt in via a connectivity tier that
needs no extra assets.
2026-07-02 12:14:39 -07:00
GitHub Action
8f567e8ef0 [automated] Apply ESLint and Oxfmt fixes 2026-07-02 18:44:36 +00:00
Nathaniel Parson Koroso
4fb282f853 docs: link custom-node suite README from browser_tests README 2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
d17a387ddb test: name each custom-node check as a pnpm script and document the suite
One script per pack tier (impact-render/impact-run/vhs-render/vhs-run) plus
the self-check, all opening the Playwright Inspector so anyone can step
through what the robot does. README covers prerequisites, every script, a
worked example, the zero-visible-errors contract, and how to add a pack.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
68ba0aa613 test: add pnpm scripts for the custom-node suite
test:custom-nodes runs the whole suite headless (the gate); :watch opens a
headed slow-motion run of the browser tiers; :debug steps through them in the
Playwright Inspector. All target the local dev server on :5173 and use the
committed system-Chrome config (no bundled-chromium download). Pass -g to
:watch / :debug to run a single test, e.g. -g 'VideoHelperSuite.*T1'.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
675140c164 test: drop io tier scaffold until assertion-node pack exists
T2a gated on the ComfyUI-test-framework 'Assert Executed' nodes, which are
not published for any backend yet, so the tier could only ever skip. A test
that cannot run anywhere is reporting noise; restore it from history when
the assertion pack lands. Suite is now 16 passed, zero skips, zero failures.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
64706c53c3 test: enforce zero visible errors across custom-node suite
Every browser tier now asserts the app's user-facing error surfaces (error
overlay, error dialog, node render errors, error toasts) are absent at test
start and after each pass, so a run is green only if a human watching the
screen sees zero errors. The harness self-check asserts the overlay IS
visible after a forced execution error, keeping the selectors provably live.

Sessions boot with a blank graph (Comfy.TutorialCompleted=false) because the
bundled default template references models absent on a scoped backend; the
tutorial path's auto-opened template browser is dismissed per test. Settings
now reach the session on single-user server-storage backends by routing
devtools set_settings to the default user, and the errors tab stays enabled
so error indicators are never suppressed in this suite. The smoke test loads
a core-only model-free workflow instead of the SD1.5 default asset.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
bfa94d4118 test: enable run tier for Impact and VHS with model-free workflows
Impact runs ImpactInt and ImpactFloat into PreviewAny as a group; VHS decodes
the existing plain_video.mp4 asset through VHS_LoadVideoPath into
VHS_VideoInfo. The executed-set check asserts each expected node individually
executed, so group workflows still verify per-node execution. Requires a
cache-disabled backend (--cache-none) with the video staged in its input dir;
documented on the manifest workflow field.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
b7708d5ad0 test: validate timeoutMs and requiresModels at manifest load 2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
564de12d46 test: harden custom-node suite per review findings
Normalize the executing event tap: its CustomEvent detail is a bare node-id
string, so the previous object-spread left the executed-set permanently empty.
The self-check now asserts a non-empty executed-set to keep that path live.

T0 now clears the graph per renderer pass, asserts exact node counts, and
verifies each added pack node's own data-node-id mounts in the Vue pass
(default-workflow nodes can no longer satisfy the assertions). Console capture
starts before the renderer toggle. Added an object_info sanity floor so a
depleted getNodeDefs fails loudly instead of skipping everything.

Run/io tiers gain test.setTimeout, requiresModels gating, and an empty-workflow
guard. Interrupted runs get pure-spec coverage; the shared console collector
moves to fixtures/utils.
2026-07-02 11:39:53 -07:00
Nathaniel Parson Koroso
5a1f788230 test: custom-node E2E regression suite (load/render, both renderers)
Data-driven Playwright harness verifying custom-node packs load and render
under both LiteGraph 1.0 and Vue Nodes 2.0 against a real ComfyUI backend.
Pure classifier/validator/manifest logic is unit-tested; the regression spec
renders each pack's nodes in both renderers (Vue via data-node-id DOM) and a
self-check runs a workflow to confirm execution-error capture. Proven against
Impact Pack + VideoHelperSuite.

Makes ComfyPage.createUser idempotent so the suite runs against a persistent
backend (Desktop server user storage).
2026-07-02 11:39:53 -07:00
36 changed files with 3720 additions and 2 deletions

View File

@@ -0,0 +1,162 @@
# Runs the custom-node regression suite against a backend that has the manifest
# packs actually installed, so the load/run tiers execute for real. This is a
# GATING check: if a pack fails to install or any tier is skipped, the job goes
# red - a regression gate that let a broken pack through as a "skip" would be
# pointless. Mark `custom-nodes-e2e` as a required status check in branch
# protection to block merges on failure.
name: 'CI: Tests Custom Nodes'
on:
pull_request:
branches-ignore: [wip/*, draft/*, temp/*]
push:
branches: [main, master]
merge_group:
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# Path gating lives here, not in a trigger-level `paths:` filter: a required
# check gated by trigger paths never creates a check run on an unrelated PR
# and leaves branch protection stuck Pending. A job-level `if:` still creates
# the check and marks it Skipped (= passing). Mirrors ci-tests-unit.yaml.
changes:
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
should-run: ${{ steps.changes.outputs.should-run }}
steps:
- uses: actions/checkout@v6
- id: changes
uses: ./.github/actions/changes-filter
# Deliberately NOT sharded yet: the suite is ~5.5 min but every shard would
# pay the full ~4.5 min setup (clone + pip-install every pack + boot the
# backend), so 2 shards buy ~2 min of wall time for double the runner cost
# and 4 shards are worse. Sharding pays once test time dwarfs setup time -
# first cut setup with a prebuilt image of the pinned packs, then shard if
# the job exceeds ~12 minutes.
custom-nodes-e2e:
needs: changes
# Run only when non-docs code changed AND the PR is same-repo. Fork PRs can
# edit the manifest's repo/pin URLs, and this job clones and pip-installs
# whatever they point at (setup.py runs at install time), so an untrusted
# fork must not be able to aim the clone at an attacker-controlled repo.
# Fork PRs still get the environment-agnostic coverage via the main e2e
# shards. A skipped job counts as passing, so this stays required-safe.
if: >-
needs.changes.outputs.should-run == 'true' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository)
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup frontend
uses: ./.github/actions/setup-frontend
with:
include_build_step: true
- name: Setup Playwright
uses: ./.github/actions/setup-playwright
# Checks out ComfyUI, installs Python/torch/requirements and ComfyUI_devtools.
# launch_server:false so we can add the manifest packs before booting.
- name: Setup ComfyUI server
uses: ./.github/actions/setup-comfyui-server
with:
launch_server: 'false'
# Install every pack the manifest declares (DRY: a new pack row installs
# itself here, no workflow change). A clone or dependency failure fails the
# job - if a pack can't be installed, its coverage can't run, and that is a
# gate failure, not something to paper over. The `jq | while` pipe hides
# failures in a subshell, so read into an array and loop with `set -e`.
- name: Install manifest custom nodes
shell: bash
run: |
set -euo pipefail
# Pin the CPU torch stack that setup-comfyui-server installed so no
# pack's requirements.txt can pull a GPU/incompatible torch onto this
# --cpu runner. A pack that genuinely needs a different torch fails
# the constrained install loudly rather than silently swapping it.
pip freeze | grep -iE '^(torch|torchvision|torchaudio)==' \
> /tmp/torch-constraints.txt || true
manifest=browser_tests/fixtures/data/customNodeManifest.json
mapfile -t entries < <(jq -c '.[]' "$manifest")
for entry in "${entries[@]}"; do
repo=$(jq -r '.repo' <<<"$entry")
pin=$(jq -r '.pin' <<<"$entry")
name=$(basename "$repo")
dir="ComfyUI/custom_nodes/$name"
echo "::group::install $name"
git clone --depth 1 "$repo" "$dir"
if [ -n "$pin" ]; then
git -C "$dir" fetch --depth 1 origin "$pin"
git -C "$dir" checkout "$pin"
fi
if [ -f "$dir/requirements.txt" ]; then
pip install -r "$dir/requirements.txt" -c /tmp/torch-constraints.txt
fi
echo "::endgroup::"
done
# The VHS run-tier workflow reads input/plain_video.mp4.
- name: Stage run-tier assets
shell: bash
run: cp browser_tests/assets/plain_video.mp4 ComfyUI/input/plain_video.mp4
# --cache-none so retried run-tier tests re-execute every node (a cached
# node emits no `executing` event and would false-fail PARTIAL).
- name: Start ComfyUI server
shell: bash
working-directory: ComfyUI
run: |
python main.py --cpu --multi-user --cache-none --front-end-root ../dist &
wait-for-it --service 127.0.0.1:8188 -t 600
- name: Run custom-node suite
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: custom-nodes-results.json
run: |
# workers=1: the auto-run tier needs exclusive backend-queue access;
# parallel workers interrupt each other's executions.
pnpm exec playwright test browser_tests/tests/customNodes/ \
--project=chromium --reporter=list,json --workers=1
# A skip here means a pack or devtools did not load: on this backend every
# tier is meant to run, so a skip is a gate failure, not an honest pass.
- name: Forbid skipped tests
if: always()
shell: bash
run: |
set -euo pipefail
skipped=$(jq '.stats.skipped' custom-nodes-results.json)
echo "skipped tests: $skipped"
if [ "$skipped" != "0" ]; then
echo "::error::$skipped test(s) skipped - a manifest pack or devtools failed to load; skips are not acceptable in the gating job"
# Recurse so specs nested under describe() blocks are found, and
# print only the specs that actually skipped.
jq -r '.. | objects
| select(has("title") and has("tests"))
| select(any(.tests[]?; .status == "skipped"))
| .title' custom-nodes-results.json | sort -u | head -40
exit 1
fi
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@v6
with:
name: playwright-report-custom-nodes
path: playwright-report/
retention-days: 7
if-no-files-found: warn

View File

@@ -123,6 +123,15 @@ Browser tests in this project follow a specific organization pattern:
- **Utilities**: Located in `utils/` - Common utility functions
- `litegraphUtils.ts` - Utilities for working with LiteGraph nodes
### Custom-node regression suite
`tests/customNodes/` holds the manifest-driven suite that proves community
custom-node packs load, render in both renderers (LiteGraph canvas and Vue
Nodes 2.0), and execute real workflows. It has its own prerequisites, pnpm
scripts (`pnpm test:custom-nodes` and per-pack variants), and a
one-JSON-row process for adding packs - see
[tests/customNodes/README.md](tests/customNodes/README.md).
## Writing Effective Tests
When writing new tests, follow these patterns:

View File

@@ -0,0 +1,53 @@
{
"last_node_id": 2,
"last_link_id": 1,
"nodes": [
{
"id": 1,
"type": "PrimitiveInt",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 80 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "INT",
"type": "INT",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "PrimitiveInt"
},
"widgets_values": [42, "fixed"]
},
{
"id": 2,
"type": "PreviewAny",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 1
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
}
],
"links": [[1, 1, 0, 2, 0, "INT"]],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,60 @@
{
"last_node_id": 2,
"last_link_id": 1,
"nodes": [
{
"id": 1,
"type": "StringFunction|pysssss",
"pos": { "0": 20, "1": 60 },
"size": { "0": 300, "1": 240 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "StringFunction|pysssss"
},
"widgets_values": ["append", "yes", "hello", " world", ""]
},
{
"id": 2,
"type": "ShowText|pysssss",
"pos": { "0": 380, "1": 60 },
"size": { "0": 220, "1": 80 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "text",
"type": "STRING",
"link": 1
}
],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"links": null,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ShowText|pysssss"
}
}
],
"links": [[1, 1, 0, 2, 0, "STRING"]],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,61 @@
{
"last_node_id": 2,
"last_link_id": 1,
"nodes": [
{
"id": 1,
"type": "SimpleMathInt+",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 60 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "INT",
"type": "INT",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "SimpleMathInt+"
},
"widgets_values": [5]
},
{
"id": 2,
"type": "DisplayAny",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 80 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "input",
"type": "*",
"link": 1
}
],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"links": null,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DisplayAny"
},
"widgets_values": ["raw value"]
}
],
"links": [[1, 1, 0, 2, 0, "INT"]],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,98 @@
{
"last_node_id": 4,
"last_link_id": 2,
"nodes": [
{
"id": 1,
"type": "ImpactInt",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 60 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "INT",
"type": "INT",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ImpactInt"
},
"widgets_values": [42]
},
{
"id": 2,
"type": "PreviewAny",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 1
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
},
{
"id": 3,
"type": "ImpactFloat",
"pos": { "0": 20, "1": 220 },
"size": { "0": 250, "1": 60 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "FLOAT",
"type": "FLOAT",
"links": [2],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ImpactFloat"
},
"widgets_values": [3.14]
},
{
"id": 4,
"type": "PreviewAny",
"pos": { "0": 340, "1": 220 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
}
],
"links": [
[1, 1, 0, 2, 0, "INT"],
[2, 3, 0, 4, 0, "FLOAT"]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,98 @@
{
"last_node_id": 4,
"last_link_id": 2,
"nodes": [
{
"id": 1,
"type": "INTConstant",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 60 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "value",
"type": "INT",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "INTConstant"
},
"widgets_values": [42]
},
{
"id": 2,
"type": "PreviewAny",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 1
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
},
{
"id": 3,
"type": "FloatConstant",
"pos": { "0": 20, "1": 220 },
"size": { "0": 250, "1": 60 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "value",
"type": "FLOAT",
"links": [2],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "FloatConstant"
},
"widgets_values": [3.14]
},
{
"id": 4,
"type": "PreviewAny",
"pos": { "0": 340, "1": 220 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
}
],
"links": [
[1, 1, 0, 2, 0, "INT"],
[2, 3, 0, 4, 0, "FLOAT"]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,53 @@
{
"last_node_id": 2,
"last_link_id": 1,
"nodes": [
{
"id": 1,
"type": "Seed (rgthree)",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 130 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "SEED",
"type": "INT",
"links": [1],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Seed (rgthree)"
},
"widgets_values": [12345]
},
{
"id": 2,
"type": "Display Any (rgthree)",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 1
}
],
"outputs": [],
"properties": {
"Node name for S&R": "Display Any (rgthree)"
}
}
],
"links": [[1, 1, 0, 2, 0, "INT"]],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,107 @@
{
"last_node_id": 3,
"last_link_id": 2,
"nodes": [
{
"id": 1,
"type": "VHS_LoadVideoPath",
"pos": { "0": 20, "1": 60 },
"size": { "0": 320, "1": 260 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": null
},
{
"name": "frame_count",
"type": "INT",
"links": null
},
{
"name": "audio",
"type": "AUDIO",
"links": null
},
{
"name": "video_info",
"type": "VHS_VIDEOINFO",
"links": [1],
"slot_index": 3
}
],
"properties": {
"Node name for S&R": "VHS_LoadVideoPath"
},
"widgets_values": ["input/plain_video.mp4", 0, 0, 0, 0, 0, 1]
},
{
"id": 2,
"type": "VHS_VideoInfo",
"pos": { "0": 400, "1": 60 },
"size": { "0": 240, "1": 260 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "video_info",
"type": "VHS_VIDEOINFO",
"link": 1
}
],
"outputs": [
{
"name": "source_fps🟨",
"type": "FLOAT",
"links": [2],
"slot_index": 0
},
{ "name": "source_frame_count🟨", "type": "INT", "links": null },
{ "name": "source_duration🟨", "type": "FLOAT", "links": null },
{ "name": "source_width🟨", "type": "INT", "links": null },
{ "name": "source_height🟨", "type": "INT", "links": null },
{ "name": "loaded_fps🟦", "type": "FLOAT", "links": null },
{ "name": "loaded_frame_count🟦", "type": "INT", "links": null },
{ "name": "loaded_duration🟦", "type": "FLOAT", "links": null },
{ "name": "loaded_width🟦", "type": "INT", "links": null },
{ "name": "loaded_height🟦", "type": "INT", "links": null }
],
"properties": {
"Node name for S&R": "VHS_VideoInfo"
}
},
{
"id": 3,
"type": "PreviewAny",
"pos": { "0": 700, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "source",
"type": "*",
"link": 2
}
],
"outputs": [],
"properties": {
"Node name for S&R": "PreviewAny"
}
}
],
"links": [
[1, 1, 3, 2, 0, "VHS_VIDEOINFO"],
[2, 2, 0, 3, 0, "FLOAT"]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -0,0 +1,103 @@
{
"last_node_id": 3,
"last_link_id": 2,
"nodes": [
{
"id": 1,
"type": "Constant Number",
"pos": { "0": 20, "1": 60 },
"size": { "0": 250, "1": 100 },
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "NUMBER",
"type": "NUMBER",
"links": [1],
"slot_index": 0
},
{
"name": "FLOAT",
"type": "FLOAT",
"links": null,
"slot_index": 1
},
{
"name": "INT",
"type": "INT",
"links": null,
"slot_index": 2
}
],
"properties": {
"Node name for S&R": "Constant Number"
},
"widgets_values": ["integer", 7]
},
{
"id": 2,
"type": "Number to Text",
"pos": { "0": 340, "1": 60 },
"size": { "0": 220, "1": 60 },
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "number",
"type": "NUMBER",
"link": 1
}
],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"links": [2],
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Number to Text"
}
},
{
"id": 3,
"type": "Text to Console",
"pos": { "0": 640, "1": 60 },
"size": { "0": 250, "1": 80 },
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "text",
"type": "STRING",
"link": 2
}
],
"outputs": [
{
"name": "STRING",
"type": "STRING",
"links": null,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Text to Console"
},
"widgets_values": ["Text Output"]
}
],
"links": [
[1, 1, 0, 2, 0, "NUMBER"],
[2, 2, 0, 3, 0, "STRING"]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}

View File

@@ -268,8 +268,16 @@ export class ComfyPage {
data: { username }
})
if (resp.status() !== 200)
throw new Error(`Failed to create user: ${await resp.text()}`)
if (resp.status() !== 200) {
const body = await resp.text()
// Persistent backends (Comfy Desktop server user storage) keep the user
// across runs and do not list it via GET /api/users, so a duplicate means
// it already exists. Returns the username since the generated id is not
// retrievable here; only reached on single-user / default-resolving backends.
if (resp.status() === 400 && body.includes('Duplicate username.'))
return username
throw new Error(`Failed to create user: ${body}`)
}
return await resp.json()
}

View File

@@ -0,0 +1,136 @@
import type { Page } from '@playwright/test'
import type { ObjectInfo } from '@e2e/fixtures/customNode/objectInfoValidator'
import type {
ExecutionError,
PromptEvent,
RunResult
} from '@e2e/fixtures/customNode/runResult'
import { classifyRun } from '@e2e/fixtures/customNode/runResult'
interface RawEvent {
type: string
node?: string | null
exception_type?: string
node_id?: string
node_type?: string
traceback?: string[]
}
const TERMINAL = [
'execution_success',
'execution_error',
'execution_interrupted'
]
function toPromptEvent(raw: RawEvent): PromptEvent {
if (raw.type === 'executing')
return { type: 'executing', node: raw.node ?? null }
if (raw.type === 'execution_error' || raw.type === 'execution_interrupted') {
const error: ExecutionError = {
exceptionType: raw.exception_type,
nodeId: raw.node_id,
nodeType: raw.node_type,
traceback: raw.traceback
}
return { type: raw.type, error }
}
return { type: raw.type as 'execution_start' | 'execution_success' }
}
/**
* Drives a real ComfyUI backend through the running frontend. The verdict logic
* lives in the pure `classifyRun`; this class is only the in-page IO plumbing.
*/
export class LocalDesktopTarget {
async getObjectInfo(page: Page): Promise<ObjectInfo> {
return await page.evaluate(async () => {
const defs = await window.app!.api.getNodeDefs()
const out: Record<
string,
{ input?: { required?: Record<string, unknown> } }
> = {}
for (const [name, def] of Object.entries(defs)) {
const required = (
def as { input?: { required?: Record<string, unknown> } }
).input?.required
out[name] = { input: { required } }
}
return out
})
}
async runWorkflow(
page: Page,
opts: { expectedNodeIds: string[]; timeoutMs: number }
): Promise<RunResult> {
await page.evaluate(
(types) => {
const sink = window as unknown as {
__cnEvents: RawEvent[]
__cnTapInstalled?: boolean
}
sink.__cnEvents = []
if (sink.__cnTapInstalled) return
sink.__cnTapInstalled = true
for (const type of types)
(window.app!.api as EventTarget).addEventListener(
type,
(event: Event) => {
const detail: unknown = (event as CustomEvent).detail
// `executing` dispatches a bare node-id string (api.ts
// dispatchCustomEvent('executing', msg.data.node)); the other
// events dispatch object payloads.
sink.__cnEvents.push(
detail !== null && typeof detail === 'object'
? { type, ...(detail as Record<string, unknown>) }
: { type, node: (detail as string | undefined) ?? null }
)
}
)
},
['execution_start', ...TERMINAL, 'executing']
)
// app.queuePrompt (NOT api.queuePrompt: that submits an empty prompt).
// false = validation reject (emits no events), but pack JS hooking the
// queue can refuse transiently - retry once; real rejects fail twice.
let queued = await page.evaluate(() => window.app!.queuePrompt(0))
if (queued === false) {
await page.evaluate(
() => new Promise((resolve) => setTimeout(resolve, 250))
)
queued = await page.evaluate(() => window.app!.queuePrompt(0))
if (queued === false)
return { outcome: 'VALIDATION_FAIL', executedNodes: [] }
}
await page
.waitForFunction(
(terminal) => {
const events =
(window as unknown as { __cnEvents?: { type: string }[] })
.__cnEvents ?? []
return events.some((event) => terminal.includes(event.type))
},
TERMINAL,
{ timeout: opts.timeoutMs }
)
.catch((error: unknown) => {
// Only a Playwright wait timeout means "no terminal event"; surface any
// other fault instead of masquerading it as a run TIMEOUT.
if (error instanceof Error && error.name === 'TimeoutError') return
throw error
})
const raw = await page.evaluate(
() => (window as unknown as { __cnEvents?: RawEvent[] }).__cnEvents ?? []
)
const timedOut = !raw.some((event) => TERMINAL.includes(event.type))
return classifyRun({
events: raw.map(toPromptEvent),
expectedNodeIds: opts.expectedNodeIds,
timedOut
})
}
}

View File

@@ -0,0 +1,102 @@
// Classifies which nodes can execute with no hand-authored fixture; the
// rest are recorded with the reason, never silently dropped.
import type { RawNodeDef } from './typePairing'
type AutoRunClass =
// Widgets cover every required input and a terminus exists.
| 'AUTO_RUNNABLE'
// A required input is a socket; needs wiring (curated workflows).
| 'NEEDS_WIRES'
// A required combo has zero options (empty model/file scan).
| 'NEEDS_MODELS'
// No outputs and not an OUTPUT_NODE - nothing the executor could watch.
| 'NO_OBSERVABLE_OUTPUT'
export interface AutoRunVerdict {
key: string
verdict: AutoRunClass
// Set for AUTO_RUNNABLE: wire output 0 to PreviewAny (false = the node is
// its own OUTPUT_NODE terminus and runs standalone).
needsPreviewSink?: boolean
reason: string
}
const WIDGET_TYPES = new Set(['INT', 'FLOAT', 'STRING', 'BOOLEAN'])
type InputSpec = [unknown, Record<string, unknown>?] | unknown
function classifyInput(
name: string,
spec: InputSpec
): 'widget' | 'socket' | 'empty-combo' {
const specArray = Array.isArray(spec) ? spec : [spec]
const rawType = specArray[0]
const options = specArray[1] as { forceInput?: boolean } | undefined
if (Array.isArray(rawType))
return rawType.length > 0 ? 'widget' : 'empty-combo'
if (typeof rawType !== 'string') return 'socket'
if (options?.forceInput) return 'socket'
return WIDGET_TYPES.has(rawType) ? 'widget' : 'socket'
}
export function classifyAutoRunnable(
key: string,
def: RawNodeDef & { output_node?: boolean }
): AutoRunVerdict {
for (const [name, spec] of Object.entries(def.input?.required ?? {})) {
const kind = classifyInput(name, spec)
if (kind === 'socket')
return {
key,
verdict: 'NEEDS_WIRES',
reason: `required input "${name}" is a socket`
}
if (kind === 'empty-combo')
return {
key,
verdict: 'NEEDS_MODELS',
reason: `required combo "${name}" has no options on this backend`
}
}
if (def.output_node === true)
return {
key,
verdict: 'AUTO_RUNNABLE',
needsPreviewSink: false,
reason: 'widgets satisfy all required inputs; node is its own terminus'
}
if ((def.output ?? []).length > 0)
return {
key,
verdict: 'AUTO_RUNNABLE',
needsPreviewSink: true,
reason: 'widgets satisfy all required inputs; output 0 -> PreviewAny'
}
return {
key,
verdict: 'NO_OBSERVABLE_OUTPUT',
reason: 'no outputs and not an OUTPUT_NODE - nothing observable to queue'
}
}
export function planAutoRuns(
defs: Record<string, RawNodeDef & { output_node?: boolean }>,
packNodeKeys: string[]
): AutoRunVerdict[] {
return packNodeKeys.map((key) => classifyAutoRunnable(key, defs[key]))
}
// Independent single-node chains per prompt so one bad node fails a batch,
// not the tier.
export function batchAutoRunnable(
verdicts: AutoRunVerdict[],
batchSize: number
): AutoRunVerdict[][] {
const runnable = verdicts.filter(
(verdict) => verdict.verdict === 'AUTO_RUNNABLE'
)
const batches: AutoRunVerdict[][] = []
for (let offset = 0; offset < runnable.length; offset += batchSize)
batches.push(runnable.slice(offset, offset + batchSize))
return batches
}

View File

@@ -0,0 +1,116 @@
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
const MANIFEST_PATH = fileURLToPath(
new URL('../data/customNodeManifest.json', import.meta.url)
)
const VALID_TIERS = ['load', 'run', 'connectivity', 'io'] as const
type CustomNodeTier = (typeof VALID_TIERS)[number]
export interface CustomNodeManifestEntry {
pack: string
repo: string
pin: string
tiers: CustomNodeTier[]
// Frontend-format workflow (path relative to browser_tests/) loaded and queued
// by the run/io tiers; empty or absent file = tier skips. Run the backend with
// --cache-none, or repeat runs classify PARTIAL when cached nodes skip executing.
workflow: string
// Runtime class_type / object_info keys, NOT Python class names (e.g. rgthree
// registers "Power Primitive (rgthree)", not RgthreePowerPrimitive).
expectedNodes: string[]
requiresGpu: boolean
requiresModels: string[]
timeoutMs: number
// Optional; absent means true. Set false ONLY with evidence that the pack's
// nodes fail to mount under Vue Nodes 2.0 (probe it - a README grumble is
// not evidence). When false, renderer-specific Vue assertions are not
// applied to this pack: its tests still run and pass their LiteGraph-canvas
// assertions, so the zero-skip gate is preserved.
vueNodesCompatible?: boolean
// Node key -> evidenced reason it cannot mount under Vue Nodes 2.0; only
// the Vue mount assertion is withheld. Stale keys fail the suite.
vueIncompatibleNodes?: Record<string, string>
// Nodes that cannot execute on pure defaults. Asserted both ways: an
// unlisted failure is a regression, a listed clean run is a stale entry.
cannotRunAlone?: string[]
}
function assertEntry(entry: CustomNodeManifestEntry, index: number): void {
const missing: string[] = []
if (typeof entry.pack !== 'string' || entry.pack.length === 0)
missing.push('pack')
// CI clones from repo, so an empty value must fail here, not mid-clone.
// pin stays optional ("" = default branch head).
if (typeof entry.repo !== 'string' || entry.repo.length === 0)
missing.push('repo')
// workflow may be an empty string until the pack gains a run-tier fixture.
if (typeof entry.workflow !== 'string') missing.push('workflow')
// A run-tier row with no workflow would otherwise skip locally, leaving
// only CI's skip gate to notice the lost coverage. Fail at load instead.
else if (
entry.workflow === '' &&
Array.isArray(entry.tiers) &&
entry.tiers.includes('run')
)
missing.push('workflow (required when tiers includes "run")')
if (!Array.isArray(entry.expectedNodes) || entry.expectedNodes.length === 0)
missing.push('expectedNodes')
if (!Array.isArray(entry.tiers) || entry.tiers.length === 0)
missing.push('tiers')
// A typo like "connectivty" would otherwise pass and silently drop that
// tier's coverage - the exact drift this manifest exists to catch.
else if (entry.tiers.some((tier) => !VALID_TIERS.includes(tier)))
missing.push(`tiers (unknown value; allowed: ${VALID_TIERS.join(', ')})`)
if (!Array.isArray(entry.requiresModels)) missing.push('requiresModels')
if (typeof entry.requiresGpu !== 'boolean') missing.push('requiresGpu')
if (!Number.isFinite(entry.timeoutMs) || entry.timeoutMs <= 0)
missing.push('timeoutMs')
if (
entry.vueNodesCompatible !== undefined &&
typeof entry.vueNodesCompatible !== 'boolean'
)
missing.push('vueNodesCompatible')
if (
entry.vueIncompatibleNodes !== undefined &&
(typeof entry.vueIncompatibleNodes !== 'object' ||
entry.vueIncompatibleNodes === null ||
Array.isArray(entry.vueIncompatibleNodes) ||
Object.values(entry.vueIncompatibleNodes).some(
(reason) => typeof reason !== 'string' || reason.length === 0
))
)
missing.push('vueIncompatibleNodes (node key -> non-empty reason string)')
if (
entry.cannotRunAlone !== undefined &&
(!Array.isArray(entry.cannotRunAlone) ||
entry.cannotRunAlone.some(
(key) => typeof key !== 'string' || key.length === 0
) ||
new Set(entry.cannotRunAlone).size !== entry.cannotRunAlone.length)
)
missing.push('cannotRunAlone (unique non-empty node keys)')
if (missing.length > 0)
throw new Error(
`custom-node manifest entry ${index} (${entry.pack ?? '?'}) missing: ${missing.join(', ')}`
)
}
// Renderer passes for the load tier: LiteGraph canvas always, Vue Nodes 2.0
// unless the pack declares itself incompatible. Conditional coverage, never a
// test.skip - the caller still runs and gates on the returned passes.
export function rendererPassesFor(
entry: Pick<CustomNodeManifestEntry, 'vueNodesCompatible'>
): boolean[] {
return entry.vueNodesCompatible === false ? [false] : [false, true]
}
export function loadManifest(): CustomNodeManifestEntry[] {
const entries = JSON.parse(
readFileSync(MANIFEST_PATH, 'utf-8')
) as CustomNodeManifestEntry[]
entries.forEach(assertEntry)
return entries
}

View File

@@ -0,0 +1,54 @@
import type { CustomNodeOutcome } from '@e2e/fixtures/customNode/runResult'
interface ObjectInfoNode {
input?: { required?: Record<string, unknown> }
}
export type ObjectInfo = Record<string, ObjectInfoNode>
export interface ApiPromptNode {
id: string
classType: string
inputs: Record<string, unknown>
}
export function expectedNodesPresent(
objectInfo: ObjectInfo,
expectedNodes: string[]
): { present: string[]; missing: string[] } {
const present: string[] = []
const missing: string[] = []
for (const name of expectedNodes) {
if (name in objectInfo) present.push(name)
else missing.push(name)
}
return { present, missing }
}
export interface PreValidationFailure {
outcome: Extract<CustomNodeOutcome, 'MISSING_NODE' | 'VALIDATION_FAIL'>
message: string
}
// Turns an opaque backend 400 into a precise infra error before submit (BE-401):
// every required input declared in object_info must be present in the fixture node.
export function preValidate(
objectInfo: ObjectInfo,
nodes: ApiPromptNode[]
): PreValidationFailure | null {
for (const node of nodes) {
const def = objectInfo[node.classType]
if (!def)
return {
outcome: 'MISSING_NODE',
message: `node ${node.id} ${node.classType} missing from object_info`
}
for (const name of Object.keys(def.input?.required ?? {})) {
if (!(name in node.inputs))
return {
outcome: 'VALIDATION_FAIL',
message: `node ${node.id} ${node.classType} missing required input "${name}"`
}
}
}
return null
}

View File

@@ -0,0 +1,72 @@
export type CustomNodeOutcome =
| 'NOT_INSTALLED'
| 'IMPORT_ERROR'
| 'MISSING_NODE'
| 'VALIDATION_FAIL'
| 'EXECUTION_ERROR'
| 'PARTIAL'
| 'TIMEOUT'
| 'PASS'
export interface ExecutionError {
exceptionType?: string
nodeId?: string
nodeType?: string
traceback?: string[]
}
export type PromptEvent =
| { type: 'execution_start' }
| { type: 'executing'; node: string | null }
| { type: 'execution_success' }
| { type: 'execution_error'; error: ExecutionError }
| { type: 'execution_interrupted'; error?: ExecutionError }
export interface RunResult {
outcome: CustomNodeOutcome
executedNodes: string[]
error?: ExecutionError
}
// `executing` with a non-null node is the only cache-safe "this node actually ran"
// signal: ComfyUI emits it solely for non-cached nodes (execution.py:493), while the
// `executed` message and /history outputs are replayed for cached nodes too.
function executedNodesFrom(events: PromptEvent[]): string[] {
const executed = new Set<string>()
for (const event of events) {
if (event.type === 'executing' && event.node !== null)
executed.add(event.node)
}
return [...executed]
}
export function classifyRun(input: {
events: PromptEvent[]
expectedNodeIds: string[]
timedOut?: boolean
}): RunResult {
const { events, expectedNodeIds, timedOut = false } = input
const executedNodes = executedNodesFrom(events)
if (timedOut) return { outcome: 'TIMEOUT', executedNodes }
const failure = events.find(
(
event
): event is Extract<
PromptEvent,
{ type: 'execution_error' | 'execution_interrupted' }
> =>
event.type === 'execution_error' || event.type === 'execution_interrupted'
)
if (failure)
return { outcome: 'EXECUTION_ERROR', executedNodes, error: failure.error }
if (!events.some((event) => event.type === 'execution_success'))
return { outcome: 'TIMEOUT', executedNodes }
const ranEveryExpected = expectedNodeIds.every((node) =>
executedNodes.includes(node)
)
return { outcome: ranEveryExpected ? 'PASS' : 'PARTIAL', executedNodes }
}

View File

@@ -0,0 +1,216 @@
// Type-driven pairing generator for the connectivity (contract) tier.
// Wildcard `*` slots are excluded from pairing: LiteGraph.isValidConnection
// short-circuits on `*` before the real type compare, so a wildcard link
// proves reachability, not type interop.
export interface RawNodeDef {
input?: {
required?: Record<string, unknown>
optional?: Record<string, unknown>
}
output?: unknown[]
output_name?: string[]
python_module?: string
}
interface NormalizedSlot {
name: string
type: string
}
export interface NormalizedNode {
type: string
pack: string
inputs: NormalizedSlot[]
outputs: NormalizedSlot[]
}
interface SlotRef {
nodeType: string
pack: string
slotName: string
slotType: string
}
export interface PlannedPair {
producer: SlotRef
consumer: SlotRef
}
export interface PairingPlan {
pairs: PlannedPair[]
// No compatible partner in the loaded corpus: a health signal, not a failure.
orphans: Array<SlotRef & { dir: 'in' | 'out' }>
// `*` / empty-typed slots, excluded by design (false confidence).
wildcards: Array<SlotRef & { dir: 'in' | 'out' }>
// COMBO-literal slots, excluded by design: isValidConnection only compares
// the string COMBO while each slot carries its own option set, so a
// type-level pairing proves nothing (a checkpoint dropdown would "connect"
// to a scheduler dropdown). Targeted fixtures cover combo behavior.
combos: Array<SlotRef & { dir: 'in' | 'out' }>
}
// Extends the shared outcome taxonomy (runResult.ts); ORPHAN_TYPE is a
// plan-time skip so it never reaches the executor.
// WIDGET_ONLY_ON_INSTANCE: the pack's own frontend JS rebuilt a declared
// input as a widget-only control, so there is no socket to wire - excluded
// like wildcards, never a failure and never a silent pass.
export type ConnectivityOutcome =
| 'PASS'
| 'CONNECT_REJECTED'
| 'ROUNDTRIP_LOST'
| 'SLOT_CONTRACT_MISMATCH'
| 'WIDGET_ONLY_ON_INSTANCE'
export function packOf(pythonModule: string | undefined): string {
if (pythonModule?.startsWith('custom_nodes.'))
return pythonModule.slice('custom_nodes.'.length)
return 'core'
}
export function isWildcard(type: string): boolean {
return type === '' || type === '*'
}
// COMBO list literals are arrays; their connectable socket type is COMBO.
function slotTypeOf(rawType: unknown): string | null {
if (Array.isArray(rawType)) return 'COMBO'
return typeof rawType === 'string' ? rawType : null
}
function inputSlots(
entries: Record<string, unknown> | undefined
): NormalizedSlot[] {
if (!entries) return []
const slots: NormalizedSlot[] = []
for (const [name, spec] of Object.entries(entries)) {
const specArray = Array.isArray(spec) ? spec : [spec]
const type = slotTypeOf(specArray[0])
if (type === null) continue
const opts = specArray[1] as { socketless?: boolean } | undefined
// socketless = widget only, no slot: not connectable, out of the matrix.
if (opts?.socketless) continue
slots.push({ name, type })
}
return slots
}
export function normalizeNodeDefs(
defs: Record<string, RawNodeDef>
): NormalizedNode[] {
return Object.entries(defs).map(([type, def]) => ({
type,
pack: packOf(def.python_module),
inputs: [
...inputSlots(def.input?.required),
...inputSlots(def.input?.optional)
],
outputs: (def.output ?? []).flatMap((rawType, index) => {
const slotType = slotTypeOf(rawType)
if (slotType === null) return []
// output_name entries can be non-strings (COMBO literals repeat the
// option array); the slot name must stay a string.
const rawName = def.output_name?.[index]
return [
{
name: typeof rawName === 'string' ? rawName : slotType,
type: slotType
}
]
})
}))
}
// Faithful mirror of LiteGraph.isValidConnection (LiteGraphGlobal.ts):
// wildcard/empty always match, comparison is case-insensitive, comma-unions
// match if any member pair matches. The live sweep still connects through the
// REAL validator, so any drift here surfaces as CONNECT_REJECTED, not a
// silent false green.
export function isTypeCompatible(a: string, b: string): boolean {
if (isWildcard(a) || isWildcard(b)) return true
const typeA = a.toLowerCase()
const typeB = b.toLowerCase()
if (typeA === typeB) return true
if (!typeA.includes(',') && !typeB.includes(',')) return false
return typeA
.split(',')
.some((memberA) =>
typeB.split(',').some((memberB) => isTypeCompatible(memberA, memberB))
)
}
function slotRef(node: NormalizedNode, slot: NormalizedSlot): SlotRef {
return {
nodeType: node.type,
pack: node.pack,
slotName: slot.name,
slotType: slot.type
}
}
// One representative compatible edge per slot, deterministically the first
// partner in (nodeType, slotName) order. This bounds cost to O(slots) but
// does NOT prove every pair; a full cross-product is an opt-in deep mode.
export function planPairs(
all: NormalizedNode[],
corpusTypes: string[]
): PairingPlan {
const sorted = [...all].sort((a, b) => a.type.localeCompare(b.type))
const pairable = (slot: NormalizedSlot) =>
!isWildcard(slot.type) && slot.type !== 'COMBO'
const producers: Array<SlotRef> = sorted.flatMap((node) =>
node.outputs.filter(pairable).map((slot) => slotRef(node, slot))
)
const consumers: Array<SlotRef> = sorted.flatMap((node) =>
node.inputs.filter(pairable).map((slot) => slotRef(node, slot))
)
const plan: PairingPlan = {
pairs: [],
orphans: [],
wildcards: [],
combos: []
}
const seen = new Set<string>()
const addPair = (producer: SlotRef, consumer: SlotRef) => {
const key = `${producer.nodeType}.${producer.slotName}->${consumer.nodeType}.${consumer.slotName}`
if (seen.has(key)) return
seen.add(key)
plan.pairs.push({ producer, consumer })
}
const corpus = all.filter((node) => corpusTypes.includes(node.type))
for (const node of corpus) {
for (const slot of node.inputs) {
if (isWildcard(slot.type)) {
plan.wildcards.push({ ...slotRef(node, slot), dir: 'in' })
continue
}
if (slot.type === 'COMBO') {
plan.combos.push({ ...slotRef(node, slot), dir: 'in' })
continue
}
const producer = producers.find((candidate) =>
isTypeCompatible(candidate.slotType, slot.type)
)
if (producer) addPair(producer, slotRef(node, slot))
else plan.orphans.push({ ...slotRef(node, slot), dir: 'in' })
}
for (const slot of node.outputs) {
if (isWildcard(slot.type)) {
plan.wildcards.push({ ...slotRef(node, slot), dir: 'out' })
continue
}
if (slot.type === 'COMBO') {
plan.combos.push({ ...slotRef(node, slot), dir: 'out' })
continue
}
const consumer = consumers.find((candidate) =>
isTypeCompatible(slot.type, candidate.slotType)
)
if (consumer) addPair(slotRef(node, slot), consumer)
else plan.orphans.push({ ...slotRef(node, slot), dir: 'out' })
}
}
return plan
}

View File

@@ -0,0 +1,132 @@
[
{
"pack": "ComfyUI-Impact-Pack",
"repo": "https://github.com/ltdrdata/ComfyUI-Impact-Pack",
"pin": "429d0159ad429e64d2b3916e6e7be9c22d025c3c",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/impact_primitives_run.json",
"expectedNodes": ["ImpactInt", "ImpactFloat"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": [
"CLIPSegDetectorProvider",
"ImpactMakeImageBatch",
"ImpactMakeMaskBatch",
"MasksToMaskList",
"NoiseInjectionDetailerHookProvider"
]
},
{
"pack": "ComfyUI-VideoHelperSuite",
"repo": "https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite",
"pin": "4ee72c065db22c9d96c2427954dc69e7b908444b",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/vhs_video_pipeline_run.json",
"expectedNodes": ["VHS_LoadVideoPath", "VHS_VideoInfo"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 90000,
"cannotRunAlone": [
"VHS_LoadAudio",
"VHS_LoadImagePath",
"VHS_LoadImages",
"VHS_LoadImagesPath",
"VHS_LoadVideoFFmpegPath",
"VHS_LoadVideoPath",
"VHS_SelectLatest"
]
},
{
"pack": "rgthree-comfy",
"repo": "https://github.com/rgthree/rgthree-comfy",
"pin": "27b4f4cdcf3b127c29d5d8135ac1536ecbd4c383",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/rgthree_seed_display_run.json",
"expectedNodes": ["Seed (rgthree)", "Display Any (rgthree)"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": ["Image or Latent Size (rgthree)"]
},
{
"pack": "ComfyUI_essentials",
"repo": "https://github.com/cubiq/ComfyUI_essentials",
"pin": "9d9f4bedfc9f0321c19faf71855e228c93bd0dc9",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/essentials_math_display_run.json",
"expectedNodes": ["SimpleMathInt+", "DisplayAny"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": [
"MaskFromList+",
"SimpleMath+",
"SimpleMathDual+",
"SimpleMathFloat+",
"SimpleMathInt+",
"SimpleMathPercent+",
"SimpleMathSlider+",
"SimpleMathSliderLowRes+"
]
},
{
"pack": "ComfyUI-KJNodes",
"repo": "https://github.com/kijai/ComfyUI-KJNodes",
"pin": "e27a505b3ba6ce42687fe00500deda103d9d6071",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/kjnodes_constants_run.json",
"expectedNodes": ["INTConstant", "FloatConstant"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": [
"CameraPoseVisualizer",
"CreateAudioMask",
"CreateMagicMask",
"CreateVoronoiMask",
"GenerateNoise",
"ImageAndMaskPreview",
"LoadImagesFromFolderKJ",
"LoadVideosFromFolder",
"MaskOrImageToWeight",
"VisualizeCUDAMemoryHistory",
"WebcamCaptureCV2",
"WidgetToString"
]
},
{
"pack": "ComfyUI-Custom-Scripts",
"repo": "https://github.com/pythongosssss/ComfyUI-Custom-Scripts",
"pin": "609f3afaa74b2f88ef9ce8d939626065e3247469",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/customscripts_string_show_run.json",
"expectedNodes": ["StringFunction|pysssss", "ShowText|pysssss"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": ["LoadText|pysssss", "MathExpression|pysssss"]
},
{
"pack": "was-node-suite-comfyui",
"repo": "https://github.com/WASasquatch/was-node-suite-comfyui",
"pin": "ea935d1044ae5a26efa54ebeb18fe9020af49a45",
"tiers": ["load", "connectivity", "run"],
"workflow": "assets/customNodes/was_number_text_run.json",
"expectedNodes": ["Constant Number", "Number to Text", "Text to Console"],
"requiresGpu": false,
"requiresModels": [],
"timeoutMs": 30000,
"cannotRunAlone": [
"Bus Node",
"Diffusers Hub Model Down-Loader",
"Image Aspect Ratio",
"Image Batch",
"Latent Batch",
"Mask Batch",
"Mask Rect Area",
"Number Counter",
"Random Number"
]
}
]

View File

@@ -0,0 +1,15 @@
import type { ConsoleMessage, Page } from '@playwright/test'
export function collectConsoleErrors(page: Page): {
errors: string[]
stop: () => void
} {
const errors: string[] = []
const listener = (message: ConsoleMessage) => {
if (message.type() !== 'error') return
const url = message.location().url
errors.push(url ? `${message.text()} [${url}]` : message.text())
}
page.on('console', listener)
return { errors, stop: () => page.off('console', listener) }
}

View File

@@ -0,0 +1,27 @@
import type { ComfyPage } from '@e2e/fixtures/ComfyPage'
import { TestIds } from '@e2e/fixtures/selectors'
// Boot every session with a blank graph (loadBlankWorkflow) instead of the
// bundled default template, whose model references error on a model-less
// harness backend and would trip the zero-visible-errors invariant. The
// backend must run --multi-user (the repo-wide prerequisite for browser
// tests): the fixture then writes these settings to the same per-worker
// user the session reads, on CI and locally alike.
// The shared fixture disables the errors tab to hide missing-model
// indicators in unrelated suites; this suite exists to SEE errors, so every
// error surface stays live.
export const customNodeSuiteSettings = {
'Comfy.TutorialCompleted': false,
'Comfy.RightSidePanel.ShowErrorsTab': true
}
// The tutorial path auto-opens the templates browser over the blank graph.
// Dismiss it deterministically so no window ever shows unexpected UI.
export async function dismissTemplatesDialog(
comfyPage: ComfyPage
): Promise<void> {
const templates = comfyPage.page.getByTestId(TestIds.templates.content)
await templates.waitFor({ state: 'visible' })
await comfyPage.page.keyboard.press('Escape')
await templates.waitFor({ state: 'hidden' })
}

View File

@@ -0,0 +1,16 @@
import type { Locator, Page } from '@playwright/test'
import { TestIds } from '@e2e/fixtures/selectors'
// The app's user-visible error surfaces. A regression run is green only if a
// human looking at the screen would see zero errors - not merely a clean
// console. The harness self-check asserts the overlay IS visible after a
// forced execution error, so these selectors are permanently proven live.
export function errorSurfaces(page: Page): Record<string, Locator> {
return {
errorOverlay: page.getByTestId(TestIds.dialogs.errorOverlay),
errorDialog: page.getByTestId(TestIds.dialogs.errorDialog),
nodeRenderErrors: page.locator('.node-error'),
errorToasts: page.locator('.p-toast-message-error')
}
}

View File

@@ -119,6 +119,11 @@ class NodeSlotReference {
const rawPos = node.getConnectionPos(type === 'input', index)
const convertedPos =
window.app!.canvas.ds!.convertOffsetToCanvas(rawPos)
// page.mouse needs page coords; pack JS can inject chrome above the
// canvas (rgthree's progress bar), shifting it off (0,0).
const rect = window.app!.canvas.canvas.getBoundingClientRect()
convertedPos[0] += rect.left
convertedPos[1] += rect.top
// Debug logging - convert Float64Arrays to regular arrays for visibility
console.warn(

View File

@@ -0,0 +1,332 @@
# Adding a custom-node pack to the regression suite
The authoritative, step-by-step process for onboarding a new pack. Written to
be followable by a human or an agent with no prior context. The suite itself
(what it asserts, how to run it) is documented in [README.md](README.md);
this file is only about adding coverage for a new pack.
The short version: install the pack on a local test backend, read the pack's
real node keys out of `/object_info`, author one small model-free workflow,
add one row to the manifest, prove it green locally, push. No new test code
is ever needed - the specs iterate the manifest.
## What a manifest row buys you (the tiers)
Adding the one row enrolls the pack in two kinds of coverage:
- **Every-node tiers (automatic, zero configuration).** The suite reads the
pack's FULL node list from the live backend and, for every registered
node: mounts it in both renderers, round-trips it through save/reload,
plans typed connections for all its concrete slots, and executes it for
real when it is self-sufficient (every required input is a widget with a
valid default; output wired to `PreviewAny` or the node is its own
terminus). Nodes that cannot run alone are classified and logged, never
silently dropped: `NEEDS_WIRES` (required socket inputs), `NEEDS_MODELS`
(empty model/file combo on the bare backend), `NO_OBSERVABLE_OUTPUT` (nothing
observable to queue), or "rejected at validation on defaults" (needs a
curated fixture).
- **Curated tiers (the row's fields).** `expectedNodes` + `workflow` drive
the hand-authored run-tier chain (Step 4) proving a real multi-node
wiring executes end to end, and serve as must-exist sentinels.
Every-node coverage means a pack update is tested the moment CI installs
it - including nodes you never listed.
## Step 0 - prerequisites
- A local test backend and dev server set up exactly per the
[README prerequisites](README.md#prerequisites). Do not skip `--multi-user`
or `--cache-none`.
- The pack's GitHub URL. The CI job clones and pip-installs it, so the repo
must be public and its `requirements.txt` must install on a CPU-only
runner. Packs that hard-require CUDA at import time cannot be onboarded
until they guard that import.
## Step 1 - install the pack on the test backend
```bash
cd <test-backend>/custom_nodes
git clone https://github.com/<owner>/<pack>
pip install -r <pack>/requirements.txt # if the pack has one
```
If you run a CPU-only backend, constrain pip so the pack cannot swap in a
different torch (CI does the same):
```bash
pip freeze | grep -iE '^(torch|torchvision|torchaudio)==' > /tmp/torch-constraints.txt
pip install -r <pack>/requirements.txt -c /tmp/torch-constraints.txt
```
Restart the backend and check its log: the `Import times for custom nodes`
block must list the pack with no `IMPORT FAILED` marker. An import failure is
a pack bug or a missing dependency - fix that first; nothing downstream can
work without a clean import.
While you are here, note whether the pack ships frontend JS:
```bash
curl -s http://127.0.0.1:8288/extensions | python3 -c '
import json, sys
print(sum(1 for p in json.load(sys.stdin) if p.startswith("/extensions/<pack-dir-name>/")))
'
```
Non-zero means the pack patches the frontend at runtime (restyled nodes,
rebuilt widgets, injected page chrome). Write that down - it decides whether
Step 6 needs the CI-parity run. Both "green locally, red on CI" failures in
the first 5-pack onboarding came from exactly this.
## Step 2 - read the pack's real node keys
The manifest's `expectedNodes` are the pack's `object_info` keys (the same
strings the API uses as `class_type`). They are NOT Python class names and
NOT display names. Get them from the running backend:
```bash
curl -s http://127.0.0.1:8288/object_info | python3 -c '
import json, sys
d = json.load(sys.stdin)
for key, node in sorted(d.items()):
if node.get("python_module") == "custom_nodes.<pack-dir-name>":
print(key)
'
```
Real traps this step catches (each one shipped in a real pack):
| Pack | Correct key | Wrong guesses that look right |
| ---------------------- | ------------------- | ------------------------------------------------------------------------------- |
| ComfyUI_essentials | `SimpleMathInt+` | `SimpleMathInt` (keys carry a trailing `+`, except `DisplayAny` which has none) |
| ComfyUI-KJNodes | `INTConstant` | `INT Constant` (that is the display name) |
| ComfyUI-Custom-Scripts | `ShowText\|pysssss` | `ShowText` (keys carry a `\|pysssss` suffix) |
| rgthree-comfy | `Seed (rgthree)` | `RgthreeSeed` (the Python class name) |
## Step 3 - pick the expected nodes
Choose 2-3 nodes that are:
- **Model-free**: no checkpoint / VAE / CLIP inputs, no file downloads. The
gate runs on CPU with no models installed. Constants, math, text, and
display nodes are ideal.
- **Wireable into a chain**: at least one producer (has a typed output) and
one terminal node. A terminal node either has `output_node: true` in
`/object_info` (it terminates a workflow by itself) or you end the chain in
the core `PreviewAny` node, which accepts any type.
Check a candidate's inputs, outputs, and `output_node` flag:
```bash
curl -s http://127.0.0.1:8288/object_info | python3 -c '
import json, sys
node = json.load(sys.stdin)["<exact key>"]
print(json.dumps({k: node[k] for k in ("input", "output", "output_name", "output_node")}, indent=1))
'
```
Every node you list in `expectedNodes` must appear in the run workflow: the
run tier asserts each one actually executes on the backend.
## Step 4 - author the run-tier workflow
Add one JSON file under `browser_tests/assets/customNodes/`, named
`<pack>_<what it does>_run.json`. Copy an existing asset as the template
(`rgthree_seed_display_run.json` is the simplest two-node example;
`was_number_text_run.json` shows a 3-node chain). It is the frontend
workflow format, hand-authorable:
- `nodes[].type` is the exact `object_info` key from Step 2.
- `widgets_values` is an array in the node's widget order: the `input`
entries from `/object_info` in declaration order (`required` first, then
`optional`), keeping only widget-type inputs (INT, FLOAT, STRING, BOOLEAN,
and combo lists) and skipping any input whose options say
`"forceInput": true` (those are sockets, never widgets). A required input
that is neither a widget type nor `forceInput` (a custom type like
`NUMBER`) is also a socket: wire a link into it or the run fails on a
missing required input.
- A link is one row in `links`: `[link_id, from_node_id, from_slot,
to_node_id, to_slot, "TYPE"]`, plus the matching `link`/`links` ids on the
two nodes' `inputs`/`outputs` entries.
- To wire INTO an input that would normally be a widget (no `forceInput`),
the input entry also needs a `"widget": { "name": "<input name>" }` key -
see `browser_tests/assets/vueNodes/linked-int-widget.json`.
- Keep it tiny. Two to four nodes proving "this pack executes" is the whole
job; feature-depth testing belongs to the pack's own repo.
- If the workflow needs a media file, reuse something already under
`browser_tests/assets/` (e.g. `plain_video.mp4`) - never commit new binary
assets. CI stages `plain_video.mp4` into the backend's `input/` dir; if
your workflow needs a different existing asset staged, extend the
`Stage run-tier assets` step in
`.github/workflows/ci-tests-custom-nodes.yaml`.
- A media path in the workflow (e.g. `input/plain_video.mp4`) resolves
against the backend process's working directory, not the repo. Locally,
copy the file into the `input/` dir of the directory you launched
`main.py` from, or the run tier fails validation with
`Invalid file path` and the test reports `TIMEOUT`.
## Step 5 - add the manifest row
Append one object to `browser_tests/fixtures/data/customNodeManifest.json`:
| Field | Meaning |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pack` | The pack's directory name under `custom_nodes/` (what `git clone` creates). |
| `repo` | The GitHub URL CI clones. Required non-empty. |
| `pin` | REQUIRED in practice: the commit SHA you verified locally. CI checks it out after cloning, so the gate tests exactly what you tested - an unpinned pack lets any upstream push red the gate for every PR. Bump deliberately, re-verifying per this doc. |
| `tiers` | Which tiers run: `load` (registers + renders in both renderers), `connectivity` (typed links + slot drags), `run` (executes the workflow). Use all three unless a tier is impossible for the pack. |
| `workflow` | Path relative to `browser_tests/` of the Step 4 file. `""` only while the pack has no `run` tier. |
| `expectedNodes` | The Step 2/3 keys. The load tier mounts each in both renderers; the run tier asserts each executes. |
| `requiresGpu` | `true` only if execution genuinely needs CUDA. Such packs cannot use the `run` tier on the CPU gate. |
| `requiresModels` | Model files the workflow needs (`[]` for the packs onboarded so far - keep it that way whenever possible). |
| `timeoutMs` | Per-test budget. `30000` unless the workflow does real work (video decode uses `90000`). |
| `vueNodesCompatible` | Optional, default `true`. See the policy below. Only ever set `false`, and only with evidence. |
`loadManifest()` (`browser_tests/fixtures/customNode/manifest.ts`) validates
every row and fails loudly on a missing field, an empty `repo`, a misspelled
tier, or a `run` tier with an empty `workflow`.
## Step 6 - prove it green locally, in both environments
### 6a - fast loop (dev server)
```bash
pnpm test:custom-nodes
```
Green means: every tier for every pack passes, zero skips, and the suite's
zero-visible-errors invariant held (no error overlay, dialog, node error, or
error toast at any point). Iterate here - it is the fastest loop.
### 6b - CI-parity run (required if the pack ships frontend JS)
The dev server never loads pack frontend JS (its `/extensions` list is
core-only), so 6a exercises vanilla nodes. If Step 1 found frontend JS, a
6a green proves nothing about the pack's real runtime behavior. CI serves
the built frontend from the backend, so reproduce that exactly:
```bash
pnpm build
# relaunch the test backend with the same flags plus:
# --front-end-root <repo>/dist
# and make sure any run-tier media is in that process's input/ dir
PLAYWRIGHT_TEST_URL=http://127.0.0.1:8288 pnpm exec playwright test \
browser_tests/tests/customNodes/ --config playwright.chrome.config.ts --workers=1
```
Both real failures during the first 5-pack onboarding only existed here:
rgthree's progress bar shifted the canvas and broke slot-drag coordinates,
and rgthree's Seed rebuilt a declared input as widget-only. Skipping 6b
means discovering that class of problem one CI round at a time.
### Failure classes and what they mean
- **T0 fails only in the Vue Nodes pass** (the LiteGraph pass is green):
suspected Vue Nodes 2.0 incompatibility. Follow the policy below - do not
delete the pack, do not skip the test.
- **Run tier fails with `PARTIAL`** (some expected nodes never executed):
either the backend is missing `--cache-none` (cached nodes emit no
`executing` event) or an expected node is not actually in the workflow.
- **Run tier fails with an execution error**: the workflow JSON is wrong
(bad key, wrong `widgets_values` order, type-mismatched link) or the pack
cannot execute model-free. Fix the workflow or drop the node for a
simpler one.
- **Connectivity reports zero planned pairs**: the pack's slots are all
wildcard or combo typed (both are excluded from pairing by design because
they bypass the real type compare). The pack still gets load/run coverage.
- **Connectivity logs `widget-only on instance` exclusions**: the pack's own
frontend JS rebuilt a declared input as a widget-only control (rgthree's
Seed does this to `seed`), so there is no socket to wire. Recorded and
excluded, like wildcards - pack design, not a regression.
- **Auto-run reports a node "not in cannotRunAlone"**: the node failed to
execute on pure defaults (validation reject, or a real exception from
degenerate defaults - empty expression, empty folder, no webcam). If the
node USED to run clean this is a regression; otherwise add it to the
row's `cannotRunAlone` baseline with the run log in the PR. The check is
two-way: a listed node that starts running clean fails the suite until
the stale entry is removed.
- **Auto-run fails with `HUNG_BACKEND`**: a node blocked forever during
execution (the canonical case downloads a model at runtime and hangs
without network). The failure names the suspects and the remedy: add the
offender to `AUTO_RUN_EXCLUDE` in `allNodes.spec.ts` with its mechanism,
and restart the test backend (the hang is non-interruptible).
- **Mount test fails on console errors**: a pack's JS logged real errors
while its nodes mounted. If it is pack-attributed noise with no visible
error surface (KJNodes' loader previews fetching `filename=undefined`),
add a scoped `CONSOLE_ERROR_ALLOWLIST` entry with the mechanism;
otherwise it is a finding.
### The exception ledgers (all reasons on the record)
Every escape hatch is a reviewed list whose entries carry the mechanism, so
the gate stays honest and none can grow silently:
| Ledger | Lives in | Covers |
| ---------------------------- | ---------------------- | ------------------------------------------------------------------------------------------ |
| `vueIncompatibleNodes` | manifest row | node cannot mount under Vue Nodes 2.0 (evidence rule below) |
| `cannotRunAlone` | manifest row | node cannot execute standalone on a bare backend; asserted both ways so entries cannot rot |
| `AUTO_RUN_EXCLUDE` | `allNodes.spec.ts` | executing the node is unsafe on a bare backend (runtime downloads, hangs) |
| `CONSOLE_ERROR_ALLOWLIST` | `allNodes.spec.ts` | pack-attributed console noise with no visible error surface |
| `CONNECT_REJECTED_ALLOWLIST` | `connectivity.spec.ts` | pack JS legitimately vetoes a planned wiring |
| `ROUNDTRIP_LOST_ALLOWLIST` | `connectivity.spec.ts` | pack's own serialize/configure drops links it manages itself |
## Step 7 - push and watch CI
The `CI: Tests Custom Nodes` job (gating) re-does Steps 1-6 from scratch on
every PR: clones every manifest `repo` at its `pin`, pip-installs under CPU
torch constraints, boots the backend, runs the suite, and fails on any
install error, any test failure, or any skipped test. A new pack row is
automatically picked up; no workflow edit is needed unless you must stage an
extra asset (Step 4).
If CI goes red where local was green, reproduce under the Step 6b
environment before changing anything - the first such failure looked like
upstream drift but was actually pack frontend JS that never loads under
the dev server. Only after 6b reproduces it, decide: adjust the suite's
expectation honestly (the way widget-only instance slots became a recorded
exclusion) or, for genuine upstream drift after a pin bump, re-pin the
pack to its last good commit. Never paper
over it with a skip.
## Vue Nodes 2.0 compatibility policy
Some packs only work under the LiteGraph canvas renderer and fail to mount
under Vue Nodes 2.0. The suite must state that fact without producing false
failures and without skipping tests:
1. **Default**: every pack is assumed compatible. New rows omit
`vueNodesCompatible`.
2. **Evidence rule**: set `"vueNodesCompatible": false` ONLY after the T0
Vue pass fails for the pack locally while the LiteGraph pass is green,
and the failure reproduces on a retry. A README grumble, a hunch, or an
old forum thread is not evidence. Record the evidence (the failing
assertion and the pack version) in the PR description of the change that
sets the flag. When only SOME of a pack's nodes fail to mount, use the
per-node `vueIncompatibleNodes` ledger in the manifest row instead of
flagging the whole pack - compatibility is per-node, not per-pack (all
823 nodes across the first 7 packs mount clean, so both mechanisms ship
unused; the every-node mount tier is what earns an entry).
3. **Effect of `false`**: the load tier runs its LiteGraph pass only, and
the connectivity drag test does not drag that pack's edges under Vue
Nodes. The tests still run and pass their canvas assertions - nothing is
`test.skip`ped, so the CI skip gate stays honest. The run tier and the
connectivity contract sweep are renderer-independent (they never toggle
the Vue Nodes setting) and run for the pack regardless of the flag - a
flagged pack must still execute and wire cleanly there.
4. **Un-flagging**: if a pack ships Vue Nodes support later, delete the flag
and prove T0 green in both passes locally.
## Checklist
- [ ] Pack installs clean on the test backend (no `IMPORT FAILED`)
- [ ] Checked whether the pack ships frontend JS (Step 1 `/extensions` probe)
- [ ] `expectedNodes` copied exactly from `/object_info` (Step 2 traps checked)
- [ ] All expected nodes are model-free and present in the run workflow
- [ ] Workflow JSON under `browser_tests/assets/customNodes/`, no new binaries
- [ ] Any media staged into the backend's own `input/` dir locally (Step 4)
- [ ] Manifest row appended with every field (Step 5 table)
- [ ] `vueNodesCompatible` omitted, or set `false` with recorded evidence
- [ ] 6a green: `pnpm test:custom-nodes` against the dev server, zero skips
- [ ] 6b green when the pack ships frontend JS: built dist + backend-served run
- [ ] Every-node tiers green: no unexplained mount/save-reload/auto-run
failures; any new ledger entry carries its mechanism
- [ ] Pushed; `CI: Tests Custom Nodes` green on the PR

View File

@@ -0,0 +1,115 @@
# Custom-node regression suite
Proves community custom-node packs work against this frontend across both
renderers: nodes register, render under LiteGraph (canvas) AND Vue Nodes 2.0
(DOM), and execute real workflows end to end. Manifest-driven: adding a pack
is one JSON row, no new test code.
## Prerequisites
1. A ComfyUI backend on `127.0.0.1:8288` with every manifest pack (the
`pack` entries in `browser_tests/fixtures/data/customNodeManifest.json`)
and ComfyUI_devtools
installed. Launch it with `--multi-user` (the repo-wide browser-test
prerequisite; the fixture writes per-worker user settings and the suite
depends on them landing), `--cache-none` (repeat runs must re-execute
every node or the executed-set check fails honestly with `PARTIAL`), and
with `browser_tests/assets/plain_video.mp4` copied into its `input/` dir.
2. The dev server proxying that backend:
`DEV_SERVER_COMFYUI_URL=http://127.0.0.1:8288 pnpm dev`
## Running
| Script | What it does |
| -------------------------------------- | ------------------------------------------------------------------------------------- |
| `pnpm test:custom-nodes` | whole suite headless - the pass/fail gate (every tier passes, zero skips) |
| `pnpm test:custom-nodes:watch` | headed slow-motion run of the browser tiers, hands-off watching |
| `pnpm test:custom-nodes:debug` | step through the browser tiers in the Playwright Inspector (F10 step, F8 resume) |
| `pnpm test:custom-nodes:impact-render` | Impact nodes render in both renderers (Inspector) |
| `pnpm test:custom-nodes:impact-run` | Impact group workflow executes on the backend (Inspector) |
| `pnpm test:custom-nodes:vhs-render` | VHS nodes render in both renderers (Inspector) |
| `pnpm test:custom-nodes:vhs-run` | VHS decodes a real video through its node chain (Inspector) |
| `pnpm test:custom-nodes:connectivity` | slot/type contract: type-paired links + real slot drags in both renderers (Inspector) |
| `pnpm test:custom-nodes:self-check` | watches the harness catch a deliberate execution error |
Example - watch the VHS video-decode run step by step:
```bash
pnpm test:custom-nodes:vhs-run
```
Two windows open: the app under test and the Playwright Inspector. Press F10
to execute one robot action at a time (workflow loads, queue fires, backend
decodes the video), F8 to run to the end. While paused, look but do not click
inside the app window - your clicks change the state the next assertion
checks.
Any `-g` pattern works against the generic scripts, e.g.
`pnpm test:custom-nodes:debug -g "Impact-Pack.*T0"`.
## What the tests assert
- **T0 load**: pack nodes are registered in `/object_info`, added to a
cleared graph, counted exactly, and each added node's own `[data-node-id]`
element mounts under Vue Nodes 2.0. Both renderer passes - unless the pack
declares `vueNodesCompatible: false` in the manifest (evidence required;
see [ADDING_CUSTOM_NODES.md](ADDING_CUSTOM_NODES.md)), in which case its tests run their
LiteGraph-canvas assertions only. Never a skip.
- **T1 run**: the manifest workflow is loaded and queued; the backend's
`executing` event stream must contain every expected node id, and the run
must end in `execution_success`.
- **Every-node tiers** (`allNodes.spec.ts`): the pack's FULL node list,
discovered live from `/object_info`, is exercised with zero
configuration - every registered node mounts in both renderers (chunked
at an empirically measured batch size), survives a serialize/configure
save-reload round-trip, and executes for real on the backend when
self-sufficient (all required inputs are widgets with valid defaults).
Nodes that cannot run alone are classified and logged
(`NEEDS_WIRES` / `NEEDS_MODELS` / `NO_OBSERVABLE_OUTPUT` / rejected-at-validation),
never silently dropped; the documented exception ledgers (see
[ADDING_CUSTOM_NODES.md](ADDING_CUSTOM_NODES.md)) carry a written mechanism for every
escape hatch.
- **connectivity (contract)**: wiring-only, no execution. A
type-pairing generator (`fixtures/customNode/typePairing.ts`) indexes
`/object_info` producers/consumers and plans one representative typed edge
per slot (wildcard `*` slots excluded - they bypass the real type compare
and prove nothing). Each planned edge must connect through the real
`isValidConnection` veto, then survive `serialize()` -> `configure()` and
appear in `graphToPrompt()` output. A curated subset is additionally
dragged for real - slot dot to slot dot - under both renderers. Orphan
types (no partner in the corpus) are reported, never fake-failed. One
representative edge per slot bounds cost; it does not prove all pairs.
- **Zero visible errors, always**: every browser test asserts the app's
error surfaces (error overlay, error dialog, node render errors, error
toasts) are absent at start and after every pass. A run is green only if a
human watching the screen sees no errors. The self-check inverts this: it
forces a real execution error and asserts the overlay IS visible, proving
the selectors stay live.
## Adding a pack
One manifest row plus one small workflow JSON - no new test code. The
authoritative step-by-step process (verifying the pack's real node keys,
authoring the run workflow, the `vueNodesCompatible` evidence rule, what CI
does with the row) lives in [ADDING_CUSTOM_NODES.md](ADDING_CUSTOM_NODES.md). Follow it
exactly; the traps it lists all shipped in real packs.
## Gotchas
- **Pack frontend JS does not load under the Vite dev server.** The dev
server's `/extensions` endpoint lists core extensions only, so nodes render
vanilla locally even when the backend has the packs installed. CI serves
the built frontend from the backend, where every pack's JS loads and can
restyle nodes, rebuild widgets, or inject page chrome. Before pushing
changes that could interact with pack JS, reproduce CI locally:
`pnpm build`, relaunch the backend with `--front-end-root <repo>/dist`,
and run the suite with `PLAYWRIGHT_TEST_URL` pointed at the backend.
- Do not run with `--trace on` against system Chrome
(`playwright.chrome.config.ts` pins trace off): the trace recorder crashes
pages under the branded Chrome channel and every test reports a bogus 15s
timeout.
- In a git worktree whose `node_modules` is symlinked from another checkout,
prefix scripts with `pnpm --config.verify-deps-before-run=false ...` to
skip pnpm's auto-install check.
- First run against a cold dev server can exceed the 15s per-test setup
budget while Vite compiles; just run again.

View File

@@ -0,0 +1,427 @@
/* oxlint-disable playwright/no-skipped-test -- tiers conditionally skip when the target backend lacks the required packs; environment gating, not a disabled test */
// Every-node coverage: the suite's core contract (mounts, survives
// save/reload, executes when self-sufficient) applied to ALL nodes a pack
// registers - not just the curated expectedNodes sentinels. Node lists come
// from the live backend, so a pack update is covered the moment it installs.
import type { Page } from '@playwright/test'
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
batchAutoRunnable,
planAutoRuns
} from '@e2e/fixtures/customNode/autoRun'
import { LocalDesktopTarget } from '@e2e/fixtures/customNode/ComfyTarget'
import { loadManifest } from '@e2e/fixtures/customNode/manifest'
import type { RawNodeDef } from '@e2e/fixtures/customNode/typePairing'
import { normalizeNodeDefs } from '@e2e/fixtures/customNode/typePairing'
import { collectConsoleErrors } from '@e2e/fixtures/utils/consoleErrorCollector'
import {
customNodeSuiteSettings,
dismissTemplatesDialog
} from '@e2e/fixtures/utils/customNodeSuite'
import { errorSurfaces } from '@e2e/fixtures/utils/errorSurfaces'
const target = new LocalDesktopTarget()
// Measured optimum (deterministic across repeats, best ms/node); see PR.
const BATCH_SIZE = 24
const AUTO_RUN_BATCH = 10
const GRID_SPACING = { x: 420, y: 360 }
// Nodes unsafe to execute on a bare backend; every entry names the mechanism.
const AUTO_RUN_EXCLUDE: Record<string, Record<string, string>> = {
'rgthree-comfy': {
'Power Primitive (rgthree)':
'requires its pack JS to build the primitive value at queue time; raw defaults KeyError. Whether a page applies pack JS varies by serving setup, so excluded unconditionally - curated-workflow candidate',
'Power Puter (rgthree)':
'requires its pack JS to compile the expression at queue time; raw defaults KeyError. Excluded unconditionally - curated-workflow candidate'
},
'ComfyUI-KJNodes': {
PointsEditor:
'requires its pack JS to inject the points JSON at queue time; raw defaults JSONDecodeError. Excluded unconditionally - curated-workflow candidate',
SplineEditor:
'requires its pack JS to inject the spline JSON at queue time; raw defaults JSONDecodeError. Excluded unconditionally - curated-workflow candidate',
StringToFloatList:
'requires its pack JS to normalize the list string at queue time; raw defaults ValueError. Excluded unconditionally - curated-workflow candidate'
},
ComfyUI_essentials: {
'RemBGSession+':
'initializes a rembg session that downloads its ONNX model at execution; hangs (non-interruptibly) on a backend without network/model access',
'TransitionMask+':
'list-expanded execution emits no per-node executing event on some runs, so the executed-set signal flip-flops between PASS and PARTIAL; mount/save-reload/connectivity tiers still cover it',
'TransparentBGSession+':
'ML-session initializer like RemBGSession+; sets up/downloads a background-removal model at execution, unstable on a bare backend'
}
}
// Pack-attributed console noise with no visible error surface.
const CONSOLE_ERROR_ALLOWLIST: Record<
string,
Array<{ pattern: RegExp; reason: string }>
> = {
'ComfyUI-KJNodes': [
{
// Image/video loader previews fetch their combo value at creation;
// on a backend with an empty input dir the value is undefined and the
// preview 404s (and retries with a fresh rand). Console-only noise,
// no visible error; upstream-report candidate.
pattern:
/Failed to load resource.*\/api\/view\?type=input&filename=undefined/,
reason: 'loader preview fetches undefined filename on empty input dir'
}
]
}
test.use({ initialSettings: customNodeSuiteSettings })
test.beforeEach(async ({ comfyPage }) => {
await dismissTemplatesDialog(comfyPage)
})
async function expectNoVisibleErrors(
page: Page,
context: string
): Promise<void> {
for (const [surface, locator] of Object.entries(errorSurfaces(page)))
await expect(locator, `${context}: ${surface}`).toHaveCount(0)
}
// null id = createNode failed for that type.
function addChunk(page: Page, types: string[]): Promise<Array<string | null>> {
return page.evaluate(
([chunk, spacingX, spacingY]) => {
window.app!.graph.clear()
const cols = Math.ceil(Math.sqrt(chunk.length))
const ids: Array<string | null> = []
for (const [index, type] of chunk.entries()) {
const node = window.LiteGraph!.createNode(type)
if (!node) {
ids.push(null)
continue
}
node.pos = [
(index % cols) * (spacingX as number),
Math.floor(index / cols) * (spacingY as number)
]
window.app!.graph.add(node)
ids.push(String(node.id))
}
const canvas = window.app!.canvas
const rect = canvas.canvas.getBoundingClientRect()
const width = cols * (spacingX as number)
const height = Math.ceil(chunk.length / cols) * (spacingY as number)
const scale = Math.min(
(rect.width / Math.max(width, 1)) * 0.9,
(rect.height / Math.max(height, 1)) * 0.9,
1
)
canvas.ds.scale = scale
canvas.ds.offset = [60 / scale, 60 / scale]
canvas.setDirty(true, true)
return ids
},
[types, GRID_SPACING.x, GRID_SPACING.y] as const
)
}
async function packNodeKeys(
page: Page,
pack: string
): Promise<{ keys: string[]; defs: Record<string, RawNodeDef> }> {
const defs = (await page.evaluate(() =>
window.app!.api.getNodeDefs()
)) as unknown as Record<string, RawNodeDef>
const keys = normalizeNodeDefs(defs)
.filter((node) => node.pack === pack)
.map((node) => node.type)
.sort()
return { keys, defs }
}
for (const entry of loadManifest()) {
test.describe(`all nodes: ${entry.pack}`, () => {
test('every registered node mounts in both renderers', async ({
comfyPage
}) => {
test.setTimeout(240_000)
const { keys } = await packNodeKeys(comfyPage.page, entry.pack)
test.skip(
keys.length === 0,
`${entry.pack} not installed on this backend`
)
const ledger = entry.vueIncompatibleNodes ?? {}
for (const ledgered of Object.keys(ledger))
expect(
keys,
`stale ledger entry: ${ledgered} is not registered by ${entry.pack}`
).toContain(ledgered)
for (const vueNodesEnabled of [false, true]) {
const consoleErrors = collectConsoleErrors(comfyPage.page)
await comfyPage.settings.setSetting(
'Comfy.VueNodes.Enabled',
vueNodesEnabled
)
const failures: string[] = []
for (let offset = 0; offset < keys.length; offset += BATCH_SIZE) {
const chunk = keys.slice(offset, offset + BATCH_SIZE)
const ids = await addChunk(comfyPage.page, chunk)
await comfyPage.nextFrame()
const count = await comfyPage.nodeOps.getGraphNodesCount()
if (count !== chunk.length)
failures.push(
`chunk@${offset}: graph has ${count} of ${chunk.length} nodes`
)
for (const [index, id] of ids.entries()) {
const key = chunk[index]
if (id === null) {
failures.push(`${key}: createNode returned null`)
continue
}
if (!vueNodesEnabled) continue
if (key in ledger) continue
const visible = await comfyPage.page
.locator(`[data-node-id="${id}"]`)
.isVisible({ timeout: 2_000 })
.catch(() => false)
if (!visible) failures.push(`${key}: no Vue mount`)
}
}
if (vueNodesEnabled && Object.keys(ledger).length > 0)
console.log(
`${entry.pack}: ${Object.keys(ledger).length} node(s) ledgered Vue-incompatible; Vue mount not asserted for them`
)
consoleErrors.stop()
expect(
failures,
`VueNodes=${vueNodesEnabled}: ${JSON.stringify(failures, null, 1)}`
).toEqual([])
const allowlist = CONSOLE_ERROR_ALLOWLIST[entry.pack] ?? []
const allowed = consoleErrors.errors.filter((error) =>
allowlist.some((rule) => rule.pattern.test(error))
)
if (allowed.length > 0)
console.log(
`${entry.pack}: ${allowed.length} console error(s) matched the pack's allowlist (${allowlist.map((rule) => rule.reason).join('; ')})`
)
expect(
consoleErrors.errors.filter(
(error) => !allowlist.some((rule) => rule.pattern.test(error))
),
`console errors with VueNodes=${vueNodesEnabled}`
).toEqual([])
await expectNoVisibleErrors(
comfyPage.page,
`after all-nodes VueNodes=${vueNodesEnabled} pass`
)
}
})
test('every registered node survives save/reload', async ({
comfyPage
}) => {
test.setTimeout(240_000)
const { keys } = await packNodeKeys(comfyPage.page, entry.pack)
test.skip(
keys.length === 0,
`${entry.pack} not installed on this backend`
)
await comfyPage.settings.setSetting('Comfy.VueNodes.Enabled', false)
const mismatches: string[] = []
for (let offset = 0; offset < keys.length; offset += BATCH_SIZE) {
const chunk = keys.slice(offset, offset + BATCH_SIZE)
const chunkMismatches = await comfyPage.page.evaluate((types) => {
window.app!.graph.clear()
const before = new Map<
string,
{ type: string; widgetValues: number }
>()
for (const type of types) {
const node = window.LiteGraph!.createNode(type)
if (!node) continue
window.app!.graph.add(node)
before.set(String(node.id), {
type,
widgetValues: (node.widgets ?? []).length
})
}
const serialized = window.app!.graph.serialize()
window.app!.graph.configure(serialized)
const problems: string[] = []
for (const [id, expected] of before) {
const restored = window.app!.graph.getNodeById(Number(id))
if (!restored) {
problems.push(`${expected.type}: lost on reload`)
continue
}
if (restored.type !== expected.type)
problems.push(
`${expected.type}: type became ${String(restored.type)}`
)
const widgets = (restored.widgets ?? []).length
if (widgets !== expected.widgetValues)
problems.push(
`${expected.type}: widgets ${expected.widgetValues} -> ${widgets}`
)
}
window.app!.graph.clear()
return problems
}, chunk)
mismatches.push(...chunkMismatches)
}
expect(mismatches, JSON.stringify(mismatches, null, 1)).toEqual([])
await expectNoVisibleErrors(comfyPage.page, 'after save/reload sweep')
})
test('every auto-runnable node executes without error', async ({
comfyPage
}) => {
test.setTimeout(900_000)
const { keys, defs } = await packNodeKeys(comfyPage.page, entry.pack)
test.skip(
keys.length === 0,
`${entry.pack} not installed on this backend`
)
await comfyPage.settings.setSetting('Comfy.VueNodes.Enabled', false)
// A leftover hung execution would false-timeout every run below.
const queueBusy = await comfyPage.page.evaluate(async () => {
const queue = (await window.app!.api.getQueue()) as {
Running?: unknown[]
}
return (queue.Running ?? []).length
})
expect(
queueBusy,
'backend queue already has a running prompt (earlier hung execution?) - restart the test backend'
).toBe(0)
const excluded = AUTO_RUN_EXCLUDE[entry.pack] ?? {}
for (const [key, reason] of Object.entries(excluded))
console.log(`${entry.pack}: ${key} excluded from auto-run (${reason})`)
const verdicts = planAutoRuns(
defs,
keys.filter((key) => !(key in excluded))
)
const counts = new Map<string, number>()
for (const verdict of verdicts)
counts.set(verdict.verdict, (counts.get(verdict.verdict) ?? 0) + 1)
console.log(
`${entry.pack} auto-run plan: ${[...counts.entries()]
.map(([verdict, count]) => `${verdict}=${count}`)
.join(' ')}`
)
const batches = batchAutoRunnable(verdicts, AUTO_RUN_BATCH)
const hardFailures: string[] = []
const cannotRun = new Map<string, string>()
const ranClean = new Set<string>()
for (const batch of batches) {
const outcome = await runBatch(comfyPage.page, batch)
if (outcome === 'PASS') {
for (const verdict of batch) ranClean.add(verdict.key)
continue
}
// A jammed queue false-timeouts everything after it - stop here.
if (outcome.startsWith('HUNG_BACKEND')) {
hardFailures.push(
`[${batch.map((verdict) => verdict.key).join(', ')}]: ${outcome} - add the offender to AUTO_RUN_EXCLUDE with its mechanism`
)
break
}
// Rerun singles so the bad node names itself.
for (const verdict of batch) {
const single = await runBatch(comfyPage.page, [verdict])
if (single === 'PASS') ranClean.add(verdict.key)
else if (single.startsWith('HUNG_BACKEND')) {
hardFailures.push(
`${verdict.key}: ${single} - add to AUTO_RUN_EXCLUDE with its mechanism`
)
break
} else cannotRun.set(verdict.key, single)
}
}
// Two-way reconciliation: unlisted failure = regression; listed node
// that runs clean (or is not auto-runnable) = stale entry.
const baseline = new Set(entry.cannotRunAlone ?? [])
const runnable = new Set(
batches.flatMap((batch) => batch.map((verdict) => verdict.key))
)
for (const [key, detail] of cannotRun)
if (!baseline.has(key))
hardFailures.push(
`${key}: ${detail} - not in cannotRunAlone; a regression, or a new baseline entry (attach the run log)`
)
for (const key of baseline) {
if (ranClean.has(key))
hardFailures.push(
`${key}: ran clean but is listed in cannotRunAlone - remove the stale entry`
)
else if (!runnable.has(key))
hardFailures.push(
`${key}: listed in cannotRunAlone but is not auto-runnable on this backend - remove the stale entry`
)
}
console.log(
`${entry.pack} auto-ran ${ranClean.size} node(s) clean; ${cannotRun.size} cannot run alone (baseline ${baseline.size})`
)
expect(hardFailures, JSON.stringify(hardFailures, null, 1)).toEqual([])
})
})
}
async function runBatch(
page: Page,
batch: Array<{ key: string; needsPreviewSink?: boolean }>
): Promise<string> {
const ids = await page.evaluate(
([nodes, spacingY]) => {
window.app!.graph.clear()
const ids: string[] = []
for (const [index, spec] of nodes.entries()) {
const node = window.LiteGraph!.createNode(spec.key)
if (!node) continue
node.pos = [0, index * (spacingY as number)]
window.app!.graph.add(node)
ids.push(String(node.id))
if (spec.needsPreviewSink) {
const sink = window.LiteGraph!.createNode('PreviewAny')!
sink.pos = [460, index * (spacingY as number)]
window.app!.graph.add(sink)
node.connect(0, sink, 0)
}
}
return ids
},
[batch, GRID_SPACING.y] as const
)
// Widget-only CPU nodes: not finished in 20s = hung.
const result = await target.runWorkflow(page, {
expectedNodeIds: ids,
timeoutMs: 20_000
})
if (result.outcome === 'TIMEOUT') {
// Interrupt and verify the queue drained; a non-interruptible hang can
// only be cleared by a backend restart, so name it.
const drained = await page.evaluate(async () => {
await window.app!.api.interrupt()
for (let attempt = 0; attempt < 10; attempt++) {
await new Promise((resolve) => setTimeout(resolve, 500))
const queue = (await window.app!.api.getQueue()) as {
Running?: unknown[]
}
if ((queue.Running ?? []).length === 0) return true
}
return false
})
if (!drained)
return 'HUNG_BACKEND (non-interruptible execution; backend restart required)'
}
return result.outcome === 'PASS'
? 'PASS'
: `${result.outcome}${result.error?.nodeType ? ` (${result.error.nodeType}: ${result.error.exceptionType ?? ''})` : ''}`
}

View File

@@ -0,0 +1,115 @@
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
batchAutoRunnable,
classifyAutoRunnable,
planAutoRuns
} from '@e2e/fixtures/customNode/autoRun'
test.describe('autoRun classifier', () => {
test('widget-only node with outputs is runnable via a PreviewAny sink', () => {
const verdict = classifyAutoRunnable('IntConstant', {
input: { required: { value: ['INT', { default: 0 }] } },
output: ['INT'],
output_node: false
})
expect(verdict.verdict).toBe('AUTO_RUNNABLE')
expect(verdict.needsPreviewSink).toBe(true)
})
test('widget-only OUTPUT_NODE runs standalone', () => {
const verdict = classifyAutoRunnable('ShowValue', {
input: {
required: {
text: ['STRING', {}],
mode: [['raw value', 'tensor shape']]
}
},
output: [],
output_node: true
})
expect(verdict.verdict).toBe('AUTO_RUNNABLE')
expect(verdict.needsPreviewSink).toBe(false)
})
test('a required socket input means NEEDS_WIRES', () => {
const verdict = classifyAutoRunnable('VaeDecode', {
input: { required: { samples: ['LATENT'], vae: ['VAE'] } },
output: ['IMAGE'],
output_node: false
})
expect(verdict.verdict).toBe('NEEDS_WIRES')
expect(verdict.reason).toContain('samples')
})
test('forceInput STRING is a socket, not a widget', () => {
const verdict = classifyAutoRunnable('TextSink', {
input: { required: { text: ['STRING', { forceInput: true }] } },
output: ['STRING'],
output_node: true
})
expect(verdict.verdict).toBe('NEEDS_WIRES')
})
test('an empty required combo means NEEDS_MODELS', () => {
const verdict = classifyAutoRunnable('CheckpointLoader', {
input: { required: { ckpt_name: [[]] } },
output: ['MODEL'],
output_node: false
})
expect(verdict.verdict).toBe('NEEDS_MODELS')
expect(verdict.reason).toContain('ckpt_name')
})
test('no outputs and not an OUTPUT_NODE means NO_OBSERVABLE_OUTPUT', () => {
const verdict = classifyAutoRunnable('SideEffectOnly', {
input: { required: { value: ['INT', {}] } },
output: [],
output_node: false
})
expect(verdict.verdict).toBe('NO_OBSERVABLE_OUTPUT')
})
test('optional socket inputs do not block auto-running', () => {
const verdict = classifyAutoRunnable('MathWithOptionalAny', {
input: {
required: { expression: ['STRING', {}] },
optional: { a: ['*'] }
},
output: ['INT', 'FLOAT'],
output_node: true
})
expect(verdict.verdict).toBe('AUTO_RUNNABLE')
})
test('planAutoRuns maps keys and batchAutoRunnable chunks only runnables', () => {
const defs = {
A: {
input: { required: { v: ['INT', {}] } },
output: ['INT'],
output_node: false
},
B: {
input: { required: { x: ['LATENT'] } },
output: ['LATENT'],
output_node: false
},
C: {
input: { required: { v: ['FLOAT', {}] } },
output: ['FLOAT'],
output_node: false
}
}
const verdicts = planAutoRuns(defs, ['A', 'B', 'C'])
expect(verdicts.map((verdict) => verdict.verdict)).toEqual([
'AUTO_RUNNABLE',
'NEEDS_WIRES',
'AUTO_RUNNABLE'
])
const batches = batchAutoRunnable(verdicts, 1)
expect(batches).toHaveLength(2)
expect(batches[0][0].key).toBe('A')
})
})

View File

@@ -0,0 +1,472 @@
import type { Page } from '@playwright/test'
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
customNodeSuiteSettings,
dismissTemplatesDialog
} from '@e2e/fixtures/utils/customNodeSuite'
import { loadManifest } from '@e2e/fixtures/customNode/manifest'
import type {
ConnectivityOutcome,
PlannedPair,
RawNodeDef
} from '@e2e/fixtures/customNode/typePairing'
import {
isWildcard,
normalizeNodeDefs,
planPairs
} from '@e2e/fixtures/customNode/typePairing'
import { collectConsoleErrors } from '@e2e/fixtures/utils/consoleErrorCollector'
import { errorSurfaces } from '@e2e/fixtures/utils/errorSurfaces'
const CORE_PROOF_NODE_COUNT = 16
// A node may legitimately veto a wiring via onConnectInput; committed
// entries here must name the veto. Green means actual rejections are a
// subset of this list.
const CONNECT_REJECTED_ALLOWLIST: string[] = [
// pysssss MathExpression only accepts INT/FLOAT-producing links into its
// expression variables; its JS vetoes text-list producers.
'AddTextPrefix.texts -> MathExpression|pysssss.expression'
]
// A pack's own serialize/configure hooks may drop links it manages itself
// (reproducible manually: wire, save, reload - link gone). Pack behavior on
// record, not frontend regressions.
const ROUNDTRIP_LOST_ALLOWLIST: string[] = [
// rgthree SDXL Power Prompt rebuilds its dimension widget-inputs during
// configure and drops inbound links to them.
'BatchCount+.INT -> SDXL Power Prompt - Positive (rgthree).target_width',
'BatchCount+.INT -> SDXL Power Prompt - Positive (rgthree).target_height',
'BatchCount+.INT -> SDXL Power Prompt - Positive (rgthree).crop_width',
'BatchCount+.INT -> SDXL Power Prompt - Positive (rgthree).crop_height',
'BatchCount+.INT -> SDXL Power Prompt - Simple / Negative (rgthree).target_width',
'BatchCount+.INT -> SDXL Power Prompt - Simple / Negative (rgthree).target_height',
'BatchCount+.INT -> SDXL Power Prompt - Simple / Negative (rgthree).crop_width',
'BatchCount+.INT -> SDXL Power Prompt - Simple / Negative (rgthree).crop_height',
// VHS_SelectLatest rebuilds its dynamic slots on configure, detaching
// links on both its inputs and outputs.
'AddTextPrefix.texts -> VHS_SelectLatest.filename_prefix',
'AddTextPrefix.texts -> VHS_SelectLatest.filename_postfix',
'VHS_SelectLatest.Filename -> AddLabel.font_color'
]
test.use({ initialSettings: customNodeSuiteSettings })
test.beforeEach(async ({ comfyPage }) => {
await dismissTemplatesDialog(comfyPage)
})
async function expectNoVisibleErrors(
page: Page,
context: string
): Promise<void> {
for (const [surface, locator] of Object.entries(errorSurfaces(page)))
await expect(locator, `${context}: ${surface}`).toHaveCount(0)
}
function concrete(slot: { type: string }): boolean {
return !isWildcard(slot.type)
}
function isEntryInstalled(
nodeTypes: Set<string>,
entry: { expectedNodes: string[] }
): boolean {
return entry.expectedNodes.every((type) => nodeTypes.has(type))
}
const connectivityEntries = loadManifest().filter((entry) =>
entry.tiers.includes('connectivity')
)
test('connectivity: every type-paired link survives model, serialize, and prompt round-trips', async ({
comfyPage
}) => {
test.setTimeout(120_000)
const defs = (await comfyPage.page.evaluate(() =>
window.app!.api.getNodeDefs()
)) as unknown as Record<string, RawNodeDef>
const nodes = normalizeNodeDefs(defs)
// Pack-specific expectations apply only where the pack is installed; on a
// backend without it (e.g. a generic CI runner) the core sweep still runs
// and the absence is reported, never fake-failed or fake-passed.
const nodeTypes = new Set(nodes.map((node) => node.type))
const installedEntries = connectivityEntries.filter((entry) =>
isEntryInstalled(nodeTypes, entry)
)
for (const entry of connectivityEntries)
if (!installedEntries.includes(entry))
console.log(`connectivity: ${entry.pack} not installed on this backend`)
// Corpus = every node the installed packs register, from the live backend.
const installedPacks = new Set(installedEntries.map((entry) => entry.pack))
const packTypes = nodes
.filter((node) => installedPacks.has(node.pack))
.map((node) => node.type)
const coreProof = nodes
.filter(
(node) =>
node.pack === 'core' &&
node.inputs.some(concrete) &&
node.outputs.some(concrete)
)
.map((node) => node.type)
.sort()
.slice(0, CORE_PROOF_NODE_COUNT)
const plan = planPairs(nodes, [...packTypes, ...coreProof])
expect(plan.pairs.length, 'pairing produced no edges').toBeGreaterThan(0)
console.log(
`connectivity plan: ${plan.pairs.length} pairs, ${plan.orphans.length} orphan slots, ${plan.wildcards.length} wildcard + ${plan.combos.length} combo slots (excluded by design)`
)
for (const entry of installedEntries) {
expect(
plan.pairs.some(
(pair) =>
pair.producer.pack === entry.pack || pair.consumer.pack === entry.pack
),
`${entry.pack} contributes no pairs - corpus or pack attribution broke`
).toBe(true)
}
const consoleErrors = collectConsoleErrors(comfyPage.page)
const results = await runPairsInPage(comfyPage.page, plan.pairs)
consoleErrors.stop()
expect(consoleErrors.errors, 'console errors during breadth sweep').toEqual(
[]
)
const widgetOnly = results.filter(
(result) =>
result.outcome ===
('WIDGET_ONLY_ON_INSTANCE' satisfies ConnectivityOutcome)
)
if (widgetOnly.length > 0)
console.log(
`connectivity sweep: ${widgetOnly.length} pair(s) excluded - pack JS made the declared input widget-only: ${widgetOnly.map((result) => result.key).join('; ')}`
)
const failures = results.filter(
(result) =>
result.outcome !== ('PASS' satisfies ConnectivityOutcome) &&
result.outcome !==
('WIDGET_ONLY_ON_INSTANCE' satisfies ConnectivityOutcome) &&
!(
result.outcome === ('CONNECT_REJECTED' satisfies ConnectivityOutcome) &&
CONNECT_REJECTED_ALLOWLIST.includes(result.key)
) &&
!(
result.outcome === ('ROUNDTRIP_LOST' satisfies ConnectivityOutcome) &&
ROUNDTRIP_LOST_ALLOWLIST.includes(result.key)
)
)
const passed = results.filter((result) => result.outcome === 'PASS').length
console.log(`connectivity sweep: ${passed}/${results.length} pairs PASS`)
expect(failures, JSON.stringify(failures, null, 1)).toEqual([])
expect(passed).toBeGreaterThan(0)
await expectNoVisibleErrors(comfyPage.page, 'after breadth sweep')
})
// First planned pair whose slots both exist on real instances (pack JS can
// rebuild declared inputs as widget-only controls).
function firstMaterializedPair(
page: Page,
pairs: PlannedPair[]
): Promise<PlannedPair | null> {
return page.evaluate((pairsInPage) => {
for (const pair of pairsInPage) {
const producer = window.LiteGraph!.createNode(pair.producer.nodeType)
const consumer = window.LiteGraph!.createNode(pair.consumer.nodeType)
const outFound = producer?.outputs.some(
(slot) => slot.name === pair.producer.slotName
)
const inFound = consumer?.inputs.some(
(slot) => slot.name === pair.consumer.slotName
)
if (outFound && inFound) return pair
}
return null
}, pairs)
}
// The self-check below runs THIS SAME executor on poisoned pairs; if it stops
// being able to reject, every green sweep above is meaningless.
function runPairsInPage(
page: Page,
pairs: PlannedPair[]
): Promise<Array<{ key: string; outcome: string; detail?: string }>> {
return page.evaluate(async (pairsInPage) => {
const graph = window.app!.graph
const report: Array<{
key: string
outcome: string
detail?: string
}> = []
for (const pair of pairsInPage) {
const key = `${pair.producer.nodeType}.${pair.producer.slotName} -> ${pair.consumer.nodeType}.${pair.consumer.slotName}`
try {
graph.clear()
const producer = window.LiteGraph!.createNode(pair.producer.nodeType)
const consumer = window.LiteGraph!.createNode(pair.consumer.nodeType)
if (!producer || !consumer) {
report.push({
key,
outcome: 'SLOT_CONTRACT_MISMATCH',
detail: 'createNode returned null for a registered type'
})
continue
}
graph.add(producer)
graph.add(consumer)
const outIndex = producer.outputs.findIndex(
(slot) => slot.name === pair.producer.slotName
)
const inIndex = consumer.inputs.findIndex(
(slot) => slot.name === pair.consumer.slotName
)
if (outIndex < 0 || inIndex < 0) {
// Pack JS may rebuild a declared input as widget-only (rgthree
// Seed.seed) - excluded; missing as slot AND widget stays a fail.
const widgetOnly =
outIndex >= 0 &&
(consumer.widgets ?? []).some(
(widget) => widget.name === pair.consumer.slotName
)
report.push({
key,
outcome: widgetOnly
? 'WIDGET_ONLY_ON_INSTANCE'
: 'SLOT_CONTRACT_MISMATCH',
detail: `declared slot missing on instance (out=${outIndex}, in=${inIndex})`
})
continue
}
const link = producer.connect(outIndex, consumer, inIndex)
if (!link || consumer.inputs[inIndex]?.link == null) {
report.push({ key, outcome: 'CONNECT_REJECTED' })
continue
}
const serialized = graph.serialize()
graph.configure(serialized)
const restored = graph.getNodeById(consumer.id)
if (restored?.inputs?.[inIndex]?.link == null) {
report.push({
key,
outcome: 'ROUNDTRIP_LOST',
detail: 'serialize/configure dropped the link'
})
continue
}
const prompt = (await window.app!.graphToPrompt()) as {
output?: Record<string, { inputs?: Record<string, unknown> }>
}
const promptInput =
prompt.output?.[String(consumer.id)]?.inputs?.[pair.consumer.slotName]
if (!Array.isArray(promptInput)) {
report.push({
key,
outcome: 'ROUNDTRIP_LOST',
detail: 'link missing from graphToPrompt output'
})
continue
}
report.push({ key, outcome: 'PASS' })
} catch (error) {
report.push({
key,
outcome: 'SLOT_CONTRACT_MISMATCH',
detail: `threw: ${String(error)}`
})
}
}
graph.clear()
return report
}, pairs)
}
test('connectivity self-check: the executor rejects broken pairs', async ({
comfyPage
}) => {
const slot = (nodeType: string, slotName: string, slotType: string) => ({
nodeType,
pack: 'core',
slotName,
slotType
})
const results = await runPairsInPage(comfyPage.page, [
{
producer: slot('CheckpointLoaderSimple', 'MODEL', 'MODEL'),
consumer: slot('KSampler', 'latent_image', 'LATENT')
},
{
producer: slot('EmptyLatentImage', 'LATENT', 'LATENT'),
consumer: slot('KSampler', 'does_not_exist', 'LATENT')
}
])
expect(results.map((result) => result.outcome)).toEqual([
'CONNECT_REJECTED',
'SLOT_CONTRACT_MISMATCH'
])
})
test('connectivity drags: curated slot-to-slot wires connect under both renderers', async ({
comfyPage
}) => {
test.setTimeout(120_000)
const defs = (await comfyPage.page.evaluate(() =>
window.app!.api.getNodeDefs()
)) as unknown as Record<string, RawNodeDef>
const nodes = normalizeNodeDefs(defs)
// Native anchor pair plus one in-pack, link-typed pair per connectivity
// pack (derived from the same generator the breadth sweep uses).
const dragEdges: PlannedPair[] = [
{
producer: {
nodeType: 'EmptyLatentImage',
pack: 'core',
slotName: 'LATENT',
slotType: 'LATENT'
},
consumer: {
nodeType: 'KSampler',
pack: 'core',
slotName: 'latent_image',
slotType: 'LATENT'
}
}
]
const nodeTypes = new Set(nodes.map((node) => node.type))
for (const entry of connectivityEntries) {
if (!isEntryInstalled(nodeTypes, entry)) {
console.log(
`connectivity drag: ${entry.pack} not installed on this backend`
)
continue
}
// Restrict the partner pool to the pack itself so the drag proves an
// in-pack wiring; widget-backed primitive inputs render real slot dots
// in Vue (verified empirically), so no slot type is excluded at plan time.
const packPlan = planPairs(
nodes.filter((node) => node.pack === entry.pack),
entry.expectedNodes
)
expect(
packPlan.pairs.length,
`${entry.pack} has no in-pack draggable pair - drag coverage lost`
).toBeGreaterThan(0)
// The plan comes from object_info, but a pack's own JS can rebuild a
// declared input as widget-only on the instance (rgthree's Seed does).
// Drag the first pair whose slots actually materialize; a pack whose
// every planned pair is customized away has no socket contract to drag.
const inPack = await firstMaterializedPair(comfyPage.page, packPlan.pairs)
if (!inPack) {
console.log(
`connectivity drag: ${entry.pack} planned pairs are widget-only on instances; drag not applicable`
)
continue
}
dragEdges.push(inPack)
}
const vueIncompatiblePacks = new Set(
connectivityEntries
.filter((entry) => entry.vueNodesCompatible === false)
.map((entry) => entry.pack)
)
for (const vueNodesEnabled of [false, true]) {
const consoleErrors = collectConsoleErrors(comfyPage.page)
await comfyPage.settings.setSetting(
'Comfy.VueNodes.Enabled',
vueNodesEnabled
)
for (const edge of dragEdges) {
if (vueNodesEnabled && vueIncompatiblePacks.has(edge.producer.pack)) {
console.log(
`connectivity drag: ${edge.producer.pack} declares vueNodesCompatible=false; Vue drag not applicable`
)
continue
}
await comfyPage.nodeOps.clearGraph()
const producer = await comfyPage.nodeOps.addNode(
edge.producer.nodeType,
undefined,
{ x: 150, y: 200 }
)
const consumer = await comfyPage.nodeOps.addNode(
edge.consumer.nodeType,
undefined,
{ x: 700, y: 200 }
)
await comfyPage.nextFrame()
const [outIndex, inIndex] = await comfyPage.page.evaluate(
([producerId, consumerId, outName, inName]) => {
const byId = (id: string) =>
window.app!.graph.nodes.find((node) => String(node.id) === id)!
const src = byId(producerId)
const dst = byId(consumerId)
return [
src.outputs.findIndex((slot) => slot.name === outName),
dst.inputs.findIndex((slot) => slot.name === inName)
]
},
[
String(producer.id),
String(consumer.id),
edge.producer.slotName,
edge.consumer.slotName
] as const
)
const key = `${edge.producer.nodeType}.${edge.producer.slotName} -> ${edge.consumer.nodeType}.${edge.consumer.slotName}`
expect(outIndex, `${key}: producer slot on instance`).toBeGreaterThan(-1)
expect(inIndex, `${key}: consumer slot on instance`).toBeGreaterThan(-1)
if (vueNodesEnabled) {
await comfyPage.vueNodes.waitForNodes(2)
// Output-side mirror of getInputSlotConnectionDot, addressed by
// data-slot-key so shared-label ambiguity cannot misfire the drag.
const outDot = comfyPage.page
.locator(`[data-node-id="${String(producer.id)}"]`)
.locator('.lg-slot--output')
.filter({
has: comfyPage.page.locator(
`[data-slot-key="${String(producer.id)}-out-${outIndex}"]`
)
})
.getByTestId('slot-connection-dot')
const inDot = comfyPage.vueNodes.getInputSlotConnectionDot(
String(consumer.id),
inIndex
)
await outDot.dragTo(inDot)
} else {
await producer.connectOutput(outIndex, consumer, inIndex)
}
const linked = await comfyPage.page.evaluate(
([consumerId, index]) => {
const node = window.app!.graph.nodes.find(
(candidate) => String(candidate.id) === consumerId
)
return node?.inputs?.[Number(index)]?.link != null
},
[String(consumer.id), String(inIndex)] as const
)
expect(linked, `${key} with VueNodes=${vueNodesEnabled}`).toBe(true)
}
consoleErrors.stop()
expect(
consoleErrors.errors,
`console errors with VueNodes=${vueNodesEnabled}`
).toEqual([])
await expectNoVisibleErrors(
comfyPage.page,
`after drag pass VueNodes=${vueNodesEnabled}`
)
}
})

View File

@@ -0,0 +1,58 @@
import { readFileSync } from 'node:fs'
import { resolve } from 'node:path'
import type { ComfyWorkflowJSON } from '@/platform/workflow/validation/schemas/workflowSchema'
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
customNodeSuiteSettings,
dismissTemplatesDialog
} from '@e2e/fixtures/utils/customNodeSuite'
import { collectConsoleErrors } from '@e2e/fixtures/utils/consoleErrorCollector'
import { errorSurfaces } from '@e2e/fixtures/utils/errorSurfaces'
import { assetPath } from '@e2e/fixtures/utils/paths'
// Core-only, model-free workflow: the bundled default template references
// model files a scoped test backend does not have, which rightly trips the
// error surfaces this suite asserts are clean.
const smokeWorkflow = JSON.parse(
readFileSync(resolve(assetPath('customNodes/core_smoke.json')), 'utf-8')
) as ComfyWorkflowJSON
test.use({ initialSettings: customNodeSuiteSettings })
test.beforeEach(async ({ comfyPage }) => {
await dismissTemplatesDialog(comfyPage)
})
test.describe('smoke: core workflow', () => {
test('loads without console errors in both renderers', async ({
comfyPage
}) => {
for (const vueNodesEnabled of [false, true]) {
const consoleErrors = collectConsoleErrors(comfyPage.page)
await comfyPage.settings.setSetting(
'Comfy.VueNodes.Enabled',
vueNodesEnabled
)
await comfyPage.workflow.loadGraphData(smokeWorkflow)
await comfyPage.nextFrame()
consoleErrors.stop()
expect(await comfyPage.nodeOps.getGraphNodesCount()).toBeGreaterThan(0)
expect(
consoleErrors.errors,
`console errors (VueNodes=${vueNodesEnabled})`
).toEqual([])
for (const [surface, locator] of Object.entries(
errorSurfaces(comfyPage.page)
))
await expect(
locator,
`${surface} (VueNodes=${vueNodesEnabled})`
).toHaveCount(0)
}
})
})

View File

@@ -0,0 +1,188 @@
/* oxlint-disable playwright/no-skipped-test -- tiers conditionally skip when the target backend lacks the required packs (installed custom nodes or devtools); this is the framework's designed environment gating, not a disabled test */
import { existsSync, readFileSync } from 'node:fs'
import { resolve } from 'node:path'
import type { Page } from '@playwright/test'
import type { ComfyWorkflowJSON } from '@/platform/workflow/validation/schemas/workflowSchema'
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
customNodeSuiteSettings,
dismissTemplatesDialog
} from '@e2e/fixtures/utils/customNodeSuite'
import { LocalDesktopTarget } from '@e2e/fixtures/customNode/ComfyTarget'
import {
loadManifest,
rendererPassesFor
} from '@e2e/fixtures/customNode/manifest'
import { expectedNodesPresent } from '@e2e/fixtures/customNode/objectInfoValidator'
import { collectConsoleErrors } from '@e2e/fixtures/utils/consoleErrorCollector'
import { errorSurfaces } from '@e2e/fixtures/utils/errorSurfaces'
import { assetPath } from '@e2e/fixtures/utils/paths'
const target = new LocalDesktopTarget()
const OBJECT_INFO_SANITY_FLOOR = 50
test.use({ initialSettings: customNodeSuiteSettings })
test.beforeEach(async ({ comfyPage }) => {
await dismissTemplatesDialog(comfyPage)
})
async function expectNoVisibleErrors(
page: Page,
context: string
): Promise<void> {
for (const [surface, locator] of Object.entries(errorSurfaces(page)))
await expect(locator, `${context}: ${surface}`).toHaveCount(0)
}
function readWorkflow(relativePath: string): ComfyWorkflowJSON {
return JSON.parse(
readFileSync(resolve(relativePath), 'utf-8')
) as ComfyWorkflowJSON
}
async function nodeIdsByType(
page: Page,
classTypes: string[]
): Promise<string[]> {
return await page.evaluate((types) => {
const nodes = window.app!.graph.nodes ?? []
return nodes
.filter((node) => {
const n = node as { comfyClass?: string; type?: string }
return types.includes(n.comfyClass ?? n.type ?? '')
})
.map((node) => String(node.id))
}, classTypes)
}
for (const entry of loadManifest()) {
const workflowRelative = `browser_tests/${entry.workflow}`
test.describe(`custom node: ${entry.pack}`, () => {
test('T0 load: expected nodes register and render in both renderers', async ({
comfyPage
}) => {
test.setTimeout(entry.timeoutMs)
const objectInfo = await target.getObjectInfo(comfyPage.page)
expect(
Object.keys(objectInfo).length,
'object_info sanity floor'
).toBeGreaterThan(OBJECT_INFO_SANITY_FLOOR)
const { missing } = expectedNodesPresent(objectInfo, entry.expectedNodes)
test.skip(
missing.length > 0,
`${entry.pack} not installed on this backend (missing: ${missing.join(', ')})`
)
await expectNoVisibleErrors(comfyPage.page, 'at startup')
// vueNodesCompatible: false = canvas-only assertions; still runs, no skip.
const rendererPasses = rendererPassesFor(entry)
if (entry.vueNodesCompatible === false)
console.log(
`${entry.pack} declares vueNodesCompatible=false; Vue Nodes pass not applicable`
)
for (const vueNodesEnabled of rendererPasses) {
const consoleErrors = collectConsoleErrors(comfyPage.page)
await comfyPage.settings.setSetting(
'Comfy.VueNodes.Enabled',
vueNodesEnabled
)
await comfyPage.nodeOps.clearGraph()
const addedIds: string[] = []
for (const classType of entry.expectedNodes) {
const node = await comfyPage.nodeOps.addNode(classType)
addedIds.push(String(node.id))
}
await comfyPage.nextFrame()
expect(await comfyPage.nodeOps.getGraphNodesCount()).toBe(
entry.expectedNodes.length
)
// Vue Nodes 2.0 mounts each node as a [data-node-id] element; assert
// the pack's own nodes rendered, not just any node count.
if (vueNodesEnabled)
for (const id of addedIds)
await expect(comfyPage.vueNodes.getNodeLocator(id)).toBeVisible()
consoleErrors.stop()
expect(
consoleErrors.errors,
`console errors with VueNodes=${vueNodesEnabled}`
).toEqual([])
await expectNoVisibleErrors(
comfyPage.page,
`after VueNodes=${vueNodesEnabled} pass`
)
}
})
test('T1 run: workflow executes without error', async ({ comfyPage }) => {
test.setTimeout(entry.timeoutMs + 15_000)
const objectInfo = await target.getObjectInfo(comfyPage.page)
const { missing } = expectedNodesPresent(objectInfo, entry.expectedNodes)
test.skip(
!entry.tiers.includes('run') ||
missing.length > 0 ||
entry.requiresGpu ||
entry.requiresModels.length > 0 ||
!entry.workflow ||
!existsSync(resolve(workflowRelative)),
`run tier unavailable for ${entry.pack}`
)
await expectNoVisibleErrors(comfyPage.page, 'at startup')
await comfyPage.workflow.loadGraphData(readWorkflow(workflowRelative))
const result = await target.runWorkflow(comfyPage.page, {
expectedNodeIds: await nodeIdsByType(
comfyPage.page,
entry.expectedNodes
),
timeoutMs: entry.timeoutMs
})
expect(result.outcome, JSON.stringify(result.error ?? {})).toBe('PASS')
await expectNoVisibleErrors(comfyPage.page, 'after run')
})
})
}
test('harness self-check: captures a real execution error', async ({
comfyPage
}) => {
test.setTimeout(30_000)
const objectInfo = await target.getObjectInfo(comfyPage.page)
expect(
Object.keys(objectInfo).length,
'object_info sanity floor'
).toBeGreaterThan(OBJECT_INFO_SANITY_FLOOR)
test.skip(
!('DevToolsErrorRaiseNode' in objectInfo),
'ComfyUI_devtools not installed on this backend'
)
await comfyPage.workflow.loadGraphData(
readWorkflow(assetPath('nodes/execution_error.json'))
)
const result = await target.runWorkflow(comfyPage.page, {
expectedNodeIds: [],
timeoutMs: 15000
})
expect(result.outcome).toBe('EXECUTION_ERROR')
expect(result.error?.exceptionType).toBeTruthy()
// Proves the event tap captures node ids from the live `executing` stream
// (its detail is a bare string): the failing node starts before it raises.
expect(result.executedNodes.length).toBeGreaterThan(0)
// Positive control for the zero-visible-errors invariant: a real execution
// error MUST surface in the app's error overlay. If this fails, the
// expectNoVisibleErrors selectors have rotted and every clean assertion in
// this suite is meaningless.
await expect(errorSurfaces(comfyPage.page).errorOverlay).toBeVisible()
})

View File

@@ -0,0 +1,29 @@
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import {
loadManifest,
rendererPassesFor
} from '@e2e/fixtures/customNode/manifest'
test.describe('customNode manifest', () => {
test('loads entries with the shape the regression spec depends on', () => {
const entries = loadManifest()
expect(entries.length).toBeGreaterThan(0)
for (const entry of entries) {
expect(entry.pack).toBeTruthy()
expect(entry.expectedNodes.length).toBeGreaterThan(0)
expect(entry.tiers.length).toBeGreaterThan(0)
}
})
test('rendererPassesFor drops only the Vue pass, only on an explicit false', () => {
expect(rendererPassesFor({})).toEqual([false, true])
expect(rendererPassesFor({ vueNodesCompatible: true })).toEqual([
false,
true
])
expect(rendererPassesFor({ vueNodesCompatible: false })).toEqual([false])
})
})

View File

@@ -0,0 +1,47 @@
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import type { ObjectInfo } from '@e2e/fixtures/customNode/objectInfoValidator'
import {
expectedNodesPresent,
preValidate
} from '@e2e/fixtures/customNode/objectInfoValidator'
const objectInfo: ObjectInfo = {
KSampler: { input: { required: { model: {}, seed: {} } } }
}
test.describe('objectInfoValidator', () => {
test('expectedNodesPresent splits present from missing', () => {
const { present, missing } = expectedNodesPresent(objectInfo, [
'KSampler',
'Missing (rgthree)'
])
expect(present).toEqual(['KSampler'])
expect(missing).toEqual(['Missing (rgthree)'])
})
test('preValidate returns MISSING_NODE for an unregistered class', () => {
const failure = preValidate(objectInfo, [
{ id: '1', classType: 'Ghost', inputs: {} }
])
expect(failure?.outcome).toBe('MISSING_NODE')
})
test('preValidate returns VALIDATION_FAIL naming the missing required input', () => {
const failure = preValidate(objectInfo, [
{ id: '3', classType: 'KSampler', inputs: { model: 0 } }
])
expect(failure?.outcome).toBe('VALIDATION_FAIL')
expect(failure?.message).toContain('missing required input "seed"')
})
test('preValidate passes when every required input is present', () => {
expect(
preValidate(objectInfo, [
{ id: '3', classType: 'KSampler', inputs: { model: 0, seed: 1 } }
])
).toBeNull()
})
})

View File

@@ -0,0 +1,71 @@
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import { classifyRun } from '@e2e/fixtures/customNode/runResult'
test.describe('classifyRun', () => {
test('PASS when every expected node appears in the executing stream', () => {
const result = classifyRun({
events: [
{ type: 'execution_start' },
{ type: 'executing', node: '1' },
{ type: 'executing', node: '2' },
{ type: 'executing', node: null },
{ type: 'execution_success' }
],
expectedNodeIds: ['1', '2']
})
expect(result.outcome).toBe('PASS')
expect(result.executedNodes).toEqual(['1', '2'])
})
test('PARTIAL when a succeeding run replays a cached node that never emitted executing', () => {
const result = classifyRun({
events: [{ type: 'executing', node: '1' }, { type: 'execution_success' }],
expectedNodeIds: ['1', '2']
})
expect(result.outcome).toBe('PARTIAL')
expect(result.executedNodes).toEqual(['1'])
})
test('EXECUTION_ERROR captures the failing node details', () => {
const result = classifyRun({
events: [
{ type: 'executing', node: '1' },
{
type: 'execution_error',
error: { exceptionType: 'ValueError', nodeId: '1' }
}
],
expectedNodeIds: ['1']
})
expect(result.outcome).toBe('EXECUTION_ERROR')
expect(result.error?.exceptionType).toBe('ValueError')
})
test('EXECUTION_ERROR when the run is interrupted', () => {
const result = classifyRun({
events: [
{ type: 'executing', node: '1' },
{ type: 'execution_interrupted' }
],
expectedNodeIds: ['1']
})
expect(result.outcome).toBe('EXECUTION_ERROR')
})
test('TIMEOUT when flagged or when no terminal event arrived', () => {
const flagged = classifyRun({
events: [{ type: 'executing', node: '1' }],
expectedNodeIds: ['1'],
timedOut: true
})
const noTerminal = classifyRun({
events: [{ type: 'executing', node: '1' }],
expectedNodeIds: ['1']
})
expect(flagged.outcome).toBe('TIMEOUT')
expect(noTerminal.outcome).toBe('TIMEOUT')
})
})

View File

@@ -0,0 +1,139 @@
import {
comfyExpect as expect,
comfyPageFixture as test
} from '@e2e/fixtures/ComfyPage'
import type { RawNodeDef } from '@e2e/fixtures/customNode/typePairing'
import {
isTypeCompatible,
normalizeNodeDefs,
packOf,
planPairs
} from '@e2e/fixtures/customNode/typePairing'
const DEFS: Record<string, RawNodeDef> = {
LatentSource: {
input: { required: {} },
output: ['LATENT'],
output_name: ['LATENT'],
python_module: 'nodes'
},
LatentSink: {
input: { required: { latent: ['LATENT', {}] } },
output: [],
python_module: 'custom_nodes.SomePack'
},
UnionSource: {
input: { required: {} },
output: ['STRING,INT'],
output_name: ['value'],
python_module: 'nodes'
},
IntSink: {
input: { required: { value: ['int', {}] } },
output: [],
python_module: 'nodes'
},
ComboNode: {
input: { required: { choice: [['a', 'b'], {}] } },
output: [],
python_module: 'nodes'
},
SocketlessNode: {
input: { required: { hidden: ['STRING', { socketless: true }] } },
output: [],
python_module: 'nodes'
},
WildcardNode: {
input: { required: { anything: ['*', {}] } },
output: ['*'],
output_name: ['out'],
python_module: 'nodes'
},
OrphanNode: {
input: { required: {} },
output: ['NOBODY_CONSUMES_THIS'],
output_name: ['orphan'],
python_module: 'custom_nodes.OrphanPack'
}
}
test.describe('typePairing', () => {
test('isTypeCompatible mirrors the real validator semantics', () => {
expect(isTypeCompatible('LATENT', 'LATENT')).toBe(true)
expect(isTypeCompatible('latent', 'LATENT')).toBe(true)
expect(isTypeCompatible('LATENT', 'IMAGE')).toBe(false)
expect(isTypeCompatible('STRING,INT', 'INT')).toBe(true)
expect(isTypeCompatible('STRING,INT', 'FLOAT')).toBe(false)
expect(isTypeCompatible('*', 'ANYTHING')).toBe(true)
expect(isTypeCompatible('', 'ANYTHING')).toBe(true)
})
test('packOf attributes core vs custom pack', () => {
expect(packOf('nodes')).toBe('core')
expect(packOf('comfy_extras.nodes_x')).toBe('core')
expect(packOf('custom_nodes.ComfyUI-Impact-Pack')).toBe(
'ComfyUI-Impact-Pack'
)
expect(packOf(undefined)).toBe('core')
})
test('normalize maps COMBO literals and drops socketless inputs', () => {
const nodes = normalizeNodeDefs(DEFS)
const combo = nodes.find((n) => n.type === 'ComboNode')!
expect(combo.inputs).toEqual([{ name: 'choice', type: 'COMBO' }])
const socketless = nodes.find((n) => n.type === 'SocketlessNode')!
expect(socketless.inputs).toEqual([])
})
test('planPairs pairs exact and union types, deterministically', () => {
const nodes = normalizeNodeDefs(DEFS)
const plan = planPairs(nodes, ['LatentSink', 'IntSink'])
const keys = plan.pairs.map(
(p) =>
`${p.producer.nodeType}.${p.producer.slotName}->${p.consumer.nodeType}.${p.consumer.slotName}`
)
expect(keys).toContain('LatentSource.LATENT->LatentSink.latent')
expect(keys).toContain('UnionSource.value->IntSink.value')
const again = planPairs(nodes, ['LatentSink', 'IntSink'])
expect(again.pairs).toEqual(plan.pairs)
})
test('COMBO literals are excluded from pairing with names coerced to strings', () => {
const nodes = normalizeNodeDefs({
ComboSource: {
input: { required: {} },
output: [['A', 'B', 'C']],
output_name: [['A', 'B', 'C'] as unknown as string],
python_module: 'nodes'
},
...DEFS
})
const source = nodes.find((n) => n.type === 'ComboSource')!
expect(source.outputs).toEqual([{ name: 'COMBO', type: 'COMBO' }])
const plan = planPairs(nodes, ['ComboSource', 'ComboNode'])
expect(plan.pairs).toEqual([])
expect(plan.combos.map((s) => `${s.nodeType}.${s.slotName}`)).toEqual([
'ComboSource.COMBO',
'ComboNode.choice'
])
})
test('wildcard slots are excluded, orphan types recorded not failed', () => {
const nodes = normalizeNodeDefs(DEFS)
const plan = planPairs(nodes, ['WildcardNode', 'OrphanNode'])
expect(plan.wildcards.map((w) => w.nodeType)).toEqual([
'WildcardNode',
'WildcardNode'
])
expect(plan.orphans).toEqual([
{
nodeType: 'OrphanNode',
pack: 'OrphanPack',
slotName: 'orphan',
slotType: 'NOBODY_CONSUMES_THIS',
dir: 'out'
}
])
expect(plan.pairs).toEqual([])
})
})

View File

@@ -52,6 +52,15 @@
"test:browser": "pnpm exec playwright test",
"test:browser:coverage": "cross-env COLLECT_COVERAGE=true pnpm test:browser",
"test:browser:local": "cross-env PLAYWRIGHT_LOCAL=1 PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm test:browser",
"test:custom-nodes": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm exec playwright test browser_tests/tests/customNodes/ --config playwright.chrome.config.ts --workers=1",
"test:custom-nodes:watch": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:5173 PLAYWRIGHT_LOCAL=1 SLOW_MO=300 pnpm exec playwright test browser_tests/tests/customNodes/customNode.regression.spec.ts browser_tests/tests/customNodes/connectivity.spec.ts --config playwright.chrome.config.ts --workers=1 --headed",
"test:custom-nodes:debug": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm exec playwright test browser_tests/tests/customNodes/customNode.regression.spec.ts browser_tests/tests/customNodes/connectivity.spec.ts --config playwright.chrome.config.ts --workers=1 --debug",
"test:custom-nodes:impact-render": "pnpm test:custom-nodes:debug -g \"ComfyUI-Impact-Pack.*T0\"",
"test:custom-nodes:impact-run": "pnpm test:custom-nodes:debug -g \"ComfyUI-Impact-Pack.*T1\"",
"test:custom-nodes:vhs-render": "pnpm test:custom-nodes:debug -g \"VideoHelperSuite.*T0\"",
"test:custom-nodes:vhs-run": "pnpm test:custom-nodes:debug -g \"VideoHelperSuite.*T1\"",
"test:custom-nodes:connectivity": "pnpm test:custom-nodes:debug -g \"connectivity\"",
"test:custom-nodes:self-check": "pnpm test:custom-nodes:watch -g \"self-check\"",
"test:coverage": "vitest run --coverage",
"test:coverage:critical": "cross-env COVERAGE_CRITICAL=true vitest run --coverage",
"test:unit": "vitest run",

View File

@@ -0,0 +1,10 @@
import { defineConfig } from '@playwright/test'
import base from './playwright.config'
// Run against the system-installed Google Chrome (no bundled-chromium download).
// trace stays off: Playwright's trace recorder crashes pages under the branded
// Chrome channel on this machine (instant browser close, reported as timeout).
export default defineConfig(base, {
use: { channel: 'chrome', video: 'off', trace: 'off' }
})

View File

@@ -140,6 +140,7 @@ describe('loadTurnstile', () => {
const promise = loadTurnstile()
scriptEl()!.dispatchEvent(new Event('load'))
// global never published; deadline elapses
// oxlint-disable-next-line vitest/valid-expect -- deliberately awaited after the timer advance below; awaiting here would deadlock fake timers
const assertion = expect(promise).rejects.toThrow(/timed out/i)
await vi.advanceTimersByTimeAsync(10_000)
@@ -177,6 +178,7 @@ describe('loadTurnstile', () => {
const loadTurnstile = await freshLoadTurnstile()
const promise = loadTurnstile()
// oxlint-disable-next-line vitest/valid-expect -- deliberately awaited after the timer advance below; awaiting here would deadlock fake timers
const assertion = expect(promise).rejects.toThrow(/timed out/i)
vi.advanceTimersByTime(10_000)
@@ -216,6 +218,7 @@ describe('loadTurnstile', () => {
const loadTurnstile = await freshLoadTurnstile()
const promise = loadTurnstile()
// oxlint-disable-next-line vitest/valid-expect -- deliberately awaited after the timer advance below; awaiting here would deadlock fake timers
const assertion = expect(promise).rejects.toThrow(/timed out/i)
await vi.advanceTimersByTimeAsync(10_000)