[PR #13] [MERGED] Exercise custom-node packs on real GPUs #11

New issue

Closed

opened 2026-10-31 22:06:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 22:06:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyci-runpod-worker/pull/13
Author: @marawan206
Created: 10/19/2026
Status: ✅ Merged
Merged: 10/19/2026
Merged by: @marawan206

Base: main ← Head: pack-workflows

📝 Commits (5)

8718304 Exercise custom-node packs on real GPUs
4328639 Add a synthetic input clip for the VideoHelperSuite workflow
19bd941 vhs_video_roundtrip: decode the synthetic clip instead of the frontend's test asset
bd15ccc Enable the four custom-node pack workflows on the cu130 lane
f12ad97 README: how to tell and flush stale workers after a repoint

📊 Changes

28 files changed (+2185 additions, -62 deletions)

View changed files

📝 .github/workflows/build-image.yml (+3 -1)
📝 .github/workflows/scripts-tests.yml (+4 -3)
📝 Dockerfile (+26 -2)
📝 README.md (+106 -7)
📝 docker/extra_model_paths.yaml (+3 -0)
➕ manifest/custom_nodes.json (+55 -0)
➕ manifest/inputs/comfyci_clip.mp4 (+0 -0)
📝 manifest/workflows.json (+105 -0)
➕ manifest/workflows/impact_sdxl_detailer_api.json (+107 -0)
➕ manifest/workflows/kjnodes_sdxl_t2i_api.json (+102 -0)
➕ manifest/workflows/manager_import_smoke_api.json (+18 -0)
➕ manifest/workflows/vhs_video_roundtrip_api.json (+43 -0)
📝 scripts/golden_baseline.py (+13 -2)
📝 scripts/index_builder.py (+4 -0)
📝 scripts/lanes.py (+73 -8)
➕ scripts/packs.py (+90 -0)
📝 scripts/run_regression.py (+20 -4)
📝 scripts/submit_and_poll.py (+10 -4)
➕ scripts/tests/test_custom_nodes_worker.py (+471 -0)
📝 scripts/tests/test_golden_baseline.py (+22 -0)

...and 8 more files

📄 Description

What

Custom-node packs exercised on real GPUs, bit-compared against goldens like every other regression workflow. This is the GPU counterpart of the frontend's custom-node gate: that gate runs ComfyUI with --cpu on GitHub runners, so anything that only breaks with CUDA kernels, half-precision paths or GPU detectors is invisible to it. The stable-release bar names breaks in KJNodes / VideoHelperSuite / Manager as blockers; after this PR a core commit or a torch lane that breaks one of them shows up on ci.comfy.org within the hour, with a diff image.

manifest/custom_nodes.json: the packs, keyed by their custom_nodes directory name and pinned by full commit to what the frontend's gate tests (browser_tests/fixtures/data/customNodeManifest.core.json): KJNodes e27a505b, VideoHelperSuite 115de7a9, Impact-Pack 429d0159; Impact-Subpack 50c7b71a (it provides the Ultralytics detector); Manager tag 4.3 f764cc0e as the pip package it now is, loaded with --enable-manager and an offline config. Pins are bumped by PR; a bump rebuilds the images and the pack goldens are re-blessed.
Image: images built from lanes.json bake every pack's requirements at its pin under the lane's torch pins (plus registry pins for the two opencv variants), fetch the sources into a cache, and record the requirements hash per pack. sam2 (an Impact-Pack requirement) installs without build isolation and without its CUDA extension; libgl1 is added for the non-headless opencv.
Worker (src/custom_nodes.py): per job, installs exactly the packs the job asks for at their pinned sha (copy from the image cache, fetch if the pin moved, pip only when the requirements hash differs from the baked one), removes packs a previous job left behind, writes pack config files, merges pack server flags, then reads ComfyUI's "Import times for custom nodes" summary (colour codes and level tags stripped) from the server log. A requested pack that failed to import or is missing turns the run into custom_node_error (verdict execution_error) even when the workflow uses none of its nodes; a pack whose pin could not be fetched or installed is custom_node_install_error (infra, retried once). A server that dies at startup is blamed on a pack only when the log proves pack loading was reached. Jobs without packs are unchanged byte for byte.
Lane filter: a new optional lanes list per workflow, honoured by run_regression.py, golden_baseline.py and lanes.py's per-commit planning. The four pack workflows run on py313-torch2.14.1-cu130 only, since the legacy cu128 image is never rebuilt; the dashboard shows "not run" there and needs no change.
Four workflows, enabled on the cu130 lane with goldens blessed at v0.42.0:
- kjnodes_sdxl_t2i: SDXL through ImageResizeKJv2, ImageConcanate and ColorMatchV2.
- vhs_video_roundtrip: VHS_LoadVideo on a small synthetic clip committed under manifest/inputs/ (the frontend's plain_video.mp4 asset is an excerpt of a film, and the golden frames would have republished it on the results branch), frames saved as PNG (bit-compared) and recombined to a gif (encode path).
- impact_sdxl_detailer: SDXL portrait, UltralyticsDetectorProvider (bbox/face_yolov8m.pt) → BboxDetectorSEGS → DetailerForEach.
- manager_import_smoke: a trivial graph whose only purpose is to fail when Manager does not load.
Run records, summaries and the index carry the packs' install source, pin and import status; blessed goldens record the pack pins they were generated with. README has a new "Custom-node packs" section and the verdict table is updated.

Verification

python -m pytest scripts/tests -q: 341 passed (253 before; the new test_custom_nodes_worker.py installs from a real local bare repo, test_packs.py, and the lint/lanes/run_regression/golden_baseline/submit_and_poll suites extended). python scripts/lanes.py validate clean.
Every class_type and input name of the four API prompts was checked against the pinned pack sources and ComfyUI core; the import-summary parser was checked against ComfyUI's current coloured log format.
GPU validation on the cu130 lane, image cu130-t2141-8718304 built from this branch (details in the follow-up comment): all four pack workflows generated goldens with identical noise floors (r1 == r2), every pack imported from the image cache (source: cache, install under 0.1 s, import: ok, Manager loaded as a baked pip package), validation clean (no missing nodes, nothing stripped); exec times kjnodes 45 s, impact detailer 39 s, VHS 0.5 s, Manager smoke 0.5 s; outputs inspected (the detailer portrait has a clearly detectable face; the KJNodes concat shows the resized, colour-matched pair; the VHS frames are the synthetic clip). The nine core workflows were re-run on the same image and all nine still match their goldens, so the pack dependencies did not move anything the core workflows depend on.

Notes

Rollout gotcha found on the way: after the repoint, RunPod kept four stale workers on the old image serving jobs for over 30 minutes while the new-image workers sat THROTTLED, so the first golden runs silently ran on the old image (pack workflows failed validation, the pack-less Manager smoke produced a golden without pack status). Scaling the endpoint's max workers to 0 and back flushed them; the README's lane notes say so.
Known soft spots, documented in the README: sam2 comes from its git HEAD (Impact-Pack does not pin it); both opencv variants are present and pinned together; a detailer that finds no face silently returns the input (the golden is inspected by eye for that reason).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyci-runpod-worker/pull/13 **Author:** [@marawan206](https://github.com/marawan206) **Created:** 10/19/2026 **Status:** ✅ Merged **Merged:** 10/19/2026 **Merged by:** [@marawan206](https://github.com/marawan206) **Base:** `main` ← **Head:** `pack-workflows` --- ### 📝 Commits (5) - [`8718304`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/8718304972ba3869024ec49c0b61b4ab58ba2374) Exercise custom-node packs on real GPUs - [`4328639`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/43286394d73914d283e5daee12d82d011d4630fe) Add a synthetic input clip for the VideoHelperSuite workflow - [`19bd941`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/19bd9414fa80451b63125f450493fc2d0695c6e9) vhs_video_roundtrip: decode the synthetic clip instead of the frontend's test asset - [`bd15ccc`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/bd15ccc1b8c84a674a812a4b40ff3083013d4069) Enable the four custom-node pack workflows on the cu130 lane - [`f12ad97`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/f12ad978680b9758b9e252693dfeaca5835e331f) README: how to tell and flush stale workers after a repoint ### 📊 Changes **28 files changed** (+2185 additions, -62 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/build-image.yml` (+3 -1) 📝 `.github/workflows/scripts-tests.yml` (+4 -3) 📝 `Dockerfile` (+26 -2) 📝 `README.md` (+106 -7) 📝 `docker/extra_model_paths.yaml` (+3 -0) ➕ `manifest/custom_nodes.json` (+55 -0) ➕ `manifest/inputs/comfyci_clip.mp4` (+0 -0) 📝 `manifest/workflows.json` (+105 -0) ➕ `manifest/workflows/impact_sdxl_detailer_api.json` (+107 -0) ➕ `manifest/workflows/kjnodes_sdxl_t2i_api.json` (+102 -0) ➕ `manifest/workflows/manager_import_smoke_api.json` (+18 -0) ➕ `manifest/workflows/vhs_video_roundtrip_api.json` (+43 -0) 📝 `scripts/golden_baseline.py` (+13 -2) 📝 `scripts/index_builder.py` (+4 -0) 📝 `scripts/lanes.py` (+73 -8) ➕ `scripts/packs.py` (+90 -0) 📝 `scripts/run_regression.py` (+20 -4) 📝 `scripts/submit_and_poll.py` (+10 -4) ➕ `scripts/tests/test_custom_nodes_worker.py` (+471 -0) 📝 `scripts/tests/test_golden_baseline.py` (+22 -0) _...and 8 more files_ </details> ### 📄 Description ## What Custom-node packs exercised on real GPUs, bit-compared against goldens like every other regression workflow. This is the GPU counterpart of the frontend's custom-node gate: that gate runs ComfyUI with `--cpu` on GitHub runners, so anything that only breaks with CUDA kernels, half-precision paths or GPU detectors is invisible to it. The stable-release bar names breaks in KJNodes / VideoHelperSuite / Manager as blockers; after this PR a core commit or a torch lane that breaks one of them shows up on ci.comfy.org within the hour, with a diff image. - **`manifest/custom_nodes.json`**: the packs, keyed by their `custom_nodes` directory name and pinned by full commit to what the frontend's gate tests (`browser_tests/fixtures/data/customNodeManifest.core.json`): KJNodes `e27a505b`, VideoHelperSuite `115de7a9`, Impact-Pack `429d0159`; Impact-Subpack `50c7b71a` (it provides the Ultralytics detector); Manager tag 4.3 `f764cc0e` as the pip package it now is, loaded with `--enable-manager` and an offline config. Pins are bumped by PR; a bump rebuilds the images and the pack goldens are re-blessed. - **Image**: images built from `lanes.json` bake every pack's requirements at its pin under the lane's torch pins (plus registry pins for the two opencv variants), fetch the sources into a cache, and record the requirements hash per pack. sam2 (an Impact-Pack requirement) installs without build isolation and without its CUDA extension; `libgl1` is added for the non-headless opencv. - **Worker** (`src/custom_nodes.py`): per job, installs exactly the packs the job asks for at their pinned sha (copy from the image cache, fetch if the pin moved, pip only when the requirements hash differs from the baked one), removes packs a previous job left behind, writes pack config files, merges pack server flags, then reads ComfyUI's "Import times for custom nodes" summary (colour codes and level tags stripped) from the server log. A requested pack that failed to import or is missing turns the run into **`custom_node_error`** (verdict `execution_error`) even when the workflow uses none of its nodes; a pack whose pin could not be fetched or installed is `custom_node_install_error` (infra, retried once). A server that dies at startup is blamed on a pack only when the log proves pack loading was reached. Jobs without packs are unchanged byte for byte. - **Lane filter**: a new optional `lanes` list per workflow, honoured by `run_regression.py`, `golden_baseline.py` and `lanes.py`'s per-commit planning. The four pack workflows run on `py313-torch2.14.1-cu130` only, since the legacy cu128 image is never rebuilt; the dashboard shows "not run" there and needs no change. - **Four workflows**, enabled on the cu130 lane with goldens blessed at v0.42.0: - `kjnodes_sdxl_t2i`: SDXL through ImageResizeKJv2, ImageConcanate and ColorMatchV2. - `vhs_video_roundtrip`: VHS_LoadVideo on a small synthetic clip committed under `manifest/inputs/` (the frontend's `plain_video.mp4` asset is an excerpt of a film, and the golden frames would have republished it on the results branch), frames saved as PNG (bit-compared) and recombined to a gif (encode path). - `impact_sdxl_detailer`: SDXL portrait, UltralyticsDetectorProvider (`bbox/face_yolov8m.pt`) → BboxDetectorSEGS → DetailerForEach. - `manager_import_smoke`: a trivial graph whose only purpose is to fail when Manager does not load. - Run records, summaries and the index carry the packs' install source, pin and import status; blessed goldens record the pack pins they were generated with. README has a new "Custom-node packs" section and the verdict table is updated. ## Verification - `python -m pytest scripts/tests -q`: 341 passed (253 before; the new `test_custom_nodes_worker.py` installs from a real local bare repo, `test_packs.py`, and the lint/lanes/run_regression/golden_baseline/submit_and_poll suites extended). `python scripts/lanes.py validate` clean. - Every `class_type` and input name of the four API prompts was checked against the pinned pack sources and ComfyUI core; the import-summary parser was checked against ComfyUI's current coloured log format. - GPU validation on the cu130 lane, image `cu130-t2141-8718304` built from this branch (details in the follow-up comment): all four pack workflows generated goldens with identical noise floors (r1 == r2), every pack imported from the image cache (`source: cache`, install under 0.1 s, `import: ok`, Manager loaded as a baked pip package), validation clean (no missing nodes, nothing stripped); exec times kjnodes 45 s, impact detailer 39 s, VHS 0.5 s, Manager smoke 0.5 s; outputs inspected (the detailer portrait has a clearly detectable face; the KJNodes concat shows the resized, colour-matched pair; the VHS frames are the synthetic clip). The nine core workflows were re-run on the same image and all nine still match their goldens, so the pack dependencies did not move anything the core workflows depend on. ## Notes - Rollout gotcha found on the way: after the repoint, RunPod kept four stale workers on the old image serving jobs for over 30 minutes while the new-image workers sat THROTTLED, so the first golden runs silently ran on the old image (pack workflows failed validation, the pack-less Manager smoke produced a golden without pack status). Scaling the endpoint's max workers to 0 and back flushed them; the README's lane notes say so. - Known soft spots, documented in the README: sam2 comes from its git HEAD (Impact-Pack does not pin it); both opencv variants are present and pinned together; a detailer that finds no face silently returns the input (the golden is inspected by eye for that reason). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 22:06:14 +00:00