[PR #5] [MERGED] Stage Krea 2, Z-Image turbo and Qwen Image 2.1 edit regression workflows #5

New issue

Closed

opened 2026-10-31 22:06:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 22:06:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyci-runpod-worker/pull/5
Author: @marawan206
Created: 10/8/2026
Status: ✅ Merged
Merged: 10/9/2026
Merged by: @marawan206

Base: main ← Head: add-image-model-workflows

📝 Commits (5)

0c1a5f0 Lint the manifest against its workflows in the script tests
1f7b607 Stage Krea 2, Z-Image turbo and Qwen Image 2.1 edit workflows
7b85dbf Lint for unused models and nodes that do not feed the SaveImage
2306d97 Merge remote-tracking branch 'origin/main' into add-image-model-workflows
a92baee Merge remote-tracking branch 'origin/main' into add-image-model-workflows

📊 Changes

7 files changed (+559 additions, -6 deletions)

View changed files

📝 .github/workflows/scripts-tests.yml (+4 -3)
📝 README.md (+15 -3)
📝 manifest/workflows.json (+105 -0)
➕ manifest/workflows/krea2_turbo_t2i_api.json (+73 -0)
➕ manifest/workflows/qwen_image_2_1_edit_api.json (+84 -0)
➕ manifest/workflows/z_image_turbo_t2i_api.json (+81 -0)
➕ scripts/tests/test_manifest_lint.py (+197 -0)

📄 Description

Summary

Adds three official Comfy template workflows to the GPU regression suite, staged ("enabled": false), so sync-models can put their models on the network volume before goldens are generated. Also adds a CPU-only manifest lint to the script tests and runs those tests on manifest/** changes.

id	Template	Models (new on volume)	Min ComfyUI tag	timeout_s
`krea2_turbo_t2i`	`image_krea2_turbo_t2i`	`krea2_turbo_fp8_scaled` 13.14 GB, `qwen3vl_4b_fp8_scaled` 5.24 GB, `qwen_image_vae` 0.25 GB	v0.26.0	1200
`z_image_turbo_t2i`	`image_z_image_turbo`	`z_image_turbo_bf16` 12.31 GB, `qwen_3_4b` 8.04 GB, `vae/ae.safetensors` (already on the volume, see below)	v0.3.74	900
`qwen_image_2_1_edit`	`image_qwen_image_2_1_image_edit`	`qwen_image_2.1_int8_convrot` 7.26 GB, `qwen3vl_8b_int8_convrot` 9.35 GB, `qwen_image_2.1_vae_bf16` 0.68 GB, 2 input PNGs (2.5 MB)	v0.37.0	2400

All model URLs come from the templates' properties.models in Comfy-Org/workflow_templates. Every new URL was HEAD-checked (302 from Hugging Face, not gated), and each entry pins the sha256 that Hugging Face reports.
New downloads total about 56.3 GB, which takes the volume to about 97 of 150 GB.
v0.39.2 (latest) has every node, so goldens can be generated there for all three.
Z-Image's vae/ae.safetensors is byte-identical to the FLUX.1-dev ae.safetensors already on the volume: same size, same sha256 afc8e282…, same LFS pointer. The entry reuses the existing FLUX URL so the sync treats them as one file. That URL needs HF_TOKEN, which is already configured for FLUX.
Output sizes: 1024x1024 for the two text-to-image workflows, 896x1152 for the edit. Each is about 1–2 MB as PNG, far below the 15 MiB inline cap.

Simplifications vs. the templates (and why)

The goal is a stable image-model regression signal that loads only the models that actually run.

Switches resolved to their default branch, and the unused branch dropped. This removes the ComfySwitchNodes along with the disabled krea2_darkbrush LoRA and the qwen3.5_9b…pe_i2i prompt-enhancer CLIP (8.8 GB that would never run).
Krea 2 prompt refinement turned off. The template's default sends the prompt through an LLM (TextGenerate, sampling on) before encoding. A tiny numeric change in the LLM can rewrite the whole prompt and change every pixel, which hides whether the image model itself regressed. The user prompt now goes straight into CLIPTextEncode, with the text unchanged. Text generation deserves its own workflow; I can add one separately.
Krea 2: ResolutionSelector(1:1, 1 MP, ×8) is replaced by its result, 1024x1024 (the node computes megapixels × 1024 × 1024). The PreviewAny, system-prompt and concatenate nodes are dropped.
Qwen edit: keeps TextEncodeQwenImage21 with both reference images (resolution: 0) and the default prompt "Replace costumes for the character". The latent comes from the encoder's reference-sized output, which is the default branch. ImageCompare is dropped. QwenImage21Cache is kept at auto/default, which is lossless.
SaveImageAdvanced (png, 8-bit, sRGB) replaced by SaveImage. For 8-bit PNG both produce the same pixels: ×255, clamp, truncate to uint8, and the colorspace option doesn't change PNG pixels. SaveImage exists in every tag and avoids the dotted format.* inputs that the runner's schema adaptation could strip.
Every filename_prefix is the workflow id. Seeds are the template values, and each has a matching seed_overrides entry (30:3, 57:3, 459:458). Node ids are kept verbatim, including the ones with :.

Input images

The worker can't receive files, and the per-commit checkout runs git clean in ComfyUI. So the two reference PNGs are listed under models with "directory": "input", pinned to workflow_templates commit 96309bd with their sha256. sync-models puts them at /runpod-volume/models/input/, and the entry sets "comfy_flags": ["--input-directory", "/runpod-volume/models/input"]. --input-directory sets ComfyUI's input folder, which LoadImage reads from (in every tag since at least v0.26.0), and the worker's missing_models check covers these files like any other model. No image rebuild is needed.

Manifest lint

scripts/tests/test_manifest_lint.py loads the real manifest and every workflow it references, and checks:

each workflow is valid API format, and every link points to an existing node with an integer output index;
every node feeds a SaveImage, so no leftover preview or helper nodes run extra models;
the files a workflow loads (*_name loader inputs, any model-file extension, LoadImage.image) match that entry's models in both directions: nothing loaded is missing, and nothing synced goes unused;
every seed override targets an existing node input;
model directories are among the 12 extra_model_paths folders, or input with --input-directory;
a directory/name shared between workflows points at one URL and sha256;
there is a SaveImage whose prefix is the workflow id;
timeout_s is below the 3600 s endpoint limit.

scripts-tests.yml now also runs on manifest/** and docker/extra_model_paths.yaml.

Feasibility (RTX 4090 24 GB, torch 2.11 cu128)

The int8 convrot weights (int8_tensorwise, convrot, group 256) run on cu128. comfy/quant_ops.py turns off comfy-kitchen's CUDA kernels below CUDA 13, and comfy-kitchen's plain-PyTorch int8_linear (Hadamard rotation plus torch._int_mm) takes over: slower, but it works.
The largest single model is 13.1 GB. ComfyUI unloads the text encoder before sampling if it needs the room, so all three fit in 24 GB.
None of these models use the parts of comfy.ldm.flux that ComfyUI #16488 changed. They only import EmbedND, apply_rope and timestep_embedding.

Rollout

Merge. sync-models runs and downloads about 56 GB. If it hits the endpoint's 3600 s limit, dispatch it again; it keeps finished files.
For each workflow, run Golden baselines with ref=v0.39.2, workflows=<id>, bless=false.
Inspect golden/<id>/v0.39.2/outputs/ and noise_floor.json (r1 vs r2), and check prompt_exec_s against the timeouts.
Re-run with bless_only=true.
Follow-up PR flips "enabled": true (and tightens timeout_s if the measured times allow).

Risks

Sync time: the first sync is large. It may need a second dispatch, which is safe to re-run.
CUDA dependence: Qwen 2.1 and Krea 2 goldens depend on the worker's CUDA. Moving the image to cu130 switches int8/fp8 to comfy-kitchen's CUDA kernels and needs a re-bless.
Timeouts: the Qwen edit timeout (2400 s) is an estimate that allows for the slower fallback path and a cold load from the network volume.
Merge conflict: scripts-tests.yml is also added by #2. Whichever merges second gets a small conflict; keep this version, which only adds triggers.
Template drift: the trimmed workflows differ from what the templates do by default (no LLM refinement in Krea 2), by design. They test the image models, not the full template experience.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyci-runpod-worker/pull/5 **Author:** [@marawan206](https://github.com/marawan206) **Created:** 10/8/2026 **Status:** ✅ Merged **Merged:** 10/9/2026 **Merged by:** [@marawan206](https://github.com/marawan206) **Base:** `main` ← **Head:** `add-image-model-workflows` --- ### 📝 Commits (5) - [`0c1a5f0`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/0c1a5f0de881c62d66b19e2dedfd85444bce7a69) Lint the manifest against its workflows in the script tests - [`1f7b607`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/1f7b607c9a0639c3ba79d5a0dcb8d19ad1af2a6a) Stage Krea 2, Z-Image turbo and Qwen Image 2.1 edit workflows - [`7b85dbf`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/7b85dbfd9ad0c47875ce289557be54c6fd582aad) Lint for unused models and nodes that do not feed the SaveImage - [`2306d97`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/2306d97354bd8dc7af4373ad428a5dc49efb1e2e) Merge remote-tracking branch 'origin/main' into add-image-model-workflows - [`a92baee`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/a92baeeb12f69a840ff29d4722d22af32dc88b4c) Merge remote-tracking branch 'origin/main' into add-image-model-workflows ### 📊 Changes **7 files changed** (+559 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/scripts-tests.yml` (+4 -3) 📝 `README.md` (+15 -3) 📝 `manifest/workflows.json` (+105 -0) ➕ `manifest/workflows/krea2_turbo_t2i_api.json` (+73 -0) ➕ `manifest/workflows/qwen_image_2_1_edit_api.json` (+84 -0) ➕ `manifest/workflows/z_image_turbo_t2i_api.json` (+81 -0) ➕ `scripts/tests/test_manifest_lint.py` (+197 -0) </details> ### 📄 Description ## Summary Adds three official Comfy template workflows to the GPU regression suite, **staged** (`"enabled": false`), so `sync-models` can put their models on the network volume before goldens are generated. Also adds a CPU-only manifest lint to the script tests and runs those tests on `manifest/**` changes. | id | Template | Models (new on volume) | Min ComfyUI tag | timeout_s | |---|---|---|---|---| | `krea2_turbo_t2i` | `image_krea2_turbo_t2i` | `krea2_turbo_fp8_scaled` 13.14 GB, `qwen3vl_4b_fp8_scaled` 5.24 GB, `qwen_image_vae` 0.25 GB | v0.26.0 | 1200 | | `z_image_turbo_t2i` | `image_z_image_turbo` | `z_image_turbo_bf16` 12.31 GB, `qwen_3_4b` 8.04 GB, `vae/ae.safetensors` (already on the volume, see below) | v0.3.74 | 900 | | `qwen_image_2_1_edit` | `image_qwen_image_2_1_image_edit` | `qwen_image_2.1_int8_convrot` 7.26 GB, `qwen3vl_8b_int8_convrot` 9.35 GB, `qwen_image_2.1_vae_bf16` 0.68 GB, 2 input PNGs (2.5 MB) | v0.37.0 | 2400 | - All model URLs come from the templates' `properties.models` in Comfy-Org/workflow_templates. Every new URL was HEAD-checked (302 from Hugging Face, not gated), and each entry pins the `sha256` that Hugging Face reports. - New downloads total about **56.3 GB**, which takes the volume to about 97 of 150 GB. - **v0.39.2 (latest) has every node**, so goldens can be generated there for all three. - Z-Image's `vae/ae.safetensors` is byte-identical to the FLUX.1-dev `ae.safetensors` already on the volume: same size, same sha256 `afc8e282…`, same LFS pointer. The entry reuses the existing FLUX URL so the sync treats them as one file. That URL needs `HF_TOKEN`, which is already configured for FLUX. - Output sizes: 1024x1024 for the two text-to-image workflows, 896x1152 for the edit. Each is about 1–2 MB as PNG, far below the 15 MiB inline cap. ## Simplifications vs. the templates (and why) The goal is a stable image-model regression signal that loads only the models that actually run. - **Switches resolved to their default branch, and the unused branch dropped.** This removes the `ComfySwitchNode`s along with the disabled `krea2_darkbrush` LoRA and the `qwen3.5_9b…pe_i2i` prompt-enhancer CLIP (8.8 GB that would never run). - **Krea 2 prompt refinement turned off.** The template's default sends the prompt through an LLM (`TextGenerate`, sampling on) before encoding. A tiny numeric change in the LLM can rewrite the whole prompt and change every pixel, which hides whether the image model itself regressed. The user prompt now goes straight into `CLIPTextEncode`, with the text unchanged. Text generation deserves its own workflow; I can add one separately. - **Krea 2:** `ResolutionSelector(1:1, 1 MP, ×8)` is replaced by its result, 1024x1024 (the node computes `megapixels × 1024 × 1024`). The `PreviewAny`, system-prompt and concatenate nodes are dropped. - **Qwen edit:** keeps `TextEncodeQwenImage21` with both reference images (`resolution: 0`) and the default prompt "Replace costumes for the character". The latent comes from the encoder's reference-sized output, which is the default branch. `ImageCompare` is dropped. `QwenImage21Cache` is kept at `auto`/`default`, which is lossless. - **`SaveImageAdvanced` (png, 8-bit, sRGB) replaced by `SaveImage`.** For 8-bit PNG both produce the same pixels: ×255, clamp, truncate to uint8, and the colorspace option doesn't change PNG pixels. `SaveImage` exists in every tag and avoids the dotted `format.*` inputs that the runner's schema adaptation could strip. - Every `filename_prefix` is the workflow id. Seeds are the template values, and each has a matching `seed_overrides` entry (`30:3`, `57:3`, `459:458`). Node ids are kept verbatim, including the ones with `:`. ## Input images The worker can't receive files, and the per-commit checkout runs `git clean` in ComfyUI. So the two reference PNGs are listed under `models` with `"directory": "input"`, pinned to workflow_templates commit `96309bd` with their sha256. `sync-models` puts them at `/runpod-volume/models/input/`, and the entry sets `"comfy_flags": ["--input-directory", "/runpod-volume/models/input"]`. `--input-directory` sets ComfyUI's input folder, which `LoadImage` reads from (in every tag since at least v0.26.0), and the worker's `missing_models` check covers these files like any other model. No image rebuild is needed. ## Manifest lint `scripts/tests/test_manifest_lint.py` loads the real manifest and every workflow it references, and checks: - each workflow is valid API format, and every link points to an existing node with an integer output index; - every node feeds a `SaveImage`, so no leftover preview or helper nodes run extra models; - the files a workflow loads (`*_name` loader inputs, any model-file extension, `LoadImage.image`) match that entry's `models` **in both directions**: nothing loaded is missing, and nothing synced goes unused; - every seed override targets an existing node input; - model directories are among the 12 `extra_model_paths` folders, or `input` with `--input-directory`; - a `directory/name` shared between workflows points at one URL and sha256; - there is a `SaveImage` whose prefix is the workflow id; - `timeout_s` is below the 3600 s endpoint limit. `scripts-tests.yml` now also runs on `manifest/**` and `docker/extra_model_paths.yaml`. ## Feasibility (RTX 4090 24 GB, torch 2.11 cu128) - The int8 convrot weights (`int8_tensorwise`, convrot, group 256) run on cu128. `comfy/quant_ops.py` turns off comfy-kitchen's CUDA kernels below CUDA 13, and comfy-kitchen's plain-PyTorch `int8_linear` (Hadamard rotation plus `torch._int_mm`) takes over: slower, but it works. - The largest single model is 13.1 GB. ComfyUI unloads the text encoder before sampling if it needs the room, so all three fit in 24 GB. - None of these models use the parts of `comfy.ldm.flux` that ComfyUI #16488 changed. They only import `EmbedND`, `apply_rope` and `timestep_embedding`. ## Rollout 1. Merge. `sync-models` runs and downloads about 56 GB. If it hits the endpoint's 3600 s limit, dispatch it again; it keeps finished files. 2. For each workflow, run **Golden baselines** with `ref=v0.39.2`, `workflows=<id>`, `bless=false`. 3. Inspect `golden/<id>/v0.39.2/outputs/` and `noise_floor.json` (r1 vs r2), and check `prompt_exec_s` against the timeouts. 4. Re-run with `bless_only=true`. 5. Follow-up PR flips `"enabled": true` (and tightens `timeout_s` if the measured times allow). ## Risks - **Sync time:** the first sync is large. It may need a second dispatch, which is safe to re-run. - **CUDA dependence:** Qwen 2.1 and Krea 2 goldens depend on the worker's CUDA. Moving the image to cu130 switches int8/fp8 to comfy-kitchen's CUDA kernels and needs a re-bless. - **Timeouts:** the Qwen edit timeout (2400 s) is an estimate that allows for the slower fallback path and a cold load from the network volume. - **Merge conflict:** `scripts-tests.yml` is also added by #2. Whichever merges second gets a small conflict; keep this version, which only adds triggers. - **Template drift:** the trimmed workflows differ from what the templates do by default (no LLM refinement in Krea 2), by design. They test the image models, not the full template experience. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 22:06:10 +00:00