[PR #11] [MERGED] Lanes as data: per-image Dockerfile, build matrix, per-lane regression jobs #12

New issue

Closed

opened 2026-10-31 22:06:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 22:06:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyci-runpod-worker/pull/11
Author: @marawan206
Created: 10/19/2026
Status: ✅ Merged
Merged: 10/19/2026
Merged by: @marawan206

Base: main ← Head: lanes-cu130

📝 Commits (7)

9166541 Lanes as data: per-image Dockerfile, build matrix, per-lane regression jobs
f3ae31f Keep the lane matrix script dependency-free for the plan jobs
530292d Create a lane's endpoint from endpoint_defaults in lanes.json
4bcaeba Provision the py313-torch2.14.1-cu130 lane: endpoint 2192t0ar23kdb5
d360061 Test lane plumbing against an unprovisioned copy of the registry
5226e3c Report worker failures under failure and carry the server log tail
aa08561 Give the worker image a C compiler for torch 2.14's Triton kernels

📊 Changes

16 files changed (+1035 additions, -99 deletions)

View changed files

📝 .github/workflows/build-image.yml (+73 -9)
📝 .github/workflows/regression.yml (+45 -9)
📝 Dockerfile (+45 -13)
📝 README.md (+56 -11)
📝 manifest/lanes.json (+63 -2)
📝 scripts/golden_baseline.py (+9 -3)
📝 scripts/index_builder.py (+21 -0)
➕ scripts/lanes.py (+151 -0)
📝 scripts/run_regression.py (+15 -3)
➕ scripts/runpod_endpoints.py (+220 -0)
📝 scripts/submit_and_poll.py (+10 -5)
➕ scripts/tests/test_lanes.py (+172 -0)
📝 scripts/tests/test_submit_and_poll.py (+16 -0)
📝 src/checkout.py (+14 -2)
📝 src/comfy_runner.py (+88 -33)
📝 src/handler.py (+37 -9)

📄 Description

Why

The matrix we agreed on in October (per-commit py3.13 / torch 2.14.1 / cu130, with the cu128 lane retiring after an overlap) has had its registry, per-lane storage prefixes, --lane flags and dashboard selector for two weeks, but only the legacy lane existed because the image, endpoint and per-lane scheduling were still hand-built. This PR makes lanes data end to end and brings up the first new lane.

What changes

manifest/lanes.json (schema 2) gains an images block (python, cuda, torch index and exact torch/torchvision pins per image), endpoint_defaults (pool, workers, volume, data center, registry credential, timeouts), and each lane names its image and endpoint_id. The new lane py313-torch2.14.1-cu130 is provisioned: endpoint 2192t0ar23kdb5. The legacy image is marked "build": false so CI never rebuilds or moves the lane that today's goldens were blessed on.

Dockerfile is parameterised by those image fields. Python comes from uv (Ubuntu 24.04 has no 3.13), torch is installed from the lane's pins and the same pins are written to /comfy-config/constraints.txt, which checkout.py passes to the runtime requirements install so a ComfyUI commit can never swap the torch build underneath a lane. The image writes /comfy-config/image.json, and every run record carries an env block (image id, build sha, python, torch, cuda, host driver, and the backend lines ComfyUI logged). No torchaudio on new images: ComfyUI dropped it from requirements in September and the cu130 index only has 2.9.1.

build-image.yml builds a matrix from the registry (scripts/lanes.py images), tags :<image> and :<image>-<sha>, caches per image in the registry, and then runs scripts/runpod_endpoints.py set-image so every provisioned, non-legacy lane on that image is repointed to the sha tag. That retires the manual PATCH after every build.

regression.yml fans out over scripts/lanes.py per-commit: one job per provisioned per-commit lane, each with its own last-tested check under its own prefix, running run_regression.py --lane.

Scripts: run_regression.py and golden_baseline.py take the endpoint from the lane (a lane with endpoint_id: null refuses to run rather than falling back to another lane's endpoint); the lane id travels in the job input and the worker echoes it. scripts/runpod_endpoints.py (status, set-image, create, --dry-run) talks to the control-plane REST v2 API. scripts/lanes.py validate is part of the test suite.

Two worker fixes the lane surfaced

Failure detail was vanishing. RunPod SDK 1.12 treats a top-level error in a handler's output as the job failing, rejects a dict there (the job-done call returns 400) and stores the output without it, so every execution error came back as execution_error with no detail. The worker now reports under failure and attaches the last 6 KB of the ComfyUI server log to every failure; the client reads failure, then the old error, then the job-level error, so the legacy image keeps working. The SDK is pinned.
torch 2.14 needs a C compiler. torch._native routes bmm_outer_product (hit by the Qwen3 and Gemma text encoders) to a Triton kernel, and Triton compiles its CUDA driver shim with the system compiler on first use. The slim image had none, so seven of nine workflows failed at CLIPTextEncode while SDXL and FLUX passed. gcc and libc6-dev are in the image now.

Verified

python -m pytest scripts/tests -q: 217 passed (12 new in test_lanes.py, 1 in test_submit_and_poll.py).
Image built three times from this branch by build-image.yml dispatches; the second and third runs repointed endpoint 2192t0ar23kdb5 automatically (visible in runpod_endpoints.py status).
Endpoint created with runpod_endpoints.py create --write from endpoint_defaults; workers come up on RTX 4090s in EU-RO-1 with the 200 GB volume mounted.
PixelDiT probe on the final image: 29 s, validation clean, env.image_sha equal to the branch head.
Goldens for all nine workflows at v0.42.0 on the lane, generated and blessed (runs linked in the first comment) (both SDXL and FLUX are byte-different from their cu128 goldens, as expected for a different torch and CUDA).

After merge

The push to main rebuilds the image and repoints the lane; the cron picks the lane up on its next tick and ci.comfy.org lists it after the first publish. Legacy stays primary until the new lane has a week of history, then primary flips in a one-line change.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyci-runpod-worker/pull/11 **Author:** [@marawan206](https://github.com/marawan206) **Created:** 10/19/2026 **Status:** ✅ Merged **Merged:** 10/19/2026 **Merged by:** [@marawan206](https://github.com/marawan206) **Base:** `main` ← **Head:** `lanes-cu130` --- ### 📝 Commits (7) - [`9166541`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/91665413677650b714fc6d77ecd83464e6ef1289) Lanes as data: per-image Dockerfile, build matrix, per-lane regression jobs - [`f3ae31f`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/f3ae31f5026bd1e8373a7c3d13060d08c079b065) Keep the lane matrix script dependency-free for the plan jobs - [`530292d`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/530292db9e73c39c378aa9c625cc03f840cb82c0) Create a lane's endpoint from endpoint_defaults in lanes.json - [`4bcaeba`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/4bcaebad0e169e2cbe839aa3452cdc3364071a62) Provision the py313-torch2.14.1-cu130 lane: endpoint 2192t0ar23kdb5 - [`d360061`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/d36006133c3b205a1e72eab735e5cde408981303) Test lane plumbing against an unprovisioned copy of the registry - [`5226e3c`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/5226e3c7004fd7febd06347683e5d1967d709030) Report worker failures under `failure` and carry the server log tail - [`aa08561`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/aa085619ece5e27b084f2c7f2435bc69503dc485) Give the worker image a C compiler for torch 2.14's Triton kernels ### 📊 Changes **16 files changed** (+1035 additions, -99 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/build-image.yml` (+73 -9) 📝 `.github/workflows/regression.yml` (+45 -9) 📝 `Dockerfile` (+45 -13) 📝 `README.md` (+56 -11) 📝 `manifest/lanes.json` (+63 -2) 📝 `scripts/golden_baseline.py` (+9 -3) 📝 `scripts/index_builder.py` (+21 -0) ➕ `scripts/lanes.py` (+151 -0) 📝 `scripts/run_regression.py` (+15 -3) ➕ `scripts/runpod_endpoints.py` (+220 -0) 📝 `scripts/submit_and_poll.py` (+10 -5) ➕ `scripts/tests/test_lanes.py` (+172 -0) 📝 `scripts/tests/test_submit_and_poll.py` (+16 -0) 📝 `src/checkout.py` (+14 -2) 📝 `src/comfy_runner.py` (+88 -33) 📝 `src/handler.py` (+37 -9) </details> ### 📄 Description ## Why The matrix we agreed on in October (per-commit py3.13 / torch 2.14.1 / cu130, with the cu128 lane retiring after an overlap) has had its registry, per-lane storage prefixes, `--lane` flags and dashboard selector for two weeks, but only the legacy lane existed because the image, endpoint and per-lane scheduling were still hand-built. This PR makes lanes data end to end and brings up the first new lane. ## What changes **`manifest/lanes.json` (schema 2)** gains an `images` block (python, cuda, torch index and exact torch/torchvision pins per image), `endpoint_defaults` (pool, workers, volume, data center, registry credential, timeouts), and each lane names its `image` and `endpoint_id`. The new lane `py313-torch2.14.1-cu130` is provisioned: endpoint `2192t0ar23kdb5`. The legacy image is marked `"build": false` so CI never rebuilds or moves the lane that today's goldens were blessed on. **`Dockerfile`** is parameterised by those image fields. Python comes from uv (Ubuntu 24.04 has no 3.13), torch is installed from the lane's pins and the same pins are written to `/comfy-config/constraints.txt`, which `checkout.py` passes to the runtime requirements install so a ComfyUI commit can never swap the torch build underneath a lane. The image writes `/comfy-config/image.json`, and every run record carries an `env` block (image id, build sha, python, torch, cuda, host driver, and the backend lines ComfyUI logged). No torchaudio on new images: ComfyUI dropped it from requirements in September and the cu130 index only has 2.9.1. **`build-image.yml`** builds a matrix from the registry (`scripts/lanes.py images`), tags `:<image>` and `:<image>-<sha>`, caches per image in the registry, and then runs `scripts/runpod_endpoints.py set-image` so every provisioned, non-legacy lane on that image is repointed to the sha tag. That retires the manual PATCH after every build. **`regression.yml`** fans out over `scripts/lanes.py per-commit`: one job per provisioned per-commit lane, each with its own last-tested check under its own prefix, running `run_regression.py --lane`. **Scripts**: `run_regression.py` and `golden_baseline.py` take the endpoint from the lane (a lane with `endpoint_id: null` refuses to run rather than falling back to another lane's endpoint); the lane id travels in the job input and the worker echoes it. `scripts/runpod_endpoints.py` (`status`, `set-image`, `create`, `--dry-run`) talks to the control-plane REST v2 API. `scripts/lanes.py validate` is part of the test suite. ## Two worker fixes the lane surfaced - **Failure detail was vanishing.** RunPod SDK 1.12 treats a top-level `error` in a handler's output as the job failing, rejects a dict there (the job-done call returns 400) and stores the output without it, so every execution error came back as `execution_error` with no detail. The worker now reports under `failure` and attaches the last 6 KB of the ComfyUI server log to every failure; the client reads `failure`, then the old `error`, then the job-level error, so the legacy image keeps working. The SDK is pinned. - **torch 2.14 needs a C compiler.** `torch._native` routes `bmm_outer_product` (hit by the Qwen3 and Gemma text encoders) to a Triton kernel, and Triton compiles its CUDA driver shim with the system compiler on first use. The slim image had none, so seven of nine workflows failed at `CLIPTextEncode` while SDXL and FLUX passed. `gcc` and `libc6-dev` are in the image now. ## Verified - `python -m pytest scripts/tests -q`: 217 passed (12 new in `test_lanes.py`, 1 in `test_submit_and_poll.py`). - Image built three times from this branch by `build-image.yml` dispatches; the second and third runs repointed endpoint `2192t0ar23kdb5` automatically (visible in `runpod_endpoints.py status`). - Endpoint created with `runpod_endpoints.py create --write` from `endpoint_defaults`; workers come up on RTX 4090s in EU-RO-1 with the 200 GB volume mounted. - PixelDiT probe on the final image: 29 s, validation clean, `env.image_sha` equal to the branch head. - Goldens for all nine workflows at v0.42.0 on the lane, generated and blessed (runs linked in the first comment) (both SDXL and FLUX are byte-different from their cu128 goldens, as expected for a different torch and CUDA). ## After merge The push to main rebuilds the image and repoints the lane; the cron picks the lane up on its next tick and ci.comfy.org lists it after the first publish. Legacy stays primary until the new lane has a week of history, then `primary` flips in a one-line change. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 22:06:14 +00:00