[PR #3] [MERGED] Add a run index so the dashboard can show regression history #2

New issue

Closed

opened 2026-10-31 22:06:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 22:06:10 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyci-runpod-worker/pull/3
Author: @marawan206
Created: 10/5/2026
Status: ✅ Merged
Merged: 10/8/2026
Merged by: @alexisrolland

Base: storage-hygiene ← Head: run-index

📝 Commits (3)

82c3033 Add a run index for the dashboard's history view
26c783a Maintain the run index on every publish and write WebP thumbnails
ff02da6 Prefetch summaries and run records in one round trip for the index backfill

📊 Changes

12 files changed (+991 additions, -32 deletions)

View changed files

➕ .github/workflows/build-index.yml (+44 -0)
📝 README.md (+25 -0)
➕ manifest/lanes.json (+18 -0)
➕ scripts/build_index.py (+58 -0)
➕ scripts/commit_meta.py (+87 -0)
➕ scripts/index_builder.py (+357 -0)
📝 scripts/run_regression.py (+59 -0)
📝 scripts/storage.py (+43 -4)
📝 scripts/tests/conftest.py (+42 -0)
➕ scripts/tests/test_index_builder.py (+229 -0)
📝 scripts/tests/test_run_regression.py (+15 -0)
📝 scripts/tests/test_storage.py (+14 -28)

📄 Description

Why

The dashboard could only read latest/<branch>.json and one summary at a time, so the 230 published runs were invisible and the first bad commit of the current FLUX regression had to be found by hand (it is b5cc8830, #16488; every run since is bit-identical).

What

index_builder.py derives compact entries (verdict, drift class, output sha, metrics, timings, commit subject/author/PR, tested range) and writes, under regression/index/: a per-lane head file with the newest 500 runs and the first bad commit of every open failing chain, complete monthly shards, and lanes.json (lane registry from the new manifest/lanes.json, per-branch latest pointer and shard list, current golden tag and output sha per workflow). Entries are ordered by commit time so re-testing an older commit never reorders history; a chain opens on a fail or an execution error and records which; a chain whose latest output is the blessed golden (accepted drift) is not reported; a partial re-run keeps the other workflows' cells; a re-test that moves an entry between months leaves no duplicate.
commit_meta.py looks up commit subject/author/PR/parents and the tested range through the GitHub API with the Actions token (null-tolerant, reports rate-limit misses).
run_regression.py resolves the lane from manifest/lanes.json before any GPU job is submitted, records lane, commit metadata and tested range in summary.json, registers the index update as a derived writer (rebuilt on the freshest tree if the push retries), and writes 256-px WebP previews under outputs/thumbs/.
build_index.py + the Build run index workflow backfill or repair a branch/lane from the committed summaries through a derived writer; GitHubStorage.prefetch fetches the needed blobs in one round trip. The workflow has its own concurrency group, a timeout, and passes inputs through the environment.
README documents the index, the lane file and the thumbnails.

Data contract for the dashboard: index/lanes.json, index/<branch>/<lane>.json (head), index/<branch>/<lane>/<YYYY-MM>.json (shards); entry fields c, t, o, infra, cv, lane, layout, m{s,a,pr,ct}, prev, range{n,url}, w{<wf>{v,d,sha,mse,psnr,pct,pv,exec,vram,g,th}}, s.

Verification

python -m pytest scripts/tests -q (entry shape, drift classification incl. golden changes, first-bad detection incl. crashes and commit-time ordering, merging of partial re-runs, backfill + incremental upsert on a real git remote, refresh after a bless, shard moves without duplicates, two concurrent publishers both landing, prefetch).
Backfill against the live results branch: 230 entries in under five minutes; first_bad.flux_dev_t2i = b5cc8830… (prev good 912fca4f…, PR 16488, 67 runs); exactly one new_drift and 66 inherited; commit metadata for 230/230; tested range for 226 (the rest have no previous run); output sha for 228 (two infra-error runs have no outputs). The resulting index is already published on the results branch so the dashboard history PR can be previewed against real data.
After merge: dispatch Build run index once (idempotent) so the published index carries the final field set.

Stacked on #2; followed by #4.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyci-runpod-worker/pull/3 **Author:** [@marawan206](https://github.com/marawan206) **Created:** 10/5/2026 **Status:** ✅ Merged **Merged:** 10/8/2026 **Merged by:** [@alexisrolland](https://github.com/alexisrolland) **Base:** `storage-hygiene` ← **Head:** `run-index` --- ### 📝 Commits (3) - [`82c3033`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/82c30330926f6fb271f970c19bc50d8e1c9cf570) Add a run index for the dashboard's history view - [`26c783a`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/26c783a63799b809725df6db0cfc56f3b79f96e9) Maintain the run index on every publish and write WebP thumbnails - [`ff02da6`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/ff02da69fc1c3a9b0094baa1cb9ad616a2fa269e) Prefetch summaries and run records in one round trip for the index backfill ### 📊 Changes **12 files changed** (+991 additions, -32 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/build-index.yml` (+44 -0) 📝 `README.md` (+25 -0) ➕ `manifest/lanes.json` (+18 -0) ➕ `scripts/build_index.py` (+58 -0) ➕ `scripts/commit_meta.py` (+87 -0) ➕ `scripts/index_builder.py` (+357 -0) 📝 `scripts/run_regression.py` (+59 -0) 📝 `scripts/storage.py` (+43 -4) 📝 `scripts/tests/conftest.py` (+42 -0) ➕ `scripts/tests/test_index_builder.py` (+229 -0) 📝 `scripts/tests/test_run_regression.py` (+15 -0) 📝 `scripts/tests/test_storage.py` (+14 -28) </details> ### 📄 Description ## Why The dashboard could only read `latest/<branch>.json` and one summary at a time, so the 230 published runs were invisible and the first bad commit of the current FLUX regression had to be found by hand (it is `b5cc8830`, #16488; every run since is bit-identical). ## What - `index_builder.py` derives compact entries (verdict, drift class, output sha, metrics, timings, commit subject/author/PR, tested range) and writes, under `regression/index/`: a per-lane head file with the newest 500 runs and the first bad commit of every open failing chain, complete monthly shards, and `lanes.json` (lane registry from the new `manifest/lanes.json`, per-branch latest pointer and shard list, current golden tag and output sha per workflow). Entries are ordered by commit time so re-testing an older commit never reorders history; a chain opens on a fail or an execution error and records which; a chain whose latest output is the blessed golden (accepted drift) is not reported; a partial re-run keeps the other workflows' cells; a re-test that moves an entry between months leaves no duplicate. - `commit_meta.py` looks up commit subject/author/PR/parents and the tested range through the GitHub API with the Actions token (null-tolerant, reports rate-limit misses). - `run_regression.py` resolves the lane from `manifest/lanes.json` before any GPU job is submitted, records lane, commit metadata and tested range in `summary.json`, registers the index update as a derived writer (rebuilt on the freshest tree if the push retries), and writes 256-px WebP previews under `outputs/thumbs/`. - `build_index.py` + the `Build run index` workflow backfill or repair a branch/lane from the committed summaries through a derived writer; `GitHubStorage.prefetch` fetches the needed blobs in one round trip. The workflow has its own concurrency group, a timeout, and passes inputs through the environment. - README documents the index, the lane file and the thumbnails. Data contract for the dashboard: `index/lanes.json`, `index/<branch>/<lane>.json` (head), `index/<branch>/<lane>/<YYYY-MM>.json` (shards); entry fields `c, t, o, infra, cv, lane, layout, m{s,a,pr,ct}, prev, range{n,url}, w{<wf>{v,d,sha,mse,psnr,pct,pv,exec,vram,g,th}}, s`. ## Verification - `python -m pytest scripts/tests -q` (entry shape, drift classification incl. golden changes, first-bad detection incl. crashes and commit-time ordering, merging of partial re-runs, backfill + incremental upsert on a real git remote, refresh after a bless, shard moves without duplicates, two concurrent publishers both landing, prefetch). - Backfill against the live results branch: 230 entries in under five minutes; `first_bad.flux_dev_t2i` = `b5cc8830…` (prev good `912fca4f…`, PR 16488, 67 runs); exactly one `new_drift` and 66 `inherited`; commit metadata for 230/230; tested range for 226 (the rest have no previous run); output sha for 228 (two infra-error runs have no outputs). The resulting index is already published on the results branch so the dashboard history PR can be previewed against real data. - After merge: dispatch **Build run index** once (idempotent) so the published index carries the final field set. Stacked on #2; followed by #4. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 22:06:10 +00:00