[PR #12] [MERGED] Publish a frontend CI index for the dashboard's Frontend tab #13

New issue

Closed

opened 2026-10-31 22:06:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 22:06:14 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfyci-runpod-worker/pull/12
Author: @marawan206
Created: 10/19/2026
Status: ✅ Merged
Merged: 10/19/2026
Merged by: @marawan206

Base: main ← Head: frontend-index

📝 Commits (2)

bdaace3 Publish a frontend CI index for the dashboard's Frontend tab
25a2602 frontend-index: split a backfill window that holds over a thousand runs

📊 Changes

14 files changed (+5139 additions, -11 deletions)

View changed files

➕ .github/workflows/frontend-index.yml (+43 -0)
📝 README.md (+88 -0)
📝 scripts/commit_meta.py (+8 -6)
➕ scripts/frontend_index.py (+1208 -0)
📝 scripts/storage.py (+8 -5)
➕ scripts/tests/fixtures/frontend/README.md (+33 -0)
➕ scripts/tests/fixtures/frontend/entries.json (+1 -0)
➕ scripts/tests/fixtures/frontend/expected_latest_cmp.json (+2576 -0)
➕ scripts/tests/fixtures/frontend/perf-20261019T103805Z-17b0d4e4.json (+1 -0)
➕ scripts/tests/fixtures/frontend/perf-20261019T105541Z-e38b73dd.json (+1 -0)
➕ scripts/tests/fixtures/frontend/perf-20261019T130522Z-dd678395.json (+1 -0)
📝 scripts/tests/test_commit_meta.py (+13 -0)
➕ scripts/tests/test_frontend_index.py (+1142 -0)
📝 scripts/tests/test_storage.py (+16 -0)

📄 Description

What

A cron publisher that gives ci.comfy.org a Frontend tab without any token or server code: scripts/frontend_index.py (run by frontend-index.yml every 30 minutes, cron 13,43 * * * *) reads three public sources of Comfy-Org/ComfyUI_frontend with the Actions GITHUB_TOKEN and publishes compact static JSON under frontend/ on the results branch, next to regression/:

frontend/perf/head.json           metric definitions, rule, newest 100 baselines, latest comparison
frontend/perf/<YYYY-MM>.json      complete monthly shards
frontend/ci/main.json             column registry, newest 300 main commits, first_bad, coverage
frontend/ci/main/<YYYY-MM>.json   complete monthly shards
frontend/ci/custom-nodes.json     newest 90 custom-nodes gate runs with shard rows, core pin, streak

Perf: every baseline on the perf-data branch (the frontend keeps only the last 20) is stored as per-scenario means for the 12 metrics its PR report reports, plus the classification the frontend's own rule gives it (scripts/perf-stats.ts: current median against the mean of the 20 preceding baselines, sample stddev, z > 2 regression / z < -2 improvement, noisy when CV > 50 %, 5-unit gate on the four count metrics). The head carries the metric list and thresholds, so the dashboard never hardcodes them.
CI on main: one row per main commit with the conclusion, start, duration and attempt of every CI: workflow that ran for it, keyed by workflow path (lint on main is ci-lint-format-queue.yaml). Fifteen known columns in groups core / website / billing / validation; any other CI: workflow seen on a main push is appended as group other. Commit metadata comes from commit_meta (now with a base parameter: the frontend merges into main, and merged_pr was hardcoded to master).
Custom-nodes gate: per run, the conclusion of every shard x proof_row job, the failed step, the verdict job, and COMFYUI_PIN read from the workflow file at the run's head sha. Proof rows 1/2/3/9/15 are detection proofs (planted break + independence assertion), so the streak is computed over the clean shards (row 0) of pinned runs only; dispatch runs test an unknown ref until phase 2 and neither start nor break it.

Behaviour

Fetch phase, then three derived writers (register_derived), so a push retry re-merges on the fresh tree and a concurrent regression publish is never clobbered. Every file is compared ignoring updated_ts: a tick with nothing new pushes nothing (exactly 3 API calls).
Incremental cursor = min(now - 6 h, oldest unsettled commit, last complete listing), so a gap between ticks never leaves a hole; workflow_dispatch with backfill_days walks day windows back from the covered range. Reruns override by attempt. Commit-meta upgrades are capped at 25 per tick and drawn from every stored entry.
Rate budget: 5-9 calls per tick in steady state; a 90-day backfill is about 500 calls, the rest spreads over the following ticks. A budget stop (403/429 or the 150-call reserve) publishes what was collected and warns; exit code stays 0.
storage.upload_json(..., compact=True) writes the files without indentation (perf entries would otherwise double in size).

Verification

python -m pytest scripts/tests -q: 253 passed (217 existing, 34 new for the publisher, 1 each for commit_meta and storage).
The perf statistics were cross-checked against the frontend's own perf-stats.ts on the 20 real baselines: the publisher reproduces every one of the 312 scenario x metric verdicts for the newest baseline (scripts/tests/fixtures/frontend/expected_latest_cmp.json was produced by running the frontend's functions; the fixtures README says how).
Live dry run (--skip-publish): ci/main.json latest commit equals main's HEAD with all 15 known columns populated (plus four other ones), perf/head.json has 20 baselines with the latest comparison, ci/custom-nodes.json has 90 runs, pin b08e6cf3, and a fail streak since 5 Sep 2026.

Notes

The scheduled custom-nodes gate has failed on every run for over 40 days (clean shards fail in the suite step, four of five proof rows fail their independence assertion); PR runs on main-derived branches pass. The tab will show this as a streak; worth raising with the frontend team.
Stored means keep 8 significant digits (not 2 dp): with 2 dp the history stddev of frame times collapses to rounding noise and every idle scenario is flagged as an improvement.
Phase 2 (needs a fine-grained PAT with Actions read on the frontend repo, FRONTEND_GH_TOKEN): read run logs for a dispatch run's comfyui_ref, Playwright result artifacts, and dispatching the gate against a core release candidate. The script already prefers that token when set.
A new workflow file cannot be dispatched from a PR branch, so the first backfill was run from this branch locally and pushed: frontend/ is on the results branch now (2,591 main commits back to 21 Jul (the full 90 days), 20 baselines, 90 gate runs), and the dashboard PR's preview already renders it. After merge the cron keeps it fresh; a backfill_days=N dispatch extends the CI coverage further back if ever wanted (each run continues from where the previous one stopped).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfyci-runpod-worker/pull/12 **Author:** [@marawan206](https://github.com/marawan206) **Created:** 10/19/2026 **Status:** ✅ Merged **Merged:** 10/19/2026 **Merged by:** [@marawan206](https://github.com/marawan206) **Base:** `main` ← **Head:** `frontend-index` --- ### 📝 Commits (2) - [`bdaace3`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/bdaace3873d3122090f5cf060f5997a2e5cc04f2) Publish a frontend CI index for the dashboard's Frontend tab - [`25a2602`](https://github.com/Comfy-Org/comfyci-runpod-worker/commit/25a2602a28218cc74de04a5890a15867140da89b) frontend-index: split a backfill window that holds over a thousand runs ### 📊 Changes **14 files changed** (+5139 additions, -11 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/frontend-index.yml` (+43 -0) 📝 `README.md` (+88 -0) 📝 `scripts/commit_meta.py` (+8 -6) ➕ `scripts/frontend_index.py` (+1208 -0) 📝 `scripts/storage.py` (+8 -5) ➕ `scripts/tests/fixtures/frontend/README.md` (+33 -0) ➕ `scripts/tests/fixtures/frontend/entries.json` (+1 -0) ➕ `scripts/tests/fixtures/frontend/expected_latest_cmp.json` (+2576 -0) ➕ `scripts/tests/fixtures/frontend/perf-20261019T103805Z-17b0d4e4.json` (+1 -0) ➕ `scripts/tests/fixtures/frontend/perf-20261019T105541Z-e38b73dd.json` (+1 -0) ➕ `scripts/tests/fixtures/frontend/perf-20261019T130522Z-dd678395.json` (+1 -0) 📝 `scripts/tests/test_commit_meta.py` (+13 -0) ➕ `scripts/tests/test_frontend_index.py` (+1142 -0) 📝 `scripts/tests/test_storage.py` (+16 -0) </details> ### 📄 Description ## What A cron publisher that gives ci.comfy.org a **Frontend** tab without any token or server code: `scripts/frontend_index.py` (run by `frontend-index.yml` every 30 minutes, cron `13,43 * * * *`) reads three public sources of `Comfy-Org/ComfyUI_frontend` with the Actions `GITHUB_TOKEN` and publishes compact static JSON under `frontend/` on the `results` branch, next to `regression/`: ``` frontend/perf/head.json metric definitions, rule, newest 100 baselines, latest comparison frontend/perf/<YYYY-MM>.json complete monthly shards frontend/ci/main.json column registry, newest 300 main commits, first_bad, coverage frontend/ci/main/<YYYY-MM>.json complete monthly shards frontend/ci/custom-nodes.json newest 90 custom-nodes gate runs with shard rows, core pin, streak ``` - **Perf**: every baseline on the `perf-data` branch (the frontend keeps only the last 20) is stored as per-scenario means for the 12 metrics its PR report reports, plus the classification the frontend's own rule gives it (`scripts/perf-stats.ts`: current median against the mean of the 20 preceding baselines, sample stddev, z > 2 regression / z < -2 improvement, noisy when CV > 50 %, 5-unit gate on the four count metrics). The head carries the metric list and thresholds, so the dashboard never hardcodes them. - **CI on main**: one row per main commit with the conclusion, start, duration and attempt of every `CI:` workflow that ran for it, keyed by workflow **path** (lint on main is `ci-lint-format-queue.yaml`). Fifteen known columns in groups core / website / billing / validation; any other `CI:` workflow seen on a main push is appended as group `other`. Commit metadata comes from `commit_meta` (now with a `base` parameter: the frontend merges into `main`, and `merged_pr` was hardcoded to `master`). - **Custom-nodes gate**: per run, the conclusion of every `shard x proof_row` job, the failed step, the verdict job, and `COMFYUI_PIN` read from the workflow file at the run's head sha. Proof rows 1/2/3/9/15 are detection proofs (planted break + independence assertion), so the streak is computed over the clean shards (row 0) of pinned runs only; dispatch runs test an unknown ref until phase 2 and neither start nor break it. ## Behaviour - Fetch phase, then three derived writers (`register_derived`), so a push retry re-merges on the fresh tree and a concurrent regression publish is never clobbered. Every file is compared ignoring `updated_ts`: a tick with nothing new pushes nothing (exactly 3 API calls). - Incremental cursor = min(now - 6 h, oldest unsettled commit, last complete listing), so a gap between ticks never leaves a hole; `workflow_dispatch` with `backfill_days` walks day windows back from the covered range. Reruns override by attempt. Commit-meta upgrades are capped at 25 per tick and drawn from every stored entry. - Rate budget: 5-9 calls per tick in steady state; a 90-day backfill is about 500 calls, the rest spreads over the following ticks. A budget stop (403/429 or the 150-call reserve) publishes what was collected and warns; exit code stays 0. - `storage.upload_json(..., compact=True)` writes the files without indentation (perf entries would otherwise double in size). ## Verification - `python -m pytest scripts/tests -q`: 253 passed (217 existing, 34 new for the publisher, 1 each for `commit_meta` and `storage`). - The perf statistics were cross-checked against the frontend's own `perf-stats.ts` on the 20 real baselines: the publisher reproduces every one of the 312 scenario x metric verdicts for the newest baseline (`scripts/tests/fixtures/frontend/expected_latest_cmp.json` was produced by running the frontend's functions; the fixtures README says how). - Live dry run (`--skip-publish`): `ci/main.json` latest commit equals `main`'s HEAD with all 15 known columns populated (plus four `other` ones), `perf/head.json` has 20 baselines with the latest comparison, `ci/custom-nodes.json` has 90 runs, pin `b08e6cf3`, and a fail streak since 5 Sep 2026. ## Notes - The scheduled custom-nodes gate has failed on every run for over 40 days (clean shards fail in the suite step, four of five proof rows fail their independence assertion); PR runs on main-derived branches pass. The tab will show this as a streak; worth raising with the frontend team. - Stored means keep 8 significant digits (not 2 dp): with 2 dp the history stddev of frame times collapses to rounding noise and every idle scenario is flagged as an improvement. - Phase 2 (needs a fine-grained PAT with Actions read on the frontend repo, `FRONTEND_GH_TOKEN`): read run logs for a dispatch run's `comfyui_ref`, Playwright result artifacts, and dispatching the gate against a core release candidate. The script already prefers that token when set. - A new workflow file cannot be dispatched from a PR branch, so the first backfill was run from this branch locally and pushed: `frontend/` is on the results branch now (2,591 main commits back to 21 Jul (the full 90 days), 20 baselines, 90 gate runs), and the dashboard PR's preview already renders it. After merge the cron keeps it fresh; a `backfill_days=N` dispatch extends the CI coverage further back if ever wanted (each run continues from where the previous one stopped). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 22:06:14 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfyci-runpod-worker#13

No description provided.

Rows
Columns