[PR #5799] [MERGED] perf(ci): pack each test tier from its own timing history #7070

Closed
opened 2026-11-01 21:45:09 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5799
Author: @morluto
Created: 10/21/2026
Status: ✅ Merged
Merged: 10/21/2026
Merged by: @morluto

Base: main ← Head: cursor/ci-fast-timing-history-f42a


📝 Commits (3)

  • f13d94c perf(ci): pack each test tier from its own timing history
  • df96a74 perf(ci): drop the shared timing-history fallback
  • 46570bc test(ci): look up fast history when only full timings exist

📊 Changes

5 files changed (+298 additions, -89 deletions)

View changed files

📝 .github/workflows/ci.yml (+18 -4)
📝 docs/testing.md (+25 -8)
📝 scripts/ci/test-history.mjs (+89 -62)
📝 tests/boundary/filesystem/ciTestHistory.test.ts (+149 -13)
📝 tests/boundary/filesystem/ciTestReports.test.ts (+17 -2)

📄 Description

Summary

Fast built shards are still packed with the latest full-suite timing artifact. That artifact stayed stale across the compiled-CLI session reuse, so some shards finish several minutes before the slowest one. Each tier now publishes and downloads its own history, and the old shared test-timings artifact is no longer read.

Human owner: @morluto.

Problem and expected behavior

On 2026-10-21, successful fast shard jobs still executed for about 5–10 minutes after runner queue. Run 39552964873 (eight built shards, after #5676, #5679, and #5784) ran its shard jobs for 319–591 seconds. The weights came from full history run 39545047294, recorded before the later session-reuse commits.

Replaying that fast run's file durations through allocateByDuration / scheduleWall (two workers, process-boundary after the main group):

  • Stale full-suite weights: 293–547 seconds of file time.
  • Weights from the same fast run: 404 seconds on every shard.

Hosted shard execution of 319–591 seconds tracks that stale spread plus setup. Fresh fast weights should cut the slowest shard's file schedule by about 140 seconds. The first green default-branch run of a tier has no tier artifact yet, so it uses Vitest's own assignment and then publishes the new one. The following run packs with that artifact.

Runner queue is separate and not fixed here. Across 108 successful shard jobs in 14 completed runs, queue was p50 198s, p95 435s, and max 988s (39542179402, shard 3/8). Cancelled main pushes in the same window had no jobs, so they were pending concurrency slots, not runners held by cancel-in-progress. Adding shards would ask for more runners while that queue is already the tail.

Change and scope

Default-branch push, schedule, and manual runs publish test-timings-fast or test-timings-full after a complete successful merge. Pull requests do not publish: the lookup reads one unpaginated artifact page. The build job downloads only the artifact for its tier. A tier with no qualifying artifact leaves history unset. There is no fallback to the previous test-timings artifact or to the other tier.

Not included: shard-count changes, a higher Vitest worker cap, or cancelling in-progress main runs. Eight fast shards stay inside the existing ten-minute job budget, and docs/testing.md already records that four workers did not remove the long-file floor.

No separate issue is linked. This continues the CI wall-time work in #5676, #5679, and #5784.

Contribution scope: not applicable. This is CI history selection inside the existing test workflow.

Contract and boundary impact

  • Semantic owner and earliest changed stage: scripts/ci/test-history.mjs owns which timing artifact the build job downloads. Shard execution is unchanged.
  • Contribution scope fit and linked maintainer decision: not applicable.
  • CLI and MCP/tool-catalog contract: none.
  • Provider, bridge, target-format, or platform compatibility: none.
  • Evidence, artifact, provenance, or reconstruction contract: none. The new artifacts are CI scheduling weights, not target Evidence.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: tests/boundary/filesystem/ciTestHistory.test.ts covers full-tier selection, fast history that does not read the full artifact, pull-request history that is ignored instead of borrowed, a fast lookup that returns empty when only full-tier history exists, a missing tier that does not query, and failed fast candidates. ciTestReports.test.ts covers the publish condition and the tier-specific download name.
  • Base reproduction or other evidence: GitHub Actions job timestamps for the runs in the summary, plus a local replay of the shipped sequencer on the test-results artifact from run 39552964873 and history artifact 12322235443 from run 39545047294. The 404s / 293–547s figures are that replay, not a hosted before/after.
  • User-visible CLI/MCP output (if applicable): not applicable. No CLI or MCP behavior changes.
  • Remaining proof gaps: this pull request's own CI has no test-timings-fast artifact yet, so its shards use Vitest's default assignment. Hosted wall-time after the first green default-branch publish is not measured yet.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:focused -- tests/boundary/filesystem/ciTestHistory.test.ts tests/boundary/filesystem/ciTestReports.test.ts — 19 tests passed.
  • npm run check:fast — typecheck and lint passed.
  • node --experimental-strip-types replay of allocateByDuration / scheduleWall — stale weights 292.6–546.8s, fresh weights 404.4s on all eight shards.

Compatibility, safety, and release

  • Breaking changes or migration steps: none for callers. Existing test-timings artifacts are no longer inputs. The first green default-branch run of each tier publishes the new artifact; until then, that tier has no duration history.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: history is still accepted only from a successful CI workflow on the default branch of this repository. Pull requests and other branches are not producers.

Author readiness

Human owner: @morluto

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.
Open in Web Open in Cursor 

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5799 **Author:** [@morluto](https://github.com/morluto) **Created:** 10/21/2026 **Status:** ✅ Merged **Merged:** 10/21/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `cursor/ci-fast-timing-history-f42a` --- ### 📝 Commits (3) - [`f13d94c`](https://github.com/morluto/rea/commit/f13d94c0a00668272a93b895da3d0ee702c50a4c) perf(ci): pack each test tier from its own timing history - [`df96a74`](https://github.com/morluto/rea/commit/df96a747efe80c0f0323909a5003378c90385859) perf(ci): drop the shared timing-history fallback - [`46570bc`](https://github.com/morluto/rea/commit/46570bc4b03b7039ed356247e9c42c0d554bbc04) test(ci): look up fast history when only full timings exist ### 📊 Changes **5 files changed** (+298 additions, -89 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+18 -4) 📝 `docs/testing.md` (+25 -8) 📝 `scripts/ci/test-history.mjs` (+89 -62) 📝 `tests/boundary/filesystem/ciTestHistory.test.ts` (+149 -13) 📝 `tests/boundary/filesystem/ciTestReports.test.ts` (+17 -2) </details> ### 📄 Description <!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Summary Fast built shards are still packed with the latest full-suite timing artifact. That artifact stayed stale across the compiled-CLI session reuse, so some shards finish several minutes before the slowest one. Each tier now publishes and downloads its own history, and the old shared `test-timings` artifact is no longer read. Human owner: @morluto. ## Problem and expected behavior On 2026-10-21, successful fast shard jobs still executed for about 5–10 minutes after runner queue. Run `39552964873` (eight built shards, after #5676, #5679, and #5784) ran its shard jobs for 319–591 seconds. The weights came from full history run `39545047294`, recorded before the later session-reuse commits. Replaying that fast run's file durations through `allocateByDuration` / `scheduleWall` (two workers, `process-boundary` after the main group): - Stale full-suite weights: 293–547 seconds of file time. - Weights from the same fast run: 404 seconds on every shard. Hosted shard execution of 319–591 seconds tracks that stale spread plus setup. Fresh fast weights should cut the slowest shard's file schedule by about 140 seconds. The first green default-branch run of a tier has no tier artifact yet, so it uses Vitest's own assignment and then publishes the new one. The following run packs with that artifact. Runner queue is separate and not fixed here. Across 108 successful shard jobs in 14 completed runs, queue was p50 198s, p95 435s, and max 988s (`39542179402`, shard 3/8). Cancelled main pushes in the same window had no jobs, so they were pending concurrency slots, not runners held by `cancel-in-progress`. Adding shards would ask for more runners while that queue is already the tail. ## Change and scope Default-branch push, schedule, and manual runs publish `test-timings-fast` or `test-timings-full` after a complete successful merge. Pull requests do not publish: the lookup reads one unpaginated artifact page. The build job downloads only the artifact for its tier. A tier with no qualifying artifact leaves history unset. There is no fallback to the previous `test-timings` artifact or to the other tier. Not included: shard-count changes, a higher Vitest worker cap, or cancelling in-progress main runs. Eight fast shards stay inside the existing ten-minute job budget, and `docs/testing.md` already records that four workers did not remove the long-file floor. No separate issue is linked. This continues the CI wall-time work in #5676, #5679, and #5784. Contribution scope: not applicable. This is CI history selection inside the existing test workflow. ## Contract and boundary impact - Semantic owner and earliest changed stage: `scripts/ci/test-history.mjs` owns which timing artifact the build job downloads. Shard execution is unchanged. - Contribution scope fit and linked maintainer decision: not applicable. - CLI and MCP/tool-catalog contract: none. - Provider, bridge, target-format, or platform compatibility: none. - Evidence, artifact, provenance, or reconstruction contract: none. The new artifacts are CI scheduling weights, not target Evidence. - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs): none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [ ] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [x] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: `tests/boundary/filesystem/ciTestHistory.test.ts` covers full-tier selection, fast history that does not read the full artifact, pull-request history that is ignored instead of borrowed, a fast lookup that returns empty when only full-tier history exists, a missing tier that does not query, and failed fast candidates. `ciTestReports.test.ts` covers the publish condition and the tier-specific download name. - Base reproduction or other evidence: GitHub Actions job timestamps for the runs in the summary, plus a local replay of the shipped sequencer on the `test-results` artifact from run `39552964873` and history artifact `12322235443` from run `39545047294`. The 404s / 293–547s figures are that replay, not a hosted before/after. - User-visible CLI/MCP output (if applicable): not applicable. No CLI or MCP behavior changes. - Remaining proof gaps: this pull request's own CI has no `test-timings-fast` artifact yet, so its shards use Vitest's default assignment. Hosted wall-time after the first green default-branch publish is not measured yet. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- tests/boundary/filesystem/ciTestHistory.test.ts tests/boundary/filesystem/ciTestReports.test.ts` — 19 tests passed. - `npm run check:fast` — typecheck and lint passed. - `node --experimental-strip-types` replay of `allocateByDuration` / `scheduleWall` — stale weights 292.6–546.8s, fresh weights 404.4s on all eight shards. ## Compatibility, safety, and release - Breaking changes or migration steps: none for callers. Existing `test-timings` artifacts are no longer inputs. The first green default-branch run of each tier publishes the new artifact; until then, that tier has no duration history. - Real Hopper/Ghidra, browser, or OS coverage: not applicable. - Package or release metadata impact: none. - Security, privacy, process, or containment review: history is still accepted only from a successful CI workflow on the default branch of this repository. Pull requests and other branches are not producers. ## Author readiness Human owner: @morluto - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-d3199c39-96f6-5d2c-8b1a-4bc227d7f42a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-d3199c39-96f6-5d2c-8b1a-4bc227d7f42a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-11-01 21:45:09 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#7070
No description provided.