morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #2210] [MERGED] ci: keep validation fresh and preserve failed-shard reports #5085

New issue

Closed

opened 2026-10-31 01:03:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:03:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2210
Author: @N0zoM1z0
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/12/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/2190-ci-test-gaps

📝 Commits (3)

aec10e8 ci: keep validation fresh and preserve failed-shard reports
1f1c7d2 chore: merge main documentation updates
2e28623 test: cover delayed startup before finalization cancellation

📊 Changes

15 files changed (+511 additions, -92 deletions)

View changed files

📝 .github/actions/portable-build-cache/action.yml (+4 -1)
📝 .github/workflows/ci.yml (+28 -27)
📝 CONTRIBUTING.md (+4 -0)
📝 docs/testing.md (+26 -9)
📝 install.sh (+22 -14)
📝 package.json (+3 -2)
➕ scripts/ci/test-report.mjs (+73 -0)
📝 tests/acceptance/process/processFinalization.test.ts (+34 -14)
📝 tests/boundary/filesystem/ciChangeScope.test.ts (+8 -0)
➕ tests/boundary/filesystem/ciTestReports.test.ts (+174 -0)
📝 tests/boundary/filesystem/installScript.test.ts (+6 -1)
➕ tests/boundary/filesystem/testTaskCache.test.ts (+108 -0)
📝 tests/fixtures/localToolsMcp.ts (+13 -3)
📝 tests/fixtures/processFinalization.mjs (+7 -1)
📝 turbo.json (+1 -20)

📄 Description

README-only changes currently bypass onboarding command consistency, cached full local gates can hide changed inputs, and a failed CI shard prevents useful blobs from being merged. This change closes those gaps and adds inexpensive producer coverage while preserving the required result and coverage contracts.

Addresses #2190. The native ownership and six-shard changes from #2131 are already on main; this builds on them.

Changes

Run the existing two-case README consistency guard from docs:check, outside the cached generated-doc task. Run the existing nine LLDB Python producer tests from the static CI job and the optional complete local gate.
Disable result caching for test:run; keep deterministic artifact preparation and static checks cached. This makes repository-input and external-dependency changes observable without hashing the ambient environment.
Merge available shard blobs before enforcing shard success. Preserve JSON/JUnit/coverage diagnostics after a failing merge, label unsuccessful or incomplete runs diagnostic-only, and keep the selected-shard and aggregate coverage gates strict.
Publish a separate compact test-timings artifact containing checkout-relative paths only after complete success. The sequencer consumes its existing report interface; old or missing artifacts fall back to native sharding.
Restore prior portable Turbo entries within an OS/architecture/toolchain/lockfile prefix so unchanged tasks remain reusable after an owner-specific input change. Turbo still validates task hashes; native outputs remain outside this cache.
Route MCP cancellation through the existing selected-tool fixture, wait for server progress before timing target readiness, observe pending rejection immediately, and force the post-cancellation catalog request through the server. Retain a deterministic delayed-startup control and all cancellation/cleanup assertions.
Preserve nonzero exit, no output, and no execution for every truncated installer prefix on Bash 3.2 and 5.2. Ordinary conditionals use [ ]; regex parsing uses a constant quoted conditional so incomplete [[ grammar cannot mask a truncated download with status 0. Input values are expanded as data, with explicit shell-payload regressions.

Validation

Development uses Node 24.18.0 / npm 11.16.0. The branch was refreshed from main 10400fdb5, then merged the testing-guide reorganization at 7ae3c50a9. Current head: 2e2862391.

On the refreshed baseline:

npm ci, npm run check:ci, npm run docs:check, and npm run test:python passed.
npm run verify:test-discovery discovered all 925 files exactly once.
Supported focused command passed 122 cases in five files, including all 32 installer cases and the unchanged every-byte truncation oracle on GNU Bash 5.2.15.
Commit/pre-push hooks and pinned actionlint 1.7.7 passed.
On the merged baseline, npm run verify:package, npm pack --dry-run --json, docs validation, complete static checks, and pinned actionlint passed.
All 23 cases in the three long-workflow control files passed on 1f1c7d2bd with two workers.
After the cancellation-fixture follow-up below, the actual CI coverage/blob command passed all 12 finalization cases on the final source, with the delayed-startup case taking 8.875s. Typecheck, complete static checks, and commit/push hooks passed. Current-head CI and the separate Go binary lane are green.

Controlled reproductions against the initial implementation baseline (315973259):

Maintained new regressions failed against the old production configuration, passed with the fix, and passed again under LC_ALL=C.
The real docs pipeline passed the original README, failed the actual incorrect command mutation twice, then passed after restoration.
The real Turbo regression fixture rejected a changed README and a changed external dependency after an earlier successful run; restoring either control passed. Preparation counters remained exactly one.
A comment-only managed-fixture input change rebuilt its owning task while five of six preparation tasks were reused, including compilation.
Removing the actual producer's stop-at-entry launch flag made the existing Python test fail; restoring it passed all nine checks.
The real installer suite passed all 32 cases on GNU Bash 3.2.57 on Linux. This establishes that Bash parser control, rather than macOS execution.
Two real CI shard commands produced blobs for one passing and one deliberately failing case. The real merge exited nonzero and still produced JSON/JUnit for both cases. The new reporter emitted partial diagnostics and no successful timing history. Coverage floors were left enabled.

Full-suite result and limits

The broad Linux run on 315973259 plus this patch executed 918 files / 9,391 cases without coverage: 9,299 passed, 89 skipped, three failed, plus one cleanup-time unhandled MCP connection error. Two long client/setup flows exceeded their unchanged 30-second deadline; the cancellation case did not observe its SIGTERM marker. The run took 27m12s on a shared host and overlapped investigation work, so it is neither a green full gate nor an isolated performance benchmark.

All 23 cases in those three files passed when rerun on this branch without changing deadlines or assertions. The clean 315973259 baseline also passed those 23 cases with two workers after giving its checkout a separate dependency installation. Its 1,361 compiled runtime files were byte-identical to this patch's initial runtime. The timing failures were not reproduced in those controls; their full-load cause remains unproven. One attempted baseline control that shared dependencies with a concurrent npm ci was invalidated and excluded.

Native Windows/macOS and real LLDB/provider coverage remain for the selected CI lanes; the Python fake-LLDB suite and Linux Bash 3.2 run do not establish those host/provider results. This PR does not reduce coverage floors, enable retries, delete slow workflow families, or promise a whole-CI speed multiplier. Queue-time measurement and broader test-family optimization from #2190 remain follow-up work.

Cancellation fixture follow-up from the first CI run

Run 38262414072 merged all 925 files / 9,457 cases: 9,370 passed, 86 skipped, and one cancellation case failed with a missing SIGTERM marker and a cleanup-time unhandled rejection. Five shards passed. The new report flow worked on that actual failure: merge exited nonzero, diagnostics and their summary uploaded successfully, successful timing history was skipped, and the final shard gate failed.

Investigation identified two relevant mechanisms rather than assuming all timing failures were the same:

A deterministic 500ms target startup reproduces the missing-marker failure with the old 300ms capture deadline. The cancellation scenarios now use the same 1.5s observation window as other finalization scenarios. The 8s cancellation bound, 20s grace interval, and 20s test deadline remain unchanged.
The cancellation case used the bare SDK client instead of the existing fixture call path. In the installed SDK, callTool() without a supplied toolDefinition compiles every listed tool's output schema before dispatch. A progress probe observed about 6.6s of client-side work before the server began capture, consuming most of the existing readiness wait. The fixture now accepts the SDK's signal/progress options while retaining the selected tool definition and schema validation. The readiness wait begins after server progress, and the follow-up tools/list explicitly refreshes.

The delayed-startup control failed before the window correction and passed after it. A separately faulted MCP readiness path still failed its primary assertion without a secondary unhandled rejection. Correcting only the window did not make the complete coverage-mode file pass; that failed control is retained in the local evidence. The final fixture/dispatch changes passed all 12 cases through the actual CI coverage/blob command. No product runtime behavior or global timeout was changed.

Current-head CI completed successfully: all 32 jobs passed, including Windows, Apple artifact E2E, darwin arm64/Linux x64/Linux arm64 packaging, all six coverage shards, the strict aggregate and required gate. The separate Go binary lane also passed. The merged report contains 935 files / 9,630 cases: 9,544 passed, 86 skipped, zero failed. The previously failing cancellation case passed in the real shard (7.682s whole-case duration).

CI validated merge commit 756940c6 against newer main 86de439b as well as this branch. The scoped build-cache restore prefix matched, and Turbo correctly rebuilt inputs changed by newer main; this run is not a partial-reuse speed benchmark.

The successful timing artifact contains one JSON file, 95,955 bytes uncompressed / 16,275 bytes zipped, compared with the diagnostic bundle's 7,725,482 compressed bytes. All 935 names are checkout-relative. The actual sequencer accepted 923 positive durations under a different checkout root; zero-duration files retain its normal fallback. The PR was marked ready for review; automated review completed with no findings. It was squash-merged into main as 428eee00fdd0406cdf2b734c9ea14a2bb02da279 on 2026-10-12 at 15:42:48 UTC. The squash author uses the GitHub anonymous email; the GitHub committer uses noreply@github.com.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2210 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/12/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/2190-ci-test-gaps` --- ### 📝 Commits (3) - [`aec10e8`](https://github.com/morluto/rea/commit/aec10e81071ff9c55cab4920946469c59254c0cd) ci: keep validation fresh and preserve failed-shard reports - [`1f1c7d2`](https://github.com/morluto/rea/commit/1f1c7d2bde6301caeb85e62c832ee936c68a1825) chore: merge main documentation updates - [`2e28623`](https://github.com/morluto/rea/commit/2e286239127cda8914652b245354ee2bbd3fe5b4) test: cover delayed startup before finalization cancellation ### 📊 Changes **15 files changed** (+511 additions, -92 deletions) <details> <summary>View changed files</summary> 📝 `.github/actions/portable-build-cache/action.yml` (+4 -1) 📝 `.github/workflows/ci.yml` (+28 -27) 📝 `CONTRIBUTING.md` (+4 -0) 📝 `docs/testing.md` (+26 -9) 📝 `install.sh` (+22 -14) 📝 `package.json` (+3 -2) ➕ `scripts/ci/test-report.mjs` (+73 -0) 📝 `tests/acceptance/process/processFinalization.test.ts` (+34 -14) 📝 `tests/boundary/filesystem/ciChangeScope.test.ts` (+8 -0) ➕ `tests/boundary/filesystem/ciTestReports.test.ts` (+174 -0) 📝 `tests/boundary/filesystem/installScript.test.ts` (+6 -1) ➕ `tests/boundary/filesystem/testTaskCache.test.ts` (+108 -0) 📝 `tests/fixtures/localToolsMcp.ts` (+13 -3) 📝 `tests/fixtures/processFinalization.mjs` (+7 -1) 📝 `turbo.json` (+1 -20) </details> ### 📄 Description README-only changes currently bypass onboarding command consistency, cached full local gates can hide changed inputs, and a failed CI shard prevents useful blobs from being merged. This change closes those gaps and adds inexpensive producer coverage while preserving the required result and coverage contracts. Addresses #2190. The native ownership and six-shard changes from #2131 are already on main; this builds on them. ## Changes - Run the existing two-case README consistency guard from `docs:check`, outside the cached generated-doc task. Run the existing nine LLDB Python producer tests from the static CI job and the optional complete local gate. - Disable result caching for `test:run`; keep deterministic artifact preparation and static checks cached. This makes repository-input and external-dependency changes observable without hashing the ambient environment. - Merge available shard blobs before enforcing shard success. Preserve JSON/JUnit/coverage diagnostics after a failing merge, label unsuccessful or incomplete runs diagnostic-only, and keep the selected-shard and aggregate coverage gates strict. - Publish a separate compact `test-timings` artifact containing checkout-relative paths only after complete success. The sequencer consumes its existing report interface; old or missing artifacts fall back to native sharding. - Restore prior portable Turbo entries within an OS/architecture/toolchain/lockfile prefix so unchanged tasks remain reusable after an owner-specific input change. Turbo still validates task hashes; native outputs remain outside this cache. - Route MCP cancellation through the existing selected-tool fixture, wait for server progress before timing target readiness, observe pending rejection immediately, and force the post-cancellation catalog request through the server. Retain a deterministic delayed-startup control and all cancellation/cleanup assertions. - Preserve nonzero exit, no output, and no execution for every truncated installer prefix on Bash 3.2 and 5.2. Ordinary conditionals use `[ ]`; regex parsing uses a constant quoted conditional so incomplete `[[` grammar cannot mask a truncated download with status 0. Input values are expanded as data, with explicit shell-payload regressions. ## Validation Development uses Node 24.18.0 / npm 11.16.0. The branch was refreshed from main `10400fdb5`, then merged the testing-guide reorganization at `7ae3c50a9`. Current head: `2e2862391`. On the refreshed baseline: - `npm ci`, `npm run check:ci`, `npm run docs:check`, and `npm run test:python` passed. - `npm run verify:test-discovery` discovered all **925 files exactly once**. - Supported focused command passed **122 cases in five files**, including all **32 installer cases** and the unchanged every-byte truncation oracle on GNU Bash 5.2.15. - Commit/pre-push hooks and pinned actionlint 1.7.7 passed. - On the merged baseline, `npm run verify:package`, `npm pack --dry-run --json`, docs validation, complete static checks, and pinned actionlint passed. - All 23 cases in the three long-workflow control files passed on `1f1c7d2bd` with two workers. - After the cancellation-fixture follow-up below, the actual CI coverage/blob command passed all 12 finalization cases on the final source, with the delayed-startup case taking 8.875s. Typecheck, complete static checks, and commit/push hooks passed. Current-head CI and the separate Go binary lane are green. Controlled reproductions against the initial implementation baseline (`315973259`): - Maintained new regressions failed against the old production configuration, passed with the fix, and passed again under `LC_ALL=C`. - The real docs pipeline passed the original README, failed the actual incorrect command mutation twice, then passed after restoration. - The real Turbo regression fixture rejected a changed README and a changed external dependency after an earlier successful run; restoring either control passed. Preparation counters remained exactly one. - A comment-only managed-fixture input change rebuilt its owning task while **five of six preparation tasks were reused**, including compilation. - Removing the actual producer's stop-at-entry launch flag made the existing Python test fail; restoring it passed all nine checks. - The real installer suite passed all **32 cases** on GNU Bash **3.2.57 on Linux**. This establishes that Bash parser control, rather than macOS execution. - Two real CI shard commands produced blobs for one passing and one deliberately failing case. The real merge exited nonzero and still produced JSON/JUnit for both cases. The new reporter emitted partial diagnostics and no successful timing history. Coverage floors were left enabled. ### Full-suite result and limits The broad Linux run on `315973259` plus this patch executed **918 files / 9,391 cases** without coverage: **9,299 passed, 89 skipped, three failed**, plus one cleanup-time unhandled MCP connection error. Two long client/setup flows exceeded their unchanged 30-second deadline; the cancellation case did not observe its SIGTERM marker. The run took 27m12s on a shared host and overlapped investigation work, so it is neither a green full gate nor an isolated performance benchmark. All **23 cases in those three files passed** when rerun on this branch without changing deadlines or assertions. The clean `315973259` baseline also passed those 23 cases with two workers after giving its checkout a separate dependency installation. Its **1,361 compiled runtime files were byte-identical** to this patch's initial runtime. The timing failures were not reproduced in those controls; their full-load cause remains unproven. One attempted baseline control that shared dependencies with a concurrent `npm ci` was invalidated and excluded. Native Windows/macOS and real LLDB/provider coverage remain for the selected CI lanes; the Python fake-LLDB suite and Linux Bash 3.2 run do not establish those host/provider results. This PR does not reduce coverage floors, enable retries, delete slow workflow families, or promise a whole-CI speed multiplier. Queue-time measurement and broader test-family optimization from #2190 remain follow-up work. ## Cancellation fixture follow-up from the first CI run [Run 38262414072](https://github.com/morluto/rea/actions/runs/38262414072) merged all 925 files / 9,457 cases: 9,370 passed, 86 skipped, and one cancellation case failed with a missing SIGTERM marker and a cleanup-time unhandled rejection. Five shards passed. The new report flow worked on that actual failure: merge exited nonzero, diagnostics and their summary uploaded successfully, successful timing history was skipped, and the final shard gate failed. Investigation identified two relevant mechanisms rather than assuming all timing failures were the same: 1. A deterministic 500ms target startup reproduces the missing-marker failure with the old 300ms capture deadline. The cancellation scenarios now use the same 1.5s observation window as other finalization scenarios. The 8s cancellation bound, 20s grace interval, and 20s test deadline remain unchanged. 2. The cancellation case used the bare SDK client instead of the existing fixture call path. In the installed SDK, `callTool()` without a supplied `toolDefinition` compiles every listed tool's output schema before dispatch. A progress probe observed about 6.6s of client-side work before the server began capture, consuming most of the existing readiness wait. The fixture now accepts the SDK's signal/progress options while retaining the selected tool definition and schema validation. The readiness wait begins after server progress, and the follow-up `tools/list` explicitly refreshes. The delayed-startup control failed before the window correction and passed after it. A separately faulted MCP readiness path still failed its primary assertion without a secondary unhandled rejection. Correcting only the window did not make the complete coverage-mode file pass; that failed control is retained in the local evidence. The final fixture/dispatch changes passed all 12 cases through the actual CI coverage/blob command. No product runtime behavior or global timeout was changed. [Current-head CI](https://github.com/morluto/rea/actions/runs/38267217515) completed successfully: **all 32 jobs passed**, including Windows, Apple artifact E2E, darwin arm64/Linux x64/Linux arm64 packaging, all six coverage shards, the strict aggregate and required gate. The separate Go binary lane also passed. The merged report contains **935 files / 9,630 cases: 9,544 passed, 86 skipped, zero failed**. The previously failing cancellation case passed in the real shard (7.682s whole-case duration). CI validated merge commit `756940c6` against newer main `86de439b` as well as this branch. The scoped build-cache restore prefix matched, and Turbo correctly rebuilt inputs changed by newer main; this run is not a partial-reuse speed benchmark. The successful timing artifact contains one JSON file, **95,955 bytes uncompressed / 16,275 bytes zipped**, compared with the diagnostic bundle's 7,725,482 compressed bytes. All 935 names are checkout-relative. The actual sequencer accepted 923 positive durations under a different checkout root; zero-duration files retain its normal fallback. The PR was marked ready for review; automated review completed with no findings. It was squash-merged into main as `428eee00fdd0406cdf2b734c9ea14a2bb02da279` on 2026-10-12 at 15:42:48 UTC. The squash author uses the GitHub anonymous email; the GitHub committer uses `noreply@github.com`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:03:35 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:43:21 +00:00

[PR #5236] [CLOSED] docs(readme): carry update alias and output bound into Korean guide #6830

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5085

No description provided.

Rows
Columns