morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3281] perf(ci): separate fast mutation preflight from full qualification and retain both reports #1173

New issue

Closed

opened 2026-10-15 18:31:44 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:31:44 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3281

Originally assigned to: @N0zoM1z0 on GitHub.

The reverse-tools quality lane dominates normal PR feedback even though the repository already distinguishes fast feedback from complete qualification. The two Stryker passes also write to the same default report paths, so the second pass replaces the first pass's machine-readable and HTML diagnostics. Neither report is retained as a CI artifact.

Observed passing baseline: CI run 38510852040, source e6bba0efea285da450b59767d5d09528ba2d8103, job 115679799382.

Observed cost	Duration
Entire reverse-tools quality job	34m 37s
Focused coverage step	1m 02s
Two complete mutation passes	33m 10s
Quality-job tail after every other job completed	27m 02s

These are step/job timestamps from a successful hosted run, not an estimate from a failed or smaller test workload. A prior run also had a roughly 30-minute quality tail. Eligible-to-start runner delay is a separate cost; the cited quality job waited approximately 4m 22s after its static prerequisite completed. The available data does not establish whether that wait came from account concurrency, runner availability, or another scheduling constraint.

The main pass generated 761 mutants across nine production files; its final report contains 760 outcomes, including compile-error mutants. The semantics pass generated 29 mutants. Both configs use Stryker's default reports/mutation/mutation.json and mutation.html, so the final files describe only the second pass. Mutation score thresholds are currently deliberately non-blocking (break: null); initial test failures and runner/configuration failures do block CI.

Proposed correction at the owning quality-job boundary:

Keep fresh focused coverage for every selected quality lane. In fast feedback, prepare the mutation fixture and run Stryker's real --dryRunOnly mode against both canonical configs. This resolves the actual production targets, instruments the code, and runs the real initial tests. It must still reject invalid targets, configuration failures, and failing initial tests.
Keep both complete mutation passes unchanged in full qualification, including scheduled, explicit ci:full, manual, and release qualification. Preserve targets, test inventories, checkers, concurrency, timeout settings, and thresholds. This deliberately moves mutant-outcome feedback to full qualification; a preflight is not a complete mutation run and must never be reported as one.
Give each canonical pass a distinct JSON/HTML destination. Retain available coverage and mutation diagnostics after failure, with tier and attempt identity and a bounded retention period. Partial diagnostics are not qualification proof. Preserve cancellation and the fail-closed required-lane gate.

Validation should include workflow tier-selection and required-gate negative controls; actual production preflights for both configs; an actual Stryker failing-base-test control; representative full mutation executions that produce and retain both real JSON/HTML reports; and a second-stage failure that does not overwrite the first stage's diagnostics. Record elapsed time, CPU, sampled process-family memory, exact source/tool versions, and scope. Compare hosted current-head results after implementation before claiming an end-to-end improvement. Do not cache live test results or reduce the portable test inventory.

Additional observed capacity cost at 2026-10-14 07:30:48 UTC: a read-only scan of 12 active/queued runs in this repository found 19 visible running jobs, including five complete reverse-tools quality jobs already occupying runners for 754, 1,088, 1,294, 1,623 and 1,993 seconds. Two visible jobs remained queued, including a portable-report merge whose test shards had already succeeded. The scan retained every jobs page and only emitted compact counts; requests span a short sampling interval rather than an atomic account-wide view. This supports reducing long-lived quality occupancy, but does not reveal account concurrency limits or jobs in other repositories. GitHub's official status API reported Actions operational with no incident when checked at approximately 07:27 UTC. That does not exclude localized scheduling delays.

Current inspected main is 29d3dcb7ea85b3be37cd81e977ae4e09cebdd2fc; the cited quality workflow and Stryker configs remain unchanged from the passing cost baseline. Implementation is queued after the current correctness fixes, with one independent branch and a complete validation/review/merge cycle.

Originally created by @N0zoM1z0 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3281 Originally assigned to: @N0zoM1z0 on GitHub. The reverse-tools quality lane dominates normal PR feedback even though the repository already distinguishes fast feedback from complete qualification. The two Stryker passes also write to the same default report paths, so the second pass replaces the first pass's machine-readable and HTML diagnostics. Neither report is retained as a CI artifact. Observed passing baseline: CI run [38510852040](https://github.com/morluto/rea/actions/runs/38510852040), source `e6bba0efea285da450b59767d5d09528ba2d8103`, job `115679799382`. | Observed cost | Duration | | --- | ---: | | Entire reverse-tools quality job | 34m 37s | | Focused coverage step | 1m 02s | | Two complete mutation passes | 33m 10s | | Quality-job tail after every other job completed | 27m 02s | These are step/job timestamps from a successful hosted run, not an estimate from a failed or smaller test workload. A prior run also had a roughly 30-minute quality tail. Eligible-to-start runner delay is a separate cost; the cited quality job waited approximately 4m 22s after its static prerequisite completed. The available data does not establish whether that wait came from account concurrency, runner availability, or another scheduling constraint. The main pass generated 761 mutants across nine production files; its final report contains 760 outcomes, including compile-error mutants. The semantics pass generated 29 mutants. Both configs use Stryker's default `reports/mutation/mutation.json` and `mutation.html`, so the final files describe only the second pass. Mutation score thresholds are currently deliberately non-blocking (`break: null`); initial test failures and runner/configuration failures do block CI. Proposed correction at the owning quality-job boundary: 1. Keep fresh focused coverage for every selected quality lane. In fast feedback, prepare the mutation fixture and run Stryker's real `--dryRunOnly` mode against **both canonical configs**. This resolves the actual production targets, instruments the code, and runs the real initial tests. It must still reject invalid targets, configuration failures, and failing initial tests. 2. Keep both complete mutation passes unchanged in full qualification, including scheduled, explicit `ci:full`, manual, and release qualification. Preserve targets, test inventories, checkers, concurrency, timeout settings, and thresholds. This deliberately moves mutant-outcome feedback to full qualification; a preflight is not a complete mutation run and must never be reported as one. 3. Give each canonical pass a distinct JSON/HTML destination. Retain available coverage and mutation diagnostics after failure, with tier and attempt identity and a bounded retention period. Partial diagnostics are not qualification proof. Preserve cancellation and the fail-closed required-lane gate. Validation should include workflow tier-selection and required-gate negative controls; actual production preflights for both configs; an actual Stryker failing-base-test control; representative full mutation executions that produce and retain both real JSON/HTML reports; and a second-stage failure that does not overwrite the first stage's diagnostics. Record elapsed time, CPU, sampled process-family memory, exact source/tool versions, and scope. Compare hosted current-head results after implementation before claiming an end-to-end improvement. Do not cache live test results or reduce the portable test inventory. Additional observed capacity cost at **2026-10-14 07:30:48 UTC**: a read-only scan of 12 active/queued runs in this repository found **19 visible running jobs**, including **five complete reverse-tools quality jobs** already occupying runners for 754, 1,088, 1,294, 1,623 and 1,993 seconds. Two visible jobs remained queued, including a portable-report merge whose test shards had already succeeded. The scan retained every jobs page and only emitted compact counts; requests span a short sampling interval rather than an atomic account-wide view. This supports reducing long-lived quality occupancy, but does not reveal account concurrency limits or jobs in other repositories. GitHub's [official status API](https://www.githubstatus.com/api/v2/summary.json) reported Actions operational with no incident when checked at approximately 07:27 UTC. That does not exclude localized scheduling delays. Current inspected main is `29d3dcb7ea85b3be37cd81e977ae4e09cebdd2fc`; the cited quality workflow and Stryker configs remain unchanged from the passing cost baseline. Implementation is queued after the current correctness fixes, with one independent branch and a complete validation/review/merge cycle.

bsakaguchi

2026-10-15 18:31:44 +00:00

closed this issue
added the
performance

P1: high
labels

bsakaguchi commented

2026-10-15 18:31:45 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

Prepared implementation: commit 117f2e4476b034cf71f318866a0b571bc680add3, exact six-file diff. Its integration base contains main ba33c706d and the independently validated #3410 / #3299 fixture prerequisites. The PR will follow those prerequisite merges.

Fast selected quality retains fresh coverage, then runs actual Stryker preflights for both canonical configs. Full-tier qualification and the standalone npm mutation command keep both complete executions, original targets/checkers/concurrency/deadlines and non-blocking score threshold. Mutant-outcome feedback is deliberately deferred to full qualification; this is a change in feedback phase, not a claim of an equivalent-workload algorithmic speedup.

Executed validation (Node 24.18.0, npm 11.16.0, Vitest 5.0.3, Stryker 10.0.0, shared Linux host):

Current integration tree: coverage + both real preflights pass in 287.72 s; CPU 301.32 s user + 70.31 s system; sampled summed process-family RSS 4,277 MiB. The clean report directory contains zero mutation-outcome reports, as expected for preflights.
Complete canonical standalone command on earlier pinned base 82f9b4299 with the same reporting/phase implementation: both full passes pass in 2,869.45 s. All four distinct JSON/HTML files exist. Main report: 9 sources, 761 entries including one ignored mutant; semantics: 2 sources, 29 entries. This proves full report writing/retention, not complete mutant outcomes for the later production-source revision.
Actual second-stage full-mode initial-run failure returns nonzero, preserves both first-stage report hashes, and creates no success report for the failed stage. Independent real Vitest failing-base-test preflight control also returns nonzero.
Four CI policy/config families: 241 tests pass. After the latest upstream integration, both directly changed families pass all 11 tests again. Required-job controls reject failed, canceled, skipped and missing selected quality jobs in both tiers.
Current check:ci, architecture guards, pinned actionlint 1.7.7 and discovery all 1,273 files exactly once pass.
Four actual application-service analyses, including the repository's Notes Electron application, pass in 3.57 s, CPU 3.43 + 0.98 s, sampled family RSS 243 MiB. Static analysis reads real source artifacts; it does not execute the application.

No live test-result cache, assertion/deadline relaxation or test inventory pruning. Available diagnostics upload after failure with tier/attempt identity and seven-day retention; canceled jobs skip upload. Both full passes have distinct explicit report names. Raw measurements/reports remain outside the tracked tree; summarized data avoids returning source-filled reports as agent context. RSS samples are 200 ms sums (shared pages can be counted twice; short-lived/detached peaks can be missed). Agent token savings and account-level runner capacity remain unmeasured.

Latest integration checkpoint

The owner has now explicitly confirmed the initial diff/evidence/ownership/readiness declarations. On fetching current main, the complete original six-file patch was already present in 1b463987959815842c99920b5801358252af19a4, with the requested anonymous author email and Kaoru co-author trailer. Binary-diff comparison confirms exact incorporation. No duplicate PR or artificial merge is needed.

Main follow-up 92a0edf1875f74528910db461ed03f782b7b59b3 additionally moves complete mutation execution to the isolated daily/manual Mutation sweep workflow. Required fast and full CI, including release qualification, now retain fresh coverage and both canonical preflights. Complete mutation execution is maintenance outside the PR/release gates; ci:full no longer requests mutant outcomes. Canonical targets, checkers, limits and the existing non-blocking mutation-score threshold remain unchanged.

Verified baseline: main 3b79af28e80785afce9f5998d8f03289402a1847. Local integration has an identical entire tree. Eight relevant test files / 279 tests / 45.72 s passed, plus check:ci and architecture guards across 1,394 files. Both completed hosted fast CI 38564501608 and full CI 38565969170 yielded 1,274 unique test files, 14,528 tests passed, 125 skipped, zero failures, no duplicate files. All 35 fast jobs succeeded; the full run had 36 successful and one skipped job. The latter is fresh full CI, not a reused qualification result.

The quality job took 193 s (fast) and 237 s (full). Its retained artifact contains valid Istanbul coverage for 13 files (2,347 / 2,470 statements hit) and no mutation-outcome reports. The full-tier job log confirms both real preflights, including 517 initial tests in the main configuration. These are different feedback phases from the older complete-sweep measurements, not equivalent-workload speedup ratios.

Four actual application-service analyses, including the tracked Notes Electron example, passed again: 6.32 s wall, 5.25 s user + 2.59 s system CPU, 244.29 MiB sampled summed family RSS (same 200 ms sampling limitations above). This is a fresh correctness observation, not a comparative performance claim.

The first real manual run of the new workflow, 38568262316, is bound to the same main SHA. Its semantics job has succeeded; the main complete sweep is still running. Final report verification will be recorded when it finishes. No passing full-sweep result is being claimed prematurely.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): Prepared implementation: commit `117f2e4476b034cf71f318866a0b571bc680add3`, [exact six-file diff](https://github.com/morluto/rea/compare/2a2c9fa574b44243cd61a5649a8c4824c055712f...perf/3281-reverse-tools-feedback-quality). Its integration base contains main `ba33c706d` and the independently validated #3410 / #3299 fixture prerequisites. The PR will follow those prerequisite merges. Fast selected quality retains fresh coverage, then runs actual Stryker preflights for **both canonical configs**. Full-tier qualification and the standalone npm mutation command keep both complete executions, original targets/checkers/concurrency/deadlines and non-blocking score threshold. Mutant-outcome feedback is deliberately deferred to full qualification; this is a change in feedback phase, not a claim of an equivalent-workload algorithmic speedup. Executed validation (Node 24.18.0, npm 11.16.0, Vitest 5.0.3, Stryker 10.0.0, shared Linux host): - Current integration tree: coverage + both real preflights **pass in 287.72 s**; CPU 301.32 s user + 70.31 s system; sampled summed process-family RSS 4,277 MiB. The clean report directory contains **zero mutation-outcome reports**, as expected for preflights. - Complete canonical standalone command on earlier pinned base `82f9b4299` with the same reporting/phase implementation: **both full passes pass in 2,869.45 s**. All four distinct JSON/HTML files exist. Main report: 9 sources, 761 entries including one ignored mutant; semantics: 2 sources, 29 entries. This proves full report writing/retention, not complete mutant outcomes for the later production-source revision. - Actual second-stage **full-mode** initial-run failure returns nonzero, preserves both first-stage report hashes, and creates no success report for the failed stage. Independent real Vitest failing-base-test preflight control also returns nonzero. - Four CI policy/config families: **241 tests pass**. After the latest upstream integration, both directly changed families pass all **11 tests** again. Required-job controls reject failed, canceled, skipped and missing selected quality jobs in both tiers. - Current `check:ci`, architecture guards, pinned actionlint 1.7.7 and discovery **all 1,273 files exactly once** pass. - Four actual application-service analyses, including the repository's Notes Electron application, **pass in 3.57 s**, CPU 3.43 + 0.98 s, sampled family RSS 243 MiB. Static analysis reads real source artifacts; it does not execute the application. No live test-result cache, assertion/deadline relaxation or test inventory pruning. Available diagnostics upload after failure with tier/attempt identity and seven-day retention; canceled jobs skip upload. Both full passes have distinct explicit report names. Raw measurements/reports remain outside the tracked tree; summarized data avoids returning source-filled reports as agent context. RSS samples are 200 ms sums (shared pages can be counted twice; short-lived/detached peaks can be missed). Agent token savings and account-level runner capacity remain unmeasured. ### Latest integration checkpoint The owner has now explicitly confirmed the initial diff/evidence/ownership/readiness declarations. On fetching current main, the complete original six-file patch was already present in `1b463987959815842c99920b5801358252af19a4`, with the requested anonymous author email and Kaoru co-author trailer. Binary-diff comparison confirms exact incorporation. No duplicate PR or artificial merge is needed. Main follow-up `92a0edf1875f74528910db461ed03f782b7b59b3` additionally moves complete mutation execution to the isolated daily/manual **Mutation sweep** workflow. Required fast and full CI, including release qualification, now retain fresh coverage and both canonical preflights. Complete mutation execution is maintenance outside the PR/release gates; `ci:full` no longer requests mutant outcomes. Canonical targets, checkers, limits and the existing non-blocking mutation-score threshold remain unchanged. Verified baseline: main `3b79af28e80785afce9f5998d8f03289402a1847`. Local integration has an identical entire tree. Eight relevant test files / **279 tests / 45.72 s** passed, plus `check:ci` and architecture guards across 1,394 files. Both completed hosted fast CI `38564501608` and full CI `38565969170` yielded **1,274 unique test files, 14,528 tests passed, 125 skipped, zero failures, no duplicate files**. All 35 fast jobs succeeded; the full run had 36 successful and one skipped job. The latter is fresh full CI, not a reused qualification result. The quality job took 193 s (fast) and 237 s (full). Its retained artifact contains valid Istanbul coverage for 13 files (2,347 / 2,470 statements hit) and no mutation-outcome reports. The full-tier job log confirms both real preflights, including 517 initial tests in the main configuration. These are different feedback phases from the older complete-sweep measurements, not equivalent-workload speedup ratios. Four actual application-service analyses, including the tracked Notes Electron example, passed again: 6.32 s wall, 5.25 s user + 2.59 s system CPU, 244.29 MiB sampled summed family RSS (same 200 ms sampling limitations above). This is a fresh correctness observation, not a comparative performance claim. The first real manual run of the new workflow, `38568262316`, is bound to the same main SHA. Its semantics job has succeeded; the main complete sweep is still running. Final report verification will be recorded when it finishes. No passing full-sweep result is being claimed prematurely.

bsakaguchi commented

2026-10-15 18:31:45 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 14, 2026):

Implemented in main; closing completed without opening a duplicate PR.

1b463987959815842c99920b5801358252af19a4 incorporates the prepared six-file patch byte-for-byte, retaining the anonymous author email and Kaoru co-author trailer.
92a0edf1875f74528910db461ed03f782b7b59b3 further isolates complete mutation sweeps in the daily/manual maintenance workflow. Required fast and full CI both keep fresh coverage and both real canonical preflights. Complete scans retain their targets, checkers, limits, and existing non-blocking score threshold. ci:full now requests regression/coverage/provider qualification, not full mutant outcomes.
On main 3b79af28e, both hosted fast CI 38564501608 and fresh full CI 38565969170 report 1,274 unique files / 14,528 passed / 125 skipped / zero failures / zero duplicates. Quality jobs took 193 and 237 s. These phases execute different work from a complete sweep; no equivalent-workload speedup ratio is claimed.
First real standalone workflow execution 38568262316 succeeded for both suites. Main: 9 sources / 761 entries, including 1 ignored; semantics: 2 sources / 29 entries. All four JSON/HTML reports are nonempty, separate, parsed, and have only terminal statuses. Main and semantics job durations were 2,032 s and 126 s. Hosted CPU/RSS were not measured.
Current later main eab789aed passed 15 relevant files / 490 tests, check:ci and 1,398-file architecture guards locally. Its CI detected a separate new cache-input omission for .github/actions/publish-mcp/registry-state.{mjs,d.mts}, introduced by subsequent release recovery work. That failure is retained and will be investigated independently; it is not a passing latest-main result and is not attributable to this mutation-workflow change.

The owner explicitly confirmed the initial review/ownership/evidence/readiness declarations. No declaration was fabricated, no tests or mutation thresholds were weakened, and no additional PR/merge is being claimed for a patch already incorporated upstream. Detailed evidence remains in the validation comment above.

@N0zoM1z0 commented on GitHub (Oct 14, 2026): Implemented in main; closing completed without opening a duplicate PR. - `1b463987959815842c99920b5801358252af19a4` incorporates the prepared six-file patch byte-for-byte, retaining the anonymous author email and Kaoru co-author trailer. - `92a0edf1875f74528910db461ed03f782b7b59b3` further isolates complete mutation sweeps in the daily/manual maintenance workflow. Required fast and full CI both keep fresh coverage and both real canonical preflights. Complete scans retain their targets, checkers, limits, and existing non-blocking score threshold. `ci:full` now requests regression/coverage/provider qualification, not full mutant outcomes. - On main `3b79af28e`, both hosted fast CI `38564501608` and fresh full CI `38565969170` report **1,274 unique files / 14,528 passed / 125 skipped / zero failures / zero duplicates**. Quality jobs took 193 and 237 s. These phases execute different work from a complete sweep; no equivalent-workload speedup ratio is claimed. - First real standalone workflow execution [38568262316](https://github.com/morluto/rea/actions/runs/38568262316) succeeded for both suites. Main: 9 sources / 761 entries, including 1 ignored; semantics: 2 sources / 29 entries. All four JSON/HTML reports are nonempty, separate, parsed, and have only terminal statuses. Main and semantics job durations were 2,032 s and 126 s. Hosted CPU/RSS were not measured. - Current later main `eab789aed` passed **15 relevant files / 490 tests**, `check:ci` and 1,398-file architecture guards locally. Its CI detected a **separate new cache-input omission** for `.github/actions/publish-mcp/registry-state.{mjs,d.mts}`, introduced by subsequent release recovery work. That failure is retained and will be investigated independently; it is not a passing latest-main result and is not attributable to this mutation-workflow change. The owner explicitly confirmed the initial review/ownership/evidence/readiness declarations. No declaration was fabricated, no tests or mutation thresholds were weakened, and no additional PR/merge is being claimed for a patch already incorporated upstream. Detailed evidence remains in the validation comment above.

bsakaguchi referenced this issue

2026-10-16 21:10:21 +00:00

[GH-ISSUE #4522] [Bug] Darwin helper cleanup rejection replaces the compiler failure #1799

bsakaguchi referenced this issue

2026-10-17 21:09:19 +00:00

[PR #1194] [MERGED] fix(process): stop cancelled Swift helper builds before removing their files #2707

bsakaguchi referenced this issue

2026-10-31 00:39:24 +00:00

[GH-ISSUE #5502] [Bug] Cancelled JavaScript analysis can still leave the macOS helper's temporary folder after #1194 #3348