[PR #6988] [MERGED] test(evaluation): locate the runner gate across cleanup wrappers #7551

New issue

Closed

opened 2026-11-03 13:38:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 13:38:38 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6988
Author: @gensokyo-dev
Created: 10/24/2026
Status: ✅ Merged
Merged: 10/24/2026
Merged by: @gensokyo-dev

Base: main ← Head: test/evaluation-gate-20261024

📝 Commits (1)

36ae757 test(evaluation): locate the runner gate across cleanup wrappers

📊 Changes

1 file changed (+77 additions, -33 deletions)

View changed files

📝 tests/evaluation/model-evals/agentEvaluationReport.test.ts (+77 -33)

📄 Description

Problem and outcome

Source CI after the analysis repair completes 17585 tests but fails agentEvaluationReport.test.ts with Runner failure gate missing. The verifier's failure predicate is intact; cleanup ownership moved its body into an async callback, while the test only searched top-level try blocks.

Walk the parsed AST and require exactly one failed = results.filter(...) predicate regardless of surrounding cleanup structure. Retain only its validated source range, then execute the actual callback with the same grading helper. Remove the fixed-depth scan. Check both accepting a correct report and rejecting incorrect factual claims, and add missing/ambiguous/decoy gate cases. The verifier and production release gate are unchanged. No separate issue was opened for this direct main-health repair.

Executed validation

Pinned Node 24.22.1 / npm 11.16.0, Linux x64, two CPUs:

Clean pre-analysis-fix worktree d3044a2df: exact same missing-gate failure, 1 failed / 5 passed (2.35 s wall / 2.35 s CPU / 380.9 MiB sampled process-family RSS).
Merged base dd02d8ff6 plus this diff: changed file 9 passed; all five model-evaluation files 114 passed (9.28 s wall / 13.02 s CPU / 1163.4 MiB).
Negative controls changed only the actual runner callback to accept every report, then reject every report. The real-runner regression failed with an assertion in both cases (1.56 s / 267.6 MiB and 1.64 s / 269.5 MiB). Both mutations were restored; no verifier change remains in the diff.
Final npm run check and git diff --check: passed (48.51 s wall / 83.17 s CPU / 2313.4 MiB; 1/6 static tasks cached).

These are deterministic fixture-grading and real-predicate wiring checks; no model, external agent or network trial was run. RSS sums descendants at 100 ms intervals; shared pages may double-count and short-lived/reparented children may be missed. Tests had a 1.5 GiB sampled stop, static checks an 8 GiB stop, and all commands had elapsed guards. No OOM occurred. Costs describe validation, not a speedup claim; agent tokens were not measured. Complete current-head CI finished: Source passed 17,689 tests in 977 files; static/build and seven of eight built shards passed. Shard 2 failed the unchanged Worker URL CLI fixture: two type-only Worker imports still expect a runtime shadow after main's intentional type-erasure correction. Runtime code and that fixture are identical to merged base dd02d8ff6; this evaluation-test-only change cannot alter their behavior. A local diagnostic stopped at its 1.5 GiB sampled guard before an assertion, so it does not count as a local reproduction. No leftover test processes were observed afterward.

Codex reviewed exact head 36ae7574d without findings. This merge repairs the evaluation gate; overall CI is still failing on the separately identified Worker fixture. Repair and post-merge main CI remain required for the maintenance task.

Author readiness

Human owner: @gensokyo-dev

The owner explicitly delegated readiness declarations and agent-managed review. Inspection and execution were performed by Codex; no separate human inspection is claimed.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6988 **Author:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Created:** 10/24/2026 **Status:** ✅ Merged **Merged:** 10/24/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `test/evaluation-gate-20261024` --- ### 📝 Commits (1) - [`36ae757`](https://github.com/morluto/rea/commit/36ae7574d4747afa6b41551859b0b611e785bd11) test(evaluation): locate the runner gate across cleanup wrappers ### 📊 Changes **1 file changed** (+77 additions, -33 deletions) <details> <summary>View changed files</summary> 📝 `tests/evaluation/model-evals/agentEvaluationReport.test.ts` (+77 -33) </details> ### 📄 Description ## Problem and outcome [Source CI after the analysis repair](https://github.com/morluto/rea/actions/runs/40083831021/job/120842824894) completes 17585 tests but fails `agentEvaluationReport.test.ts` with `Runner failure gate missing`. The verifier's failure predicate is intact; cleanup ownership moved its body into an async callback, while the test only searched top-level try blocks. Walk the parsed AST and require exactly one `failed = results.filter(...)` predicate regardless of surrounding cleanup structure. Retain only its validated source range, then execute the actual callback with the same grading helper. Remove the fixed-depth scan. Check both accepting a correct report and rejecting incorrect factual claims, and add missing/ambiguous/decoy gate cases. The verifier and production release gate are unchanged. No separate issue was opened for this direct main-health repair. ## Executed validation Pinned Node 24.22.1 / npm 11.16.0, Linux x64, two CPUs: - Clean pre-analysis-fix worktree `d3044a2df`: exact same missing-gate failure, **1 failed / 5 passed** (2.35 s wall / 2.35 s CPU / 380.9 MiB sampled process-family RSS). - Merged base `dd02d8ff6` plus this diff: changed file **9 passed**; all five model-evaluation files **114 passed** (9.28 s wall / 13.02 s CPU / 1163.4 MiB). - Negative controls changed only the actual runner callback to accept every report, then reject every report. The real-runner regression failed with an assertion in both cases (1.56 s / 267.6 MiB and 1.64 s / 269.5 MiB). Both mutations were restored; no verifier change remains in the diff. - Final `npm run check` and `git diff --check`: passed (48.51 s wall / 83.17 s CPU / 2313.4 MiB; 1/6 static tasks cached). These are deterministic fixture-grading and real-predicate wiring checks; no model, external agent or network trial was run. RSS sums descendants at 100 ms intervals; shared pages may double-count and short-lived/reparented children may be missed. Tests had a 1.5 GiB sampled stop, static checks an 8 GiB stop, and all commands had elapsed guards. No OOM occurred. Costs describe validation, not a speedup claim; agent tokens were not measured. [Complete current-head CI](https://github.com/morluto/rea/actions/runs/40085480702) finished: Source passed **17,689 tests in 977 files**; static/build and seven of eight built shards passed. Shard 2 failed the unchanged Worker URL CLI fixture: two type-only Worker imports still expect a runtime shadow after main's intentional type-erasure correction. Runtime code and that fixture are identical to merged base `dd02d8ff6`; this evaluation-test-only change cannot alter their behavior. A local diagnostic stopped at its 1.5 GiB sampled guard before an assertion, so it does not count as a local reproduction. No leftover test processes were observed afterward. Codex reviewed exact head `36ae7574d` without findings. This merge repairs the evaluation gate; overall CI is still failing on the separately identified Worker fixture. Repair and post-merge main CI remain required for the maintenance task. ## Author readiness Human owner: @gensokyo-dev The owner explicitly delegated readiness declarations and agent-managed review. Inspection and execution were performed by Codex; no separate human inspection is claimed. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-03 13:38:38 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7551

No description provided.

Rows
Columns