[PR #5154] [MERGED] fix(tests): complete independent agent evaluation cleanup #6789

New issue

Closed

opened 2026-11-01 21:43:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:43:02 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5154
Author: @gensokyo-dev
Created: 10/19/2026
Status: ✅ Merged
Merged: 10/19/2026
Merged by: @gensokyo-dev

Base: main ← Head: audit/ci-tests-20261019

📝 Commits (6)

5a91c19 fix(tests): complete independent agent evaluation cleanup
2535f8b fix(tests): preserve evaluation fixture startup diagnostics
7ebbd8e fix(tests): retain cooperative evaluation coverage for root
27ca6fb chore: merge latest main for CI verification
79df05d chore: merge latest main for CI verification
35296a5 Merge branch 'main' into audit/ci-tests-20261019

📊 Changes

3 files changed (+170 additions, -4 deletions)

View changed files

📝 docs/client-verification.md (+3 -1)
📝 scripts/verify-agent-experience.mjs (+14 -3)
➕ tests/boundary/process/agentEvaluationCleanup.test.ts (+153 -0)

📄 Description

Problem and change

Fixes #5095. If verify:agent cannot remove its disposable auth file, its finalizer previously skipped the remaining run tree and replaced the original verification error. Reuse closeVerifierResources to attempt independent cleanups and retain their errors alongside the primary failure. Fixture retention keeps the run tree while still attempting auth removal.

The regression launches the actual verifier with an inert, failing --version executable and real filesystem permission denial. It covers ordinary cleanup and both retention modes before authentication copying, setup, or model requests.

Validation

Current head: 35296a58f, with main b999e7247 integrated. The PR still changes only the verifier, its regression file and the documentation.

Local focused verification: 28/28 tests pass across 6 files, covering verifier/helper cleanup and the previously failing process-snapshot families.
Current-head CI passes, including source tests, all five portable shards, build, static checks, docs and required aggregation. Complete job inventory: 12 succeeded, 19 conditionally skipped.
Codex review completed with no unresolved findings; Greptile 5/5 at the same head. Both review threads are resolved.
Failure control at baseline d6ca7b3f2: both permission-denial cases fail while both cooperative controls pass. Verifier/helper regressions pass after the fix.
Root control at 27ca6fb9f: 2 cooperative cases pass, 2 denial cases skip. Startup negative control: all 4 cases fail with the original ERR_MODULE_NOT_FOUND, without a receipt error; required module restored before passing runs.

Linux, Node 24.18.0/npm 11.16.0; heavy commands restricted to two CPUs and serialized. Current focused verification reused prepared artifacts: 40.2 s elapsed, 47.0 s CPU, 1,134 MiB peak sampled process-family summed RSS. Summed RSS may double-count shared pages and miss short-lived/reparented children. Agent-token cost is unmeasured; no suite speedup is claimed.

Coverage limits: synthetic executable, no live Codex/model/provider run. The POSIX executable fixture skips Windows; only denial cases skip root. Conditional provider lanes did not run. Process-timeout ownership in #5094 is a separate, unchanged boundary.

Author readiness

Human owner: @gensokyo-dev

The owner explicitly delegated these declarations and authorized opening ready, following CI/review, and merging in this session. Codex reviewed the diff and ran the stated validation; independent human review was not observed.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5154 **Author:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Created:** 10/19/2026 **Status:** ✅ Merged **Merged:** 10/19/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `audit/ci-tests-20261019` --- ### 📝 Commits (6) - [`5a91c19`](https://github.com/morluto/rea/commit/5a91c190b37421c532f8a13a35659706bbdaeec6) fix(tests): complete independent agent evaluation cleanup - [`2535f8b`](https://github.com/morluto/rea/commit/2535f8bdfe612677583bcd364c11a0ac96e35610) fix(tests): preserve evaluation fixture startup diagnostics - [`7ebbd8e`](https://github.com/morluto/rea/commit/7ebbd8e9f95814162e7ab9d265331f3caf63ca52) fix(tests): retain cooperative evaluation coverage for root - [`27ca6fb`](https://github.com/morluto/rea/commit/27ca6fb9f633f0774ee1906adacb72b28b7cf9b4) chore: merge latest main for CI verification - [`79df05d`](https://github.com/morluto/rea/commit/79df05dbcfa009a45eb19d85e0f1c949484cd1de) chore: merge latest main for CI verification - [`35296a5`](https://github.com/morluto/rea/commit/35296a58f479bbdcd14a61e60cd254f1a2054f85) Merge branch 'main' into audit/ci-tests-20261019 ### 📊 Changes **3 files changed** (+170 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `docs/client-verification.md` (+3 -1) 📝 `scripts/verify-agent-experience.mjs` (+14 -3) ➕ `tests/boundary/process/agentEvaluationCleanup.test.ts` (+153 -0) </details> ### 📄 Description ## Problem and change Fixes #5095. If `verify:agent` cannot remove its disposable auth file, its finalizer previously skipped the remaining run tree and replaced the original verification error. Reuse `closeVerifierResources` to attempt independent cleanups and retain their errors alongside the primary failure. Fixture retention keeps the run tree while still attempting auth removal. The regression launches the actual verifier with an inert, failing `--version` executable and real filesystem permission denial. It covers ordinary cleanup and both retention modes before authentication copying, setup, or model requests. ## Validation Current head: `35296a58f`, with main `b999e7247` integrated. The PR still changes only the verifier, its regression file and the documentation. - Local focused verification: **28/28 tests pass across 6 files**, covering verifier/helper cleanup and the previously failing process-snapshot families. - [Current-head CI](https://github.com/morluto/rea/actions/runs/39245430920) **passes**, including source tests, all five portable shards, build, static checks, docs and required aggregation. Complete job inventory: **12 succeeded, 19 conditionally skipped**. - Codex review completed with no unresolved findings; Greptile **5/5** at the same head. Both review threads are resolved. - Failure control at baseline `d6ca7b3f2`: both permission-denial cases fail while both cooperative controls pass. Verifier/helper regressions pass after the fix. - Root control at `27ca6fb9f`: **2 cooperative cases pass, 2 denial cases skip**. Startup negative control: all 4 cases fail with the original `ERR_MODULE_NOT_FOUND`, without a receipt error; required module restored before passing runs. Linux, Node 24.18.0/npm 11.16.0; heavy commands restricted to two CPUs and serialized. Current focused verification reused prepared artifacts: **40.2 s elapsed, 47.0 s CPU, 1,134 MiB peak sampled process-family summed RSS**. Summed RSS may double-count shared pages and miss short-lived/reparented children. Agent-token cost is unmeasured; no suite speedup is claimed. Coverage limits: synthetic executable, no live Codex/model/provider run. The POSIX executable fixture skips Windows; only denial cases skip root. Conditional provider lanes did not run. Process-timeout ownership in #5094 is a separate, unchanged boundary. ## Author readiness Human owner: @gensokyo-dev The owner explicitly delegated these declarations and authorized opening ready, following CI/review, and merging in this session. Codex reviewed the diff and ran the stated validation; independent human review was not observed. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:43:02 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6789

No description provided.

Rows
Columns