morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5095] [Bug] Agent evaluation skips disposable sibling cleanup when auth-file removal fails #3132

New issue

Closed

opened 2026-10-31 00:36:59 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:36:59 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5095

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

SOURCE-ONLY caller tracing, with an executed extracted cleanup-boundary probe. No real Codex evaluation ran.

The finalizer of scripts/verify-agent-experience.mjs awaits removal of codex/auth.json before attempting recursive removal of its disposable run directory. A permission denial on the first removal skips cleanup of the unaffected account and targets siblings, even when fixture retention is unset.

Using the exact finalizer extracted from source and real filesystem operations, a synthetic codex directory with mode 0500 caused EACCES. Only the auth removal was attempted; account and targets remained. A separate recursive-removal control under the same denial removed both siblings while still reporting EACCES for the denied subtree.

Reproduction

From the source checkout at the revision below, run as an ordinary non-root user. This executes only the extracted finalizer and synthetic filesystem fixtures; it does not launch Codex or load REA build artifacts.

WORK="$(mktemp -d)"
cat > "$WORK/probe.mjs" <<'JS'
import { readFile, mkdtemp, mkdir, writeFile, chmod, rm, access } from "node:fs/promises";
import { join } from "node:path";
const checkout = process.argv[2];
const scratch = process.argv[3];
const source = await readFile(join(checkout, "scripts/verify-agent-experience.mjs"), "utf8");
const start = source.indexOf("} finally {\n  // Even a deliberately retained fixture");
if (start < 0) throw Error("finalizer marker missing");
const body = source.slice(start + "} finally {".length, source.indexOf("\n}\n\nasync function createTargets", start));
const cleanup = new (Object.getPrototypeOf(async function(){}).constructor)("rm", "authPath", "evaluationRoot", "process", body);
const exists = async p => access(p).then(() => true, () => false);
for (const denied of [false, true]) {
 const root = await mkdtemp(join(scratch, "run-"));
 const codex = join(root, "codex");
 const auth = join(codex, "auth.json");
 await Promise.all([mkdir(codex), mkdir(join(root,"account")), mkdir(join(root,"targets"))]);
 await writeFile(auth, "synthetic fixture");
 if (denied) await chmod(codex, 0o500);
 const calls = [];
 let error = null;
 try { await cleanup(async (p,o) => {calls.push(p===auth?"auth":"root"); return rm(p,o)},auth,root,{env:{}}); }
 catch (e) {error = e.code;}
 console.log(JSON.stringify({denied,calls,error,rootRemains:await exists(root),accountRemains:await exists(join(root,"account")),targetsRemain:await exists(join(root,"targets"))}));
 if (denied) {
   let recursiveError=null;
   try {await rm(root,{recursive:true,force:true})}catch(e){recursiveError=e.code}
   console.log(JSON.stringify({recursiveControlError:recursiveError,accountRemains:await exists(join(root,"account")),targetsRemain:await exists(join(root,"targets"))}));
   await chmod(codex,0o700);
 }
 await rm(root,{recursive:true,force:true});
}
JS
node "$WORK/probe.mjs" "$PWD" "$WORK"

The probe restores permissions and removes its run fixtures. The probe script remains in $WORK for inspection.

Expected behavior

With retention unset, attempt removal of the owned run tree even if the earlier auth-file cleanup rejects. Preserve any unresolved cleanup failure. A denied subtree may remain unresolved, but its error should not prevent removing unrelated disposable siblings.

The lane documentation promises run-directory removal unless retention is requested. The lifecycle contract requires preserving successful child cleanup and unresolved owners.

Impact and scope

Bounded disposable filesystem state remains after a failed verifier cleanup. The probe demonstrates unnecessary retention of account and target directories. Restoring permissions and manually removing the owned tree worked in the probe. Natural incidence, retained bytes in actual evaluations, and behavior on other platforms are NOT_ENOUGH_INFO. No confidentiality incident or incorrect analysis result is claimed.

REA version

Source revision 3537e58c1ab29309f06eabf50bc52adba9817a0c, package version 6.7.0. The affected script is identical in tag rea-agents-6.7.0; release runtime behavior was not tested.

Runtime and platform

Executed boundary probe: Node.js v26.12.0, macOS 27.0.1, arm64. The repository pins Node 24.18.0; execution on that pinned runtime is NOT_ENOUGH_INFO.

Target and artifact context

Synthetic disposable account/codex/targets tree. The auth-named file contains inert text, with no credentials or real user configuration.

Interface

Other

Earliest failure stage

Cleanup, cancellation, or lifecycle

Relevant output or evidence

Observed probe output:

{"denied":false,"calls":["auth","root"],"error":null,"rootRemains":false,"accountRemains":false,"targetsRemain":false}
{"denied":true,"calls":["auth"],"error":"EACCES","rootRemains":true,"accountRemains":true,"targetsRemain":true}
{"recursiveControlError":"EACCES","accountRemains":false,"targetsRemain":false}

Owned path creation, the top-level try, and the sole finalizer establish the production caller path. package.json:229 exposes it through npm run verify:agent. The probe instruments removal-call order and uses actual filesystem errors, without mocking a provider or rm failure.

Typed outcome or recovery expected

Attempt remaining owned cleanup and report unresolved resources. Do not report successful removal of the denied subtree. Complete deletion is not established merely by attempting recursive removal.

Analysis or suspected cause

Source-derived: the first rejected await exits the finalizer before root removal. No other run-root removal or retry exists in this script.

Additional context

No full verifier run, build, live model request, real-provider observation, or stable-release runtime reproduction was performed. The permission denial is deliberately constructed; its occurrence during a normal evaluation remains NOT_ENOUGH_INFO. Nearby ADB-verifier and Windows package-verifier cleanup issues concern separate owners and failure contracts.

Originally created by @coygeek on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5095 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area MCP or agent integration ## Observed behavior SOURCE-ONLY caller tracing, with an executed extracted cleanup-boundary probe. No real Codex evaluation ran. The finalizer of `scripts/verify-agent-experience.mjs` awaits removal of `codex/auth.json` before attempting recursive removal of its disposable run directory. A permission denial on the first removal skips cleanup of the unaffected `account` and `targets` siblings, even when fixture retention is unset. Using the exact finalizer extracted from source and real filesystem operations, a synthetic codex directory with mode 0500 caused EACCES. Only the auth removal was attempted; account and targets remained. A separate recursive-removal control under the same denial removed both siblings while still reporting EACCES for the denied subtree. ## Reproduction From the source checkout at the revision below, run as an ordinary non-root user. This executes only the extracted finalizer and synthetic filesystem fixtures; it does not launch Codex or load REA build artifacts. ```sh WORK="$(mktemp -d)" cat > "$WORK/probe.mjs" <<'JS' import { readFile, mkdtemp, mkdir, writeFile, chmod, rm, access } from "node:fs/promises"; import { join } from "node:path"; const checkout = process.argv[2]; const scratch = process.argv[3]; const source = await readFile(join(checkout, "scripts/verify-agent-experience.mjs"), "utf8"); const start = source.indexOf("} finally {\n // Even a deliberately retained fixture"); if (start < 0) throw Error("finalizer marker missing"); const body = source.slice(start + "} finally {".length, source.indexOf("\n}\n\nasync function createTargets", start)); const cleanup = new (Object.getPrototypeOf(async function(){}).constructor)("rm", "authPath", "evaluationRoot", "process", body); const exists = async p => access(p).then(() => true, () => false); for (const denied of [false, true]) { const root = await mkdtemp(join(scratch, "run-")); const codex = join(root, "codex"); const auth = join(codex, "auth.json"); await Promise.all([mkdir(codex), mkdir(join(root,"account")), mkdir(join(root,"targets"))]); await writeFile(auth, "synthetic fixture"); if (denied) await chmod(codex, 0o500); const calls = []; let error = null; try { await cleanup(async (p,o) => {calls.push(p===auth?"auth":"root"); return rm(p,o)},auth,root,{env:{}}); } catch (e) {error = e.code;} console.log(JSON.stringify({denied,calls,error,rootRemains:await exists(root),accountRemains:await exists(join(root,"account")),targetsRemain:await exists(join(root,"targets"))})); if (denied) { let recursiveError=null; try {await rm(root,{recursive:true,force:true})}catch(e){recursiveError=e.code} console.log(JSON.stringify({recursiveControlError:recursiveError,accountRemains:await exists(join(root,"account")),targetsRemain:await exists(join(root,"targets"))})); await chmod(codex,0o700); } await rm(root,{recursive:true,force:true}); } JS node "$WORK/probe.mjs" "$PWD" "$WORK" ``` The probe restores permissions and removes its run fixtures. The probe script remains in `$WORK` for inspection. ## Expected behavior With retention unset, attempt removal of the owned run tree even if the earlier auth-file cleanup rejects. Preserve any unresolved cleanup failure. A denied subtree may remain unresolved, but its error should not prevent removing unrelated disposable siblings. [The lane documentation](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/docs/client-verification.md#L455-L464) promises run-directory removal unless retention is requested. [The lifecycle contract](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/docs/tool-design.md#L119-L134) requires preserving successful child cleanup and unresolved owners. ## Impact and scope Bounded disposable filesystem state remains after a failed verifier cleanup. The probe demonstrates unnecessary retention of account and target directories. Restoring permissions and manually removing the owned tree worked in the probe. Natural incidence, retained bytes in actual evaluations, and behavior on other platforms are NOT_ENOUGH_INFO. No confidentiality incident or incorrect analysis result is claimed. ## REA version Source revision `3537e58c1ab29309f06eabf50bc52adba9817a0c`, package version 6.7.0. The affected script is identical in tag `rea-agents-6.7.0`; release runtime behavior was not tested. ## Runtime and platform Executed boundary probe: Node.js v26.12.0, macOS 27.0.1, arm64. The repository pins Node 24.18.0; execution on that pinned runtime is NOT_ENOUGH_INFO. ## Target and artifact context Synthetic disposable account/codex/targets tree. The auth-named file contains inert text, with no credentials or real user configuration. ## Interface Other ## Earliest failure stage Cleanup, cancellation, or lifecycle ## Relevant output or evidence Observed probe output: ```jsonl {"denied":false,"calls":["auth","root"],"error":null,"rootRemains":false,"accountRemains":false,"targetsRemain":false} {"denied":true,"calls":["auth"],"error":"EACCES","rootRemains":true,"accountRemains":true,"targetsRemain":true} {"recursiveControlError":"EACCES","accountRemains":false,"targetsRemain":false} ``` [Owned path creation](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/scripts/verify-agent-experience.mjs#L41-L50), [the top-level try](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/scripts/verify-agent-experience.mjs#L71-L88), and [the sole finalizer](https://github.com/morluto/rea/blob/3537e58c1ab29309f06eabf50bc52adba9817a0c/scripts/verify-agent-experience.mjs#L394-L399) establish the production caller path. `package.json:229` exposes it through `npm run verify:agent`. The probe instruments removal-call order and uses actual filesystem errors, without mocking a provider or rm failure. ## Typed outcome or recovery expected Attempt remaining owned cleanup and report unresolved resources. Do not report successful removal of the denied subtree. Complete deletion is not established merely by attempting recursive removal. ## Analysis or suspected cause Source-derived: the first rejected await exits the finalizer before root removal. No other run-root removal or retry exists in this script. ## Additional context No full verifier run, build, live model request, real-provider observation, or stable-release runtime reproduction was performed. The permission denial is deliberately constructed; its occurrence during a normal evaluation remains NOT_ENOUGH_INFO. Nearby ADB-verifier and Windows package-verifier cleanup issues concern separate owners and failure contracts.

bsakaguchi closed this issue

2026-10-31 00:36:59 +00:00

bsakaguchi commented

2026-10-31 00:37:00 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 19, 2026):

Agreed — a cleanup path that lets one EACCES on auth.json abort the whole finalizer leaks the account and targets siblings, which is worse than the original failure. The control case in your probe shows recursive removal with a per-subtree error report is achievable; restructuring the finalizer so each sibling removal is independent (and best-effort with aggregated errors) would match the lifecycle contract's intent.

@XinwangTao commented on GitHub (Oct 19, 2026): Agreed — a cleanup path that lets one EACCES on auth.json abort the whole finalizer leaks the account and targets siblings, which is worse than the original failure. The control case in your probe shows recursive removal with a per-subtree error report is achievable; restructuring the finalizer so each sibling removal is independent (and best-effort with aggregated errors) would match the lifecycle contract's intent.

bsakaguchi referenced this issue

2026-10-31 00:38:43 +00:00

[GH-ISSUE #5411] [Bug] A zero-byte raw/NN_s.txt (session with no response) makes inspect_web_network_capture reject the whole SAZ archive #3279

bsakaguchi referenced this issue

2026-10-31 01:09:48 +00:00

[PR #3584] [MERGED] test(perf): measure genuine SAZ selection and retained recovery #5836