morluto/rea
1
0
Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00
Code Issues 1.3k Projects Packages Wiki Activity

[PR #1971] feat(evaluation): run agent evaluation against Claude Code and Gemini CLI #4943

New issue
Open
opened 2026-10-31 00:57:13 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-31 00:57:13 +00:00
Owner
Copy link

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1971
Author: @Nick-800
Created: 10/11/2026
Status: 🔄 Open

Base: main ← Head: feat/agent-eval-clients

📝 Commits (10+)

  • 95cbdef feat(evaluation): run agent evaluation against Claude Code and Gemini CLI
  • 7e5869d fix(evaluation): own agent client processes and credentials
  • 05cdc28 fix(evaluation): preserve client authority and isolated permissions
  • a84907b fix(evaluation): retain verified process ownership and selected authentication
  • b79e445 fix(eval): admit only unambiguous terminal Codex routing
  • a3d2ea9 fix(evaluation): preserve client identity and process ownership
  • a1576a3 Merge commit 'ebaaa79ca0' into HEAD
  • bab7827 fix(evaluation): preserve client command and evidence identity
  • 1ce664c Merge branch 'main' into feat/agent-eval-clients
  • b17b9b6 Merge branch 'main' into feat/agent-eval-clients

📊 Changes

20 files changed (+3174 additions, -470 deletions)

View changed files

📝 docs/client-verification.md (+68 -5)
📝 docs/testing.md (+5 -3)
➕ scripts/lib/agent-eval-clients.d.mts (+53 -0)
➕ scripts/lib/agent-eval-clients.mjs (+394 -0)
📝 scripts/lib/agent-evaluation-commands.mjs (+16 -32)
📝 scripts/verify-agent-experience.mjs (+85 -74)
📝 tests/boundary/mcp/agentKnownAnswerMcp.test.ts (+12 -1)
📝 tests/boundary/process/agentEvaluationCleanup.test.ts (+1 -1)
📝 tests/boundary/process/agentEvaluationCommands.test.ts (+10 -9)
📝 tests/evaluation/model-evals/AgentEval.ts (+55 -128)
📝 tests/evaluation/model-evals/AgentEvaluationReport.ts (+3 -0)
📝 tests/evaluation/model-evals/AgentEvaluationScenarios.ts (+29 -5)
➕ tests/evaluation/model-evals/AgentTranscript.ts (+786 -0)
📝 tests/evaluation/model-evals/KnownAnswerEvaluation.ts (+10 -209)
📝 tests/evaluation/model-evals/agentEval.test.ts (+15 -1)
📝 tests/evaluation/model-evals/agentEvaluationReport.test.ts (+15 -0)
📝 tests/evaluation/model-evals/agentEvaluationScenarios.test.ts (+60 -1)
➕ tests/evaluation/model-evals/agentTranscript.test.ts (+1098 -0)
📝 tests/evaluation/model-evals/knownAnswerEvaluation.test.ts (+22 -1)
➕ tests/process-global/agentEvalClients.test.ts (+437 -0)

📄 Description

Summary

npm run verify:agent can now run the same scenarios, fixture claims and gates through Claude Code or Gemini CLI as well as Codex. Choose the client with REA_AGENT_EVAL_CLIENT=codex|claude-code|gemini-cli. Codex is still the default.

Fixes #1932

Problem and expected behavior

verify:agent was the only lane that measures REA's agent experience against a real model, and it only drove Codex. The verify:claude-client / verify:gemini-client lanes use a local model fixture, so they check integration, not investigation quality. Parsing was Codex-shaped in two places: evaluateCodexEvents and the known-answer grader's completedCall / deliveredCodexEvidence. That meant a regression that only shows up with another client went unmeasured.

Change and scope

  • One client-neutral transcript (tests/evaluation/model-evals/AgentTranscript.ts, beside the other development-owned evaluation modules main moved there; the verifier loads it from source like the fixtures and scenarios). It owns result parsing for every client: failure, error-code certainty (no code is chosen among duplicate JSON members) and the successful content used for Evidence authentication are derived once per call. Each client's events are parsed once into MCP calls, retained results, the final answer and token usage. parseCodexEvents, parseClaudeCodeEvents and parseGeminiCliEvents each follow their client's own grammar:
    • Codex item.completed / mcp_tool_call.
    • Claude tool_use / tool_result, plus the raw MCP tool_use_result that keeps structuredContent even when model-visible content was spilled to a file. The raw detail is attached only when one result block makes it unambiguous; batched parallel results each stand on their own block content.
    • Gemini tool_use / tool_result, where only joined MCP text is kept.
  • Both graders consume the transcript. evaluateAgentTranscript (tests/evaluation/model-evals/AgentEval.ts, replacing CodexAgentEval.ts) and evaluateKnownAnswers read the shared record; KnownAnswerEvaluation authenticates Evidence from it instead of parsing Codex items itself. The Codex-only interpretation is removed; main's grader tests keep their event shapes through a test-local evaluateCodexEvents wrapper.
  • Behavior changes:
    • A completed call is delivered only when the client retained one consistent result record, and any non-null result error, even a bare string, is a failure that delivers no Evidence. A Codex item.completed without result/output (or with a non-object one), and any result whose structuredContent and structured_content contradict each other, keeps its routing facts but is not a completed workflow step, so two such records cannot satisfy the native open_binary/close_binary sequence.
    • If a client reports the same call twice with conflicting results, the call is a failure with no certain error code: it establishes neither Evidence delivery nor an expected refusal. Previously the metric path kept the first report and the grader saw both.
    • The transcript carries the client's own failures (clientFailures): Claude Code permission_denials entries and is_error results, a Gemini CLI result status other than success, a Gemini tool result that failed without REA's own MCP tool error type, and a Claude Code error result without REA's own error envelope (a policy refusal, an unavailable or unregistered server, a client exception). REA's own error results stay ordinary call failures, and an error code is exposed only when every simultaneously present error record agrees on it. Any entry fails agentEvaluationPassed even when the client still returned an answer, so a refused dontAsk request cannot pass a scenario on an "unavailable" explanation.
    • REA server attribution must be explicit, as on main since 443c7aff; this branch's earlier server-less legacy Codex routing is gone.
  • Client runner (scripts/lib/agent-eval-clients.mjs). Each client gets an isolated profile configured through public rea setup --client <id> (dry run, then --yes), and only its authentication is copied in. Processes run through main's shared createAgentEvaluationCommands owner, generalized from runCodex to runClient/readCommandVersion over the selected client's launch; a selected JavaScript entry point (cli.js, bundle/gemini.js) runs under the verifier's Node for the preflight and every scenario. Each client names its credential destination before preparation, so a run that fails earlier still removes it. The version preflight receives the client's non-authentication containment controls (no self-update, no nonessential traffic, no relaunch) without credentials.
    • Claude Code: limited to read-only built-ins (Read,Grep,Glob,Skill,ToolSearch,WaitForMcpServers) in dontAsk mode. claude.ai account connectors are disabled. ANTHROPIC_AUTH_TOKEN is forwarded beside ANTHROPIC_API_KEY for gateway authentication. An empty CLAUDE_CONFIG_DIR counts as unset, as REA's own client resolver reads it.
    • Gemini CLI: runs headless in its default approval mode with a policy file. The policy also allows activate_skill so the REA skill can load. The caller's settings file is read through the owning client-configuration parser, so an empty placeholder counts as no settings.
    • Codex: arguments are unchanged.
  • Tool allow-list. Claude Code and Gemini CLI receive a permission policy taken from each tool's audited effects: a tool that may run foreign code, change a device, discard caller state, launch a process, or reach the network is refused unless the scenario approves it by name. Every scenario additionally approves by name the tools its own contract requires (expected first tool and required subsequence, beside explicit approvals), so the JavaScript scenarios may run analyze_javascript_application, which main's audit marks as launching a worker process. The MCP hints derived from those effects are not consulted, so a hint edit cannot change permissions.
  • Token usage includes cached input for every client. Claude's cache-creation and cache-read counts are added to its uncached input.
  • Report schema 5 adds client, clientCommand and clientVersion, names the setup method rea_setup_<client>, and records each scenario's clientFailures with their total in clientFailureCount. Codex reports keep codex, codexVersion and isolatedCodexHome.
  • Not included: no new scenarios, no gate or threshold changes, and no CI requirement for paid runs. The lane stays manual.

Contract and boundary impact

  • Semantic owner and earliest changed stage: transcript interpretation at the client boundary (tests/evaluation/model-evals/AgentTranscript.ts).
  • CLI and MCP/tool-catalog contract: none.
  • Provider, bridge, target-format, or platform compatibility: none. This is verifier-only. The Gemini lane needs a POSIX host for the same reasons as verify:gemini-client.
  • Evidence, artifact, provenance, or reconstruction contract: grading semantics are unchanged, except that conflicting duplicate reports of one call now fail closed.
  • Process execution, authorization, cleanup, or containment impact:
    • Copied credential files (auth.json, .credentials.json, .gemini/oauth_creds.json) are removed in finally, as before.
    • Client, provider and REA environment variables are stripped, except the selected client's API key and gateway endpoint (OPENAI_BASE_URL, ANTHROPIC_BASE_URL or GOOGLE_GEMINI_BASE_URL).
    • When Gemini CLI keeps an API key in the OS keychain, only the caller's selected auth type is carried over, and the keychain credential is used where it is.
  • Generated metadata, package, or installation impact: none. The evaluator stays development-owned under tests/evaluation/model-evals/ and is not part of the shipped runtime, so knip.json is unchanged.

Evidence and regression coverage

  • Tests added or updated:
    • tests/evaluation/model-evals/agentTranscript.test.ts adds Claude/Gemini parser tests on shapes trimmed from real transcripts. They cover call/result pairing, client-reported failures with REA error codes, non-rea servers, subagent text, token totals, missing usage, permission denials and terminal client errors, delivery requiring a retained result (missing, non-object and empty-object Codex results), duplicate JSON error members, conflicting duplicate reports, and reused call IDs.
    • agentEval.test.ts (main's codexAgentEval.test.ts through the wrapper), knownAnswerEvaluation.test.ts, agentEvaluationScenarios.test.ts and tests/boundary/mcp/agentKnownAnswerMcp.test.ts (main's Notes cases included) go through parseCodexEvents → evaluateAgentTranscript. Two of main's lifecycle-accounting cases now supply result: {} on their completed steps, matching the delivery contract above.
    • agentEvaluationReport.test.ts rejects a result with client failures; agentEvaluationScenarios.test.ts covers scenario approvals and grades an unconfirmed open or close as failed.
    • tests/process-global/agentEvalClients.test.ts covers the effects-based allow-list against TOOL_CONTRACTS[].effects, the JavaScript entry launch, the Claude auth token, and credential destinations; tests/boundary/process/agentEvaluationCommands.test.ts drives the generalized command owner.
  • Base reproduction or other evidence: real runs below.
  • Remaining proof gaps:
    • Codex was not run (not installed here). Codex behavior is covered by the existing tests.
    • Gemini's asar and missing-target scenarios did not complete because of the free-tier daily quota. The activate_skill policy fix has not been checked on a real run.
    • Gemini MCP failures carry a generic client message instead of REA's error record. Validation failures there are counted from the call's arguments against the REA schema.

Validation performed

Re-fit onto main at ff6d96bce (2026-10-25, merge b04661d54, follow-ups 07f44cae8, 5763d60ef, 69c22a621, 0963d13c2, d2e1a7aa5, 86ae9b4bf, ed2e3ba1b, 4dbebd70c, 38a51aa91, 6d2d2c328 and d00eca532 for the Greptile and Codex findings on the pushed heads), then re-fit again onto main at d6f88becd (2026-10-27, merge f36dcbd4e; one conflict, in the verifier's import block, where main now loads parseSkillManifestMetadata through importVerifierSource; follow-up fbd8e10fc for the Greptile finding on that head), Linux, Node 24.22.1 from .nvmrc, npm ci, npm run build:cached:

  • npm run test:focused on agentEval, knownAnswerEvaluation, agentEvaluationReport, agentTranscript, agentEvaluationScenarios, agentEvalClients, agentKnownAnswerMcp, agentEvaluationCommands and agentEvaluationCleanup: 9 files, 170 tests pass (the last two drive verify-agent-experience.mjs with fixture clients, including the early-failure credential cleanup).
  • Every test file under tests/evaluation/: 21 files, 450 passed, 1 skipped (pre-existing).
  • node scripts/run-typescript.mjs --noEmit, npm run lint (no errors; the complexity/length warnings on the verifier and transcript parser are of the same kind main already carries), oxfmt --check ., npx knip: clean.
  • On fbd8e10fc: node scripts/run-typescript.mjs --noEmit, oxlint on the parser and its test, the pre-push check:fast, and the six evaluation suites that import the transcript parser: 166 tests pass.
  • On the merged head f36dcbd4e: npm run build:cached, node scripts/run-typescript.mjs --noEmit, oxfmt --check and oxlint on the lane's files, the pre-push check:fast (turbo typecheck and lint), and the same nine focused suites: 196 tests pass. The first agentEvaluationCleanup case timed out once on a cold cache right after the rebuild and passed on re-run (4/4). Not re-run on this head: every file under tests/evaluation/, oxfmt --check . over the whole repo, npx knip.
  • Not run: a real Claude Code, Gemini CLI or Codex session, npm run docs:check, Windows or macOS. The permission_denials and tool_use_result shapes are those scripts/verify-claude-client.mjs already asserts; the Gemini result.status shape is the one recorded in agentTranscript.test.ts.

Earlier evidence, recorded on the pre-merge head b17b9b6ce before the delivery and client-failure gates existed:

  • Real Claude Code 2.1.296 (REA_AGENT_EVAL_CLIENT=claude-code REA_AGENT_EVAL_SCENARIOS=asar,native,missing-target):
    • asar: passed. All configured fixture claims were correct, against authenticated delivered Evidence.
    • native: failed correctFirstTool. Claude called binary_session before open_binary. The lifecycle subsequence was met, with no repeats or validation failures.
    • missing-target: failed targetClarificationPassed. Claude correctly asked for the target, but wrapped the required JSON in prose and a code fence.
    • Both failures are model behavior measured by the shared gates, which is what #1932 asks for.
  • Real Gemini CLI 0.63.0 (gemini-3-flash-preview, API-key auth), run before the activate_skill fix:
    • native: open_binary was the first call, followed by 13 calls, 1 repeat and 1 validation failure (~1.6M input tokens).
    • asar: Evidence was parsed from Gemini's text-only results, then the run stopped at the daily quota.
    • missing-target: stopped at the daily quota.

Compatibility, safety, and release

  • Breaking changes or migration steps: report schemaVersion 4 → 5, which is additive. Codex reports keep their existing fields.
  • Real Hopper/Ghidra, browser, or OS coverage: Linux only. No provider changes.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: credentials are copied with mode 0600 and removed after the run. No user configuration is copied beyond auth (and Gemini's selected auth type). Account-level claude.ai connectors are disabled in the isolated Claude profile.

Author readiness

Human owner: @Nick-800

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1971 **Author:** [@Nick-800](https://github.com/Nick-800) **Created:** 10/11/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/agent-eval-clients` --- ### 📝 Commits (10+) - [`95cbdef`](https://github.com/morluto/rea/commit/95cbdef60b7d3a97cec133de656430e9efe0f458) feat(evaluation): run agent evaluation against Claude Code and Gemini CLI - [`7e5869d`](https://github.com/morluto/rea/commit/7e5869d8e62642e7bd191cee8cbbd0db0cb2b503) fix(evaluation): own agent client processes and credentials - [`05cdc28`](https://github.com/morluto/rea/commit/05cdc283c080cf750ba259fca55a8f80f3dc1bd0) fix(evaluation): preserve client authority and isolated permissions - [`a84907b`](https://github.com/morluto/rea/commit/a84907bf06b9fb5cb23d389e3c0cea3bbd64aecb) fix(evaluation): retain verified process ownership and selected authentication - [`b79e445`](https://github.com/morluto/rea/commit/b79e44522c64304de5047b5d44cebae2ccc51a15) fix(eval): admit only unambiguous terminal Codex routing - [`a3d2ea9`](https://github.com/morluto/rea/commit/a3d2ea930d4bb0d83974444e558eb58e5f7ecd18) fix(evaluation): preserve client identity and process ownership - [`a1576a3`](https://github.com/morluto/rea/commit/a1576a34c621ee45137156f1c286a48ceebba2ee) Merge commit 'ebaaa79ca02bde4a73013a2a9a781d0b07a57ee4' into HEAD - [`bab7827`](https://github.com/morluto/rea/commit/bab7827263910a1934b15a301c75714133e90ba4) fix(evaluation): preserve client command and evidence identity - [`1ce664c`](https://github.com/morluto/rea/commit/1ce664c6558b14410bda6e13f0ef8da2c1badd2b) Merge branch 'main' into feat/agent-eval-clients - [`b17b9b6`](https://github.com/morluto/rea/commit/b17b9b6cead74c474d5c802890c7bd9a9ee2978c) Merge branch 'main' into feat/agent-eval-clients ### 📊 Changes **20 files changed** (+3174 additions, -470 deletions) <details> <summary>View changed files</summary> 📝 `docs/client-verification.md` (+68 -5) 📝 `docs/testing.md` (+5 -3) ➕ `scripts/lib/agent-eval-clients.d.mts` (+53 -0) ➕ `scripts/lib/agent-eval-clients.mjs` (+394 -0) 📝 `scripts/lib/agent-evaluation-commands.mjs` (+16 -32) 📝 `scripts/verify-agent-experience.mjs` (+85 -74) 📝 `tests/boundary/mcp/agentKnownAnswerMcp.test.ts` (+12 -1) 📝 `tests/boundary/process/agentEvaluationCleanup.test.ts` (+1 -1) 📝 `tests/boundary/process/agentEvaluationCommands.test.ts` (+10 -9) 📝 `tests/evaluation/model-evals/AgentEval.ts` (+55 -128) 📝 `tests/evaluation/model-evals/AgentEvaluationReport.ts` (+3 -0) 📝 `tests/evaluation/model-evals/AgentEvaluationScenarios.ts` (+29 -5) ➕ `tests/evaluation/model-evals/AgentTranscript.ts` (+786 -0) 📝 `tests/evaluation/model-evals/KnownAnswerEvaluation.ts` (+10 -209) 📝 `tests/evaluation/model-evals/agentEval.test.ts` (+15 -1) 📝 `tests/evaluation/model-evals/agentEvaluationReport.test.ts` (+15 -0) 📝 `tests/evaluation/model-evals/agentEvaluationScenarios.test.ts` (+60 -1) ➕ `tests/evaluation/model-evals/agentTranscript.test.ts` (+1098 -0) 📝 `tests/evaluation/model-evals/knownAnswerEvaluation.test.ts` (+22 -1) ➕ `tests/process-global/agentEvalClients.test.ts` (+437 -0) </details> ### 📄 Description ## Summary `npm run verify:agent` can now run the same scenarios, fixture claims and gates through Claude Code or Gemini CLI as well as Codex. Choose the client with `REA_AGENT_EVAL_CLIENT=codex|claude-code|gemini-cli`. Codex is still the default. Fixes #1932 ## Problem and expected behavior `verify:agent` was the only lane that measures REA's agent experience against a real model, and it only drove Codex. The `verify:claude-client` / `verify:gemini-client` lanes use a local model fixture, so they check integration, not investigation quality. Parsing was Codex-shaped in two places: `evaluateCodexEvents` and the known-answer grader's `completedCall` / `deliveredCodexEvidence`. That meant a regression that only shows up with another client went unmeasured. ## Change and scope - **One client-neutral transcript** (`tests/evaluation/model-evals/AgentTranscript.ts`, beside the other development-owned evaluation modules main moved there; the verifier loads it from source like the fixtures and scenarios). It owns result parsing for every client: failure, error-code certainty (no code is chosen among duplicate JSON members) and the successful content used for Evidence authentication are derived once per call. Each client's events are parsed once into MCP calls, retained results, the final answer and token usage. `parseCodexEvents`, `parseClaudeCodeEvents` and `parseGeminiCliEvents` each follow their client's own grammar: - Codex `item.completed` / `mcp_tool_call`. - Claude `tool_use` / `tool_result`, plus the raw MCP `tool_use_result` that keeps `structuredContent` even when model-visible content was spilled to a file. The raw detail is attached only when one result block makes it unambiguous; batched parallel results each stand on their own block content. - Gemini `tool_use` / `tool_result`, where only joined MCP text is kept. - **Both graders consume the transcript.** `evaluateAgentTranscript` (`tests/evaluation/model-evals/AgentEval.ts`, replacing `CodexAgentEval.ts`) and `evaluateKnownAnswers` read the shared record; `KnownAnswerEvaluation` authenticates Evidence from it instead of parsing Codex items itself. The Codex-only interpretation is removed; main's grader tests keep their event shapes through a test-local `evaluateCodexEvents` wrapper. - **Behavior changes:** - A completed call is delivered only when the client retained one consistent result record, and any non-null result error, even a bare string, is a failure that delivers no Evidence. A Codex `item.completed` without `result`/`output` (or with a non-object one), and any result whose `structuredContent` and `structured_content` contradict each other, keeps its routing facts but is not a completed workflow step, so two such records cannot satisfy the native `open_binary`/`close_binary` sequence. - If a client reports the same call twice with conflicting results, the call is a failure with no certain error code: it establishes neither Evidence delivery nor an expected refusal. Previously the metric path kept the first report and the grader saw both. - The transcript carries the client's own failures (`clientFailures`): Claude Code `permission_denials` entries and `is_error` results, a Gemini CLI result status other than `success`, a Gemini tool result that failed without REA's own MCP tool error type, and a Claude Code error result without REA's own error envelope (a policy refusal, an unavailable or unregistered server, a client exception). REA's own error results stay ordinary call failures, and an error code is exposed only when every simultaneously present error record agrees on it. Any entry fails `agentEvaluationPassed` even when the client still returned an answer, so a refused `dontAsk` request cannot pass a scenario on an "unavailable" explanation. - REA server attribution must be explicit, as on main since 443c7aff; this branch's earlier server-less legacy Codex routing is gone. - **Client runner** (`scripts/lib/agent-eval-clients.mjs`). Each client gets an isolated profile configured through public `rea setup --client <id>` (dry run, then `--yes`), and only its authentication is copied in. Processes run through main's shared `createAgentEvaluationCommands` owner, generalized from `runCodex` to `runClient`/`readCommandVersion` over the selected client's `launch`; a selected JavaScript entry point (`cli.js`, `bundle/gemini.js`) runs under the verifier's Node for the preflight and every scenario. Each client names its credential destination before preparation, so a run that fails earlier still removes it. The version preflight receives the client's non-authentication containment controls (no self-update, no nonessential traffic, no relaunch) without credentials. - Claude Code: limited to read-only built-ins (`Read,Grep,Glob,Skill,ToolSearch,WaitForMcpServers`) in `dontAsk` mode. claude.ai account connectors are disabled. `ANTHROPIC_AUTH_TOKEN` is forwarded beside `ANTHROPIC_API_KEY` for gateway authentication. An empty `CLAUDE_CONFIG_DIR` counts as unset, as REA's own client resolver reads it. - Gemini CLI: runs headless in its default approval mode with a policy file. The policy also allows `activate_skill` so the REA skill can load. The caller's settings file is read through the owning client-configuration parser, so an empty placeholder counts as no settings. - Codex: arguments are unchanged. - **Tool allow-list.** Claude Code and Gemini CLI receive a permission policy taken from each tool's audited `effects`: a tool that may run foreign code, change a device, discard caller state, launch a process, or reach the network is refused unless the scenario approves it by name. Every scenario additionally approves by name the tools its own contract requires (expected first tool and required subsequence, beside explicit approvals), so the JavaScript scenarios may run `analyze_javascript_application`, which main's audit marks as launching a worker process. The MCP hints derived from those effects are not consulted, so a hint edit cannot change permissions. - **Token usage** includes cached input for every client. Claude's cache-creation and cache-read counts are added to its uncached input. - **Report schema 5** adds `client`, `clientCommand` and `clientVersion`, names the setup method `rea_setup_<client>`, and records each scenario's `clientFailures` with their total in `clientFailureCount`. Codex reports keep `codex`, `codexVersion` and `isolatedCodexHome`. - **Not included:** no new scenarios, no gate or threshold changes, and no CI requirement for paid runs. The lane stays manual. ## Contract and boundary impact - Semantic owner and earliest changed stage: transcript interpretation at the client boundary (`tests/evaluation/model-evals/AgentTranscript.ts`). - CLI and MCP/tool-catalog contract: none. - Provider, bridge, target-format, or platform compatibility: none. This is verifier-only. The Gemini lane needs a POSIX host for the same reasons as `verify:gemini-client`. - Evidence, artifact, provenance, or reconstruction contract: grading semantics are unchanged, except that conflicting duplicate reports of one call now fail closed. - Process execution, authorization, cleanup, or containment impact: - Copied credential files (`auth.json`, `.credentials.json`, `.gemini/oauth_creds.json`) are removed in `finally`, as before. - Client, provider and REA environment variables are stripped, except the selected client's API key and gateway endpoint (`OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL` or `GOOGLE_GEMINI_BASE_URL`). - When Gemini CLI keeps an API key in the OS keychain, only the caller's selected auth type is carried over, and the keychain credential is used where it is. - Generated metadata, package, or installation impact: none. The evaluator stays development-owned under `tests/evaluation/model-evals/` and is not part of the shipped runtime, so `knip.json` is unchanged. ## Evidence and regression coverage - Tests added or updated: - `tests/evaluation/model-evals/agentTranscript.test.ts` adds Claude/Gemini parser tests on shapes trimmed from real transcripts. They cover call/result pairing, client-reported failures with REA error codes, non-`rea` servers, subagent text, token totals, missing usage, permission denials and terminal client errors, delivery requiring a retained result (missing, non-object and empty-object Codex results), duplicate JSON error members, conflicting duplicate reports, and reused call IDs. - `agentEval.test.ts` (main's `codexAgentEval.test.ts` through the wrapper), `knownAnswerEvaluation.test.ts`, `agentEvaluationScenarios.test.ts` and `tests/boundary/mcp/agentKnownAnswerMcp.test.ts` (main's Notes cases included) go through `parseCodexEvents` → `evaluateAgentTranscript`. Two of main's lifecycle-accounting cases now supply `result: {}` on their completed steps, matching the delivery contract above. - `agentEvaluationReport.test.ts` rejects a result with client failures; `agentEvaluationScenarios.test.ts` covers scenario approvals and grades an unconfirmed open or close as failed. - `tests/process-global/agentEvalClients.test.ts` covers the effects-based allow-list against `TOOL_CONTRACTS[].effects`, the JavaScript entry launch, the Claude auth token, and credential destinations; `tests/boundary/process/agentEvaluationCommands.test.ts` drives the generalized command owner. - Base reproduction or other evidence: real runs below. - Remaining proof gaps: - Codex was not run (not installed here). Codex behavior is covered by the existing tests. - Gemini's `asar` and `missing-target` scenarios did not complete because of the free-tier daily quota. The `activate_skill` policy fix has not been checked on a real run. - Gemini MCP failures carry a generic client message instead of REA's error record. Validation failures there are counted from the call's arguments against the REA schema. ## Validation performed Re-fit onto main at ff6d96bce (2026-10-25, merge b04661d54, follow-ups 07f44cae8, 5763d60ef, 69c22a621, 0963d13c2, d2e1a7aa5, 86ae9b4bf, ed2e3ba1b, 4dbebd70c, 38a51aa91, 6d2d2c328 and d00eca532 for the Greptile and Codex findings on the pushed heads), then re-fit again onto main at d6f88becd (2026-10-27, merge f36dcbd4e; one conflict, in the verifier's import block, where main now loads `parseSkillManifestMetadata` through `importVerifierSource`; follow-up fbd8e10fc for the Greptile finding on that head), Linux, Node 24.22.1 from `.nvmrc`, `npm ci`, `npm run build:cached`: - `npm run test:focused` on `agentEval`, `knownAnswerEvaluation`, `agentEvaluationReport`, `agentTranscript`, `agentEvaluationScenarios`, `agentEvalClients`, `agentKnownAnswerMcp`, `agentEvaluationCommands` and `agentEvaluationCleanup`: 9 files, 170 tests pass (the last two drive `verify-agent-experience.mjs` with fixture clients, including the early-failure credential cleanup). - Every test file under `tests/evaluation/`: 21 files, 450 passed, 1 skipped (pre-existing). - `node scripts/run-typescript.mjs --noEmit`, `npm run lint` (no errors; the complexity/length warnings on the verifier and transcript parser are of the same kind main already carries), `oxfmt --check .`, `npx knip`: clean. - On fbd8e10fc: `node scripts/run-typescript.mjs --noEmit`, `oxlint` on the parser and its test, the pre-push `check:fast`, and the six evaluation suites that import the transcript parser: 166 tests pass. - On the merged head f36dcbd4e: `npm run build:cached`, `node scripts/run-typescript.mjs --noEmit`, `oxfmt --check` and `oxlint` on the lane's files, the pre-push `check:fast` (turbo typecheck and lint), and the same nine focused suites: 196 tests pass. The first `agentEvaluationCleanup` case timed out once on a cold cache right after the rebuild and passed on re-run (4/4). Not re-run on this head: every file under `tests/evaluation/`, `oxfmt --check .` over the whole repo, `npx knip`. - Not run: a real Claude Code, Gemini CLI or Codex session, `npm run docs:check`, Windows or macOS. The `permission_denials` and `tool_use_result` shapes are those `scripts/verify-claude-client.mjs` already asserts; the Gemini `result.status` shape is the one recorded in `agentTranscript.test.ts`. Earlier evidence, recorded on the pre-merge head b17b9b6ce before the delivery and client-failure gates existed: - **Real Claude Code 2.1.296** (`REA_AGENT_EVAL_CLIENT=claude-code REA_AGENT_EVAL_SCENARIOS=asar,native,missing-target`): - `asar`: **passed**. All configured fixture claims were correct, against authenticated delivered Evidence. - `native`: failed `correctFirstTool`. Claude called `binary_session` before `open_binary`. The lifecycle subsequence was met, with no repeats or validation failures. - `missing-target`: failed `targetClarificationPassed`. Claude correctly asked for the target, but wrapped the required JSON in prose and a code fence. - Both failures are model behavior measured by the shared gates, which is what #1932 asks for. - **Real Gemini CLI 0.63.0** (`gemini-3-flash-preview`, API-key auth), run before the `activate_skill` fix: - `native`: `open_binary` was the first call, followed by 13 calls, 1 repeat and 1 validation failure (~1.6M input tokens). - `asar`: Evidence was parsed from Gemini's text-only results, then the run stopped at the daily quota. - `missing-target`: stopped at the daily quota. ## Compatibility, safety, and release - Breaking changes or migration steps: report `schemaVersion` 4 → 5, which is additive. Codex reports keep their existing fields. - Real Hopper/Ghidra, browser, or OS coverage: Linux only. No provider changes. - Package or release metadata impact: none. - Security, privacy, process, or containment review: credentials are copied with mode 0600 and removed after the run. No user configuration is copied beyond auth (and Gemini's selected auth type). Account-level claude.ai connectors are disabled in the isolated Claude profile. ## Author readiness Human owner: @Nick-800 - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
codex/procedure-reference-stacked
release-please--branches--main--components--rea-agents
release-please--branches--main--components--rea-agents--release-notes
codex/kuna-independent-retry
codex/byte-search-overlay-coverage
codex/open-failure-target-receipt
codex/kuna-xref-evidence
codex/ghidra-ambiguous-procedure-recovery
codex/function-source-bundle
codex/trace-function-integration-prototype
codex/snapshot-source-closure
codex/recorded-evidence-array-pages
codex/evidence-recovery-redesign
codex/trace-feature-address-scope
codex/call-discovery-scope
codex/ghidra-literal-candidate-evidence
codex/decompile-directory-independent-retry
codex/retry-provenance-base
codex/value-flow-capability-experiment
codex/ghidra-containing-instruction
cursor/readme-discord-button-ef9f
dependabot/github_actions/actions/setup-node-7.1.0
dependabot/github_actions/actions/upload-artifact-7.0.2
dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df
codex/kuna-1989
fix/8559-native-ui-fixture-ownership
fix/8555-go-verifier-mcp-ownership
codex/provider-capability-recovery
fix/8642-rizin-target-stdin
codex/ghidra-memory-effect-scope
codex/ghidra-annotation-entry-selector
codex/ghidra-readonly-recovery
codex/ghidra-regex-verifier-recovery
codex/evidence-discovery-view-guidance
fix/8545-bulk-append-overflow
fix/8428-navigation-refusal-alternatives
codex/address-context-recovery
codex/matching-member-limitations
codex/unknown-scope-contract
codex/evidence-view-guidance
feat/8373-native-trace-views
codex/ghidra-snapshot-verifier-binding
codex/elf-export-visibility
codex/evidence-view-recovery
codex/native-reference-cli-help
codex/provider-candidate-observation-scope
codex/kuna-session-verification
codex/ghidra-switch-lifetime-guidance
codex/pseudocode-paging-discovery
codex/clarify-evidence-view-support
codex/kuna-xrefs-selector-recovery
codex/context-capacity-recovery
codex/evidence-closure-large-links
codex/same-function-evidence-identity
codex/ghidra-verifier-inspected-jdk
codex/audit-guided-workflows
codex/crash-prompt-retained-dossiers
codex/analysis-view-error-recovery
codex/kuna-verifier-target-selection
codex/ghidra-selected-project-call-coverage
codex/docs-native-analysis-scope
codex/ghidra-java-socket-boundary
codex/ghidra-retained-snapshot-verifier
codex/ghidra-search-inventory-facts
fix/8284-zero-argument-parameter-bindings
fix/8198-ui-action-handler-offset
fix/8257-elf-entry-label-identity
fix/8304-deferred-output-schema-projection
fix/8277-call-path-computed-target-uncertainty
fix/pr-8089-conflicts
fix/ci-static-duplicate-export
dependabot/npm_and_yarn/lint-staged-17.6.0
dependabot/github_actions/actions/download-artifact-8.0.2
dependabot/npm_and_yarn/write-file-atomic-8.0.0
dependabot/npm_and_yarn/canonicalize-5.1.0
dependabot/github_actions/actions/cache/restore-6.1.0
dependabot/npm_and_yarn/toon-format/toon-4.4.0
dependabot/github_actions/actions/deploy-pages-5.0.1
codex/review-fixes-pr-2403
codex/review-fixes-pr-2400
codex/process-terminal-width
codex/test-brittleness-cleanup
verify/7167-browser-connect-macos
fix/7167-browser-connect
feat/native-matching-allowance-setting
feat/browser-scenario-dry-run
test/managed-caller-parameters-20261024
codex/parser-fortythird-boundaries
fix/6926-pinvoke-platform-decorations-20261023
fix/6889-har-body-encoding-20261023
fix/6888-sqlite-schema-budget-20261023
fix/dependency-security-gates-6708
cursor/evidence-overwrite-rebase-b4ac
feat/4057-verify-native-exports
fix/windows-acl-observer-overhead-20261022
cursor/erlang-beam-restack-4476
cursor/pe-certificate-review-bf32
fix/6116-javascript-application-fs-errors
fix/6061-unmapped-runtime-script-unknown
feat/browser-scenario-summary
fix/pe-certificates
codex/verify-3407
cursor/rebase-5699
cursor/rebase-5352
cursor/rebase-5600
cursor/fix-cli-schema-omission-10a8
codex/audit-hopper-shutdown-reasons-20261021
fix/demangle-swift-omitted-record
fix/nib-boolean-value-types
cursor/cancellation-partial-observation-a5ac
fix/ib-object-limit-reasons
fix/electron-fuse-limit-advice
fix/kuna-doctor-dangling-cache
fix/fixed-budget-refusal-advice
fix/js-ipc-listener-dedup
fix/js-return-member-provenance
fix/dmg-integrity-retry-advice
fix/ida-doctor-readiness
fix/kuna-version-probe-output-drain
fix/legacy-swift-runtime-name-matching
fix/dylib-lazy-load-commands
fix/browser-scenario-startup-cancel
fix/swift-nested-punycode-classification
fix/compare-unselected-scopes
fix/objc-method-budget-coverage
fix/dmg-mounted-root-observations
audit/ci-tests-20261019-round2
release/6.7.0
fix/wabt-cleanup-retry-resolution
fix/electron-fuse-elf-pe-architecture
fix/native-plist-descriptor-ownership
fix/kuna-caller-cleanup-ownership
experiment/atomic-codemode-20261016
review/first-contributor-3776
fix/call-resolution-full-read-cycles
fix/member-mutation-selection-repeat
experiment/atomic-re-tools-20261015
perf/member-mutation-lazy-origins
perf/static-network-mutation-index
fix/hopper-startup-lifecycle
audit/3050-windows-proof-20261015
feat/windows-window-capture-2183
audit/3049-windows-proof-20261015
audit/3735-macos-fixtures-20261015
fix/batch-decompile-null-pseudocode
fix/kuna-inventory-honors-jobs
experiment/browser-codemode-tool-design-20261015
fix/hopper-clean-runtime-argv
codex/review-2049-27f1b367
review-fixes/pr-2787-routing
fix/3360-scoped-this-emitters
fix/3367-replaced-global-effects
fix/3838-compact-any-value-items
test/hopper-download-probe-20261015
codex/windows-runtime-creation-red-green-20261015
codex/windows-readonly-root-red-green-20261015
codex/windows-root-share-red-green-20261015
codex/exp-1685-win-mcp-startup
codex/bug-sweep-20261015
investigate/2190-windows-package-phases
investigate/3543-macos-cli-deadline
perf/3281-reverse-tools-feedback-quality
fix/3406-hopper-doctor-fixture
fix/3341-wrapper-require-test-context
fix/3301-owned-uninstall-fixtures
fix/3164-compiler-input-ownership
fix/pcap-cleanup-stdout-newline-20261014
perf/interface-builder-summary
codex/review-fixes-pr-1977
fix/ghidra-effective-jvm-platform
fix/docs-turkish-rust-link-20261013
fix/main-maintenance-ci-20261010
fix/recover-binary-session-process-effects
codex/review-fixes-pr-2457
codex/ci-required-cancellation-gate
fix/main-format-baseline-20261013
fix/setup-preserve-entry-fields
cursor/restore-main-checks-1ffc
fix/ghidra-bridge-hashmap-import
codex/shared-ci-jadx-json-default-20261013
codex/shared-ci-ghidra-test-groups-20261013
codex/shared-ci-java-fixture-format-20261013
codex/fix-intrinsic-invocation-boundaries
test/release-preflight-reusable-jobs
test/js-parameter-provenance-expectation
fix/linux-hopper-verification-current
test/2419-ghidra-elf-probe-size
followup/pr-1544-native-ida
codex/fold-pr-2230-contribution
docs/readme-responsible-use
fix/issue-2064
fix/issue-2050
fix/issue-2053
fix/issue-1968
fix/issue-1945
experiment/codemode-bridge-20261011
fix/tool-effects-negative-wording-1904-20261012
fix/1842-electron-interop-provenance
feat/recover-javascript-multi-input
refactor/website-i18n-prep
fix/nativeaot-pe-virtual-extent
fix/browser-verifier-ci-hang
fix/pe-resources-input-selection
docs/beginner-friendly-guides
docs/beginner-setup-guide
fix/javascript-analysis-worker-1563
fix/1649-ghidra-duplicate-names
fix/javascript-export-keys-integration-1369
test/maintenance-boundary-contracts-20261011
review/javascript-export-keys-1369
review/process-live-progress-1483
test/javascript-1495-growth-ratio
fix/ci-escape-timing
docs/normalized-terminal-geometry-1450
release/6.3.0
archive/release-6.3.0-pr1381
release/6.2.0
fix/electron-preload-dynamic-override
fix/javascript-self-referential-bundler-edges
release/6.1.0
cursor/grok-client-cherry-pick-0b56
feat/signature-posture
feat/dyld-shared-cache
feat/artifact-integrity-policy
feat/pkg-inventory
release/6.0.0
chore/sync-release-5.0.0
release/5.0.0
cursor/cloud-agent-node-787e
fix/javascript-xhr-open-endpoints
review/pr-711-realpath-fallback-wording
cleanup/replay-confinement
codex/test-suite-performance
codex/issue-foundations
codex/address-open-issues-2026-07-20
agent/javascript-vite-rollup-esbuild-adapters
agent/process-cleanup
agent/triage-integration
agent/unknown-commands-registry
feat/mcp-identity-metadata
morlvps/issue_cleanup
codex/installer-redesign
morl/roadmap-investigation-core
morl/roadmap-investigation-docs
release-please--branches--main--components--rea
rea-agents-6.8.0
rea-agents-6.7.0
rea-agents-6.6.0
experiment/atomic-codemode-wait-runtime-20261016
rea-agents-6.5.0
rea-agents-6.4.0
rea-agents-6.3.0
rea-agents-6.2.0
rea-agents-6.1.0
rea-agents-6.0.0
rea-agents-5.0.0
rea-agents-4.1.0
rea-agents-4.0.1
rea-agents-4.0.0
rea-agents-3.2.1
rea-agents-3.2.0
rea-agents-3.1.0
rea-agents-3.0.0
rea-agents-2.7.0
rea-agents-2.6.0
rea-agents-2.5.1
rea-agents-2.5.0
rea-agents-2.4.0
rea-agents-2.3.0
rea-agents-2.2.0
rea-agents-2.1.0
rea-agents-2.0.0
rea-agents-1.7.0
rea-agents-1.6.0
rea-agents-1.5.0
rea-agents-1.4.0
rea-agents-1.3.0
rea-agents-1.2.0
rea-agents-1.1.0
rea-agents-1.0.0
rea-agents-0.5.0
rea-agents-0.4.0
rea-agents-0.3.0
rea-0.2.1
rea-0.2.0
Labels
Clear labels   
area: bridge

   
area: bridge

   
area: build

   
area: build

   
area: build

   
area: cli

   
area: domain

   
area: evidence

   
area: mcp

   
area: native

   
area: providers

   
area: runtime

   
area: server

   
area: setup

   
area: tests

   
bug

   
design

   
documentation

   
enhancement

   
enhancement

   
enhancement

   
experiment

   
help wanted

   
i18n

   
invalid

   
P0: critical

   
P1: high

   
P2: medium

   
P2: medium

   
P2: medium

   
P3: low

   
performance

   
provider-request

   
pull-request
Mirrored from GitHub Pull Request

  
question

   
roadmap

   
triage
No labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
question
roadmap
triage
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#4943
No description provided.