morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1932] [Feature] Run the agent evaluation against Claude Code and Gemini CLI, not only Codex #640

New issue

Open

opened 2026-10-15 18:26:52 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:26:52 +00:00

Owner

Originally created by @balkan16 on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1932

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Current limitation

npm run verify:agent is the only lane that measures REA's agent experience with a real model, and it runs only through Codex. scripts/verify-agent-experience.mjs spawns Codex directly (runCodex, isolated CODEX_HOME) and scoring parses Codex JSONL events (evaluateCodexEvents).

The per-client lanes (verify:claude-client, verify:gemini-client, ...) use a local model fixture with synthetic usage, so they verify integration, not real-model investigation quality. Routing, validation-failure, repetition and factual-claim regressions in non-Codex clients go unmeasured.

Use cases

rea setup registers REA with Claude Code, Gemini CLI and others. Tool descriptions and workflows that route well for one model can mislead another, so maintainers and users currently cannot see whether a contract change helps Codex while hurting other supported clients.

Desired outcome and success criteria

The same scenarios, golden claims and gates run against a selected client, e.g. REA_AGENT_EVAL_CLIENT=claude-code.
Codex remains the default; existing report fields are unchanged, plus a client field.
Each client runs in an isolated home configured via public rea setup, matching the current Codex isolation.
Results are comparable across clients because scenarios and grading are shared.

Authority boundaries and non-goals

No new scenarios, no gate threshold changes, no CI requirement for paid model runs (stays a manual lane, as today). No user configuration copied beyond authentication, same as the current Codex lane.

Alternatives considered

Running cross-client comparisons outside the repo. That works for one-off results but cannot catch regressions when REA's contracts change.

Possible direction

A client adapter with two responsibilities: (1) run a scenario in an isolated home via rea setup, (2) normalize the client transcript into the event shape the existing evaluator needs (tool calls, results, final answer, token use). Reuse the isolation and setup code already in the per-client verifiers instead of writing new setup paths. Claude Code first, then Gemini CLI.

Question: is a client-adapter boundary acceptable here, or would you prefer the evaluation to stay Codex-only? I'm happy to implement it and report which scenarios and clients I verified locally.

References

docs/testing.md, "Agent evaluation and conformance records" and the client lane sections
scripts/verify-agent-experience.mjs
scripts/verify-claude-client.mjs
scripts/verify-gemini-client.mjs

Originally created by @balkan16 on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1932 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Current limitation `npm run verify:agent` is the only lane that measures REA's agent experience with a real model, and it runs only through Codex. `scripts/verify-agent-experience.mjs` spawns Codex directly (`runCodex`, isolated `CODEX_HOME`) and scoring parses Codex JSONL events (`evaluateCodexEvents`). The per-client lanes (`verify:claude-client`, `verify:gemini-client`, ...) use a local model fixture with synthetic usage, so they verify integration, not real-model investigation quality. Routing, validation-failure, repetition and factual-claim regressions in non-Codex clients go unmeasured. ### Use cases `rea setup` registers REA with Claude Code, Gemini CLI and others. Tool descriptions and workflows that route well for one model can mislead another, so maintainers and users currently cannot see whether a contract change helps Codex while hurting other supported clients. ### Desired outcome and success criteria - The same scenarios, golden claims and gates run against a selected client, e.g. `REA_AGENT_EVAL_CLIENT=claude-code`. - Codex remains the default; existing report fields are unchanged, plus a `client` field. - Each client runs in an isolated home configured via public `rea setup`, matching the current Codex isolation. - Results are comparable across clients because scenarios and grading are shared. ### Authority boundaries and non-goals No new scenarios, no gate threshold changes, no CI requirement for paid model runs (stays a manual lane, as today). No user configuration copied beyond authentication, same as the current Codex lane. ### Alternatives considered Running cross-client comparisons outside the repo. That works for one-off results but cannot catch regressions when REA's contracts change. ### Possible direction A client adapter with two responsibilities: (1) run a scenario in an isolated home via `rea setup`, (2) normalize the client transcript into the event shape the existing evaluator needs (tool calls, results, final answer, token use). Reuse the isolation and setup code already in the per-client verifiers instead of writing new setup paths. Claude Code first, then Gemini CLI. Question: is a client-adapter boundary acceptable here, or would you prefer the evaluation to stay Codex-only? I'm happy to implement it and report which scenarios and clients I verified locally. ### References - docs/testing.md, "Agent evaluation and conformance records" and the client lane sections - scripts/verify-agent-experience.mjs - scripts/verify-claude-client.mjs - scripts/verify-gemini-client.mjs

bsakaguchi added the

P3: low

area: cli

enhancement

labels

2026-10-15 18:26:52 +00:00

bsakaguchi commented

2026-10-15 18:26:53 +00:00

Author

Owner

@Nick-800 commented on GitHub (Oct 11, 2026):

I'd like to take this one. Plan: normalize Codex / Claude Code / Gemini CLI transcripts into one client-neutral transcript consumed by both the metric and known-answer graders, select the client with REA_AGENT_EVAL_CLIENT (Codex default), configure each isolated home through rea setup, and add a client field to the report. Scenarios, claims and gates stay shared and unchanged.

@Nick-800 commented on GitHub (Oct 11, 2026): I'd like to take this one. Plan: normalize Codex / Claude Code / Gemini CLI transcripts into one client-neutral transcript consumed by both the metric and known-answer graders, select the client with `REA_AGENT_EVAL_CLIENT` (Codex default), configure each isolated home through `rea setup`, and add a `client` field to the report. Scenarios, claims and gates stay shared and unchanged.

bsakaguchi referenced this issue

2026-10-15 18:28:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi commented

2026-10-16 21:02:28 +00:00

Author

Owner

@pablodiazjorge commented on GitHub (Oct 16, 2026):

@Nick-800, two things from running Claude Code as the client in rea-lab's studies (rea 6.4.0, Claude Code 2.1.296; harness in bench/run.ts and bench/claude.ts) that may save you time on the Claude Code lane:

Large MCP results do not reach the stream. When a result is too big, Claude Code returns a tool error instead ("result (104,841 characters) exceeds maximum allowed tokens. Output has been saved to …") and writes the output under its config directory, in projects/<project>/<session>/tool-results/. The agent then reads that file in chunks. In study 3 this happened in 32 of 72 runs. Our grader read only the stream events, so it missed two Evidence ids that reached the agent through one of those files. A client-neutral transcript needs to resolve those files, or the known-answer grader undercounts.
Isolation that held for 144 runs:
- a per-run CLAUDE_CONFIG_DIR with its own settings.json, loaded with --setting-sources user --settings <file>;
- --strict-mcp-config --mcp-config <file>, so only REA is connected;
- defaultMode: "dontAsk" with an explicit allow list;
- additionalDirectories for the tool-results folder, otherwise those reads are denied;
- claude -p --output-format stream-json --verbose for the events.

@pablodiazjorge commented on GitHub (Oct 16, 2026): @Nick-800, two things from running Claude Code as the client in [rea-lab](https://github.com/pablodiazjorge/rea-lab)'s studies (rea 6.4.0, Claude Code 2.1.296; harness in `bench/run.ts` and `bench/claude.ts`) that may save you time on the Claude Code lane: - **Large MCP results do not reach the stream.** When a result is too big, Claude Code returns a tool error instead ("result (104,841 characters) exceeds maximum allowed tokens. Output has been saved to …") and writes the output under its config directory, in `projects/<project>/<session>/tool-results/`. The agent then reads that file in chunks. In study 3 this happened in 32 of 72 runs. Our grader read only the stream events, so it missed two Evidence ids that reached the agent through one of those files. A client-neutral transcript needs to resolve those files, or the known-answer grader undercounts. - **Isolation that held for 144 runs:** - a per-run `CLAUDE_CONFIG_DIR` with its own `settings.json`, loaded with `--setting-sources user --settings <file>`; - `--strict-mcp-config --mcp-config <file>`, so only REA is connected; - `defaultMode: "dontAsk"` with an explicit allow list; - `additionalDirectories` for the `tool-results` folder, otherwise those reads are denied; - `claude -p --output-format stream-json --verbose` for the events.

bsakaguchi referenced this issue

2026-10-16 21:14:46 +00:00

[PR #647] [MERGED] fix(ghidra): preserve request and shutdown failure diagnostics #2138

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-17 21:01:24 +00:00