[PR #1734] [MERGED] fix(evaluation): authenticate delivered Evidence citations #4784

New issue

Closed

opened 2026-10-31 00:55:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:55:00 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1734
Author: @N0zoM1z0
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/evaluation-evidence-1728

📝 Commits (1)

d1785cf fix(evaluation): authenticate delivered Evidence citations

📊 Changes

4 files changed (+269 additions, -37 deletions)

View changed files

📝 src/evaluation/CodexAgentEval.ts (+15 -12)
📝 src/evaluation/KnownAnswerEvaluation.ts (+61 -19)
📝 tests/evaluation/model-evals/codexAgentEval.test.ts (+180 -6)
📝 tests/evaluation/model-evals/knownAnswerEvaluation.test.ts (+13 -0)

📄 Description

A diagnostic ev_ token, including one inside a failed tool response, could satisfy the produced-Evidence citation requirement and pass the unassessed-answer release gate. Authenticate completed REA deliveries through the same parser used for fixture grading, and remove the whole-result regex scan.

Accept full structured or sole complete JSON text Evidence with consistent aliases and operation binding. Preserve the nested inventory Evidence delivered by inspect_artifact, checking its semantic ID, substep operation/status and parent link. Bare references, error responses, tampered records and partial previews cannot count as delivery. Existing scoring tests now use authentic records rather than ID-only stubs.

Closes #1728.

Validation: actual compiled red/green probes with diagnostic and error-response negatives plus complete Evidence captured from a 218-byte real ZIP via the CLI/stdio MCP. Negatives now fail the citation/release gate; both authentic forms retain the root and nested IDs. All 63 evaluation/fixture-grading/gate tests pass, including text-only grading and nested-record tampering. Fresh compile, single-threaded typecheck, full lint and architecture guards pass under the CPU/RSS limits; complete CLI/MCP result, Evidence ID and source bytes are preserved. No live-model correctness claim.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1734 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/evaluation-evidence-1728` --- ### 📝 Commits (1) - [`d1785cf`](https://github.com/morluto/rea/commit/d1785cf45f5ec2c72352bd2fa17e5045d17fb1e2) fix(evaluation): authenticate delivered Evidence citations ### 📊 Changes **4 files changed** (+269 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `src/evaluation/CodexAgentEval.ts` (+15 -12) 📝 `src/evaluation/KnownAnswerEvaluation.ts` (+61 -19) 📝 `tests/evaluation/model-evals/codexAgentEval.test.ts` (+180 -6) 📝 `tests/evaluation/model-evals/knownAnswerEvaluation.test.ts` (+13 -0) </details> ### 📄 Description A diagnostic `ev_` token, including one inside a failed tool response, could satisfy the produced-Evidence citation requirement and pass the unassessed-answer release gate. Authenticate completed REA deliveries through the same parser used for fixture grading, and remove the whole-result regex scan. Accept full structured or sole complete JSON text Evidence with consistent aliases and operation binding. Preserve the nested inventory Evidence delivered by `inspect_artifact`, checking its semantic ID, substep operation/status and parent link. Bare references, error responses, tampered records and partial previews cannot count as delivery. Existing scoring tests now use authentic records rather than ID-only stubs. Closes #1728. Validation: actual compiled red/green probes with diagnostic and error-response negatives plus complete Evidence captured from a 218-byte real ZIP via the CLI/stdio MCP. Negatives now fail the citation/release gate; both authentic forms retain the root and nested IDs. All 63 evaluation/fixture-grading/gate tests pass, including text-only grading and nested-record tampering. Fresh compile, single-threaded typecheck, full lint and architecture guards pass under the CPU/RSS limits; complete CLI/MCP result, Evidence ID and source bytes are preserved. No live-model correctness claim. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:55:00 +00:00