[GH-ISSUE #1728] fix(evaluation): diagnostic tokens pass the produced Evidence citation gate #581

New issue

Closed

opened 2026-10-15 18:26:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:26:23 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1728

evaluateCodexEvents(..., {requireEvidence:true}) treats every ev_<64 hex> substring anywhere in a tool result as a produced Evidence ID. A final message quoting a diagnostic token can therefore pass the Evidence-required answer heuristic and the release gate, despite no delivered Evidence record. An error response carrying the same token also passes the citation check.

Reproduced with freshly compiled evaluation code from main at 092de8f4b336b537579bd48d7d30135b7114371d (Linux, Node.js 24.18.0). Four actual compiled probes were run, including complete authentic Evidence captured from a small REA CLI/MCP ZIP inspection as positive controls.

Minimal negative input:

const fake = "ev_" + "a".repeat(64);
const events = [
  {type:"item.completed", item:{type:"mcp_tool_call", status:"completed", server:"rea", tool:"inspect_artifact", arguments:{}, result:{content:[{type:"text", text:"unrelated diagnostic: " + fake}]}}},
  {type:"item.completed", item:{type:"agent_message", text:`Observed artifact evidence ${fake} is cited here; runtime behavior remains unknown because no runtime authority was supplied.`}},
  {type:"turn.completed", usage:{input_tokens:100, output_tokens:20}}
];
const metrics = evaluateCodexEvents(events, "inspect_artifact", {requireEvidence:true});
agentEvaluationPassed({...metrics, exitCode:0, expectedFirstTool:"inspect_artifact"});

Observed: evidenceIds:[fake], finalCitesEvidence:true, answerHeuristicsMet:true, factualCorrectness:"not_assessed", and release gate true. Replacing the result with {isError:true, structuredContent:{error:{code:"analysis_failed",message:fake}}} still marks that token as produced Evidence and passes this gate.

Expected: diagnostic strings, metadata-only IDs and failed responses cannot establish that Evidence was delivered. Validate complete Evidence at the owning transcript boundary, preserve real structured/snake-case and complete JSON text-only success shapes, bind the result to its tool operation, and reject conflicting aliases or tampering. Reuse the adjacent known-answer path’s validation where appropriate instead of introducing a second interpretation.

The root cause is evidenceIdsFrom in src/evaluation/CodexAgentEval.ts, which JSON-stringifies the entire result and regex-matches tokens. The neighboring known-answer parser is stricter. This defect concerns citation/heuristic scoring; it does not establish factual correctness for free-form answers or a live model trial. No matching open issue/PR was found through gh; #1065 and #740 are broader roadmaps.

Originally created by @N0zoM1z0 on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1728 `evaluateCodexEvents(..., {requireEvidence:true})` treats every `ev_<64 hex>` substring anywhere in a tool result as a produced Evidence ID. A final message quoting a diagnostic token can therefore pass the Evidence-required answer heuristic and the release gate, despite no delivered Evidence record. An error response carrying the same token also passes the citation check. Reproduced with freshly compiled evaluation code from `main` at `092de8f4b336b537579bd48d7d30135b7114371d` (Linux, Node.js 24.18.0). Four actual compiled probes were run, including complete authentic Evidence captured from a small REA CLI/MCP ZIP inspection as positive controls. Minimal negative input: ```js const fake = "ev_" + "a".repeat(64); const events = [ {type:"item.completed", item:{type:"mcp_tool_call", status:"completed", server:"rea", tool:"inspect_artifact", arguments:{}, result:{content:[{type:"text", text:"unrelated diagnostic: " + fake}]}}}, {type:"item.completed", item:{type:"agent_message", text:`Observed artifact evidence ${fake} is cited here; runtime behavior remains unknown because no runtime authority was supplied.`}}, {type:"turn.completed", usage:{input_tokens:100, output_tokens:20}} ]; const metrics = evaluateCodexEvents(events, "inspect_artifact", {requireEvidence:true}); agentEvaluationPassed({...metrics, exitCode:0, expectedFirstTool:"inspect_artifact"}); ``` Observed: `evidenceIds:[fake]`, `finalCitesEvidence:true`, `answerHeuristicsMet:true`, `factualCorrectness:"not_assessed"`, and release gate `true`. Replacing the result with `{isError:true, structuredContent:{error:{code:"analysis_failed",message:fake}}}` still marks that token as produced Evidence and passes this gate. Expected: diagnostic strings, metadata-only IDs and failed responses cannot establish that Evidence was delivered. Validate complete Evidence at the owning transcript boundary, preserve real structured/snake-case and complete JSON text-only success shapes, bind the result to its tool operation, and reject conflicting aliases or tampering. Reuse the adjacent known-answer path’s validation where appropriate instead of introducing a second interpretation. The root cause is `evidenceIdsFrom` in `src/evaluation/CodexAgentEval.ts`, which JSON-stringifies the entire result and regex-matches tokens. The neighboring known-answer parser is stricter. This defect concerns citation/heuristic scoring; it does not establish factual correctness for free-form answers or a live model trial. No matching open issue/PR was found through `gh`; #1065 and #740 are broader roadmaps.

bsakaguchi

2026-10-15 18:26:23 +00:00