morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6611] [Bug] Missing-target evaluation rejects answers that obey its instruction to end with JSON #3995

New issue

Closed

opened 2026-10-31 00:45:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:45:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6611

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Other

Observed behavior

Observed with the production clarification validator and scorer: a missing-target response containing a plain clarification sentence followed by the requested JSON object fails targetClarificationPassed and the evaluation gate. The identical JSON object alone passes. Both responses make zero REA calls. The prompt asks to “End with one JSON object”; it does not request a JSON-only answer.

This was an isolated synthetic-transcript replay, not an end-to-end Codex/model run.

Reproduction

In a source checkout with locked dependencies installed, save this as clarification-repro.mts at the repository root and run node_modules/.bin/tsx clarification-repro.mts. It loads the runner's actual validator without executing setup or starting any process under test.

import { readFileSync } from 'node:fs';
import { parse } from '@babel/parser';
import { z } from 'zod';
import { evaluateCodexEvents } from './src/evaluation/CodexAgentEval.ts';
import { agentEvaluationPassed } from './src/evaluation/AgentEvaluationReport.ts';
const source = readFileSync('scripts/verify-agent-experience.mjs', 'utf8');
const ast = parse(source, { sourceType: 'module' });
const node = ast.program.body.find(x =>
  x.type === 'FunctionDeclaration' && x.id?.name === 'validTargetClarification'
)!;
const validate = new Function('z',
  `return (${source.slice(node.start!, node.end!)});`)(z);
const answer = JSON.stringify({
  status: 'needs_target', requested_input: 'app_name_or_artifact_path',
  question: 'Which app name or artifact path should I inspect?'
});
for (const [label, message] of [
  ['bare-json', answer],
  ['prose-then-json',
   'Please provide the app name or artifact path so I can inspect the right target.\n\n' + answer]
]) {
  const metrics = evaluateCodexEvents([
    { type: 'item.completed', item: { type: 'agent_message', text: message } },
    { type: 'turn.completed', usage: {
      input_tokens: 100, cached_input_tokens: 0, output_tokens: 30
    } }
  ], null);
  const targetClarificationPassed = validate(message);
  console.log(JSON.stringify({
    label, naturalUse: metrics.naturalUse,
    correctFirstTool: metrics.correctFirstTool,
    inputValidationFailureCount: metrics.inputValidationFailureCount,
    requiredToolSubsequenceMet: metrics.requiredToolSubsequenceMet,
    inputTokens: metrics.inputTokens, targetClarificationPassed,
    gate: agentEvaluationPassed({
      ...metrics, expectedFirstTool: null, targetClarificationPassed, exitCode: 0
    })
  }));
}

Both cases executed once before this draft was saved. The run used the same imports and validator extraction with absolute imports to an inspection checkout; the runnable version above uses repository-relative imports and no private paths.

Expected behavior

An answer that asks for the app name or artifact path, makes zero REA calls, and ends with the exact requested JSON should satisfy the missing-target scenario. Alternatively, a JSON-only output requirement must be expressed in its prompt. The current prompt in scripts/verify-agent-experience.mjs, scenario missing-target, says: End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).

Impact and scope

Correct missing-target routing can fail the benchmark because of preceding prose that the prompt permits. The default scenario set includes this case, so this false failure can fail the overall evaluation. The directly observed scope is a pair of synthetic transcripts; actual model frequency is unknown. A bare JSON response is an observed passing workaround.

REA version

6.7.0, source baseline.

Runtime and platform

Node.js v26.12.1; Linux x64; repository's locked tsx, @babel/parser, and zod. No target, analysis engine, live home/configuration, or model execution was used.

Target and artifact context

Built-in missing-target scenario and synthetic completed Codex agent-message events.

Interface

Other

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Observed output:

{"label":"bare-json","naturalUse":false,"correctFirstTool":true,"inputValidationFailureCount":0,"requiredToolSubsequenceMet":true,"inputTokens":100,"targetClarificationPassed":true,"gate":true}
{"label":"prose-then-json","naturalUse":false,"correctFirstTool":true,"inputValidationFailureCount":0,"requiredToolSubsequenceMet":true,"inputTokens":100,"targetClarificationPassed":false,"gate":false}

The production validTargetClarification function at scripts/verify-agent-experience.mjs:492-504 applies JSON.parse(message) to the entire message. Its SyntaxError path returns false before the correct trailing object's schema can be assessed. src/evaluation/AgentEvaluationReport.ts:34-35 requires this boolean for the null-first-tool scenario.

Typed outcome or recovery expected

The example prose-plus-JSON clarification should pass the advertised prompt contract. Proposed restoration check, not executed: accept that answer while continuing to reject an analyzed status, missing requested input, an empty question, or any REA tool call for this scenario.

Analysis or suspected cause

Source-derived: the prompt describes a suffix while the validator requires the whole response to be JSON. The other closed factual scenarios expressly instruct the agent to return only one JSON object; the missing-target prompt lacks that restriction.

Additional context

No known-good version or real-model run was tested. Open and closed issues were searched for missing-target evaluation and clarification-validator reports before submission.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A prompt-compliant target clarification fails the actual scorer's gate despite correct zero-call routing and all remaining gate inputs; this is a bounded evaluation false failure.
Workaround status: available
Workaround basis: The identical requested JSON object without preceding prose passed the actual validator and gate in the paired control.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6611 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Other ## Observed behavior Observed with the production clarification validator and scorer: a missing-target response containing a plain clarification sentence followed by the requested JSON object fails `targetClarificationPassed` and the evaluation gate. The identical JSON object alone passes. Both responses make zero REA calls. The prompt asks to “End with one JSON object”; it does not request a JSON-only answer. This was an isolated synthetic-transcript replay, not an end-to-end Codex/model run. ## Reproduction In a source checkout with locked dependencies installed, save this as `clarification-repro.mts` at the repository root and run `node_modules/.bin/tsx clarification-repro.mts`. It loads the runner's actual validator without executing setup or starting any process under test. ```ts import { readFileSync } from 'node:fs'; import { parse } from '@babel/parser'; import { z } from 'zod'; import { evaluateCodexEvents } from './src/evaluation/CodexAgentEval.ts'; import { agentEvaluationPassed } from './src/evaluation/AgentEvaluationReport.ts'; const source = readFileSync('scripts/verify-agent-experience.mjs', 'utf8'); const ast = parse(source, { sourceType: 'module' }); const node = ast.program.body.find(x => x.type === 'FunctionDeclaration' && x.id?.name === 'validTargetClarification' )!; const validate = new Function('z', `return (${source.slice(node.start!, node.end!)});`)(z); const answer = JSON.stringify({ status: 'needs_target', requested_input: 'app_name_or_artifact_path', question: 'Which app name or artifact path should I inspect?' }); for (const [label, message] of [ ['bare-json', answer], ['prose-then-json', 'Please provide the app name or artifact path so I can inspect the right target.\n\n' + answer] ]) { const metrics = evaluateCodexEvents([ { type: 'item.completed', item: { type: 'agent_message', text: message } }, { type: 'turn.completed', usage: { input_tokens: 100, cached_input_tokens: 0, output_tokens: 30 } } ], null); const targetClarificationPassed = validate(message); console.log(JSON.stringify({ label, naturalUse: metrics.naturalUse, correctFirstTool: metrics.correctFirstTool, inputValidationFailureCount: metrics.inputValidationFailureCount, requiredToolSubsequenceMet: metrics.requiredToolSubsequenceMet, inputTokens: metrics.inputTokens, targetClarificationPassed, gate: agentEvaluationPassed({ ...metrics, expectedFirstTool: null, targetClarificationPassed, exitCode: 0 }) })); } ``` Both cases executed once before this draft was saved. The run used the same imports and validator extraction with absolute imports to an inspection checkout; the runnable version above uses repository-relative imports and no private paths. ## Expected behavior An answer that asks for the app name or artifact path, makes zero REA calls, and ends with the exact requested JSON should satisfy the missing-target scenario. Alternatively, a JSON-only output requirement must be expressed in its prompt. The current prompt in `scripts/verify-agent-experience.mjs`, scenario `missing-target`, says: `End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).` ## Impact and scope Correct missing-target routing can fail the benchmark because of preceding prose that the prompt permits. The default scenario set includes this case, so this false failure can fail the overall evaluation. The directly observed scope is a pair of synthetic transcripts; actual model frequency is unknown. A bare JSON response is an observed passing workaround. ## REA version 6.7.0, source [baseline](https://github.com/morluto/rea/commit/21b836a32710a575357715a798beadd0f3976b8a). ## Runtime and platform Node.js v26.12.1; Linux x64; repository's locked `tsx`, `@babel/parser`, and `zod`. No target, analysis engine, live home/configuration, or model execution was used. ## Target and artifact context Built-in `missing-target` scenario and synthetic completed Codex agent-message events. ## Interface Other ## Earliest failure stage Request parsing or validation ## Relevant output or evidence Observed output: ```text {"label":"bare-json","naturalUse":false,"correctFirstTool":true,"inputValidationFailureCount":0,"requiredToolSubsequenceMet":true,"inputTokens":100,"targetClarificationPassed":true,"gate":true} {"label":"prose-then-json","naturalUse":false,"correctFirstTool":true,"inputValidationFailureCount":0,"requiredToolSubsequenceMet":true,"inputTokens":100,"targetClarificationPassed":false,"gate":false} ``` The production `validTargetClarification` function at `scripts/verify-agent-experience.mjs:492-504` applies `JSON.parse(message)` to the entire message. Its `SyntaxError` path returns false before the correct trailing object's schema can be assessed. `src/evaluation/AgentEvaluationReport.ts:34-35` requires this boolean for the null-first-tool scenario. ## Typed outcome or recovery expected The example prose-plus-JSON clarification should pass the advertised prompt contract. Proposed restoration check, not executed: accept that answer while continuing to reject an `analyzed` status, missing requested input, an empty question, or any REA tool call for this scenario. ## Analysis or suspected cause Source-derived: the prompt describes a suffix while the validator requires the whole response to be JSON. The other closed factual scenarios expressly instruct the agent to return only one JSON object; the missing-target prompt lacks that restriction. ## Additional context No known-good version or real-model run was tested. Open and closed issues were searched for missing-target evaluation and clarification-validator reports before submission. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A prompt-compliant target clarification fails the actual scorer's gate despite correct zero-call routing and all remaining gate inputs; this is a bounded evaluation false failure. - Workaround status: available - Workaround basis: The identical requested JSON object without preceding prose passed the actual validator and gate in the paired control.

bsakaguchi closed this issue

2026-10-31 00:45:53 +00:00

bsakaguchi commented

2026-10-31 00:45:54 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

At d87e73f2db50be72d33a4b3c7199af35518b441c, the missing-target scenario now explicitly requests only one JSON object, without prose or Markdown fences, alongside the whole-answer validator (scenario). This implements the alternative requested here: make the JSON-only requirement explicit. The regression checks the prompt/validator agreement, valid clarification gate and invalid/prose responses.

I ran the cited regression file on this revision; all passed as part of the focused 28-file, 603-test run. Provider-facing checks here use controlled fixtures; this does not claim a run with the real analysis engine. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): At `d87e73f2db50be72d33a4b3c7199af35518b441c`, the missing-target scenario now explicitly requests only one JSON object, without prose or Markdown fences, alongside the whole-answer validator ([scenario](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/evaluation/model-evals/AgentEvaluationScenarios.ts#L11)). This implements the alternative requested here: make the JSON-only requirement explicit. The [regression](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/evaluation/model-evals/agentEvaluationScenarios.test.ts#L96) checks the prompt/validator agreement, valid clarification gate and invalid/prose responses. I ran the cited regression file on this revision; all passed as part of the focused 28-file, 603-test run. Provider-facing checks here use controlled fixtures; this does not claim a run with the real analysis engine. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:10:56 +00:00

[PR #3995] [CLOSED] fix(mcp): advertise a screenshot example with a valid PNG checksum #5984

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425