morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6608] [Bug] Agent evaluation rejects the native missing-provider outcome because it requires a successful open #3993

New issue

Closed

opened 2026-10-31 00:45:51 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:45:51 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6608

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Other

Observed behavior

Observed by replaying synthetic completed Codex transcript events through the production source scorer: the native scenario fails when open_binary reports provider unavailability and the agent correctly explains that no analysis session was opened. Making an additional successful close_binary call does not change the failure. A successful-open control with the same final answer passes. No real Codex/model session or analysis engine was launched for this reproduction.

Reproduction

From a source checkout with its locked dependencies installed, save the following as eval-repro.mts at the repository root and run node_modules/.bin/tsx eval-repro.mts. These are the native scenario's actual scoring options, copied from scripts/verify-agent-experience.mjs:140-148.

import assert from 'node:assert/strict';
import { evaluateCodexEvents } from './src/evaluation/CodexAgentEval.ts';
import { agentEvaluationPassed } from './src/evaluation/AgentEvaluationReport.ts';
import { TOOL_CONTRACTS } from './src/contracts/toolContracts.ts';
const args = { path: '/bin/true' };
assert(TOOL_CONTRACTS.find(x => x.name === 'open_binary')!
  .inputSchema.safeParse(args).success);
const final = 'The provider is unavailable, so no native analysis session was opened. Static artifact facts from that provider remain unknown; I did not execute the target or install tools.';
const call = (id, tool, arguments_, result) => ({
  type: 'item.completed', item: {
    id, type: 'mcp_tool_call', server: 'rea', tool,
    arguments: arguments_, status: 'completed', result
  }
});
const failedOpen = call('open', 'open_binary', args, {
  isError: true, structuredContent: {
    error: { code: 'provider_unavailable', message: 'No native analysis provider is available.' }
  }
});
const successOpen = call('open', 'open_binary', args, {
  content: [{ type: 'text', text: 'Session opened.' }]
});
const close = call('close', 'close_binary', {}, {
  content: [{ type: 'text', text: 'Session closed.' }]
});
for (const [label, calls] of [
  ['report-unavailable', [failedOpen]],
  ['also-close', [failedOpen, close]],
  ['successful-open-control', [successOpen, close]]
]) {
  const metrics = evaluateCodexEvents([
    ...calls,
    { type: 'item.completed', item: { type: 'agent_message', text: final } },
    { type: 'turn.completed', usage: {
      input_tokens: 100, cached_input_tokens: 0, output_tokens: 30
    } }
  ], 'open_binary', {
    requiredToolSubsequence: ['open_binary', 'close_binary'],
    requiredAnswerTermGroups: [['unavailable', 'could not', 'provider']],
    forbidInputValidationFailures: true
  });
  console.log(label, JSON.stringify({
    requiredToolSubsequenceMet: metrics.requiredToolSubsequenceMet,
    answerTermCoverageMet: metrics.answerTermCoverageMet,
    inputValidationFailureCount: metrics.inputValidationFailureCount,
    gate: agentEvaluationPassed({
      ...metrics, expectedFirstTool: 'open_binary', exitCode: 0
    })
  }));
}

The isolated scorer probe executed once initially and once with the successful-open control added, before this report was saved. The full npm run verify:agent workflow was not executed. The synthetic result records exercise success/error classification; they do not claim to reproduce an actual provider response payload.

Expected behavior

The documented native missing-provider scenario should accept a schema-valid attempted open followed by an accurate availability limitation when no session was opened. It should still require cleanup when a session was successfully opened. docs/client-verification.md:478-481 explicitly calls this a native missing-provider scenario, and the scenario prompt says to close any session the agent opened.

Impact and scope

A correctly handled expected provider failure cannot pass this scenario. Because native is included by default, this can fail the overall agent evaluation on hosts without a usable native provider. The observed scope is the scorer and release-gate functions with the exact native scenario configuration; real-model frequency and an end-to-end failing command were not measured. Selecting other scenario IDs is an available way to run the remaining benchmarks, but leaves native missing-provider behavior unevaluated.

REA version

6.7.0, source baseline.

Runtime and platform

Node.js v26.12.1; Linux x64; source executed with the repository's locked tsx. No Hopper, Ghidra, target execution, or live user configuration was used.

Target and artifact context

Synthetic Codex JSONL-shaped events for the built-in /bin/true static-analysis target. The target itself was not opened or executed.

Interface

Other

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed scorer outputs, projected to the relevant fields:

report-unavailable {"requiredToolSubsequenceMet":false,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":false}
also-close {"requiredToolSubsequenceMet":false,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":false}
successful-open-control {"requiredToolSubsequenceMet":true,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":true}

In all three observed cases, natural routing and the first-tool check passed, repeated-call and validation-failure counts were zero, input tokens were positive, the epistemic cue was present, and factual correctness was not_assessed. src/evaluation/CodexAgentEval.ts:191-194 removes error calls before matching the required sequence. src/evaluation/AgentEvaluationReport.ts:22-39 requires that sequence and the runner rejects any failed scenario at scripts/verify-agent-experience.mjs:392-396.

Typed outcome or recovery expected

Distinguish an expected missing-provider outcome from a workflow failure, while preserving successful-session cleanup checks. Proposed restoration check, not executed: the same unavailable-provider transcript should pass; a successful open with omitted close should fail.

Analysis or suspected cause

Source-derived: the scenario requires open_binary in the successful-call subsequence, but its advertised missing-provider outcome necessarily has an unsuccessful open. This contradicts the scenario's expected availability path rather than indicating an invalid agent request.

Additional context

Open and closed issues were searched for related agent-evaluation/scoring reports before submission. No known-good version was tested. This report concerns scoring, not whether any native provider should be available.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The scorer rejects the expected missing-provider outcome and therefore can fail the default evaluation despite correct agent routing and availability reporting; the directly observed boundary is synthetic transcript evaluation.
Workaround status: unverified
Workaround basis: The documented REA_AGENT_EVAL_SCENARIOS selector can omit native and run the other evaluations, but that bypass does not validate native missing-provider handling. No equivalent workaround was established.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6608 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Other ## Observed behavior Observed by replaying synthetic completed Codex transcript events through the production source scorer: the `native` scenario fails when `open_binary` reports provider unavailability and the agent correctly explains that no analysis session was opened. Making an additional successful `close_binary` call does not change the failure. A successful-open control with the same final answer passes. No real Codex/model session or analysis engine was launched for this reproduction. ## Reproduction From a source checkout with its locked dependencies installed, save the following as `eval-repro.mts` at the repository root and run `node_modules/.bin/tsx eval-repro.mts`. These are the native scenario's actual scoring options, copied from `scripts/verify-agent-experience.mjs:140-148`. ```ts import assert from 'node:assert/strict'; import { evaluateCodexEvents } from './src/evaluation/CodexAgentEval.ts'; import { agentEvaluationPassed } from './src/evaluation/AgentEvaluationReport.ts'; import { TOOL_CONTRACTS } from './src/contracts/toolContracts.ts'; const args = { path: '/bin/true' }; assert(TOOL_CONTRACTS.find(x => x.name === 'open_binary')! .inputSchema.safeParse(args).success); const final = 'The provider is unavailable, so no native analysis session was opened. Static artifact facts from that provider remain unknown; I did not execute the target or install tools.'; const call = (id, tool, arguments_, result) => ({ type: 'item.completed', item: { id, type: 'mcp_tool_call', server: 'rea', tool, arguments: arguments_, status: 'completed', result } }); const failedOpen = call('open', 'open_binary', args, { isError: true, structuredContent: { error: { code: 'provider_unavailable', message: 'No native analysis provider is available.' } } }); const successOpen = call('open', 'open_binary', args, { content: [{ type: 'text', text: 'Session opened.' }] }); const close = call('close', 'close_binary', {}, { content: [{ type: 'text', text: 'Session closed.' }] }); for (const [label, calls] of [ ['report-unavailable', [failedOpen]], ['also-close', [failedOpen, close]], ['successful-open-control', [successOpen, close]] ]) { const metrics = evaluateCodexEvents([ ...calls, { type: 'item.completed', item: { type: 'agent_message', text: final } }, { type: 'turn.completed', usage: { input_tokens: 100, cached_input_tokens: 0, output_tokens: 30 } } ], 'open_binary', { requiredToolSubsequence: ['open_binary', 'close_binary'], requiredAnswerTermGroups: [['unavailable', 'could not', 'provider']], forbidInputValidationFailures: true }); console.log(label, JSON.stringify({ requiredToolSubsequenceMet: metrics.requiredToolSubsequenceMet, answerTermCoverageMet: metrics.answerTermCoverageMet, inputValidationFailureCount: metrics.inputValidationFailureCount, gate: agentEvaluationPassed({ ...metrics, expectedFirstTool: 'open_binary', exitCode: 0 }) })); } ``` The isolated scorer probe executed once initially and once with the successful-open control added, before this report was saved. The full `npm run verify:agent` workflow was not executed. The synthetic result records exercise success/error classification; they do not claim to reproduce an actual provider response payload. ## Expected behavior The documented native missing-provider scenario should accept a schema-valid attempted open followed by an accurate availability limitation when no session was opened. It should still require cleanup when a session was successfully opened. `docs/client-verification.md:478-481` explicitly calls this a native missing-provider scenario, and the scenario prompt says to close any session the agent opened. ## Impact and scope A correctly handled expected provider failure cannot pass this scenario. Because `native` is included by default, this can fail the overall agent evaluation on hosts without a usable native provider. The observed scope is the scorer and release-gate functions with the exact native scenario configuration; real-model frequency and an end-to-end failing command were not measured. Selecting other scenario IDs is an available way to run the remaining benchmarks, but leaves native missing-provider behavior unevaluated. ## REA version 6.7.0, source [baseline](https://github.com/morluto/rea/commit/21b836a32710a575357715a798beadd0f3976b8a). ## Runtime and platform Node.js v26.12.1; Linux x64; source executed with the repository's locked `tsx`. No Hopper, Ghidra, target execution, or live user configuration was used. ## Target and artifact context Synthetic Codex JSONL-shaped events for the built-in `/bin/true` static-analysis target. The target itself was not opened or executed. ## Interface Other ## Earliest failure stage Transport or result projection ## Relevant output or evidence Observed scorer outputs, projected to the relevant fields: ```text report-unavailable {"requiredToolSubsequenceMet":false,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":false} also-close {"requiredToolSubsequenceMet":false,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":false} successful-open-control {"requiredToolSubsequenceMet":true,"answerTermCoverageMet":true,"inputValidationFailureCount":0,"gate":true} ``` In all three observed cases, natural routing and the first-tool check passed, repeated-call and validation-failure counts were zero, input tokens were positive, the epistemic cue was present, and factual correctness was `not_assessed`. `src/evaluation/CodexAgentEval.ts:191-194` removes error calls before matching the required sequence. `src/evaluation/AgentEvaluationReport.ts:22-39` requires that sequence and the runner rejects any failed scenario at `scripts/verify-agent-experience.mjs:392-396`. ## Typed outcome or recovery expected Distinguish an expected missing-provider outcome from a workflow failure, while preserving successful-session cleanup checks. Proposed restoration check, not executed: the same unavailable-provider transcript should pass; a successful open with omitted close should fail. ## Analysis or suspected cause Source-derived: the scenario requires `open_binary` in the successful-call subsequence, but its advertised missing-provider outcome necessarily has an unsuccessful open. This contradicts the scenario's expected availability path rather than indicating an invalid agent request. ## Additional context Open and closed issues were searched for related agent-evaluation/scoring reports before submission. No known-good version was tested. This report concerns scoring, not whether any native provider should be available. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The scorer rejects the expected missing-provider outcome and therefore can fail the default evaluation despite correct agent routing and availability reporting; the directly observed boundary is synthetic transcript evaluation. - Workaround status: unverified - Workaround basis: The documented REA_AGENT_EVAL_SCENARIOS selector can omit native and run the other evaluations, but that bypass does not validate native missing-provider handling. No equivalent workaround was established.

bsakaguchi closed this issue

2026-10-31 00:45:52 +00:00

bsakaguchi commented

2026-10-31 00:45:52 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

The native scenario no longer requires successful open/close as an unconditional subsequence. Its outcome owner accepts a lone attempted open with provider_unavailable, requires close after successful acquisition, and rejects unexpected errors or unclosed sessions. The runner passes this explicit outcome into the release gate.

Implementation: AgentEvaluationScenarios.ts, AgentEvaluationScenarios.ts.

The relevant regression checks in agentEvaluationScenarios.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. The native scenario no longer requires successful open/close as an unconditional subsequence. Its outcome owner accepts a lone attempted open with provider_unavailable, requires close after successful acquisition, and rejects unexpected errors or unclosed sessions. The runner passes this explicit outcome into the release gate. Implementation: [AgentEvaluationScenarios.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/evaluation/model-evals/AgentEvaluationScenarios.ts#L34), [AgentEvaluationScenarios.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/evaluation/model-evals/AgentEvaluationScenarios.ts#L57). The relevant regression checks in [agentEvaluationScenarios.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/evaluation/model-evals/agentEvaluationScenarios.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:11:46 +00:00

[PR #4150] [CLOSED] fix(gdb): ignore quoted groups=[ when reading inferior lists #6090

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3993

No description provided.

Rows
Columns