[GH-ISSUE #4636] [Bug] MCP browser scenario validation omits the typed input-error diagnostic #2416

New issue

Closed

opened 2026-10-17 21:04:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:04:21 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4636

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

Calling capture_browser_scenario with an undeclared secret reference returns isError: true, but its text contains a raw Zod issue array rather than REA's canonical error object. The validation message and issue path remain visible; the REA code/category, operation, retryability, and remediation fields are missing. The same scenario through capture-browser-scenario returns invalid_request / invalid_input with correction guidance and exits 1. The MCP call resolves normally, and a subsequent ping succeeds.

Reproduction

From a source checkout, install the locked dependencies with npm ci and build with npm run build:cached. Run the following from the checkout root. These commands use the installed MCP client over stdio and fresh temporary homes. The selected executable intentionally does not exist; validation rejects the input before browser capture. No browser, fixture server, or secret value is required.

scenario='{"browser":{"mode":"launch","executable_path":"/nonexistent/chromium"},"start_url":{"url":"http://127.0.0.1:1/","query":[]},"actions":[{"step_id":"pw","action":"fill","locator":{"kind":"css","selector":"#pw"},"value":{"source":"secret","secret_id":"pw"}}],"secrets":[]}'
work=$(mktemp -d)
env -i PATH="$PATH" HOME="$(mktemp -d "$work/home.XXXXXX")" SCENARIO="$scenario" node --input-type=module <<'JS'
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';
import { Ajv2020 } from 'ajv/dist/2020.js';
import { resolve } from 'node:path';
const client = new Client({ name: 'scenario-validation-repro', version: '1' });
const transport = new StdioClientTransport({
  command: process.execPath,
  args: [resolve('scripts/rea.mjs'), 'mcp'],
  env: { PATH: process.env.PATH, HOME: process.env.HOME, REA_LOG_LEVEL: 'silent' },
  stderr: 'pipe'
});
transport.stderr?.on('data', () => {});
try {
  await client.connect(transport);
  const tool = (await client.listTools()).tools.find(t => t.name === 'capture_browser_scenario');
  const input = JSON.parse(process.env.SCENARIO);
  const validate = new Ajv2020({ strict: false, validateFormats: false }).compile(tool.inputSchema);
  console.log('advertised schema accepts:', validate(input));
  console.log(JSON.stringify(await client.callTool({ name: tool.name, arguments: input })));
  console.log('ping:', JSON.stringify(await client.ping()));
} finally {
  await client.close();
  await transport.close();
}
JS
env -i PATH="$PATH" HOME="$(mktemp -d "$work/home.XXXXXX")" node scripts/rea.mjs capture-browser-scenario "$scenario" --json
status=$?
printf 'CLI exit: %s\n' "$status"
rm -rf "$work"

The advertised schema accepts the input, then MCP returns the raw issue array shown below. CLI reports the typed invalid-input result. A separate stdio replay against each tested build also used two wait_for_timeout actions with the same step_id; both returned a raw issue array at path actions. Changing the second ID reached a canonical execution_failure result for the unavailable executable. Each invalid case reproduced once per build in that comparison; the standalone commands above additionally reproduced the undeclared-secret case on the source build.

Expected behavior

Both adapters should reject the same cross-field-invalid scenario through REA's typed input-error diagnostic, retaining the issue path and correction action. The browser scenario contract specifies the shared schema and rejection of undeclared secret references and duplicate action IDs. The MCP contract promises canonical JSON error diagnostics for tool execution failures. CLI demonstrates the expected semantic classification; the adapter envelopes need not be byte-identical.

Impact and scope

MCP consumers lose the machine-readable REA classification and recovery fields for these cross-field-invalid requests. Rejection, a useful message/path, and connection usability remain intact. Only validation and the unavailable-executable control were exercised; successful browser capture was not tested.

Impact level: P2
Assessment status: supported
Impact basis: The bounded adapter defect prevents canonical machine-readable classification and recovery for the tested invalid requests; validation still rejects them and the server remains usable.
Workaround status: available
Workaround basis: The CLI returned the typed diagnosis, issue path, and correction action for both invalid cases on both tested builds, providing a diagnostic workaround without acquiring a target.

REA version

Published rea-agents 6.5.0 installed separately from npm, and a fresh source build of main at 71929ed37c. Both MCP initialization replies identified REA 6.5.0. The npm latest tag pointed to this release when checked.

Runtime and platform

Node v24.18.0 (the checkout's development pin), npm 11.16.0, macOS arm64. All REA invocations used a fresh scratch HOME and a cleared environment containing only PATH, HOME, and the explicit reproduction variables. Locked MCP client/server dependencies are 2.3.1.

Target and artifact context

Declarative browser scenario with a fill action referencing secret ID pw and an empty secrets declaration array. The URL is loopback, no real secret value is supplied, and input validation fails before capture.

Interface

MCP

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Expected source: docs/browser-scenario-contract.md:3-7 and :178-181 specify shared adapter inputs and cross-field rejection; docs/mcp-contracts.md:72-77 requires canonical error text; AGENTS.md, Architecture and Cleanup, requires tagged errors for expected boundary failures.
Failure source: The executed public MCP and CLI commands above; src/server/registerBrowserScenarioTool.ts:38 parses with the refined schema before REA result delivery, while src/cliBrowserScenarioCommands.ts:39-50 maps failed validation into AnalysisInputError.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Both tested builds returned advertised schema accepts: true. The MCP result contained isError: true, the normal target-content provenance marker, no structuredContent, and this decoded content[0].text:

[{"code":"custom","path":["secrets"],"message":"Secret reference pw is not declared"}]

The same fixture through CLI returned the following JSON and exited 1:

{"error":"Browser scenario capture failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"capture_browser_scenario","issues":[{"path":["secrets"],"reason":"invalid_value","message":"Secret reference pw is not declared"}]}}

The duplicate-ID case returned MCP text [{"code":"custom","path":["actions"],"message":"Action step IDs must be unique"}]; CLI returned the same typed classification with an actions issue path. The unique-ID control returned a canonical MCP error object with code/category execution_failure. Pings after the cases returned {}, and server/CLI stderr was empty in the comparison.

Typed outcome or recovery expected

Repeat the undeclared-secret request: it should still be rejected with isError: true, but its text should contain the canonical MCP error object with invalid_request / invalid_input, operation capture_browser_scenario, the secrets issue path, and correction guidance. Duplicate action IDs should preserve the actions issue path through the same typed outcome. Preserve omission of error structuredContent and post-error ping. A scenario without those cross-field violations should continue beyond input validation, as the unavailable-executable control did.

Analysis or suspected cause

Source-derived explanation: src/contracts/browserScenarioToolContracts.ts:31 advertises browserScenarioInputSchema, while src/domain/browserScenario.ts:134 adds cross-field refinements through superRefine. The MCP handler invokes that refined schema's throwing parser before admission/result delivery. The observed exception becomes SDK error text. CLI explicitly projects refined validation failures into AnalysisInputError. This is an isError tool result, not a JSON-RPC internal error.

Additional context

No matching report was found in open/closed issue and PR searches for capture_browser_scenario, capture-browser-scenario, ZodError, zod, input validation, and typed error. The oversized SDK validation diagnostic in #3221 and lost comparison refinements in #1364 concern different failure paths. No last-known-good release was established.

Originally created by @coygeek on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4636 ### Before submitting - [X] I searched the open and closed issues for a related report. - [X] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Observed behavior Calling `capture_browser_scenario` with an undeclared secret reference returns `isError: true`, but its text contains a raw Zod issue array rather than REA's canonical error object. The validation message and issue path remain visible; the REA code/category, operation, retryability, and remediation fields are missing. The same scenario through `capture-browser-scenario` returns `invalid_request` / `invalid_input` with correction guidance and exits 1. The MCP call resolves normally, and a subsequent ping succeeds. ### Reproduction From a source checkout, install the locked dependencies with `npm ci` and build with `npm run build:cached`. Run the following from the checkout root. These commands use the installed MCP client over stdio and fresh temporary homes. The selected executable intentionally does not exist; validation rejects the input before browser capture. No browser, fixture server, or secret value is required. ```sh scenario='{"browser":{"mode":"launch","executable_path":"/nonexistent/chromium"},"start_url":{"url":"http://127.0.0.1:1/","query":[]},"actions":[{"step_id":"pw","action":"fill","locator":{"kind":"css","selector":"#pw"},"value":{"source":"secret","secret_id":"pw"}}],"secrets":[]}' work=$(mktemp -d) env -i PATH="$PATH" HOME="$(mktemp -d "$work/home.XXXXXX")" SCENARIO="$scenario" node --input-type=module <<'JS' import { Client } from '@modelcontextprotocol/client'; import { StdioClientTransport } from '@modelcontextprotocol/client/stdio'; import { Ajv2020 } from 'ajv/dist/2020.js'; import { resolve } from 'node:path'; const client = new Client({ name: 'scenario-validation-repro', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: [resolve('scripts/rea.mjs'), 'mcp'], env: { PATH: process.env.PATH, HOME: process.env.HOME, REA_LOG_LEVEL: 'silent' }, stderr: 'pipe' }); transport.stderr?.on('data', () => {}); try { await client.connect(transport); const tool = (await client.listTools()).tools.find(t => t.name === 'capture_browser_scenario'); const input = JSON.parse(process.env.SCENARIO); const validate = new Ajv2020({ strict: false, validateFormats: false }).compile(tool.inputSchema); console.log('advertised schema accepts:', validate(input)); console.log(JSON.stringify(await client.callTool({ name: tool.name, arguments: input }))); console.log('ping:', JSON.stringify(await client.ping())); } finally { await client.close(); await transport.close(); } JS env -i PATH="$PATH" HOME="$(mktemp -d "$work/home.XXXXXX")" node scripts/rea.mjs capture-browser-scenario "$scenario" --json status=$? printf 'CLI exit: %s\n' "$status" rm -rf "$work" ``` The advertised schema accepts the input, then MCP returns the raw issue array shown below. CLI reports the typed invalid-input result. A separate stdio replay against each tested build also used two `wait_for_timeout` actions with the same `step_id`; both returned a raw issue array at path `actions`. Changing the second ID reached a canonical `execution_failure` result for the unavailable executable. Each invalid case reproduced once per build in that comparison; the standalone commands above additionally reproduced the undeclared-secret case on the source build. ### Expected behavior Both adapters should reject the same cross-field-invalid scenario through REA's typed input-error diagnostic, retaining the issue path and correction action. The browser scenario contract specifies the shared schema and rejection of undeclared secret references and duplicate action IDs. The MCP contract promises canonical JSON error diagnostics for tool execution failures. CLI demonstrates the expected semantic classification; the adapter envelopes need not be byte-identical. ### Impact and scope MCP consumers lose the machine-readable REA classification and recovery fields for these cross-field-invalid requests. Rejection, a useful message/path, and connection usability remain intact. Only validation and the unavailable-executable control were exercised; successful browser capture was not tested. - Impact level: P2 - Assessment status: supported - Impact basis: The bounded adapter defect prevents canonical machine-readable classification and recovery for the tested invalid requests; validation still rejects them and the server remains usable. - Workaround status: available - Workaround basis: The CLI returned the typed diagnosis, issue path, and correction action for both invalid cases on both tested builds, providing a diagnostic workaround without acquiring a target. ### REA version Published `rea-agents` 6.5.0 installed separately from npm, and a fresh source build of `main` at 71929ed37cdc560b6b8aa62b5f7182e417c8a850. Both MCP initialization replies identified REA 6.5.0. The npm `latest` tag pointed to this release when checked. ### Runtime and platform Node v24.18.0 (the checkout's development pin), npm 11.16.0, macOS arm64. All REA invocations used a fresh scratch HOME and a cleared environment containing only PATH, HOME, and the explicit reproduction variables. Locked MCP client/server dependencies are 2.3.1. ### Target and artifact context Declarative browser scenario with a fill action referencing secret ID `pw` and an empty secrets declaration array. The URL is loopback, no real secret value is supplied, and input validation fails before capture. ### Interface MCP ### Earliest failure stage Request parsing or validation ### Relevant output or evidence - Expected source: `docs/browser-scenario-contract.md:3-7` and `:178-181` specify shared adapter inputs and cross-field rejection; `docs/mcp-contracts.md:72-77` requires canonical error text; `AGENTS.md`, Architecture and Cleanup, requires tagged errors for expected boundary failures. - Failure source: The executed public MCP and CLI commands above; `src/server/registerBrowserScenarioTool.ts:38` parses with the refined schema before REA result delivery, while `src/cliBrowserScenarioCommands.ts:39-50` maps failed validation into AnalysisInputError. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Both tested builds returned `advertised schema accepts: true`. The MCP result contained `isError: true`, the normal target-content provenance marker, no `structuredContent`, and this decoded `content[0].text`: ```json [{"code":"custom","path":["secrets"],"message":"Secret reference pw is not declared"}] ``` The same fixture through CLI returned the following JSON and exited 1: ```json {"error":"Browser scenario capture failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"capture_browser_scenario","issues":[{"path":["secrets"],"reason":"invalid_value","message":"Secret reference pw is not declared"}]}} ``` The duplicate-ID case returned MCP text `[{"code":"custom","path":["actions"],"message":"Action step IDs must be unique"}]`; CLI returned the same typed classification with an actions issue path. The unique-ID control returned a canonical MCP `error` object with code/category `execution_failure`. Pings after the cases returned `{}`, and server/CLI stderr was empty in the comparison. ### Typed outcome or recovery expected Repeat the undeclared-secret request: it should still be rejected with `isError: true`, but its text should contain the canonical MCP error object with `invalid_request` / `invalid_input`, operation `capture_browser_scenario`, the secrets issue path, and correction guidance. Duplicate action IDs should preserve the actions issue path through the same typed outcome. Preserve omission of error `structuredContent` and post-error ping. A scenario without those cross-field violations should continue beyond input validation, as the unavailable-executable control did. ### Analysis or suspected cause Source-derived explanation: `src/contracts/browserScenarioToolContracts.ts:31` advertises `browserScenarioInputSchema`, while `src/domain/browserScenario.ts:134` adds cross-field refinements through `superRefine`. The MCP handler invokes that refined schema's throwing parser before admission/result delivery. The observed exception becomes SDK error text. CLI explicitly projects refined validation failures into AnalysisInputError. This is an `isError` tool result, not a JSON-RPC internal error. ### Additional context No matching report was found in open/closed issue and PR searches for `capture_browser_scenario`, `capture-browser-scenario`, `ZodError`, `zod`, `input validation`, and `typed error`. The oversized SDK validation diagnostic in #3221 and lost comparison refinements in #1364 concern different failure paths. No last-known-good release was established.

bsakaguchi closed this issue

2026-10-17 21:04:22 +00:00

bsakaguchi referenced this issue

2026-10-31 01:04:51 +00:00

[PR #2416] [MERGED] test(perf): distinguish Codex tool context and token accounting #5211