morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5897] [Docs] compare_web_captures example marks attached browser captures complete #3576

New issue

Closed

opened 2026-10-31 00:42:01 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5897

Before submitting

I searched the existing issues for a related documentation report.
I removed credentials, tokens, private paths, and sensitive target data.

Documentation location

The default MCP tools/list input example for compare_web_captures: example constructor and comparison contract example.

Problem

The compare_web_captures example returned by the live MCP tools/list catalog describes two attached-browser captures with browser.mode: "connect", completeness.status: "complete", equality_eligible: true, no missing sections, and limitations: [].

That example does not match the production capture contract: connected captures always declare missing event coverage and attachment limitations. Submitting the advertised example unchanged through MCP succeeds and returns overall_status: "unchanged". Changing only both capture-wide completeness records to the producer's incomplete state makes the comparison return unknown.

The defect is the advertised example. This does not show a real browser capture being incorrectly marked complete.

P3: the advertised example gives clients misleading guidance about attached capture coverage. The observed false completeness is confined to synthetic example metadata. The production provider's conservative completeness behavior passed its existing simulated-session test, and the incomplete-input comparison control correctly returned unknown. No consequential downstream behavior or incorrect real-browser equality was observed.

Clients can preserve the actual capture's completeness and limitations when submitting its normalized result for comparison.

Expected content

The example should describe a capture the normal producer can emit. An attached-browser example should retain incomplete, equality_eligible: false, missing events coverage, and the applicable limitations. A launch-mode example would need consistent mode, process ownership, cleanup, completeness, and limitations.

The production completeness rule always adds missing event coverage for connect mode. The capture projection supplies general and attachment-specific limitations. The comparison contract requires incomplete coverage to remain unknown rather than prove equality.

The catalog example should preserve the same coverage and limitation facts as the corresponding producer mode. Real incomplete captures should continue comparing as unknown when no observed difference establishes a changed result.

Supporting evidence

From an installed and built source checkout at the revision below, run this with the repository's locked dependencies. It starts the actual stdio MCP entry, obtains the example from the catalog, submits it unchanged, and submits an incomplete-coverage control. It closes the client afterward. No browser is started or attached.

node --input-type=module <<'JS'
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';

const client = new Client({ name: 'scenario-example-check', version: '1' });
const transport = new StdioClientTransport({
  command: process.execPath,
  args: ['scripts/rea.mjs', 'mcp'],
  cwd: process.cwd(),
  stderr: 'pipe',
});
try {
  await client.connect(transport);
  const tool = (await client.listTools()).tools.find(
    ({ name }) => name === 'compare_web_captures',
  );
  const example = tool.inputSchema.examples[0];
  console.log(JSON.stringify({
    browser: example.before_scenario.browser,
    completeness: example.before_scenario.completeness,
    limitations: example.before_scenario.limitations,
  }));
  const original = await client.callTool({ name: tool.name, arguments: example });
  console.log('advertised:', original.structuredContent.normalized_result.overall_status);
  const control = structuredClone(example);
  for (const capture of [control.before_scenario, control.after_scenario]) {
    capture.completeness = {
      status: 'incomplete', equality_eligible: false,
      missing_sections: ['events'], truncated_sections: [],
    };
  }
  const checked = await client.callTool({ name: tool.name, arguments: control });
  console.log('incomplete:', checked.structuredContent.normalized_result.overall_status);
} finally {
  await client.close();
}
JS

Both comparison calls were executed, in separate stdio sessions, with the results below. The script combines those calls for convenient replay.

Also ran the existing focused simulated-session provider test:

node node_modules/vitest/vitest.mjs run src/browser/PlaywrightBrowserScenarioProvider.test.ts -t 'disconnects external CDP sessions and marks pre-attach events missing' --maxWorkers=1

Result: one passed, thirty skipped. No real-browser capture was run.

The live catalog's example contained:

{
  "browser": {
    "mode": "connect",
    "process_ownership": "external",
    "cleanup": "disconnected-external",
    "product": "Chromium",
    "version": "149"
  },
  "completeness": {
    "status": "complete",
    "equality_eligible": true,
    "missing_sections": [],
    "truncated_sections": []
  },
  "limitations": []
}

Both MCP calls succeeded without isError. The unchanged advertised example returned overall_status: "unchanged"; the incomplete control returned overall_status: "unknown". Both aligned two steps with no alignment failures.

The example constructor supplies the inconsistent state, and the comparison contract publishes two copies as inputs. The existing provider regression test asserts the correct attached-capture state.

Source analysis: Source-derived: the example constructor reuses a complete coverage object for a connected capture. The schema metadata layer publishes these examples, while runtime capture separately enforces mode-specific completeness. Schema-valid synthetic metadata therefore reaches the catalog without matching the producer's coverage rules.

Additional context

Area: Documentation.

REA version: 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e, using its prepared build and locked dependencies.

Runtime and platform: macOS 27.0.1 arm64; Node.js v26.12.0. This supported runtime differs from the checkout's contributor toolchain pin of Node.js 24.18.0.

Target and artifact context: Synthetic browser scenario captures obtained directly from the default MCP catalog example. No external target, browser executable, or CDP endpoint was accessed.

Interface: MCP.

Earliest observed stage: Transport or result projection.

Fetched upstream main at c324d31b45fcab77e4ed7320ed60f0bf5a4ba188; the example, capture completeness/projection, and comparison files were unchanged from the tested baseline.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5897 ### Before submitting - [x] I searched the existing issues for a related documentation report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Documentation location The default MCP `tools/list` input example for `compare_web_captures`: [example constructor](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/browserToolContracts.ts#L57-L133) and [comparison contract example](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/browserToolContracts.ts#L235-L247). ### Problem The `compare_web_captures` example returned by the live MCP `tools/list` catalog describes two attached-browser captures with `browser.mode: "connect"`, `completeness.status: "complete"`, `equality_eligible: true`, no missing sections, and `limitations: []`. That example does not match the production capture contract: connected captures always declare missing event coverage and attachment limitations. Submitting the advertised example unchanged through MCP succeeds and returns `overall_status: "unchanged"`. Changing only both capture-wide completeness records to the producer's incomplete state makes the comparison return `unknown`. The defect is the advertised example. This does not show a real browser capture being incorrectly marked complete. P3: the advertised example gives clients misleading guidance about attached capture coverage. The observed false completeness is confined to synthetic example metadata. The production provider's conservative completeness behavior passed its existing simulated-session test, and the incomplete-input comparison control correctly returned `unknown`. No consequential downstream behavior or incorrect real-browser equality was observed. Clients can preserve the actual capture's completeness and limitations when submitting its normalized result for comparison. ### Expected content The example should describe a capture the normal producer can emit. An attached-browser example should retain `incomplete`, `equality_eligible: false`, missing `events` coverage, and the applicable limitations. A launch-mode example would need consistent mode, process ownership, cleanup, completeness, and limitations. The [production completeness rule](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightBrowserScenarioProvider.ts#L226-L239) always adds missing event coverage for connect mode. The [capture projection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/BrowserScenarioPartialObservation.ts#L34-L58) supplies general and attachment-specific limitations. The [comparison contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/browser-scenario-contract.md#L378-L419) requires incomplete coverage to remain unknown rather than prove equality. The catalog example should preserve the same coverage and limitation facts as the corresponding producer mode. Real incomplete captures should continue comparing as unknown when no observed difference establishes a changed result. ### Supporting evidence From an installed and built source checkout at the revision below, run this with the repository's locked dependencies. It starts the actual stdio MCP entry, obtains the example from the catalog, submits it unchanged, and submits an incomplete-coverage control. It closes the client afterward. No browser is started or attached. ```sh node --input-type=module <<'JS' import { Client } from '@modelcontextprotocol/client'; import { StdioClientTransport } from '@modelcontextprotocol/client/stdio'; const client = new Client({ name: 'scenario-example-check', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: ['scripts/rea.mjs', 'mcp'], cwd: process.cwd(), stderr: 'pipe', }); try { await client.connect(transport); const tool = (await client.listTools()).tools.find( ({ name }) => name === 'compare_web_captures', ); const example = tool.inputSchema.examples[0]; console.log(JSON.stringify({ browser: example.before_scenario.browser, completeness: example.before_scenario.completeness, limitations: example.before_scenario.limitations, })); const original = await client.callTool({ name: tool.name, arguments: example }); console.log('advertised:', original.structuredContent.normalized_result.overall_status); const control = structuredClone(example); for (const capture of [control.before_scenario, control.after_scenario]) { capture.completeness = { status: 'incomplete', equality_eligible: false, missing_sections: ['events'], truncated_sections: [], }; } const checked = await client.callTool({ name: tool.name, arguments: control }); console.log('incomplete:', checked.structuredContent.normalized_result.overall_status); } finally { await client.close(); } JS ``` Both comparison calls were executed, in separate stdio sessions, with the results below. The script combines those calls for convenient replay. Also ran the existing focused simulated-session provider test: ```sh node node_modules/vitest/vitest.mjs run src/browser/PlaywrightBrowserScenarioProvider.test.ts -t 'disconnects external CDP sessions and marks pre-attach events missing' --maxWorkers=1 ``` Result: one passed, thirty skipped. No real-browser capture was run. The live catalog's example contained: ```json { "browser": { "mode": "connect", "process_ownership": "external", "cleanup": "disconnected-external", "product": "Chromium", "version": "149" }, "completeness": { "status": "complete", "equality_eligible": true, "missing_sections": [], "truncated_sections": [] }, "limitations": [] } ``` Both MCP calls succeeded without `isError`. The unchanged advertised example returned `overall_status: "unchanged"`; the incomplete control returned `overall_status: "unknown"`. Both aligned two steps with no alignment failures. The [example constructor](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/browserToolContracts.ts#L57-L133) supplies the inconsistent state, and [the comparison contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/browserToolContracts.ts#L235-L247) publishes two copies as inputs. The [existing provider regression test](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightBrowserScenarioProvider.test.ts#L371-L394) asserts the correct attached-capture state. Source analysis: Source-derived: the example constructor reuses a complete coverage object for a connected capture. The schema metadata layer [publishes these examples](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/toolSchemaMetadata.ts#L61-L98), while runtime capture separately enforces mode-specific completeness. Schema-valid synthetic metadata therefore reaches the catalog without matching the producer's coverage rules. ### Additional context Area: Documentation. REA version: 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`, using its prepared build and locked dependencies. Runtime and platform: macOS 27.0.1 arm64; Node.js v26.12.0. This supported runtime differs from the checkout's contributor toolchain pin of Node.js 24.18.0. Target and artifact context: Synthetic browser scenario captures obtained directly from the default MCP catalog example. No external target, browser executable, or CDP endpoint was accessed. Interface: MCP. Earliest observed stage: Transport or result projection. Fetched upstream main at `c324d31b45fcab77e4ed7320ed60f0bf5a4ba188`; the example, capture completeness/projection, and comparison files were unchanged from the tested baseline.

bsakaguchi closed this issue

2026-10-31 00:42:01 +00:00

bsakaguchi commented

2026-10-31 00:42:02 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

The actual compare_web_captures scenario example now has connect/external ownership together with incomplete, equality_eligible=false, missing events, and attachment limitations. Its contract test parses the advertised example and actually passes it to compareBrowserCaptures, asserting unknown rather than equality. Step-local completeness remains distinct from overall missing attachment events; no real browser correctness claim is made.

Implementation: browserToolContracts.ts, browserToolContracts.test.ts.

The relevant regression checks in browserToolContracts.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. The actual compare_web_captures scenario example now has connect/external ownership together with incomplete, equality_eligible=false, missing events, and attachment limitations. Its contract test parses the advertised example and actually passes it to compareBrowserCaptures, asserting unknown rather than equality. Step-local completeness remains distinct from overall missing attachment events; no real browser correctness claim is made. Implementation: [browserToolContracts.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/contracts/browserToolContracts.ts#L146), [browserToolContracts.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/contracts/browserToolContracts.test.ts#L8). The relevant regression checks in [browserToolContracts.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/contracts/browserToolContracts.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3576

No description provided.

Rows
Columns