morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6449] [Bug] compare-web-captures reports WebMCP changes from capture-local frame IDs #3862

New issue

Closed

opened 2026-10-31 00:44:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:44:44 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6449

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

node scripts/rea.mjs compare-web-captures before.json after.json --format json reports one WebMCP tool removed and one added when the registration facts are identical and only the capture-local frame ID and derived tool_key change. The WebMCP dimension becomes changed, making overall_status changed as well. The DOM frame comparison correctly remains unchanged.

This was observed with synthetic normalized capture files accepted by the public CLI. The forward case reproduced twice and the reverse case once; a live pair of browser sessions was not captured.

Reproduction

From a fresh source checkout, prepare the baseline with its pinned toolchain:

git checkout 9eb829aca34be36ff112826337996f6139f82eff
npm ci && npm run build:cached
export WORK="$(mktemp -d)"
mkdir -p "$WORK/home"
export HOME="$WORK/home"

The verification used an already prepared checkout; the dependency/build setup above was not rerun. Run the following inline fixture from the checkout root. It validates every input with the production snapshot schema and computes both the producer's frame-derived key and canonical schema digest. The URLs are inert recorded identities; no browser or service runs.

node --input-type=module <<'JS'
import { writeFileSync } from 'node:fs';
import { createHash } from 'node:crypto';
import { pathToFileURL } from 'node:url';
const load = p => import(pathToFileURL(`${process.cwd()}/dist/domain/${p}.js`));
const { captureSnapshotSchema } = await load('webCaptureDiffSchemas');
const { digestCanonicalValue } = await load('canonicalDigest');
const hash = v => createHash('sha256').update(JSON.stringify(v)).digest('hex');
const origin = 'https://app.example.test', url = `${origin}/`;
const at = new Date(0).toISOString();
const browser = { product: 'fixture', protocol_version: '1.3', revision: 'fixture', user_agent: 'fixture', js_version: 'fixture' };
const completeness = {
  status: 'complete_within_window', conditions: ['complete_within_window'],
  policy_filtered_sections: [], attach_limited_sections: [], truncated_sections: [], unavailable_sections: [], excluded: [],
  dropped_events: { scripts: 0, network_requests: 0, console_events: 0, websocket_connections: 0, websocket_frames: 0, webmcp_tools: 0, timeline_events: 0, total: 0 },
};
function snapshot(frameId, inputSchema = { type: 'object', properties: { q: { type: 'string' } } }, removed = false) {
  const tool = {
    tool_key: `webmcp_${hash([url, frameId, 'search'])}`, name: 'search', description: 'Search',
    frame_id: frameId, frame_url: url, owner_origin: origin, declaration_kind: 'declarative',
    input_schema: inputSchema, input_schema_sha256: digestCanonicalValue(inputSchema),
    annotations: { read_only: true, untrusted_content: null, consequential: false, autosubmit: null },
    registration_source: { url: `${origin}/app.js`, line: 3, column: 1 }, trust: 'page-declared-untrusted',
  };
  return captureSnapshotSchema.parse({
    inspection: {
      browser, target: { target_id: 'page', type: 'page', title: 'Fixture', url, origin, attached: false },
      capture_window: { started_at: at, ended_at: at, observation_ms: 0 }, completeness,
      network: { requests: [], websocket_connections: [], coverage_started_at: at, prior_activity_available: false },
      frames: [{ frame_id: frameId, parent_frame_id: null, url, origin }], dom: { total_nodes: 0, nodes: [] },
      accessibility: { total_nodes: 0, nodes: [], text_capture: { status: 'not_approved', retained_bytes: 0, excluded_fields: 0 } },
      scripts: { total: 0, items: [] }, resources: [], workers: [],
      console: { events: [], coverage_started_at: at, prior_activity_available: false },
      metadata: { responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true },
      storage: { origin, usage_bytes: null, quota_bytes: null, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], values_redacted: true },
      limitations: ['Synthetic fixture'],
    },
    webmcp: { browser, target: { target_id: 'page', url, origin }, status: 'available',
      tools: { total: removed ? 0 : 1, items: removed ? [] : [tool] }, completeness, limitations: ['Synthetic fixture'] },
  });
}
const captures = {
  before: snapshot('frame-A'), after: snapshot('frame-B'),
  same: snapshot('frame-A'), schema: snapshot('frame-A', { type: 'string' }),
  removed: snapshot('frame-A', null, true),
};
for (const [name, value] of Object.entries(captures))
  writeFileSync(`${process.env.WORK}/${name}.json`, JSON.stringify(value));
JS
python3 <<'PYTHON'
import os, subprocess, json
work = os.environ['WORK']
for label, left, right in [
    ('frame ID changed', 'before', 'after'),
    ('repeat', 'before', 'after'),
    ('same ID', 'before', 'same'),
    ('schema changed', 'before', 'schema'),
    ('tool removed', 'before', 'removed'),
    ('reverse', 'after', 'before'),
]:
    result = subprocess.run([
        'node', 'scripts/rea.mjs', 'compare-web-captures',
        f'{work}/{left}.json', f'{work}/{right}.json', '--format', 'json'
    ], text=True, capture_output=True, check=True)
    value = json.loads(result.stdout)['normalized_result']
    print(label, json.dumps({
        'exit': result.returncode, 'overall_status': value['overall_status'],
        'webmcp': value['dimensions']['webmcp']
    }))
PYTHON

Expected behavior

A change in capture-local frame identity alone should not prove that a page registration was removed and another added. Compare stable registration facts where correspondence is established, or return an explicit unknown identity result where it cannot be established. Preserve the separate registrations and counts of distinct frame owners.

The discovery contract explicitly says tool_key is unstable across navigation/browser sessions. The identity invariant requires unresolved identity to remain unknown instead of being treated as absent or aligned. The comparison result contract says changed proves an observed difference. The related DOM frame contract already excludes capture-local frame IDs as evidence of change.

Impact and scope

P2: this bounded passive-comparison defect gives a false WebMCP addition/removal result and promotes the overall comparison to changed. Stored captures remain intact. For this single-registration pair, direct comparison of every tool field except frame_id and tool_key was observed to establish equality and provides a manual workaround. That does not establish an alignment method for arbitrary duplicate frames.

All six CLI comparisons exited 0. The same-ID control returns WebMCP unchanged; a real schema change returns one modified change; removing the registration returns one removed change. Browser-session frequency, other providers, and MCP transport were not tested.

REA version

6.7.0 (source 9eb829aca3)

Runtime and platform

Node.js v26.12.1; macOS 27.0.1; arm64. The checkout pins Node.js 24.18.0; this observation used the stated installed runtime, so pin-matched reproduction is not claimed.

Target and artifact context

Schema-validated synthetic normalized passive capture JSON: one page frame and one WebMCP registration, with complete WebMCP coverage and no dropped events. Name, description, frame URL, origin, declaration kind, input schema/digest, annotations and declaration source remain identical between the frame-ID variants.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed summary from the six commands above:

frame ID changed: overall=changed, webmcp=changed,   changes=added + removed
repeat:           overall=changed, webmcp=changed,   changes=added + removed
same ID:          overall=unknown, webmcp=unchanged, changes=0
schema changed:   overall=changed, webmcp=changed,   changes=modified
tool removed:     overall=changed, webmcp=changed,   changes=removed
reverse:          overall=changed, webmcp=changed,   changes=removed + added

For the forward case, dimensions.webmcp has status: "changed", total_changes: 2, reason: null, and these changes:

[
  {"identity":"webmcp_1e0eaba8c395f6c83423c4044cd00c4a84c286aaadf88f67fc7b36eb44f16886","change":"added"},
  {"identity":"webmcp_be402495cffc5726cc1d7a54cd1584acecc21ffe9d915c2c5670f377aa1a71c3","change":"removed"}
]

Every other dimension has the same status in the forward and same-ID controls. Accessibility and storage are unknown in both; this is why the same-ID control's overall result is unknown rather than unchanged.

Typed outcome or recovery expected

The frame-ID-only case must avoid a proven WebMCP change. Return unchanged after justified stable correspondence, or unknown with an identity limitation. Keep real schema differences and inventory removals reportable, and retain multiplicity for separate frame registrations.

Analysis or suspected cause

Source-derived: webMcpDimension matches registrations by tool_key. Its projection retains the complete tool object; keyed hashes that object, including transient identity fields. The producer hashes frame URL, frame ID and name into tool_key.

Both matching and compared content therefore retain capture-local identity. Changing only the map key while retaining frame_id and tool_key in its content fingerprint would not address the complete cause.

Additional context

Related merged PR #813 preserves separate same-URL registration owners in discovery and explicitly documents key instability. That fix must remain intact; this report concerns consuming those keys across captures. #3989 concerns false equality after dropped observations; #5736 concerns initial subframe replay coverage. Neither describes this complete-inventory false change.

Last known good version: NOT_ENOUGH_INFO; none established. No fix or restored runtime result is claimed.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6449 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `node scripts/rea.mjs compare-web-captures before.json after.json --format json` reports one WebMCP tool removed and one added when the registration facts are identical and only the capture-local frame ID and derived `tool_key` change. The WebMCP dimension becomes `changed`, making `overall_status` `changed` as well. The DOM frame comparison correctly remains `unchanged`. This was observed with synthetic normalized capture files accepted by the public CLI. The forward case reproduced twice and the reverse case once; a live pair of browser sessions was not captured. ### Reproduction From a fresh source checkout, prepare the baseline with its pinned toolchain: ```sh git checkout 9eb829aca34be36ff112826337996f6139f82eff npm ci && npm run build:cached export WORK="$(mktemp -d)" mkdir -p "$WORK/home" export HOME="$WORK/home" ``` The verification used an already prepared checkout; the dependency/build setup above was not rerun. Run the following inline fixture from the checkout root. It validates every input with the production snapshot schema and computes both the producer's frame-derived key and canonical schema digest. The URLs are inert recorded identities; no browser or service runs. ```sh node --input-type=module <<'JS' import { writeFileSync } from 'node:fs'; import { createHash } from 'node:crypto'; import { pathToFileURL } from 'node:url'; const load = p => import(pathToFileURL(`${process.cwd()}/dist/domain/${p}.js`)); const { captureSnapshotSchema } = await load('webCaptureDiffSchemas'); const { digestCanonicalValue } = await load('canonicalDigest'); const hash = v => createHash('sha256').update(JSON.stringify(v)).digest('hex'); const origin = 'https://app.example.test', url = `${origin}/`; const at = new Date(0).toISOString(); const browser = { product: 'fixture', protocol_version: '1.3', revision: 'fixture', user_agent: 'fixture', js_version: 'fixture' }; const completeness = { status: 'complete_within_window', conditions: ['complete_within_window'], policy_filtered_sections: [], attach_limited_sections: [], truncated_sections: [], unavailable_sections: [], excluded: [], dropped_events: { scripts: 0, network_requests: 0, console_events: 0, websocket_connections: 0, websocket_frames: 0, webmcp_tools: 0, timeline_events: 0, total: 0 }, }; function snapshot(frameId, inputSchema = { type: 'object', properties: { q: { type: 'string' } } }, removed = false) { const tool = { tool_key: `webmcp_${hash([url, frameId, 'search'])}`, name: 'search', description: 'Search', frame_id: frameId, frame_url: url, owner_origin: origin, declaration_kind: 'declarative', input_schema: inputSchema, input_schema_sha256: digestCanonicalValue(inputSchema), annotations: { read_only: true, untrusted_content: null, consequential: false, autosubmit: null }, registration_source: { url: `${origin}/app.js`, line: 3, column: 1 }, trust: 'page-declared-untrusted', }; return captureSnapshotSchema.parse({ inspection: { browser, target: { target_id: 'page', type: 'page', title: 'Fixture', url, origin, attached: false }, capture_window: { started_at: at, ended_at: at, observation_ms: 0 }, completeness, network: { requests: [], websocket_connections: [], coverage_started_at: at, prior_activity_available: false }, frames: [{ frame_id: frameId, parent_frame_id: null, url, origin }], dom: { total_nodes: 0, nodes: [] }, accessibility: { total_nodes: 0, nodes: [], text_capture: { status: 'not_approved', retained_bytes: 0, excluded_fields: 0 } }, scripts: { total: 0, items: [] }, resources: [], workers: [], console: { events: [], coverage_started_at: at, prior_activity_available: false }, metadata: { responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true }, storage: { origin, usage_bytes: null, quota_bytes: null, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], values_redacted: true }, limitations: ['Synthetic fixture'], }, webmcp: { browser, target: { target_id: 'page', url, origin }, status: 'available', tools: { total: removed ? 0 : 1, items: removed ? [] : [tool] }, completeness, limitations: ['Synthetic fixture'] }, }); } const captures = { before: snapshot('frame-A'), after: snapshot('frame-B'), same: snapshot('frame-A'), schema: snapshot('frame-A', { type: 'string' }), removed: snapshot('frame-A', null, true), }; for (const [name, value] of Object.entries(captures)) writeFileSync(`${process.env.WORK}/${name}.json`, JSON.stringify(value)); JS python3 <<'PYTHON' import os, subprocess, json work = os.environ['WORK'] for label, left, right in [ ('frame ID changed', 'before', 'after'), ('repeat', 'before', 'after'), ('same ID', 'before', 'same'), ('schema changed', 'before', 'schema'), ('tool removed', 'before', 'removed'), ('reverse', 'after', 'before'), ]: result = subprocess.run([ 'node', 'scripts/rea.mjs', 'compare-web-captures', f'{work}/{left}.json', f'{work}/{right}.json', '--format', 'json' ], text=True, capture_output=True, check=True) value = json.loads(result.stdout)['normalized_result'] print(label, json.dumps({ 'exit': result.returncode, 'overall_status': value['overall_status'], 'webmcp': value['dimensions']['webmcp'] })) PYTHON ``` ### Expected behavior A change in capture-local frame identity alone should not prove that a page registration was removed and another added. Compare stable registration facts where correspondence is established, or return an explicit unknown identity result where it cannot be established. Preserve the separate registrations and counts of distinct frame owners. The [discovery contract](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/browser-observation.md#L21) explicitly says `tool_key` is unstable across navigation/browser sessions. The [identity invariant](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/AGENTS.md#L21) requires unresolved identity to remain unknown instead of being treated as absent or aligned. The [comparison result contract](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L44) says `changed` proves an observed difference. The related [DOM frame contract](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/browser-observation.md#L47) already excludes capture-local frame IDs as evidence of change. ### Impact and scope P2: this bounded passive-comparison defect gives a false WebMCP addition/removal result and promotes the overall comparison to `changed`. Stored captures remain intact. For this single-registration pair, direct comparison of every tool field except `frame_id` and `tool_key` was observed to establish equality and provides a manual workaround. That does not establish an alignment method for arbitrary duplicate frames. All six CLI comparisons exited 0. The same-ID control returns WebMCP `unchanged`; a real schema change returns one `modified` change; removing the registration returns one `removed` change. Browser-session frequency, other providers, and MCP transport were not tested. ### REA version 6.7.0 (source 9eb829aca34be36ff112826337996f6139f82eff) ### Runtime and platform Node.js v26.12.1; macOS 27.0.1; arm64. The checkout pins Node.js 24.18.0; this observation used the stated installed runtime, so pin-matched reproduction is not claimed. ### Target and artifact context Schema-validated synthetic normalized passive capture JSON: one page frame and one WebMCP registration, with complete WebMCP coverage and no dropped events. Name, description, frame URL, origin, declaration kind, input schema/digest, annotations and declaration source remain identical between the frame-ID variants. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed summary from the six commands above: ```text frame ID changed: overall=changed, webmcp=changed, changes=added + removed repeat: overall=changed, webmcp=changed, changes=added + removed same ID: overall=unknown, webmcp=unchanged, changes=0 schema changed: overall=changed, webmcp=changed, changes=modified tool removed: overall=changed, webmcp=changed, changes=removed reverse: overall=changed, webmcp=changed, changes=removed + added ``` For the forward case, `dimensions.webmcp` has `status: "changed"`, `total_changes: 2`, `reason: null`, and these changes: ```json [ {"identity":"webmcp_1e0eaba8c395f6c83423c4044cd00c4a84c286aaadf88f67fc7b36eb44f16886","change":"added"}, {"identity":"webmcp_be402495cffc5726cc1d7a54cd1584acecc21ffe9d915c2c5670f377aa1a71c3","change":"removed"} ] ``` Every other dimension has the same status in the forward and same-ID controls. Accessibility and storage are `unknown` in both; this is why the same-ID control's overall result is `unknown` rather than `unchanged`. ### Typed outcome or recovery expected The frame-ID-only case must avoid a proven WebMCP change. Return `unchanged` after justified stable correspondence, or `unknown` with an identity limitation. Keep real schema differences and inventory removals reportable, and retain multiplicity for separate frame registrations. ### Analysis or suspected cause Source-derived: [webMcpDimension](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L704) matches registrations by `tool_key`. Its [projection](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L723) retains the complete tool object; [keyed](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L220) hashes that object, including transient identity fields. The [producer](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/browser/CdpWebMcpDiscovery.ts#L556) hashes frame URL, frame ID and name into `tool_key`. Both matching and compared content therefore retain capture-local identity. Changing only the map key while retaining `frame_id` and `tool_key` in its content fingerprint would not address the complete cause. ### Additional context Related merged PR #813 preserves separate same-URL registration owners in discovery and explicitly documents key instability. That fix must remain intact; this report concerns consuming those keys across captures. #3989 concerns false equality after dropped observations; #5736 concerns initial subframe replay coverage. Neither describes this complete-inventory false change. Last known good version: NOT_ENOUGH_INFO; none established. No fix or restored runtime result is claimed.

bsakaguchi closed this issue

2026-10-31 00:44:44 +00:00

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:42:56 +00:00

[PR #5115] [CLOSED] fix(setup): accept and preserve Codex per-tool settings (#3862, #3863) #6771