morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3989] [Bug] Dropped WebMCP observations still compare as unchanged #1582

New issue

Closed

opened 2026-10-16 21:07:26 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:07:26 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3989

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The supported offline compare-web-captures command reports the WebMCP dimension as unchanged when one normalized capture explicitly records a dropped WebMCP observation but has no corresponding section-list marker. The retained tool inventories are empty, so the result asserts equality despite known missing evidence. This reproduced with the incomplete capture on either side, and also when its reported tool total was reset to match the retained count.

The baseline run observed all six child CLI calls exit successfully with code 0. The omission case, reversed omission case, and drop-count-only case each returned this WebMCP dimension:

{"total_changes":0,"changes":[],"status":"unchanged","reason":null}

The explicit-section control returned:

{"total_changes":0,"changes":[],"status":"unknown","reason":"WebMCP discovery was unavailable or incomplete in at least one capture."}

Complete empty inventories and the unrelated network-drop control both returned unchanged. Each comparison ran once. The focused acceptance probe failed at its final assertion with expected 'unchanged' to be 'unknown'; its preceding control assertions passed. This establishes the WebMCP dimension's false equality result, not that the whole comparison necessarily reports overall equality.

Reproduction

Use the source baseline under Runtime or environment with its locked dependencies and compiled runtime already prepared. No live browser or provider connection is needed. These are synthetic normalized fixtures accepted by the production CLI, rather than captured browser sessions.

Save the following probe as tests/acceptance/browser/captureCoverage.test.ts.

import { it, expect } from "vitest";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { passiveNetworkCapture, withPassiveDroppedEvents } from "../../fixtures/passiveNetworkCapture.js";

const exec = promisify(execFile);
it("preserves incomplete WebMCP coverage through the compiled comparison CLI", async () => {
  const root = process.env.REA_VERIFY_SCRATCH!;
  const inspection = passiveNetworkCapture([]);
  const complete = {
    browser: inspection.browser,
    target: { target_id: inspection.target.target_id, url: inspection.target.url, origin: inspection.target.origin },
    status: "available", tools: { total: 0, items: [] },
    completeness: inspection.completeness, limitations: [],
  };
  const omitted = structuredClone(complete);
  omitted.tools.total = 1;
  omitted.completeness = withPassiveDroppedEvents(inspection, { webmcp_tools: 1 }).completeness;
  const dropOnly = structuredClone(omitted);
  dropOnly.tools.total = 0;
  const explicit = structuredClone(omitted);
  explicit.completeness.truncated_sections.push("webmcp_tools");
  const unrelated = structuredClone(complete);
  unrelated.completeness = withPassiveDroppedEvents(inspection, { network_requests: 1 }).completeness;
  const inputs = { complete, omitted, dropOnly, explicit, unrelated };
  for (const [name, webmcp] of Object.entries(inputs)) {
    await writeFile(join(root, `${name}.json`), JSON.stringify({ inspection, webmcp }, null, 2));
  }
  const results: Record<string, unknown> = {};
  for (const [name, before, after] of [
    ["omission", "complete", "omitted"],
    ["reverseOmission", "omitted", "complete"],
    ["dropOnly", "complete", "dropOnly"],
    ["explicitSection", "complete", "explicit"],
    ["complete", "complete", "complete"],
    ["unrelatedDrop", "complete", "unrelated"],
  ]) {
    const result = await exec(process.execPath, ["scripts/rea.mjs", "compare-web-captures", join(root, `${before}.json`), join(root, `${after}.json`), "--json"], { env: { ...process.env, HOME: join(root, "home") } });
    const envelope = JSON.parse(result.stdout);
    results[name!] = { exitCode: 0, webmcp: envelope.normalized_result.dimensions.webmcp };
    await writeFile(join(root, `${name}.stdout.json`), result.stdout);
  }
  await writeFile(join(root, "results.json"), JSON.stringify(results, null, 2));
  console.log(JSON.stringify(results, null, 2));
  expect((results.explicitSection as any).webmcp.status).toBe("unknown");
  expect((results.complete as any).webmcp.status).toBe("unchanged");
  expect((results.unrelatedDrop as any).webmcp.status).toBe("unchanged");
  expect((results.omission as any).webmcp.status).toBe("unknown");
}, 30_000);

Run the probe using the repository's pinned Node.js. The directory variable gives every child CLI call an isolated home directory. The test should fail at the final assertion on the affected baseline, after recording all six comparisons.

export REA_VERIFY_SCRATCH="$(mktemp -d)"
mkdir -p "$REA_VERIFY_SCRATCH/home"
npx --no-install vitest run tests/acceptance/browser/captureCoverage.test.ts
cat "$REA_VERIFY_SCRATCH/results.json"
rm tests/acceptance/browser/captureCoverage.test.ts

The baseline verification executed this probe with an assigned disposable directory and a temporary test filename. The fixture data, child CLI command, six comparisons, and assertions were as shown above.

Expected behavior

A positive recorded WebMCP drop count must prevent an equality conclusion when no retained difference establishes a change. For these empty retained inventories, the WebMCP dimension should report unknown whether the incomplete input is before or after. The comparison contract expressly says missing or truncated evidence never proves equality. A redundant section-list marker should not be necessary to preserve the recorded omission.

Impact and scope

The supported offline comparison gives an incorrect WebMCP equality result for captures that explicitly record missing WebMCP observations. The demonstrated reach is synthetic normalized inputs, with no established real-provider frequency or data loss.

Workaround status: available. The executed control returns unknown after a derived copy also records webmcp_tools in its truncated sections. This preserves the existing drop evidence and leaves the original capture unchanged.

REA version

rea 6.5.0 at source commit 4692bb3330

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Synthetic normalized passive capture JSON accepted by the offline compare-web-captures command; live emission of this marker combination was not tested.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: src/contracts/browserToolContracts.ts:238 and docs/browser-observation.md:22 promise that missing or truncated evidence never establishes equality; docs/mcp-contracts.md:787 documents the passive normalized input family used here.
Failure source: Executed compiled CLI compare-web-captures on the synthetic normalized fixtures above; src/domain/webCaptureDiff.ts:728 checks discovery availability and section lists without consulting the WebMCP dropped-event count.

Source-derived evidence explains the accepted completeness shape. src/domain/browserCompleteness.ts:100 classifies any positive dropped-event count as truncation and preserves the supplied section lists independently. The fixture helper uses that production classifier. The probe records status: "truncated", conditions: ["truncated"], one WebMCP drop and a matching total drop count, with empty section lists.

Live discovery emission of this exact count/section combination was not tested. The verified reach is caller-supplied normalized capture files through the supported offline CLI.

Typed outcome or recovery expected

Return unknown for the WebMCP dimension when recorded WebMCP drops prevent equality. Keep complete empty inventories and unrelated network-drop controls unchanged.

Analysis or suspected cause

Source-derived: src/domain/webCaptureDiff.ts:728 checks discovery availability and section lists without consulting the WebMCP dropped-event count. src/domain/browserCompleteness.ts:100,125-130 classifies a positive count as truncation independently of section lists.

Additional context

Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision.

Restoration check: Repeat the six comparisons. The omission, reversed omission, and drop-count-only cases must preserve unknown WebMCP coverage instead of reporting equality. The explicit-section case must stay unknown; complete empty inventories and an unrelated network drop must stay unchanged. Preserve the original captures when applying or evaluating a workaround. This check concerns known missing WebMCP evidence and does not require rejecting the entire normalized input family.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3989 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The supported offline `compare-web-captures` command reports the WebMCP dimension as `unchanged` when one normalized capture explicitly records a dropped WebMCP observation but has no corresponding section-list marker. The retained tool inventories are empty, so the result asserts equality despite known missing evidence. This reproduced with the incomplete capture on either side, and also when its reported tool total was reset to match the retained count. The baseline run observed all six child CLI calls exit successfully with code 0. The omission case, reversed omission case, and drop-count-only case each returned this WebMCP dimension: ```json {"total_changes":0,"changes":[],"status":"unchanged","reason":null} ``` The explicit-section control returned: ```json {"total_changes":0,"changes":[],"status":"unknown","reason":"WebMCP discovery was unavailable or incomplete in at least one capture."} ``` Complete empty inventories and the unrelated network-drop control both returned `unchanged`. Each comparison ran once. The focused acceptance probe failed at its final assertion with `expected 'unchanged' to be 'unknown'`; its preceding control assertions passed. This establishes the WebMCP dimension's false equality result, not that the whole comparison necessarily reports overall equality. ### Reproduction Use the source baseline under Runtime or environment with its locked dependencies and compiled runtime already prepared. No live browser or provider connection is needed. These are synthetic normalized fixtures accepted by the production CLI, rather than captured browser sessions. 1. Save the following probe as `tests/acceptance/browser/captureCoverage.test.ts`. ```ts import { it, expect } from "vitest"; import { execFile } from "node:child_process"; import { promisify } from "node:util"; import { writeFile } from "node:fs/promises"; import { join } from "node:path"; import { passiveNetworkCapture, withPassiveDroppedEvents } from "../../fixtures/passiveNetworkCapture.js"; const exec = promisify(execFile); it("preserves incomplete WebMCP coverage through the compiled comparison CLI", async () => { const root = process.env.REA_VERIFY_SCRATCH!; const inspection = passiveNetworkCapture([]); const complete = { browser: inspection.browser, target: { target_id: inspection.target.target_id, url: inspection.target.url, origin: inspection.target.origin }, status: "available", tools: { total: 0, items: [] }, completeness: inspection.completeness, limitations: [], }; const omitted = structuredClone(complete); omitted.tools.total = 1; omitted.completeness = withPassiveDroppedEvents(inspection, { webmcp_tools: 1 }).completeness; const dropOnly = structuredClone(omitted); dropOnly.tools.total = 0; const explicit = structuredClone(omitted); explicit.completeness.truncated_sections.push("webmcp_tools"); const unrelated = structuredClone(complete); unrelated.completeness = withPassiveDroppedEvents(inspection, { network_requests: 1 }).completeness; const inputs = { complete, omitted, dropOnly, explicit, unrelated }; for (const [name, webmcp] of Object.entries(inputs)) { await writeFile(join(root, `${name}.json`), JSON.stringify({ inspection, webmcp }, null, 2)); } const results: Record<string, unknown> = {}; for (const [name, before, after] of [ ["omission", "complete", "omitted"], ["reverseOmission", "omitted", "complete"], ["dropOnly", "complete", "dropOnly"], ["explicitSection", "complete", "explicit"], ["complete", "complete", "complete"], ["unrelatedDrop", "complete", "unrelated"], ]) { const result = await exec(process.execPath, ["scripts/rea.mjs", "compare-web-captures", join(root, `${before}.json`), join(root, `${after}.json`), "--json"], { env: { ...process.env, HOME: join(root, "home") } }); const envelope = JSON.parse(result.stdout); results[name!] = { exitCode: 0, webmcp: envelope.normalized_result.dimensions.webmcp }; await writeFile(join(root, `${name}.stdout.json`), result.stdout); } await writeFile(join(root, "results.json"), JSON.stringify(results, null, 2)); console.log(JSON.stringify(results, null, 2)); expect((results.explicitSection as any).webmcp.status).toBe("unknown"); expect((results.complete as any).webmcp.status).toBe("unchanged"); expect((results.unrelatedDrop as any).webmcp.status).toBe("unchanged"); expect((results.omission as any).webmcp.status).toBe("unknown"); }, 30_000); ``` 2. Run the probe using the repository's pinned Node.js. The directory variable gives every child CLI call an isolated home directory. The test should fail at the final assertion on the affected baseline, after recording all six comparisons. ```sh export REA_VERIFY_SCRATCH="$(mktemp -d)" mkdir -p "$REA_VERIFY_SCRATCH/home" npx --no-install vitest run tests/acceptance/browser/captureCoverage.test.ts cat "$REA_VERIFY_SCRATCH/results.json" rm tests/acceptance/browser/captureCoverage.test.ts ``` The baseline verification executed this probe with an assigned disposable directory and a temporary test filename. The fixture data, child CLI command, six comparisons, and assertions were as shown above. ### Expected behavior A positive recorded WebMCP drop count must prevent an equality conclusion when no retained difference establishes a change. For these empty retained inventories, the WebMCP dimension should report `unknown` whether the incomplete input is before or after. The comparison contract expressly says missing or truncated evidence never proves equality. A redundant section-list marker should not be necessary to preserve the recorded omission. ### Impact and scope The supported offline comparison gives an incorrect WebMCP equality result for captures that explicitly record missing WebMCP observations. The demonstrated reach is synthetic normalized inputs, with no established real-provider frequency or data loss. Workaround status: available. The executed control returns unknown after a derived copy also records `webmcp_tools` in its truncated sections. This preserves the existing drop evidence and leaves the original capture unchanged. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Synthetic normalized passive capture JSON accepted by the offline compare-web-captures command; live emission of this marker combination was not tested. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence - Expected source: `src/contracts/browserToolContracts.ts:238` and `docs/browser-observation.md:22` promise that missing or truncated evidence never establishes equality; `docs/mcp-contracts.md:787` documents the passive normalized input family used here. - Failure source: Executed compiled CLI `compare-web-captures` on the synthetic normalized fixtures above; `src/domain/webCaptureDiff.ts:728` checks discovery availability and section lists without consulting the WebMCP dropped-event count. Source-derived evidence explains the accepted completeness shape. `src/domain/browserCompleteness.ts:100` classifies any positive dropped-event count as truncation and preserves the supplied section lists independently. The fixture helper uses that production classifier. The probe records `status: "truncated"`, `conditions: ["truncated"]`, one WebMCP drop and a matching total drop count, with empty section lists. Live discovery emission of this exact count/section combination was not tested. The verified reach is caller-supplied normalized capture files through the supported offline CLI. ### Typed outcome or recovery expected Return unknown for the WebMCP dimension when recorded WebMCP drops prevent equality. Keep complete empty inventories and unrelated network-drop controls unchanged. ### Analysis or suspected cause Source-derived: src/domain/webCaptureDiff.ts:728 checks discovery availability and section lists without consulting the WebMCP dropped-event count. src/domain/browserCompleteness.ts:100,125-130 classifies a positive count as truncation independently of section lists. ### Additional context Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750cbe16755c4f78c1f90fdf09fbaca31. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision. Restoration check: Repeat the six comparisons. The omission, reversed omission, and drop-count-only cases must preserve unknown WebMCP coverage instead of reporting equality. The explicit-section case must stay `unknown`; complete empty inventories and an unrelated network drop must stay `unchanged`. Preserve the original captures when applying or evaluating a workaround. This check concerns known missing WebMCP evidence and does not require rejecting the entire normalized input family.

bsakaguchi commented

2026-10-16 21:07:27 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

Confirmed: a recorded dropped WebMCP observation means the dimension was not fully captured, so claiming unchanged overstates confidence. The compare should treat an explicit drop record like the section-list marker path and report unknown with the incomplete-discovery reason, reserving unchanged for genuinely complete empty inventories.

@XinwangTao commented on GitHub (Oct 15, 2026): Confirmed: a recorded dropped WebMCP observation means the dimension was not fully captured, so claiming unchanged overstates confidence. The compare should treat an explicit drop record like the section-list marker path and report unknown with the incomplete-discovery reason, reserving unchanged for genuinely complete empty inventories.

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:12:35 +00:00

[PR #1582] [MERGED] feat(flutter): identify Dart snapshot builds and inspect AOT payloads by pure APK parsing #2987

bsakaguchi closed this issue

2026-10-31 00:31:36 +00:00

bsakaguchi commented

2026-10-31 00:31:37 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

webMcpComplete now independently requires dropped_events.webmcp_tools === 0 as well as available status and no incomplete section marker, so a positive drop count cannot imply complete coverage even when tools.total matches retained rows and section lists are empty. With no retained differences, compareDimension returns unknown for incomplete inputs. The existing acceptance case injects a drop count using the production completeness helper without adding a section marker, and expects unknown, exercised by the shared stdio MCP case loop.

Implementation: webCaptureDiff.ts, webCaptureDiff.ts.

The relevant regression checks in passiveCaptureCorrespondence.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. webMcpComplete now independently requires dropped_events.webmcp_tools === 0 as well as available status and no incomplete section marker, so a positive drop count cannot imply complete coverage even when tools.total matches retained rows and section lists are empty. With no retained differences, compareDimension returns unknown for incomplete inputs. The existing acceptance case injects a drop count using the production completeness helper without adding a section marker, and expects unknown, exercised by the shared stdio MCP case loop. Implementation: [webCaptureDiff.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/webCaptureDiff.ts#L912), [webCaptureDiff.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/webCaptureDiff.ts#L898). The relevant regression checks in [passiveCaptureCorrespondence.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/acceptance/tools/passiveCaptureCorrespondence.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:04:31 +00:00

[PR #2334] [MERGED] fix(flutter): reject ambiguous ELF mappings and bound note reads #5170

bsakaguchi referenced this issue

2026-10-31 01:06:24 +00:00

[PR #2804] [MERGED] fix(flutter): find snapshot headers that a read chunk boundary splits #5407