morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #2169] fix(browser): mismatched event-family coverage compares as unchanged #715

New issue

Closed

opened 2026-10-15 18:27:39 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:27:39 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2169

Summary

compare-web-captures returns overall_status: "unchanged" for complete launch-scenario captures taken with different capture.events selections when neither selected family emitted an event. In particular, selecting [] on one side and ["console"] on the other, or ["console"] versus ["page-errors"], produces no event artifact difference or coverage warning. This treats different observation coverage as equality, although the scenario comparison contract requires mismatched coverage to remain unknown.

The failure affects browser scenario comparison through the CLI and the shared workflow used by MCP. The earliest visible failure is comparison result projection. The target in the executed reproduction is a simulated browser session producing valid scenario capture artifacts; no actual browser page or application regression was observed.

Runtime or environment

Verified on source revision c22111eca2c4bc84a084f12711f405b3e7422a40 and the published rea-agents@6.4.0 package, using Node.js v24.21.0 on macOS 27.0.1 arm64. The source build was already prepared with npm run build:cached. The measured runtime satisfies the package's supported Node range; it is not the checkout's exact contributor toolchain pin. Both builds produced the same comparison results.

Steps to reproduce

Use either tested build as the package root. Save the following as verify-event-coverage.mjs, then run node verify-event-coverage.mjs /path/to/rea/package/root ./event-coverage-output. The script calls the production scenario capture provider and event collector with an in-memory session, writes their normalized_result captures, and executes the public comparison CLI. It does not open a browser or contact the example URL. The output directory retains the inputs and results for inspection.

import assert from "node:assert/strict";
import { EventEmitter } from "node:events";
import { mkdir, writeFile } from "node:fs/promises";
import { resolve } from "node:path";
import { spawnSync } from "node:child_process";
import { pathToFileURL } from "node:url";

const root = resolve(process.argv[2]);
const output = resolve(process.argv[3]);
await mkdir(output, { recursive: true });
const load = (name) => import(pathToFileURL(`${root}/dist/${name}.js`));
const { browserScenarioSchema } = await load("domain/browserScenario");
const { PlaywrightBrowserScenarioProvider } = await load("browser/PlaywrightBrowserScenarioProvider");
const { PlaywrightScenarioEvents } = await load("browser/PlaywrightScenarioEvents");
const { BrowserScenarioSecrets } = await load("browser/BrowserScenarioSecrets");
const { captureBrowserScenario } = await load("application/BrowserScenarioCaptureService");
const url = "https://fixture.example.test/";

async function capture(families, emitConsole = false) {
  const scenario = browserScenarioSchema.parse({
    browser: { mode: "launch", executable_path: "/opt/chromium/chrome" },
    start_url: { url },
    actions: [{ step_id: "wait", action: "wait_for_timeout", duration_ms: 1 }],
    capture: { events: families },
  });
  const provider = new PlaywrightBrowserScenarioProvider(async () => {
    const page = new EventEmitter();
    const context = new EventEmitter();
    const secrets = BrowserScenarioSecrets.resolve(scenario, {});
    const events = new PlaywrightScenarioEvents({
      page, context, ownsContext: true, secrets,
      enabled: new Set(scenario.capture.events), network: scenario.capture.network,
    });
    return {
      mode: "launch", processOwnership: "provider-owned",
      product: "Chromium", version: "fixture", initialUrl: url,
      currentUrl: () => url, sanitizeUrl: (value) => secrets.sanitizeUrl(value),
      setStep: (index) => events.setStep(index),
      nextEventSequence: () => events.nextSequence(),
      lastEventSequence: () => events.lastSequence(),
      events: () => events.result(), eventLimitations: () => events.limitations(),
      redactError: (error) => String(error),
      perform: async () => {
        if (emitConsole) page.emit("console", {
          location: () => ({ url }), type: () => "error", text: () => "fixture message",
        });
      },
      capture: async (requested) => Object.fromEntries(
        ["screenshot", "dom", "accessibility", "url", "history", "storage"].map(
          (kind) => [kind, requested.has(kind)
            ? { state: "captured", value: secrets.sanitizeUrl(url) }
            : { state: "not_requested" }],
        ),
      ),
      close: async () => { await events.finish(); return "terminated-owned-process"; },
    };
  });
  try {
    const captured = await captureBrowserScenario(provider, scenario);
    assert.equal(captured.ok, true);
    assert.deepEqual(captured.value.parameters.capture.events, families);
    assert.equal(captured.value.normalized_result.completeness.equality_eligible, true);
    return captured.value.normalized_result;
  } finally { await provider.close(); }
}

const empty = await capture([]);
const consoleEmpty = await capture(["console"]);
const errorsEmpty = await capture(["page-errors"]);
const consoleEvent = await capture(["console"], true);
async function compare(name, before, after) {
  const beforeFile = `${output}/${name}-before.json`;
  const afterFile = `${output}/${name}-after.json`;
  await writeFile(beforeFile, JSON.stringify(before));
  await writeFile(afterFile, JSON.stringify(after));
  const run = spawnSync(process.execPath, [
    `${root}/scripts/rea.mjs`, "compare-web-captures", beforeFile, afterFile, "--json",
  ], { encoding: "utf8", timeout: 20000, maxBuffer: 1024 * 1024 });
  assert.equal(run.status, 0, run.stderr);
  const evidence = JSON.parse(run.stdout);
  await writeFile(`${output}/${name}-result.json`, JSON.stringify(evidence, null, 2));
  const diff = evidence.normalized_result;
  console.log(name, diff.overall_status, diff.artifact_diffs.total);
}
await compare("none-vs-console", empty, consoleEmpty);
await compare("console-vs-page-errors", consoleEmpty, errorsEmpty);
await compare("same-console-empty", consoleEmpty, consoleEmpty);
await compare("same-console-difference", consoleEmpty, consoleEvent);

The browser executable path in the validated request is not accessed by this simulated-session fixture. For an actual browser replay, use a caller-selected supported Chromium executable and a local quiet HTTP page, run the same one-step launch scenario with the different event selections, extract each returned Evidence object's normalized_result, and compare those captures. Those real-browser steps were not executed during this verification. Omitting capture.events also parses to []; an explicit empty array is valid.

Expected behavior

For captures with mismatched event-family coverage and no independently observed difference, event evidence must remain unknown and prevent a definite overall_status: "unchanged". The result should identify the coverage limitation and allow the caller to recover by capturing matching families. Complete empty observations taken with the same selected families must remain distinguishable from mismatched observation coverage.

This does not require deliberately unrequested artifacts on both sides to force an unknown overall comparison. The existing comparator allows two not_requested snapshot artifacts to be excluded from comparison. A result about selected evidence need not claim anything about other families that neither side requested.

Actual behavior

The inline reproduction completed successfully through each build's CLI and printed:

none-vs-console unchanged 0
console-vs-page-errors unchanged 0
same-console-empty unchanged 0
same-console-difference changed 1

Both mismatched-selection cases have complete, equality-eligible captures, aligned steps and identical browser/origin context. Each step is unchanged, and artifact_diffs contains no event record. The matching-selection difference control retains the simulated console emission and reports one changed event artifact.

The full capture Evidence preserves the requested families in parameters.capture.events. Its normalized_result omits that selection, and the documented comparison input takes those normalized capture objects rather than full Evidence. Consequently, the comparator cannot distinguish different empty family selections. This statement concerns the executed producer and CLI behavior, not a claim that a real page logged an error.

Additional public-CLI controls on both builds returned unknown for explicit missing event coverage, dropped events, mismatched network-content selection, incompatible browser context and incomplete capture-wide coverage. Those existing guards do not detect event-family selection mismatch. Connect captures are already marked event-incomplete; the failing complete-production condition is launch mode without recorded event gaps.

Evidence

Expected source: docs/browser-scenario-contract.md:294 describes scenario comparison and :313 requires mismatched coverage to remain unknown. src/contracts/browserToolContracts.ts:226 specifies that scenario comparison receives complete capture_browser_scenario.normalized_result objects.
Failure source: The inline production-producer/public-CLI fixture returns the output above on both tested builds. src/domain/browserScenarioDiffEvidence.ts:69, src/domain/browserScenarioDiff.ts:324 and :356 classify complete event arrays as captured and compare them without an event-family coverage check.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The reproduction is complete for the public comparison failure using production-generated capture artifacts. Browser session acquisition and page behavior were simulated. Actual browser launch, navigation, process cleanup and MCP transport were not run. Their relevant reachability and guards were checked statically, so the result does not claim real-provider verification.

Producer reachability follows the public capture adapters in src/cliBrowserScenarioCommands.ts:26 and src/server/registerBrowserScenarioTool.ts:32, through the strict request schema and src/composition/browserScenario.ts:8. src/domain/browserScenarioValues.ts:292 accepts an empty family list and defaults to it. src/browser/PlaywrightScenarioSession.ts:244 passes the caller's selection into the event collector before initial navigation. src/browser/PlaywrightScenarioEvents.ts:73 and :186 gate subscriptions by family. src/browser/PlaywrightBrowserScenarioProvider.ts:212 can classify a successful quiet launch as complete. src/browser/BrowserScenarioPartialObservation.ts:85 and src/domain/browserScenarioCapture.ts:20 omit the selection from normalized captures; src/application/BrowserScenarioEvidence.ts:12 retains it in full Evidence parameters. Identical unique step IDs, action kinds, product/version, origin and network-content settings satisfy comparison guards.

Restoration check

Repeat the mismatched empty-family cases through the supported capture-to-comparison workflow. They must identify incomparable event coverage and return overall unknown while no artifact proves a change. Matching selected families with complete empty observations must still compare unchanged; matching selected families with the retained console emission must still compare changed. Missing/truncated-event and network-content-selection controls must preserve their unknown results. Keep two deliberately unrequested snapshot artifacts excluded from comparison, and retain the ability to prove independently observed differences. The implementation may choose how to preserve or recover coverage information.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The caller receives a definite equality result after changing supported event-family selection, contrary to the documented comparison contract. This can mislead scenario comparison consumers about observation coverage. The observed cases are bounded to complete launch captures with empty selected-family observations; no actual application regression or downstream harm was demonstrated.
Workaround status: available
Workaround basis: Before comparing, check the original capture Evidence's parameters.capture.events and select the same relevant families on both runs, with matching network-content settings. The same-selection empty and observed-difference controls pass on both tested builds. If only normalized captures remain and selection is unavailable, rerun the captures with matching explicit selections instead of relying on an unchanged result.

Originally created by @coygeek on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2169 ## Summary `compare-web-captures` returns `overall_status: "unchanged"` for complete launch-scenario captures taken with different `capture.events` selections when neither selected family emitted an event. In particular, selecting `[]` on one side and `["console"]` on the other, or `["console"]` versus `["page-errors"]`, produces no event artifact difference or coverage warning. This treats different observation coverage as equality, although the scenario comparison contract requires mismatched coverage to remain unknown. The failure affects browser scenario comparison through the CLI and the shared workflow used by MCP. The earliest visible failure is comparison result projection. The target in the executed reproduction is a simulated browser session producing valid scenario capture artifacts; no actual browser page or application regression was observed. ## Runtime or environment Verified on source revision `c22111eca2c4bc84a084f12711f405b3e7422a40` and the published `rea-agents@6.4.0` package, using Node.js v24.21.0 on macOS 27.0.1 arm64. The source build was already prepared with `npm run build:cached`. The measured runtime satisfies the package's supported Node range; it is not the checkout's exact contributor toolchain pin. Both builds produced the same comparison results. ## Steps to reproduce Use either tested build as the package root. Save the following as `verify-event-coverage.mjs`, then run `node verify-event-coverage.mjs /path/to/rea/package/root ./event-coverage-output`. The script calls the production scenario capture provider and event collector with an in-memory session, writes their `normalized_result` captures, and executes the public comparison CLI. It does not open a browser or contact the example URL. The output directory retains the inputs and results for inspection. ```js import assert from "node:assert/strict"; import { EventEmitter } from "node:events"; import { mkdir, writeFile } from "node:fs/promises"; import { resolve } from "node:path"; import { spawnSync } from "node:child_process"; import { pathToFileURL } from "node:url"; const root = resolve(process.argv[2]); const output = resolve(process.argv[3]); await mkdir(output, { recursive: true }); const load = (name) => import(pathToFileURL(`${root}/dist/${name}.js`)); const { browserScenarioSchema } = await load("domain/browserScenario"); const { PlaywrightBrowserScenarioProvider } = await load("browser/PlaywrightBrowserScenarioProvider"); const { PlaywrightScenarioEvents } = await load("browser/PlaywrightScenarioEvents"); const { BrowserScenarioSecrets } = await load("browser/BrowserScenarioSecrets"); const { captureBrowserScenario } = await load("application/BrowserScenarioCaptureService"); const url = "https://fixture.example.test/"; async function capture(families, emitConsole = false) { const scenario = browserScenarioSchema.parse({ browser: { mode: "launch", executable_path: "/opt/chromium/chrome" }, start_url: { url }, actions: [{ step_id: "wait", action: "wait_for_timeout", duration_ms: 1 }], capture: { events: families }, }); const provider = new PlaywrightBrowserScenarioProvider(async () => { const page = new EventEmitter(); const context = new EventEmitter(); const secrets = BrowserScenarioSecrets.resolve(scenario, {}); const events = new PlaywrightScenarioEvents({ page, context, ownsContext: true, secrets, enabled: new Set(scenario.capture.events), network: scenario.capture.network, }); return { mode: "launch", processOwnership: "provider-owned", product: "Chromium", version: "fixture", initialUrl: url, currentUrl: () => url, sanitizeUrl: (value) => secrets.sanitizeUrl(value), setStep: (index) => events.setStep(index), nextEventSequence: () => events.nextSequence(), lastEventSequence: () => events.lastSequence(), events: () => events.result(), eventLimitations: () => events.limitations(), redactError: (error) => String(error), perform: async () => { if (emitConsole) page.emit("console", { location: () => ({ url }), type: () => "error", text: () => "fixture message", }); }, capture: async (requested) => Object.fromEntries( ["screenshot", "dom", "accessibility", "url", "history", "storage"].map( (kind) => [kind, requested.has(kind) ? { state: "captured", value: secrets.sanitizeUrl(url) } : { state: "not_requested" }], ), ), close: async () => { await events.finish(); return "terminated-owned-process"; }, }; }); try { const captured = await captureBrowserScenario(provider, scenario); assert.equal(captured.ok, true); assert.deepEqual(captured.value.parameters.capture.events, families); assert.equal(captured.value.normalized_result.completeness.equality_eligible, true); return captured.value.normalized_result; } finally { await provider.close(); } } const empty = await capture([]); const consoleEmpty = await capture(["console"]); const errorsEmpty = await capture(["page-errors"]); const consoleEvent = await capture(["console"], true); async function compare(name, before, after) { const beforeFile = `${output}/${name}-before.json`; const afterFile = `${output}/${name}-after.json`; await writeFile(beforeFile, JSON.stringify(before)); await writeFile(afterFile, JSON.stringify(after)); const run = spawnSync(process.execPath, [ `${root}/scripts/rea.mjs`, "compare-web-captures", beforeFile, afterFile, "--json", ], { encoding: "utf8", timeout: 20000, maxBuffer: 1024 * 1024 }); assert.equal(run.status, 0, run.stderr); const evidence = JSON.parse(run.stdout); await writeFile(`${output}/${name}-result.json`, JSON.stringify(evidence, null, 2)); const diff = evidence.normalized_result; console.log(name, diff.overall_status, diff.artifact_diffs.total); } await compare("none-vs-console", empty, consoleEmpty); await compare("console-vs-page-errors", consoleEmpty, errorsEmpty); await compare("same-console-empty", consoleEmpty, consoleEmpty); await compare("same-console-difference", consoleEmpty, consoleEvent); ``` The browser executable path in the validated request is not accessed by this simulated-session fixture. For an actual browser replay, use a caller-selected supported Chromium executable and a local quiet HTTP page, run the same one-step launch scenario with the different event selections, extract each returned Evidence object's `normalized_result`, and compare those captures. Those real-browser steps were not executed during this verification. Omitting `capture.events` also parses to `[]`; an explicit empty array is valid. ## Expected behavior For captures with mismatched event-family coverage and no independently observed difference, event evidence must remain unknown and prevent a definite `overall_status: "unchanged"`. The result should identify the coverage limitation and allow the caller to recover by capturing matching families. Complete empty observations taken with the same selected families must remain distinguishable from mismatched observation coverage. This does not require deliberately unrequested artifacts on both sides to force an unknown overall comparison. The existing comparator allows two `not_requested` snapshot artifacts to be excluded from comparison. A result about selected evidence need not claim anything about other families that neither side requested. ## Actual behavior The inline reproduction completed successfully through each build's CLI and printed: ```text none-vs-console unchanged 0 console-vs-page-errors unchanged 0 same-console-empty unchanged 0 same-console-difference changed 1 ``` Both mismatched-selection cases have complete, equality-eligible captures, aligned steps and identical browser/origin context. Each step is `unchanged`, and `artifact_diffs` contains no event record. The matching-selection difference control retains the simulated console emission and reports one changed event artifact. The full capture Evidence preserves the requested families in `parameters.capture.events`. Its `normalized_result` omits that selection, and the documented comparison input takes those normalized capture objects rather than full Evidence. Consequently, the comparator cannot distinguish different empty family selections. This statement concerns the executed producer and CLI behavior, not a claim that a real page logged an error. Additional public-CLI controls on both builds returned `unknown` for explicit missing event coverage, dropped events, mismatched network-content selection, incompatible browser context and incomplete capture-wide coverage. Those existing guards do not detect event-family selection mismatch. Connect captures are already marked event-incomplete; the failing complete-production condition is launch mode without recorded event gaps. ## Evidence - Expected source: `docs/browser-scenario-contract.md:294` describes scenario comparison and `:313` requires mismatched coverage to remain unknown. `src/contracts/browserToolContracts.ts:226` specifies that scenario comparison receives complete `capture_browser_scenario.normalized_result` objects. - Failure source: The inline production-producer/public-CLI fixture returns the output above on both tested builds. `src/domain/browserScenarioDiffEvidence.ts:69`, `src/domain/browserScenarioDiff.ts:324` and `:356` classify complete event arrays as captured and compare them without an event-family coverage check. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The reproduction is complete for the public comparison failure using production-generated capture artifacts. Browser session acquisition and page behavior were simulated. Actual browser launch, navigation, process cleanup and MCP transport were not run. Their relevant reachability and guards were checked statically, so the result does not claim real-provider verification. Producer reachability follows the public capture adapters in `src/cliBrowserScenarioCommands.ts:26` and `src/server/registerBrowserScenarioTool.ts:32`, through the strict request schema and `src/composition/browserScenario.ts:8`. `src/domain/browserScenarioValues.ts:292` accepts an empty family list and defaults to it. `src/browser/PlaywrightScenarioSession.ts:244` passes the caller's selection into the event collector before initial navigation. `src/browser/PlaywrightScenarioEvents.ts:73` and `:186` gate subscriptions by family. `src/browser/PlaywrightBrowserScenarioProvider.ts:212` can classify a successful quiet launch as complete. `src/browser/BrowserScenarioPartialObservation.ts:85` and `src/domain/browserScenarioCapture.ts:20` omit the selection from normalized captures; `src/application/BrowserScenarioEvidence.ts:12` retains it in full Evidence parameters. Identical unique step IDs, action kinds, product/version, origin and network-content settings satisfy comparison guards. ## Restoration check Repeat the mismatched empty-family cases through the supported capture-to-comparison workflow. They must identify incomparable event coverage and return overall unknown while no artifact proves a change. Matching selected families with complete empty observations must still compare unchanged; matching selected families with the retained console emission must still compare changed. Missing/truncated-event and network-content-selection controls must preserve their unknown results. Keep two deliberately unrequested snapshot artifacts excluded from comparison, and retain the ability to prove independently observed differences. The implementation may choose how to preserve or recover coverage information. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The caller receives a definite equality result after changing supported event-family selection, contrary to the documented comparison contract. This can mislead scenario comparison consumers about observation coverage. The observed cases are bounded to complete launch captures with empty selected-family observations; no actual application regression or downstream harm was demonstrated. - Workaround status: available - Workaround basis: Before comparing, check the original capture Evidence's `parameters.capture.events` and select the same relevant families on both runs, with matching network-content settings. The same-selection empty and observed-difference controls pass on both tested builds. If only normalized captures remain and selection is unavailable, rerun the captures with matching explicit selections instead of relying on an unchanged result.

bsakaguchi

2026-10-15 18:27:39 +00:00

closed this issue
added the
area: runtime

bug
labels

bsakaguchi commented

2026-10-15 18:27:40 +00:00

Author

Owner

@morluto commented on GitHub (Oct 12, 2026):

Consolidation update: #2239 now folds #2253 with contributor credit while retaining the earlier PR/commit. Recorded scenario.event_families distinguishes empty selections from omitted historical coverage; selection mismatches and missing historical selections remain unknown. Standards/spec review is complete; 29 focused browser/domain/CLI/MCP tests and source/docs checks pass. Keeping this issue open until the canonical PR merges.

@morluto commented on GitHub (Oct 12, 2026): Consolidation update: #2239 now folds #2253 with contributor credit while retaining the earlier PR/commit. Recorded scenario.event_families distinguishes empty selections from omitted historical coverage; selection mismatches and missing historical selections remain unknown. Standards/spec review is complete; 29 focused browser/domain/CLI/MCP tests and source/docs checks pass. Keeping this issue open until the canonical PR merges.

bsakaguchi referenced this issue

2026-10-16 21:15:32 +00:00

[PR #715] [MERGED] fix(trace): match typed seeds against member arrays #2196

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#715

No description provided.

Rows
Columns