[PR #2253] [CLOSED] fix(browser): keep mismatched event-family coverage unknown in scenario comparison #5120

New issue

Closed

opened 2026-10-31 01:04:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:04:07 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2253
Author: @itsF4LCON
Created: 10/12/2026
Status: ❌ Closed

Base: main ← Head: fix/2169-event-family-coverage

📝 Commits (1)

aeffaec fix(browser): keep mismatched event-family coverage unknown in scenario comparison

📊 Changes

13 files changed (+206 additions, -15 deletions)

View changed files

📝 docs/browser-scenario-contract.md (+6 -0)
📝 src/browser/BrowserScenarioPartialObservation.ts (+1 -0)
📝 src/browser/PlaywrightBrowserScenarioProvider.test.ts (+53 -0)
📝 src/contracts/browserToolContracts.ts (+1 -0)
📝 src/domain/browserScenarioCapture.ts (+2 -0)
📝 src/domain/browserScenarioCaptureValues.ts (+15 -0)
📝 src/domain/browserScenarioDiff.test.ts (+90 -0)
📝 src/domain/browserScenarioDiff.ts (+13 -1)
📝 src/domain/browserScenarioDiffEvidence.ts (+20 -0)
📝 src/domain/browserScenarioValues.ts (+2 -14)
📝 tests/boundary/cli/browserCli.test.ts (+1 -0)
📝 tests/boundary/mcp/browserScenarioMcp.test.ts (+1 -0)
📝 tests/fixtures/webScriptCapture.ts (+1 -0)

📄 Description

Summary

Fixes #2169.

compare-web-captures / compare_web_captures returned overall_status: "unchanged" for two complete launch-mode scenario captures that selected different capture.events families, when none of the selected families emitted an event (for example [] vs ["console"], or ["console"] vs ["page-errors"]). The normalized capture did not record the selection, so the comparator could not tell "no console messages" apart from "console was not observed".

Normalized scenario captures now record the selection as scenario.event_families. Scenario comparison reports the event artifact as unknown, with a recovery instruction, when the selections differ.

#2239 also targets this issue. The main difference: #2239 defaults a missing selection to [], so a capture saved before the fix that recorded console messages but logged none still compares unchanged against a new capture with nothing selected. Here, a missing selection keeps event evidence unknown. This PR also accepts only the eight real event families, de-duplicates the selection, and documents the field in the scenario contract.

Problem and expected behavior

Smallest trigger: capture the same quiet page twice in launch mode, once with capture.events: [] and once with ["console"], then compare the two normalized_result objects.
Violated invariant: docs/browser-scenario-contract.md says mismatched capture coverage remains unknown instead of being treated as equality. A difference in observation coverage was reported as equality.
Expected: mismatched selections give unknown event evidence and prevent an unchanged overall result. Matching selections still compare unchanged when both sides are quiet, and changed when the retained events differ.

Change and scope

browserScenarioCaptureData (the single producer for complete and partial captures) records scenario.event_families: the sorted, de-duplicated capture.events selection.
eventFamilyCoverageReason in browserScenarioDiffEvidence.ts sits next to the existing networkContentCoverageReason and feeds the same incomparableReason path. Selections are compared as sets. When both the event-family and network-content selections differ, both reasons are returned.
The event-family enum moves into browserScenarioCaptureValues.ts as browserScenarioEventFamilySchema, so the request schema and the capture schema share one list.
Compatibility: event_families is optional in browserScenarioCaptureSchema, so captures saved by earlier versions still parse as comparison input. If either side omits it, event evidence is unknown ("Event family selection is unknown in a legacy capture"), because the selection cannot be established. This follows the precedent #797 set when it added scenario.network_content.
Not included: comparing only the families selected on both sides. Like the network-content precedent, any mismatch keeps event evidence unknown. Other artifacts (DOM, URL, screenshots, …) can still prove changed.

Follow the contribution scope. Not applicable: ordinary fix within existing boundaries.

Contract and boundary impact

Semantic owner and earliest changed stage: browser scenario capture projection (browserScenarioCaptureData), consumed by scenario comparison (browserScenarioDiff).
Contribution scope fit and linked maintainer decision: routine fix within existing boundaries; no decision needed.
CLI and MCP/tool-catalog contract: capture_browser_scenario normalized results gain scenario.event_families. compare_web_captures accepts captures with or without it. The compare_web_captures example capture includes it.
Provider, bridge, target-format, or platform compatibility: none.
Evidence, artifact, provenance, or reconstruction contract: event evidence for mismatched or unrecorded selections is now unknown instead of unchanged. Documented in docs/browser-scenario-contract.md (Scenario comparison).
Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable.
Generated metadata (docs/public/product-catalog.json), package, or installation impact: none committed. npm run docs:check passes.

Expected effects:

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated:
- src/browser/PlaywrightBrowserScenarioProvider.test.ts: captures through the production provider with a fake session, then compares. [] vs ["console"] and ["console"] vs ["page-errors"] give unknown with the event-family reason. A duplicated, unordered selection ["page-errors", "console", "console"] is recorded as ["console", "page-errors"], and reordered selections compare unchanged.
- src/domain/browserScenarioDiff.test.ts: the mismatch reason names both selections. A reordered selection compares unchanged. A legacy capture (field omitted) on one or both sides gives unknown. An unknown family name is rejected. Differing events under mismatched selections are unknown, not changed. Event-family and network-content mismatches report both reasons.
- Existing capture fixtures, CLI/MCP boundary fixtures, and the tool-contract example set event_families.
Base reproduction or other evidence: on the base revision, the new provider test fails with expected 'unchanged' to be 'unknown'. Real Chrome through the public CLI (below) also returns unchanged for all three cases on base.

User-visible CLI/MCP output: real google-chrome-stable, launch mode, a quiet local page, capture-browser-scenario then compare-web-captures:

Base (upstream/main):

none-vs-console        unchanged
console-vs-page-errors unchanged
same-console-empty     unchanged

This branch:

none-vs-console        unknown   Event family selections differ (none before, console after); capture both sides with matching event families before comparing event evidence.
console-vs-page-errors unknown   Event family selections differ (console before, page-errors after); capture both sides with matching event families before comparing event evidence.
same-console-empty     unchanged

All captures were complete and equality_eligible: true.

Remaining proof gaps: MCP stdio was exercised only by the existing boundary tests (tests/boundary/mcp/browserScenarioMcp.test.ts), not with a real browser. Connect mode was not run against a real browser; its captures are already event-incomplete.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Linux x86_64, Node 24.21.0 (the repo pins 24.18.0), npm 11.19.0.

npm run test:focused -- on the 82 test files that touch scenario capture, comparison, or tool contracts: 82 files passed (579 tests, 5 skipped).
npm run check:fast: passed.
npm run check: passed.
npm run docs:check: passed.
npm run verify:test-discovery: passed.
Real Chrome CLI comparison above, on base and on this branch.

Compatibility, safety, and release

Breaking changes or migration steps: none. The new field is optional on input. Captures without it still parse, but their event evidence now compares as unknown. Re-capture both sides to get a definite event comparison.
Real Hopper/Ghidra, browser, or OS coverage: real Chrome (launch mode) through the public CLI on Linux. No Hopper/Ghidra/IDA involvement.
Package or release metadata impact: none.
Security, privacy, process, or containment review: no new process, network, or file effects. The field records only the caller's family selection.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2253 **Author:** [@itsF4LCON](https://github.com/itsF4LCON) **Created:** 10/12/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/2169-event-family-coverage` --- ### 📝 Commits (1) - [`aeffaec`](https://github.com/morluto/rea/commit/aeffaec0f25ba7434071509619f401010d60b2c8) fix(browser): keep mismatched event-family coverage unknown in scenario comparison ### 📊 Changes **13 files changed** (+206 additions, -15 deletions) <details> <summary>View changed files</summary> 📝 `docs/browser-scenario-contract.md` (+6 -0) 📝 `src/browser/BrowserScenarioPartialObservation.ts` (+1 -0) 📝 `src/browser/PlaywrightBrowserScenarioProvider.test.ts` (+53 -0) 📝 `src/contracts/browserToolContracts.ts` (+1 -0) 📝 `src/domain/browserScenarioCapture.ts` (+2 -0) 📝 `src/domain/browserScenarioCaptureValues.ts` (+15 -0) 📝 `src/domain/browserScenarioDiff.test.ts` (+90 -0) 📝 `src/domain/browserScenarioDiff.ts` (+13 -1) 📝 `src/domain/browserScenarioDiffEvidence.ts` (+20 -0) 📝 `src/domain/browserScenarioValues.ts` (+2 -14) 📝 `tests/boundary/cli/browserCli.test.ts` (+1 -0) 📝 `tests/boundary/mcp/browserScenarioMcp.test.ts` (+1 -0) 📝 `tests/fixtures/webScriptCapture.ts` (+1 -0) </details> ### 📄 Description ## Summary Fixes #2169. `compare-web-captures` / `compare_web_captures` returned `overall_status: "unchanged"` for two complete launch-mode scenario captures that selected different `capture.events` families, when none of the selected families emitted an event (for example `[]` vs `["console"]`, or `["console"]` vs `["page-errors"]`). The normalized capture did not record the selection, so the comparator could not tell "no console messages" apart from "console was not observed". Normalized scenario captures now record the selection as `scenario.event_families`. Scenario comparison reports the event artifact as `unknown`, with a recovery instruction, when the selections differ. #2239 also targets this issue. The main difference: #2239 defaults a missing selection to `[]`, so a capture saved before the fix that recorded console messages but logged none still compares `unchanged` against a new capture with nothing selected. Here, a missing selection keeps event evidence `unknown`. This PR also accepts only the eight real event families, de-duplicates the selection, and documents the field in the scenario contract. ## Problem and expected behavior - Smallest trigger: capture the same quiet page twice in launch mode, once with `capture.events: []` and once with `["console"]`, then compare the two `normalized_result` objects. - Violated invariant: `docs/browser-scenario-contract.md` says mismatched capture coverage remains unknown instead of being treated as equality. A difference in observation coverage was reported as equality. - Expected: mismatched selections give `unknown` event evidence and prevent an `unchanged` overall result. Matching selections still compare `unchanged` when both sides are quiet, and `changed` when the retained events differ. ## Change and scope - `browserScenarioCaptureData` (the single producer for complete and partial captures) records `scenario.event_families`: the sorted, de-duplicated `capture.events` selection. - `eventFamilyCoverageReason` in `browserScenarioDiffEvidence.ts` sits next to the existing `networkContentCoverageReason` and feeds the same `incomparableReason` path. Selections are compared as sets. When both the event-family and network-content selections differ, both reasons are returned. - The event-family enum moves into `browserScenarioCaptureValues.ts` as `browserScenarioEventFamilySchema`, so the request schema and the capture schema share one list. - **Compatibility:** `event_families` is optional in `browserScenarioCaptureSchema`, so captures saved by earlier versions still parse as comparison input. If either side omits it, event evidence is `unknown` ("Event family selection is unknown in a legacy capture"), because the selection cannot be established. This follows the precedent #797 set when it added `scenario.network_content`. - Not included: comparing only the families selected on both sides. Like the network-content precedent, any mismatch keeps event evidence unknown. Other artifacts (DOM, URL, screenshots, …) can still prove `changed`. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Not applicable: ordinary fix within existing boundaries. ## Contract and boundary impact - Semantic owner and earliest changed stage: browser scenario capture projection (`browserScenarioCaptureData`), consumed by scenario comparison (`browserScenarioDiff`). - Contribution scope fit and linked maintainer decision: routine fix within existing boundaries; no decision needed. - CLI and MCP/tool-catalog contract: `capture_browser_scenario` normalized results gain `scenario.event_families`. `compare_web_captures` accepts captures with or without it. The `compare_web_captures` example capture includes it. - Provider, bridge, target-format, or platform compatibility: none. - Evidence, artifact, provenance, or reconstruction contract: event evidence for mismatched or unrecorded selections is now `unknown` instead of `unchanged`. Documented in `docs/browser-scenario-contract.md` (Scenario comparison). - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none committed. `npm run docs:check` passes. Expected effects: - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `src/browser/PlaywrightBrowserScenarioProvider.test.ts`: captures through the production provider with a fake session, then compares. `[]` vs `["console"]` and `["console"]` vs `["page-errors"]` give `unknown` with the event-family reason. A duplicated, unordered selection `["page-errors", "console", "console"]` is recorded as `["console", "page-errors"]`, and reordered selections compare `unchanged`. - `src/domain/browserScenarioDiff.test.ts`: the mismatch reason names both selections. A reordered selection compares `unchanged`. A legacy capture (field omitted) on one or both sides gives `unknown`. An unknown family name is rejected. Differing events under mismatched selections are `unknown`, not `changed`. Event-family and network-content mismatches report both reasons. - Existing capture fixtures, CLI/MCP boundary fixtures, and the tool-contract example set `event_families`. - Base reproduction or other evidence: on the base revision, the new provider test fails with `expected 'unchanged' to be 'unknown'`. Real Chrome through the public CLI (below) also returns `unchanged` for all three cases on base. - User-visible CLI/MCP output: real `google-chrome-stable`, launch mode, a quiet local page, `capture-browser-scenario` then `compare-web-captures`: Base (`upstream/main`): ``` none-vs-console unchanged console-vs-page-errors unchanged same-console-empty unchanged ``` This branch: ``` none-vs-console unknown Event family selections differ (none before, console after); capture both sides with matching event families before comparing event evidence. console-vs-page-errors unknown Event family selections differ (console before, page-errors after); capture both sides with matching event families before comparing event evidence. same-console-empty unchanged ``` All captures were complete and `equality_eligible: true`. - Remaining proof gaps: MCP stdio was exercised only by the existing boundary tests (`tests/boundary/mcp/browserScenarioMcp.test.ts`), not with a real browser. Connect mode was not run against a real browser; its captures are already event-incomplete. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Linux x86_64, Node 24.21.0 (the repo pins 24.18.0), npm 11.19.0. - `npm run test:focused --` on the 82 test files that touch scenario capture, comparison, or tool contracts: 82 files passed (579 tests, 5 skipped). - `npm run check:fast`: passed. - `npm run check`: passed. - `npm run docs:check`: passed. - `npm run verify:test-discovery`: passed. - Real Chrome CLI comparison above, on base and on this branch. ## Compatibility, safety, and release - Breaking changes or migration steps: none. The new field is optional on input. Captures without it still parse, but their event evidence now compares as `unknown`. Re-capture both sides to get a definite event comparison. - Real Hopper/Ghidra, browser, or OS coverage: real Chrome (launch mode) through the public CLI on Linux. No Hopper/Ghidra/IDA involvement. - Package or release metadata impact: none. - Security, privacy, process, or containment review: no new process, network, or file effects. The field records only the caller's family selection. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:04:07 +00:00