morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5414] [Bug] compare-web-captures rejects legacy scenarios without event-family selections #3281

New issue

Closed

opened 2026-10-31 00:38:44 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:38:44 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5414

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

compare-web-captures before.json legacy.json --json rejects a saved scenario capture that omits scenario.event_families, returning invalid_request instead of a comparison. The omission was independently reproduced on the earlier side, later side, and both sides; all three commands exited 1. An otherwise identical current/current pair exited 0 and reported unchanged.

Reproduction

From the source checkout at the revision below, with locked dependencies installed and the CLI built, generate the suite fixture copies and run the public CLI. These are the commands used by the independent verifier; the inputs are synthetic, not archived captures from an older release.

capture_dir=$(mktemp -d)
mkdir -p "$capture_dir"/{home,config,data,cache,state}
export XDG_CONFIG_HOME="$capture_dir/config"
export XDG_DATA_HOME="$capture_dir/data"
export XDG_CACHE_HOME="$capture_dir/cache"
export XDG_STATE_HOME="$capture_dir/state"

HOME="$capture_dir/home" node --import tsx --input-type=module -e '
import { writeFileSync } from "node:fs";
import { scenarioCaptureFixture } from "./tests/fixtures/browserScenarioCapture.ts";
const before = scenarioCaptureFixture();
const legacy = scenarioCaptureFixture();
delete legacy.scenario.event_families;
const different = scenarioCaptureFixture();
different.scenario.event_families = ["network", "console"];
for (const [name, value] of Object.entries({ before, legacy, different })) {
  writeFileSync(process.env.XDG_DATA_HOME + "/" + name + ".json", JSON.stringify(value));
}
'

# Accepted unchanged control.
HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/before.json" --json

# Rejected: only the documented legacy field is absent.
HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/legacy.json" --json

# Accepted differing-selection control.
HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/different.json" --json

The verifier also ran the failing comparison with the pair reversed and with legacy.json on both sides. Each omission placement failed; no repeated-run rate was measured.

Expected behavior

Return a scenario comparison with unknown event evidence, retaining comparison of other artifact families. For these otherwise identical fixtures, the overall result should be unknown.

The explicit compatibility contract in docs/browser-scenario-contract.md:403-414 says missing coverage remains unknown and captures written before scenario.event_families existed omit it, so their event evidence is also unknown. A source-derived expectation follows from that contract; no historical selection should be fabricated from observed event kinds.

Impact and scope

The missing field blocks the entire saved-scenario comparison, including other evidence families. This is a bounded compatibility failure (P2); accepted current captures still compare successfully, and no saved-data alteration or loss was observed. No practical workaround for existing historical records was established: recapturing requires access to historical targets and produces new observations. MCP runtime behavior was not tested.

REA version

6.7.0; source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8.

Runtime and platform

Node.js v26.12.0 on macOS. NOT_ENOUGH_INFO: the independent verification record does not specify the macOS version or architecture. No browser, provider, Hopper, or Ghidra was used.

Target and artifact context

Local saved browser scenario JSON generated from tests/fixtures/browserScenarioCapture.ts. The legacy-shaped copy differs only by deletion of scenario.event_families; no browser or external service was contacted.

Interface

CLI

Earliest failure stage

Request parsing or validation

Relevant output or evidence

The public CLI returned exit 1 for current/legacy:

{
  "code": "invalid_request",
  "category": "invalid_input",
  "message": "Analysis input is invalid. Check the arguments and try again.",
  "details": {
    "operation": "compare_web_captures",
    "issues": [
      {
        "path": ["after_scenario", "scenario", "event_families"],
        "reason": "missing_argument",
        "expected": "array"
      }
    ]
  }
}

With the omission on the earlier side the path starts with before_scenario; with omissions on both sides, both issues appear. The current/current control returned overall_status: unchanged, two aligned steps, and zero artifact differences. The differing-selection control returned overall_status: unknown and unknown events artifacts on both aligned steps with matching-selection recovery guidance.

Typed outcome or recovery expected

A successful scenario comparison with unknown event evidence, rather than a missing-argument refusal. Preserve the unchanged current/current control and the unknown differing-selection control with its recovery guidance.

Analysis or suspected cause

Source-derived: src/cli/browserAdvancedCommands.ts:130-151 validates recognized scenario captures; src/domain/browserScenarioDiffValues.ts:76-85 applies the shared capture schema to both sides; src/domain/browserScenarioCapture.ts:28 requires event_families. The documented legacy omission therefore fails admission before comparison. MCP shares the canonical contract, but its runtime was not reproduced.

Additional context

Prepared with AI assistance and independently reproduced through the public CLI. The submitter fetched upstream main (e84468c0bb0476412ac0c8161f7c2ecc9a6864ca); no commits since the tested baseline changed the cited contract or validation files.

Related #2169 and merged #2239 concern mismatched selected-family coverage being reported as unchanged. #2239 explicitly describes historical records as readable with unknown event evidence. This report concerns the current missing-field validation refusal, rather than that earlier equality result. A last known good runtime version was not established.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5414 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `compare-web-captures before.json legacy.json --json` rejects a saved scenario capture that omits `scenario.event_families`, returning `invalid_request` instead of a comparison. The omission was independently reproduced on the earlier side, later side, and both sides; all three commands exited 1. An otherwise identical current/current pair exited 0 and reported `unchanged`. ### Reproduction From the source checkout at the revision below, with locked dependencies installed and the CLI built, generate the suite fixture copies and run the public CLI. These are the commands used by the independent verifier; the inputs are synthetic, not archived captures from an older release. ```bash capture_dir=$(mktemp -d) mkdir -p "$capture_dir"/{home,config,data,cache,state} export XDG_CONFIG_HOME="$capture_dir/config" export XDG_DATA_HOME="$capture_dir/data" export XDG_CACHE_HOME="$capture_dir/cache" export XDG_STATE_HOME="$capture_dir/state" HOME="$capture_dir/home" node --import tsx --input-type=module -e ' import { writeFileSync } from "node:fs"; import { scenarioCaptureFixture } from "./tests/fixtures/browserScenarioCapture.ts"; const before = scenarioCaptureFixture(); const legacy = scenarioCaptureFixture(); delete legacy.scenario.event_families; const different = scenarioCaptureFixture(); different.scenario.event_families = ["network", "console"]; for (const [name, value] of Object.entries({ before, legacy, different })) { writeFileSync(process.env.XDG_DATA_HOME + "/" + name + ".json", JSON.stringify(value)); } ' # Accepted unchanged control. HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/before.json" --json # Rejected: only the documented legacy field is absent. HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/legacy.json" --json # Accepted differing-selection control. HOME="$capture_dir/home" node scripts/rea.mjs compare-web-captures "$capture_dir/data/before.json" "$capture_dir/data/different.json" --json ``` The verifier also ran the failing comparison with the pair reversed and with `legacy.json` on both sides. Each omission placement failed; no repeated-run rate was measured. ### Expected behavior Return a scenario comparison with unknown event evidence, retaining comparison of other artifact families. For these otherwise identical fixtures, the overall result should be `unknown`. The explicit compatibility contract in `docs/browser-scenario-contract.md:403-414` says missing coverage remains unknown and captures written before `scenario.event_families` existed omit it, so their event evidence is also unknown. A source-derived expectation follows from that contract; no historical selection should be fabricated from observed event kinds. ### Impact and scope The missing field blocks the entire saved-scenario comparison, including other evidence families. This is a bounded compatibility failure (P2); accepted current captures still compare successfully, and no saved-data alteration or loss was observed. No practical workaround for existing historical records was established: recapturing requires access to historical targets and produces new observations. MCP runtime behavior was not tested. ### REA version 6.7.0; source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`. ### Runtime and platform Node.js v26.12.0 on macOS. NOT_ENOUGH_INFO: the independent verification record does not specify the macOS version or architecture. No browser, provider, Hopper, or Ghidra was used. ### Target and artifact context Local saved browser scenario JSON generated from `tests/fixtures/browserScenarioCapture.ts`. The legacy-shaped copy differs only by deletion of `scenario.event_families`; no browser or external service was contacted. ### Interface CLI ### Earliest failure stage Request parsing or validation ### Relevant output or evidence The public CLI returned exit 1 for current/legacy: ```json { "code": "invalid_request", "category": "invalid_input", "message": "Analysis input is invalid. Check the arguments and try again.", "details": { "operation": "compare_web_captures", "issues": [ { "path": ["after_scenario", "scenario", "event_families"], "reason": "missing_argument", "expected": "array" } ] } } ``` With the omission on the earlier side the path starts with `before_scenario`; with omissions on both sides, both issues appear. The current/current control returned `overall_status: unchanged`, two aligned steps, and zero artifact differences. The differing-selection control returned `overall_status: unknown` and unknown `events` artifacts on both aligned steps with matching-selection recovery guidance. ### Typed outcome or recovery expected A successful scenario comparison with unknown event evidence, rather than a missing-argument refusal. Preserve the unchanged current/current control and the unknown differing-selection control with its recovery guidance. ### Analysis or suspected cause Source-derived: `src/cli/browserAdvancedCommands.ts:130-151` validates recognized scenario captures; `src/domain/browserScenarioDiffValues.ts:76-85` applies the shared capture schema to both sides; `src/domain/browserScenarioCapture.ts:28` requires `event_families`. The documented legacy omission therefore fails admission before comparison. MCP shares the canonical contract, but its runtime was not reproduced. ### Additional context Prepared with AI assistance and independently reproduced through the public CLI. The submitter fetched upstream main (`e84468c0bb0476412ac0c8161f7c2ecc9a6864ca`); no commits since the tested baseline changed the cited contract or validation files. Related #2169 and merged #2239 concern mismatched selected-family coverage being reported as unchanged. #2239 explicitly describes historical records as readable with unknown event evidence. This report concerns the current missing-field validation refusal, rather than that earlier equality result. A last known good runtime version was not established.

bsakaguchi closed this issue

2026-10-31 00:38:45 +00:00

bsakaguchi commented

2026-10-31 00:38:46 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with canonical issue #5322 (compare_web_captures: advertised normalization default is rejected for passive before/after comparisons). Both reports uncover option validation discrepancies in compare-web-captures when handling default event-family selections and normalization profiles. Cataloged under Cluster BK in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with canonical issue #5322 (`compare_web_captures: advertised normalization default is rejected for passive before/after comparisons`). Both reports uncover option validation discrepancies in `compare-web-captures` when handling default event-family selections and normalization profiles. Cataloged under Cluster BK in consolidation tracker #4330.

bsakaguchi commented

2026-10-31 00:38:46 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

Retained-capture consumers now use a schema with optional scenario.event_families, while new capture producers still require it. Both comparison input sides use the retained schema, and the CLI scenario branch validates through that shared comparison schema. Missing selection explicitly yields unknown event evidence without inventing historical selection; other artifact families continue through normal step comparison.

Implementation: browserScenarioCapture.ts, browserScenarioDiffValues.ts.

The relevant regression checks in browserScenarioDiff.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. Retained-capture consumers now use a schema with optional scenario.event_families, while new capture producers still require it. Both comparison input sides use the retained schema, and the CLI scenario branch validates through that shared comparison schema. Missing selection explicitly yields unknown event evidence without inventing historical selection; other artifact families continue through normal step comparison. Implementation: [browserScenarioCapture.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/browserScenarioCapture.ts#L13), [browserScenarioDiffValues.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/browserScenarioDiffValues.ts#L76). The relevant regression checks in [browserScenarioDiff.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/browserScenarioDiff.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3281

No description provided.

Rows
Columns