morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #7415] [Bug] compare-web-captures classifies an ambiguous retained step ID as absent #4421

New issue

Open

opened 2026-10-31 00:49:39 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:39 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7415

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public compare-web-captures CLI accepts complete retained scenario captures containing two occurrences of save in the earlier capture and one in the later capture. It reports the correct duplicate_before_step_id failure, but also returns missing_before_step, the explanation Step save is missing from the before capture., and after_only: ["save"]. The earlier capture contains that ID twice. Reversing the captures produces the symmetric false absence from the later capture.

The comparison exits 0 with partial alignment and an overall unknown result. Those statuses correctly avoid an equality claim; the defect is the additional definite absence classification in the alignment result.

Reproduction

Use Node 24.22.1 and the already-built baseline checkout identified below. In an otherwise empty working directory, make that checkout available under the relative name rea (a directory or symlink). The commands below create synthetic local capture JSON and redirect HOME. The recorded loopback URL is an inert identity; comparison does not access it.
Create the fixtures inline, compare the duplicated ID on each side, and run the unique-ID self-comparison control:

mkdir -p home
cat > make-fixtures.mjs <<'JS'
import { writeFileSync } from 'node:fs';
const url = { url: 'http://127.0.0.1:3000/', origin: 'http://127.0.0.1:3000', query_parameter_names: [], redacted: false };
const completeness = { status: 'complete', equality_eligible: true, missing_sections: [], truncated_sections: [] };
const limits = { max_events: 10000, max_event_allocation_bytes: 4194304, max_total_event_allocation_bytes: 16777216, max_network_transactions: 1024, max_network_body_bytes: 65536, max_network_body_allocation_bytes: 1048576, max_network_header_allocation_bytes: 1048576, max_network_metadata_allocation_bytes: 1048576 };
function capture(ids) {
  return {
    browser: { mode: 'launch', process_ownership: 'provider-owned', cleanup: 'terminated-owned-process', product: 'chromium', version: 'fixture' },
    scenario: { start_origin: url.origin, action_count: ids.length - 1, secret_references: [], network_content: { request_body: false, response_body: false, header_values: false }, event_families: [], limits },
    duration_ms: 1,
    steps: ids.map((step_id, step_index) => ({ step_id, step_index, action: step_index === 0 ? 'navigate' : 'click', elapsed_ms: 1, before_url: url, after_url: url, event_sequence_start: 1, event_sequence_end: 0, status: 'completed', error: null, completeness, artifacts: { screenshot: { state: 'not_requested' }, dom: { state: 'not_requested' }, accessibility: { state: 'not_requested' }, url: { state: 'captured', value: url }, history: { state: 'not_requested' }, storage: { state: 'not_requested' } } })),
    events: { retained: 0, dropped: 0, items: [], omissions: { by_kind: [], network_transactions: [], network_transaction_range: null } },
    completeness, limitations: []
  };
}
for (const [name, ids] of Object.entries({ duplicate: ['scenario_start', 'save', 'save'], unique: ['scenario_start', 'save'], missing: ['scenario_start', 'other'], repaired: ['scenario_start', 'save', 'second-save'] })) writeFileSync(`${name}.json`, JSON.stringify(capture(ids)));
JS
HOME="$PWD/home" node make-fixtures.mjs
HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures duplicate.json unique.json --json > report-before.json
HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures unique.json duplicate.json --json > report-after.json
HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures unique.json unique.json --json > report-control.json
python3 - <<'PYOUT'
import json
for name in ['before','after','control']:
    r=json.load(open(f'report-{name}.json'))['normalized_result']
    print(json.dumps({'case':name,'overall_status':r['overall_status'],'alignment':r['alignment']},separators=(',',':')))
PYOUT

Observe the output shown under Relevant output or evidence. The exact reduced script above completed once; across all independent CLI runs of these inputs, the false missing-before classification occurred in 4/4 comparisons and the symmetric missing-after classification in 3/3. The unique-ID self-comparison returned unchanged/aligned in 3/3 runs. No browser capture or MCP request was executed.

Expected behavior

docs/browser-scenario-contract.md, under “Scenario comparison” (lines 740–744), says: “Steps align only by exact, unique step_id. Duplicate IDs, missing counterparts, changed action kinds, and incompatible browser/origin capture context are returned as alignment failures.” The duplicated identity must remain ineligible for alignment, with no unsupported equality claim. It must not be described as absent from a capture that retains it twice.

AGENTS.md, under “Architecture and Cleanup” (line 21), states: “Unresolved identity or applicability must remain unknown rather than be treated as absent, foreign, or aligned.” That distinction applies to the ambiguous counterpart here. Neither the scenario-comparison paragraph nor docs/browser-observation.md under “Shipped surfaces” defines a missing-step diagnostic as “present but nonunique” or promises simultaneous duplicate and missing failures for this input. The existing tests do not explicitly establish that combined behavior as intended.

Impact and scope

Observed impact is an incorrect machine-readable missing-step failure, one-sided membership entry, and absence explanation for a retained ID. The affected comparison still reports unknown, so this reproduction demonstrates neither false equality nor a false changed result. Callers can inspect the original IDs and the accompanying duplicate diagnosis to distinguish ambiguity from actual absence. No downstream caller action, data loss, or live-browser occurrence was observed; frequency outside these synthetic retained inputs is unknown.

REA version

6.8.0 (source commit a905a71e31)

Runtime and platform

Node 24.22.1; macOS 27.0.1 arm64

Target and artifact context

Synthetic retained browser-scenario JSON with complete/equality-eligible coverage, no events, URL-only artifacts, contiguous step indices, and matching browser/origin metadata. Browser metadata is synthetic; no installed browser was exercised.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed output from the exact reduced script above; all three comparison commands exited 0:

{"case":"before","overall_status":"unknown","alignment":{"status":"partial","aligned_steps":1,"before_only":[],"after_only":["save"],"failures":[{"code":"duplicate_before_step_id","step_id":"save","reason":"before capture contains duplicate step_id save."},{"code":"missing_before_step","step_id":"save","reason":"Step save is missing from the before capture."}]}}
{"case":"after","overall_status":"unknown","alignment":{"status":"partial","aligned_steps":1,"before_only":["save"],"after_only":[],"failures":[{"code":"duplicate_after_step_id","step_id":"save","reason":"after capture contains duplicate step_id save."},{"code":"missing_after_step","step_id":"save","reason":"Step save is missing from the after capture."}]}}
{"case":"control","overall_status":"unchanged","alignment":{"status":"aligned","aligned_steps":2,"before_only":[],"after_only":[],"failures":[]}}

Source-only explanation (not a separate execution): src/cli/browserAdvancedCommands.ts:130 parses the retained captures through the canonical comparison schema, whose retained-capture admission at src/domain/browserScenarioCapture.ts:15 does not reject duplicate step IDs. src/domain/browserScenarioDiff.ts:150 computes one-sided IDs from the maps after duplicate removal; src/domain/browserScenarioDiff.ts:262 removes every occurrence of a duplicate identity, and src/domain/browserScenarioDiff.ts:285 turns the resulting absent map entry into an assertion that the step is missing from the capture. The literal captures remain unchanged.

The focused source suite npm run test:local -- src/domain/browserScenarioDiff.test.ts passed all 32 tests. A repository-wide search for the exact duplicate-step failure codes found their schema and production generator, with no test asserting their combination with missing-step failures. The passing suite therefore does not establish this combined diagnostic as intentional.

Typed outcome or recovery expected

Preserve the duplicate-step alignment failure and the unknown comparison status. Distinguish the unavailable unique counterpart from an ID actually absent from the original capture; do not classify save as definitely after-only or missing-before in the first pair, or as definitely before-only or missing-after in the reversed pair. Continue reporting genuine one-sided IDs and preserve the aligned/unchanged unique-ID control.

Analysis or suspected cause

Source-derived: src/domain/browserScenarioDiff.ts:150 uses the filtered unique-step maps both for comparison eligibility and for original-capture membership. Those are different facts when an ID is duplicated. The observed symmetric false absence diagnoses agree with this loss of the distinction between present-but-ambiguous and absent. A repair should preserve that distinction at the alignment owner rather than only changing the reason string.

Additional context

Related (closed):

#3391 concerned producing duplicate initial-state IDs from an accepted reserved action ID; merged PR #3493 fixes that producer admission on this baseline. This report instead exercises retained-capture comparison, which still accepts duplicate IDs and returns the false absence classification shown above. Current scenario-request validation also rejects duplicate action IDs; this report does not claim a new capture request can generate these fixtures.

Closed #2169, #3173, #3385, and #3386 concern event-family equality, network-content recovery guidance, incomplete accessibility, and passive HTTP multiplicity respectively. Open #7342 concerns repeated event-coverage wording with unique aligned steps. None has the same input condition and acceptance criteria as this false step-absence classification. Open and closed issues and all-state pull requests were searched; no exact open duplicate was found.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7415 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The public `compare-web-captures` CLI accepts complete retained scenario captures containing two occurrences of `save` in the earlier capture and one in the later capture. It reports the correct `duplicate_before_step_id` failure, but also returns `missing_before_step`, the explanation `Step save is missing from the before capture.`, and `after_only: ["save"]`. The earlier capture contains that ID twice. Reversing the captures produces the symmetric false absence from the later capture. The comparison exits 0 with partial alignment and an overall unknown result. Those statuses correctly avoid an equality claim; the defect is the additional definite absence classification in the alignment result. ### Reproduction 1. Use Node 24.22.1 and the already-built baseline checkout identified below. In an otherwise empty working directory, make that checkout available under the relative name `rea` (a directory or symlink). The commands below create synthetic local capture JSON and redirect HOME. The recorded loopback URL is an inert identity; comparison does not access it. 2. Create the fixtures inline, compare the duplicated ID on each side, and run the unique-ID self-comparison control: ```sh mkdir -p home cat > make-fixtures.mjs <<'JS' import { writeFileSync } from 'node:fs'; const url = { url: 'http://127.0.0.1:3000/', origin: 'http://127.0.0.1:3000', query_parameter_names: [], redacted: false }; const completeness = { status: 'complete', equality_eligible: true, missing_sections: [], truncated_sections: [] }; const limits = { max_events: 10000, max_event_allocation_bytes: 4194304, max_total_event_allocation_bytes: 16777216, max_network_transactions: 1024, max_network_body_bytes: 65536, max_network_body_allocation_bytes: 1048576, max_network_header_allocation_bytes: 1048576, max_network_metadata_allocation_bytes: 1048576 }; function capture(ids) { return { browser: { mode: 'launch', process_ownership: 'provider-owned', cleanup: 'terminated-owned-process', product: 'chromium', version: 'fixture' }, scenario: { start_origin: url.origin, action_count: ids.length - 1, secret_references: [], network_content: { request_body: false, response_body: false, header_values: false }, event_families: [], limits }, duration_ms: 1, steps: ids.map((step_id, step_index) => ({ step_id, step_index, action: step_index === 0 ? 'navigate' : 'click', elapsed_ms: 1, before_url: url, after_url: url, event_sequence_start: 1, event_sequence_end: 0, status: 'completed', error: null, completeness, artifacts: { screenshot: { state: 'not_requested' }, dom: { state: 'not_requested' }, accessibility: { state: 'not_requested' }, url: { state: 'captured', value: url }, history: { state: 'not_requested' }, storage: { state: 'not_requested' } } })), events: { retained: 0, dropped: 0, items: [], omissions: { by_kind: [], network_transactions: [], network_transaction_range: null } }, completeness, limitations: [] }; } for (const [name, ids] of Object.entries({ duplicate: ['scenario_start', 'save', 'save'], unique: ['scenario_start', 'save'], missing: ['scenario_start', 'other'], repaired: ['scenario_start', 'save', 'second-save'] })) writeFileSync(`${name}.json`, JSON.stringify(capture(ids))); JS HOME="$PWD/home" node make-fixtures.mjs HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures duplicate.json unique.json --json > report-before.json HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures unique.json duplicate.json --json > report-after.json HOME="$PWD/home" node rea/scripts/rea.mjs compare-web-captures unique.json unique.json --json > report-control.json python3 - <<'PYOUT' import json for name in ['before','after','control']: r=json.load(open(f'report-{name}.json'))['normalized_result'] print(json.dumps({'case':name,'overall_status':r['overall_status'],'alignment':r['alignment']},separators=(',',':'))) PYOUT ``` 3. Observe the output shown under Relevant output or evidence. The exact reduced script above completed once; across all independent CLI runs of these inputs, the false missing-before classification occurred in 4/4 comparisons and the symmetric missing-after classification in 3/3. The unique-ID self-comparison returned unchanged/aligned in 3/3 runs. No browser capture or MCP request was executed. ### Expected behavior `docs/browser-scenario-contract.md`, under “Scenario comparison” (lines 740–744), says: “Steps align only by exact, unique `step_id`. Duplicate IDs, missing counterparts, changed action kinds, and incompatible browser/origin capture context are returned as alignment failures.” The duplicated identity must remain ineligible for alignment, with no unsupported equality claim. It must not be described as absent from a capture that retains it twice. `AGENTS.md`, under “Architecture and Cleanup” (line 21), states: “Unresolved identity or applicability must remain unknown rather than be treated as absent, foreign, or aligned.” That distinction applies to the ambiguous counterpart here. Neither the scenario-comparison paragraph nor `docs/browser-observation.md` under “Shipped surfaces” defines a missing-step diagnostic as “present but nonunique” or promises simultaneous duplicate and missing failures for this input. The existing tests do not explicitly establish that combined behavior as intended. ### Impact and scope Observed impact is an incorrect machine-readable missing-step failure, one-sided membership entry, and absence explanation for a retained ID. The affected comparison still reports unknown, so this reproduction demonstrates neither false equality nor a false changed result. Callers can inspect the original IDs and the accompanying duplicate diagnosis to distinguish ambiguity from actual absence. No downstream caller action, data loss, or live-browser occurrence was observed; frequency outside these synthetic retained inputs is unknown. ### REA version 6.8.0 (source commit a905a71e31f71457cb21fd056b234f872126974e) ### Runtime and platform Node 24.22.1; macOS 27.0.1 arm64 ### Target and artifact context Synthetic retained browser-scenario JSON with complete/equality-eligible coverage, no events, URL-only artifacts, contiguous step indices, and matching browser/origin metadata. Browser metadata is synthetic; no installed browser was exercised. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed output from the exact reduced script above; all three comparison commands exited 0: ```json {"case":"before","overall_status":"unknown","alignment":{"status":"partial","aligned_steps":1,"before_only":[],"after_only":["save"],"failures":[{"code":"duplicate_before_step_id","step_id":"save","reason":"before capture contains duplicate step_id save."},{"code":"missing_before_step","step_id":"save","reason":"Step save is missing from the before capture."}]}} {"case":"after","overall_status":"unknown","alignment":{"status":"partial","aligned_steps":1,"before_only":["save"],"after_only":[],"failures":[{"code":"duplicate_after_step_id","step_id":"save","reason":"after capture contains duplicate step_id save."},{"code":"missing_after_step","step_id":"save","reason":"Step save is missing from the after capture."}]}} {"case":"control","overall_status":"unchanged","alignment":{"status":"aligned","aligned_steps":2,"before_only":[],"after_only":[],"failures":[]}} ``` Source-only explanation (not a separate execution): `src/cli/browserAdvancedCommands.ts:130` parses the retained captures through the canonical comparison schema, whose retained-capture admission at `src/domain/browserScenarioCapture.ts:15` does not reject duplicate step IDs. `src/domain/browserScenarioDiff.ts:150` computes one-sided IDs from the maps after duplicate removal; `src/domain/browserScenarioDiff.ts:262` removes every occurrence of a duplicate identity, and `src/domain/browserScenarioDiff.ts:285` turns the resulting absent map entry into an assertion that the step is missing from the capture. The literal captures remain unchanged. The focused source suite `npm run test:local -- src/domain/browserScenarioDiff.test.ts` passed all 32 tests. A repository-wide search for the exact duplicate-step failure codes found their schema and production generator, with no test asserting their combination with missing-step failures. The passing suite therefore does not establish this combined diagnostic as intentional. ### Typed outcome or recovery expected Preserve the duplicate-step alignment failure and the unknown comparison status. Distinguish the unavailable unique counterpart from an ID actually absent from the original capture; do not classify `save` as definitely after-only or missing-before in the first pair, or as definitely before-only or missing-after in the reversed pair. Continue reporting genuine one-sided IDs and preserve the aligned/unchanged unique-ID control. ### Analysis or suspected cause Source-derived: `src/domain/browserScenarioDiff.ts:150` uses the filtered unique-step maps both for comparison eligibility and for original-capture membership. Those are different facts when an ID is duplicated. The observed symmetric false absence diagnoses agree with this loss of the distinction between present-but-ambiguous and absent. A repair should preserve that distinction at the alignment owner rather than only changing the reason string. ### Additional context Related (closed): [#3391](https://github.com/morluto/rea/issues/3391) concerned producing duplicate initial-state IDs from an accepted reserved action ID; merged PR #3493 fixes that producer admission on this baseline. This report instead exercises retained-capture comparison, which still accepts duplicate IDs and returns the false absence classification shown above. Current scenario-request validation also rejects duplicate action IDs; this report does not claim a new capture request can generate these fixtures. Closed [#2169](https://github.com/morluto/rea/issues/2169), [#3173](https://github.com/morluto/rea/issues/3173), [#3385](https://github.com/morluto/rea/issues/3385), and [#3386](https://github.com/morluto/rea/issues/3386) concern event-family equality, network-content recovery guidance, incomplete accessibility, and passive HTTP multiplicity respectively. Open [#7342](https://github.com/morluto/rea/issues/7342) concerns repeated event-coverage wording with unique aligned steps. None has the same input condition and acceptance criteria as this false step-absence classification. Open and closed issues and all-state pull requests were searched; no exact open duplicate was found.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4421

No description provided.

Rows
Columns