morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #7342] [Bug] Scenario event comparison repeats identical coverage instructions #4365

New issue

Closed

opened 2026-10-31 00:49:14 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:14 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7342

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public compare-web-captures command repeats an identical coverage sentence inside each affected events.reason. A synthetic retained scenario with an explicitly unknown redirect predecessor compares with itself as unknown, but its explanation contains the same sentence twice. A separate comparison with both event-family and network-content selection mismatches repeats the family instruction before the content instruction. The comparison statuses remain correct in these cases.

Reproduction

Use the prepared built checkout at the source revision below, with its pinned Node runtime. The following reproduces the independently executed synthetic inputs and six comparison calls. It creates local files only; no browser or website is accessed. Each failing pair was run once, and both aligned steps contained the repetition.

REA_ROOT=/path/to/rea
WORK=$(mktemp -d)
mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/data" "$WORK/tmp"
export REA_ROOT WORK
node --input-type=module <<'JS'
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
const root = process.env.REA_ROOT;
const work = process.env.WORK;
const load = (p) => import(pathToFileURL(join(root, 'dist', p)).href);
const { browserScenarioCaptureSchema } = await load('domain/browserScenarioCapture.js');
const { DEFAULT_BROWSER_SCENARIO_CAPTURE_LIMITS: limits } = await load('domain/browserScenarioCaptureLimits.js');
const complete = { status: 'complete', equality_eligible: true, missing_sections: [], truncated_sections: [] };
const url = { url: 'https://example.test/', origin: 'https://example.test', query_parameter_names: [], redacted: false };
function capture(state = 'unknown', families = ['network'], requestBody = false, includeRequest = true) {
  return browserScenarioCaptureSchema.parse({
    browser: { mode: 'launch', process_ownership: 'provider-owned', cleanup: 'terminated-owned-process', product: 'Chromium', version: 'fixture' },
    scenario: { start_origin: url.origin, action_count: 1, secret_references: [], network_content: { request_body: requestBody, response_body: false, header_values: false }, event_families: families, limits },
    duration_ms: 1,
    steps: ['scenario_start', 'wait'].map((step_id, step_index) => ({
      step_id, step_index, action: step_index ? 'wait_for_timeout' : 'scenario_start', status: 'completed', elapsed_ms: 0,
      before_url: url, after_url: url, error: null,
      event_sequence_start: 1, event_sequence_end: step_index && includeRequest ? 1 : 0,
      artifacts: Object.fromEntries(['screenshot', 'dom', 'accessibility', 'url', 'history', 'storage'].map(x => [x, { state: 'not_requested' }])),
      completeness: complete,
    })),
    events: { retained: includeRequest ? 1 : 0, dropped: 0, items: includeRequest ? [{ sequence: 1, step_index: 1, kind: 'request', transaction_id: 'request-1', redirected_from_transaction_id: null, redirected_from_transaction_state: state, method: 'GET', url, resource_type: 'fetch', header_names: [], status: null, failure: null }] : [], omissions: { by_kind: [], network_transactions: [], network_transaction_range: null } },
    completeness: complete, limitations: [],
  });
}
const captures = {
  unknown: capture(), root: capture('root'),
  console: capture('root', ['console'], false, false),
  families: capture('root', ['console', 'network'], false, false),
  content: capture('root', ['console'], true, false),
  both: capture('root', ['console', 'network'], true, false),
};
for (const [name, value] of Object.entries(captures)) writeFileSync(join(work, `${name}.json`), JSON.stringify(value, null, 2));
JS
cd "$WORK"
run_compare() {
  env -i PATH="$PATH" HOME="$WORK/home" TMPDIR="$WORK/tmp" \
    XDG_CONFIG_HOME="$WORK/config" XDG_CACHE_HOME="$WORK/cache" \
    XDG_DATA_HOME="$WORK/data" NO_COLOR=1 \
    node "$REA_ROOT/scripts/rea.mjs" compare-web-captures \
    "$WORK/$1.json" "$WORK/$2.json" --json > "$WORK/$3.json"
}
run_compare unknown unknown unknown-self
run_compare root root root-self
run_compare console families family-only
run_compare console content content-only
run_compare console both family-and-content
run_compare console console console-self
node --input-type=module <<'JS'
import { readFileSync } from 'node:fs';
for (const name of ['unknown-self', 'root-self', 'family-only', 'content-only', 'family-and-content', 'console-self']) {
  const r = JSON.parse(readFileSync(`${name}.json`, 'utf8')).normalized_result;
  console.log(name, r.overall_status, r.alignment.status,
    r.steps.flatMap(s => s.artifact_diffs.filter(d => d.artifact === 'events').map(d => d.reason)));
}
JS

Expected behavior

Report each distinct coverage condition once while retaining its recovery instruction. Keep the correct unknown status for unknown ancestry and mismatched selections, and preserve current alignment and digests. docs/browser-scenario-contract.md:489, :591, and :617 define the ancestry and selection uncertainty contracts; AGENTS.md requires the smallest sufficient answer with necessary coverage and limitations. src/domain/browserScenarioDiff.ts:616 already de-duplicates incomparable reason values, supporting one explanation per distinct reason.

Impact and scope

P3: redundant diagnostic wording in the two observed comparison conditions. No false equality, incorrect change decision, lost evidence or digest corruption was observed. The complete instruction remains readable, so callers can treat identical repeated sentences as one explanation. Observed through the CLI on synthetic retained captures; actual browser frequency and MCP transport behavior were not measured.

REA version

6.8.0, source revision dbe2c97af34361b11b9504e37611dd47715b52cb.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1 arm64. Used the existing compiled runtime with the supplied successful build receipt; did not install dependencies or build during verification. Chromium metadata in the fixture is synthetic; no browser version was exercised.

Target and artifact context

Schema-validated retained browser-scenario JSON with matching browser/origin context, two unique aligned steps, complete inventories and no dropped events. The ancestry case has one request with redirected_from_transaction_id: null and redirected_from_transaction_state: "unknown", matching the current producer's representation of an unseen predecessor. Selection controls have empty event inventories. The example URL is an inert recorded identity.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

All six CLI calls exited 0 and returned alignment.status: "aligned". The relevant observed results were:

Comparison	Overall status	Events explanation in each aligned step
unknown-self	unknown	Redirect sentence twice
root-self	unchanged	No events diff
family-only	unknown	Family instruction once
content-only	unknown	Content instruction once
family-and-content	unknown	Family instruction twice, content instruction once
console-self	unchanged	No events diff

Exact unknown-self reason:

Redirect predecessor evidence is missing; network transaction identity remains unknown. Redirect predecessor evidence is missing; network transaction identity remains unknown.

The combined case returned the following full reason:

Event family selections differ (console before, console, network after); capture both sides with matching event families before comparing event evidence. Event family selections differ (console before, console, network after); capture both sides with matching event families before comparing event evidence. Network content selections differ; capture both sides with matching content selection before comparing network evidence.

The ancestry self-comparison has equal before/after event digests at each step. The combined-selection event digests match the family-only control. These observations establish redundant text without an accompanying evidence identity change in these cases.

Typed outcome or recovery expected

The same comparison result with each distinct coverage instruction included once. Preserve the independent family and content instructions and the existing uncertainty decisions.

Analysis or suspected cause

Source-derived: eventCoverageReason at src/domain/browserScenarioDiff.ts:505 concatenates the before/after network coverage reasons, allowing identical per-capture sentences twice. The unknown-events branch at :464 also combines familyIncomparableReason with a composite string that already contains it. joinIncomparableReasons at :616 removes exact duplicate strings but cannot remove the family sentence embedded within that longer string. src/domain/browserScenarioNetworkComparison.ts:134 supplies the ancestry instruction. This source trace is consistent with the observed CLI output.

Additional context

Expected source: The scenario comparison contract and reason de-duplication boundary cited above apply to complete retained captures accepted by the CLI's canonical parser.
Failure source: The independently executed public CLI comparisons above return the exact repeated instructions; source locators explain their construction.
Evidence provenance: observed. Producer representation and shared MCP reachability are source-derived.
Local verification: reproduced. Each fixture also passed browserScenarioCaptureSchema and each pair passed compareBrowserScenariosInputSchema before CLI execution.
Reproduction completeness: complete for the public comparison failure on synthetic retained captures. A real browser capture and MCP transport were not executed, and no fixed implementation was tested.
Restoration check: Re-run both failing cases and require each distinct sentence once. Preserve the current unknown statuses, aligned steps and event digests. Keep the family-only and content-only instructions once, and both matching-input controls unchanged.

Related closed #3173 concerns missing network-content guidance alongside console changes; the present cases retain the guidance but repeat text. Closed #2169, #2239 and #2253 concern false equality from mismatched event-family coverage; these inputs correctly return unknown. No identical repetition report was found in the complete saved history or final live open/closed issue and pull-request searches.

Triage assessment

Impact level: P3
Assessment status: supported
Impact basis: The observed defect repeats recovery wording while preserving the correct uncertainty decision and complete instructions; no consequential behavior change was demonstrated.
Workaround status: available
Workaround basis: Read the identical repeated instruction once. The full instruction remains present in the observed result, so no capture change or loss of evidence is required.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7342 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The public `compare-web-captures` command repeats an identical coverage sentence inside each affected `events.reason`. A synthetic retained scenario with an explicitly unknown redirect predecessor compares with itself as `unknown`, but its explanation contains the same sentence twice. A separate comparison with both event-family and network-content selection mismatches repeats the family instruction before the content instruction. The comparison statuses remain correct in these cases. ## Reproduction Use the prepared built checkout at the source revision below, with its pinned Node runtime. The following reproduces the independently executed synthetic inputs and six comparison calls. It creates local files only; no browser or website is accessed. Each failing pair was run once, and both aligned steps contained the repetition. ```sh REA_ROOT=/path/to/rea WORK=$(mktemp -d) mkdir -p "$WORK/home" "$WORK/config" "$WORK/cache" "$WORK/data" "$WORK/tmp" export REA_ROOT WORK node --input-type=module <<'JS' import { writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; const root = process.env.REA_ROOT; const work = process.env.WORK; const load = (p) => import(pathToFileURL(join(root, 'dist', p)).href); const { browserScenarioCaptureSchema } = await load('domain/browserScenarioCapture.js'); const { DEFAULT_BROWSER_SCENARIO_CAPTURE_LIMITS: limits } = await load('domain/browserScenarioCaptureLimits.js'); const complete = { status: 'complete', equality_eligible: true, missing_sections: [], truncated_sections: [] }; const url = { url: 'https://example.test/', origin: 'https://example.test', query_parameter_names: [], redacted: false }; function capture(state = 'unknown', families = ['network'], requestBody = false, includeRequest = true) { return browserScenarioCaptureSchema.parse({ browser: { mode: 'launch', process_ownership: 'provider-owned', cleanup: 'terminated-owned-process', product: 'Chromium', version: 'fixture' }, scenario: { start_origin: url.origin, action_count: 1, secret_references: [], network_content: { request_body: requestBody, response_body: false, header_values: false }, event_families: families, limits }, duration_ms: 1, steps: ['scenario_start', 'wait'].map((step_id, step_index) => ({ step_id, step_index, action: step_index ? 'wait_for_timeout' : 'scenario_start', status: 'completed', elapsed_ms: 0, before_url: url, after_url: url, error: null, event_sequence_start: 1, event_sequence_end: step_index && includeRequest ? 1 : 0, artifacts: Object.fromEntries(['screenshot', 'dom', 'accessibility', 'url', 'history', 'storage'].map(x => [x, { state: 'not_requested' }])), completeness: complete, })), events: { retained: includeRequest ? 1 : 0, dropped: 0, items: includeRequest ? [{ sequence: 1, step_index: 1, kind: 'request', transaction_id: 'request-1', redirected_from_transaction_id: null, redirected_from_transaction_state: state, method: 'GET', url, resource_type: 'fetch', header_names: [], status: null, failure: null }] : [], omissions: { by_kind: [], network_transactions: [], network_transaction_range: null } }, completeness: complete, limitations: [], }); } const captures = { unknown: capture(), root: capture('root'), console: capture('root', ['console'], false, false), families: capture('root', ['console', 'network'], false, false), content: capture('root', ['console'], true, false), both: capture('root', ['console', 'network'], true, false), }; for (const [name, value] of Object.entries(captures)) writeFileSync(join(work, `${name}.json`), JSON.stringify(value, null, 2)); JS cd "$WORK" run_compare() { env -i PATH="$PATH" HOME="$WORK/home" TMPDIR="$WORK/tmp" \ XDG_CONFIG_HOME="$WORK/config" XDG_CACHE_HOME="$WORK/cache" \ XDG_DATA_HOME="$WORK/data" NO_COLOR=1 \ node "$REA_ROOT/scripts/rea.mjs" compare-web-captures \ "$WORK/$1.json" "$WORK/$2.json" --json > "$WORK/$3.json" } run_compare unknown unknown unknown-self run_compare root root root-self run_compare console families family-only run_compare console content content-only run_compare console both family-and-content run_compare console console console-self node --input-type=module <<'JS' import { readFileSync } from 'node:fs'; for (const name of ['unknown-self', 'root-self', 'family-only', 'content-only', 'family-and-content', 'console-self']) { const r = JSON.parse(readFileSync(`${name}.json`, 'utf8')).normalized_result; console.log(name, r.overall_status, r.alignment.status, r.steps.flatMap(s => s.artifact_diffs.filter(d => d.artifact === 'events').map(d => d.reason))); } JS ``` ## Expected behavior Report each distinct coverage condition once while retaining its recovery instruction. Keep the correct `unknown` status for unknown ancestry and mismatched selections, and preserve current alignment and digests. `docs/browser-scenario-contract.md:489`, `:591`, and `:617` define the ancestry and selection uncertainty contracts; `AGENTS.md` requires the smallest sufficient answer with necessary coverage and limitations. `src/domain/browserScenarioDiff.ts:616` already de-duplicates incomparable reason values, supporting one explanation per distinct reason. ## Impact and scope P3: redundant diagnostic wording in the two observed comparison conditions. No false equality, incorrect change decision, lost evidence or digest corruption was observed. The complete instruction remains readable, so callers can treat identical repeated sentences as one explanation. Observed through the CLI on synthetic retained captures; actual browser frequency and MCP transport behavior were not measured. ## REA version 6.8.0, source revision `dbe2c97af34361b11b9504e37611dd47715b52cb`. ## Runtime and platform Node.js v24.22.1; macOS 27.0.1 arm64. Used the existing compiled runtime with the supplied successful build receipt; did not install dependencies or build during verification. Chromium metadata in the fixture is synthetic; no browser version was exercised. ## Target and artifact context Schema-validated retained browser-scenario JSON with matching browser/origin context, two unique aligned steps, complete inventories and no dropped events. The ancestry case has one request with `redirected_from_transaction_id: null` and `redirected_from_transaction_state: "unknown"`, matching the current producer's representation of an unseen predecessor. Selection controls have empty event inventories. The example URL is an inert recorded identity. ## Interface CLI ## Earliest failure stage Transport or result projection ## Relevant output or evidence All six CLI calls exited 0 and returned `alignment.status: "aligned"`. The relevant observed results were: | Comparison | Overall status | Events explanation in each aligned step | | --- | --- | --- | | unknown-self | unknown | Redirect sentence twice | | root-self | unchanged | No events diff | | family-only | unknown | Family instruction once | | content-only | unknown | Content instruction once | | family-and-content | unknown | Family instruction twice, content instruction once | | console-self | unchanged | No events diff | Exact `unknown-self` reason: ```text Redirect predecessor evidence is missing; network transaction identity remains unknown. Redirect predecessor evidence is missing; network transaction identity remains unknown. ``` The combined case returned the following full reason: ```text Event family selections differ (console before, console, network after); capture both sides with matching event families before comparing event evidence. Event family selections differ (console before, console, network after); capture both sides with matching event families before comparing event evidence. Network content selections differ; capture both sides with matching content selection before comparing network evidence. ``` The ancestry self-comparison has equal before/after event digests at each step. The combined-selection event digests match the family-only control. These observations establish redundant text without an accompanying evidence identity change in these cases. ## Typed outcome or recovery expected The same comparison result with each distinct coverage instruction included once. Preserve the independent family and content instructions and the existing uncertainty decisions. ## Analysis or suspected cause Source-derived: `eventCoverageReason` at `src/domain/browserScenarioDiff.ts:505` concatenates the before/after network coverage reasons, allowing identical per-capture sentences twice. The unknown-events branch at `:464` also combines `familyIncomparableReason` with a composite string that already contains it. `joinIncomparableReasons` at `:616` removes exact duplicate strings but cannot remove the family sentence embedded within that longer string. `src/domain/browserScenarioNetworkComparison.ts:134` supplies the ancestry instruction. This source trace is consistent with the observed CLI output. ## Additional context - Expected source: The scenario comparison contract and reason de-duplication boundary cited above apply to complete retained captures accepted by the CLI's canonical parser. - Failure source: The independently executed public CLI comparisons above return the exact repeated instructions; source locators explain their construction. - Evidence provenance: observed. Producer representation and shared MCP reachability are source-derived. - Local verification: reproduced. Each fixture also passed `browserScenarioCaptureSchema` and each pair passed `compareBrowserScenariosInputSchema` before CLI execution. - Reproduction completeness: complete for the public comparison failure on synthetic retained captures. A real browser capture and MCP transport were not executed, and no fixed implementation was tested. - Restoration check: Re-run both failing cases and require each distinct sentence once. Preserve the current unknown statuses, aligned steps and event digests. Keep the family-only and content-only instructions once, and both matching-input controls unchanged. Related closed [#3173](https://github.com/morluto/rea/issues/3173) concerns missing network-content guidance alongside console changes; the present cases retain the guidance but repeat text. Closed [#2169](https://github.com/morluto/rea/issues/2169), [#2239](https://github.com/morluto/rea/pull/2239) and [#2253](https://github.com/morluto/rea/pull/2253) concern false equality from mismatched event-family coverage; these inputs correctly return unknown. No identical repetition report was found in the complete saved history or final live open/closed issue and pull-request searches. ## Triage assessment - Impact level: P3 - Assessment status: supported - Impact basis: The observed defect repeats recovery wording while preserving the correct uncertainty decision and complete instructions; no consequential behavior change was demonstrated. - Workaround status: available - Workaround basis: Read the identical repeated instruction once. The full instruction remains present in the observed result, so no capture change or loss of evidence is required.

bsakaguchi closed this issue

2026-10-31 00:49:14 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:41 +00:00

[PR #4365] [CLOSED] fix(domain): stop reporting a named-versus-unnamed reference endpoint as a change #6206

bsakaguchi referenced this issue

2026-11-01 21:10:53 +00:00

[GH-ISSUE #3582] [Tracker] Ready to close or merge: issues and pull requests (October 2026) #1330