morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3173] [Bug] Scenario comparison loses network-content recovery guidance when console events differ #1113

New issue

Closed

opened 2026-10-15 18:31:10 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:31:10 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3173

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

compare-web-captures omits the documented network-content comparability warning when complete, aligned scenario captures declare different request-body selections and have different console text at both steps. The observed CLI comparison exits zero, returns overall_status: "changed", and gives both steps an events diff with status: "changed" and reason: null. The full response contains no network-selection recovery instruction.

The console differences support a changed result. The failure is the loss of the independent network uncertainty and the instruction to capture matching content selections. A matching-selection control also returns changed records with null reasons. A quiet control with mismatched content selection reports unknown at both steps and retains the recovery instruction.

Reproduction

Use a clean checkout of revision 58300885207133422af235e0f2b10658cf958b03 with its pinned toolchain. These commands create only synthetic local capture files; no browser or reachable website is required. The independent verifier replayed the fixture and comparison commands against the prepared built runtime; dependency installation and the build were not rerun during that verification.

Install and build, using a disposable local home for the commands:

mkdir -p repro-content-selection/home
repro_run() {
  env HOME="$PWD/repro-content-selection/home" \
    XDG_CONFIG_HOME="$PWD/repro-content-selection/home/.config" \
    XDG_CACHE_HOME="$PWD/repro-content-selection/home/.cache" \
    XDG_DATA_HOME="$PWD/repro-content-selection/home/.local/share" "$@"
}
repro_run npm ci
repro_run npm run build:cached

Create complete captures with the same selected event families, browser identity, steps and artifacts. Request-body selection differs, and the console message differs at both steps. Also create controls with matching content selection and matching console messages:

repro_run node --input-type=module <<'JS'
import { createHash } from 'node:crypto';
import { writeFileSync } from 'node:fs';
const directory = 'repro-content-selection';
const url = {
  url: 'https://app.example.test/settings',
  origin: 'https://app.example.test',
  query_parameter_names: [],
  redacted: false,
};
const completeness = {
  status: 'complete', equality_eligible: true,
  missing_sections: [], truncated_sections: [],
};
const text = '<main>Ready</main>';
const artifacts = {
  screenshot: { state: 'not_requested' },
  dom: { state: 'captured', value: {
    text, bytes: Buffer.byteLength(text),
    sha256: createHash('sha256').update(text).digest('hex'),
  } },
  accessibility: { state: 'not_requested' },
  url: { state: 'captured', value: url },
  history: { state: 'not_requested' },
  storage: { state: 'not_requested' },
};
function capture(requestBody, message) {
  return {
    browser: {
      mode: 'launch', process_ownership: 'provider-owned',
      cleanup: 'terminated-owned-process',
      product: 'Chromium', version: '149.0',
    },
    scenario: {
      start_origin: url.origin, action_count: 1, secret_references: [],
      network_content: {
        request_body: requestBody, response_body: false, header_values: false,
      },
      event_families: ['console', 'network'],
    },
    duration_ms: 10,
    steps: ['scenario_start', 'open-settings'].map((step_id, step_index) => ({
      step_id, step_index,
      action: step_index === 0 ? 'scenario_start' : 'click',
      status: 'completed', elapsed_ms: 10,
      before_url: url, after_url: url, error: null,
      event_sequence_start: step_index + 1,
      event_sequence_end: step_index + 1,
      artifacts, completeness,
    })),
    events: {
      retained: 2, dropped: 0,
      items: [0, 1].map(step_index => ({
        sequence: step_index + 1, step_index,
        kind: 'console', level: 'info', text: message, url: null,
      })),
    },
    completeness, limitations: [],
  };
}
for (const [name, value] of [
  ['before', capture(true, 'before')],
  ['after', capture(false, 'after')],
  ['after-matching-selection', capture(true, 'after')],
  ['after-quiet', capture(false, 'before')],
]) writeFileSync(`${directory}/${name}.json`, JSON.stringify(value));
JS

Run the failing comparison and both controls, then inspect the returned event records:

repro_run node scripts/rea.mjs compare-web-captures \
  repro-content-selection/before.json repro-content-selection/after.json \
  --json > repro-content-selection/result.json
repro_run node scripts/rea.mjs compare-web-captures \
  repro-content-selection/before.json repro-content-selection/after-matching-selection.json \
  --json > repro-content-selection/matching-selection-result.json
repro_run node scripts/rea.mjs compare-web-captures \
  repro-content-selection/before.json repro-content-selection/after-quiet.json \
  --json > repro-content-selection/quiet-result.json
repro_run node --input-type=module <<'JS'
import { readFileSync } from 'node:fs';
for (const name of ['result', 'matching-selection-result', 'quiet-result']) {
  const raw = readFileSync(`repro-content-selection/${name}.json`, 'utf8');
  const result = JSON.parse(raw).normalized_result;
  console.log(JSON.stringify({
    name, overall_status: result.overall_status,
    events: result.steps.flatMap(step => step.artifact_diffs
      .filter(diff => diff.artifact === 'events')
      .map(diff => ({ step: step.step_id, status: diff.status, reason: diff.reason }))),
    recovery_present: raw.includes('Network content selections differ'),
  }));
}
JS

Expected behavior

docs/browser-scenario-contract.md:265-267 promises unknown network evidence and a recovery instruction when resolved scenario.network_content selections differ. docs/browser-scenario-contract.md:306-307 admits complete scenario capture objects through compare-web-captures. The result must expose the network uncertainty and tell the caller to recapture with matching content selections even when console events differ. Independently observed console differences can still support changed step and overall results.

Impact and scope

This is a P2 evidence-reporting defect for a bounded comparison input. Callers receive no network-content mismatch diagnosis or recovery guidance anywhere in the failing comparison result. The observed console changes remain valid. No caller misattribution or security impact was demonstrated.

Both input files retain their explicit network-content selection fields. Callers can compare those fields and recapture with matching selections before interpreting network evidence. The executed matching-selection control shows that console changes remain reportable after alignment. Only the local CLI was exercised; MCP transport was not replayed.

REA version

6.4.0 (main at 5830088520)

Runtime and platform

Node 26.10.0; macOS. The verifier did not record an OS version or architecture in the draft.

Target and artifact context

Synthetic complete browser scenario captures with identical browser identity, origin, two aligned steps, and event-family selection ["console", "network"]. The captures differ in request-body selection and console messages. Chromium 149.0 is synthetic capture metadata, not a browser version exercised by this reproduction.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

All three comparison commands exited zero.

result: overall_status=changed
  scenario_start: events status=changed, reason=null
  open-settings: events status=changed, reason=null
  recovery_present=false
matching-selection-result: overall_status=changed
  Both event records: status=changed, reason=null
  recovery_present=false
quiet-result: overall_status=unknown
  Both event records: status=unknown
  reason=Network content selections differ; capture both sides with matching content selection before comparing network evidence.
  recovery_present=true

Typed outcome or recovery expected

The failing pair must explicitly report unknown network comparability caused by the selection mismatch and include Network content selections differ; capture both sides with matching content selection before comparing network evidence. The observed console changes must remain reportable. The matching-selection control must keep its changed result without a mismatch warning, and the quiet control must keep its unknown result with that warning. The contract can be restored through an artifact record or another explicit result field; it does not require forcing the entire comparison to unknown.

Analysis or suspected cause

Source-derived at the reported baseline: src/domain/browserScenarioDiff.ts:351-365 passes the content-selection warning through incomparableReason, while familyIncomparableReason carries the separate family warning. The return at src/domain/browserScenarioDiff.ts:417-430 reports changed ordinary events with a null reason before the unknown/reason branch at src/domain/browserScenarioDiff.ts:431-453. src/domain/browserScenarioDiffEvidence.ts:148-158 defines the omitted recovery instruction. This control flow is consistent with the independently observed CLI outputs.

Additional context

Related closed issue #2169 concerns false equality from mismatched event-family coverage; this report uses matching event families and requires retaining network-content recovery guidance alongside valid console changes.

Related: #3172 reports a separate failure in the same area; the two have independent fixes.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3173 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `compare-web-captures` omits the documented network-content comparability warning when complete, aligned scenario captures declare different request-body selections and have different console text at both steps. The observed CLI comparison exits zero, returns `overall_status: "changed"`, and gives both steps an `events` diff with `status: "changed"` and `reason: null`. The full response contains no network-selection recovery instruction. The console differences support a changed result. The failure is the loss of the independent network uncertainty and the instruction to capture matching content selections. A matching-selection control also returns changed records with null reasons. A quiet control with mismatched content selection reports unknown at both steps and retains the recovery instruction. ### Reproduction Use a clean checkout of revision `58300885207133422af235e0f2b10658cf958b03` with its pinned toolchain. These commands create only synthetic local capture files; no browser or reachable website is required. The independent verifier replayed the fixture and comparison commands against the prepared built runtime; dependency installation and the build were not rerun during that verification. 1. Install and build, using a disposable local home for the commands: ```sh mkdir -p repro-content-selection/home repro_run() { env HOME="$PWD/repro-content-selection/home" \ XDG_CONFIG_HOME="$PWD/repro-content-selection/home/.config" \ XDG_CACHE_HOME="$PWD/repro-content-selection/home/.cache" \ XDG_DATA_HOME="$PWD/repro-content-selection/home/.local/share" "$@" } repro_run npm ci repro_run npm run build:cached ``` 2. Create complete captures with the same selected event families, browser identity, steps and artifacts. Request-body selection differs, and the console message differs at both steps. Also create controls with matching content selection and matching console messages: ```sh repro_run node --input-type=module <<'JS' import { createHash } from 'node:crypto'; import { writeFileSync } from 'node:fs'; const directory = 'repro-content-selection'; const url = { url: 'https://app.example.test/settings', origin: 'https://app.example.test', query_parameter_names: [], redacted: false, }; const completeness = { status: 'complete', equality_eligible: true, missing_sections: [], truncated_sections: [], }; const text = '<main>Ready</main>'; const artifacts = { screenshot: { state: 'not_requested' }, dom: { state: 'captured', value: { text, bytes: Buffer.byteLength(text), sha256: createHash('sha256').update(text).digest('hex'), } }, accessibility: { state: 'not_requested' }, url: { state: 'captured', value: url }, history: { state: 'not_requested' }, storage: { state: 'not_requested' }, }; function capture(requestBody, message) { return { browser: { mode: 'launch', process_ownership: 'provider-owned', cleanup: 'terminated-owned-process', product: 'Chromium', version: '149.0', }, scenario: { start_origin: url.origin, action_count: 1, secret_references: [], network_content: { request_body: requestBody, response_body: false, header_values: false, }, event_families: ['console', 'network'], }, duration_ms: 10, steps: ['scenario_start', 'open-settings'].map((step_id, step_index) => ({ step_id, step_index, action: step_index === 0 ? 'scenario_start' : 'click', status: 'completed', elapsed_ms: 10, before_url: url, after_url: url, error: null, event_sequence_start: step_index + 1, event_sequence_end: step_index + 1, artifacts, completeness, })), events: { retained: 2, dropped: 0, items: [0, 1].map(step_index => ({ sequence: step_index + 1, step_index, kind: 'console', level: 'info', text: message, url: null, })), }, completeness, limitations: [], }; } for (const [name, value] of [ ['before', capture(true, 'before')], ['after', capture(false, 'after')], ['after-matching-selection', capture(true, 'after')], ['after-quiet', capture(false, 'before')], ]) writeFileSync(`${directory}/${name}.json`, JSON.stringify(value)); JS ``` 3. Run the failing comparison and both controls, then inspect the returned event records: ```sh repro_run node scripts/rea.mjs compare-web-captures \ repro-content-selection/before.json repro-content-selection/after.json \ --json > repro-content-selection/result.json repro_run node scripts/rea.mjs compare-web-captures \ repro-content-selection/before.json repro-content-selection/after-matching-selection.json \ --json > repro-content-selection/matching-selection-result.json repro_run node scripts/rea.mjs compare-web-captures \ repro-content-selection/before.json repro-content-selection/after-quiet.json \ --json > repro-content-selection/quiet-result.json repro_run node --input-type=module <<'JS' import { readFileSync } from 'node:fs'; for (const name of ['result', 'matching-selection-result', 'quiet-result']) { const raw = readFileSync(`repro-content-selection/${name}.json`, 'utf8'); const result = JSON.parse(raw).normalized_result; console.log(JSON.stringify({ name, overall_status: result.overall_status, events: result.steps.flatMap(step => step.artifact_diffs .filter(diff => diff.artifact === 'events') .map(diff => ({ step: step.step_id, status: diff.status, reason: diff.reason }))), recovery_present: raw.includes('Network content selections differ'), })); } JS ``` ### Expected behavior `docs/browser-scenario-contract.md:265-267` promises unknown network evidence and a recovery instruction when resolved `scenario.network_content` selections differ. `docs/browser-scenario-contract.md:306-307` admits complete scenario capture objects through `compare-web-captures`. The result must expose the network uncertainty and tell the caller to recapture with matching content selections even when console events differ. Independently observed console differences can still support changed step and overall results. ### Impact and scope This is a P2 evidence-reporting defect for a bounded comparison input. Callers receive no network-content mismatch diagnosis or recovery guidance anywhere in the failing comparison result. The observed console changes remain valid. No caller misattribution or security impact was demonstrated. Both input files retain their explicit network-content selection fields. Callers can compare those fields and recapture with matching selections before interpreting network evidence. The executed matching-selection control shows that console changes remain reportable after alignment. Only the local CLI was exercised; MCP transport was not replayed. ### REA version 6.4.0 (main at 58300885207133422af235e0f2b10658cf958b03) ### Runtime and platform Node 26.10.0; macOS. The verifier did not record an OS version or architecture in the draft. ### Target and artifact context Synthetic complete browser scenario captures with identical browser identity, origin, two aligned steps, and event-family selection `["console", "network"]`. The captures differ in request-body selection and console messages. Chromium 149.0 is synthetic capture metadata, not a browser version exercised by this reproduction. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence ```text All three comparison commands exited zero. result: overall_status=changed scenario_start: events status=changed, reason=null open-settings: events status=changed, reason=null recovery_present=false matching-selection-result: overall_status=changed Both event records: status=changed, reason=null recovery_present=false quiet-result: overall_status=unknown Both event records: status=unknown reason=Network content selections differ; capture both sides with matching content selection before comparing network evidence. recovery_present=true ``` ### Typed outcome or recovery expected The failing pair must explicitly report unknown network comparability caused by the selection mismatch and include `Network content selections differ; capture both sides with matching content selection before comparing network evidence.` The observed console changes must remain reportable. The matching-selection control must keep its changed result without a mismatch warning, and the quiet control must keep its unknown result with that warning. The contract can be restored through an artifact record or another explicit result field; it does not require forcing the entire comparison to unknown. ### Analysis or suspected cause Source-derived at the reported baseline: `src/domain/browserScenarioDiff.ts:351-365` passes the content-selection warning through `incomparableReason`, while `familyIncomparableReason` carries the separate family warning. The return at `src/domain/browserScenarioDiff.ts:417-430` reports changed ordinary events with a null reason before the unknown/reason branch at `src/domain/browserScenarioDiff.ts:431-453`. `src/domain/browserScenarioDiffEvidence.ts:148-158` defines the omitted recovery instruction. This control flow is consistent with the independently observed CLI outputs. ### Additional context Related closed issue [#2169](https://github.com/morluto/rea/issues/2169) concerns false equality from mismatched event-family coverage; this report uses matching event families and requires retaining network-content recovery guidance alongside valid console changes. Related: #3172 reports a separate failure in the same area; the two have independent fixes.

bsakaguchi

2026-10-15 18:31:10 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-15 18:31:11 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 14, 2026):

Heads up: this looks fixed on main in 2b60e097f. The change is in src/domain/browserScenarioDiff.ts, and retains network guidance beside a changed console event covers it. It is listed on tracker #3582. If it still happens on main, please say so here and I'll take it off the list.

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@CassieFox23 commented on GitHub (Oct 14, 2026): Heads up: this looks fixed on main in 2b60e097f. The change is in `src/domain/browserScenarioDiff.ts`, and `retains network guidance beside a changed console event` covers it. It is listed on tracker #3582. If it still happens on main, please say so here and I'll take it off the list. 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

bsakaguchi commented

2026-10-15 18:31:11 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Resolved on main by 2b60e097f.

Known console/event changes retain the independent network-content recovery guidance.

Rechecked on main b8510ee00 in the current focused run: 20 test files passed, 225 tests passed and one platform-dependent test skipped. Relevant regression coverage: src/domain/browserScenarioDiff.test.ts and tests/boundary/cli/browserCli.test.ts.

@morluto commented on GitHub (Oct 14, 2026): Resolved on main by 2b60e097f. Known console/event changes retain the independent network-content recovery guidance. Rechecked on main b8510ee00 in the current focused run: 20 test files passed, 225 tests passed and one platform-dependent test skipped. Relevant regression coverage: src/domain/browserScenarioDiff.test.ts and tests/boundary/cli/browserCli.test.ts.

bsakaguchi referenced this issue

2026-10-17 21:08:46 +00:00

[PR #1116] [MERGED] fix(build): exclude the test-only generated catalog from packages #2661

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1113

No description provided.

Rows
Columns