morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3879] [Bug] compare-web-captures reports scripts unchanged when retained source digests differ #1500

New issue

Closed

opened 2026-10-16 21:06:13 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:06:13 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3879

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

rea compare-web-captures reports the script dimension as unchanged when two passive captures retain different script source artifacts (different text and SHA-256) for the same URL, length, and metadata with cdp_hash: null. It also reports unchanged when neither capture retains source and script_sources is listed as unavailable, so missing content identity is treated as equivalence.

All four comparisons exit 0 and report scripts as status: "unchanged", total_changes: 0, changes: [], reason: null, including the two pairs with different source SHA-256 values and the pair with unavailable source. The before vs before control gives the same expected unchanged result. The fixtures' overall comparison status is unknown for unrelated reasons (other dimensions have no observations); the failure is specific to the script dimension.

Reproduction

Save the script below as repro.mjs in an empty work directory outside a built REA source checkout. It imports the checkout's built captureSnapshotSchema and stableWebScriptKey so every fixture is schema-valid, writes three normalized passive captures (before, after with different script text, and unavailable with no retained source), and runs the public CLI on four pairs. The example URLs are never fetched.
From the checkout root, run node <work>/repro.mjs "$PWD" <work>.

import { createHash } from 'node:crypto';
import { writeFile } from 'node:fs/promises';
import { spawnSync } from 'node:child_process';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const repo = resolve(process.argv[2]);
const work = resolve(process.argv[3]);
const { captureSnapshotSchema } = await import(pathToFileURL(resolve(repo, 'dist/domain/webCaptureDiffSchemas.js')));
const { stableWebScriptKey } = await import(pathToFileURL(resolve(repo, 'dist/domain/webInventory.js')));
const artifact = text => ({text, sha256: createHash('sha256').update(text).digest('hex'), bytes: Buffer.byteLength(text), media_type: 'text/javascript', charset: 'utf-8'});
const origin = 'https://capture.test';
const time = new Date().toISOString();
const metadata = {url: origin + '/app.js', cdp_hash: null, length: 18, is_module: true, language: 'JavaScript', source_map_url: null};
const capture = captureSnapshotSchema.parse({inspection: {
  browser: {product: 'normalized-fixture', protocol_version: '1.3', revision: 'fixture', user_agent: 'fixture', js_version: 'fixture'},
  target: {target_id: 'page', type: 'page', title: 'Fixture', url: origin, origin, attached: false},
  capture_window: {started_at: time, ended_at: time, observation_ms: 0},
  completeness: {
    status: 'complete_within_window', conditions: ['complete_within_window'],
    policy_filtered_sections: [], attach_limited_sections: [], truncated_sections: [], unavailable_sections: [], excluded: [],
    dropped_events: {scripts: 0, network_requests: 0, console_events: 0, websocket_connections: 0, websocket_frames: 0, webmcp_tools: 0, timeline_events: 0, total: 0}
  },
  frames: [], dom: {total_nodes: 0, nodes: []},
  accessibility: {total_nodes: 0, nodes: [], text_capture: {status: 'not_approved', retained_bytes: 0, excluded_fields: 0}},
  scripts: {total: 1, items: [{...metadata, script_key: stableWebScriptKey(metadata), origin, frame_id: null, resource_reconciliation: {status: 'unmatched', reason: 'no_exact_sanitized_url'}, source: {included: true, artifact: artifact('export let x = 1;\n')}}]},
  resources: [], network: {requests: [], websocket_connections: [], coverage_started_at: time, prior_activity_available: false},
  console: {events: [], coverage_started_at: time, prior_activity_available: false}, workers: [],
  metadata: {responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true},
  storage: {origin, usage_bytes: null, quota_bytes: null, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], values_redacted: true},
  limitations: []
}, webmcp: null});
const changed = structuredClone(capture);
changed.inspection.scripts.items[0].source.artifact = artifact('export let x = 2;\n');
const unavailable = structuredClone(capture);
unavailable.inspection.scripts.items[0].source = {included: false, reason: 'source unavailable'};
unavailable.inspection.completeness.unavailable_sections.push('script_sources');
for (const [name, value] of [['before', capture], ['after', changed], ['unavailable', unavailable]])
  await writeFile(resolve(work, name + '.json'), JSON.stringify(captureSnapshotSchema.parse(value)));
for (const [left, right] of [['before', 'after'], ['after', 'before'], ['before', 'before'], ['unavailable', 'unavailable']]) {
  const result = spawnSync('node', [resolve(repo, 'scripts/rea.mjs'), 'compare-web-captures', resolve(work, left+'.json'), resolve(work, right+'.json'), '--json'], {encoding: 'utf8', timeout: 30000});
  const body = JSON.parse(result.stdout);
  console.log(left+' vs '+right, JSON.stringify({exit: result.status, scripts: body.normalized_result.dimensions.scripts}));
}
console.log('source digests', capture.inspection.scripts.items[0].source.artifact.sha256, changed.inspection.scripts.items[0].source.artifact.sha256);

Expected behavior

before vs after (and the reverse) reports a script change, because the retained, self-verifying source artifacts differ. unavailable vs unavailable does not claim equivalence; with no producer hash and no retained source, the script dimension keeps its content identity unknown. docs/browser-observation.md says capture comparison never treats missing or truncated evidence as equivalence, and the capture schema describes a null cdp_hash as unavailable rather than equal.

Impact and scope

Analysts comparing passive captures receive false script equivalence despite an observed source change or unavailable content identity; the defect is limited to comparison output, and the stored captures keep their correct source artifacts.

Workaround: available. Comparing scripts.items[*].source.artifact.sha256 in the two capture files directly shows the difference in this reproduction.

Reporter's impact estimate: P2 (supported).

REA version

6.5.0, built from source at main commit 034342ba28 (latest npm release is 6.5.0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64

Target and artifact context

Normalized passive web capture JSON files (synthetic, schema-valid)

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Expected source: docs/browser-observation.md:22 "Shipped surfaces" (missing or truncated evidence is never treated as equivalence); src/domain/browserObservationSchemas.ts:97 (cdp_hash may be null when unavailable)
Failure source: observed rea compare-web-captures script-dimension output for the schema-valid fixtures above
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

before vs after {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
after vs before {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
before vs before {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
unavailable vs unavailable {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
source digests fa74d908f2a3b8df62bdca39067055c8920c08b334ab01c7e98e60b65ef5ee1e 108158e7f1fa75a400b4e20555174d75d62e03e472f271e93402475c2b7866cf

The inputs are synthetic normalized captures, not captures from a live Chromium process, so how often real producers omit the script hash was not measured. Source-derived reasoning: src/domain/webCaptureDiff.ts:184 fingerprints only scriptProjection and checks only the scripts inventory section for completeness, and scriptProjection at src/domain/webCaptureDiff.ts:813 excludes the retained source artifact, its digest, and the source availability state. The stable script key at src/domain/webInventory.ts:101 also excludes the artifact digest.

Typed outcome or recovery expected

The script dimension should report a change when retained source artifacts differ, and an unknown or limited status when neither a producer hash nor retained source establishes content identity.

Restoration check:

On the fixtures above, before vs after and after vs before currently report "status":"unchanged" for scripts; after a fix they report a script change (or at least a non-equivalent status) that reflects the different source digests. unavailable vs unavailable currently reports unchanged; after a fix it reports an unknown or limited status rather than equivalence. Control: before vs before continues to report scripts unchanged.

Analysis or suspected cause

Source-derived: src/domain/webCaptureDiff.ts:184 fingerprints only scriptProjection, which at src/domain/webCaptureDiff.ts:813 excludes the retained source artifact, digest, and availability; src/domain/webInventory.ts:101 also excludes the digest from the stable key.

Additional context

This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3879 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `rea compare-web-captures` reports the script dimension as `unchanged` when two passive captures retain different script source artifacts (different text and SHA-256) for the same URL, length, and metadata with `cdp_hash: null`. It also reports `unchanged` when neither capture retains source and `script_sources` is listed as unavailable, so missing content identity is treated as equivalence. All four comparisons exit 0 and report `scripts` as `status: "unchanged"`, `total_changes: 0`, `changes: []`, `reason: null`, including the two pairs with different source SHA-256 values and the pair with unavailable source. The `before` vs `before` control gives the same expected unchanged result. The fixtures' overall comparison status is unknown for unrelated reasons (other dimensions have no observations); the failure is specific to the script dimension. ### Reproduction 1. Save the script below as `repro.mjs` in an empty work directory outside a built REA source checkout. It imports the checkout's built `captureSnapshotSchema` and `stableWebScriptKey` so every fixture is schema-valid, writes three normalized passive captures (`before`, `after` with different script text, and `unavailable` with no retained source), and runs the public CLI on four pairs. The example URLs are never fetched. 2. From the checkout root, run `node <work>/repro.mjs "$PWD" <work>`. ```javascript import { createHash } from 'node:crypto'; import { writeFile } from 'node:fs/promises'; import { spawnSync } from 'node:child_process'; import { resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; const repo = resolve(process.argv[2]); const work = resolve(process.argv[3]); const { captureSnapshotSchema } = await import(pathToFileURL(resolve(repo, 'dist/domain/webCaptureDiffSchemas.js'))); const { stableWebScriptKey } = await import(pathToFileURL(resolve(repo, 'dist/domain/webInventory.js'))); const artifact = text => ({text, sha256: createHash('sha256').update(text).digest('hex'), bytes: Buffer.byteLength(text), media_type: 'text/javascript', charset: 'utf-8'}); const origin = 'https://capture.test'; const time = new Date().toISOString(); const metadata = {url: origin + '/app.js', cdp_hash: null, length: 18, is_module: true, language: 'JavaScript', source_map_url: null}; const capture = captureSnapshotSchema.parse({inspection: { browser: {product: 'normalized-fixture', protocol_version: '1.3', revision: 'fixture', user_agent: 'fixture', js_version: 'fixture'}, target: {target_id: 'page', type: 'page', title: 'Fixture', url: origin, origin, attached: false}, capture_window: {started_at: time, ended_at: time, observation_ms: 0}, completeness: { status: 'complete_within_window', conditions: ['complete_within_window'], policy_filtered_sections: [], attach_limited_sections: [], truncated_sections: [], unavailable_sections: [], excluded: [], dropped_events: {scripts: 0, network_requests: 0, console_events: 0, websocket_connections: 0, websocket_frames: 0, webmcp_tools: 0, timeline_events: 0, total: 0} }, frames: [], dom: {total_nodes: 0, nodes: []}, accessibility: {total_nodes: 0, nodes: [], text_capture: {status: 'not_approved', retained_bytes: 0, excluded_fields: 0}}, scripts: {total: 1, items: [{...metadata, script_key: stableWebScriptKey(metadata), origin, frame_id: null, resource_reconciliation: {status: 'unmatched', reason: 'no_exact_sanitized_url'}, source: {included: true, artifact: artifact('export let x = 1;\n')}}]}, resources: [], network: {requests: [], websocket_connections: [], coverage_started_at: time, prior_activity_available: false}, console: {events: [], coverage_started_at: time, prior_activity_available: false}, workers: [], metadata: {responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true}, storage: {origin, usage_bytes: null, quota_bytes: null, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], values_redacted: true}, limitations: [] }, webmcp: null}); const changed = structuredClone(capture); changed.inspection.scripts.items[0].source.artifact = artifact('export let x = 2;\n'); const unavailable = structuredClone(capture); unavailable.inspection.scripts.items[0].source = {included: false, reason: 'source unavailable'}; unavailable.inspection.completeness.unavailable_sections.push('script_sources'); for (const [name, value] of [['before', capture], ['after', changed], ['unavailable', unavailable]]) await writeFile(resolve(work, name + '.json'), JSON.stringify(captureSnapshotSchema.parse(value))); for (const [left, right] of [['before', 'after'], ['after', 'before'], ['before', 'before'], ['unavailable', 'unavailable']]) { const result = spawnSync('node', [resolve(repo, 'scripts/rea.mjs'), 'compare-web-captures', resolve(work, left+'.json'), resolve(work, right+'.json'), '--json'], {encoding: 'utf8', timeout: 30000}); const body = JSON.parse(result.stdout); console.log(left+' vs '+right, JSON.stringify({exit: result.status, scripts: body.normalized_result.dimensions.scripts})); } console.log('source digests', capture.inspection.scripts.items[0].source.artifact.sha256, changed.inspection.scripts.items[0].source.artifact.sha256); ``` ### Expected behavior `before` vs `after` (and the reverse) reports a script change, because the retained, self-verifying source artifacts differ. `unavailable` vs `unavailable` does not claim equivalence; with no producer hash and no retained source, the script dimension keeps its content identity unknown. `docs/browser-observation.md` says capture comparison never treats missing or truncated evidence as equivalence, and the capture schema describes a null `cdp_hash` as unavailable rather than equal. ### Impact and scope Analysts comparing passive captures receive false script equivalence despite an observed source change or unavailable content identity; the defect is limited to comparison output, and the stored captures keep their correct source artifacts. Workaround: available. Comparing `scripts.items[*].source.artifact.sha256` in the two capture files directly shows the difference in this reproduction. Reporter's impact estimate: P2 (supported). ### REA version 6.5.0, built from source at main commit 034342ba28c8e56c3746be92b78aa7299afa02bd (latest npm release is 6.5.0) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64 ### Target and artifact context Normalized passive web capture JSON files (synthetic, schema-valid) ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence - Expected source: docs/browser-observation.md:22 "Shipped surfaces" (missing or truncated evidence is never treated as equivalence); src/domain/browserObservationSchemas.ts:97 (`cdp_hash` may be null when unavailable) - Failure source: observed `rea compare-web-captures` script-dimension output for the schema-valid fixtures above - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete ```text before vs after {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} after vs before {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} before vs before {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} unavailable vs unavailable {"exit":0,"scripts":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} source digests fa74d908f2a3b8df62bdca39067055c8920c08b334ab01c7e98e60b65ef5ee1e 108158e7f1fa75a400b4e20555174d75d62e03e472f271e93402475c2b7866cf ``` The inputs are synthetic normalized captures, not captures from a live Chromium process, so how often real producers omit the script hash was not measured. Source-derived reasoning: `src/domain/webCaptureDiff.ts:184` fingerprints only `scriptProjection` and checks only the `scripts` inventory section for completeness, and `scriptProjection` at `src/domain/webCaptureDiff.ts:813` excludes the retained source artifact, its digest, and the source availability state. The stable script key at `src/domain/webInventory.ts:101` also excludes the artifact digest. ### Typed outcome or recovery expected The script dimension should report a change when retained source artifacts differ, and an unknown or limited status when neither a producer hash nor retained source establishes content identity. Restoration check: On the fixtures above, `before vs after` and `after vs before` currently report `"status":"unchanged"` for scripts; after a fix they report a script change (or at least a non-equivalent status) that reflects the different source digests. `unavailable vs unavailable` currently reports `unchanged`; after a fix it reports an unknown or limited status rather than equivalence. Control: `before vs before` continues to report scripts `unchanged`. ### Analysis or suspected cause Source-derived: src/domain/webCaptureDiff.ts:184 fingerprints only scriptProjection, which at src/domain/webCaptureDiff.ts:813 excludes the retained source artifact, digest, and availability; src/domain/webInventory.ts:101 also excludes the digest from the stable key. ### Additional context This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

bsakaguchi commented

2026-10-16 21:06:14 +00:00

Author

Owner

@z2Ace0107 commented on GitHub (Oct 15, 2026):

I'd like to take this on. I reproduced the issue on Windows with the pinned Node 24.18.0/npm 11.16.0 toolchain: different retained source digests are reported as unchanged, and unavailable source identity is also treated as unchanged. I'll keep the fix scoped to offline script comparison and add public CLI regression coverage, preserving unknown results when content identity cannot be established.

@z2Ace0107 commented on GitHub (Oct 15, 2026): I'd like to take this on. I reproduced the issue on Windows with the pinned Node 24.18.0/npm 11.16.0 toolchain: different retained source digests are reported as unchanged, and unavailable source identity is also treated as unchanged. I'll keep the fix scoped to offline script comparison and add public CLI regression coverage, preserving unknown results when content identity cannot be established.

bsakaguchi referenced this issue

2026-10-17 21:11:59 +00:00

[PR #1513] [MERGED] fix: bind identity and coverage to the producing rule #2933

bsakaguchi closed this issue

2026-10-31 00:31:05 +00:00

bsakaguchi commented

2026-10-31 00:31:05 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

scriptsDimension groups by URL and, when both corresponding groups retain source, fingerprints artifact.sha256; cdp_hash:null no longer discards retained source identity. When neither side has source and producer hash is null, valuesComplete becomes false and the dimension cannot report unchanged absent another proved difference. The acceptance fixtures explicitly use null cdp_hash, same metadata and same-length different text (1 vs 2), assert changed for digest differences, unchanged for agreement, and unknown for unavailable source on one/both sides.

Implementation: webCaptureDiff.ts, webCaptureDiff.ts.

The relevant regression checks in passiveCaptureCorrespondence.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. scriptsDimension groups by URL and, when both corresponding groups retain source, fingerprints artifact.sha256; cdp_hash:null no longer discards retained source identity. When neither side has source and producer hash is null, valuesComplete becomes false and the dimension cannot report unchanged absent another proved difference. The acceptance fixtures explicitly use null cdp_hash, same metadata and same-length different text (1 vs 2), assert changed for digest differences, unchanged for agreement, and unknown for unavailable source on one/both sides. Implementation: [webCaptureDiff.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/webCaptureDiff.ts#L998), [webCaptureDiff.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/webCaptureDiff.ts#L1037). The relevant regression checks in [passiveCaptureCorrespondence.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/acceptance/tools/passiveCaptureCorrespondence.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 00:40:52 +00:00

[GH-ISSUE #5725] [Bug] Script-source reads mask the target_changed outcome after a page reload #3473

bsakaguchi referenced this issue

2026-10-31 00:44:07 +00:00

[GH-ISSUE #6231] [Bug] Source-enabled page inspection counts a late script without returning it or reporting its omission #3793