morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3385] [Bug] Truncated accessibility trees falsely report modifications #1252

New issue

Closed

opened 2026-10-16 21:03:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:03:23 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3385

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

compare-web-captures reports an accessibility modification when the later capture omits one node and explicitly marks accessibility coverage truncated. The remaining node has identical semantics. The command exits with status 0 and returns overall_status: "changed", with one accessibility_tree modification and no reason. The incomplete tree cannot establish that modification.

The same result states that accessibility roles, text, and hierarchy are compared only when the tree was fully captured. Comparing the complete fixture with itself returns dimensions.accessibility.status: "unchanged".

A separate executed production-provider check used a local synthetic CDP peer. The first accessibility reply contained a root and button; the second retained the root's child reference but omitted the button record. REA marked accessibility truncated and still returned the same modification claim through the CLI. This confirms that the incomplete normalized fixture represents a reachable provider outcome.

Reproduction

Use a clean checkout at 8dde6370d0809ad04cda3342d4e6f290f48aa063, install the locked dependencies, and build the runtime.

npm ci
npm run build:cached

From the repository root, create both normalized capture inputs inline and compare them through the public CLI. The URLs are inert recorded identities; this command does not fetch them. No browser is required.

REA_ENTRY="$PWD/scripts/rea.mjs"
mkdir capture-comparison-repro
cd capture-comparison-repro
mkdir isolated-home
node --input-type=module <<'JS'
import { writeFileSync } from 'node:fs';
const origin = 'https://example.test';
const url = origin + '/page';
const time = '2026-01-01T00:00:00.000Z';
const completeness = () => ({
  status: 'attach_limited', conditions: ['attach_limited'],
  policy_filtered_sections: [],
  attach_limited_sections: ['network_requests', 'console_events', 'websocket_connections', 'websocket_frames'],
  truncated_sections: [], unavailable_sections: [], excluded: [],
  dropped_events: Object.fromEntries(['scripts', 'network_requests', 'console_events', 'websocket_connections', 'websocket_frames', 'webmcp_tools', 'timeline_events', 'total'].map(key => [key, 0]))
});
const node = (id, parent, role, name) => ({node_id: id, parent_id: parent, role, name, description: null, ignored: false, states: []});
const inspection = () => ({
  browser: {product: 'Fixture/1', protocol_version: '1.3', revision: 'fixture', user_agent: 'Fixture', js_version: '1'},
  target: {target_id: 'page', type: 'page', title: 'Page', url, origin, attached: false},
  capture_window: {started_at: time, ended_at: time, observation_ms: 0},
  completeness: completeness(),
  frames: [{frame_id: 'main', parent_frame_id: null, url, origin}],
  dom: {total_nodes: 0, nodes: []},
  accessibility: {total_nodes: 2, text_capture: {status: 'included', retained_bytes: 10, excluded_fields: 0}, nodes: [node('root', null, 'RootWebArea', 'Page'), node('button', 'root', 'button', 'Submit')]},
  scripts: {total: 0, items: []}, resources: [],
  network: {requests: [], websocket_connections: [], coverage_started_at: time, prior_activity_available: false},
  console: {events: [], coverage_started_at: time, prior_activity_available: false}, workers: [],
  metadata: {responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true},
  storage: {origin, usage_bytes: 0, quota_bytes: 1024, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], content_fingerprints: [], fingerprint_algorithm: 'sha256', fingerprints_complete: true, values_redacted: true},
  limitations: []
});
const save = (name, value) => writeFileSync(name, JSON.stringify(value));
const before = {inspection: inspection()};
const after = structuredClone(before);
after.inspection.accessibility.nodes.pop();
after.inspection.accessibility.total_nodes = 1;
after.inspection.accessibility.text_capture.retained_bytes = 4;
after.inspection.completeness.status = 'truncated';
after.inspection.completeness.conditions.push('truncated');
after.inspection.completeness.truncated_sections.push('accessibility');
save('before.json', before); save('after.json', after);
JS
env HOME="$PWD/isolated-home" node "$REA_ENTRY" compare-web-captures before.json after.json --format json --full-output

Run the complete self-comparison control from the same directory.

env HOME="$PWD/isolated-home" node "$REA_ENTRY" compare-web-captures before.json before.json --format json --full-output

Expected behavior

The accessibility dimension should remain unknown for this pair, with an explanation of incomplete tree coverage. An omitted node must not establish a tree modification. Other dimensions may still report independently supported differences.

src/domain/webCaptureDiff.ts:43 states that accessibility semantics and hierarchy are compared only with a fully captured tree and selected, untruncated text. docs/tool-design.md:57 requires missing coverage to remain unknown.

Impact and scope

P2, supported. Incomplete accessibility observations produce a false positive website-change claim, including an overall changed result. The executed case is bounded to incomplete tree coverage; no source or capture bytes are modified.

A workaround is available. Callers can inspect the existing accessibility completeness markers and decline the unsupported modification claim. The reproduction directly verifies the truncated marker; this workaround requires checking the retained captures instead of trusting the comparison status.

REA version

6.4.0 (main at 8dde6370d0)

Runtime and platform

Node.js v26.10.0 and macOS 27.0.1, using the compiled runtime and locked dependencies. Architecture was not recorded. The CLI and a local synthetic CDP peer were exercised; these observations do not establish a particular real Chrome version's occurrence frequency.

Target and artifact context

Two local normalized passive page-capture JSON fixtures. The complete fixture contains a root and Submit button. The truncated fixture omits the button, retains identical semantics for the root, reduces the node and retained-text counts, and explicitly includes accessibility in completeness.truncated_sections. The recorded identity is https://example.test/page; the offline comparison never fetches it. Additional provider verification used a local synthetic CDP peer with an omitted child record.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Comparison exit status: 0
 overall_status: "changed"
 accessibility dimension:
 {
   "status": "changed",
   "total_changes": 1,
   "changes": [{"identity": "accessibility_tree", "change": "modified"}],
   "reason": null
 }
Complete self-comparison: dimensions.accessibility.status: "unchanged"

The evidence comes from observed CLI reproductions. The local reproduction is complete, and the separate synthetic-CDP provider check also returned the modification claim with accessibility marked truncated.

Typed outcome or recovery expected

Rerun the pair above. Accessibility should return unknown with no modification inferred from the omitted node and a reason explaining incomplete tree coverage. Keep the complete self-comparison control unchanged. With two complete captures that actually differ in a button role or name, a real modification must still be reported. The complete-tree change is a restoration check, not an additional observed result.

Analysis or suspected cause

Source-derived reasoning: src/domain/webCaptureDiff.ts:102 fingerprints the accessibility tree as one aggregate, and src/domain/webCaptureDiff.ts:77 retains a modified fingerprint despite incomplete coverage. A node omitted from an incomplete tree changes that aggregate without proving a semantic modification. This reasoning agrees with the observed CLI result.

The provider's missing-child guard is src/browser/CdpCaptureDocuments.ts:496, with truncation recorded at src/browser/CdpPageCapture.ts:424. The separate executed synthetic-CDP check confirms that this producer path can supply the incomplete tree and marker consumed by the failing comparison.

Additional context

Related closed feature #377 introduced accessibility and storage comparison; this report identifies a specific false modification from an explicitly truncated accessibility tree.

Related open issue #3172 concerns unavailable redirect response fields causing false network conclusions; this report concerns accessibility node coverage and requires a separate comparison correction.

Related closed issue #1830 corrected false changes from unavailable top-level response metadata; this report concerns incomplete accessibility tree aggregates.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3385 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `compare-web-captures` reports an accessibility modification when the later capture omits one node and explicitly marks accessibility coverage truncated. The remaining node has identical semantics. The command exits with status 0 and returns `overall_status: "changed"`, with one `accessibility_tree` modification and no reason. The incomplete tree cannot establish that modification. The same result states that accessibility roles, text, and hierarchy are compared only when the tree was fully captured. Comparing the complete fixture with itself returns `dimensions.accessibility.status: "unchanged"`. A separate executed production-provider check used a local synthetic CDP peer. The first accessibility reply contained a root and button; the second retained the root's child reference but omitted the button record. REA marked accessibility truncated and still returned the same modification claim through the CLI. This confirms that the incomplete normalized fixture represents a reachable provider outcome. ### Reproduction 1. Use a clean checkout at `8dde6370d0809ad04cda3342d4e6f290f48aa063`, install the locked dependencies, and build the runtime. ```sh npm ci npm run build:cached ``` 2. From the repository root, create both normalized capture inputs inline and compare them through the public CLI. The URLs are inert recorded identities; this command does not fetch them. No browser is required. ```sh REA_ENTRY="$PWD/scripts/rea.mjs" mkdir capture-comparison-repro cd capture-comparison-repro mkdir isolated-home node --input-type=module <<'JS' import { writeFileSync } from 'node:fs'; const origin = 'https://example.test'; const url = origin + '/page'; const time = '2026-01-01T00:00:00.000Z'; const completeness = () => ({ status: 'attach_limited', conditions: ['attach_limited'], policy_filtered_sections: [], attach_limited_sections: ['network_requests', 'console_events', 'websocket_connections', 'websocket_frames'], truncated_sections: [], unavailable_sections: [], excluded: [], dropped_events: Object.fromEntries(['scripts', 'network_requests', 'console_events', 'websocket_connections', 'websocket_frames', 'webmcp_tools', 'timeline_events', 'total'].map(key => [key, 0])) }); const node = (id, parent, role, name) => ({node_id: id, parent_id: parent, role, name, description: null, ignored: false, states: []}); const inspection = () => ({ browser: {product: 'Fixture/1', protocol_version: '1.3', revision: 'fixture', user_agent: 'Fixture', js_version: '1'}, target: {target_id: 'page', type: 'page', title: 'Page', url, origin, attached: false}, capture_window: {started_at: time, ended_at: time, observation_ms: 0}, completeness: completeness(), frames: [{frame_id: 'main', parent_frame_id: null, url, origin}], dom: {total_nodes: 0, nodes: []}, accessibility: {total_nodes: 2, text_capture: {status: 'included', retained_bytes: 10, excluded_fields: 0}, nodes: [node('root', null, 'RootWebArea', 'Page'), node('button', 'root', 'button', 'Submit')]}, scripts: {total: 0, items: []}, resources: [], network: {requests: [], websocket_connections: [], coverage_started_at: time, prior_activity_available: false}, console: {events: [], coverage_started_at: time, prior_activity_available: false}, workers: [], metadata: {responses: [], dom_urls: [], agent_hints: [], excluded_dom_urls: 0, headers_allowlisted: true}, storage: {origin, usage_bytes: 0, quota_bytes: 1024, local_storage_keys: [], session_storage_keys: [], indexed_db_names: [], cache_names: [], content_fingerprints: [], fingerprint_algorithm: 'sha256', fingerprints_complete: true, values_redacted: true}, limitations: [] }); const save = (name, value) => writeFileSync(name, JSON.stringify(value)); const before = {inspection: inspection()}; const after = structuredClone(before); after.inspection.accessibility.nodes.pop(); after.inspection.accessibility.total_nodes = 1; after.inspection.accessibility.text_capture.retained_bytes = 4; after.inspection.completeness.status = 'truncated'; after.inspection.completeness.conditions.push('truncated'); after.inspection.completeness.truncated_sections.push('accessibility'); save('before.json', before); save('after.json', after); JS env HOME="$PWD/isolated-home" node "$REA_ENTRY" compare-web-captures before.json after.json --format json --full-output ``` 3. Run the complete self-comparison control from the same directory. ```sh env HOME="$PWD/isolated-home" node "$REA_ENTRY" compare-web-captures before.json before.json --format json --full-output ``` ### Expected behavior The accessibility dimension should remain `unknown` for this pair, with an explanation of incomplete tree coverage. An omitted node must not establish a tree modification. Other dimensions may still report independently supported differences. `src/domain/webCaptureDiff.ts:43` states that accessibility semantics and hierarchy are compared only with a fully captured tree and selected, untruncated text. `docs/tool-design.md:57` requires missing coverage to remain unknown. ### Impact and scope P2, supported. Incomplete accessibility observations produce a false positive website-change claim, including an overall changed result. The executed case is bounded to incomplete tree coverage; no source or capture bytes are modified. A workaround is available. Callers can inspect the existing accessibility completeness markers and decline the unsupported modification claim. The reproduction directly verifies the truncated marker; this workaround requires checking the retained captures instead of trusting the comparison status. ### REA version 6.4.0 (main at 8dde6370d0809ad04cda3342d4e6f290f48aa063) ### Runtime and platform Node.js v26.10.0 and macOS 27.0.1, using the compiled runtime and locked dependencies. Architecture was not recorded. The CLI and a local synthetic CDP peer were exercised; these observations do not establish a particular real Chrome version's occurrence frequency. ### Target and artifact context Two local normalized passive page-capture JSON fixtures. The complete fixture contains a root and Submit button. The truncated fixture omits the button, retains identical semantics for the root, reduces the node and retained-text counts, and explicitly includes accessibility in `completeness.truncated_sections`. The recorded identity is `https://example.test/page`; the offline comparison never fetches it. Additional provider verification used a local synthetic CDP peer with an omitted child record. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence ```text Comparison exit status: 0 overall_status: "changed" accessibility dimension: { "status": "changed", "total_changes": 1, "changes": [{"identity": "accessibility_tree", "change": "modified"}], "reason": null } Complete self-comparison: dimensions.accessibility.status: "unchanged" ``` The evidence comes from observed CLI reproductions. The local reproduction is complete, and the separate synthetic-CDP provider check also returned the modification claim with accessibility marked truncated. ### Typed outcome or recovery expected Rerun the pair above. Accessibility should return `unknown` with no modification inferred from the omitted node and a reason explaining incomplete tree coverage. Keep the complete self-comparison control `unchanged`. With two complete captures that actually differ in a button role or name, a real modification must still be reported. The complete-tree change is a restoration check, not an additional observed result. ### Analysis or suspected cause Source-derived reasoning: `src/domain/webCaptureDiff.ts:102` fingerprints the accessibility tree as one aggregate, and `src/domain/webCaptureDiff.ts:77` retains a modified fingerprint despite incomplete coverage. A node omitted from an incomplete tree changes that aggregate without proving a semantic modification. This reasoning agrees with the observed CLI result. The provider's missing-child guard is `src/browser/CdpCaptureDocuments.ts:496`, with truncation recorded at `src/browser/CdpPageCapture.ts:424`. The separate executed synthetic-CDP check confirms that this producer path can supply the incomplete tree and marker consumed by the failing comparison. ### Additional context Related closed feature [#377](https://github.com/morluto/rea/issues/377) introduced accessibility and storage comparison; this report identifies a specific false modification from an explicitly truncated accessibility tree. Related open issue [#3172](https://github.com/morluto/rea/issues/3172) concerns unavailable redirect response fields causing false network conclusions; this report concerns accessibility node coverage and requires a separate comparison correction. Related closed issue [#1830](https://github.com/morluto/rea/issues/1830) corrected false changes from unavailable top-level response metadata; this report concerns incomplete accessibility tree aggregates.

bsakaguchi closed this issue

2026-10-16 21:03:23 +00:00

bsakaguchi commented

2026-10-16 21:03:25 +00:00

Author

Owner

@morluto commented on GitHub (Oct 16, 2026):

Fixed on main in 0123d5976. Accessibility comparison now reports unknown before fingerprinting incomplete trees instead of inventing a structural modification.

Public CLI controls preserve independent known changes and distinguish incomplete accessibility coverage from text permissions.

Hosted main CI is still pending; this closure is based on the implemented fix and the scoped regression evidence.

@morluto commented on GitHub (Oct 16, 2026): Fixed on `main` in [0123d5976](https://github.com/morluto/rea/commit/0123d5976828bfe621bd51daa4cf39fee48a1349). Accessibility comparison now reports unknown before fingerprinting incomplete trees instead of inventing a structural modification. Public CLI controls preserve independent known changes and distinguish incomplete accessibility coverage from text permissions. Hosted main CI is still pending; this closure is based on the implemented fix and the scoped regression evidence.

bsakaguchi referenced this issue

2026-10-17 21:11:52 +00:00

[PR #1483] [MERGED] feat(process): report bounded live capture progress #2921

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1252

No description provided.

Rows
Columns