morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6450] [Bug] compare-web-captures incorrectly says structural accessibility comparison requires text capture #3864

New issue

Closed

opened 2026-10-31 00:44:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:44:44 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6450

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Documentation

Observed behavior

compare-web-captures reports a known accessibility role change when both captures explicitly omit accessibility text, but the same result says roles are compared only when text capture was selected. This makes the returned coverage limitation inaccurate. The role comparison itself agrees with the existing intended behavior.

Observed three times with synthetic normalized passive captures: a one-node tree changes from button to link, while both inputs retain text_capture.status: "not_approved" and an accessibility/not_approved producer-style exclusion. All three CLI calls exited 0 and returned one accessibility modification alongside the contradictory limitation.

Reproduction

From a fresh source checkout at 9eb829aca34be36ff112826337996f6139f82eff, prepare the locked dependencies and runtime:

npm ci
npm run build:cached

These are setup instructions; the observed runs used the prepared dependencies and build. From the checkout root, create two local capture files and compare them. The example URL is an inert recorded identity; no browser or URL is accessed.

export WORK="${WORK:-$(mktemp -d)}"
mkdir -p "$WORK/home"
export CAPTURE_AT="$(TZ=America/Los_Angeles date +%Y-%m-%dT%H:%M:%S%z)"
node --input-type=module <<'JS'
import {writeFileSync} from 'node:fs';
import {join} from 'node:path';
import {webPageInspectionSchema} from './dist/domain/browserObservationSchemas.js';
const origin = 'https://example.test';
const at = new Date(process.env.CAPTURE_AT).toISOString();
const inspection = webPageInspectionSchema.parse({
  browser: {product:'fixture',protocol_version:'1.3',revision:'fixture',user_agent:'fixture',js_version:'fixture'},
  target: {target_id:'page',type:'page',title:'Fixture',url:origin+'/',origin,attached:false},
  capture_window: {started_at:at,ended_at:at,observation_ms:0},
  completeness: {
    status:'policy_filtered',conditions:['policy_filtered'],
    policy_filtered_sections:['accessibility'],attach_limited_sections:[],truncated_sections:[],unavailable_sections:[],
    excluded:[{section:'accessibility',reason:'not_approved',count:1}],
    dropped_events:Object.fromEntries(['scripts','network_requests','console_events','websocket_connections','websocket_frames','webmcp_tools','timeline_events','total'].map(key=>[key,0]))
  },
  frames:[],dom:{total_nodes:0,nodes:[]},
  accessibility: {
    total_nodes:1,text_capture:{status:'not_approved',retained_bytes:0,excluded_fields:1},
    nodes:[{node_id:'n1',parent_id:null,role:'button',name:null,description:null,ignored:false,states:[]}]
  },
  scripts:{total:0,items:[]},resources:[],workers:[],
  network:{requests:[],websocket_connections:[],coverage_started_at:at,prior_activity_available:false},
  console:{events:[],coverage_started_at:at,prior_activity_available:false},
  metadata:{responses:[],dom_urls:[],agent_hints:[],excluded_dom_urls:0,headers_allowlisted:true},
  storage:{origin,usage_bytes:null,quota_bytes:null,local_storage_keys:[],session_storage_keys:[],indexed_db_names:[],cache_names:[],values_redacted:true},
  limitations:[]
});
const before = {inspection};
const after = structuredClone(before);
after.inspection.accessibility.nodes[0].role = 'link';
for (const [name,value] of Object.entries({before,after}))
  writeFileSync(join(process.env.WORK,name+'.json'),JSON.stringify(value));
JS
HOME="$WORK/home" node scripts/rea.mjs compare-web-captures "$WORK/before.json" "$WORK/after.json" --format json --full-output

Inspect data.normalized_result.dimensions.accessibility and data.normalized_result.limitations in the returned envelope.

Expected behavior

The limitation should distinguish structural accessibility coverage from selected text coverage: complete structural observations can establish role, ignored-state, state, or hierarchy changes without permission to retain names and descriptions. Text comparisons require jointly available text. Missing text should still prevent a claim of full accessibility equality when no known structural difference exists.

The existing public CLI regression explicitly expects known structure to be compared when only text was not approved. The returned limitation should describe that behavior accurately.

Impact and scope

P3: misleading wording in passive comparison results. The observed role change is supported by both retained trees; no incorrect comparison status, data loss, or consequential downstream failure was demonstrated. A caller reading the limitation alone could incorrectly conclude that the reported structural change was never compared.

The directly observed workaround is to inspect the retained role values, text-capture states, and dimension result together. Real-browser frequency and MCP transport behavior were not tested.

REA version

6.7.0 (source 9eb829aca3)

Runtime and platform

Node.js v26.12.1; macOS 27.0.1; arm64. Locked dependencies and the prepared compiled runtime were used.

Target and artifact context

Two schema-validated normalized passive page-capture JSON files with complete one-node accessibility inventories and text explicitly not approved. Only the retained role differs. Other empty inventories are synthetic fixture data, not observations from a live browser.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Observed through the CLI above; exit 0:

{
  "status": "changed",
  "total_changes": 1,
  "changes": [{"identity":"accessibility_tree","change":"modified"}],
  "reason": null
}

The same normalized result includes:

Accessibility roles, ignored state, text, and hierarchy are compared only when the accessibility tree was fully captured and text capture was selected and not truncated.

Both inputs have text_capture.status: "not_approved", zero retained text bytes, and one excluded text field. Controls also exited 0: identical inputs without text returned accessibility unknown; a role change with text included returned changed; marking the role-change tree truncated returned unknown.

Typed outcome or recovery expected

Keep the supported structural modification visible and return an accurate limitation explaining which fields require text capture. Preserve unknown results for incomplete trees and for unobserved text when no structural difference proves change.

Analysis or suspected cause

Source-derived: accessibilityDimension computes structural and text comparability separately. The comparability helper and projection admit complete structure with text-only policy filtering and conditionally omit only names/descriptions from the fingerprint. The unconditional limitation still combines those separate requirements.

The fixture's text exclusion matches the producer's source path. This is a source check of producer consistency, not a live-capture reproduction.

Additional context

Related closed #3385 and PR #3824 corrected false changes from incomplete trees and intentionally preserved non-text comparisons. This report concerns the inaccurate limitation that remains after that fix. Closed #3376 concerns malformed inventories; open #5897 concerns scenario example completeness. Neither describes this passive-result wording mismatch.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6450 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Documentation ### Observed behavior `compare-web-captures` reports a known accessibility role change when both captures explicitly omit accessibility text, but the same result says roles are compared only when text capture was selected. This makes the returned coverage limitation inaccurate. The role comparison itself agrees with the existing intended behavior. Observed three times with synthetic normalized passive captures: a one-node tree changes from `button` to `link`, while both inputs retain `text_capture.status: "not_approved"` and an accessibility/not_approved producer-style exclusion. All three CLI calls exited 0 and returned one accessibility modification alongside the contradictory limitation. ### Reproduction From a fresh source checkout at `9eb829aca34be36ff112826337996f6139f82eff`, prepare the locked dependencies and runtime: ```sh npm ci npm run build:cached ``` These are setup instructions; the observed runs used the prepared dependencies and build. From the checkout root, create two local capture files and compare them. The example URL is an inert recorded identity; no browser or URL is accessed. ```sh export WORK="${WORK:-$(mktemp -d)}" mkdir -p "$WORK/home" export CAPTURE_AT="$(TZ=America/Los_Angeles date +%Y-%m-%dT%H:%M:%S%z)" node --input-type=module <<'JS' import {writeFileSync} from 'node:fs'; import {join} from 'node:path'; import {webPageInspectionSchema} from './dist/domain/browserObservationSchemas.js'; const origin = 'https://example.test'; const at = new Date(process.env.CAPTURE_AT).toISOString(); const inspection = webPageInspectionSchema.parse({ browser: {product:'fixture',protocol_version:'1.3',revision:'fixture',user_agent:'fixture',js_version:'fixture'}, target: {target_id:'page',type:'page',title:'Fixture',url:origin+'/',origin,attached:false}, capture_window: {started_at:at,ended_at:at,observation_ms:0}, completeness: { status:'policy_filtered',conditions:['policy_filtered'], policy_filtered_sections:['accessibility'],attach_limited_sections:[],truncated_sections:[],unavailable_sections:[], excluded:[{section:'accessibility',reason:'not_approved',count:1}], dropped_events:Object.fromEntries(['scripts','network_requests','console_events','websocket_connections','websocket_frames','webmcp_tools','timeline_events','total'].map(key=>[key,0])) }, frames:[],dom:{total_nodes:0,nodes:[]}, accessibility: { total_nodes:1,text_capture:{status:'not_approved',retained_bytes:0,excluded_fields:1}, nodes:[{node_id:'n1',parent_id:null,role:'button',name:null,description:null,ignored:false,states:[]}] }, scripts:{total:0,items:[]},resources:[],workers:[], network:{requests:[],websocket_connections:[],coverage_started_at:at,prior_activity_available:false}, console:{events:[],coverage_started_at:at,prior_activity_available:false}, metadata:{responses:[],dom_urls:[],agent_hints:[],excluded_dom_urls:0,headers_allowlisted:true}, storage:{origin,usage_bytes:null,quota_bytes:null,local_storage_keys:[],session_storage_keys:[],indexed_db_names:[],cache_names:[],values_redacted:true}, limitations:[] }); const before = {inspection}; const after = structuredClone(before); after.inspection.accessibility.nodes[0].role = 'link'; for (const [name,value] of Object.entries({before,after})) writeFileSync(join(process.env.WORK,name+'.json'),JSON.stringify(value)); JS HOME="$WORK/home" node scripts/rea.mjs compare-web-captures "$WORK/before.json" "$WORK/after.json" --format json --full-output ``` Inspect `data.normalized_result.dimensions.accessibility` and `data.normalized_result.limitations` in the returned envelope. ### Expected behavior The limitation should distinguish structural accessibility coverage from selected text coverage: complete structural observations can establish role, ignored-state, state, or hierarchy changes without permission to retain names and descriptions. Text comparisons require jointly available text. Missing text should still prevent a claim of full accessibility equality when no known structural difference exists. The existing [public CLI regression](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/tests/boundary/cli/accessibilityCaptureComparison.test.ts#L146-L190) explicitly expects known structure to be compared when only text was not approved. The [returned limitation](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L46) should describe that behavior accurately. ### Impact and scope P3: misleading wording in passive comparison results. The observed role change is supported by both retained trees; no incorrect comparison status, data loss, or consequential downstream failure was demonstrated. A caller reading the limitation alone could incorrectly conclude that the reported structural change was never compared. The directly observed workaround is to inspect the retained role values, text-capture states, and dimension result together. Real-browser frequency and MCP transport behavior were not tested. ### REA version 6.7.0 (source 9eb829aca34be36ff112826337996f6139f82eff) ### Runtime and platform Node.js v26.12.1; macOS 27.0.1; arm64. Locked dependencies and the prepared compiled runtime were used. ### Target and artifact context Two schema-validated normalized passive page-capture JSON files with complete one-node accessibility inventories and text explicitly not approved. Only the retained role differs. Other empty inventories are synthetic fixture data, not observations from a live browser. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Observed through the CLI above; exit 0: ```json { "status": "changed", "total_changes": 1, "changes": [{"identity":"accessibility_tree","change":"modified"}], "reason": null } ``` The same normalized result includes: ```text Accessibility roles, ignored state, text, and hierarchy are compared only when the accessibility tree was fully captured and text capture was selected and not truncated. ``` Both inputs have `text_capture.status: "not_approved"`, zero retained text bytes, and one excluded text field. Controls also exited 0: identical inputs without text returned accessibility `unknown`; a role change with text included returned `changed`; marking the role-change tree truncated returned `unknown`. ### Typed outcome or recovery expected Keep the supported structural modification visible and return an accurate limitation explaining which fields require text capture. Preserve unknown results for incomplete trees and for unobserved text when no structural difference proves change. ### Analysis or suspected cause Source-derived: [accessibilityDimension](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L109-L140) computes structural and text comparability separately. The [comparability helper and projection](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/domain/webCaptureDiff.ts#L865-L919) admit complete structure with text-only policy filtering and conditionally omit only names/descriptions from the fingerprint. The unconditional limitation still combines those separate requirements. The fixture's text exclusion matches the [producer's source path](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/browser/CdpPageCapture.ts#L422-L434). This is a source check of producer consistency, not a live-capture reproduction. ### Additional context Related closed [#3385](https://github.com/morluto/rea/issues/3385) and [PR #3824](https://github.com/morluto/rea/pull/3824) corrected false changes from incomplete trees and intentionally preserved non-text comparisons. This report concerns the inaccurate limitation that remains after that fix. Closed [#3376](https://github.com/morluto/rea/issues/3376) concerns malformed inventories; open [#5897](https://github.com/morluto/rea/issues/5897) concerns scenario example completeness. Neither describes this passive-result wording mismatch.

bsakaguchi closed this issue

2026-10-31 00:44:44 +00:00

bsakaguchi referenced this issue

2026-10-31 00:47:25 +00:00

[GH-ISSUE #6826] [Bug] import-reference-source imports the wrong tree when .. follows a directory symlink #4173

bsakaguchi referenced this issue

2026-10-31 01:11:20 +00:00

[PR #4067] [MERGED] fix(paths): select caller paths through symlinks and .. as the filesystem does #6032

bsakaguchi referenced this issue

2026-11-01 21:45:44 +00:00

[PR #6089] fix(adb): preserve filesystem meaning of host paths #7146