morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6786] [Bug] Missing accessibility roles become definite changes or equality in capture comparison #4138

New issue

Closed

opened 2026-10-31 00:47:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:06 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6786

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The offline compare-web-captures CLI treats an omitted accessibility role as a comparable value. Comparing otherwise identical records with role: "button" and an omitted producer role returns accessibility.status: "changed" and one accessibility_tree modification, in both directions. Comparing two omitted-role records returns accessibility.status: "unchanged". The production captureAccessibility parser converts the omitted role to null and leaves the capture completeness ledger complete.

These results were observed on generated retained fixtures through the production parser and public CLI. No browser was launched or attached. Each of the five comparisons below ran once before this draft was written.

Reproduction

Use the prepared build of the source revision below. Set REA_ROOT to that checkout. Save the following as /tmp/rea-accessibility-repro.mjs; all URLs are inert recorded identities.

import {writeFileSync,readFileSync,mkdirSync} from 'node:fs';
import {spawnSync,execFileSync} from 'node:child_process';
import {resolve} from 'node:path';
import {pathToFileURL} from 'node:url';
const repo=resolve(process.argv[2]), work=resolve(process.argv[3]);
mkdirSync(work,{recursive:true});
const load=p=>import(pathToFileURL(resolve(repo,'dist',p)));
const {captureAccessibility}=await load('browser/CdpCaptureDocuments.js');
const {CdpCaptureCompleteness}=await load('browser/CdpCaptureCompleteness.js');
const {captureSnapshotSchema}=await load('domain/webCaptureDiffSchemas.js');
const origin='https://app.example.test';
const at=new Date(execFileSync('date',['--iso-8601=seconds'],{encoding:'utf8',env:{...process.env,TZ:'America/Los_Angeles'}}).trim()).toISOString();
const base={browser:{product:'normalized-fixture',protocol_version:'1.3',revision:'fixture',user_agent:'fixture',js_version:'fixture'},target:{target_id:'page',type:'page',title:'Fixture',url:origin,origin,attached:false},capture_window:{started_at:at,ended_at:at,observation_ms:0},frames:[],dom:{total_nodes:0,nodes:[]},scripts:{total:0,items:[]},resources:[],workers:[],network:{requests:[],websocket_connections:[],coverage_started_at:at,prior_activity_available:false},console:{events:[],coverage_started_at:at,prior_activity_available:false},metadata:{responses:[],dom_urls:[],agent_hints:[],excluded_dom_urls:0,headers_allowlisted:true},storage:{origin,usage_bytes:null,quota_bytes:null,local_storage_keys:[],session_storage_keys:[],indexed_db_names:[],cache_names:[],values_redacted:true},limitations:[]};
for(const [name,role] of [['unknown',undefined],['button','button'],['link','link']]){
 const completeness=new CdpCaptureCompleteness();
 const raw={nodeId:'ax-one',ignored:false,name:{type:'string',value:'Status'},description:{type:'string',value:''},properties:[],...(role===undefined?{}:{role:{type:'role',value:role}})};
 const captured=captureAccessibility([{nodes:[raw]}],{includeText:true,completeness});
 const capture=captureSnapshotSchema.parse({inspection:{...base,completeness:completeness.snapshot(),accessibility:{total_nodes:captured.total,nodes:captured.nodes,text_capture:captured.textCapture}},webmcp:null});
 writeFileSync(resolve(work,name+'.json'),JSON.stringify(capture));
 console.log('producer-'+name,JSON.stringify({accessibility:capture.inspection.accessibility,completeness:capture.inspection.completeness,hasUnobservedChildren:captured.hasUnobservedChildren}));
}
for(const [left,right] of [['button','button'],['button','unknown'],['unknown','button'],['unknown','unknown'],['button','link']]){
 const run=spawnSync(process.execPath,[resolve(repo,'scripts/rea.mjs'),'compare-web-captures',resolve(work,left+'.json'),resolve(work,right+'.json'),'--json'],{encoding:'utf8',timeout:30000});
 writeFileSync(resolve(work,left+'-'+right+'.output.json'),run.stdout);
 if(run.status!==0)throw new Error(JSON.stringify({status:run.status,stdout:run.stdout,stderr:run.stderr}));
 const result=JSON.parse(run.stdout).normalized_result;
 console.log(left+' vs '+right,JSON.stringify({exit:run.status,overall:result.overall_status,accessibility:result.dimensions.accessibility}));
}

Run:

node /tmp/rea-accessibility-repro.mjs "$REA_ROOT" /tmp/rea-accessibility-evidence

The generator calls actual captureAccessibility, CdpCaptureCompleteness, and captureSnapshotSchema exports. Its raw AX nodes have a stable ID, ignored: false, a known name and description, no children, and no properties. Only the optional producer role differs. Both missing-role and known-role inputs yield one retained node, complete text capture, no unobserved children, no exclusions, and an identical complete completeness ledger. The built protocol declaration in node_modules/playwright-core/types/protocol.d.ts declares AXNode.role?: AXValue; omission is a supported protocol shape, not an invented role string.

The CLI invocation executed by each pair is:

node "$REA_ROOT/scripts/rea.mjs" compare-web-captures /tmp/rea-accessibility-evidence/button.json /tmp/rea-accessibility-evidence/unknown.json --json

Expected behavior

An unavailable role cannot establish either a role modification or role equality. Preserve the known accessibility facts and keep the unknown role explicit; these omission-only pairs should remain unknown unless an independent retained fact proves a difference. The complete button self-comparison should remain unchanged, and the known button versus link change should remain observable.

The comparison contract says missing evidence is never treated as equivalence. The repository boundary contract requires explicit unknowns to survive normalization. The producer projection obtains role through axText(node.role), which maps the absent value to null; it does not observe an alternative role or the absence of a target role. A role that was not supplied is not evidence that the target changed roles.

Impact and scope

P2: retained accessibility evidence produces unsupported definite changes and equality. In the known-versus-missing pair, this also changes overall_status to changed. In the missing-versus-missing pair, only the accessibility dimension asserts equality; unrelated incomplete dimensions keep the overall result unknown. The stored captures retain the correct null role, and no input mutation or data loss was observed.

A practical workaround for these fixtures is to inspect accessibility.nodes[*].role and treat a null role as unavailable before interpreting the comparison result. This manual distinction was directly observed in the generated inputs. No automatic CLI workaround or frequency in live browser captures was tested.

REA version

Package 6.7.0; source revision 98ca6c82705bf8c44f814620636852f938d827ca. The shared compiled runtime from this checkout was used. The published package was not independently executed for this finding.

Runtime and platform

Node.js v26.12.1; Linux x64. No browser, native analysis provider, or external service was started.

Target and artifact context

Three schema-validated normalized passive-capture files containing one accessibility node each. Accessibility records and completeness ledgers use real production parser functions; other empty inventories are synthetic fixture data. No live provider capture integration was executed.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The producer returned this node for an omitted role:

{"node_id":"ax-one","parent_id":null,"role":null,"name":"Status","description":"","ignored":false,"states":[]}

Its accessibility envelope was total_nodes: 1, text_capture: {"status":"included","retained_bytes":6,"excluded_fields":0}. hasUnobservedChildren was false. The completeness ledger was complete_within_window, with empty section/exclusion arrays and zero dropped events, identical to both known-role fixtures.

Exact relevant CLI result projections:

button vs button {"exit":0,"overall":"unknown","accessibility":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
button vs unknown {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}}
unknown vs button {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}}
unknown vs unknown {"exit":0,"overall":"unknown","accessibility":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}}
button vs link {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}}

Evidence provenance: observed production-parser results and public CLI output. Local verification: reproduced. Reproduction completeness: complete for the retained-input workflow, with live browser integration not executed. No probe or comparison was rerun after drafting.

Analysis or suspected cause

Source-derived: accessibilityComparable checks text coverage, section lists, and node counts, but not missing role facts. accessibilityProjection hashes node.role directly, including null. accessibilityDimension therefore reports a digest difference as a modification, and equal nulls as unchanged. The public CLI route accepts these normalized captures and calls this comparator through the browser application workflow.

This report concerns the availability of a role actually used in the digest, rather than demanding unrelated facts from every AX node. Missing-role coverage should preserve independently proven differences in name, structure, ignored state, or known roles.

Additional context

Related closed reports #3376 and #3385 concern malformed or missing node inventories and explicitly truncated trees. This report retains one valid node and a complete tree; only its protocol-optional role is unavailable. Those inventory and truncation guards do not distinguish an unknown role from a known value.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed retained-input CLI returns definite accessibility modification or equality from missing role evidence; the known-versus-missing result also changes the aggregate classification. The effect is bounded to comparison output, with inputs intact and live-provider frequency unknown.
Workaround status: available
Workaround basis: The executed parser and saved captures retain role: null, so manual inspection can identify that the role is unavailable and avoid interpreting the unsupported verdict; no recapture, target execution, or input mutation is required for that distinction.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6786 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The offline `compare-web-captures` CLI treats an omitted accessibility role as a comparable value. Comparing otherwise identical records with `role: "button"` and an omitted producer role returns `accessibility.status: "changed"` and one `accessibility_tree` modification, in both directions. Comparing two omitted-role records returns `accessibility.status: "unchanged"`. The production `captureAccessibility` parser converts the omitted role to null and leaves the capture completeness ledger complete. These results were observed on generated retained fixtures through the production parser and public CLI. No browser was launched or attached. Each of the five comparisons below ran once before this draft was written. ## Reproduction Use the prepared build of the source revision below. Set `REA_ROOT` to that checkout. Save the following as `/tmp/rea-accessibility-repro.mjs`; all URLs are inert recorded identities. ```javascript import {writeFileSync,readFileSync,mkdirSync} from 'node:fs'; import {spawnSync,execFileSync} from 'node:child_process'; import {resolve} from 'node:path'; import {pathToFileURL} from 'node:url'; const repo=resolve(process.argv[2]), work=resolve(process.argv[3]); mkdirSync(work,{recursive:true}); const load=p=>import(pathToFileURL(resolve(repo,'dist',p))); const {captureAccessibility}=await load('browser/CdpCaptureDocuments.js'); const {CdpCaptureCompleteness}=await load('browser/CdpCaptureCompleteness.js'); const {captureSnapshotSchema}=await load('domain/webCaptureDiffSchemas.js'); const origin='https://app.example.test'; const at=new Date(execFileSync('date',['--iso-8601=seconds'],{encoding:'utf8',env:{...process.env,TZ:'America/Los_Angeles'}}).trim()).toISOString(); const base={browser:{product:'normalized-fixture',protocol_version:'1.3',revision:'fixture',user_agent:'fixture',js_version:'fixture'},target:{target_id:'page',type:'page',title:'Fixture',url:origin,origin,attached:false},capture_window:{started_at:at,ended_at:at,observation_ms:0},frames:[],dom:{total_nodes:0,nodes:[]},scripts:{total:0,items:[]},resources:[],workers:[],network:{requests:[],websocket_connections:[],coverage_started_at:at,prior_activity_available:false},console:{events:[],coverage_started_at:at,prior_activity_available:false},metadata:{responses:[],dom_urls:[],agent_hints:[],excluded_dom_urls:0,headers_allowlisted:true},storage:{origin,usage_bytes:null,quota_bytes:null,local_storage_keys:[],session_storage_keys:[],indexed_db_names:[],cache_names:[],values_redacted:true},limitations:[]}; for(const [name,role] of [['unknown',undefined],['button','button'],['link','link']]){ const completeness=new CdpCaptureCompleteness(); const raw={nodeId:'ax-one',ignored:false,name:{type:'string',value:'Status'},description:{type:'string',value:''},properties:[],...(role===undefined?{}:{role:{type:'role',value:role}})}; const captured=captureAccessibility([{nodes:[raw]}],{includeText:true,completeness}); const capture=captureSnapshotSchema.parse({inspection:{...base,completeness:completeness.snapshot(),accessibility:{total_nodes:captured.total,nodes:captured.nodes,text_capture:captured.textCapture}},webmcp:null}); writeFileSync(resolve(work,name+'.json'),JSON.stringify(capture)); console.log('producer-'+name,JSON.stringify({accessibility:capture.inspection.accessibility,completeness:capture.inspection.completeness,hasUnobservedChildren:captured.hasUnobservedChildren})); } for(const [left,right] of [['button','button'],['button','unknown'],['unknown','button'],['unknown','unknown'],['button','link']]){ const run=spawnSync(process.execPath,[resolve(repo,'scripts/rea.mjs'),'compare-web-captures',resolve(work,left+'.json'),resolve(work,right+'.json'),'--json'],{encoding:'utf8',timeout:30000}); writeFileSync(resolve(work,left+'-'+right+'.output.json'),run.stdout); if(run.status!==0)throw new Error(JSON.stringify({status:run.status,stdout:run.stdout,stderr:run.stderr})); const result=JSON.parse(run.stdout).normalized_result; console.log(left+' vs '+right,JSON.stringify({exit:run.status,overall:result.overall_status,accessibility:result.dimensions.accessibility})); } ``` Run: ```sh node /tmp/rea-accessibility-repro.mjs "$REA_ROOT" /tmp/rea-accessibility-evidence ``` The generator calls actual `captureAccessibility`, `CdpCaptureCompleteness`, and `captureSnapshotSchema` exports. Its raw AX nodes have a stable ID, `ignored: false`, a known name and description, no children, and no properties. Only the optional producer role differs. Both missing-role and known-role inputs yield one retained node, complete text capture, no unobserved children, no exclusions, and an identical complete completeness ledger. The built protocol declaration in `node_modules/playwright-core/types/protocol.d.ts` declares `AXNode.role?: AXValue`; omission is a supported protocol shape, not an invented role string. The CLI invocation executed by each pair is: ```sh node "$REA_ROOT/scripts/rea.mjs" compare-web-captures /tmp/rea-accessibility-evidence/button.json /tmp/rea-accessibility-evidence/unknown.json --json ``` ## Expected behavior An unavailable role cannot establish either a role modification or role equality. Preserve the known accessibility facts and keep the unknown role explicit; these omission-only pairs should remain unknown unless an independent retained fact proves a difference. The complete `button` self-comparison should remain unchanged, and the known `button` versus `link` change should remain observable. The [comparison contract](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/docs/browser-observation.md#L23) says missing evidence is never treated as equivalence. The [repository boundary contract](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/AGENTS.md#L33) requires explicit unknowns to survive normalization. The [producer projection](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/browser/CdpCaptureDocuments.ts#L566-L584) obtains role through `axText(node.role)`, which maps the absent value to null; it does not observe an alternative role or the absence of a target role. A role that was not supplied is not evidence that the target changed roles. ## Impact and scope P2: retained accessibility evidence produces unsupported definite changes and equality. In the known-versus-missing pair, this also changes `overall_status` to changed. In the missing-versus-missing pair, only the accessibility dimension asserts equality; unrelated incomplete dimensions keep the overall result unknown. The stored captures retain the correct null role, and no input mutation or data loss was observed. A practical workaround for these fixtures is to inspect `accessibility.nodes[*].role` and treat a null role as unavailable before interpreting the comparison result. This manual distinction was directly observed in the generated inputs. No automatic CLI workaround or frequency in live browser captures was tested. ## REA version Package 6.7.0; source revision `98ca6c82705bf8c44f814620636852f938d827ca`. The shared compiled runtime from this checkout was used. The published package was not independently executed for this finding. ## Runtime and platform Node.js v26.12.1; Linux x64. No browser, native analysis provider, or external service was started. ## Target and artifact context Three schema-validated normalized passive-capture files containing one accessibility node each. Accessibility records and completeness ledgers use real production parser functions; other empty inventories are synthetic fixture data. No live provider capture integration was executed. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence The producer returned this node for an omitted role: ```json {"node_id":"ax-one","parent_id":null,"role":null,"name":"Status","description":"","ignored":false,"states":[]} ``` Its accessibility envelope was `total_nodes: 1`, `text_capture: {"status":"included","retained_bytes":6,"excluded_fields":0}`. `hasUnobservedChildren` was false. The completeness ledger was `complete_within_window`, with empty section/exclusion arrays and zero dropped events, identical to both known-role fixtures. Exact relevant CLI result projections: ```text button vs button {"exit":0,"overall":"unknown","accessibility":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} button vs unknown {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}} unknown vs button {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}} unknown vs unknown {"exit":0,"overall":"unknown","accessibility":{"total_changes":0,"changes":[],"status":"unchanged","reason":null}} button vs link {"exit":0,"overall":"changed","accessibility":{"status":"changed","total_changes":1,"changes":[{"identity":"accessibility_tree","change":"modified"}],"reason":null}} ``` Evidence provenance: observed production-parser results and public CLI output. Local verification: reproduced. Reproduction completeness: complete for the retained-input workflow, with live browser integration not executed. No probe or comparison was rerun after drafting. ## Analysis or suspected cause Source-derived: [accessibilityComparable](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/domain/webCaptureDiff.ts#L866-L894) checks text coverage, section lists, and node counts, but not missing role facts. [accessibilityProjection](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/domain/webCaptureDiff.ts#L897-L918) hashes `node.role` directly, including null. [accessibilityDimension](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/domain/webCaptureDiff.ts#L110-L141) therefore reports a digest difference as a modification, and equal nulls as unchanged. The [public CLI route](https://github.com/morluto/rea/blob/98ca6c82705bf8c44f814620636852f938d827ca/src/cli/browserAdvancedCommands.ts#L105-L163) accepts these normalized captures and calls this comparator through the browser application workflow. This report concerns the availability of a role actually used in the digest, rather than demanding unrelated facts from every AX node. Missing-role coverage should preserve independently proven differences in name, structure, ignored state, or known roles. ## Additional context Related closed reports [#3376](https://github.com/morluto/rea/issues/3376) and [#3385](https://github.com/morluto/rea/issues/3385) concern malformed or missing node inventories and explicitly truncated trees. This report retains one valid node and a complete tree; only its protocol-optional role is unavailable. Those inventory and truncation guards do not distinguish an unknown role from a known value. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed retained-input CLI returns definite accessibility modification or equality from missing role evidence; the known-versus-missing result also changes the aggregate classification. The effect is bounded to comparison output, with inputs intact and live-provider frequency unknown. - Workaround status: available - Workaround basis: The executed parser and saved captures retain `role: null`, so manual inspection can identify that the role is unavailable and avoid interpreting the unsupported verdict; no recapture, target execution, or input mutation is required for that distinction.

bsakaguchi closed this issue

2026-10-31 00:47:06 +00:00

bsakaguchi referenced this issue

2026-11-01 21:43:07 +00:00

[PR #5173] [CLOSED] Fix/psx zero gp unknown 4138 #6800