[GH-ISSUE #6541] [Bug] Native UI capture silently replaces numeric accessibility values with null #3931

New issue

Closed

opened 2026-10-31 00:45:20 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:20 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6541

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

Source-only finding; no native UI runtime reproduction was performed. observe_native_ui and the snapshots in capture_native_ui_scenario discard successfully read non-string accessibility values. A standard slider whose AXValue is a number is emitted as "value": null, with no value-related gap or truncation flag. Distinct numeric states therefore lose their accessibility value distinction.

Reproduction

Actually performed: inspected the published release's production helper, provider dispatch, input validation, response schema, and report construction; established the same reachable helper and nullable-string output on main. Searched the downloaded open-issue inventory and queried open and closed GitHub issues with native/value/accessibility, accessibility/numeric, and native/checkbox terms. No REA command, Swift helper, native application, or macOS accessibility API was executed.

Proposed supported-host reproduction, unexecuted:

On a supported macOS host, open a caller-owned Mach-O application containing a standard slider that exposes a numeric AXValue. Grant the host Accessibility permission and open the application's executable as REA's active target. Select its actual PID and uniquely matched window ID.
Call observe_native_ui with {"pid":123,"window_id":456,"screenshot":false,"accessibility":true,"max_nodes":500}, replacing only PID and window ID with the selected application's actual values. Keep the complete window tree within the node and depth budgets.
Inspect the slider node in initial.nodes. For any successful numeric AXValue read, the inspected helper deterministically emits value: null. A second numeric slider state follows the same conversion.

NOT_ENOUGH_INFO: live macOS output and occurrence frequency were not measured. The sample PID/window numbers are placeholders for this unexecuted recipe.

Expected behavior

Preserve a successfully read numeric accessibility value, or explicitly record that its representation is unsupported. Do not silently represent an available numeric value as an absent value. A lossless numeric string would fit the current output type; choosing a richer typed representation is a separate implementation decision.

Apple documents numeric slider values in kAXValueDescriptionAttribute and defines kAXValueAttribute as the element's user-visible setting. REA's repository instructions require preserving source values and explicit unknowns when normalization loses information.

Impact and scope

The supported source-derived scope is numeric AXValue observations on otherwise admitted native macOS windows, including passive snapshots and scenario before/after snapshots. Missing numeric state is consequential for inspecting slider positions and changes while the result supplies no reason for the loss. String values follow a separate successful cast and are unaffected by this specific cause.

NOT_ENOUGH_INFO: no supported-host reproduction, distribution of affected controls, or practical workaround preserving exact numeric state was demonstrated. Screenshots may show some state visually, but that is unverified and is disabled in the concrete recipe.

REA version

Latest published rea-agents 6.7.0 source: 27dc42d10249f0410904a75c0997842eb03b931c. Main presence established at 1635eadfcc49bbb80178838ab74667c2981de383.

Runtime and platform

Source inspection on Linux x86_64 with Node.js v26.12.1. The affected runtime path targets supported macOS hosts. NOT_ENOUGH_INFO: macOS, Swift, and native helper runtime versions were not measured because no native reproduction was executed.

Target and artifact context

A caller-owned running Mach-O application with a uniquely selected window and an ordinary numeric accessibility slider value; passive accessibility capture with screenshots disabled. No private target or artifact is required.

Interface

MCP

Earliest failure stage

Capture or observation

Relevant output or evidence

No runtime output is claimed. The following concrete data path establishes the source-derived loss:

The helper preserves the returned CF value when AXUIElementCopyAttributeValue succeeds, then text casts it to String and substitutes NSNull when the cast fails: release helper, lines 21–37.
Every captured node sends kAXValueAttribute through that string-only function. The adjacent gap handling covers child enumeration, not unsupported value conversion: release node capture, lines 116–135.
The same helper on main is byte-identical to the release helper; its Git blob is 2e4cc834144203937de26b0ee330b83c8ff8e4cc.
Both snapshots accept value as a nullable string, so the substituted null passes validation rather than producing an error: release output schema, main output schema.
The production observation validates a Mach-O executable target and selected PID/window identity, then retains the parsed snapshots without recovering their values: release observation workflow, main provider dispatch.

The concrete input is a successful AXValue read returning a numeric object such as 0.25. It passes the helper's process, executable digest, launch identity, window ownership/geometry, and Accessibility guards. Screenshot permission is irrelevant when screenshot is false. The numeric object cannot satisfy as? String, so its emitted value is null independently of timing or provider failure.

Typed outcome or recovery expected

Retain the observed numeric setting or expose a value-conversion gap distinguishing it from an absent or failed AXValue read.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A reachable string-only conversion deterministically loses numeric accessibility state from admitted native UI observations without reporting that loss; impact is bounded to non-string values.
Workaround status: unverified
Workaround basis: No supported-host check demonstrated a practical way to retain the exact numeric value through another REA path; screenshots may provide visual context but were not tested and do not establish preservation of the numeric fact.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6541 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior Source-only finding; no native UI runtime reproduction was performed. `observe_native_ui` and the snapshots in `capture_native_ui_scenario` discard successfully read non-string accessibility values. A standard slider whose `AXValue` is a number is emitted as `"value": null`, with no value-related gap or truncation flag. Distinct numeric states therefore lose their accessibility value distinction. ### Reproduction Actually performed: inspected the published release's production helper, provider dispatch, input validation, response schema, and report construction; established the same reachable helper and nullable-string output on main. Searched the downloaded open-issue inventory and queried open and closed GitHub issues with native/value/accessibility, accessibility/numeric, and native/checkbox terms. No REA command, Swift helper, native application, or macOS accessibility API was executed. Proposed supported-host reproduction, unexecuted: 1. On a supported macOS host, open a caller-owned Mach-O application containing a standard slider that exposes a numeric `AXValue`. Grant the host Accessibility permission and open the application's executable as REA's active target. Select its actual PID and uniquely matched window ID. 2. Call `observe_native_ui` with `{"pid":123,"window_id":456,"screenshot":false,"accessibility":true,"max_nodes":500}`, replacing only PID and window ID with the selected application's actual values. Keep the complete window tree within the node and depth budgets. 3. Inspect the slider node in `initial.nodes`. For any successful numeric AXValue read, the inspected helper deterministically emits `value: null`. A second numeric slider state follows the same conversion. NOT_ENOUGH_INFO: live macOS output and occurrence frequency were not measured. The sample PID/window numbers are placeholders for this unexecuted recipe. ### Expected behavior Preserve a successfully read numeric accessibility value, or explicitly record that its representation is unsupported. Do not silently represent an available numeric value as an absent value. A lossless numeric string would fit the current output type; choosing a richer typed representation is a separate implementation decision. Apple documents numeric slider values in [kAXValueDescriptionAttribute](https://developer.apple.com/documentation/applicationservices/kaxvaluedescriptionattribute) and defines [kAXValueAttribute](https://developer.apple.com/documentation/applicationservices/kaxvalueattribute) as the element's user-visible setting. REA's repository instructions require preserving source values and explicit unknowns when normalization loses information. ### Impact and scope The supported source-derived scope is numeric AXValue observations on otherwise admitted native macOS windows, including passive snapshots and scenario before/after snapshots. Missing numeric state is consequential for inspecting slider positions and changes while the result supplies no reason for the loss. String values follow a separate successful cast and are unaffected by this specific cause. NOT_ENOUGH_INFO: no supported-host reproduction, distribution of affected controls, or practical workaround preserving exact numeric state was demonstrated. Screenshots may show some state visually, but that is unverified and is disabled in the concrete recipe. ### REA version Latest published rea-agents 6.7.0 source: `27dc42d10249f0410904a75c0997842eb03b931c`. Main presence established at `1635eadfcc49bbb80178838ab74667c2981de383`. ### Runtime and platform Source inspection on Linux x86_64 with Node.js v26.12.1. The affected runtime path targets supported macOS hosts. NOT_ENOUGH_INFO: macOS, Swift, and native helper runtime versions were not measured because no native reproduction was executed. ### Target and artifact context A caller-owned running Mach-O application with a uniquely selected window and an ordinary numeric accessibility slider value; passive accessibility capture with screenshots disabled. No private target or artifact is required. ### Interface MCP ### Earliest failure stage Capture or observation ### Relevant output or evidence No runtime output is claimed. The following concrete data path establishes the source-derived loss: - The helper preserves the returned CF value when `AXUIElementCopyAttributeValue` succeeds, then `text` casts it to `String` and substitutes `NSNull` when the cast fails: [release helper, lines 21–37](https://github.com/morluto/rea/blob/27dc42d10249f0410904a75c0997842eb03b931c/bridge/native/ReaNativeUI.swift#L21-L37). - Every captured node sends `kAXValueAttribute` through that string-only function. The adjacent gap handling covers child enumeration, not unsupported value conversion: [release node capture, lines 116–135](https://github.com/morluto/rea/blob/27dc42d10249f0410904a75c0997842eb03b931c/bridge/native/ReaNativeUI.swift#L116-L135). - [The same helper on main](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/bridge/native/ReaNativeUI.swift#L21-L37) is byte-identical to the release helper; its Git blob is `2e4cc834144203937de26b0ee330b83c8ff8e4cc`. - Both snapshots accept `value` as a nullable string, so the substituted null passes validation rather than producing an error: [release output schema](https://github.com/morluto/rea/blob/27dc42d10249f0410904a75c0997842eb03b931c/src/domain/native/nativeUiObservation.ts#L116-L138), [main output schema](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/domain/native/nativeUiObservation.ts#L119-L139). - The production observation validates a Mach-O executable target and selected PID/window identity, then retains the parsed snapshots without recovering their values: [release observation workflow](https://github.com/morluto/rea/blob/27dc42d10249f0410904a75c0997842eb03b931c/src/native/NativeUiObservation.ts#L119-L207), [main provider dispatch](https://github.com/morluto/rea/blob/1635eadfcc49bbb80178838ab74667c2981de383/src/native/NativeMacOSProvider.ts#L419-L471). The concrete input is a successful AXValue read returning a numeric object such as 0.25. It passes the helper's process, executable digest, launch identity, window ownership/geometry, and Accessibility guards. Screenshot permission is irrelevant when `screenshot` is false. The numeric object cannot satisfy `as? String`, so its emitted value is null independently of timing or provider failure. ### Typed outcome or recovery expected Retain the observed numeric setting or expose a value-conversion gap distinguishing it from an absent or failed AXValue read. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A reachable string-only conversion deterministically loses numeric accessibility state from admitted native UI observations without reporting that loss; impact is bounded to non-string values. - Workaround status: unverified - Workaround basis: No supported-host check demonstrated a practical way to retain the exact numeric value through another REA path; screenshots may provide visual context but were not tested and do not establish preservation of the numeric fact.

bsakaguchi closed this issue

2026-10-31 00:45:21 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:57 +00:00

[PR #4178] [MERGED] fix(process): preserve unrelated trace comparison uncertainty #6112