[GH-ISSUE #497] [Feature] Add approved UI observation for native desktop applications #268

New issue

Closed

opened 2026-10-15 18:23:14 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:23:14 +00:00

Owner

Originally created by @morluto on GitHub (Oct 2, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/497

Area

Runtime observation and reconstruction

Current limitation

REA can observe browsers and run controlled browser scenarios, while native application work is largely static: bundle inventory, decoded Interface Builder resources, symbols, and code traces. There is no REA workflow for inspecting the rendered UI of a user-selected native desktop app, scrolling through its pages, or exercising an explicitly chosen control and retaining the resulting visual/accessibility evidence.

Use cases

Confirm which UI route a visible map marker, tab, or card opens.
Inspect long native screens by scrolling and capture result states after a user-approved action.
Compare runtime appearance with compiled storyboard/NIB and code evidence without treating static layout as final rendering.

Desired outcome

Provide an opt-in, bounded observation capability for one explicitly selected native desktop app/window. It should support passive screenshot and accessibility-tree capture, plus a separately authorized declarative scenario with a small set of actions such as click, scroll, key entry, and bounded wait. Return ordered before/after evidence per step, target-window identity, action outcome, and explicit capture gaps.

Authority and safety boundaries

Bind the session to one exact app/process/window; do not capture or interact with unrelated windows.
Passive capture is the default. Mutating input actions require an explicit scenario and declared side effects; do not accept arbitrary scripts or shell commands.
Respect OS accessibility/screen-recording permissions. Permission denial must fail with actionable guidance without silently switching to broad screen control.
Bound action count, wait time, screenshot size, and retained output. Cancellation must stop pending actions and clean up only processes REA started.
Declare whether app data, network state, or persistent UI changes may be affected. Require explicit user-selected cleanup/restore behavior rather than guessing.

Success criteria

The CLI and MCP expose equivalent target, permission, action, cancellation, and evidence contracts.
Screenshots and accessibility records identify the exact target and action step; hidden/private UI content remains unretained unless explicitly requested.
Fixtures cover permission denial, ambiguous windows, scrolling, delayed UI, failed actions, cancellation, and owned-process cleanup.
Browser observation remains on its existing provider path; mobile emulator/device workflows remain separately scoped under #41.

#41 mobile application investigation providers and workflows
#480 compiled Apple storyboard and nib UI graphs
#483 native UI control dispatch traces
Browser observation and controlled browser scenarios

Originally created by @morluto on GitHub (Oct 2, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/497 ## Area Runtime observation and reconstruction ## Current limitation REA can observe browsers and run controlled browser scenarios, while native application work is largely static: bundle inventory, decoded Interface Builder resources, symbols, and code traces. There is no REA workflow for inspecting the rendered UI of a user-selected native desktop app, scrolling through its pages, or exercising an explicitly chosen control and retaining the resulting visual/accessibility evidence. ## Use cases - Confirm which UI route a visible map marker, tab, or card opens. - Inspect long native screens by scrolling and capture result states after a user-approved action. - Compare runtime appearance with compiled storyboard/NIB and code evidence without treating static layout as final rendering. ## Desired outcome Provide an opt-in, bounded observation capability for one explicitly selected native desktop app/window. It should support passive screenshot and accessibility-tree capture, plus a separately authorized declarative scenario with a small set of actions such as click, scroll, key entry, and bounded wait. Return ordered before/after evidence per step, target-window identity, action outcome, and explicit capture gaps. ## Authority and safety boundaries - Bind the session to one exact app/process/window; do not capture or interact with unrelated windows. - Passive capture is the default. Mutating input actions require an explicit scenario and declared side effects; do not accept arbitrary scripts or shell commands. - Respect OS accessibility/screen-recording permissions. Permission denial must fail with actionable guidance without silently switching to broad screen control. - Bound action count, wait time, screenshot size, and retained output. Cancellation must stop pending actions and clean up only processes REA started. - Declare whether app data, network state, or persistent UI changes may be affected. Require explicit user-selected cleanup/restore behavior rather than guessing. ## Success criteria - The CLI and MCP expose equivalent target, permission, action, cancellation, and evidence contracts. - Screenshots and accessibility records identify the exact target and action step; hidden/private UI content remains unretained unless explicitly requested. - Fixtures cover permission denial, ambiguous windows, scrolling, delayed UI, failed actions, cancellation, and owned-process cleanup. - Browser observation remains on its existing provider path; mobile emulator/device workflows remain separately scoped under #41. ## Related work - #41 mobile application investigation providers and workflows - #480 compiled Apple storyboard and nib UI graphs - #483 native UI control dispatch traces - Browser observation and controlled browser scenarios

bsakaguchi

2026-10-15 18:23:14 +00:00

closed this issue
added the
triage

enhancement
labels

bsakaguchi commented

2026-10-15 18:23:15 +00:00

Author

Owner

@morluto commented on GitHub (Oct 8, 2026):

Verified fixed on main (7a7460f0). observe_native_ui + capture_native_ui_scenario ship via nativeToolContracts + CLI with NativeUiObservation module and passing tests (16/16). Closing per audit.

@morluto commented on GitHub (Oct 8, 2026): Verified fixed on main (7a7460f0). observe_native_ui + capture_native_ui_scenario ship via nativeToolContracts + CLI with NativeUiObservation module and passing tests (16/16). Closing per audit.

bsakaguchi referenced this issue

2026-10-16 21:06:20 +00:00

[GH-ISSUE #3889] [Bug] Reconstruction coverage reports ready when a passing proof conflicts with a failed one #1509

bsakaguchi referenced this issue

2026-10-16 21:07:56 +00:00

[GH-ISSUE #4069] [RFC] Evidence-backed UI state/transition observation primitives for agent-driven behavioral reconstruction #1610

bsakaguchi referenced this issue

2026-10-16 21:12:34 +00:00

[PR #285] [MERGED] feat: add managed characterization and reconstruction coverage #1927

bsakaguchi added

enhancement

and removed