morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4069] [RFC] Evidence-backed UI state/transition observation primitives for agent-driven behavioral reconstruction #1610

New issue

Open

opened 2026-10-16 21:07:56 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-16 21:07:56 +00:00

Owner

Originally created by @kangyishuai on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4069

Problem / motivation

REA already provides valuable evidence-backed static and runtime analysis. For an external agent attempting large-scale behavioral reconstruction of an application it owns or is authorized to examine, the missing piece is often not another decompiler: it is a composable, evidence-backed application interaction and observation contract that can enumerate UI states and transitions, connect observations to REA evidence, and support repeatable comparisons.

This is not a request for REA to implement a one-click commercial software clone, generate replacement applications, bypass protections, or own an autonomous orchestration loop. The external agent remains responsible for exploration strategy, implementation, and iterative repair.

Requested capability / desired outcome

Please consider defining an opt-in CLI/MCP contract (or documenting existing composable equivalents) for:

UI state and action inventory: observable windows/pages, accessible controls, actions, navigation, modal dialogs, and state identifiers, with platform/provider and visibility limitations explicitly recorded.
Evidence-linked transitions: given a caller-selected action and state, return before/after observations and references to relevant process, filesystem, IPC, network, and static-analysis evidence where available; unresolved associations must remain unknown rather than guessed.
Repeatable scenario primitives: caller-controlled initial conditions, action sequences, bounded execution, cancellation, cleanup, and replay/comparison metadata. Do not silently promise deterministic reset of external state.
Coverage export: machine-readable observed states/transitions, unvisited or unreachable candidates, source evidence, and gaps, so an external agent can prioritize follow-up experiments.
Incremental use: stable artifact/session references and import/export suitable for long-running investigations without requiring a monolithic in-memory analysis.

A minimal first milestone could be a read-only UI state/transition evidence schema plus an adapter for one supported target class, using existing browser/process observations, rather than a universal UI automation engine.

Example workflow (proposed, not tested)

For a self-owned sample Electron app:

An agent asks REA for an observed UI state inventory.
It chooses a search action and supplies a bounded interaction scenario.
REA returns before/after UI observations, available IPC/network/process evidence, and explicit unknowns.
The agent uses those observations to implement its own equivalent feature, then compares recorded outcomes using existing comparison primitives.
Subsequent runs reuse prior evidence and expose unexplored states without claiming exhaustive coverage.

Relationship to existing issues

I searched for related issues and do not want to duplicate them:

#3718: universal static-to-runtime semantic correlation (this request would consume/extend its evidence links).
#405 and #268: reconstruction obligation/coverage workspace (this request focuses on collecting and exporting UI interaction state/transition evidence rather than owning closure policy).
#37 and #48: comparison and process observation foundations.
#69: resettable macOS environment (possible optional provider, not a requirement for every target).
#3636: Windows XAML structural reconstruction (format-specific, complementary).

If maintainers believe existing primitives already suffice, a documented end-to-end composition example and explicit gap inventory would also address much of this request.

Scope / safety / verification

Authorized, self-owned or suitably licensed sample applications only.
No anti-debugging/DRM/license bypass, stealth instrumentation, credential extraction, or attack automation.
Any execution, input injection, filesystem/network mutation, or state reset must declare its authority, side effects, time/resource limits, and cleanup; request explicit maintainer scope review before adding a provider with new execution powers.
Validate against a small self-owned sample with known UI transitions and intentionally missing/ambiguous observations. Check that unknowns remain unknown, that cancellation/cleanup work, and that output is stable enough for external agents.
No runtime implementation or validation has been performed for this proposal.

Questions for maintainers

Is a provider-neutral UI interaction/observation evidence contract within REA's intended infrastructure scope?
Should this be tracked under #3718 / #405 instead of as a separate feature?
Which initial platform would best fit existing providers (browser, Electron, or macOS native)?

Originally created by @kangyishuai on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4069 ## Problem / motivation REA already provides valuable evidence-backed static and runtime analysis. For an external agent attempting **large-scale behavioral reconstruction** of an application it owns or is authorized to examine, the missing piece is often not another decompiler: it is a **composable, evidence-backed application interaction and observation contract** that can enumerate UI states and transitions, connect observations to REA evidence, and support repeatable comparisons. This is **not** a request for REA to implement a one-click commercial software clone, generate replacement applications, bypass protections, or own an autonomous orchestration loop. The external agent remains responsible for exploration strategy, implementation, and iterative repair. ## Requested capability / desired outcome Please consider defining an opt-in CLI/MCP contract (or documenting existing composable equivalents) for: 1. **UI state and action inventory**: observable windows/pages, accessible controls, actions, navigation, modal dialogs, and state identifiers, with platform/provider and visibility limitations explicitly recorded. 2. **Evidence-linked transitions**: given a caller-selected action and state, return before/after observations and references to relevant process, filesystem, IPC, network, and static-analysis evidence where available; unresolved associations must remain unknown rather than guessed. 3. **Repeatable scenario primitives**: caller-controlled initial conditions, action sequences, bounded execution, cancellation, cleanup, and replay/comparison metadata. Do not silently promise deterministic reset of external state. 4. **Coverage export**: machine-readable observed states/transitions, unvisited or unreachable candidates, source evidence, and gaps, so an external agent can prioritize follow-up experiments. 5. **Incremental use**: stable artifact/session references and import/export suitable for long-running investigations without requiring a monolithic in-memory analysis. A minimal first milestone could be a **read-only UI state/transition evidence schema plus an adapter for one supported target class**, using existing browser/process observations, rather than a universal UI automation engine. ## Example workflow (proposed, not tested) For a self-owned sample Electron app: - An agent asks REA for an observed UI state inventory. - It chooses a search action and supplies a bounded interaction scenario. - REA returns before/after UI observations, available IPC/network/process evidence, and explicit unknowns. - The agent uses those observations to implement its own equivalent feature, then compares recorded outcomes using existing comparison primitives. - Subsequent runs reuse prior evidence and expose unexplored states without claiming exhaustive coverage. ## Relationship to existing issues I searched for related issues and **do not want to duplicate them**: - #3718: universal static-to-runtime semantic correlation (this request would consume/extend its evidence links). - #405 and #268: reconstruction obligation/coverage workspace (this request focuses on *collecting and exporting UI interaction state/transition evidence* rather than owning closure policy). - #37 and #48: comparison and process observation foundations. - #69: resettable macOS environment (possible optional provider, not a requirement for every target). - #3636: Windows XAML structural reconstruction (format-specific, complementary). If maintainers believe existing primitives already suffice, a documented end-to-end composition example and explicit gap inventory would also address much of this request. ## Scope / safety / verification - Authorized, self-owned or suitably licensed sample applications only. - No anti-debugging/DRM/license bypass, stealth instrumentation, credential extraction, or attack automation. - Any execution, input injection, filesystem/network mutation, or state reset must declare its authority, side effects, time/resource limits, and cleanup; request explicit maintainer scope review before adding a provider with new execution powers. - Validate against a small self-owned sample with known UI transitions and intentionally missing/ambiguous observations. Check that unknowns remain unknown, that cancellation/cleanup work, and that output is stable enough for external agents. - **No runtime implementation or validation has been performed for this proposal.** ## Questions for maintainers 1. Is a provider-neutral UI interaction/observation evidence contract within REA's intended infrastructure scope? 2. Should this be tracked under #3718 / #405 instead of as a separate feature? 3. Which initial platform would best fit existing providers (browser, Electron, or macOS native)?

bsakaguchi commented

2026-10-16 21:07:57 +00:00

Author

Owner

@SoGeRiEn commented on GitHub (Oct 15, 2026):

Draft #4107: read-only browser-page v1. Pyppeteer stays external; no second browser engine or new execution authority.

@SoGeRiEn commented on GitHub (Oct 15, 2026): Draft #4107: read-only browser-page v1. Pyppeteer stays external; no second browser engine or new execution authority.

bsakaguchi commented

2026-10-16 21:07:58 +00:00

Author

Owner

@kangyishuai commented on GitHub (Oct 15, 2026):

A follow-up from the RFC author: here is a gap inventory against current main and a first increment that stays inside existing authority. Each row is labelled source-derived or measured.

What already exists (source-derived).

Browser: capture_browser_scenario declares actions and records post-action artifacts per step. compare_web_captures aligns two runs by step_id.
macOS: observe_native_ui and capture_native_ui_scenario record the AX tree with action names, and before/after snapshots for each step. decode_interface_builder and trace_native_ui_action cover static UI and the paths from controls to handlers.
Electron: capture_electron_scenario records action results and a main-process timeline.
Process: capture_process_scenario and compare_process_captures.
Cross-session reuse: obligation and coverage ledgers, residual unknowns, and evidence bundles.

Gaps (source-derived).

Steps within one run. A completed browser scenario could not be read step by step. inspect_analysis_view rejected it, and the only comparison is run against run.
Controls and available actions. Accessibility in scenario captures is retained only as text.
Native UI Evidence labelling. It records authority: shipped-artifact with environment: null, while the other scenario producers record controlled-replay.
Electron. There is one final window snapshot, and actions are not linked to events.
Planning and correlation. There is no UI candidate or frontier for obligations (#405/#268), and no step- or time-keyed join to network, IPC or filesystem Evidence (#3718).

First increment: draft #4163 (derived only; no new tool, provider or execution authority). inspect_analysis_view now reads retained or inline capture_browser_scenario Evidence:

a summary;
step rows: the recorded declaration, outcome, per-artifact coverage, and the change from the previous capture using the compare_web_captures digests;
events received in each step's window;
network transactions attributed by request, reported as null rather than 0 when network was not observed;
UTF-16 pages of accessibility or DOM text.

On a 20-action capture with screenshot, DOM, accessibility and URL after every step (about 2.2 MB, measured), the summary plus the step page is about 39 KB. That capture is still delivered inline. A separate detail: "summary" delivery for capture_browser_scenario would address it.

Relationship to #4107. #4107 derives one transition from two passive inspect_web_page records plus a caller-declared external action. The draft above projects one REA-controlled run. Neither introduces state keys or a shared "UI state" owner. If you want a provider-neutral state/transition vocabulary, I'd propose one owner that both surfaces consume, rather than two.

Proposed next steps, each as a separate PR:

(a) detail: "summary" delivery for capture_browser_scenario.
(b) A structured accessibility snapshot kind at the browser producer, recording observed facts only, with no locator or action suggestions.
(c) Fix the authority and environment of native UI Evidence at the producer.
(d) A native UI scenario parent for the same views.

Electron renderer UI reads, and any new launch, attach or input, would wait for your scope decision.

Questions:

Is (a) through (d) acceptable as routine work within existing boundaries?
Should UI state identity and coverage export be tracked here, or under #405/#3718?
Should derived views of controlled-replay captures inherit that authority (the current view rule), given that bundle scenarios manifests list every controlled-replay record?

@kangyishuai commented on GitHub (Oct 15, 2026): A follow-up from the RFC author: here is a gap inventory against current `main` and a first increment that stays inside existing authority. Each row is labelled source-derived or measured. **What already exists (source-derived).** - Browser: `capture_browser_scenario` declares actions and records post-action artifacts per step. `compare_web_captures` aligns two runs by `step_id`. - macOS: `observe_native_ui` and `capture_native_ui_scenario` record the AX tree with action names, and before/after snapshots for each step. `decode_interface_builder` and `trace_native_ui_action` cover static UI and the paths from controls to handlers. - Electron: `capture_electron_scenario` records action results and a main-process timeline. - Process: `capture_process_scenario` and `compare_process_captures`. - Cross-session reuse: obligation and coverage ledgers, residual unknowns, and evidence bundles. **Gaps (source-derived).** 1. *Steps within one run.* A completed browser scenario could not be read step by step. `inspect_analysis_view` rejected it, and the only comparison is run against run. 2. *Controls and available actions.* Accessibility in scenario captures is retained only as text. 3. *Native UI Evidence labelling.* It records `authority: shipped-artifact` with `environment: null`, while the other scenario producers record `controlled-replay`. 4. *Electron.* There is one final window snapshot, and actions are not linked to events. 5. *Planning and correlation.* There is no UI candidate or frontier for obligations (#405/#268), and no step- or time-keyed join to network, IPC or filesystem Evidence (#3718). **First increment: draft #4163 (derived only; no new tool, provider or execution authority).** `inspect_analysis_view` now reads retained or inline `capture_browser_scenario` Evidence: - a summary; - step rows: the recorded declaration, outcome, per-artifact coverage, and the change from the previous capture using the `compare_web_captures` digests; - events received in each step's window; - network transactions attributed by request, reported as `null` rather than `0` when network was not observed; - UTF-16 pages of accessibility or DOM text. On a 20-action capture with screenshot, DOM, accessibility and URL after every step (about 2.2 MB, measured), the summary plus the step page is about 39 KB. That capture is still delivered inline. A separate `detail: "summary"` delivery for `capture_browser_scenario` would address it. **Relationship to #4107.** #4107 derives one transition from two passive `inspect_web_page` records plus a caller-declared external action. The draft above projects one REA-controlled run. Neither introduces state keys or a shared "UI state" owner. If you want a provider-neutral state/transition vocabulary, I'd propose one owner that both surfaces consume, rather than two. **Proposed next steps, each as a separate PR:** - (a) `detail: "summary"` delivery for `capture_browser_scenario`. - (b) A structured accessibility snapshot kind at the browser producer, recording observed facts only, with no locator or action suggestions. - (c) Fix the authority and environment of native UI Evidence at the producer. - (d) A native UI scenario parent for the same views. Electron renderer UI reads, and any new launch, attach or input, would wait for your scope decision. **Questions:** 1. Is (a) through (d) acceptable as routine work within existing boundaries? 2. Should UI state identity and coverage export be tracked here, or under #405/#3718? 3. Should derived views of `controlled-replay` captures inherit that authority (the current view rule), given that bundle `scenarios` manifests list every `controlled-replay` record?

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:12:43 +00:00

[PR #1601] [MERGED] perf(javascript): reduce retained semantic analysis allocations #2996

bsakaguchi referenced this issue

2026-10-17 21:12:47 +00:00

[PR #1610] [CLOSED] fix(javascript): bound module source payload before deep semantic analysis #3002