[PR #3004] feat(adb): add UI Automator discovery, waits and actions #5555

New issue

Open

opened 2026-10-31 01:07:32 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:07:32 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3004
Author: @3QL
Created: 10/13/2026
Status: 🔄 Open

Base: main ← Head: feat/adb-ui-automation

📝 Commits (10+)

95f9950 fix(adb): preserve window focus across Android dump sections
801ad30 test(adb): prototype UI hierarchy and explicit input workflows
75ada4a fix(adb): exclude saved ANR state from live window focus
8a1e608 fix(adb): bind UI probe admission and cleanup ownership
b488a14 chore(adb): merge main and retain upstream focus inspection
427594b fix(adb): settle installers before serialized fixture cleanup
b5b35bd fix(adb): anchor verifier launches to its checkout
2eb9db3 feat(adb): add UI Automator discovery and actions
2302c74 fix(adb): release UI requests that never launch
010b22a fix(adb): validate password state before UI text entry

📊 Changes

138 files changed (+11387 additions, -447 deletions)

View changed files

📝 .oxlintrc.json (+5 -1)
📝 README.md (+1 -0)
📝 THIRD_PARTY_NOTICES.md (+1 -0)
➕ bridge/adb-ui/.gitignore (+3 -0)
➕ bridge/adb-ui/README.md (+50 -0)
➕ bridge/adb-ui/build.gradle (+32 -0)
➕ bridge/adb-ui/gradle.properties (+3 -0)
➕ bridge/adb-ui/settings.gradle (+8 -0)
➕ bridge/adb-ui/src/main/AndroidManifest.xml (+6 -0)
➕ bridge/adb-ui/src/main/java/dev/rea/uiautomator/ReaUiRunner.java (+1245 -0)
➕ bridge/adb-ui/src/main/java/dev/rea/uiautomator/ReaUiSocket.java (+134 -0)
📝 docs/adb-device-analysis.md (+11 -7)
➕ docs/adb-ui-automation.md (+286 -0)
📝 docs/client-configuration.md (+1 -1)
📝 docs/readme/README_ar.md (+1 -0)
📝 docs/readme/README_de.md (+1 -0)
📝 docs/readme/README_el.md (+1 -0)
📝 docs/readme/README_es.md (+1 -0)
📝 docs/readme/README_fa.md (+1 -0)
📝 docs/readme/README_fr.md (+1 -0)

...and 80 more files

📄 Description

Summary

Adds package-scoped UI inspection, discovery and waits through shared CLI/MCP workflows using an operator-started UI Automator runner.

Actions remain unshipped: perform_adb_ui_action is absent from normal CLI/MCP registration and the public catalog. Only experiments/adb-ui/mcp.mjs binds the proposal; npm excludes experiment and runner sources. Maintainer mutation-scope approval remains pending.
Definitive refusals stay definitive: the transport distinguishes uncertain handoff, acceptance and refusal. A busy reply remains a capacity refusal through cleanup recovery; genuinely unknown outcomes retain replay safeguards.
Reconciles main at 3ce311757, adopts upstream comparison and fixture repairs, and removes the unrelated JavaScript patch from this PR’s final diff.
Preserves shared input-error projection, display observations, Evidence provenance and MCP cleanup ownership.

Validation — `2c8149173`

Runtime validation was performed at 5f746544d; 2c8149173 is a documentation-only follow-up.

The 22 failed CDP/HTTP comparisons reproduce on exact CI base f2db494bf. Upstream fixes preserve uncertainty for incomplete captures and retain positive change-detection assertions. All selected comparison tests now pass.
984 focused tests across 62 files covered successfully: 940 tests in 61 files passed. After replacing a brittle recursive diagnostic-count assertion with checks of the documented error fields, all 44 runner tests passed.
npm run check, npm run docs:check, test discovery, 18 experiment/lifecycle tests and installed-package verification passed.
The package inventory contains 2,319 files, excluding runner, experiment and test sources.
After the commit hook normalized an equivalent ADB numeric regex, five affected tests and static/docs checks passed again.
GitHub reports the main conflicts resolved. Current-head CI is not yet verified green.

Verification limits

No full local-suite pass or new Android runtime run is claimed. Earlier API 36 evidence predates this reconciliation. Native runner contention, physical devices/OEMs, older Android and native Windows/macOS UI behavior remain unverified.

Pinned main also has independent CI failures. An isolated JavaScript reference-lifetime test exceeded a bounded 30-second run, and a separate stored-getter provenance probe fails on main. Their evidence is retained separately from this UI change.

Coordination with #3840

#3840 covers wait_adb_boot and trace_android_resource_references; resource IDs from UI observations can feed tracing against a caller-selected APK.

Original scope request and historical follow-up

Maintainer scope decision remains pending before integration. This capability observes and mutates a caller-selected target application's UI (including text replacement and key events), although the helper itself is operator-built and does not inject code into the target process. Please decide whether REA should integrate this interactive capability under the documented operator-owned boundary, and define any limits for supported targets/actions, sensitive text, focus/identity, and permission dialogs. The existing implementation exposes no automatic permission-dialog watcher, arbitrary shell command, or caller-authored code. The prior PR body reported approval relayed by the human owner; this draft does not treat that report as the maintainer decision required for integration.

Current follow-up evidence

Input-submission uncertainty is now recorded at the shared owned-command stdin handoff, rather than before a command can launch. Definitive pre-handoff cancellation remains cancellation. Recovered action outcomes have their own durable owner, independent of remote request cleanup, so a lost forget acknowledgement or later cancellation cannot discard them. A matching fresh inspection clears the exact unknown request from both its gate and durable map; unrelated outcomes remain retained. The direct unknown→matching inspect→next action regression verifies there is no extra stale refusal.

The runner/owned-command set passed 67 tests, fixture lifecycle tests passed 12/12, and the final inspection-clearing runner suite passed 38/38, all on Node 24.18.0/npm 11.16.0. check:fast and push gates passed. Independent review found the stale-map issue and the final patch fixes it; no device/emulator was run.

The installer-fixture finding is already structurally prevented: the fixture awaits exact command-owner settlement before package reconciliation; unsuccessful settlement prevents inventory/uninstall and retains its lease/journal. Late-install and retained-owner regressions passed in the 12-test fixture suite. A stale mock was updated to the current unparsed_lines Evidence contract. This evidence does not supply the pending maintainer scope decision.

The older lifecycle findings are also covered by the current owners: CLI execution delegates to finishAnalysisCommand, which combines operation and close failures while preserving partial observations; receiver-setup failure closes the socket and retains cleanup as a suppressed cause; terminal runner completions are bounded before being retained so status/cancel envelopes remain releasable. These are source-derived ownership checks plus the scoped regression evidence above, not a real Android runtime reproduction.

Author readiness

Human owner: @3QL

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3004 **Author:** [@3QL](https://github.com/3QL) **Created:** 10/13/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/adb-ui-automation` --- ### 📝 Commits (10+) - [`95f9950`](https://github.com/morluto/rea/commit/95f9950ed32466131cca67b49ec9ad10baa40c91) fix(adb): preserve window focus across Android dump sections - [`801ad30`](https://github.com/morluto/rea/commit/801ad3060aa35b58649838e6977d4b2fa0a35f89) test(adb): prototype UI hierarchy and explicit input workflows - [`75ada4a`](https://github.com/morluto/rea/commit/75ada4a3a841cfcbf1404131dc0785a7d18955e3) fix(adb): exclude saved ANR state from live window focus - [`8a1e608`](https://github.com/morluto/rea/commit/8a1e608f1a182570413e016e50db733e5572be57) fix(adb): bind UI probe admission and cleanup ownership - [`b488a14`](https://github.com/morluto/rea/commit/b488a14d1746a846138288f8f84d0c5514e28afd) chore(adb): merge main and retain upstream focus inspection - [`427594b`](https://github.com/morluto/rea/commit/427594be806e8341000be73908eec5cf59bdabb9) fix(adb): settle installers before serialized fixture cleanup - [`b5b35bd`](https://github.com/morluto/rea/commit/b5b35bda82d7700c2f399f3bea76cd285b0d9ea9) fix(adb): anchor verifier launches to its checkout - [`2eb9db3`](https://github.com/morluto/rea/commit/2eb9db36ffc600efe2d5fea29ec2668fc7a0bc28) feat(adb): add UI Automator discovery and actions - [`2302c74`](https://github.com/morluto/rea/commit/2302c7449a7cf0c8656b2bd7acac422eeaef10c4) fix(adb): release UI requests that never launch - [`010b22a`](https://github.com/morluto/rea/commit/010b22a6d3f2f97cb614fc9a2ccb4c3730d2e609) fix(adb): validate password state before UI text entry ### 📊 Changes **138 files changed** (+11387 additions, -447 deletions) <details> <summary>View changed files</summary> 📝 `.oxlintrc.json` (+5 -1) 📝 `README.md` (+1 -0) 📝 `THIRD_PARTY_NOTICES.md` (+1 -0) ➕ `bridge/adb-ui/.gitignore` (+3 -0) ➕ `bridge/adb-ui/README.md` (+50 -0) ➕ `bridge/adb-ui/build.gradle` (+32 -0) ➕ `bridge/adb-ui/gradle.properties` (+3 -0) ➕ `bridge/adb-ui/settings.gradle` (+8 -0) ➕ `bridge/adb-ui/src/main/AndroidManifest.xml` (+6 -0) ➕ `bridge/adb-ui/src/main/java/dev/rea/uiautomator/ReaUiRunner.java` (+1245 -0) ➕ `bridge/adb-ui/src/main/java/dev/rea/uiautomator/ReaUiSocket.java` (+134 -0) 📝 `docs/adb-device-analysis.md` (+11 -7) ➕ `docs/adb-ui-automation.md` (+286 -0) 📝 `docs/client-configuration.md` (+1 -1) 📝 `docs/readme/README_ar.md` (+1 -0) 📝 `docs/readme/README_de.md` (+1 -0) 📝 `docs/readme/README_el.md` (+1 -0) 📝 `docs/readme/README_es.md` (+1 -0) 📝 `docs/readme/README_fa.md` (+1 -0) 📝 `docs/readme/README_fr.md` (+1 -0) _...and 80 more files_ </details> ### 📄 Description ## Summary Adds package-scoped UI inspection, discovery and waits through shared CLI/MCP workflows using an operator-started UI Automator runner. - **Actions remain unshipped:** `perform_adb_ui_action` is absent from normal CLI/MCP registration and the public catalog. Only `experiments/adb-ui/mcp.mjs` binds the proposal; npm excludes experiment and runner sources. Maintainer mutation-scope approval remains pending. - **Definitive refusals stay definitive:** the transport distinguishes uncertain handoff, acceptance and refusal. A `busy` reply remains a capacity refusal through cleanup recovery; genuinely unknown outcomes retain replay safeguards. - Reconciles main at `3ce311757`, adopts upstream comparison and fixture repairs, and removes the unrelated JavaScript patch from this PR’s final diff. - Preserves shared input-error projection, display observations, Evidence provenance and MCP cleanup ownership. ## Validation — `2c8149173` Runtime validation was performed at `5f746544d`; `2c8149173` is a documentation-only follow-up. - The 22 failed CDP/HTTP comparisons reproduce on exact CI base `f2db494bf`. Upstream fixes preserve uncertainty for incomplete captures and retain positive change-detection assertions. All selected comparison tests now pass. - **984 focused tests across 62 files covered successfully:** 940 tests in 61 files passed. After replacing a brittle recursive diagnostic-count assertion with checks of the documented error fields, all 44 runner tests passed. - `npm run check`, `npm run docs:check`, test discovery, 18 experiment/lifecycle tests and installed-package verification passed. - The package inventory contains **2,319 files**, excluding runner, experiment and test sources. - After the commit hook normalized an equivalent ADB numeric regex, five affected tests and static/docs checks passed again. - GitHub reports the main conflicts resolved. [Current-head CI](https://github.com/morluto/rea/actions/runs/40080915862) is **not yet verified green**. ### Verification limits No full local-suite pass or new Android runtime run is claimed. Earlier API 36 evidence predates this reconciliation. Native runner contention, physical devices/OEMs, older Android and native Windows/macOS UI behavior remain unverified. Pinned main also has independent CI failures. An isolated JavaScript reference-lifetime test exceeded a bounded 30-second run, and a separate stored-getter provenance probe fails on main. Their evidence is retained separately from this UI change. ## Coordination with #3840 #3840 covers `wait_adb_boot` and `trace_android_resource_references`; resource IDs from UI observations can feed tracing against a caller-selected APK. <details> <summary>Original scope request and historical follow-up</summary> **Maintainer scope decision remains pending before integration.** This capability observes and mutates a caller-selected target application's UI (including text replacement and key events), although the helper itself is operator-built and does not inject code into the target process. Please decide whether REA should integrate this interactive capability under the documented operator-owned boundary, and define any limits for supported targets/actions, sensitive text, focus/identity, and permission dialogs. The existing implementation exposes no automatic permission-dialog watcher, arbitrary shell command, or caller-authored code. The prior PR body reported approval relayed by the human owner; this draft does not treat that report as the maintainer decision required for integration.  ### Current follow-up evidence Input-submission uncertainty is now recorded at the shared owned-command stdin handoff, rather than before a command can launch. Definitive pre-handoff cancellation remains cancellation. Recovered action outcomes have their own durable owner, independent of remote request cleanup, so a lost forget acknowledgement or later cancellation cannot discard them. A matching fresh inspection clears the exact unknown request from both its gate and durable map; unrelated outcomes remain retained. The direct unknown→matching inspect→next action regression verifies there is no extra stale refusal. The runner/owned-command set passed 67 tests, fixture lifecycle tests passed 12/12, and the final inspection-clearing runner suite passed 38/38, all on Node 24.18.0/npm 11.16.0. `check:fast` and push gates passed. Independent review found the stale-map issue and the final patch fixes it; no device/emulator was run. The installer-fixture finding is already structurally prevented: the fixture awaits exact command-owner settlement before package reconciliation; unsuccessful settlement prevents inventory/uninstall and retains its lease/journal. Late-install and retained-owner regressions passed in the 12-test fixture suite. A stale mock was updated to the current `unparsed_lines` Evidence contract. This evidence does not supply the pending maintainer scope decision. The older lifecycle findings are also covered by the current owners: CLI execution delegates to `finishAnalysisCommand`, which combines operation and close failures while preserving partial observations; receiver-setup failure closes the socket and retains cleanup as a suppressed cause; terminal runner completions are bounded before being retained so status/cancel envelopes remain releasable. These are source-derived ownership checks plus the scoped regression evidence above, not a real Android runtime reproduction. </details> ## Author readiness Human owner: @3QL - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:07:32 +00:00

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:44:19 +00:00

[PR #5548] [MERGED] fix(native): remove completed signing capture limitation #6956

bsakaguchi referenced this issue

2026-11-01 21:44:19 +00:00