morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #5049] feat(adb): inspect the on-screen view hierarchy #6760

New issue

Open

opened 2026-11-01 21:42:51 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:42:51 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5049
Author: @RoKenobi
Created: 10/18/2026
Status: 🔄 Open

Base: main ← Head: feat/adb-view-hierarchy

📝 Commits (2)

1d36647 feat(adb): inspect the on-screen view hierarchy
0ab78ff docs(adb): note system UI and Compose coverage in view hierarchies

📊 Changes

14 files changed (+757 additions, -26 deletions)

View changed files

📝 docs/adb-device-analysis.md (+54 -26)
📝 scripts/verify-adb.mjs (+10 -0)
📝 src/adb/AdbProvider.ts (+56 -0)
➕ src/adb/AdbViewHierarchy.test.ts (+152 -0)
➕ src/adb/AdbViewHierarchy.ts (+220 -0)
📝 src/cli/adbCommands.ts (+10 -0)
📝 src/cliCommandNames.ts (+1 -0)
📝 src/contracts/adb/adbToolContracts.ts (+15 -0)
📝 src/contracts/toolEffects.ts (+5 -0)
📝 src/domain/adb/adbDeviceAnalysis.ts (+78 -0)
📝 src/server/registerAdbTools.ts (+1 -0)
➕ tests/boundary/cli/adbViewHierarchy.test.ts (+127 -0)
📝 tests/boundary/mcp/adbMcp.test.ts (+27 -0)
📝 tests/boundary/process/adbVerifierCleanup.test.ts (+1 -0)

📄 Description

Summary

Draft prototype for #5033: a read-only inspect_adb_view_hierarchy tool (CLI inspect-adb-view-hierarchy) that returns the on-screen accessibility view hierarchy as Evidence. Opened as a draft to inform the scope decision requested in #5033; it does not ask for integration before that decision.

Problem and expected behavior

REA's adb tools report focus (inspect_adb_window), device and display facts, and screenshots, but not which elements are on screen and where. Studying an app's layout on a device currently means leaving REA for a manual uiautomator dump, so those facts have no Evidence record.

With this change, one call returns document-order nodes (parent, depth, class, package, resource-id, text, content-desc, pixel bounds, boolean states) plus the SHA-256 of the exact document bytes, with explicit limitations.

Change and scope

Runs a fixed argv: adb -s SERIAL exec-out uiautomator dump /dev/tty. No caller-controlled command text, no device-side file, no interaction.
Why exec-out, not adb shell: on the tested device, adb shell uiautomator dump /dev/tty (and /proc/self/fd/1) returned only the 33-byte status line, while exec-out returned the full document (about 13–29 KB). The existing shell helper therefore cannot be reused. This also refines the proposal in #5033.
exec-out merges the device's stdout and stderr and does not return the device exit status. Non-document lines are kept verbatim in device_messages. The parser classifies:
- captured: a <hierarchy> document was found and parsed with the shared parseXmlDocument (doctype rejected).
- not_idle: no document, and uiautomator printed could not get idle state (screens with continuous animation). Returned as a successful observation with an "unknown, not empty" limitation, not as an error.
- unavailable: no document and no recognized reason.
- A malformed document is an AnalysisProtocolError.
Known boolean attributes declared true go to flags; known attributes a node did not declare go to undeclared_flags (unknown, not false). Other non-empty attributes, and known values that fail to parse, go to other_attributes verbatim.
New parser lives in src/adb/AdbViewHierarchy.ts; contract, effects, MCP registration, CLI command, docs and the verify:adb lane follow the inspect_adb_window pattern.
Not included: interaction (taps, keys), --compressed, multi-display selection, paging, or any change to existing tools.

Contract and boundary impact

Semantic owner and earliest changed stage: ADB adapter (src/adb/) parses device output; domain schema in src/domain/adb/adbDeviceAnalysis.ts.
Contribution scope fit and linked maintainer decision: requested in #5033; pending.
CLI and MCP/tool-catalog contract: additive. New tool inspect_adb_view_hierarchy and CLI command inspect-adb-view-hierarchy; no existing contract changes.
Provider, bridge, target-format, or platform compatibility: adb provider only. Verified on one physical device (see below); emulators and other Android versions unknown.
Evidence, artifact, provenance, or reconstruction contract: new result schema; document digest and size retained; observed values only.
Target selection, authority, upstream behavior, and cleanup details: caller-selected serial; one bounded adb invocation (30 s, 8 MiB); no persistent processes or files. uiautomator itself waits up to 10 s for an idle UI. Unknown: behaviour when another UiAutomation client (for example a running instrumentation test) is connected.
Generated metadata, package, or installation impact: none committed; catalogs regenerate in CI.

Expected effects:

Read existing files or captured data.
Launch or attach to a process. (Starts the device-side uiautomator command; does not attach to the foreground app.)
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services. (Local adb server, as for the existing adb tools.)
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated:
- src/adb/AdbViewHierarchy.test.ts (6): document order/parents/depth, exact-byte digest, flags vs undeclared flags, unparsable values retained, not_idle, unavailable, empty self-closing hierarchy, malformed and DOCTYPE documents rejected.
- tests/boundary/cli/adbViewHierarchy.test.ts (3): CLI and MCP with a fixture adb for captured and not-idle devices (schemas validated against the advertised MCP schemas), and malformed output surfacing as unreadable_output.
- tests/boundary/mcp/adbMcp.test.ts: catalog and advertised-contract check include the new tool.
- tests/boundary/process/adbVerifierCleanup.test.ts: verifier stub covers the new call.
Base reproduction or other evidence (executed, OnePlus CPH2581, Android 16 / SDK 36, adb 28.0.2-debian):
- Settings screen: captured, 77 nodes, depth up to 13, 29,502-byte document.
- Clock app with a running stopwatch: not_idle after about 12 s with device_messages: ["ERROR: could not get idle state."].
- Launcher home screen, two consecutive runs: identical document_sha256 and identical nodes (35). Each CLI run took about 2.95 s, of which about 2.4 s is uiautomator dump itself (measured with plain adb exec-out); an existing adb CLI tool (inspect-adb-display) takes about 0.7 s on the same host.
- Output size: 14,688-byte Evidence JSON for a 13,038-byte raw document (compact nodes array 13,136 bytes), so the projection does not expand the payload materially.
- A third-party Jetpack Compose app: captured, 189 nodes. Compose nodes with semantics appear as android.view.View / Button / TextView descendants of each ComposeView, without resource IDs.
- Screen off (always-on display, mWakefulness=Dozing): captured, 46 nodes, all com.android.systemui. Lock screen after wake: captured, 107 nodes, all com.android.systemui. The docs now say that the hierarchy is what is on screen, not necessarily the foreground app.
- After all runs, /sdcard/window_dump.xml did not exist.
User-visible CLI/MCP output (real device, trimmed):

{
  "hierarchy_status": "captured",
  "rotation": 0,
  "document_sha256": "9a24617aaca1d50cfe289a4bbc3b94569fa1f2116f7a10d7ff28e04af537aa9b",
  "document_bytes": 29502,
  "device_messages": ["UI hierchary dumped to: /dev/tty"],
  "nodes": [
    "… 77 nodes; one shown …",
    {
      "index": 9, "parent": 7, "depth": 8, "sibling_index": 1,
      "class_name": "android.widget.TextView",
      "package": "com.android.settings",
      "resource_id": null,
      "text": "Display & brightness",
      "content_desc": null,
      "bounds": { "left": 168, "top": 160, "right": 1032, "bottom": 233 },
      "flags": ["enabled"],
      "undeclared_flags": [],
      "other_attributes": [{ "name": "drawing-order", "value": "1" }]
    }
  ]
}

Remaining proof gaps: emulator and other Android versions; Windows and macOS hosts (fixture tests skip on Windows like the existing window test); concurrent UiAutomation clients; multi-display devices; landscape rotation; very large hierarchies against the 8 MiB cap (largest observed: about 29 KB). npm run verify:adb was not run: by default it pulls the first third-party APK from the device and pushes a file, which was out of scope for a personal device; the new check was added to that lane but has not run through it.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run test:focused -- src/adb/AdbViewHierarchy.test.ts tests/boundary/cli/adbViewHierarchy.test.ts tests/boundary/cli/adbWindowFocus.test.ts tests/boundary/mcp/adbMcp.test.ts tests/boundary/process/adbVerifierCleanup.test.ts — 5 files, 20 tests passed.
npm run check:changed — 78 files, 1461 passed, 3 skipped.
npm run check:fast — typecheck and lint passed.
npm run docs:check — passed.
npm run build:cached, then node scripts/rea.mjs inspect-adb-view-hierarchy <serial> --json on the device above — captured (Settings, launcher, a Compose app, always-on display, lock screen) and not_idle (running stopwatch) as described.
npm run docs:check after the follow-up docs commit — passed.

Compatibility, safety, and release

Breaking changes or migration steps: none; additive tool.
Real Hopper/Ghidra, browser, or OS coverage: not applicable; Linux host with one physical Android 16 device.
Package or release metadata impact: new tool in generated catalogs.
Security, privacy, process, or containment review: returns on-screen text, which can include personal data shown on the device; same exposure class as capture_adb_screen. No device writes.

Author readiness

Human owner: Rokenobi

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5049 **Author:** [@RoKenobi](https://github.com/RoKenobi) **Created:** 10/18/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/adb-view-hierarchy` --- ### 📝 Commits (2) - [`1d36647`](https://github.com/morluto/rea/commit/1d36647cd39bcaba44b763ea321e947b1c33746d) feat(adb): inspect the on-screen view hierarchy - [`0ab78ff`](https://github.com/morluto/rea/commit/0ab78ffb335cdf97aeabf9c8eeede72a2415294c) docs(adb): note system UI and Compose coverage in view hierarchies ### 📊 Changes **14 files changed** (+757 additions, -26 deletions) <details> <summary>View changed files</summary> 📝 `docs/adb-device-analysis.md` (+54 -26) 📝 `scripts/verify-adb.mjs` (+10 -0) 📝 `src/adb/AdbProvider.ts` (+56 -0) ➕ `src/adb/AdbViewHierarchy.test.ts` (+152 -0) ➕ `src/adb/AdbViewHierarchy.ts` (+220 -0) 📝 `src/cli/adbCommands.ts` (+10 -0) 📝 `src/cliCommandNames.ts` (+1 -0) 📝 `src/contracts/adb/adbToolContracts.ts` (+15 -0) 📝 `src/contracts/toolEffects.ts` (+5 -0) 📝 `src/domain/adb/adbDeviceAnalysis.ts` (+78 -0) 📝 `src/server/registerAdbTools.ts` (+1 -0) ➕ `tests/boundary/cli/adbViewHierarchy.test.ts` (+127 -0) 📝 `tests/boundary/mcp/adbMcp.test.ts` (+27 -0) 📝 `tests/boundary/process/adbVerifierCleanup.test.ts` (+1 -0) </details> ### 📄 Description ## Summary Draft prototype for #5033: a read-only `inspect_adb_view_hierarchy` tool (CLI `inspect-adb-view-hierarchy`) that returns the on-screen accessibility view hierarchy as Evidence. Opened as a draft to inform the scope decision requested in #5033; it does not ask for integration before that decision. ## Problem and expected behavior REA's adb tools report focus (`inspect_adb_window`), device and display facts, and screenshots, but not which elements are on screen and where. Studying an app's layout on a device currently means leaving REA for a manual `uiautomator dump`, so those facts have no Evidence record. With this change, one call returns document-order nodes (parent, depth, class, package, resource-id, text, content-desc, pixel bounds, boolean states) plus the SHA-256 of the exact document bytes, with explicit limitations. ## Change and scope - Runs a fixed argv: `adb -s SERIAL exec-out uiautomator dump /dev/tty`. No caller-controlled command text, no device-side file, no interaction. - **Why exec-out, not `adb shell`:** on the tested device, `adb shell uiautomator dump /dev/tty` (and `/proc/self/fd/1`) returned only the 33-byte status line, while `exec-out` returned the full document (about 13–29 KB). The existing `shell` helper therefore cannot be reused. This also refines the proposal in #5033. - exec-out merges the device's stdout and stderr and does not return the device exit status. Non-document lines are kept verbatim in `device_messages`. The parser classifies: - `captured`: a `<hierarchy>` document was found and parsed with the shared `parseXmlDocument` (doctype rejected). - `not_idle`: no document, and uiautomator printed `could not get idle state` (screens with continuous animation). Returned as a successful observation with an "unknown, not empty" limitation, not as an error. - `unavailable`: no document and no recognized reason. - A malformed document is an `AnalysisProtocolError`. - Known boolean attributes declared `true` go to `flags`; known attributes a node did not declare go to `undeclared_flags` (unknown, not false). Other non-empty attributes, and known values that fail to parse, go to `other_attributes` verbatim. - New parser lives in `src/adb/AdbViewHierarchy.ts`; contract, effects, MCP registration, CLI command, docs and the `verify:adb` lane follow the `inspect_adb_window` pattern. - **Not included:** interaction (taps, keys), `--compressed`, multi-display selection, paging, or any change to existing tools. ## Contract and boundary impact - Semantic owner and earliest changed stage: ADB adapter (`src/adb/`) parses device output; domain schema in `src/domain/adb/adbDeviceAnalysis.ts`. - Contribution scope fit and linked maintainer decision: requested in #5033; pending. - CLI and MCP/tool-catalog contract: additive. New tool `inspect_adb_view_hierarchy` and CLI command `inspect-adb-view-hierarchy`; no existing contract changes. - Provider, bridge, target-format, or platform compatibility: adb provider only. Verified on one physical device (see below); emulators and other Android versions unknown. - Evidence, artifact, provenance, or reconstruction contract: new result schema; document digest and size retained; observed values only. - Target selection, authority, upstream behavior, and cleanup details: caller-selected serial; one bounded adb invocation (30 s, 8 MiB); no persistent processes or files. uiautomator itself waits up to 10 s for an idle UI. **Unknown:** behaviour when another UiAutomation client (for example a running instrumentation test) is connected. - Generated metadata, package, or installation impact: none committed; catalogs regenerate in CI. Expected effects: - [ ] Read existing files or captured data. - [x] Launch or attach to a process. (Starts the device-side `uiautomator` command; does not attach to the foreground app.) - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [x] Contact network services. (Local adb server, as for the existing adb tools.) - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `src/adb/AdbViewHierarchy.test.ts` (6): document order/parents/depth, exact-byte digest, flags vs undeclared flags, unparsable values retained, `not_idle`, `unavailable`, empty self-closing hierarchy, malformed and DOCTYPE documents rejected. - `tests/boundary/cli/adbViewHierarchy.test.ts` (3): CLI and MCP with a fixture adb for captured and not-idle devices (schemas validated against the advertised MCP schemas), and malformed output surfacing as `unreadable_output`. - `tests/boundary/mcp/adbMcp.test.ts`: catalog and advertised-contract check include the new tool. - `tests/boundary/process/adbVerifierCleanup.test.ts`: verifier stub covers the new call. - Base reproduction or other evidence (executed, OnePlus CPH2581, Android 16 / SDK 36, adb 28.0.2-debian): - Settings screen: `captured`, 77 nodes, depth up to 13, 29,502-byte document. - Clock app with a running stopwatch: `not_idle` after about 12 s with `device_messages: ["ERROR: could not get idle state."]`. - Launcher home screen, two consecutive runs: identical `document_sha256` and identical nodes (35). Each CLI run took about 2.95 s, of which about 2.4 s is `uiautomator dump` itself (measured with plain `adb exec-out`); an existing adb CLI tool (`inspect-adb-display`) takes about 0.7 s on the same host. - Output size: 14,688-byte Evidence JSON for a 13,038-byte raw document (compact `nodes` array 13,136 bytes), so the projection does not expand the payload materially. - A third-party Jetpack Compose app: `captured`, 189 nodes. Compose nodes with semantics appear as `android.view.View` / `Button` / `TextView` descendants of each `ComposeView`, without resource IDs. - Screen off (always-on display, `mWakefulness=Dozing`): `captured`, 46 nodes, all `com.android.systemui`. Lock screen after wake: `captured`, 107 nodes, all `com.android.systemui`. The docs now say that the hierarchy is what is on screen, not necessarily the foreground app. - After all runs, `/sdcard/window_dump.xml` did not exist. - User-visible CLI/MCP output (real device, trimmed): ```json { "hierarchy_status": "captured", "rotation": 0, "document_sha256": "9a24617aaca1d50cfe289a4bbc3b94569fa1f2116f7a10d7ff28e04af537aa9b", "document_bytes": 29502, "device_messages": ["UI hierchary dumped to: /dev/tty"], "nodes": [ "… 77 nodes; one shown …", { "index": 9, "parent": 7, "depth": 8, "sibling_index": 1, "class_name": "android.widget.TextView", "package": "com.android.settings", "resource_id": null, "text": "Display & brightness", "content_desc": null, "bounds": { "left": 168, "top": 160, "right": 1032, "bottom": 233 }, "flags": ["enabled"], "undeclared_flags": [], "other_attributes": [{ "name": "drawing-order", "value": "1" }] } ] } ``` - Remaining proof gaps: emulator and other Android versions; Windows and macOS hosts (fixture tests skip on Windows like the existing window test); concurrent UiAutomation clients; multi-display devices; landscape rotation; very large hierarchies against the 8 MiB cap (largest observed: about 29 KB). `npm run verify:adb` was not run: by default it pulls the first third-party APK from the device and pushes a file, which was out of scope for a personal device; the new check was added to that lane but has not run through it. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- src/adb/AdbViewHierarchy.test.ts tests/boundary/cli/adbViewHierarchy.test.ts tests/boundary/cli/adbWindowFocus.test.ts tests/boundary/mcp/adbMcp.test.ts tests/boundary/process/adbVerifierCleanup.test.ts` — 5 files, 20 tests passed. - `npm run check:changed` — 78 files, 1461 passed, 3 skipped. - `npm run check:fast` — typecheck and lint passed. - `npm run docs:check` — passed. - `npm run build:cached`, then `node scripts/rea.mjs inspect-adb-view-hierarchy <serial> --json` on the device above — `captured` (Settings, launcher, a Compose app, always-on display, lock screen) and `not_idle` (running stopwatch) as described. - `npm run docs:check` after the follow-up docs commit — passed. ## Compatibility, safety, and release - Breaking changes or migration steps: none; additive tool. - Real Hopper/Ghidra, browser, or OS coverage: not applicable; Linux host with one physical Android 16 device. - Package or release metadata impact: new tool in generated catalogs. - Security, privacy, process, or containment review: returns on-screen text, which can include personal data shown on the device; same exposure class as `capture_adb_screen`. No device writes. ## Author readiness Human owner: Rokenobi - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-11-01 21:42:51 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6760

No description provided.

Rows
Columns