morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5033] [Feature] Read-only Android view hierarchy capture over adb #3092

New issue

Open

opened 2026-10-31 00:36:31 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:36:31 +00:00

Owner

Originally created by @RoKenobi on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5033

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Process capture or runtime observation

Expected interaction with targets

Not sure yet (explained under "Target and scope details")

Verification available so far

I have a prototype or observed results

Current limitation

REA's adb tools can report the focused window (inspect_adb_window), device and display
facts, and a screenshot (capture_adb_screen), but there is no tool for the on-screen view
hierarchy: which elements are visible, their bounds, resource IDs, classes, text and
content descriptions.

While studying an installed app's screen layout on a real device, I had to leave REA and run
adb shell uiautomator dump by hand. Those measurements then had no Evidence record, so a
REA-based report mixed cited results with uncited manual output.

Desired outcome and success criteria

A read-only tool (for example inspect_adb_view_hierarchy) that, for one device serial:

returns the current hierarchy as Evidence: per node, bounds, resource-id, class, package,
text, content-desc and the boolean state attributes, plus the raw XML digest;
writes no file on the device (stream the dump back, like capture_adb_screen's exec-out);
reports the display size/density context needed to convert px to dp, or links to
inspect_adb_display Evidence;
reports Android's dump failures as typed limitations instead of a generic error,
especially ERROR: could not get idle state. on screens that never go idle;
states the coverage limit that Jetpack Compose screens expose only nodes with semantics,
so an absent node is not evidence that nothing is drawn there.

Target and scope details

Runs one device-side shell command (uiautomator dump). It does not launch, attach to,
inject into or change the foreground app; it reads what the accessibility service exposes.
With /dev/tty as the output path there is no device-side file. The default command writes
/sdcard/window_dump.xml, which the tool should avoid.
No interaction (taps, keys). Navigation stays with the caller or existing tools.

Verification details

Observed on a OnePlus CPH2581, Android 16 (SDK 36), adb 28.0.2-debian, REA 6.5.0
(gap also confirmed in source on main @ 23f984b):

adb exec-out uiautomator dump /dev/tty returned 18,560 bytes: a complete XML document
followed by the plain-text line UI hierchary dumped to: /dev/tty (Android's spelling),
which a parser must strip. /sdcard/window_dump.xml did not exist afterwards.
On an app screen with continuous animation (a media player with a looping video
background), uiautomator dump failed with ERROR: could not get idle state.
On Jetpack Compose screens, the dump contained the outer View frame (app bar, navigation
bar, containers) but not the individual Compose elements inside a ComposeView.

Not yet checked: other Android versions, multi-display devices, --compressed output, and
behaviour while the screen is locked.

Alternatives considered

capture_adb_screen: gives pixels only; element bounds must be measured by eye.
inspect_adb_window: gives only the focused window and activity.
collect_adb_bugreport: far broader than needed and does not contain the view tree.
Manual adb outside REA: works, but produces no Evidence or limitations.

Possible direction

Follow capture_adb_screen's pattern: fixed argv, exec-out, no device-side file, bounded
output size and timeout. Parse the XML into node rows, keep the raw XML's SHA-256, and map
the known failure strings to typed limitations.

References

Android uiautomator dump usage text (adb shell uiautomator): "creates an XML dump of
current UI hierarchy ... default is /sdcard/window_dump.xml".

Originally created by @RoKenobi on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5033 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Process capture or runtime observation ### Expected interaction with targets Not sure yet (explained under "Target and scope details") ### Verification available so far I have a prototype or observed results ### Current limitation REA's adb tools can report the focused window (`inspect_adb_window`), device and display facts, and a screenshot (`capture_adb_screen`), but there is no tool for the on-screen view hierarchy: which elements are visible, their bounds, resource IDs, classes, text and content descriptions. While studying an installed app's screen layout on a real device, I had to leave REA and run `adb shell uiautomator dump` by hand. Those measurements then had no Evidence record, so a REA-based report mixed cited results with uncited manual output. ### Desired outcome and success criteria A read-only tool (for example `inspect_adb_view_hierarchy`) that, for one device serial: - returns the current hierarchy as Evidence: per node, bounds, resource-id, class, package, text, content-desc and the boolean state attributes, plus the raw XML digest; - writes no file on the device (stream the dump back, like `capture_adb_screen`'s exec-out); - reports the display size/density context needed to convert px to dp, or links to `inspect_adb_display` Evidence; - reports Android's dump failures as typed limitations instead of a generic error, especially `ERROR: could not get idle state.` on screens that never go idle; - states the coverage limit that Jetpack Compose screens expose only nodes with semantics, so an absent node is not evidence that nothing is drawn there. ### Target and scope details - Runs one device-side shell command (`uiautomator dump`). It does not launch, attach to, inject into or change the foreground app; it reads what the accessibility service exposes. - With `/dev/tty` as the output path there is no device-side file. The default command writes `/sdcard/window_dump.xml`, which the tool should avoid. - No interaction (taps, keys). Navigation stays with the caller or existing tools. ### Verification details Observed on a OnePlus CPH2581, Android 16 (SDK 36), adb 28.0.2-debian, REA 6.5.0 (gap also confirmed in source on main @ 23f984b): - `adb exec-out uiautomator dump /dev/tty` returned 18,560 bytes: a complete XML document followed by the plain-text line `UI hierchary dumped to: /dev/tty` (Android's spelling), which a parser must strip. `/sdcard/window_dump.xml` did not exist afterwards. - On an app screen with continuous animation (a media player with a looping video background), `uiautomator dump` failed with `ERROR: could not get idle state.` - On Jetpack Compose screens, the dump contained the outer View frame (app bar, navigation bar, containers) but not the individual Compose elements inside a `ComposeView`. Not yet checked: other Android versions, multi-display devices, `--compressed` output, and behaviour while the screen is locked. ### Alternatives considered - `capture_adb_screen`: gives pixels only; element bounds must be measured by eye. - `inspect_adb_window`: gives only the focused window and activity. - `collect_adb_bugreport`: far broader than needed and does not contain the view tree. - Manual `adb` outside REA: works, but produces no Evidence or limitations. ### Possible direction Follow `capture_adb_screen`'s pattern: fixed argv, exec-out, no device-side file, bounded output size and timeout. Parse the XML into node rows, keep the raw XML's SHA-256, and map the known failure strings to typed limitations. ### References - Android `uiautomator dump` usage text (`adb shell uiautomator`): "creates an XML dump of current UI hierarchy ... default is /sdcard/window_dump.xml".

bsakaguchi commented

2026-10-31 00:36:32 +00:00

Author

Owner

@RoKenobi commented on GitHub (Oct 18, 2026):

Correction to the proposal above, from testing on the same device: adb shell uiautomator dump /dev/tty (and /proc/self/fd/1) returns only the 33-byte status line; the document arrives only through adb exec-out. exec-out also merges the device's stdout and stderr and does not return the device exit status, so the idle-state message has to be recognized from the merged text.

A draft prototype that follows this is in #5049, opened to help with the scope decision here, not to ask for integration before it.

@RoKenobi commented on GitHub (Oct 18, 2026): Correction to the proposal above, from testing on the same device: `adb shell uiautomator dump /dev/tty` (and `/proc/self/fd/1`) returns only the 33-byte status line; the document arrives only through `adb exec-out`. exec-out also merges the device's stdout and stderr and does not return the device exit status, so the idle-state message has to be recognized from the merged text. A draft prototype that follows this is in #5049, opened to help with the scope decision here, not to ask for integration before it.

bsakaguchi commented

2026-10-31 00:36:32 +00:00

Author

Owner

@RoKenobi commented on GitHub (Oct 18, 2026):

Correction to the "Verification details" in the issue text: the statement that Jetpack Compose screens exposed only the outer View frame was wrong. It came from a filter in my own inspection script, not from uiautomator. Re-checked with the prototype in #5049: a Compose app produced 189 nodes, and Compose nodes with semantics appear under each ComposeView (as android.view.View / Button / TextView, without resource IDs). Only content without semantics is absent.

@RoKenobi commented on GitHub (Oct 18, 2026): Correction to the "Verification details" in the issue text: the statement that Jetpack Compose screens exposed only the outer View frame was wrong. It came from a filter in my own inspection script, not from uiautomator. Re-checked with the prototype in #5049: a Compose app produced 189 nodes, and Compose nodes with semantics appear under each `ComposeView` (as `android.view.View` / `Button` / `TextView`, without resource IDs). Only content without semantics is absent.

bsakaguchi referenced this issue

2026-10-31 01:07:57 +00:00

[PR #3092] [CLOSED] fix(hopper): prepare verified thin FAT32 slices for arm64 analysis #5610

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3092

No description provided.

Rows
Columns