morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5306] [Bug] observe_native_ui treats a wrong PID or window as provider unavailability and withdraws the tool in binary_session #3233

New issue

Open

opened 2026-10-31 00:38:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 00:38:08 +00:00

Owner

Originally created by @cathrynlavery on GitHub (Oct 19, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5306

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

When the native UI helper cannot match the caller's pid or window_id, observe_native_ui and capture_native_ui_scenario fail with capability_unavailable / unsupported_provider and the message "This analysis is unavailable for the current target. Choose another analysis or target." The reason the helper gave is only in details.reason.

The session also records the failure as runtime unavailability. After one call with a wrong PID, binary_session lists the operation as available: false, reason: "provider_unavailable" until a later call to the same operation succeeds. Each operation is marked separately, so observe_native_ui can be advertised as unavailable while capture_native_ui_scenario is still advertised as available.

Reproduction

Ran on main at b4938aed5 (built with npm run build:cached), driving node scripts/rea.mjs mcp over stdio with @modelcontextprotocol/client. Step 5 against the published 6.4.0 package returns the same error. Each case below reproduced every time I ran it.

open -a Calculator. Read its PID and window ID from CGWindowListCopyWindowInfo (owner Calculator, layer 0).
sleep 300 & to get a PID that is not an application.
open_binary {"path": "/System/Applications/Calculator.app"}
observe_native_ui {"pid": <Calculator PID>, "window_id": <window ID>, "screenshot": false, "max_nodes": 5} succeeds.
observe_native_ui with the sleep PID and the same window ID fails (output below).
binary_session {} now reports the observe_native_ui group as unavailable.

The same thing happens with Calculator's PID and window_id: 1 (window-mismatch), with TextEdit's PID and Calculator's window (target-mismatch: Selected PID executable /System/Applications/TextEdit.app/Contents/MacOS/TextEdit differs from active target /System/Applications/Calculator.app/Contents/MacOS/Calculator), and with capture_native_ui_scenario and a single wait step.

Expected behavior

A PID or window ID that does not match the active target is a problem with the call's arguments. I expected invalid_input that names pid or window_id and carries the helper's reason, with advertised availability left alone. For host failures such as Accessibility or Screen Recording denial, I expected the helper's own remediation ("Grant Accessibility to the REA host in System Settings > Privacy & Security, then retry") as the message. docs/native-investigation.md says missing permissions "produce actionable errors".

Impact and scope

An agent that checks binary_session after one wrong PID is told the operation is unavailable on this host, and the error text tells it to choose another target or analysis, when the fix is a different PID or window ID. Every wrong PID or window I tried was affected. I did not test the element-path failures inside scenario steps.

REA version

main b4938aed58 (rea --version 6.7.0); also rea-agents 6.4.0 via npx -y rea-agents@6.4.0 mcp

Runtime and platform

Node 24.18.0; macOS 27.0 (26A428) arm64; swiftc from Xcode command-line tools

Target and artifact context

/System/Applications/Calculator.app (universal Mach-O, arm64 selected)

Interface

MCP

Earliest failure stage

Capture or observation

Relevant output or evidence

Step 5, main:

{"error":{"code":"capability_unavailable","category":"unsupported_provider","message":"This analysis is unavailable for the current target. Choose another analysis or target.","retryable":false,"remediation":{"action":"This analysis is unavailable for the current target. Choose another analysis or target."},"details":{"provider_id":"native-macos","operation":"observe_native_ui","reason":"target-mismatch: Selected PID does not expose an application executable URL"}}}

Step 6, tool_availability.groups entries for the native UI tools before and after step 5:

{"names":["observe_native_ui"],"available":true,"reason":"available","remediation":null}
{"names":["observe_native_ui"],"available":false,"reason":"provider_unavailable","remediation":"Provider native-macos cannot execute observe_native_ui: target-mismatch: Selected PID does not expose an application executable URL"}

Analysis or suspected cause

Source-derived, at b4938aed5:

bridge/native/ReaNativeUI.swift#L46-L48: one guard covers NSRunningApplication(processIdentifier:) returning nil and a nil executableURL, and both produce target-mismatch: Selected PID does not expose an application executable URL. The same code is used for a PID that runs another executable (L53) and for changed executable bytes (L63).
src/native/NativeUiObservation.ts#L174-L181: every {ok: false} helper reply becomes AnalysisCapabilityUnavailableError without a userMessage, so presentation falls back to the generic unsupported-provider text (analysisErrorPresentation.ts#L373-L374).
src/application/binary/BinarySession.ts#L811-L815: any AnalysisCapabilityUnavailableError marks that operation unavailable in session status.

Permissions do not explain the PID message. The PID check runs before the Screen Recording and Accessibility checks (L73-L78), and NSRunningApplication needs neither. A small Swift probe run over SSH on the same Mac, where CGPreflightScreenCaptureAccess() and AXIsProcessTrusted() both returned false, still got Calculator's NSRunningApplication with its executable URL and launch date. Path canonicalization, App Translocation, and universal versus thin paths would fail later with "differs from active target", and universal Calculator passes.

Additional context

I first saw this message with a Developer ID-signed third-party app in /Applications that was launched with open -a. Its PID and window ID came from CGWindowListCopyWindowInfo (owner name matched, layer 0). I have not found out why NSRunningApplication returned nil or a nil executableURL for that process: NOT_ENOUGH_INFO. The current message does not say which of the two happened or whether the PID was still running, which is why I could not tell.

Proposed fix, with a PR to follow:

Give the helper separate codes for "no LaunchServices application for this PID" (including whether the PID is running and its proc_pidpath), "application has no executable URL", and "PID runs another executable".
When the first capture cannot resolve the caller's PID or window, return AnalysisInputError with an issue on pid or window_id. A window or process that changes after a successful capture inside a scenario stays a capability failure, since the arguments were valid.
Pass the helper's message as userMessage for the remaining helper failures, so permission denials show their remediation.

Originally created by @cathrynlavery on GitHub (Oct 19, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5306 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior When the native UI helper cannot match the caller's `pid` or `window_id`, `observe_native_ui` and `capture_native_ui_scenario` fail with `capability_unavailable` / `unsupported_provider` and the message "This analysis is unavailable for the current target. Choose another analysis or target." The reason the helper gave is only in `details.reason`. The session also records the failure as runtime unavailability. After one call with a wrong PID, `binary_session` lists the operation as `available: false, reason: "provider_unavailable"` until a later call to the same operation succeeds. Each operation is marked separately, so `observe_native_ui` can be advertised as unavailable while `capture_native_ui_scenario` is still advertised as available. ### Reproduction Ran on main at b4938aed5 (built with `npm run build:cached`), driving `node scripts/rea.mjs mcp` over stdio with `@modelcontextprotocol/client`. Step 5 against the published 6.4.0 package returns the same error. Each case below reproduced every time I ran it. 1. `open -a Calculator`. Read its PID and window ID from `CGWindowListCopyWindowInfo` (owner Calculator, layer 0). 2. `sleep 300 &` to get a PID that is not an application. 3. `open_binary {"path": "/System/Applications/Calculator.app"}` 4. `observe_native_ui {"pid": <Calculator PID>, "window_id": <window ID>, "screenshot": false, "max_nodes": 5}` succeeds. 5. `observe_native_ui` with the `sleep` PID and the same window ID fails (output below). 6. `binary_session {}` now reports the `observe_native_ui` group as unavailable. The same thing happens with Calculator's PID and `window_id: 1` (`window-mismatch`), with TextEdit's PID and Calculator's window (`target-mismatch: Selected PID executable /System/Applications/TextEdit.app/Contents/MacOS/TextEdit differs from active target /System/Applications/Calculator.app/Contents/MacOS/Calculator`), and with `capture_native_ui_scenario` and a single wait step. ### Expected behavior A PID or window ID that does not match the active target is a problem with the call's arguments. I expected `invalid_input` that names `pid` or `window_id` and carries the helper's reason, with advertised availability left alone. For host failures such as Accessibility or Screen Recording denial, I expected the helper's own remediation ("Grant Accessibility to the REA host in System Settings > Privacy & Security, then retry") as the message. [docs/native-investigation.md](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/docs/native-investigation.md#native-desktop-observation) says missing permissions "produce actionable errors". ### Impact and scope An agent that checks `binary_session` after one wrong PID is told the operation is unavailable on this host, and the error text tells it to choose another target or analysis, when the fix is a different PID or window ID. Every wrong PID or window I tried was affected. I did not test the element-path failures inside scenario steps. ### REA version main b4938aed58737293a6987630476632619713f5cd (`rea --version` 6.7.0); also rea-agents 6.4.0 via `npx -y rea-agents@6.4.0 mcp` ### Runtime and platform Node 24.18.0; macOS 27.0 (26A428) arm64; swiftc from Xcode command-line tools ### Target and artifact context /System/Applications/Calculator.app (universal Mach-O, arm64 selected) ### Interface MCP ### Earliest failure stage Capture or observation ### Relevant output or evidence Step 5, main: ```json {"error":{"code":"capability_unavailable","category":"unsupported_provider","message":"This analysis is unavailable for the current target. Choose another analysis or target.","retryable":false,"remediation":{"action":"This analysis is unavailable for the current target. Choose another analysis or target."},"details":{"provider_id":"native-macos","operation":"observe_native_ui","reason":"target-mismatch: Selected PID does not expose an application executable URL"}}} ``` Step 6, `tool_availability.groups` entries for the native UI tools before and after step 5: ```json {"names":["observe_native_ui"],"available":true,"reason":"available","remediation":null} {"names":["observe_native_ui"],"available":false,"reason":"provider_unavailable","remediation":"Provider native-macos cannot execute observe_native_ui: target-mismatch: Selected PID does not expose an application executable URL"} ``` ### Analysis or suspected cause Source-derived, at b4938aed5: - [`bridge/native/ReaNativeUI.swift#L46-L48`](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/bridge/native/ReaNativeUI.swift#L46-L48): one guard covers `NSRunningApplication(processIdentifier:)` returning nil and a nil `executableURL`, and both produce `target-mismatch: Selected PID does not expose an application executable URL`. The same code is used for a PID that runs another executable ([L53](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/bridge/native/ReaNativeUI.swift#L53)) and for changed executable bytes ([L63](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/bridge/native/ReaNativeUI.swift#L63)). - [`src/native/NativeUiObservation.ts#L174-L181`](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/src/native/NativeUiObservation.ts#L174-L181): every `{ok: false}` helper reply becomes `AnalysisCapabilityUnavailableError` without a `userMessage`, so presentation falls back to the generic unsupported-provider text ([`analysisErrorPresentation.ts#L373-L374`](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/src/domain/analysisErrorPresentation.ts#L373-L374)). - [`src/application/binary/BinarySession.ts#L811-L815`](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/src/application/binary/BinarySession.ts#L811-L815): any `AnalysisCapabilityUnavailableError` marks that operation unavailable in session status. Permissions do not explain the PID message. The PID check runs before the Screen Recording and Accessibility checks ([L73-L78](https://github.com/morluto/rea/blob/b4938aed58737293a6987630476632619713f5cd/bridge/native/ReaNativeUI.swift#L73-L78)), and `NSRunningApplication` needs neither. A small Swift probe run over SSH on the same Mac, where `CGPreflightScreenCaptureAccess()` and `AXIsProcessTrusted()` both returned false, still got Calculator's `NSRunningApplication` with its executable URL and launch date. Path canonicalization, App Translocation, and universal versus thin paths would fail later with "differs from active target", and universal Calculator passes. ### Additional context I first saw this message with a Developer ID-signed third-party app in /Applications that was launched with `open -a`. Its PID and window ID came from `CGWindowListCopyWindowInfo` (owner name matched, layer 0). I have not found out why `NSRunningApplication` returned nil or a nil `executableURL` for that process: NOT_ENOUGH_INFO. The current message does not say which of the two happened or whether the PID was still running, which is why I could not tell. Proposed fix, with a PR to follow: 1. Give the helper separate codes for "no LaunchServices application for this PID" (including whether the PID is running and its `proc_pidpath`), "application has no executable URL", and "PID runs another executable". 2. When the first capture cannot resolve the caller's PID or window, return `AnalysisInputError` with an issue on `pid` or `window_id`. A window or process that changes after a successful capture inside a scenario stays a capability failure, since the arguments were valid. 3. Pass the helper's message as `userMessage` for the remaining helper failures, so permission denials show their remediation.

bsakaguchi commented

2026-10-31 00:38:09 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 20, 2026):

This classification matters for agent behavior: a wrong PID/window is retryable within the same target (fix the arguments), while provider_unavailable tells the agent to give up or switch targets. Conflating the two makes an agent burn a session thinking the host lacks the capability. The #5307 direction of surfacing an invalid_input with the helper's reason (and reserving capability_unavailable for actual host-permission failures like Accessibility/Screen Recording denial) matches the expected behavior described here and keeps binary_session availability accurate.

@XinwangTao commented on GitHub (Oct 20, 2026): This classification matters for agent behavior: a wrong PID/window is retryable within the same target (fix the arguments), while provider_unavailable tells the agent to give up or switch targets. Conflating the two makes an agent burn a session thinking the host lacks the capability. The #5307 direction of surfacing an invalid_input with the helper's reason (and reserving capability_unavailable for actual host-permission failures like Accessibility/Screen Recording denial) matches the expected behavior described here and keeps binary_session availability accurate.

bsakaguchi referenced this issue

2026-10-31 01:08:24 +00:00

[PR #3233] [CLOSED] fix(cli): advertise the outline-depth minimum the parser enforces #5662

bsakaguchi commented

2026-11-03 01:12:17 +00:00

Author

Owner

@chordpli commented on GitHub (Nov 3, 2026):

Thanks for the detailed write-up. It made this easy to re-check.

I re-ran your steps on current main (9bea563e3), and the behavior you reported no longer reproduces.

Setup: macOS 26.5.2 (arm64), Node 24.22.1, built with npm run build:cached, driving node scripts/rea.mjs mcp over stdio with @modelcontextprotocol/client. I opened Calculator with open -a Calculator and took its PID and window ID from CGWindowListCopyWindowInfo.

What I saw:

Step	Call	Result
4	`observe_native_ui` with Calculator's PID and window	Success
5	`sleep 300` PID	`invalid_request`, issue on `pid` (`pid-not-application`)
5	Calculator's PID with `window_id: 1`	`invalid_request`, issue on `window_id` (`window-mismatch: Selected window is absent or belongs to another process`)
5	TextEdit's PID with Calculator's window	`invalid_request`, issue on `pid` (`target-mismatch: Selected PID executable .../TextEdit differs from active target .../Calculator`)
5	`capture_native_ui_scenario` with the `sleep` PID and one `wait` step	`invalid_request`, issue on `pid`
6	`binary_session` after each failure	`observe_native_ui` and `capture_native_ui_scenario` both stay `available: true`, `reason: "available"`

Retrying with the correct PID and window on the same connection succeeded.

One extra data point, by accident: on my first run, Calculator had already quit before the calls went out. Its stale PID came back as invalid_request on pid with pid-not-application, and availability stayed untouched. That lines up with the helper's wording that this code "does not establish that the process is absent."

Source only, not verified at runtime: other helper failures, including accessibility-denied and screen-recording-denied, now pass the helper's message through as userMessage (NativeUiObservation.ts L205-L210), and the helper now separates pid-not-application from application-executable-unavailable (ReaNativeUI.swift L46-L51). Accessibility is already granted on this machine, so I didn't hit a denial. The helper also doesn't report whether the PID is still running, which was part of your first proposed fix. These changes landed in cc90771af (2026-10-24).

As far as I can tell, the issue as reported is resolved on main. The permission-denial message might still be worth a quick check on a host without Accessibility granted.

@chordpli commented on GitHub (Nov 3, 2026): Thanks for the detailed write-up. It made this easy to re-check. I re-ran your steps on current `main` (`9bea563e3`), and the behavior you reported no longer reproduces. **Setup:** macOS 26.5.2 (arm64), Node 24.22.1, built with `npm run build:cached`, driving `node scripts/rea.mjs mcp` over stdio with `@modelcontextprotocol/client`. I opened Calculator with `open -a Calculator` and took its PID and window ID from `CGWindowListCopyWindowInfo`. **What I saw:** | Step | Call | Result | |---|---|---| | 4 | `observe_native_ui` with Calculator's PID and window | Success | | 5 | `sleep 300` PID | `invalid_request`, issue on `pid` (`pid-not-application`) | | 5 | Calculator's PID with `window_id: 1` | `invalid_request`, issue on `window_id` (`window-mismatch: Selected window is absent or belongs to another process`) | | 5 | TextEdit's PID with Calculator's window | `invalid_request`, issue on `pid` (`target-mismatch: Selected PID executable .../TextEdit differs from active target .../Calculator`) | | 5 | `capture_native_ui_scenario` with the `sleep` PID and one `wait` step | `invalid_request`, issue on `pid` | | 6 | `binary_session` after each failure | `observe_native_ui` and `capture_native_ui_scenario` both stay `available: true`, `reason: "available"` | Retrying with the correct PID and window on the same connection succeeded. One extra data point, by accident: on my first run, Calculator had already quit before the calls went out. Its stale PID came back as `invalid_request` on `pid` with `pid-not-application`, and availability stayed untouched. That lines up with the helper's wording that this code "does not establish that the process is absent." **Source only, not verified at runtime:** other helper failures, including `accessibility-denied` and `screen-recording-denied`, now pass the helper's message through as `userMessage` ([`NativeUiObservation.ts` L205-L210](https://github.com/morluto/rea/blob/9bea563e30fd255d639d6877a91109ba9ff2eb00/src/native/NativeUiObservation.ts#L205-L210)), and the helper now separates `pid-not-application` from `application-executable-unavailable` ([`ReaNativeUI.swift` L46-L51](https://github.com/morluto/rea/blob/9bea563e30fd255d639d6877a91109ba9ff2eb00/bridge/native/ReaNativeUI.swift#L46-L51)). Accessibility is already granted on this machine, so I didn't hit a denial. The helper also doesn't report whether the PID is still running, which was part of your first proposed fix. These changes landed in cc90771af (2026-10-24). As far as I can tell, the issue as reported is resolved on `main`. The permission-denial message might still be worth a quick check on a host without Accessibility granted.

bsakaguchi commented

2026-11-04 04:56:15 +00:00

Author

Owner

@XinwangTao commented on GitHub (Nov 4, 2026):

Verified results look solid — returning invalid_request with an issue on pid/window_id while leaving availability untouched is exactly the classification this issue asked for, and it makes agent retry behavior correct instead of burning a session. Agreed the permission-denial path deserves a runtime check on a host without Accessibility/Screen Recording granted. One small gap from the original proposal remains: the helper still doesn't report whether the PID is running (the stale-PID case was only caught by accident), so a stale PID could be misread as a different problem. Adding a pid-running / proc_pidpath detail to the reason would close that. Happy to help verify on a non-granted host if that's useful.

@XinwangTao commented on GitHub (Nov 4, 2026): Verified results look solid — returning invalid_request with an issue on pid/window_id while leaving availability untouched is exactly the classification this issue asked for, and it makes agent retry behavior correct instead of burning a session. Agreed the permission-denial path deserves a runtime check on a host without Accessibility/Screen Recording granted. One small gap from the original proposal remains: the helper still doesn't report whether the PID is running (the stale-PID case was only caught by accident), so a stale PID could be misread as a different problem. Adding a pid-running / proc_pidpath detail to the reason would close that. Happy to help verify on a non-granted host if that's useful.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3233

No description provided.

Rows
Columns