[PR #991] [MERGED] fix(native)!: preserve incomplete native UI captures #2560

New issue

Closed

opened 2026-10-17 21:07:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:07:36 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/991
Author: @morluto
Created: 10/7/2026
Status: ✅ Merged
Merged: 10/7/2026
Merged by: @morluto

Base: main ← Head: codex/native-ui-correctness-pr

📝 Commits (5)

d83cc97 fix(native): preserve unknown accessibility child counts
1b76cef fix(native): align helper buffer with capture budget
e7c2c76 fix(native): align Swift helper JSON wire size
fcbe25f fix(native): bound raw helper output to scenario budget
f3193d2 Merge remote-tracking branch 'origin/main' into codex/native-ui-correctness-pr

📊 Changes

13 files changed (+178 additions, -21 deletions)

View changed files

📝 bridge/native/NativeUIChildren.swift (+16 -0)
📝 bridge/native/ReaNativeUI.swift (+19 -9)
📝 docs/product-catalog.json (+2 -2)
📝 docs/verification/managed-conformance-manifest.json (+2 -2)
📝 scripts/verify-native-ui.mjs (+33 -1)
📝 skills/reverse-engineer-anything/SKILL.md (+1 -1)
📝 src/domain/native/nativeUiObservation.ts (+1 -1)
📝 src/generatedMcpToolCatalog.ts (+1 -1)
📝 src/native/NativeUiHelperRuntime.ts (+2 -1)
📝 src/native/NativeUiObservation.test.ts (+52 -0)
📝 src/native/NativeUiObservation.ts (+8 -2)
➕ src/native/NativeUiOutputBudget.ts (+15 -0)
📝 tests/conformance/native/native-ui-children/main.swift (+26 -1)

📄 Description

When macOS Accessibility child-count queries fail, native UI capture previously reported zero children. That could make an incomplete tree look like a complete leaf. The helper now reports an unknown count, marks the capture truncated, and includes the exact AX error and path in gaps; action-path validation preserves the AX error as well. A successful count of zero remains a valid leaf.

The helper's raw output buffer is derived from the existing 64 MiB scenario output budget plus the exact JSON response envelope. Swift serialization leaves slashes unescaped to avoid inflating base64 and other slash-rich fields. The application still charges normalized capture output at the existing 2× rate and returns the actionable aggregate-budget failure.

BREAKING CHANGE: nodes[].children_count is now number | null. Existing successful counts retain their values; null means the native API could not determine the count. The CLI/MCP output schemas and generated catalog metadata are updated.

Verification:

npm run test:focused -- src/native/NativeUiObservation.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts
npm run check:fast
npm run docs:check
npm run format:check
npm run verify:native-ui (macOS source-owned fixture: capture, selected action, CLI parity, target-digest mismatch)
Paired source-owned 2048×2048 high-entropy screenshot: old 16 MiB helper limit fails; the budget-derived helper limit succeeds. Foundation slash-rich serialization and numeric-format behavior are checked against the JavaScript consumer representation.

Real macOS native UI verification was performed. Windows native execution was not performed. No speedup claim is made.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/991 **Author:** [@morluto](https://github.com/morluto) **Created:** 10/7/2026 **Status:** ✅ Merged **Merged:** 10/7/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `codex/native-ui-correctness-pr` --- ### 📝 Commits (5) - [`d83cc97`](https://github.com/morluto/rea/commit/d83cc9779d3dba1dc93bbd9f974815307db6b0b2) fix(native): preserve unknown accessibility child counts - [`1b76cef`](https://github.com/morluto/rea/commit/1b76cef917238bf3d0b721df5911cdcd1f1ab708) fix(native): align helper buffer with capture budget - [`e7c2c76`](https://github.com/morluto/rea/commit/e7c2c76a9cd8ac7ace6ab252f65cd0e2fcae339c) fix(native): align Swift helper JSON wire size - [`fcbe25f`](https://github.com/morluto/rea/commit/fcbe25f0fda0d1168c8cd74263e88370ea9dff45) fix(native): bound raw helper output to scenario budget - [`f3193d2`](https://github.com/morluto/rea/commit/f3193d27f59457202477519c85203e81660401e3) Merge remote-tracking branch 'origin/main' into codex/native-ui-correctness-pr ### 📊 Changes **13 files changed** (+178 additions, -21 deletions) <details> <summary>View changed files</summary> 📝 `bridge/native/NativeUIChildren.swift` (+16 -0) 📝 `bridge/native/ReaNativeUI.swift` (+19 -9) 📝 `docs/product-catalog.json` (+2 -2) 📝 `docs/verification/managed-conformance-manifest.json` (+2 -2) 📝 `scripts/verify-native-ui.mjs` (+33 -1) 📝 `skills/reverse-engineer-anything/SKILL.md` (+1 -1) 📝 `src/domain/native/nativeUiObservation.ts` (+1 -1) 📝 `src/generatedMcpToolCatalog.ts` (+1 -1) 📝 `src/native/NativeUiHelperRuntime.ts` (+2 -1) 📝 `src/native/NativeUiObservation.test.ts` (+52 -0) 📝 `src/native/NativeUiObservation.ts` (+8 -2) ➕ `src/native/NativeUiOutputBudget.ts` (+15 -0) 📝 `tests/conformance/native/native-ui-children/main.swift` (+26 -1) </details> ### 📄 Description When macOS Accessibility child-count queries fail, native UI capture previously reported zero children. That could make an incomplete tree look like a complete leaf. The helper now reports an unknown count, marks the capture truncated, and includes the exact AX error and path in `gaps`; action-path validation preserves the AX error as well. A successful count of zero remains a valid leaf. The helper's raw output buffer is derived from the existing 64 MiB scenario output budget plus the exact JSON response envelope. Swift serialization leaves slashes unescaped to avoid inflating base64 and other slash-rich fields. The application still charges normalized capture output at the existing 2× rate and returns the actionable aggregate-budget failure. BREAKING CHANGE: `nodes[].children_count` is now `number | null`. Existing successful counts retain their values; `null` means the native API could not determine the count. The CLI/MCP output schemas and generated catalog metadata are updated. Verification: - `npm run test:focused -- src/native/NativeUiObservation.test.ts tests/boundary/mcp/toolSchemaValidity.test.ts` - `npm run check:fast` - `npm run docs:check` - `npm run format:check` - `npm run verify:native-ui` (macOS source-owned fixture: capture, selected action, CLI parity, target-digest mismatch) - Paired source-owned 2048×2048 high-entropy screenshot: old 16 MiB helper limit fails; the budget-derived helper limit succeeds. Foundation slash-rich serialization and numeric-format behavior are checked against the JavaScript consumer representation. Real macOS native UI verification was performed. Windows native execution was not performed. No speedup claim is made. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:07:36 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#2560

No description provided.

Rows
Columns