morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8104] [Bug] Desktop inspection stops at 4–7k entries because it overestimates row size #4672

New issue

Closed

opened 2026-10-31 00:52:18 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:52:18 +00:00

Owner

Originally created by @kaoru0822-kitauji on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8104

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

On large real macOS applications, inspect-desktop-application stops after 3,700–7,200 entries with coverage.status: "partial" and this limitation:

Directory metadata reached the 32 MiB representation budget; remaining entry and omission facts are unavailable.

That is well below the default max_entries of 20,000. Raising --max_entries to 100,000 changes nothing. Some top-level trees are never descended: in Keynote, Contents/Frameworks/, SharedSupport/ and XPCServices/; in iMovie, Contents/Frameworks/ and Contents/Resources/.

Bundle (installed app)	Entries in bundle	Entries examined (`max_entries` 20,000 or 100,000)	Output JSON
Keynote	37,072	7,109	3.3 MiB
iMovie	58,206	3,727	2.3 MiB
Pages	~36,000	7,246	—
GarageBand	~45,000	6,089	—
ChatGPT (Electron)	5,484	4,896	2.7 MiB

Peak RSS was about 237 MiB for every run, about the same as a 563-entry app (227 MiB).

Reproduction

At origin/main 317bf3fe2, with the runtime built:

node scripts/rea.mjs inspect-desktop-application /Applications/Keynote.app --max_entries 100000 --format json

It reproduced on every run (1–3 runs per app above).

Expected behavior

The limit should follow the real retained cost. Bundles of these sizes should reach max_entries, and the representation budget should stop only inputs whose retained rows really approach 32 MiB.

Impact and scope

Every macOS app bundle above roughly 4,000–7,000 entries is reported as partial before the caller's max_entries. Which entries are skipped depends on directory order. On Keynote and iMovie, that left the framework trees unexamined. Coverage stays honest (partial, with the reason), so no fact is wrong, but the advertised entry limits can't be reached on real bundles.

REA version

6.8.0 (source origin/main 317bf3fe2)

Runtime and platform

Node.js v26.5.0, macOS 26.7.1, arm64

Target and artifact context

macOS .app bundles (Apple iWork/iLife apps and an Electron app), selected as installation directories

Interface

CLI

Earliest failure stage

Capture or observation

Relevant output or evidence

scanDirectory in src/artifacts/DesktopApplicationReader.ts reserves 1024 + (entry.path.length + entry.adapterKey.length) * 24 bytes per entry against RETAINED_METADATA_BYTES = 32 MiB. Omissions and the root use the same 24-bytes-per-character charge. I measured what the retained rows actually cost by loading the compiled reader with the cap lifted, then comparing heap after GC and JSON.stringify of the report:

Bundle	Rows	Retained heap per row	JSON per row	Current charge per row
Keynote	37,072	803 B	513 B	5,146 B
iMovie	58,206	900 B	639 B	8,334 B
VS Code	4,304	941 B	518 B	5,691 B

The charge overstates the retained heap 6–9× and the serialized JSON 10–13×.

Typed outcome or recovery expected

The same partial coverage and limitation, but only when the retained representation really reaches the budget.

Analysis or suspected cause

Source-derived: the per-character multiplier of 24 is far above any single representation of a path. A UTF-16 heap string costs at most 2 bytes per code unit. Escaped UTF-8 JSON costs at most 6 bytes per code unit (control characters), and JSON byte counting already exists in jsonStringByteLength. Charging each path the larger of those two keeps an upper bound on both the retained heap and the serialized result, without the compounding multiplier.

Additional context

Related but distinct: #1801 (an over-conservative per-byte charge in the managed provider). Draft PR #7907 edits the same file for engine-editor detection, but not this estimate.

Originally created by @kaoru0822-kitauji on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8104 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior On large real macOS applications, `inspect-desktop-application` stops after 3,700–7,200 entries with `coverage.status: "partial"` and this limitation: > Directory metadata reached the 32 MiB representation budget; remaining entry and omission facts are unavailable. That is well below the default `max_entries` of 20,000. Raising `--max_entries` to 100,000 changes nothing. Some top-level trees are never descended: in Keynote, `Contents/Frameworks/`, `SharedSupport/` and `XPCServices/`; in iMovie, `Contents/Frameworks/` and `Contents/Resources/`. | Bundle (installed app) | Entries in bundle | Entries examined (`max_entries` 20,000 or 100,000) | Output JSON | | --- | ---: | ---: | ---: | | Keynote | 37,072 | 7,109 | 3.3 MiB | | iMovie | 58,206 | 3,727 | 2.3 MiB | | Pages | ~36,000 | 7,246 | — | | GarageBand | ~45,000 | 6,089 | — | | ChatGPT (Electron) | 5,484 | 4,896 | 2.7 MiB | Peak RSS was about 237 MiB for every run, about the same as a 563-entry app (227 MiB). ### Reproduction At `origin/main` 317bf3fe2, with the runtime built: ```sh node scripts/rea.mjs inspect-desktop-application /Applications/Keynote.app --max_entries 100000 --format json ``` It reproduced on every run (1–3 runs per app above). ### Expected behavior The limit should follow the real retained cost. Bundles of these sizes should reach `max_entries`, and the representation budget should stop only inputs whose retained rows really approach 32 MiB. ### Impact and scope Every macOS app bundle above roughly 4,000–7,000 entries is reported as partial before the caller's `max_entries`. Which entries are skipped depends on directory order. On Keynote and iMovie, that left the framework trees unexamined. Coverage stays honest (`partial`, with the reason), so no fact is wrong, but the advertised entry limits can't be reached on real bundles. ### REA version 6.8.0 (source `origin/main` 317bf3fe2) ### Runtime and platform Node.js v26.5.0, macOS 26.7.1, arm64 ### Target and artifact context macOS `.app` bundles (Apple iWork/iLife apps and an Electron app), selected as installation directories ### Interface CLI ### Earliest failure stage Capture or observation ### Relevant output or evidence `scanDirectory` in `src/artifacts/DesktopApplicationReader.ts` reserves `1024 + (entry.path.length + entry.adapterKey.length) * 24` bytes per entry against `RETAINED_METADATA_BYTES = 32 MiB`. Omissions and the root use the same 24-bytes-per-character charge. I measured what the retained rows actually cost by loading the compiled reader with the cap lifted, then comparing heap after GC and `JSON.stringify` of the report: | Bundle | Rows | Retained heap per row | JSON per row | Current charge per row | | --- | ---: | ---: | ---: | ---: | | Keynote | 37,072 | 803 B | 513 B | 5,146 B | | iMovie | 58,206 | 900 B | 639 B | 8,334 B | | VS Code | 4,304 | 941 B | 518 B | 5,691 B | The charge overstates the retained heap 6–9× and the serialized JSON 10–13×. ### Typed outcome or recovery expected The same `partial` coverage and limitation, but only when the retained representation really reaches the budget. ### Analysis or suspected cause Source-derived: the per-character multiplier of 24 is far above any single representation of a path. A UTF-16 heap string costs at most 2 bytes per code unit. Escaped UTF-8 JSON costs at most 6 bytes per code unit (control characters), and JSON byte counting already exists in `jsonStringByteLength`. Charging each path the larger of those two keeps an upper bound on both the retained heap and the serialized result, without the compounding multiplier. ### Additional context Related but distinct: #1801 (an over-conservative per-byte charge in the managed provider). Draft PR #7907 edits the same file for engine-editor detection, but not this estimate.

bsakaguchi

2026-10-31 00:52:18 +00:00

closed this issue
added the
bug
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4672

No description provided.

Rows
Columns