[GH-ISSUE #2483] [Design] macOS: every REA process recompiles the ownership helper (~380 ms, 34–47% of short CLI commands) #826

Closed
opened 2026-10-15 18:28:48 +00:00 by bsakaguchi · 2 comments
Owner

Originally created by @kaoru0822-kitauji on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2483

Originally assigned to: @kaoru0822-kitauji on GitHub.

Summary

On macOS, the first owned command in every REA process compiles bridge/process/ProcessRunTokenReader.swift and ProcessRunTokenReaderMain.swift with xcrun swiftc into a fresh temporary root (createDarwinProcessRunTokenReader in src/process/DarwinProcessRunTokenReader.ts). The compiled reader is deleted when the process exits. The MCP server pays this once per server. The CLI pays it on every command, and on short static commands it is the largest single cost.

Changing this means persisting or shipping an executable that ownership decisions rely on, which is an architecture and authority decision. So I'm opening an issue with data instead of a PR, as CONTRIBUTING asks.

Measurements

Main at a6efa4fce, macOS 26 (Apple silicon), Xcode Command Line Tools (no full Xcode), Node 26.5.0, local build, idle machine. All values are medians of 3 runs. The Swift module cache was warm, as in normal use. #1137 measured an 18 s fully cold compile.

Wall time
rea inspect-macho /bin/ls --format json 1,124 ms
rea inspect-signature /bin/ls --format json 880 ms
rea list-architectures /bin/ls --format json 801 ms
xcrun swiftc of the two reader sources alone 378 ms

So the helper compile is 34% / 43% / 47% of those three commands.

I logged the spawn trace for one rea inspect-macho /bin/ls with a ChildProcess.prototype.spawn hook:

Process Count Each
xcrun swiftc …ProcessRunTokenReader*.swift 1 376 ms
ps -axo pid=,ppid=,pgid=,uid=,stat=,command= 15 15–16 ms
xcrun --find <tool> 8 5 ms
the analysis tools themselves (otool, nm, dyld_info ×2, lipo, file, …) 8 2–5 ms

The ownership ps sweeps (about 230 ms here) are a second, smaller cost: roughly one sweep per supervised launch, each 3–7× the duration of the command it supervises.

In an MCP server, the compile shows up only on the first native call. inspect_macho on /bin/ls takes about 755 ms cold versus 350 ms warm, and in-process timing of the first resolveXcrunTool call is 398 ms versus 21 ms.

  • #1137 / #1139: an 18 s fully cold compile exhausted deadlines. That was fixed by deadline handling, not by reuse.
  • #2184: the compile deadline under load. A commenter there suggested a content-hashed cache or precompilation, without data.

Options (for maintainers to choose)

  1. Per-user, content-addressed cache. Key it on the source digest, the compiler and SDK identity, and the architecture, under a REA-owned 0700 directory. Before every exec, verify owner, mode and non-symlink. Publish with an atomic rename, and recompile on any mismatch. This removes the compile from every process after the first. Open questions: the trust given to a persisted executable that a same-user process could replace (it decides ownership before signals and cleanup), cache lifecycle under rea setup / rea uninstall, and recording which helper build produced the ownership facts.
  2. Compile once at rea setup (or doctor) into a REA-owned location, using the same verification as option 1 and falling back to the per-process compile. This makes setup the explicit owner of the artifact. It still needs the same integrity checks.
  3. Keep the per-process compile and reduce the ps sweeps. For example, read the process table through the already-compiled sysctl helper, or sweep once per batch of commands. This doesn't help the compile, but it removes about 15 ms × N launches from every operation, CLI and MCP alike.

Committing a prebuilt binary isn't an option (binaries must not be committed).

Not measured

  • A loaded machine (#2184) or a fully cold Swift module cache (#1137). Both make the compile much slower than here.
  • Full Xcode installs.
  • Linux and Windows ownership paths, which don't compile this helper.

AI agent note

An AI agent (Claude Code) collected these measurements and wrote this issue during a scheduled maintenance loop. It made no code changes for this issue.

Originally created by @kaoru0822-kitauji on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2483 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Summary On macOS, the first owned command in every REA process compiles `bridge/process/ProcessRunTokenReader.swift` and `ProcessRunTokenReaderMain.swift` with `xcrun swiftc` into a fresh temporary root (`createDarwinProcessRunTokenReader` in `src/process/DarwinProcessRunTokenReader.ts`). The compiled reader is deleted when the process exits. The MCP server pays this once per server. The CLI pays it on every command, and on short static commands it is the largest single cost. Changing this means persisting or shipping an executable that ownership decisions rely on, which is an architecture and authority decision. So I'm opening an issue with data instead of a PR, as CONTRIBUTING asks. ### Measurements Main at a6efa4fce, macOS 26 (Apple silicon), Xcode Command Line Tools (no full Xcode), Node 26.5.0, local build, idle machine. All values are medians of 3 runs. The Swift module cache was warm, as in normal use. #1137 measured an 18 s fully cold compile. | | Wall time | | --- | ---: | | `rea inspect-macho /bin/ls --format json` | 1,124 ms | | `rea inspect-signature /bin/ls --format json` | 880 ms | | `rea list-architectures /bin/ls --format json` | 801 ms | | `xcrun swiftc` of the two reader sources alone | 378 ms | So the helper compile is 34% / 43% / 47% of those three commands. I logged the spawn trace for one `rea inspect-macho /bin/ls` with a `ChildProcess.prototype.spawn` hook: | Process | Count | Each | | --- | ---: | ---: | | `xcrun swiftc …ProcessRunTokenReader*.swift` | 1 | 376 ms | | `ps -axo pid=,ppid=,pgid=,uid=,stat=,command=` | 15 | 15–16 ms | | `xcrun --find <tool>` | 8 | 5 ms | | the analysis tools themselves (`otool`, `nm`, `dyld_info` ×2, `lipo`, `file`, …) | 8 | 2–5 ms | The ownership `ps` sweeps (about 230 ms here) are a second, smaller cost: roughly one sweep per supervised launch, each 3–7× the duration of the command it supervises. In an MCP server, the compile shows up only on the first native call. `inspect_macho` on `/bin/ls` takes about 755 ms cold versus 350 ms warm, and in-process timing of the first `resolveXcrunTool` call is 398 ms versus 21 ms. ### Related - #1137 / #1139: an 18 s fully cold compile exhausted deadlines. That was fixed by deadline handling, not by reuse. - #2184: the compile deadline under load. A commenter there suggested a content-hashed cache or precompilation, without data. ### Options (for maintainers to choose) 1. **Per-user, content-addressed cache.** Key it on the source digest, the compiler and SDK identity, and the architecture, under a REA-owned 0700 directory. Before every exec, verify owner, mode and non-symlink. Publish with an atomic rename, and recompile on any mismatch. This removes the compile from every process after the first. Open questions: the trust given to a persisted executable that a same-user process could replace (it decides ownership before signals and cleanup), cache lifecycle under `rea setup` / `rea uninstall`, and recording which helper build produced the ownership facts. 2. **Compile once at `rea setup` (or `doctor`)** into a REA-owned location, using the same verification as option 1 and falling back to the per-process compile. This makes setup the explicit owner of the artifact. It still needs the same integrity checks. 3. **Keep the per-process compile and reduce the `ps` sweeps.** For example, read the process table through the already-compiled sysctl helper, or sweep once per batch of commands. This doesn't help the compile, but it removes about 15 ms × N launches from every operation, CLI and MCP alike. Committing a prebuilt binary isn't an option (binaries must not be committed). ### Not measured - A loaded machine (#2184) or a fully cold Swift module cache (#1137). Both make the compile much slower than here. - Full Xcode installs. - Linux and Windows ownership paths, which don't compile this helper. ### AI agent note An AI agent (Claude Code) collected these measurements and wrote this issue during a scheduled maintenance loop. It made no code changes for this issue.
Author
Owner

@kaoru0822-kitauji commented on GitHub (Oct 25, 2026):

6.8.0 data point: this cost grew in 6.8.0. Published rea-agents@6.8.0 vs 6.7.0, macOS 26.7.1 arm64, Command Line Tools, Node 24.22.1. Medians of 5 alternating runs. The host was not idle (another agent session was running jobs), so absolute numbers are noisy, but both versions were interleaved.

Command (Calculator.app unless noted) 6.7.0 6.8.0 Change
inspect-plist 0.92 s 1.39 s +51%
inspect-signature 1.00 s 1.51 s +51%
inspect-macho 1.11 s 1.57 s +41%
inspect-sqlite-database (53 KB DB) 0.80 s 1.25 s +56%
inspect-go-binary (no native command) 0.38 s 0.43 s +13%

Where the extra time goes (child-process trace from a ChildProcess.prototype.spawn preload, inspect-plist):

6.7.0: xcrun swiftc …/ProcessRunTokenReader*.swift -o $TMPDIR/rea-process-token-*/reader   447 ms
       xcrun --find file; ps -axo …; file -b …; ps -axo …; xcrun --find plutil; ps -axo …; plutil …; ps -axo …
6.8.0: xcrun swiftc … (same)                                                               469 ms
       xcrun --find file; ps -axo …; reader --identities 1 323 324 …                       317–324 ms  ← new
       file -b …; ps -axo …; reader --identities …                                          6 ms
       (every later ps snapshot is followed by reader --identities, ~6 ms each)

6.7.0 compiles the reader but never executes it on this path. 6.8.0 now runs reader --identities <all ps PIDs> after every ps snapshot, and the first exec of the freshly compiled, unsigned binary costs about 320 ms. That's consistent with the first-launch policy check on a new executable. Later execs cost about 6 ms. A CPU profile shows matching idle time: 914 ms vs 596 ms.

So short native-command CLI calls now pay about 0.5 s (compile) plus 0.32 s (first exec), on every invocation. MCP pays it once per server. Persisting or reusing the reader across invocations would remove both costs.

<!-- gh-comment-id:6396240766 --> @kaoru0822-kitauji commented on GitHub (Oct 25, 2026): 6.8.0 data point: this cost grew in 6.8.0. Published `rea-agents@6.8.0` vs `6.7.0`, macOS 26.7.1 arm64, Command Line Tools, Node 24.22.1. Medians of 5 alternating runs. The host was not idle (another agent session was running jobs), so absolute numbers are noisy, but both versions were interleaved. | Command (Calculator.app unless noted) | 6.7.0 | 6.8.0 | Change | |---|---|---|---| | `inspect-plist` | 0.92 s | 1.39 s | +51% | | `inspect-signature` | 1.00 s | 1.51 s | +51% | | `inspect-macho` | 1.11 s | 1.57 s | +41% | | `inspect-sqlite-database` (53 KB DB) | 0.80 s | 1.25 s | +56% | | `inspect-go-binary` (no native command) | 0.38 s | 0.43 s | +13% | Where the extra time goes (child-process trace from a `ChildProcess.prototype.spawn` preload, `inspect-plist`): ``` 6.7.0: xcrun swiftc …/ProcessRunTokenReader*.swift -o $TMPDIR/rea-process-token-*/reader 447 ms xcrun --find file; ps -axo …; file -b …; ps -axo …; xcrun --find plutil; ps -axo …; plutil …; ps -axo … 6.8.0: xcrun swiftc … (same) 469 ms xcrun --find file; ps -axo …; reader --identities 1 323 324 … 317–324 ms ← new file -b …; ps -axo …; reader --identities … 6 ms (every later ps snapshot is followed by reader --identities, ~6 ms each) ``` 6.7.0 compiles the reader but never executes it on this path. 6.8.0 now runs `reader --identities <all ps PIDs>` after every `ps` snapshot, and the first exec of the freshly compiled, unsigned binary costs about 320 ms. That's consistent with the first-launch policy check on a new executable. Later execs cost about 6 ms. A CPU profile shows matching idle time: 914 ms vs 596 ms. So short native-command CLI calls now pay about 0.5 s (compile) plus 0.32 s (first exec), on every invocation. MCP pays it once per server. Persisting or reusing the reader across invocations would remove both costs.
bsakaguchi 2026-11-01 21:10:52 +00:00
Author
Owner

@kaoru0822-kitauji commented on GitHub (Oct 31, 2026):

Re-measured on main 23de10bb2 (macOS 26 arm64, Command Line Tools, Node 24.22.1). Child-process trace of one rea inspect-macho /bin/ls --json, total 1,427 ms:

+408 ms  xcrun swiftc …/ProcessRunTokenReader.swift …ProcessRunTokenReaderMain.swift -o $TMP/rea-process-token-*/reader
+325 ms  $TMP/rea-process-token-*/reader --identities …   ← first exec of the fresh binary
  +4–7 ms each for the 11 later reader execs; ps sweeps 14–28 ms each

Compile plus first exec is 733 ms, or 51% of the command. I'm taking this on with option 1 and a deliberately narrow trust story:

Proposal: a per-user, content-addressed helper cache, with fail-closed verification and the current compile as the fallback.

  • Location: ~/Library/Caches/rea/process-helper/<key>/. HOME comes from the launch environment, next to the existing ~/Library/Caches/rea/kuna. The process ownership host turns it on in production. Readers created directly (tests) stay uncached unless they pass a directory.
  • Key: sha256 over a helper-format version, the bytes of both Swift sources, the compiler selection (xcrun, or REA_DARWIN_SWIFTC_PATH + SDKROOT), process.arch and os.release(). An OS update or a helper source change gives a new key. No extra compiler subprocess is spawned to build the key.
  • Publish: compile into a fresh mkdtemp staging directory inside the cache root, so the rename stays on one filesystem. Write a manifest with the binary's sha256 and the key inputs, then atomically rename the staging directory to <key>. If another process published first, verify and use its entry, then remove our staging directory.
  • Use (once per process, before the first exec): lstat the cache root, the <key> directory and reader. All must be owned by the current uid, not symlinks, not group- or world-writable, and reader must be a regular executable file. The manifest must parse, its key must match, and the binary digest must match. Any failure means the entry is never executed.
  • Fallback: if anything about the cache is unavailable, the reader uses today's per-process temp compile with unchanged cleanup semantics. That includes an unset or relative HOME, EACCES/EROFS/ENOSPC, a foreign or loosened entry, or a digest mismatch. Cache problems never fail an ownership operation. A REA-owned entry that fails validation is removed (Malformed owned data may still be safely removable). Entries with foreign ownership or modes are left alone, but are not executed.
  • Lifecycle: close() and process exit remove only per-process staging and temp roots, never a published entry. Nothing is pruned automatically, because removing an entry another live REA process may still exec would turn a cache into an ownership failure. Growth is one ~100 KB entry per helper-source × OS-release × compiler × arch combination. The location is documented, and it is safe to delete while no REA process runs.
  • Trust: this adds no new boundary. Today's helper already lives for the whole process, which is days for an MCP server, in a same-user 0700 temp root, and REA's own JavaScript is equally same-user-writable. The checks reject accidental or foreign corruption and loosened permissions. They don't claim to stop a same-user attacker. TOCTOU between verification and exec is the same same-user class and stays out of scope.

Verification plan: module tests with a fake compiler for a hit (no compile), a miss (compile once, then publish), a concurrent publish race, a corrupt digest, a loosened mode, a symlink, a foreign key, an unwritable cache root (fallback), cancellation during compile (no published entry, staging removed), and close not removing the published entry. Plus the real xcrun swiftc boundary test and before/after CLI timings (cold first run vs warm) for inspect-macho, inspect-signature and inspect-plist.

<!-- gh-comment-id:6544486519 --> @kaoru0822-kitauji commented on GitHub (Oct 31, 2026): Re-measured on main `23de10bb2` (macOS 26 arm64, Command Line Tools, Node 24.22.1). Child-process trace of one `rea inspect-macho /bin/ls --json`, total 1,427 ms: ``` +408 ms xcrun swiftc …/ProcessRunTokenReader.swift …ProcessRunTokenReaderMain.swift -o $TMP/rea-process-token-*/reader +325 ms $TMP/rea-process-token-*/reader --identities … ← first exec of the fresh binary +4–7 ms each for the 11 later reader execs; ps sweeps 14–28 ms each ``` Compile plus first exec is 733 ms, or 51% of the command. I'm taking this on with option 1 and a deliberately narrow trust story: **Proposal: a per-user, content-addressed helper cache, with fail-closed verification and the current compile as the fallback.** - **Location**: `~/Library/Caches/rea/process-helper/<key>/`. HOME comes from the launch environment, next to the existing `~/Library/Caches/rea/kuna`. The process ownership host turns it on in production. Readers created directly (tests) stay uncached unless they pass a directory. - **Key**: sha256 over a helper-format version, the bytes of both Swift sources, the compiler selection (`xcrun`, or `REA_DARWIN_SWIFTC_PATH` + `SDKROOT`), `process.arch` and `os.release()`. An OS update or a helper source change gives a new key. No extra compiler subprocess is spawned to build the key. - **Publish**: compile into a fresh `mkdtemp` staging directory inside the cache root, so the rename stays on one filesystem. Write a manifest with the binary's sha256 and the key inputs, then atomically `rename` the staging directory to `<key>`. If another process published first, verify and use its entry, then remove our staging directory. - **Use (once per process, before the first exec)**: `lstat` the cache root, the `<key>` directory and `reader`. All must be owned by the current uid, not symlinks, not group- or world-writable, and `reader` must be a regular executable file. The manifest must parse, its key must match, and the binary digest must match. Any failure means the entry is never executed. - **Fallback**: if anything about the cache is unavailable, the reader uses today's per-process temp compile with unchanged cleanup semantics. That includes an unset or relative HOME, `EACCES`/`EROFS`/`ENOSPC`, a foreign or loosened entry, or a digest mismatch. Cache problems never fail an ownership operation. A REA-owned entry that fails validation is removed (`Malformed owned data may still be safely removable`). Entries with foreign ownership or modes are left alone, but are not executed. - **Lifecycle**: `close()` and process exit remove only per-process staging and temp roots, never a published entry. Nothing is pruned automatically, because removing an entry another live REA process may still exec would turn a cache into an ownership failure. Growth is one ~100 KB entry per helper-source × OS-release × compiler × arch combination. The location is documented, and it is safe to delete while no REA process runs. - **Trust**: this adds no new boundary. Today's helper already lives for the whole process, which is days for an MCP server, in a same-user 0700 temp root, and REA's own JavaScript is equally same-user-writable. The checks reject accidental or foreign corruption and loosened permissions. They don't claim to stop a same-user attacker. TOCTOU between verification and exec is the same same-user class and stays out of scope. Verification plan: module tests with a fake compiler for a hit (no compile), a miss (compile once, then publish), a concurrent publish race, a corrupt digest, a loosened mode, a symlink, a foreign key, an unwritable cache root (fallback), cancellation during compile (no published entry, staging removed), and close not removing the published entry. Plus the real `xcrun swiftc` boundary test and before/after CLI timings (cold first run vs warm) for `inspect-macho`, `inspect-signature` and `inspect-plist`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#826
No description provided.