morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #3032] [Bug] macOS: closing a large Ghidra session reports cleanup_incomplete (sysctl_errno_22) while the JVM is mid-exit #1061

New issue

Closed

opened 2026-10-15 18:30:45 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:30:45 +00:00

Owner

Originally created by @visualmindpt on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3032

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

On macOS arm64, closing a Ghidra session for a large target returns cleanup_incomplete even though the Ghidra JVM exits normally (exit_code: 0). The provider runtime directory rea-ghidra-XXXXXX under the temporary directory (about 84 MB) is retained every time.

It happened on two consecutive runs with the same target, once through the CLI (rea decompile <target> entry --provider ghidra) and once through MCP close_binary. The analysis result itself was returned correctly; with the CLI it arrives as partial_observation inside the error.

Closing a session for a small target (14 MB x86_64 Mach-O, default 2G heap) on the same host did not show the problem.

Reproduction

macOS arm64, Ghidra 12.1.4, JDK 21, with GHIDRA_HEADLESS_MAXMEM=6G and REA_GHIDRA_STARTUP_TIMEOUT_MS=1800000.
Run rea decompile /path/to/large-arm64-macho entry --provider ghidra --format json on a target whose auto-analysis uses several GB of JVM memory (here: a 74 MB thin arm64 Mach-O, about 10 minutes of analysis, 3.4 GB peak RSS for the process family).
Observe the cleanup_incomplete error on close and the leftover rea-ghidra-XXXXXX directory.

This is timing dependent. A provider-free demonstration of the underlying OS behaviour is in "Analysis or suspected cause".

Expected behavior

When the only unresolved group member is a process the kernel is already tearing down, cleanup should wait for it to become a zombie or disappear, then report cleaned: true and remove the runtime directory. A process that is live and unreadable for any other reason should keep failing closed as it does today.

Impact and scope

Affected: macOS cleanup of owned process groups when a member with a large address space is exiting at the moment ownership is revalidated. Each occurrence leaves a runtime directory behind, and the CLI reports an error for an analysis that succeeded.
Not affected: evidence content (the pseudocode was correct), process isolation (nothing unowned was signalled; the check failed closed), Linux (procfs path), small targets on the same host.
Not verified: Hopper and IDA sessions, the x86_64 host path, and how often this occurs across runs.

REA version

6.4.0, source checkout at 06739aeaf

Runtime and platform

Node 26.0.0; macOS 26.5.2 arm64; Ghidra 12.1.4; Temurin JDK 21.0.12

Target and artifact context

Thin arm64 Mach-O executable of a Qt/C++ desktop application, 74 MB (52,775 functions after analysis)

Interface

CLI (also observed through MCP close_binary)

Earliest failure stage

Cleanup, cancellation, or lifecycle

Relevant output or evidence

code: cleanup_incomplete
category: execution_failure
message: Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target.
details.operation: close_binary
details.resources: ["ghidra-process", "<tmp>/rea-ghidra-XXXXXX"]
details.diagnostics.reason: Ghidra process cleanup was incomplete: process ownership token could not be read for 1 live process(es): sysctl_errno_22=1; live candidates <pid>=sysctl_errno_22
details.diagnostics.diagnostics.exit_code: 0
details.diagnostics.diagnostics.exit_signal: null

Typed outcome or recovery expected

cleaned: true once the exiting process has settled. If it is still exiting after a bounded wait, the current cleanup_incomplete with the same diagnostic is the right outcome.

Analysis or suspected cause

Observed on this host, independent of REA:

sysctl(KERN_PROCARGS2) returns EINVAL (22) for a zombie, which matches sysctl_failed_22 from bridge/process/ProcessRunTokenReader.swift.
A process with about 3 GB of touched memory, killed with SIGKILL, was reported by ps -o stat= as ?E (exit in progress) before it became Z. During that interval KERN_PROCARGS2 also returned EINVAL. The interval was about 30 ms for that process.

Source-derived reasoning:

liveProcesses in src/process/ProcessOwnershipProcessTree.ts excludes only entries whose state starts with Z, so a process in the E state counts as live.
recheckOwnershipFailures in src/process/ProcessOwnership.ts resolves an unreadable observation only when the PID is absent from one fresh snapshot. A member that is mid-exit in that snapshot stays environment-unreadable, and cleanupValidationFailure reports it.
The sweep path has settleUnreadableTokenCandidates (3 attempts, 40 ms apart), but its final liveness snapshot has the same gap, and the group-member path has no settling at all.

Hypothesis, not confirmed inside REA: the Ghidra JVM was in the exit-in-progress state when ownership was revalidated, and tearing down a multi-GB heap made that window long enough to hit. I did not capture the process state at the moment of the two failures.

Candidate fix, tested locally: make the liveness snapshot that resolves unreadable observations repeat, within a fixed budget, while a selected process carries the BSD exit flag (!state.startsWith("Z") && state.slice(1).includes("E")). Processes that are not exiting keep the single-snapshot behaviour, so the existing "one fresh snapshot" tests still pass. With that change the process module tests pass and one real Ghidra 12.1.4 CLI run on the same target closed cleanly with no leftover directory; one clean run does not establish the race is closed. I can open a PR with the change and a regression test if that is useful.

Originally created by @visualmindpt on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3032 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior On macOS arm64, closing a Ghidra session for a large target returns `cleanup_incomplete` even though the Ghidra JVM exits normally (`exit_code: 0`). The provider runtime directory `rea-ghidra-XXXXXX` under the temporary directory (about 84 MB) is retained every time. It happened on two consecutive runs with the same target, once through the CLI (`rea decompile <target> entry --provider ghidra`) and once through MCP `close_binary`. The analysis result itself was returned correctly; with the CLI it arrives as `partial_observation` inside the error. Closing a session for a small target (14 MB x86_64 Mach-O, default 2G heap) on the same host did not show the problem. ### Reproduction 1. macOS arm64, Ghidra 12.1.4, JDK 21, with `GHIDRA_HEADLESS_MAXMEM=6G` and `REA_GHIDRA_STARTUP_TIMEOUT_MS=1800000`. 2. Run `rea decompile /path/to/large-arm64-macho entry --provider ghidra --format json` on a target whose auto-analysis uses several GB of JVM memory (here: a 74 MB thin arm64 Mach-O, about 10 minutes of analysis, 3.4 GB peak RSS for the process family). 3. Observe the `cleanup_incomplete` error on close and the leftover `rea-ghidra-XXXXXX` directory. This is timing dependent. A provider-free demonstration of the underlying OS behaviour is in "Analysis or suspected cause". ### Expected behavior When the only unresolved group member is a process the kernel is already tearing down, cleanup should wait for it to become a zombie or disappear, then report `cleaned: true` and remove the runtime directory. A process that is live and unreadable for any other reason should keep failing closed as it does today. ### Impact and scope - Affected: macOS cleanup of owned process groups when a member with a large address space is exiting at the moment ownership is revalidated. Each occurrence leaves a runtime directory behind, and the CLI reports an error for an analysis that succeeded. - Not affected: evidence content (the pseudocode was correct), process isolation (nothing unowned was signalled; the check failed closed), Linux (procfs path), small targets on the same host. - Not verified: Hopper and IDA sessions, the x86_64 host path, and how often this occurs across runs. ### REA version 6.4.0, source checkout at `06739aeaf` ### Runtime and platform Node 26.0.0; macOS 26.5.2 arm64; Ghidra 12.1.4; Temurin JDK 21.0.12 ### Target and artifact context Thin arm64 Mach-O executable of a Qt/C++ desktop application, 74 MB (52,775 functions after analysis) ### Interface CLI (also observed through MCP `close_binary`) ### Earliest failure stage Cleanup, cancellation, or lifecycle ### Relevant output or evidence ```text code: cleanup_incomplete category: execution_failure message: Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target. details.operation: close_binary details.resources: ["ghidra-process", "<tmp>/rea-ghidra-XXXXXX"] details.diagnostics.reason: Ghidra process cleanup was incomplete: process ownership token could not be read for 1 live process(es): sysctl_errno_22=1; live candidates <pid>=sysctl_errno_22 details.diagnostics.diagnostics.exit_code: 0 details.diagnostics.diagnostics.exit_signal: null ``` ### Typed outcome or recovery expected `cleaned: true` once the exiting process has settled. If it is still exiting after a bounded wait, the current `cleanup_incomplete` with the same diagnostic is the right outcome. ### Analysis or suspected cause Observed on this host, independent of REA: - `sysctl(KERN_PROCARGS2)` returns `EINVAL` (22) for a zombie, which matches `sysctl_failed_22` from `bridge/process/ProcessRunTokenReader.swift`. - A process with about 3 GB of touched memory, killed with SIGKILL, was reported by `ps -o stat=` as `?E` (exit in progress) before it became `Z`. During that interval `KERN_PROCARGS2` also returned `EINVAL`. The interval was about 30 ms for that process. Source-derived reasoning: - `liveProcesses` in `src/process/ProcessOwnershipProcessTree.ts` excludes only entries whose state starts with `Z`, so a process in the `E` state counts as live. - `recheckOwnershipFailures` in `src/process/ProcessOwnership.ts` resolves an unreadable observation only when the PID is absent from one fresh snapshot. A member that is mid-exit in that snapshot stays `environment-unreadable`, and `cleanupValidationFailure` reports it. - The sweep path has `settleUnreadableTokenCandidates` (3 attempts, 40 ms apart), but its final liveness snapshot has the same gap, and the group-member path has no settling at all. Hypothesis, not confirmed inside REA: the Ghidra JVM was in the exit-in-progress state when ownership was revalidated, and tearing down a multi-GB heap made that window long enough to hit. I did not capture the process state at the moment of the two failures. Candidate fix, tested locally: make the liveness snapshot that resolves unreadable observations repeat, within a fixed budget, while a selected process carries the BSD exit flag (`!state.startsWith("Z") && state.slice(1).includes("E")`). Processes that are not exiting keep the single-snapshot behaviour, so the existing "one fresh snapshot" tests still pass. With that change the process module tests pass and one real Ghidra 12.1.4 CLI run on the same target closed cleanly with no leftover directory; one clean run does not establish the race is closed. I can open a PR with the change and a regression test if that is useful.

bsakaguchi

2026-10-15 18:30:45 +00:00

closed this issue
added the
area: native

bug
labels

bsakaguchi commented

2026-10-15 18:30:46 +00:00

Author

Owner

@ashishkattamuri commented on GitHub (Oct 14, 2026):

It looks like this may already be addressed on main in commit 599a8ee11. recheckOwnershipFailures now re-snapshots (up to 3 × 40 ms) while an unreadable member is in the Darwin E (exit-in-progress) state, and there's a regression test for a ?E member in ProcessOwnership.validation.test.ts. That commit doesn't reference this issue, so I wanted to link them.

@visualmindpt, could you retest on the current main with the large target? If it still reproduces, I'm happy to help with a follow-up that uses a time-based bound instead @morluto

@ashishkattamuri commented on GitHub (Oct 14, 2026): It looks like this may already be addressed on main in commit 599a8ee11. recheckOwnershipFailures now re-snapshots (up to 3 × 40 ms) while an unreadable member is in the Darwin E (exit-in-progress) state, and there's a regression test for a ?E member in ProcessOwnership.validation.test.ts. That commit doesn't reference this issue, so I wanted to link them. @visualmindpt, could you retest on the current main with the large target? If it still reproduces, I'm happy to help with a follow-up that uses a time-based bound instead @morluto

bsakaguchi commented

2026-10-15 18:30:46 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Resolved on main by 599a8ee11.

Darwin process ownership observations now allow the exiting-in-kernel state to settle. The issue also contains confirmation of this fix. The recheck exercises controlled ownership observations, not a fresh large Ghidra session.

Rechecked on main b8510ee00 in the current focused run: 20 test files passed, 225 tests passed and one platform-dependent test skipped. Relevant regression coverage: src/process/ProcessOwnership.validation.test.ts.

@morluto commented on GitHub (Oct 14, 2026): Resolved on main by 599a8ee11. Darwin process ownership observations now allow the exiting-in-kernel state to settle. The issue also contains confirmation of this fix. The recheck exercises controlled ownership observations, not a fresh large Ghidra session. Rechecked on main b8510ee00 in the current focused run: 20 test files passed, 225 tests passed and one platform-dependent test skipped. Relevant regression coverage: src/process/ProcessOwnership.validation.test.ts.

bsakaguchi referenced this issue

2026-10-17 21:08:13 +00:00

[PR #1061] [MERGED] fix(install): guard empty prefix_args expansion on macOS Bash 3.2 under set -u #2614

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1061

No description provided.

Rows
Columns