[PR #4275] [CLOSED] fix(ghidra): report JVM heap exhaustion as a memory constraint #6152

Closed
opened 2026-10-31 01:12:16 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4275
Author: @RishTheFish21
Created: 10/15/2026
Status: ❌ Closed

Base: main ← Head: fix/4241-ghidra-heap


📝 Commits (2)

  • cd2ee37 fix(ghidra): report JVM heap exhaustion as a memory constraint
  • cf884f4 fix(ghidra): advise a larger heap only for heap exhaustion

📊 Changes

8 files changed (+362 additions, -7 deletions)

View changed files

📝 docs/installation.md (+14 -0)
📝 src/ghidra/GhidraDiagnostics.ts (+3 -0)
📝 src/ghidra/GhidraLauncher.ts (+33 -6)
➕ src/ghidra/GhidraMemoryExhaustion.test.ts (+90 -0)
➕ src/ghidra/GhidraMemoryExhaustion.ts (+93 -0)
📝 src/ghidra/GhidraProvider.behavior.test.ts (+123 -1)
📝 src/ghidra/GhidraProviderClient.ts (+4 -0)
📝 tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts (+2 -0)

📄 Description

Summary

When Ghidra's JVM runs out of heap during import or analysis, REA reports a generic execution_failure that tells the caller to run rea doctor. Doctor reports Ghidra as healthy, so the advice leads nowhere. The real cause (java.lang.OutOfMemoryError: Java heap space) is only visible deep inside details.diagnostics.stdout.text.

Now, when a failed Ghidra session's own output shows a JVM OutOfMemoryError, REA reports resource_constraint with resource: "memory", the heap size it launched Ghidra with, and memory-specific remediation.

Fixes #4241

Problem and expected behavior

docs/binary-diagnostics.md says memory allocation failures use resource_constraint, with effective limits (or an explicit unknown) and memory-specific guidance. No Ghidra path recognized OutOfMemoryError, so every heap exhaustion was projected as ProviderAdapterError (execution_failure), or as a timeout if the deadline hit first.

Smallest trigger: any target large enough to exhaust a small heap, e.g. GHIDRA_HEADLESS_MAXMEM=128M rea inspect "$(command -v node)" --provider ghidra --json.

Change and scope

  • src/ghidra/GhidraMemoryExhaustion.ts: looks for java.lang.OutOfMemoryError (with its detail, e.g. "Java heap space") in the retained Ghidra process stdout/stderr. When found, the failure becomes AnalysisResourceConstraintError("memory"):
    • reported_limits.jvm_max_heap is the heap REA launched with, or null if unknown.
    • The remediation is to raise GHIDRA_HEADLESS_MAXMEM above that value or skip heavy analyzers with REA_GHIDRA_DISABLED_ANALYZERS, then reopen. It also notes that rea doctor doesn't check heap size.
    • It applies to start, process, protocol and timeout failures, after cancellation is handled. Bridge-reported (remote) failures and cancellations are unchanged.
  • The launcher records the selected heap on the launch, and diagnostics expose it as jvm_max_heap.
    • On macOS/Linux it's the last -Xmx actually passed to the JVM.
    • On Windows, the headless script picks the heap itself, so only an explicit GHIDRA_HEADLESS_MAXMEM/GHIDRA_MAXMEM is reported. Otherwise it stays unknown.
  • docs/installation.md: documents the new classification under "Ghidra heap and CPU controls".

Not included: the issue's second request (naming REA_GHIDRA_STARTUP_TIMEOUT_MS in startup-timeout remediation) and the optional "Untested Go version" limitation. Both touch different code paths, so I'd do them separately.

Follow the contribution scope.
Unresolved scope, authority, side effects, or required verification block
integration. Link the required maintainer scope decision for new providers or
capabilities involving target execution, injection, mutation, or protection
recovery; ordinary fixes within existing boundaries may state "not applicable".
Draft PRs may leave the scope decision pending and the readiness and merge checklists
unchecked. Maintainers can help resolve open questions.

Contract and boundary impact

  • Semantic owner and earliest changed stage: Ghidra session error projection (projectPrimarySessionError in src/ghidra/GhidraProviderClient.ts); launch diagnostics in GhidraLauncher.ts / GhidraDiagnostics.ts.
  • Contribution scope fit and linked maintainer decision: Bug fix within the existing Ghidra provider boundary. Maintainer decision not applicable.
  • CLI and MCP/tool-catalog contract: Ghidra heap exhaustion now returns the existing resource_constraint code instead of execution_failure/provider_timeout. No schema change.
  • Provider, bridge, target-format, or platform compatibility: Ghidra only. No bridge (Java) change. Windows reports the heap only when it's explicitly configured.
  • Evidence, artifact, provenance, or reconstruction contract: Error-path only. No Evidence or analysis-profile change.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): Not applicable. Cleanup behavior is unchanged.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): None.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated:
    • src/ghidra/GhidraMemoryExhaustion.test.ts: reads OutOfMemoryError with and without detail from stdout and stderr; bounds long details; ignores mentions outside the process streams. Heap selection uses the last POSIX -Xmx, and Windows reports only explicit settings.
    • src/ghidra/GhidraProvider.behavior.test.ts: a process failure with heap exhaustion in stdout projects to resource_constraint with jvm_max_heap: "4G" and the new remediation. A timeout with GC overhead limit exceeded and no recorded heap reports jvm_max_heap: null. Failures without an OOM, and bridge-reported failures, are unchanged.
    • tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts: a real launch records jvmMaxHeap: "1G" and diagnostics expose it.
  • Base reproduction or other evidence: Real Ghidra 12.1.4 on macOS arm64, GHIDRA_HEADLESS_MAXMEM=128M rea inspect "$(command -v node)" --provider ghidra --json:
    • upstream/main: "code":"execution_failure". The only mention of the cause is buried in stdout diagnostics: OutOfMemoryError: Java heap space\n\tat java.base/java.lang.Long.valueOf(...).
    • this branch: "code":"resource_constraint", "jvm_max_heap":"128M", message Ghidra's JVM ran out of memory (java.lang.OutOfMemoryError: Java heap space) with a maximum heap of 128M.
  • User-visible CLI/MCP output (if applicable): see above.
  • Remaining proof gaps: Not run on Linux or Windows. I didn't reproduce the issue's Go/Hugo target or confirm that a larger heap lets it finish. An OutOfMemoryError printed earlier in a long session will classify a later process failure as memory too, since the classification reads the session's retained output.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:focused -- src/ghidra/GhidraMemoryExhaustion.test.ts src/ghidra/GhidraProvider.behavior.test.ts tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts: passed
  • npm run check:fast: passed
  • npm run lint: passed
  • npm run knip: passed
  • npm run docs:check: passed
  • Real Ghidra 12.1.4 before/after run on macOS arm64 (above): execution_failure → resource_constraint with jvm_max_heap: "128M"

Compatibility, safety, and release

  • Breaking changes or migration steps: Callers that matched execution_failure for Ghidra heap exhaustion now receive resource_constraint (resource: "memory"). Other failures are unchanged.
  • Real Hopper/Ghidra, browser, or OS coverage: Real Ghidra 12.1.4 on macOS arm64. Hopper is unaffected.
  • Package or release metadata impact: None.
  • Security, privacy, process, or containment review: Reads already-retained, already-redacted process output. No new process or target interaction.

Author readiness

Human owner: @RishTheFish21

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4275 **Author:** [@RishTheFish21](https://github.com/RishTheFish21) **Created:** 10/15/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/4241-ghidra-heap` --- ### 📝 Commits (2) - [`cd2ee37`](https://github.com/morluto/rea/commit/cd2ee3785fae75567e084449dc3659653472b2b1) fix(ghidra): report JVM heap exhaustion as a memory constraint - [`cf884f4`](https://github.com/morluto/rea/commit/cf884f41efb17170b32d47a1f4b8c3846f91c3c3) fix(ghidra): advise a larger heap only for heap exhaustion ### 📊 Changes **8 files changed** (+362 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `docs/installation.md` (+14 -0) 📝 `src/ghidra/GhidraDiagnostics.ts` (+3 -0) 📝 `src/ghidra/GhidraLauncher.ts` (+33 -6) ➕ `src/ghidra/GhidraMemoryExhaustion.test.ts` (+90 -0) ➕ `src/ghidra/GhidraMemoryExhaustion.ts` (+93 -0) 📝 `src/ghidra/GhidraProvider.behavior.test.ts` (+123 -1) 📝 `src/ghidra/GhidraProviderClient.ts` (+4 -0) 📝 `tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts` (+2 -0) </details> ### 📄 Description ## Summary When Ghidra's JVM runs out of heap during import or analysis, REA reports a generic `execution_failure` that tells the caller to run `rea doctor`. Doctor reports Ghidra as healthy, so the advice leads nowhere. The real cause (`java.lang.OutOfMemoryError: Java heap space`) is only visible deep inside `details.diagnostics.stdout.text`. Now, when a failed Ghidra session's own output shows a JVM `OutOfMemoryError`, REA reports `resource_constraint` with `resource: "memory"`, the heap size it launched Ghidra with, and memory-specific remediation. Fixes #4241 ## Problem and expected behavior `docs/binary-diagnostics.md` says memory allocation failures use `resource_constraint`, with effective limits (or an explicit unknown) and memory-specific guidance. No Ghidra path recognized `OutOfMemoryError`, so every heap exhaustion was projected as `ProviderAdapterError` (`execution_failure`), or as a timeout if the deadline hit first. Smallest trigger: any target large enough to exhaust a small heap, e.g. `GHIDRA_HEADLESS_MAXMEM=128M rea inspect "$(command -v node)" --provider ghidra --json`. ## Change and scope - `src/ghidra/GhidraMemoryExhaustion.ts`: looks for `java.lang.OutOfMemoryError` (with its detail, e.g. "Java heap space") in the retained Ghidra process stdout/stderr. When found, the failure becomes `AnalysisResourceConstraintError("memory")`: - `reported_limits.jvm_max_heap` is the heap REA launched with, or `null` if unknown. - The remediation is to raise `GHIDRA_HEADLESS_MAXMEM` above that value or skip heavy analyzers with `REA_GHIDRA_DISABLED_ANALYZERS`, then reopen. It also notes that `rea doctor` doesn't check heap size. - It applies to start, process, protocol and timeout failures, after cancellation is handled. Bridge-reported (`remote`) failures and cancellations are unchanged. - The launcher records the selected heap on the launch, and diagnostics expose it as `jvm_max_heap`. - On macOS/Linux it's the last `-Xmx` actually passed to the JVM. - On Windows, the headless script picks the heap itself, so only an explicit `GHIDRA_HEADLESS_MAXMEM`/`GHIDRA_MAXMEM` is reported. Otherwise it stays unknown. - `docs/installation.md`: documents the new classification under "Ghidra heap and CPU controls". Not included: the issue's second request (naming `REA_GHIDRA_STARTUP_TIMEOUT_MS` in startup-timeout remediation) and the optional "Untested Go version" limitation. Both touch different code paths, so I'd do them separately. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". Draft PRs may leave the scope decision pending and the readiness and merge checklists unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact - Semantic owner and earliest changed stage: Ghidra session error projection (`projectPrimarySessionError` in `src/ghidra/GhidraProviderClient.ts`); launch diagnostics in `GhidraLauncher.ts` / `GhidraDiagnostics.ts`. - Contribution scope fit and linked maintainer decision: Bug fix within the existing Ghidra provider boundary. Maintainer decision not applicable. - CLI and MCP/tool-catalog contract: Ghidra heap exhaustion now returns the existing `resource_constraint` code instead of `execution_failure`/`provider_timeout`. No schema change. - Provider, bridge, target-format, or platform compatibility: Ghidra only. No bridge (Java) change. Windows reports the heap only when it's explicitly configured. - Evidence, artifact, provenance, or reconstruction contract: Error-path only. No Evidence or analysis-profile change. - Target selection, authority, upstream behavior, and cleanup details (if applicable): Not applicable. Cleanup behavior is unchanged. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs): None. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: - `src/ghidra/GhidraMemoryExhaustion.test.ts`: reads `OutOfMemoryError` with and without detail from stdout and stderr; bounds long details; ignores mentions outside the process streams. Heap selection uses the last POSIX `-Xmx`, and Windows reports only explicit settings. - `src/ghidra/GhidraProvider.behavior.test.ts`: a process failure with heap exhaustion in stdout projects to `resource_constraint` with `jvm_max_heap: "4G"` and the new remediation. A timeout with `GC overhead limit exceeded` and no recorded heap reports `jvm_max_heap: null`. Failures without an OOM, and bridge-reported failures, are unchanged. - `tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts`: a real launch records `jvmMaxHeap: "1G"` and diagnostics expose it. - Base reproduction or other evidence: Real Ghidra 12.1.4 on macOS arm64, `GHIDRA_HEADLESS_MAXMEM=128M rea inspect "$(command -v node)" --provider ghidra --json`: - upstream/main: `"code":"execution_failure"`. The only mention of the cause is buried in stdout diagnostics: `OutOfMemoryError: Java heap space\n\tat java.base/java.lang.Long.valueOf(...)`. - this branch: `"code":"resource_constraint"`, `"jvm_max_heap":"128M"`, message `Ghidra's JVM ran out of memory (java.lang.OutOfMemoryError: Java heap space) with a maximum heap of 128M.` - User-visible CLI/MCP output (if applicable): see above. - Remaining proof gaps: Not run on Linux or Windows. I didn't reproduce the issue's Go/Hugo target or confirm that a larger heap lets it finish. An `OutOfMemoryError` printed earlier in a long session will classify a later process failure as memory too, since the classification reads the session's retained output. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- src/ghidra/GhidraMemoryExhaustion.test.ts src/ghidra/GhidraProvider.behavior.test.ts tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts`: passed - `npm run check:fast`: passed - `npm run lint`: passed - `npm run knip`: passed - `npm run docs:check`: passed - Real Ghidra 12.1.4 before/after run on macOS arm64 (above): `execution_failure` → `resource_constraint` with `jvm_max_heap: "128M"` ## Compatibility, safety, and release - Breaking changes or migration steps: Callers that matched `execution_failure` for Ghidra heap exhaustion now receive `resource_constraint` (`resource: "memory"`). Other failures are unchanged. - Real Hopper/Ghidra, browser, or OS coverage: Real Ghidra 12.1.4 on macOS arm64. Hopper is unaffected. - Package or release metadata impact: None. - Security, privacy, process, or containment review: Reads already-retained, already-redacted process output. No new process or target interaction. ## Author readiness Human owner: @RishTheFish21 - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:12:16 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#6152
No description provided.