morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4682] [Bug] Ghidra: an OutOfMemoryError is not recognized once the session's 8 MiB output budget is used up #2440

New issue

Closed

opened 2026-10-17 21:04:42 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-17 21:04:42 +00:00

Owner

Originally created by @RishTheFish21 on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4682

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

When a Ghidra session has written more than 8 MiB of output before an operation fails with java.lang.OutOfMemoryError, REA doesn't recognize the OOM. The failure becomes a generic execution_failure telling the user to run rea doctor, instead of the resource_constraint error and memory advice from d52a488. The same OOM line with less output before it is recognized correctly.

Cause (source):

Ghidra process output is kept head-only, with one 8 MiB budget per session shared by stdout and stderr (GHIDRA_PROCESS_DIAGNOSTIC_BYTES, ProviderProcessSupervisor.#capture). Bytes after the budget are counted but dropped.
projectObservedJvmMemoryFailure only searches the retained failure_output text for java.lang.OutOfMemoryError, and ignores failure_output.complete.

Codex raised this as a P2 on #4275.

Reproduction

Regression test on main (cc2e18a2). It runs list_procedures through the real GhidraProvider, GhidraClient, process capture and error projection. Only the Ghidra JVM is simulated, by the repo's existing tests/fixtures/fakeGhidra.mjs with two new modes:

oom_exit_tools: write "ERROR Abort due to Headless analyzer error: Java heap space (HeadlessAnalyzer) java.lang.OutOfMemoryError: Java heap space" to stderr, then exit 1.
flood_then_oom_exit_tools: first write ~10.6 MiB of "INFO fixture analyzer progress line" to stdout, then the same OOM line, then exit 1.

npx vitest run tests/boundary/providers/ghidra/ghidraOomRetention.test.ts

Mode 1 → resource_constraint (passes). Mode 2 → execution_failure (fails). Reproduced on every run (2/2).

A real Ghidra run that logs >8 MiB and then runs out of memory on demand wasn't attempted; that's why the JVM is simulated.

Expected behavior

An OutOfMemoryError reported by the Ghidra process should give resource_constraint with the d52a488 remediation, however much output came before it. If output was dropped and no OOM can be confirmed, the error should say a memory cause can't be ruled out rather than pointing at rea doctor.

Impact and scope

Affects long Ghidra sessions and large targets, where import/analysis logging uses most of the budget before the failing request. Those are the cases where an OOM is most likely. The user gets the wrong error class and remediation; analysis results aren't affected. How often real sessions pass 8 MiB is unknown. Other providers weren't checked.

REA version

source commit cc2e18a2 (main), package 6.5.0

Runtime and platform

Node 24.18.0; macOS arm64; vitest 5.0.3 (simulated Ghidra 12.1.4 process)

Target and artifact context

Minimal ELF fixture (elf(2, 1, 62)); any Ghidra target whose session output exceeds 8 MiB

Interface

MCP

Earliest failure stage

Transport or result projection

Relevant output or evidence

Mode 2 projection (vitest diff, test expects resource_constraint):

{
"code": "execution_failure",
"diagnostic_truncated": true,
"failure_output_complete": false,
"message": "Analysis could not complete. Run rea doctor and review the reported details; repeating the unchanged request is not expected to succeed.",
}

Mode 1 (same OOM line, no earlier output): resource_constraint, test passes.

Source, main cc2e18a2:

src/process/ProviderProcess.ts #capture: retained = min(chunk, maxDiagnosticBytes - stdout.bytes - stderr.bytes); the rest is dropped.
src/ghidra/GhidraProviderClient.ts projectObservedJvmMemoryFailure: tests only the retained failure_output text for /java.lang.OutOfMemoryError/ and never reads failure_output.complete.

Typed outcome or recovery expected

resource_constraint (memory) with the d52a488 remediation, as mode 1 already gets. If output was incomplete and no OOM was seen, an error that says a memory cause can't be ruled out.

Analysis or suspected cause

Source-derived: the OOM check depends on retained text, and retention is head-only. A likely fix is to check each chunk for the OOM signature in #capture before the retention limit applies, and keep a small flag or the matching line, scoped to the current operation's checkpoint. Then project from that, using failure_output.complete for the "can't rule out" case.

Additional context

Raised by Codex on #4275 (superseded by d52a488, which has the same dependency on retained output). I have the reproduction test above and would like to take the fix.

Originally created by @RishTheFish21 on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4682 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior When a Ghidra session has written more than 8 MiB of output before an operation fails with java.lang.OutOfMemoryError, REA doesn't recognize the OOM. The failure becomes a generic execution_failure telling the user to run `rea doctor`, instead of the resource_constraint error and memory advice from d52a488. The same OOM line with less output before it is recognized correctly. Cause (source): - Ghidra process output is kept head-only, with one 8 MiB budget per session shared by stdout and stderr (GHIDRA_PROCESS_DIAGNOSTIC_BYTES, ProviderProcessSupervisor.#capture). Bytes after the budget are counted but dropped. - projectObservedJvmMemoryFailure only searches the retained failure_output text for java.lang.OutOfMemoryError, and ignores failure_output.complete. Codex raised this as a P2 on #4275. ### Reproduction Regression test on main (cc2e18a2). It runs list_procedures through the real GhidraProvider, GhidraClient, process capture and error projection. Only the Ghidra JVM is simulated, by the repo's existing tests/fixtures/fakeGhidra.mjs with two new modes: 1. oom_exit_tools: write "ERROR Abort due to Headless analyzer error: Java heap space (HeadlessAnalyzer) java.lang.OutOfMemoryError: Java heap space" to stderr, then exit 1. 2. flood_then_oom_exit_tools: first write ~10.6 MiB of "INFO fixture analyzer progress line" to stdout, then the same OOM line, then exit 1. npx vitest run tests/boundary/providers/ghidra/ghidraOomRetention.test.ts Mode 1 → resource_constraint (passes). Mode 2 → execution_failure (fails). Reproduced on every run (2/2). A real Ghidra run that logs >8 MiB and then runs out of memory on demand wasn't attempted; that's why the JVM is simulated. ### Expected behavior An OutOfMemoryError reported by the Ghidra process should give resource_constraint with the d52a488 remediation, however much output came before it. If output was dropped and no OOM can be confirmed, the error should say a memory cause can't be ruled out rather than pointing at `rea doctor`. ### Impact and scope Affects long Ghidra sessions and large targets, where import/analysis logging uses most of the budget before the failing request. Those are the cases where an OOM is most likely. The user gets the wrong error class and remediation; analysis results aren't affected. How often real sessions pass 8 MiB is unknown. Other providers weren't checked. ### REA version source commit cc2e18a2 (main), package 6.5.0 ### Runtime and platform Node 24.18.0; macOS arm64; vitest 5.0.3 (simulated Ghidra 12.1.4 process) ### Target and artifact context Minimal ELF fixture (elf(2, 1, 62)); any Ghidra target whose session output exceeds 8 MiB ### Interface MCP ### Earliest failure stage Transport or result projection ### Relevant output or evidence Mode 2 projection (vitest diff, test expects resource_constraint): { "code": "execution_failure", "diagnostic_truncated": true, "failure_output_complete": false, "message": "Analysis could not complete. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.", } Mode 1 (same OOM line, no earlier output): resource_constraint, test passes. Source, main cc2e18a2: - src/process/ProviderProcess.ts #capture: retained = min(chunk, maxDiagnosticBytes - stdout.bytes - stderr.bytes); the rest is dropped. - src/ghidra/GhidraProviderClient.ts projectObservedJvmMemoryFailure: tests only the retained failure_output text for /java\.lang\.OutOfMemoryError/ and never reads failure_output.complete. ### Typed outcome or recovery expected resource_constraint (memory) with the d52a488 remediation, as mode 1 already gets. If output was incomplete and no OOM was seen, an error that says a memory cause can't be ruled out. ### Analysis or suspected cause Source-derived: the OOM check depends on retained text, and retention is head-only. A likely fix is to check each chunk for the OOM signature in #capture before the retention limit applies, and keep a small flag or the matching line, scoped to the current operation's checkpoint. Then project from that, using failure_output.complete for the "can't rule out" case. ### Additional context Raised by Codex on #4275 (superseded by d52a488, which has the same dependency on retained output). I have the reproduction test above and would like to take the fix.

bsakaguchi added the

triage

bug

labels

2026-10-17 21:04:42 +00:00

bsakaguchi closed this issue

2026-10-31 00:33:58 +00:00

bsakaguchi commented

2026-10-31 00:33:59 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 17, 2026):

Cross-referencing with canonical JVM heap exhaustion reports in Cluster B: #3517 and #4241 ([Bug] Ghidra "Java heap space" during import is reported as execution_failure ("run rea doctor"), not a memory resource constraint). This report isolates an edge case in ProviderProcessSupervisor.#capture where logging beyond the 8 MiB budget causes OutOfMemoryError lines to be truncated from the diagnostic buffer, reverting the error projection back to generic execution_failure.

Consolidated under Cluster B in tracker #4330.

@matbrgz commented on GitHub (Oct 17, 2026): Cross-referencing with canonical JVM heap exhaustion reports in **Cluster B**: #3517 and #4241 (`[Bug] Ghidra "Java heap space" during import is reported as execution_failure ("run rea doctor"), not a memory resource constraint`). This report isolates an edge case in `ProviderProcessSupervisor.#capture` where logging beyond the 8 MiB budget causes `OutOfMemoryError` lines to be truncated from the diagnostic buffer, reverting the error projection back to generic `execution_failure`. Consolidated under Cluster B in tracker #4330.

bsakaguchi commented

2026-10-31 00:34:00 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

This is a real UX bug: once the shared 8 MiB diagnostic budget is exhausted, stderr from the Ghidra process is truncated before the OOM line, so projectObservedJvmMemoryFailure only sees the retained head and maps it to a generic execution_failure instead of resource_constraint. The fact that the same OOM is recognized correctly when there is less prior output confirms the failure is in the budgeted capture/retention logic, not in the JVM error detection itself. Keeping a small tail buffer for the failure path (or checking failure_output.complete before deciding the search scope) would make the classification consistent regardless of how much output the session wrote earlier. Good to see the fix in #4774 targets exactly this.

@XinwangTao commented on GitHub (Oct 18, 2026): This is a real UX bug: once the shared 8 MiB diagnostic budget is exhausted, stderr from the Ghidra process is truncated before the OOM line, so projectObservedJvmMemoryFailure only sees the retained head and maps it to a generic execution_failure instead of resource_constraint. The fact that the same OOM is recognized correctly when there is less prior output confirms the failure is in the budgeted capture/retention logic, not in the JVM error detection itself. Keeping a small tail buffer for the failure path (or checking failure_output.complete before deciding the search scope) would make the classification consistent regardless of how much output the session wrote earlier. Good to see the fix in #4774 targets exactly this.

bsakaguchi commented

2026-10-31 00:34:00 +00:00

Author

Owner

@morluto commented on GitHub (Oct 19, 2026):

Fixed on main in 91884c3743, adapting #4774 with the contributor attribution preserved.

The producing process owner now scans all drained bytes before retention and retains bounded confirmed OOME token observations. Classification consumes those facts instead of reinterpreting retained text. Review reproduced and corrected valid-marker overwrite by a later identifier, prior-operation cause text crossing the checkpoint, and a budget-cut ...ErrorCustom prefix falsely appearing complete. The scanner distinguishes a pending token from a confirmed boundary and finishes bare tokens only at observed stream EOF. Directional line completeness keeps prefix clipping from triggering credential-tail redaction.

Incomplete operation output without a confirmed OOME retains its original error code/cause and explicitly says a memory cause cannot be ruled out. This is not classified as a memory constraint. Current checkpoint coverage is separate from historical dropped output. #5220 also binds cause details/advice to owner-observed OOME lines.

Executed: 268 focused tests across process capture, scanner, Ghidra provider/client, real Node pipes/socket and MCP failure/cancellation controls; npm run check, npm run docs:check, test discovery (all 1639 files exactly once), and normal commit/push hooks pass. The Ghidra JVM was simulated; real Ghidra and native Linux/Windows behavior remain unverified. Ghidra 12.1.4 logging source was checked separately; source browsing does not establish runtime reproduction.

Completed provider probe medians: elapsed 0.239→0.267s; CPU 0.256→0.276s; sampled family RSS 515→467MB (brief peaks may be missed; individual-process rusage peaks 597→556MB). Full internal provider projection grew 484 bytes / about115 estimated cl100k tokens while retaining its existing large diagnostics. These are not MCP wire/model-context counts: oversized MCP errors already use bounded retained-result delivery. Three nonalternating runs per version after warmup establish no speedup; real JVM, full agent task and transport/recovery costs remain unmeasured.

@morluto commented on GitHub (Oct 19, 2026): Fixed on main in 91884c374385182726a93d56059720d5e78a0e42, adapting #4774 with the contributor attribution preserved. The producing process owner now scans all drained bytes before retention and retains bounded confirmed OOME token observations. Classification consumes those facts instead of reinterpreting retained text. Review reproduced and corrected valid-marker overwrite by a later identifier, prior-operation cause text crossing the checkpoint, and a budget-cut `...ErrorCustom` prefix falsely appearing complete. The scanner distinguishes a pending token from a confirmed boundary and finishes bare tokens only at observed stream EOF. Directional line completeness keeps prefix clipping from triggering credential-tail redaction. Incomplete operation output without a confirmed OOME retains its original error code/cause and explicitly says a memory cause cannot be ruled out. This is not classified as a memory constraint. Current checkpoint coverage is separate from historical dropped output. #5220 also binds cause details/advice to owner-observed OOME lines. Executed: 268 focused tests across process capture, scanner, Ghidra provider/client, real Node pipes/socket and MCP failure/cancellation controls; `npm run check`, `npm run docs:check`, test discovery (all 1639 files exactly once), and normal commit/push hooks pass. The Ghidra JVM was simulated; real Ghidra and native Linux/Windows behavior remain unverified. Ghidra 12.1.4 logging source was checked separately; source browsing does not establish runtime reproduction. Completed provider probe medians: elapsed 0.239→0.267s; CPU 0.256→0.276s; sampled family RSS 515→467MB (brief peaks may be missed; individual-process rusage peaks 597→556MB). Full internal provider projection grew 484 bytes / about115 estimated cl100k tokens while retaining its existing large diagnostics. These are not MCP wire/model-context counts: oversized MCP errors already use bounded retained-result delivery. Three nonalternating runs per version after warmup establish no speedup; real JVM, full agent task and transport/recovery costs remain unmeasured.

bsakaguchi referenced this issue

2026-10-31 01:05:01 +00:00

[PR #2440] [MERGED] fix(android): clean Windows workspaces after owner termination #5233

bsakaguchi referenced this issue

2026-10-31 01:07:44 +00:00

[PR #3048] fix(android): decode guardian readiness across UTF-8 chunks #5580