morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5859] [Bug] Ghidra killed by the kernel OOM killer reports execution_failure with `rea doctor` advice instead of a memory resource_constraint #3552

New issue

Closed

opened 2026-10-31 00:41:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:41:46 +00:00

Owner

Originally created by @Rinkia on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5859

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Related: #4241 (the JVM java.lang.OutOfMemoryError: Java heap space case, fixed in 6.6.0 by matching that output). This report is the remaining case: the kernel SIGKILLs Ghidra, so no OOME text exists to match.

Area

Native analysis or decompilation

Observed behavior

When the Ghidra process is killed by the Linux cgroup OOM killer (container memory limit), rea analyze returns:

code: execution_failure
message: Analysis could not complete. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.

The diagnostics already show the cause: exit_code: null, exit_signal: "SIGKILL", configured_jvm_max_heap: "2G". The cgroup confirms the kill: memory.events reports oom_kill 1, memory.peak equals memory.max (3221225472), and docker inspect reports OOMKilled=true. rea doctor gives no hint about memory; it only reports unrelated missing optional engines.

The JVM heap is not what runs out. A per-process sample taken once a second up to the kill:

java=1x/661MB  MainThread(node)=1x/357MB  decompile=10x/2150MB  total≈3166MB   <- next sample: java gone

Ten native decompile processes (auto-analysis, scaled to nproc=12) use about 2.1 GB, while the JVM sits at 661 MB, well under its 2G cap. Lowering the heap to GHIDRA_HEADLESS_MAXMEM=1G is still SIGKILLed. So the advice REA already gives for JVM OutOfMemoryError (raise GHIDRA_HEADLESS_MAXMEM) would be the wrong fix here.

Reproduction

Public target: cargo-auditable.exe from the cargo-auditable v0.7.7 release asset cargo-auditable-x86_64-pc-windows-msvc.zip (asset digest sha256:717ba2d3ed7997c501e9987ee7c94bebf9d7219b18b185a03009420ec7029c7d, https://github.com/rust-secure-code/cargo-auditable/releases/tag/v0.7.7). Executable sha256 fc35804634fc76ddec6bd2d7dde2bdd2c0b18e7fe2e7bc96d223751f6a72cefc, 1,069,568 bytes.

Image built from the bundled Docker recipe with rea-agents@6.7.0. The Docker VM has 12 CPUs.

docker run --rm --network none --memory 3g --memory-swap 3g \
  -e GHIDRA_HEADLESS_MAXMEM=2G -v "$PWD:/targets:ro" \
  rea:6.7.0 analyze /targets/cargo-auditable.exe --format json --full-output

This exact command failed after 72 s. The same limits failed in every one of 6 runs (55–120 s), including variants with --read-only --tmpfs /tmp (the tmpfs held only 32K, so it plays no part). Controls on the same target and machine:

Run	Limits	Result
A	3g, swap off, heap 2G	SIGKILL, `oom_kill 1`, execution_failure + doctor advice
C	3g, swap off, heap 1G	SIGKILL, `oom_kill 1`, same message
D	A + `REA_GHIDRA_DISABLED_ANALYZERS=Decompiler Switch Analysis`	SIGKILL, same message
B	3g, swap at Docker default (6g total)	ok in 312 s (`memory.swap.peak` ≈ 298 MiB)
E	A + `--cpus 2`	no OOM; `provider_timeout` at the 330 s startup deadline
F	A + `--cpus 2` + `REA_GHIDRA_STARTUP_TIMEOUT_MS=1200000`	ok in 595 s, 2704 procedures

Expected behavior

A Ghidra process that ends with SIGKILL during import or analysis is reported as a typed resource failure, not as an installation problem:

On Linux, compare the cgroup v2 memory.events oom_kill count before and after the run. If it went up, project resource_constraint with resource: "memory" and the observed limit (memory.max). The advice should say the memory limit was hit by the whole process tree, Ghidra's native decompiler processes included, and suggest raising the container or host memory limit, or reducing CPUs/parallelism. It should not suggest a bigger Java heap.
If the kill cannot be attributed, at least say "Ghidra was terminated by SIGKILL (possibly out of memory)" instead of "Run rea doctor".

Impact and scope

Anyone running REA in a memory-limited container or CI runner on a many-core host. Decompiler parallelism follows the CPU count, so the same target that works on a small machine fails on a bigger one.
The current message tells users that repeating the request will not help and points them at rea doctor. Doctor finds nothing, and the GHIDRA_HEADLESS_MAXMEM knob users would try next does not help either.

REA version

rea-agents 6.7.0 (executed). Source checked on main at f801b2fbf (2026-10-21): unchanged, see below.

Runtime and platform

Docker Desktop on Windows 11 (WSL2 VM, 12 CPUs, about 7.6 GB). Container Ubuntu 24.04, Node 24.22.0, Ghidra 12.1.4, cgroup v2.

Target and artifact context

Public PE32+ Rust binary (above). Nothing target-specific is involved; any medium-size binary with a few thousand functions should do.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

From --full-output, trimmed:

{"ok":false,"error":{"code":"execution_failure","details":{"provider_id":"ghidra","operation":"read_binary_inventory_summary",
 "diagnostics":{"configured_jvm_max_heap":"2G","exit_code":null,"exit_signal":"SIGKILL", "...": "..."}}}}

The stdout tail ends in ordinary auto-analysis lines (WARN Decompiling 14005c790, pcode error ...), with no java.lang.OutOfMemoryError. The cgroup after the run:

memory.max 3221225472 · memory.swap.max 0 · memory.peak 3221225472
memory.events: max 772, oom 3, oom_kill 1

Typed outcome or recovery expected

resource_constraint (resource: "memory") with the observed cgroup limit and process-tree-aware advice, as described under Expected behavior.

Analysis or suspected cause

Source-derived on main f801b2fbf; not stepped through:

src/ghidra/GhidraDiagnostics.ts recognizes memory failures only through the java.lang.OutOfMemoryError output marker (GHIDRA_JVM_OOME_MARKER). The existing "Ghidra OOME error projection" tests in GhidraProvider.behavior.test.ts cover heap, GC-overhead, Metaspace and native-thread cases, all text-based.
An external SIGKILL leaves no such text, so the failure falls through to the generic projection in src/domain/analysisErrorPresentation.ts (the "Run rea doctor … not expected to succeed" message).
exit_signal is already captured in the diagnostics (GhidraClient.ts / GhidraDiagnostics.ts), so the signal is available at projection time.

Possibly worth a separate feature request: a knob for Ghidra analysis/decompiler parallelism, so memory use can be bounded without limiting the container's CPUs (today --cpus is the only lever, and it pushes runs past the 330 s startup deadline).

Originally created by @Rinkia on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5859 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. Related: #4241 (the JVM `java.lang.OutOfMemoryError: Java heap space` case, fixed in 6.6.0 by matching that output). This report is the remaining case: the kernel SIGKILLs Ghidra, so no OOME text exists to match. ### Area Native analysis or decompilation ### Observed behavior When the Ghidra process is killed by the Linux cgroup OOM killer (container memory limit), `rea analyze` returns: ``` code: execution_failure message: Analysis could not complete. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed. ``` The diagnostics already show the cause: `exit_code: null`, `exit_signal: "SIGKILL"`, `configured_jvm_max_heap: "2G"`. The cgroup confirms the kill: `memory.events` reports `oom_kill 1`, `memory.peak` equals `memory.max` (3221225472), and `docker inspect` reports `OOMKilled=true`. `rea doctor` gives no hint about memory; it only reports unrelated missing optional engines. The JVM heap is not what runs out. A per-process sample taken once a second up to the kill: ``` java=1x/661MB MainThread(node)=1x/357MB decompile=10x/2150MB total≈3166MB <- next sample: java gone ``` Ten native `decompile` processes (auto-analysis, scaled to `nproc`=12) use about 2.1 GB, while the JVM sits at 661 MB, well under its 2G cap. Lowering the heap to `GHIDRA_HEADLESS_MAXMEM=1G` is still SIGKILLed. So the advice REA already gives for JVM OutOfMemoryError (raise `GHIDRA_HEADLESS_MAXMEM`) would be the wrong fix here. ### Reproduction Public target: `cargo-auditable.exe` from the cargo-auditable v0.7.7 release asset `cargo-auditable-x86_64-pc-windows-msvc.zip` (asset digest `sha256:717ba2d3ed7997c501e9987ee7c94bebf9d7219b18b185a03009420ec7029c7d`, https://github.com/rust-secure-code/cargo-auditable/releases/tag/v0.7.7). Executable sha256 `fc35804634fc76ddec6bd2d7dde2bdd2c0b18e7fe2e7bc96d223751f6a72cefc`, 1,069,568 bytes. Image built from the bundled Docker recipe with `rea-agents@6.7.0`. The Docker VM has 12 CPUs. ```bash docker run --rm --network none --memory 3g --memory-swap 3g \ -e GHIDRA_HEADLESS_MAXMEM=2G -v "$PWD:/targets:ro" \ rea:6.7.0 analyze /targets/cargo-auditable.exe --format json --full-output ``` This exact command failed after 72 s. The same limits failed in every one of 6 runs (55–120 s), including variants with `--read-only --tmpfs /tmp` (the tmpfs held only 32K, so it plays no part). Controls on the same target and machine: | Run | Limits | Result | |---|---|---| | A | 3g, swap off, heap 2G | SIGKILL, `oom_kill 1`, execution_failure + doctor advice | | C | 3g, swap off, heap **1G** | SIGKILL, `oom_kill 1`, same message | | D | A + `REA_GHIDRA_DISABLED_ANALYZERS=Decompiler Switch Analysis` | SIGKILL, same message | | B | 3g, swap at Docker default (6g total) | ok in 312 s (`memory.swap.peak` ≈ 298 MiB) | | E | A + `--cpus 2` | no OOM; `provider_timeout` at the 330 s startup deadline | | F | A + `--cpus 2` + `REA_GHIDRA_STARTUP_TIMEOUT_MS=1200000` | ok in 595 s, 2704 procedures | ### Expected behavior A Ghidra process that ends with SIGKILL during import or analysis is reported as a typed resource failure, not as an installation problem: - On Linux, compare the cgroup v2 `memory.events` `oom_kill` count before and after the run. If it went up, project `resource_constraint` with `resource: "memory"` and the observed limit (`memory.max`). The advice should say the memory limit was hit by the whole process tree, Ghidra's native decompiler processes included, and suggest raising the container or host memory limit, or reducing CPUs/parallelism. It should not suggest a bigger Java heap. - If the kill cannot be attributed, at least say "Ghidra was terminated by SIGKILL (possibly out of memory)" instead of "Run `rea doctor`". ### Impact and scope - Anyone running REA in a memory-limited container or CI runner on a many-core host. Decompiler parallelism follows the CPU count, so the same target that works on a small machine fails on a bigger one. - The current message tells users that repeating the request will not help and points them at `rea doctor`. Doctor finds nothing, and the `GHIDRA_HEADLESS_MAXMEM` knob users would try next does not help either. ### REA version rea-agents 6.7.0 (executed). Source checked on `main` at f801b2fbf (2026-10-21): unchanged, see below. ### Runtime and platform Docker Desktop on Windows 11 (WSL2 VM, 12 CPUs, about 7.6 GB). Container Ubuntu 24.04, Node 24.22.0, Ghidra 12.1.4, cgroup v2. ### Target and artifact context Public PE32+ Rust binary (above). Nothing target-specific is involved; any medium-size binary with a few thousand functions should do. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence From `--full-output`, trimmed: ```json {"ok":false,"error":{"code":"execution_failure","details":{"provider_id":"ghidra","operation":"read_binary_inventory_summary", "diagnostics":{"configured_jvm_max_heap":"2G","exit_code":null,"exit_signal":"SIGKILL", "...": "..."}}}} ``` The stdout tail ends in ordinary auto-analysis lines (`WARN Decompiling 14005c790, pcode error ...`), with no `java.lang.OutOfMemoryError`. The cgroup after the run: ``` memory.max 3221225472 · memory.swap.max 0 · memory.peak 3221225472 memory.events: max 772, oom 3, oom_kill 1 ``` ### Typed outcome or recovery expected `resource_constraint` (`resource: "memory"`) with the observed cgroup limit and process-tree-aware advice, as described under Expected behavior. ### Analysis or suspected cause Source-derived on `main` f801b2fbf; not stepped through: - `src/ghidra/GhidraDiagnostics.ts` recognizes memory failures only through the `java.lang.OutOfMemoryError` output marker (`GHIDRA_JVM_OOME_MARKER`). The existing "Ghidra OOME error projection" tests in `GhidraProvider.behavior.test.ts` cover heap, GC-overhead, Metaspace and native-thread cases, all text-based. - An external SIGKILL leaves no such text, so the failure falls through to the generic projection in `src/domain/analysisErrorPresentation.ts` (the "Run `rea doctor` … not expected to succeed" message). - `exit_signal` is already captured in the diagnostics (`GhidraClient.ts` / `GhidraDiagnostics.ts`), so the signal is available at projection time. Possibly worth a separate feature request: a knob for Ghidra analysis/decompiler parallelism, so memory use can be bounded without limiting the container's CPUs (today `--cpus` is the only lever, and it pushes runs past the 330 s startup deadline).

bsakaguchi closed this issue

2026-10-31 00:41:46 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:40 +00:00

[PR #3552] [CLOSED] fix(javascript): retain exports through renamed module parameters #5818

bsakaguchi referenced this issue

2026-10-31 01:11:01 +00:00

[PR #4020] [CLOSED] fix(javascript): preserve structural fingerprint semantics #5994

bsakaguchi referenced this issue

2026-11-03 13:39:25 +00:00

[PR #7117] [MERGED] fix(javascript): bind defaulted and destructured Webpack factory roles #7655