morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3413] [Feature] Static packer identification and UPX extraction as a derived artifact for PE/ELF #1272

New issue

Open

opened 2026-10-16 21:03:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:03:36 +00:00

Owner

Originally created by @dickens88 on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3413

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

We run REA 6.4.0 (CLI + Ghidra provider, Linux x64) as the static engine behind an agent that triages suspicious Windows PE / Linux ELF files. Many real-world samples are compressed with UPX or a similar packer. Today:

REA gives no packer signal. The agent has to guess from indirect hints (few imports, odd section names, unusual sizes), and different runs reach different conclusions.
Once a sample is packed, Ghidra only sees the decompression stub. String search is empty or misleading (as GhidraProviderCapabilities already notes), and the agent spends dozens of tool calls decompiling the stub before giving up.
docs/ghidra-dos.md recommends analyzing "a separately prepared unpacked copy" while retaining both digests and the transformation evidence, but REA has no first-party way to produce or record that copy. The handoff happens outside REA and loses provenance.

This is related to, but much narrower than, #2080. We are not asking for commercial-protector defeat or runtime dumping. This request covers only the two parts that #2080's closing comment and CONTRIBUTING.md describe as in scope: static packer identification, and ordinary standard-compression extraction.

Use cases

Malware triage / incident response: an analyst submits a PE that turns out to be UPX-packed and expects analysis of the real program, with evidence showing that it was unpacked and how.
Any native binary (self-built tools, legacy utilities) shipped UPX-compressed to reduce size.

Desired outcome and success criteria

Two independently decidable parts:

A. Static packer identification (no target interaction beyond reading bytes)
For an admitted PE or ELF, report packer indicators as evidence, for example:

per-section raw/virtual size, flags, and Shannon entropy;
known packer section names / magic (e.g. UPX0/UPX1, UPX! header) with byte offsets;
import count and whether imports are limited to loader-style APIs (LoadLibrary*, GetProcAddress, VirtualAlloc/VirtualProtect);
entry point located in a non-first or writable+executable section;
overlay presence and size.

Output is a verdict such as likely_packed / not_observed / unknown, with the supporting facts and explicit limitations. An empty result does not establish that the file is unpacked.

B. UPX extraction as a derived artifact (static decompression, no execution)
Following the firmware Binwalk/Unblob pattern:

a caller-supplied UPX binary (e.g. REA_UPX_COMMAND, version-checked, not installed by REA);
upx -d runs on a private copy, never the admitted original;
the result is a new artifact with its own digest, linked to the original digest, the tool version, and the command outcome, and can then be selected for normal Ghidra analysis;
modified or corrupted UPX headers fail explicitly (unsupported / tool error retained) instead of being worked around.

Success: for a UPX-packed self-built PE and ELF, an agent can go from "packed original" to "Ghidra analysis of the decompressed program" through REA CLI/MCP, and the evidence chain records both digests.

Expected interaction with targets

Read existing files or captured data
Write or change files/configuration (B only: writes a derived copy into REA's workspace; the original snapshot is unchanged)

Target and scope details (optional)

Targets: Windows PE (x86/x64) and Linux ELF (x86/x64/ARM) on a Linux x64 host.

Non-goals:

commercial protectors / code virtualization (Themida, VMProtect, Enigma, etc.);
emulation, process launch, memory dumping, or OEP/IAT reconstruction;
anti-debug / anti-tamper bypass;
automatically replacing the admitted target.

Open question for maintainers: does UPX -d count as "ordinary standard compression extraction" under the current scope? If not, would part A alone be acceptable?

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Self-built, freely licensed fixtures: compile a small C program for Linux and Windows (mingw), then upx --best it. Expected results:

A reports UPX0/UPX1 sections, a high-entropy UPX1, and a minimal import table on the packed file, and not_observed on the unpacked build;
B produces a file whose digest matches upx -d run directly, whose functions match the unpacked build, and whose evidence links to the packed original;
a fixture with a deliberately corrupted UPX! header fails explicitly.

Alternatives considered

Running upx -d outside REA before submission: this works, but the derived artifact has no provenance inside REA's evidence, and the agent has to decide on its own when to do it.
Having the agent infer packing from inspect output plus read_bytes: costly and inconsistent, and REA does not surface section entropy today.
PPEE as a provider (#2729) might cover part A for PE only. It does not cover ELF or extraction.

Possible direction

Part A could extend the existing native inspection / binary diagnostics output. Part B could reuse the content-addressed extraction and caller-supplied-tool plumbing from firmware analysis and the .NET single-file bundle work (#2352).

References

#2080 (closing comment invites a detection-only proposal)
docs/ghidra-dos.md – "analyze a separately prepared unpacked copy… retain both artifact digests and the transformation evidence"
docs/firmware-analysis.md – caller-supplied extraction tool pattern
UPX: https://github.com/upx/upx

Additional real-provider observations

On macOS arm64, direct Kuna 1.986 unpack succeeded on its public test fixtures: a UPX-packed i386 PE and an x86-64 ELF. The ELF result was a separate output with SHA-256 747e8dfcd05f51b77def83e961ba57c56bc89bf1602db9783b6453ad65387acb; its packed input had SHA-256 907b4dc30be0c0e53fda3ad1905ef3d804735a472f1c9484d3cfb5d3180e26c1. Neither was executed. In real REA stdio MCP, Kuna-backed list_procedures on the packed ELF failed with the upstream diagnostic that the image appears UPX-packed and suggests kuna unpack; after a separate direct Kuna unpack and explicit open_binary of the derived file, REA listed 70 procedures and decompiled main. REA did not perform the unpack or create a provenance link between these two artifacts. The diagnostic's input path names REA's private temporary copy, so a user taking the suggestion outside REA must instead select their original file.

A separate source-built PE32+ fixture was packed by UPX 5.2.1; upx -t and upx -d both succeeded, but Kuna 1.833 and 1.986 reported corrupt UPX image: PackHeader declares an overlay outside the file. Upstream investigation indicates Kuna's PE unpacker currently supports 32-bit PE, so successful PE32+ extraction must not be assumed; the misleading error is being investigated upstream. The UPX-restored file's digest differed from the original, so its success does not prove byte-for-byte reconstruction.

These observations compare possible components for the analyst workflow; they do not authorize target-recovery integration or establish that arbitrary UPX files can be unpacked. Kuna's direct command also documents overwriting an existing output by default, so an REA-owned derived-artifact design would need its own explicit destination, ownership, publication and provenance rules rather than forwarding that behavior. The applicable maintainer scope decision remains open.

PE32+ control (2026-11-04)

On macOS arm64, a self-built x86-64 PE packed with UPX 5.3.0 passed upx -t. Kuna 1.990's standalone unpack succeeded, and its output matched upx -d byte for byte; the packed input's SHA-256 remained unchanged. This is one fixture, not broad format verification. Kuna's earlier PE32+ diagnostic issue and support work are Noelo-Lab/kuna#1294 and Noelo-Lab/kuna#1298.

Through REA's public MCP on the Kuna 1.990 validation branch, explicitly selecting Kuna rejected both packed and unpacked PE with the documented target_format_unsupported admission reason. Selecting Ghidra 12.1.4 opened the separately unpacked copy and decompiled a recovered function in both output-schema profiles; both input hashes remained unchanged. This demonstrates a manual handoff, not a provenance-linked REA transformation. It does not establish scope approval for an integrated unpacking workflow or PE support in REA's Kuna provider.

Existing inspection workflow on that PE control

On the same fixture and REA validation branch, public MCP inspect_pe_imports reported the packed PE's four KERNEL32 imports (LoadLibraryA, ExitProcess, GetProcAddress, VirtualProtect), while the independently unpacked PE had no import directory. Ghidra binary_overview identified UPX0, UPX1 and UPX2 blocks, and inspect_native_load_image reported the UPX0 mapping as uninitialized, executable and writable and the entry at 0x140006150. The latter explicitly marked independent PE load-image verification unsupported; these are Ghidra observations. The unpacked copy instead had .text and entry 0x140001060.

inspect_byte_entropy reported 4.1515 bits/byte for the entire 2,048-byte packed file and no high-entropy runs at the default 4 KiB block size, versus 0.9713 for the 3,072-byte unpacked file. The tool correctly notes small blocks can underestimate entropy and that entropy cannot distinguish compression from encryption. These are clues for one deliberately small sample, not a detector or a general entropy comparison. The workflow uses multiple separate calls and leaves interpretation and artifact transformation provenance to the caller. No target was executed.

Cross-target evidence handoff control

In one public MCP session, I opened the packed PE with Ghidra, called inspect_pe_imports and binary_overview, then switched to the separately unpacked copy and repeated both calls. After close_binary, get_evidence_bundle retained two distinct artifact SHA-256 identities and six records, including the inventory records linked by each overview; every record retained the correct subject digest. It did not infer a transformation between the two artifacts, as expected for independently supplied files. The complete bundle returned inline was saved as JSON, imported into a fresh target-free server with import_evidence_bundle, and the resulting bundle matched the original exactly. Both target digests stayed unchanged. This tests evidence retention and portability, not an automatic transformation record or a caller-declared relationship mechanism.

Preserving the manual handoff as a question

A target-free server imported that two-artifact bundle. Using record_unknown, I recorded an open question asking whether the output digest came from unpacking the input digest with Kuna 1.990, linked both existing inspect_pe_imports Evidence IDs, and requested controlled-replay authority and independently captured transformation evidence. A complete bundle containing the unknown and its mutation record was imported into another target-free server; the open status and both evidence links survived. This is a usable manual uncertainty record, not a typed transformation edge or proof of the invocation: the linked PE observations establish properties of each file, not the relationship between them. The requested controlled replay was not performed through REA in this control.

Originally created by @dickens88 on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3413 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation We run REA 6.4.0 (CLI + Ghidra provider, Linux x64) as the static engine behind an agent that triages suspicious Windows PE / Linux ELF files. Many real-world samples are compressed with UPX or a similar packer. Today: - REA gives no packer signal. The agent has to guess from indirect hints (few imports, odd section names, unusual sizes), and different runs reach different conclusions. - Once a sample is packed, Ghidra only sees the decompression stub. String search is empty or misleading (as `GhidraProviderCapabilities` already notes), and the agent spends dozens of tool calls decompiling the stub before giving up. - `docs/ghidra-dos.md` recommends analyzing "a separately prepared unpacked copy" while retaining both digests and the transformation evidence, but REA has no first-party way to produce or record that copy. The handoff happens outside REA and loses provenance. This is related to, but much narrower than, #2080. We are **not** asking for commercial-protector defeat or runtime dumping. This request covers only the two parts that #2080's closing comment and CONTRIBUTING.md describe as in scope: static packer identification, and ordinary standard-compression extraction. ### Use cases - Malware triage / incident response: an analyst submits a PE that turns out to be UPX-packed and expects analysis of the real program, with evidence showing that it was unpacked and how. - Any native binary (self-built tools, legacy utilities) shipped UPX-compressed to reduce size. ### Desired outcome and success criteria Two independently decidable parts: **A. Static packer identification (no target interaction beyond reading bytes)** For an admitted PE or ELF, report packer indicators as evidence, for example: - per-section raw/virtual size, flags, and Shannon entropy; - known packer section names / magic (e.g. `UPX0`/`UPX1`, `UPX!` header) with byte offsets; - import count and whether imports are limited to loader-style APIs (`LoadLibrary*`, `GetProcAddress`, `VirtualAlloc`/`VirtualProtect`); - entry point located in a non-first or writable+executable section; - overlay presence and size. Output is a verdict such as `likely_packed` / `not_observed` / `unknown`, with the supporting facts and explicit limitations. An empty result does not establish that the file is unpacked. **B. UPX extraction as a derived artifact (static decompression, no execution)** Following the firmware Binwalk/Unblob pattern: - a caller-supplied UPX binary (e.g. `REA_UPX_COMMAND`, version-checked, not installed by REA); - `upx -d` runs on a private copy, never the admitted original; - the result is a new artifact with its own digest, linked to the original digest, the tool version, and the command outcome, and can then be selected for normal Ghidra analysis; - modified or corrupted UPX headers fail explicitly (`unsupported` / tool error retained) instead of being worked around. Success: for a UPX-packed self-built PE and ELF, an agent can go from "packed original" to "Ghidra analysis of the decompressed program" through REA CLI/MCP, and the evidence chain records both digests. ### Expected interaction with targets - Read existing files or captured data - Write or change files/configuration (B only: writes a derived copy into REA's workspace; the original snapshot is unchanged) ### Target and scope details (optional) Targets: Windows PE (x86/x64) and Linux ELF (x86/x64/ARM) on a Linux x64 host. Non-goals: - commercial protectors / code virtualization (Themida, VMProtect, Enigma, etc.); - emulation, process launch, memory dumping, or OEP/IAT reconstruction; - anti-debug / anti-tamper bypass; - automatically replacing the admitted target. Open question for maintainers: does UPX `-d` count as "ordinary standard compression extraction" under the current scope? If not, would part A alone be acceptable? ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Self-built, freely licensed fixtures: compile a small C program for Linux and Windows (mingw), then `upx --best` it. Expected results: - A reports `UPX0`/`UPX1` sections, a high-entropy `UPX1`, and a minimal import table on the packed file, and `not_observed` on the unpacked build; - B produces a file whose digest matches `upx -d` run directly, whose functions match the unpacked build, and whose evidence links to the packed original; - a fixture with a deliberately corrupted `UPX!` header fails explicitly. ### Alternatives considered - Running `upx -d` outside REA before submission: this works, but the derived artifact has no provenance inside REA's evidence, and the agent has to decide on its own when to do it. - Having the agent infer packing from `inspect` output plus `read_bytes`: costly and inconsistent, and REA does not surface section entropy today. - PPEE as a provider (#2729) might cover part A for PE only. It does not cover ELF or extraction. ### Possible direction Part A could extend the existing native inspection / binary diagnostics output. Part B could reuse the content-addressed extraction and caller-supplied-tool plumbing from firmware analysis and the .NET single-file bundle work (#2352). ### References - #2080 (closing comment invites a detection-only proposal) - `docs/ghidra-dos.md` – "analyze a separately prepared unpacked copy… retain both artifact digests and the transformation evidence" - `docs/firmware-analysis.md` – caller-supplied extraction tool pattern - UPX: https://github.com/upx/upx ### Additional real-provider observations On macOS arm64, direct Kuna 1.986 `unpack` succeeded on its public test fixtures: a UPX-packed i386 PE and an x86-64 ELF. The ELF result was a separate output with SHA-256 `747e8dfcd05f51b77def83e961ba57c56bc89bf1602db9783b6453ad65387acb`; its packed input had SHA-256 `907b4dc30be0c0e53fda3ad1905ef3d804735a472f1c9484d3cfb5d3180e26c1`. Neither was executed. In real REA stdio MCP, Kuna-backed `list_procedures` on the packed ELF failed with the upstream diagnostic that the image appears UPX-packed and suggests `kuna unpack`; after a separate direct Kuna unpack and explicit `open_binary` of the derived file, REA listed 70 procedures and decompiled `main`. REA did not perform the unpack or create a provenance link between these two artifacts. The diagnostic's input path names REA's private temporary copy, so a user taking the suggestion outside REA must instead select their original file. A separate source-built PE32+ fixture was packed by UPX 5.2.1; `upx -t` and `upx -d` both succeeded, but Kuna 1.833 and 1.986 reported `corrupt UPX image: PackHeader declares an overlay outside the file`. Upstream investigation indicates Kuna's PE unpacker currently supports 32-bit PE, so successful PE32+ extraction must not be assumed; the misleading error is being investigated upstream. The UPX-restored file's digest differed from the original, so its success does not prove byte-for-byte reconstruction. These observations compare possible components for the analyst workflow; they do not authorize target-recovery integration or establish that arbitrary UPX files can be unpacked. Kuna's direct command also documents overwriting an existing output by default, so an REA-owned derived-artifact design would need its own explicit destination, ownership, publication and provenance rules rather than forwarding that behavior. The applicable maintainer scope decision remains open. ### PE32+ control (2026-11-04) On macOS arm64, a self-built x86-64 PE packed with UPX 5.3.0 passed `upx -t`. Kuna 1.990's standalone `unpack` succeeded, and its output matched `upx -d` byte for byte; the packed input's SHA-256 remained unchanged. This is one fixture, not broad format verification. Kuna's earlier PE32+ diagnostic issue and support work are [Noelo-Lab/kuna#1294](https://github.com/Noelo-Lab/kuna/issues/1294) and [Noelo-Lab/kuna#1298](https://github.com/Noelo-Lab/kuna/pull/1298). Through REA's public MCP on the Kuna 1.990 validation branch, explicitly selecting Kuna rejected both packed and unpacked PE with the documented `target_format_unsupported` admission reason. Selecting Ghidra 12.1.4 opened the separately unpacked copy and decompiled a recovered function in both output-schema profiles; both input hashes remained unchanged. This demonstrates a manual handoff, not a provenance-linked REA transformation. It does not establish scope approval for an integrated unpacking workflow or PE support in REA's Kuna provider. ### Existing inspection workflow on that PE control On the same fixture and REA validation branch, public MCP `inspect_pe_imports` reported the packed PE's four KERNEL32 imports (`LoadLibraryA`, `ExitProcess`, `GetProcAddress`, `VirtualProtect`), while the independently unpacked PE had no import directory. Ghidra `binary_overview` identified `UPX0`, `UPX1` and `UPX2` blocks, and `inspect_native_load_image` reported the `UPX0` mapping as uninitialized, executable and writable and the entry at `0x140006150`. The latter explicitly marked independent PE load-image verification unsupported; these are Ghidra observations. The unpacked copy instead had `.text` and entry `0x140001060`. `inspect_byte_entropy` reported 4.1515 bits/byte for the entire 2,048-byte packed file and no high-entropy runs at the default 4 KiB block size, versus 0.9713 for the 3,072-byte unpacked file. The tool correctly notes small blocks can underestimate entropy and that entropy cannot distinguish compression from encryption. These are clues for one deliberately small sample, not a detector or a general entropy comparison. The workflow uses multiple separate calls and leaves interpretation and artifact transformation provenance to the caller. No target was executed. ### Cross-target evidence handoff control In one public MCP session, I opened the packed PE with Ghidra, called `inspect_pe_imports` and `binary_overview`, then switched to the separately unpacked copy and repeated both calls. After `close_binary`, `get_evidence_bundle` retained two distinct artifact SHA-256 identities and six records, including the inventory records linked by each overview; every record retained the correct subject digest. It did not infer a transformation between the two artifacts, as expected for independently supplied files. The complete bundle returned inline was saved as JSON, imported into a fresh target-free server with `import_evidence_bundle`, and the resulting bundle matched the original exactly. Both target digests stayed unchanged. This tests evidence retention and portability, not an automatic transformation record or a caller-declared relationship mechanism. ### Preserving the manual handoff as a question A target-free server imported that two-artifact bundle. Using `record_unknown`, I recorded an open question asking whether the output digest came from unpacking the input digest with Kuna 1.990, linked both existing `inspect_pe_imports` Evidence IDs, and requested controlled-replay authority and independently captured transformation evidence. A complete bundle containing the unknown and its mutation record was imported into another target-free server; the open status and both evidence links survived. This is a usable manual uncertainty record, not a typed transformation edge or proof of the invocation: the linked PE observations establish properties of each file, not the relationship between them. The requested controlled replay was not performed through REA in this control.

bsakaguchi referenced this issue

2026-10-17 21:09:53 +00:00

[PR #1272] [MERGED] docs(website): continue the Calculator example with reconstruction #2756

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1272

No description provided.

Rows
Columns