[PR #1868] [MERGED] feat(native): admit raw binaries with caller-supplied architecture and entry #4864

Closed
opened 2026-10-31 00:56:09 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1868
Author: @mvanhorn
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @morluto

Base: main ← Head: fix/raw-binary-architecture


📝 Commits (4)

  • 04cbf4d feat(native): admit raw binaries with caller-supplied architecture and entry
  • 90da1ce fix(native): honor raw-binary preparation and entry identity
  • 039c9c9 fix(ghidra): stream raw snapshot attestation with cancellation
  • 0e58a88 fix(native): preserve and validate raw lifecycle entries

📊 Changes

57 files changed (+2452 additions, -335 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+1 -1)
📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+7 -0)
📝 bridge/ghidra/ReaGhidraBridge.java (+11 -4)
➕ bridge/ghidra/ReaGhidraPrepareRaw.java (+36 -0)
📝 docs/.vitepress/config.ts (+1 -0)
📝 docs/cli.md (+2 -1)
➕ docs/raw-binary.md (+72 -0)
📝 docs/testing.md (+8 -0)
📝 package.json (+2 -0)
📝 scripts/verify-packaged-ghidra-windows.mjs (+1 -0)
➕ scripts/verify-real-ghidra-raw.mjs (+219 -0)
📝 src/application/BinaryTargetResolver.ts (+53 -3)
📝 src/application/DirectAnalysis.ts (+37 -0)
📝 src/application/binary/BinarySessionOpen.ts (+9 -0)
📝 src/application/binary/BinarySessionPort.ts (+6 -1)
📝 src/application/binary/BinarySessionStatus.ts (+2 -5)
➕ src/application/binary/BinaryTargetSummary.ts (+11 -0)
📝 src/cli/coreBinaryCommands.ts (+22 -66)
📝 src/cli/coreNativeCommands.ts (+22 -83)
📝 src/cli/options.ts (+55 -1)

...and 37 more files

📄 Description

Summary

Callers can now open a headerless executable by supplying the CPU family and the address where file offset 0 is mapped. Header-based detection is unchanged when format is omitted. Hopper and IDA refuse this interpretation; Ghidra imports it with BinaryLoader and a little-endian generic language, then seeds an entry function before auto-analysis.

Fixes #1836

Demo

demo

Problem and expected behavior

A local image without ELF, Mach-O, PE, or MZ headers is rejected as truncated or unsupported binary header. Embedded firmware often has those headers removed, so inventory and function analysis cannot start even when the caller already knows the CPU family and load address.

format=raw-binary with architecture (x86, x86_64, arm, or arm64) and entry_point (0x hex) admits the complete file as an executable. File offset 0 maps at that address. Architecture or entry without format=raw-binary is rejected. The analysis profile includes the bound Ghidra language and canonical entry, so a different architecture or entry is a different snapshot identity.

Change and scope

New domain admission (src/domain/rawBinary.ts) composed through parseBinaryTarget, session open, and shared CLI/MCP adapters:

  • open_binary remains a single type: object input. Pairing is enforced with superRefine, not an input anyOf.
  • CLI --target-format raw-binary --architecture … --entry-point … on native inspect, function, and load-image commands.
  • Ghidra profile: BinaryLoader, language_selection: caller-supplied-v1, canonical entry_point, load_image_evidence: independent-raw-mapping-context-v1. Languages: x86:LE:32:default, x86:LE:64:default, ARM:LE:32:v7, AARCH64:LE:64:v8A.
  • Packaged ReaGhidraPrepareRaw.java maps file offset 0 at the entry, then addEntryPoint / disassemble / createFunction("entry") before auto-analysis.
  • Hopper and IDA return target_format_unsupported with a Ghidra-directed reason. Windows Ghidra P0 remains PE-only (applications and DLLs).
  • Independent load-image attestation covers source identity, mapped bytes, language, and caller-supplied entry. Original executable bytes are unchanged.
  • Authored skill text in .agents/skills/reverse-engineer-anything/ (skill version 39). Catalog regenerated; build outputs not committed.

Out of scope: a load address independent of entry, big-endian / Thumb-only / Cortex-M / PowerPC languages, analyzer overrides, guessing vector tables or memory maps, and a new MCP tool.

Contract and boundary impact

  • Semantic owner and earliest changed stage: domain src/domain/rawBinary.ts, then parseBinaryTarget / session open, CLI/MCP adapters, Ghidra BinaryLoader + prepare script.
  • CLI and MCP/tool-catalog contract: open_binary gains optional architecture and entry_point; format enum adds raw-binary. CLI --architecture and --entry-point join --target-format. Output format enums and native load-image results add a raw-binary verified/mismatch variant. Input schema stays a single object.
  • Provider, bridge, target-format, or platform compatibility: Ghidra Linux/macOS admit raw-binary; Hopper and IDA refuse; Windows Ghidra P0 still PE-only. Packaged bridge/ghidra/ReaGhidraPrepareRaw.java.
  • Evidence, artifact, provenance, or reconstruction contract: snapshot/profile identity includes canonical entry and language selection; source spelling is preserved on the target as entryPointSource and does not change the digest. Load-image checks keep observed vs expected facts distinct.
  • Process execution, authorization, cleanup, or containment impact: ephemeral Ghidra import remains read-only and delete-on-exit; no executable-byte writes.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): skill version 39 in .agents/skills/reverse-engineer-anything/ and src/generatedPackageMetadata.ts; package.json files list includes ReaGhidraPrepareRaw.java.

Evidence and regression coverage

  • Tests added or updated: domain admission, pairing, and address-width overflow; open_binary input contract; filesystem target resolution including misleading suffixes; Ghidra language map, profile digest, Windows P0 refusal, launcher argv, and load-image attestation; Hopper/IDA target_format_unsupported; CLI inspect before/missing-flags/Hopper-refusal; MCP open success and omitted architecture.
  • Base reproduction or other evidence: on 58767790, rea inspect of a four-byte headerless image reports truncated or unsupported binary header; --target-format raw-binary is not a format hint until architecture and entry are also supplied.
  • User-visible CLI/MCP output (if applicable): generated local ecu.bin (1eff2fe1). Inspect without flags: target_unavailable / truncated or unsupported binary header. --target-format raw-binary alone: raw-binary format requires architecture (x86, x86_64, arm, or arm64). Full flags with --provider hopper: capability_unavailable / target_format_unsupported (“Headerless raw-binary analysis requires the Ghidra adapter with a caller-supplied architecture and entry point.”), not unsupported binary format. Architecture without format: architecture and entry_point apply only with format=raw-binary. MCP open_binary with format, architecture, and entry_point returns format: "raw-binary".
  • Remaining proof gaps: real Ghidra import of a caller-supplied raw-binary image was not run on this host. No verify:ghidra:raw lane in this increment.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm ci — pass (377 packages added, 378 audited)
  • npm run test:focused -- src/domain/rawBinary.test.ts src/contracts/sessionLifecycleInputs.test.ts src/ghidra/GhidraRawBinaryLoadImage.test.ts src/ghidra/GhidraProvider.behavior.test.ts src/hopper/HopperProvider.test.ts src/ida/IdaProvider.test.ts tests/boundary/cli/rawBinaryCli.test.ts tests/boundary/mcp/rawBinaryMcp.test.ts tests/boundary/filesystem/rawBinaryTarget.test.ts tests/boundary/providers/ghidra/ghidraLauncher.test.ts tests/boundary/providers/hopper/hopperAnalysisProfile.test.ts — pass (11 files, 98 passed, 1 skipped)
  • npm run check:fast — pass
  • npm run check — pass (typecheck, lint, format, knip, metadata:check; 6/6 turbo tasks; architecture-guards)
  • npm run docs:check — pass
  • npm run verify:test-discovery — pass (“Discovered all 843 test files exactly once.”)

Compatibility, safety, and release

  • Breaking changes or migration steps: none. Additive format=raw-binary plus optional architecture/entry. Omitted format still uses header-based detection. Existing dos-com and headered targets are unchanged.
  • Real Hopper/Ghidra, browser, or OS coverage: not run. Hopper/IDA refusals and Ghidra argv/profile/attestation are covered with fixtures. Live BinaryLoader import remains a proof gap.
  • Package or release metadata impact: skill version 39; packaged files include bridge/ghidra/ReaGhidraPrepareRaw.java.
  • Security, privacy, process, or containment review: caller-selected path, architecture, and entry are preserved. Ghidra import stays ephemeral and source-immutable. Transport credentials are not introduced.

Review checklist

  • This PR addresses a concrete problem or an agreed enhancement.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1868 **Author:** [@mvanhorn](https://github.com/mvanhorn) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/raw-binary-architecture` --- ### 📝 Commits (4) - [`04cbf4d`](https://github.com/morluto/rea/commit/04cbf4d394a8d94f55e657d73235bd43b67c9825) feat(native): admit raw binaries with caller-supplied architecture and entry - [`90da1ce`](https://github.com/morluto/rea/commit/90da1cec77c551a88b6dc288ae57d0b5f62286ad) fix(native): honor raw-binary preparation and entry identity - [`039c9c9`](https://github.com/morluto/rea/commit/039c9c9f89e98a7958cda3cd6b70bb42510ab3cb) fix(ghidra): stream raw snapshot attestation with cancellation - [`0e58a88`](https://github.com/morluto/rea/commit/0e58a8846cfb9dc5c7405e9483ed28965e08e656) fix(native): preserve and validate raw lifecycle entries ### 📊 Changes **57 files changed** (+2452 additions, -335 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+1 -1) 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+7 -0) 📝 `bridge/ghidra/ReaGhidraBridge.java` (+11 -4) ➕ `bridge/ghidra/ReaGhidraPrepareRaw.java` (+36 -0) 📝 `docs/.vitepress/config.ts` (+1 -0) 📝 `docs/cli.md` (+2 -1) ➕ `docs/raw-binary.md` (+72 -0) 📝 `docs/testing.md` (+8 -0) 📝 `package.json` (+2 -0) 📝 `scripts/verify-packaged-ghidra-windows.mjs` (+1 -0) ➕ `scripts/verify-real-ghidra-raw.mjs` (+219 -0) 📝 `src/application/BinaryTargetResolver.ts` (+53 -3) 📝 `src/application/DirectAnalysis.ts` (+37 -0) 📝 `src/application/binary/BinarySessionOpen.ts` (+9 -0) 📝 `src/application/binary/BinarySessionPort.ts` (+6 -1) 📝 `src/application/binary/BinarySessionStatus.ts` (+2 -5) ➕ `src/application/binary/BinaryTargetSummary.ts` (+11 -0) 📝 `src/cli/coreBinaryCommands.ts` (+22 -66) 📝 `src/cli/coreNativeCommands.ts` (+22 -83) 📝 `src/cli/options.ts` (+55 -1) _...and 37 more files_ </details> ### 📄 Description ## Summary Callers can now open a headerless executable by supplying the CPU family and the address where file offset 0 is mapped. Header-based detection is unchanged when `format` is omitted. Hopper and IDA refuse this interpretation; Ghidra imports it with `BinaryLoader` and a little-endian generic language, then seeds an `entry` function before auto-analysis. Fixes #1836 ## Demo ![demo](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/raw-binary-architecture/demo.gif) <video src="https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/raw-binary-architecture/demo.mp4" controls width="720"></video> ## Problem and expected behavior A local image without ELF, Mach-O, PE, or MZ headers is rejected as `truncated or unsupported binary header`. Embedded firmware often has those headers removed, so inventory and function analysis cannot start even when the caller already knows the CPU family and load address. `format=raw-binary` with `architecture` (`x86`, `x86_64`, `arm`, or `arm64`) and `entry_point` (`0x` hex) admits the complete file as an executable. File offset 0 maps at that address. Architecture or entry without `format=raw-binary` is rejected. The analysis profile includes the bound Ghidra language and canonical entry, so a different architecture or entry is a different snapshot identity. ## Change and scope New domain admission (`src/domain/rawBinary.ts`) composed through `parseBinaryTarget`, session open, and shared CLI/MCP adapters: - `open_binary` remains a single `type: object` input. Pairing is enforced with `superRefine`, not an input `anyOf`. - CLI `--target-format raw-binary --architecture … --entry-point …` on native inspect, function, and load-image commands. - Ghidra profile: `BinaryLoader`, `language_selection: caller-supplied-v1`, canonical `entry_point`, `load_image_evidence: independent-raw-mapping-context-v1`. Languages: `x86:LE:32:default`, `x86:LE:64:default`, `ARM:LE:32:v7`, `AARCH64:LE:64:v8A`. - Packaged `ReaGhidraPrepareRaw.java` maps file offset 0 at the entry, then `addEntryPoint` / `disassemble` / `createFunction("entry")` before auto-analysis. - Hopper and IDA return `target_format_unsupported` with a Ghidra-directed reason. Windows Ghidra P0 remains PE-only (applications and DLLs). - Independent load-image attestation covers source identity, mapped bytes, language, and caller-supplied entry. Original executable bytes are unchanged. - Authored skill text in `.agents/skills/reverse-engineer-anything/` (skill version 39). Catalog regenerated; build outputs not committed. Out of scope: a load address independent of entry, big-endian / Thumb-only / Cortex-M / PowerPC languages, analyzer overrides, guessing vector tables or memory maps, and a new MCP tool. ## Contract and boundary impact - Semantic owner and earliest changed stage: domain `src/domain/rawBinary.ts`, then `parseBinaryTarget` / session open, CLI/MCP adapters, Ghidra `BinaryLoader` + prepare script. - CLI and MCP/tool-catalog contract: `open_binary` gains optional `architecture` and `entry_point`; `format` enum adds `raw-binary`. CLI `--architecture` and `--entry-point` join `--target-format`. Output format enums and native load-image results add a `raw-binary` verified/mismatch variant. Input schema stays a single object. - Provider, bridge, target-format, or platform compatibility: Ghidra Linux/macOS admit raw-binary; Hopper and IDA refuse; Windows Ghidra P0 still PE-only. Packaged `bridge/ghidra/ReaGhidraPrepareRaw.java`. - Evidence, artifact, provenance, or reconstruction contract: snapshot/profile identity includes canonical entry and language selection; source spelling is preserved on the target as `entryPointSource` and does not change the digest. Load-image checks keep observed vs expected facts distinct. - Process execution, authorization, cleanup, or containment impact: ephemeral Ghidra import remains read-only and delete-on-exit; no executable-byte writes. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs): skill version 39 in `.agents/skills/reverse-engineer-anything/` and `src/generatedPackageMetadata.ts`; `package.json` files list includes `ReaGhidraPrepareRaw.java`. ## Evidence and regression coverage - Tests added or updated: domain admission, pairing, and address-width overflow; `open_binary` input contract; filesystem target resolution including misleading suffixes; Ghidra language map, profile digest, Windows P0 refusal, launcher argv, and load-image attestation; Hopper/IDA `target_format_unsupported`; CLI inspect before/missing-flags/Hopper-refusal; MCP open success and omitted architecture. - Base reproduction or other evidence: on `58767790`, `rea inspect` of a four-byte headerless image reports `truncated or unsupported binary header`; `--target-format raw-binary` is not a format hint until architecture and entry are also supplied. - User-visible CLI/MCP output (if applicable): generated local `ecu.bin` (`1eff2fe1`). Inspect without flags: `target_unavailable` / `truncated or unsupported binary header`. `--target-format raw-binary` alone: `raw-binary format requires architecture (x86, x86_64, arm, or arm64)`. Full flags with `--provider hopper`: `capability_unavailable` / `target_format_unsupported` (“Headerless raw-binary analysis requires the Ghidra adapter with a caller-supplied architecture and entry point.”), not `unsupported binary format`. Architecture without format: `architecture and entry_point apply only with format=raw-binary`. MCP `open_binary` with `format`, `architecture`, and `entry_point` returns `format: "raw-binary"`. - Remaining proof gaps: real Ghidra import of a caller-supplied raw-binary image was not run on this host. No `verify:ghidra:raw` lane in this increment. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm ci` — pass (377 packages added, 378 audited) - `npm run test:focused -- src/domain/rawBinary.test.ts src/contracts/sessionLifecycleInputs.test.ts src/ghidra/GhidraRawBinaryLoadImage.test.ts src/ghidra/GhidraProvider.behavior.test.ts src/hopper/HopperProvider.test.ts src/ida/IdaProvider.test.ts tests/boundary/cli/rawBinaryCli.test.ts tests/boundary/mcp/rawBinaryMcp.test.ts tests/boundary/filesystem/rawBinaryTarget.test.ts tests/boundary/providers/ghidra/ghidraLauncher.test.ts tests/boundary/providers/hopper/hopperAnalysisProfile.test.ts` — pass (11 files, 98 passed, 1 skipped) - `npm run check:fast` — pass - `npm run check` — pass (typecheck, lint, format, knip, metadata:check; 6/6 turbo tasks; architecture-guards) - `npm run docs:check` — pass - `npm run verify:test-discovery` — pass (“Discovered all 843 test files exactly once.”) ## Compatibility, safety, and release - Breaking changes or migration steps: none. Additive `format=raw-binary` plus optional architecture/entry. Omitted `format` still uses header-based detection. Existing `dos-com` and headered targets are unchanged. - Real Hopper/Ghidra, browser, or OS coverage: not run. Hopper/IDA refusals and Ghidra argv/profile/attestation are covered with fixtures. Live BinaryLoader import remains a proof gap. - Package or release metadata impact: skill version 39; packaged files include `bridge/ghidra/ReaGhidraPrepareRaw.java`. - Security, privacy, process, or containment review: caller-selected path, architecture, and entry are preserved. Ghidra import stays ephemeral and source-immutable. Transport credentials are not introduced. ## Review checklist - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 00:56:09 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#4864
No description provided.