[GH-ISSUE #718] [Feature] Integrate IoT firmware analysis tools into REA: research and roadmap #316

New issue

Closed

opened 2026-10-15 18:23:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:23:49 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/718

Goal

Add IoT firmware analysis to REA by integrating mature identification, extraction, metadata, native-analysis, and runtime tools. Make their results composable for agents through the same CLI/MCP workflows.

This issue records the research and a staged roadmap under REA — One MCP for reverse engineering across binaries, applications, and runtime behavior. Related roadmap: #32.

Distinguish the inputs

Input	Analysis path
Linux-device update/flash image	Identify containers/regions → selected rootfs extraction → settings/artifacts/ELF → existing native analysis
MCU/bare-metal image	Explicit CPU/endian/base/entry/memory map → native inspection; target-specific runtime later
Raw NAND/hardware dump	Explicit page/OOB/ECC/geometry handling → logical image; do not assume ordinary filesystem extraction covers it

A .bin extension does not establish any of these semantics. Inspection of an arbitrary blob is distinct from admitting it as a supported executable.

Recommended first integrations

Tool	Useful capability	REA integration
Binwalk v3	Signatures, offsets, entropy, JSON log	Region inspection with evidence and unknowns
Unblob	Recursive extraction, chunk boundaries, unknown chunks, JSON reports	Derived-artifact graph, errors, scoped extraction
Existing REA artifact/Ghidra providers	Identity, selected extraction, supported native analysis	Reuse for extracted native files

Prefer one selected extraction engine per operation. Binwalk and Unblob overlap; verify whether both are necessary for a particular outcome rather than running both automatically.

Version and resource findings

Binwalk v3 is a Rust rewrite under MIT. Research observed latest GitHub release v3.1.0 (2024-10-31) and default-branch activity in 2026. The release and main differ in CLI/log behavior. Bind to the actual version; do not reuse Binwalk v2 Python/CLI assumptions.
v3.1.0 exposes --log and --threads; analysis results carry file_path, file_map, and extractions keyed by signature identity. Validate empty, multiple, interrupted, and partial JSON with the real selected version. Release CLI, result source.
Unblob is MIT; research observed release 26.6.4. It has JSON reports and external-extractor dependencies; installing its Python package alone does not establish all-format coverage. Check release-specific behavior instead of assuming current main features are available. Installation, API.
Both have parallel execution settings. Start with one worker, bounded workspace/input handling, and owned-descendant cleanup. Binwalk main reads file data into memory; one thread is not an RSS bound. Report actual budget enforcement and partial results.

Format-specific helpers and metadata

Candidate	Use
squashfs-tools, ONEKEY sasquatch	Standard and vendor SquashFS inspection/extraction
ONEKEY UBI Reader	UBI/UBIFS blocks, volumes, extraction; explicit geometry/overrides
ONEKEY Jefferson	JFFS2; preserve the tool's actual CRC coverage
dtc/libfdt	DT nodes/properties, memory/partition descriptions, raw cells/phandles
U-Boot dumpimage	Boot component/configuration metadata and selected extraction

Use these as extraction dependencies or distinct metadata providers when they answer an additional question. Avoid duplicating identical extraction tools in the MCP inventory. Listing a boot image does not prove its signature is verified. Report declared load/entry metadata separately from actual imported bytes.

Confirmed REA gaps

Research baseline: main af48fe879161.

BinaryArchitecture currently contains x86/x86_64/arm/arm64. MIPS is absent; PPC/RISC-V are also absent. Ghidra's upstream processor support does not establish REA admission, contracts, or real workflow coverage.
GhidraAnalysisProfile uses header-based import plus explicit DOS handling. Raw flash needs caller-owned processor/endian, file-to-memory regions, entry/context, and loaded-image verification.

Staged work

First: firmware region inspection through a versioned Binwalk v3 adapter, without implicit recursive extraction.
First: caller-selected Unblob extraction, normalized JSON errors/unknown chunks, and derived-artifact provenance.
First: hand extracted supported native files to existing admission/Ghidra workflows.
Follow-up: complete MIPS ELF support across header parsing, domain/contracts, target admission, provider profile, and real Ghidra verification; select exact endian/ABI variants.
Follow-up: boot component/configuration and DT metadata through dumpimage/libfdt/dtc.
Follow-up: explicit raw-firmware load profiles and loaded-image attestation; select exact processors/context support.
Later: selected ELF/rootfs observation through Qiling; known-machine guests through QEMU/QMP.
Later: selected results from a caller-provided local FACT instance or EMBA reports.

Evidence and operational requirements

Preserve parent/child digests, original paths, chunk ranges, handler/provider/version, errors, unknown regions, and actionable missing dependencies.
Separate parent-file offsets, decompressed child offsets, and runtime addresses. Preserve derivation even when a byte-level mapping is unknown; compression does not produce a one-to-one offset mapping.
Signature hits and entropy are observations/inferences with limitations, not proof of a valid filesystem or encryption.
Reuse shared application workflows, named contracts, existing artifact relationships, and process lifecycle primitives. Keep provider protocols outside domain/application code.
Bring-your-own tools and isolated workspaces; no automatic root installers, host mounts, global configuration changes, or execution of extracted scripts during inspection. Diagnose only dependencies needed for the selected outcome.
Component licenses need checking before code reuse/redistribution. For example, dtc and libfdt have different licensing; legacy firmware-mod-kit has unresolved mixed components.

Other useful research references

Firmadyne/FirmAE provide Linux rehosting prior art; neither was archived in the research snapshot. Unicorn is a CPU library, not a full OS. avatar² and HALucinator address target-specific hardware/HAL boundaries. FirmWire is specialized baseband tooling. Keep these for observed needs and specific fixtures.

FACT's REST API is unversioned: pin representative producer data for a local connector. Heavy platforms should remain optional. Firmware Mod Kit, ImHex, Rizin, radare2, and LIEF are further format/interface references, not an initial mandatory stack.

Acceptance for initial milestones

Small source-generated fixtures have known regions, padding/unknown bytes, compressed filesystem content, and selected ELF children; compare offsets/bytes/digests to an independent generator oracle.
Exercise missing extractors, partial extraction, malformed inputs, links/path boundaries, duplicate names, depth/resource termination, and cancellation. Preserve useful partial evidence.
Real Binwalk/Unblob and selected native Ghidra workflows through CLI/MCP, with recorded budgets and unchanged source bytes.
Real MIPS/raw/runtime lanes are required before claiming their respective coverage; cross-toolchain prerequisites belong only to the lane using them.
Track fixture generators/source, not private firmware, generated binaries, or project databases. Use focused iteration checks; full CI remains a separate gate.

Research status: documentation, selected upstream code, and version/license metadata reviewed. No firmware extraction/emulation or candidate-tool execution has been performed. Deliver the roadmap as small independently useful PRs.

Originally created by @N0zoM1z0 on GitHub (Oct 6, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/718 ## Goal Add IoT firmware analysis to REA by integrating mature identification, extraction, metadata, native-analysis, and runtime tools. Make their results composable for agents through the same CLI/MCP workflows. This issue records the research and a staged roadmap under **REA — One MCP for reverse engineering across binaries, applications, and runtime behavior**. Related roadmap: #32. ## Distinguish the inputs | Input | Analysis path | | --- | --- | | Linux-device update/flash image | Identify containers/regions → selected rootfs extraction → settings/artifacts/ELF → existing native analysis | | MCU/bare-metal image | Explicit CPU/endian/base/entry/memory map → native inspection; target-specific runtime later | | Raw NAND/hardware dump | Explicit page/OOB/ECC/geometry handling → logical image; do not assume ordinary filesystem extraction covers it | A `.bin` extension does not establish any of these semantics. Inspection of an arbitrary blob is distinct from admitting it as a supported executable. ## Recommended first integrations | Tool | Useful capability | REA integration | | --- | --- | --- | | [Binwalk v3](https://github.com/ReFirmLabs/binwalk) | Signatures, offsets, entropy, JSON log | Region inspection with evidence and unknowns | | [Unblob](https://github.com/onekey-sec/unblob) | Recursive extraction, chunk boundaries, unknown chunks, JSON reports | Derived-artifact graph, errors, scoped extraction | | Existing REA artifact/Ghidra providers | Identity, selected extraction, supported native analysis | Reuse for extracted native files | Prefer one selected extraction engine per operation. Binwalk and Unblob overlap; verify whether both are necessary for a particular outcome rather than running both automatically. ### Version and resource findings - Binwalk v3 is a Rust rewrite under MIT. Research observed latest GitHub release `v3.1.0` (2024-10-31) and default-branch activity in 2026. The release and main differ in CLI/log behavior. Bind to the actual version; do not reuse Binwalk v2 Python/CLI assumptions. - `v3.1.0` exposes `--log` and `--threads`; analysis results carry `file_path`, `file_map`, and extractions keyed by signature identity. Validate empty, multiple, interrupted, and partial JSON with the real selected version. [Release CLI](https://github.com/ReFirmLabs/binwalk/blob/v3.1.0/src/cliparser.rs), [result source](https://github.com/ReFirmLabs/binwalk/blob/v3.1.0/src/binwalk.rs). - Unblob is MIT; research observed release `26.6.4`. It has JSON reports and external-extractor dependencies; installing its Python package alone does not establish all-format coverage. Check release-specific behavior instead of assuming current main features are available. [Installation](https://unblob.org/installation/), [API](https://unblob.org/api/). - Both have parallel execution settings. Start with one worker, bounded workspace/input handling, and owned-descendant cleanup. Binwalk main reads file data into memory; one thread is not an RSS bound. Report actual budget enforcement and partial results. ## Format-specific helpers and metadata | Candidate | Use | | --- | --- | | [squashfs-tools](https://github.com/plougher/squashfs-tools), [ONEKEY sasquatch](https://github.com/onekey-sec/sasquatch) | Standard and vendor SquashFS inspection/extraction | | [ONEKEY UBI Reader](https://github.com/onekey-sec/ubi_reader) | UBI/UBIFS blocks, volumes, extraction; explicit geometry/overrides | | [ONEKEY Jefferson](https://github.com/onekey-sec/jefferson) | JFFS2; preserve the tool's actual CRC coverage | | [dtc/libfdt](https://github.com/dgibson/dtc) | DT nodes/properties, memory/partition descriptions, raw cells/phandles | | [U-Boot dumpimage](https://github.com/u-boot/u-boot) | Boot component/configuration metadata and selected extraction | Use these as extraction dependencies or distinct metadata providers when they answer an additional question. Avoid duplicating identical extraction tools in the MCP inventory. Listing a boot image does not prove its signature is verified. Report declared load/entry metadata separately from actual imported bytes. ## Confirmed REA gaps Research baseline: main `af48fe879161`. 1. [BinaryArchitecture](https://github.com/morluto/rea/blob/af48fe879161/src/domain/binaryTargetTypes.ts) currently contains x86/x86_64/arm/arm64. **MIPS is absent**; PPC/RISC-V are also absent. Ghidra's upstream processor support does not establish REA admission, contracts, or real workflow coverage. 2. [GhidraAnalysisProfile](https://github.com/morluto/rea/blob/af48fe879161/src/ghidra/GhidraAnalysisProfile.ts) uses header-based import plus explicit DOS handling. Raw flash needs caller-owned processor/endian, file-to-memory regions, entry/context, and loaded-image verification. ## Staged work - [ ] **First:** firmware region inspection through a versioned Binwalk v3 adapter, without implicit recursive extraction. - [ ] **First:** caller-selected Unblob extraction, normalized JSON errors/unknown chunks, and derived-artifact provenance. - [ ] **First:** hand extracted supported native files to existing admission/Ghidra workflows. - [ ] **Follow-up:** complete MIPS ELF support across header parsing, domain/contracts, target admission, provider profile, and real Ghidra verification; select exact endian/ABI variants. - [ ] **Follow-up:** boot component/configuration and DT metadata through dumpimage/libfdt/dtc. - [ ] **Follow-up:** explicit raw-firmware load profiles and loaded-image attestation; select exact processors/context support. - [ ] **Later:** selected ELF/rootfs observation through [Qiling](https://github.com/qilingframework/qiling); known-machine guests through [QEMU/QMP](https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html). - [ ] **Later:** selected results from a caller-provided local [FACT](https://github.com/fkie-cad/FACT_core) instance or [EMBA](https://github.com/e-m-b-a/emba) reports. ## Evidence and operational requirements - Preserve parent/child digests, original paths, chunk ranges, handler/provider/version, errors, unknown regions, and actionable missing dependencies. - Separate parent-file offsets, decompressed child offsets, and runtime addresses. Preserve derivation even when a byte-level mapping is unknown; compression does not produce a one-to-one offset mapping. - Signature hits and entropy are observations/inferences with limitations, not proof of a valid filesystem or encryption. - Reuse shared application workflows, named contracts, existing artifact relationships, and process lifecycle primitives. Keep provider protocols outside domain/application code. - Bring-your-own tools and isolated workspaces; no automatic root installers, host mounts, global configuration changes, or execution of extracted scripts during inspection. Diagnose only dependencies needed for the selected outcome. - Component licenses need checking before code reuse/redistribution. For example, dtc and libfdt have different licensing; legacy firmware-mod-kit has unresolved mixed components. ## Other useful research references [Firmadyne](https://github.com/firmadyne/firmadyne)/[FirmAE](https://github.com/pr0v3rbs/FirmAE) provide Linux rehosting prior art; neither was archived in the research snapshot. [Unicorn](https://github.com/unicorn-engine/unicorn) is a CPU library, not a full OS. [avatar²](https://github.com/avatartwo/avatar2) and [HALucinator](https://github.com/halucinator/halucinator) address target-specific hardware/HAL boundaries. [FirmWire](https://github.com/FirmWire/FirmWire) is specialized baseband tooling. Keep these for observed needs and specific fixtures. FACT's [REST API](https://github.com/fkie-cad/FACT_core/wiki/Rest-API) is unversioned: pin representative producer data for a local connector. Heavy platforms should remain optional. [Firmware Mod Kit](https://github.com/rampageX/firmware-mod-kit), [ImHex](https://github.com/WerWolv/ImHex), [Rizin](https://github.com/rizinorg/rizin), [radare2](https://github.com/radareorg/radare2), and [LIEF](https://github.com/lief-project/LIEF) are further format/interface references, not an initial mandatory stack. ## Acceptance for initial milestones - Small source-generated fixtures have known regions, padding/unknown bytes, compressed filesystem content, and selected ELF children; compare offsets/bytes/digests to an independent generator oracle. - Exercise missing extractors, partial extraction, malformed inputs, links/path boundaries, duplicate names, depth/resource termination, and cancellation. Preserve useful partial evidence. - Real Binwalk/Unblob and selected native Ghidra workflows through CLI/MCP, with recorded budgets and unchanged source bytes. - Real MIPS/raw/runtime lanes are required before claiming their respective coverage; cross-toolchain prerequisites belong only to the lane using them. - Track fixture generators/source, not private firmware, generated binaries, or project databases. Use focused iteration checks; full CI remains a separate gate. **Research status:** documentation, selected upstream code, and version/license metadata reviewed. No firmware extraction/emulation or candidate-tool execution has been performed. Deliver the roadmap as small independently useful PRs.

bsakaguchi

2026-10-15 18:23:49 +00:00