[PR #741] [MERGED] feat(firmware): integrate Binwalk and Unblob analysis #2207

New issue

Closed

opened 2026-10-16 21:15:41 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:15:41 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/741
Author: @N0zoM1z0
Created: 10/6/2026
Status: ✅ Merged
Merged: 10/6/2026
Merged by: @N0zoM1z0

Base: main ← Head: feat/firmware-analysis-adapters

📝 Commits (5)

6354fe4 feat(firmware): integrate Binwalk and Unblob analysis
a7fa0a2 fix(firmware): verify published files and namespace source references
247e6df test(firmware): isolate extractor dependency failure from process cleanup
a81f558 fix(firmware): retain actionable report and cleanup diagnostics
7bc98cd test: assert retained batch output failure diagnostics

📊 Changes

58 files changed (+3230 additions, -16 deletions)

View changed files

📝 .gitmodules (+6 -0)
📝 .jscpd.json (+1 -1)
📝 .oxfmtrc.json (+1 -1)
📝 .oxlintrc.json (+1 -1)
📝 README.md (+5 -0)
📝 README_ar.md (+1 -0)
📝 README_ja.md (+1 -0)
📝 README_ko.md (+1 -0)
📝 README_zh.md (+1 -0)
📝 docs/architecture.mermaid (+4 -0)
➕ docs/firmware-analysis.md (+173 -0)
📝 docs/product-catalog.json (+27 -7)
📝 docs/testing.md (+12 -0)
📝 docs/verification/managed-conformance-manifest.json (+2 -2)
📝 package.json (+2 -0)
➕ scripts/generate-firmware-fixtures.mjs (+42 -0)
📝 scripts/lib/catalog-builders.mjs (+17 -0)
📝 scripts/lib/catalog-core.mjs (+2 -0)
➕ scripts/lib/generate-firmware-fixture.py (+43 -0)
➕ scripts/verify-real-firmware.mjs (+254 -0)

...and 38 more files

📄 Description

Changes

First firmware integration milestone for #718:

Add inspect_firmware_regions (Binwalk 3.1.0) and extract_firmware (Unblob 26.6.4), sharing application workflows across CLI and MCP.
Preserve unmodified MIT upstream sources as optional submodules at fixed release commits. Engines/extractors remain bring-your-own; npm packages include neither upstream source nor binaries.
Bind observations to private source snapshots and digests. Retain Binwalk's reported lengths with explicit unknown validation; retain unordered Unblob task lineage, root byte ranges, unknown/encrypted chunks, depth stops and extractor diagnostics. Decompressed offsets and runtime addresses remain unknown.
Publish regular files into an exclusively created output directory with SHA-256 readback. Preserve link/special-file metadata without publishing them; never overwrite existing output or execute extracted programs.
Preserve stderr/exit status when failed extraction produces no usable report, and retain cleanup process coordinates. Keep typed output constraints in shared CLI/MCP error details without exposing internal causes.
Queue expensive operations; use one engine worker, Linux prlimit, a deadline and checked staging/report budgets. Keep Unblob's default sandbox enabled. Cancellation verifies owned process cleanup; uncertain cleanup retains resources and blocks later launches.
Update exact named contracts, capability availability, generated catalog, localized inventories, architecture and firmware/testing/provenance guides.

This milestone is Linux-only. Memory/file limits are per process/file; staging budgets are polled and may transiently overshoot, rather than aggregate quotas. Generated gzip/USTAR/ext4 and a host ELF are verified; SquashFS/UBIFS, MIPS/PPC/RISC-V, raw flash loading and emulation remain follow-ups. #718 stays open.

MCP examples

{"name":"inspect_firmware_regions","arguments":{"path":"/tmp/firmware.bin"}}

{"name":"extract_firmware","arguments":{"path":"/tmp/firmware.bin","output_directory":"/tmp/new-output","range":{"offset":64,"length":4096},"max_depth":3}}

Results include inline Evidence and child paths/digests. Select a supported native child separately with open_binary and provider_id:"ghidra".

Verification

npm run check:fast, generated-document checks, formatting and Knip.
85 distinct focused regressions across firmware provider/lifecycle, report normalization, MCP, capability, canonical catalog and SDK-converted JSON Schema validity. Includes nonexistent reported output, reserved filename $input, cancellation while queued, uncertain cleanup blocking and batch error projection.
npm run fixtures:firmware; real CLI/MCP Binwalk/Unblob verification with independent offset/hash oracle, whole-file and selected-range extraction, unknown chunks and depth limits.
Optional real ext4/debugfs lane, including useful partial diagnostics when debugfs is unavailable (Unblob exit 1).
Real Ghidra 12.1.4 handoff: extracted ELF digest, function dossier and direct pseudocode provider identity. No real Hopper verification: no Hopper integration changes.
Isolated npm pack/install (754 shipped files; no firmware/upstream/tool binaries) and the same real firmware + ext4 + Ghidra workflow against its installed CLI/MCP entrypoint.

No firmware, private game data, generated binaries, toolchain installation or host configuration changes are committed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/741 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/6/2026 **Status:** ✅ Merged **Merged:** 10/6/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `feat/firmware-analysis-adapters` --- ### 📝 Commits (5) - [`6354fe4`](https://github.com/morluto/rea/commit/6354fe43158499155511580f1ed9f4cdca407dca) feat(firmware): integrate Binwalk and Unblob analysis - [`a7fa0a2`](https://github.com/morluto/rea/commit/a7fa0a24ee6d2489bd8f6a38d9a3008bb54e9157) fix(firmware): verify published files and namespace source references - [`247e6df`](https://github.com/morluto/rea/commit/247e6dfa32a00e8dd3a9c7eb86ccbbb63f2b5394) test(firmware): isolate extractor dependency failure from process cleanup - [`a81f558`](https://github.com/morluto/rea/commit/a81f558cf687b83156f3ca798c1b17efb91ffb34) fix(firmware): retain actionable report and cleanup diagnostics - [`7bc98cd`](https://github.com/morluto/rea/commit/7bc98cd657585cd5ab093587db28bf9914469d56) test: assert retained batch output failure diagnostics ### 📊 Changes **58 files changed** (+3230 additions, -16 deletions) <details> <summary>View changed files</summary> 📝 `.gitmodules` (+6 -0) 📝 `.jscpd.json` (+1 -1) 📝 `.oxfmtrc.json` (+1 -1) 📝 `.oxlintrc.json` (+1 -1) 📝 `README.md` (+5 -0) 📝 `README_ar.md` (+1 -0) 📝 `README_ja.md` (+1 -0) 📝 `README_ko.md` (+1 -0) 📝 `README_zh.md` (+1 -0) 📝 `docs/architecture.mermaid` (+4 -0) ➕ `docs/firmware-analysis.md` (+173 -0) 📝 `docs/product-catalog.json` (+27 -7) 📝 `docs/testing.md` (+12 -0) 📝 `docs/verification/managed-conformance-manifest.json` (+2 -2) 📝 `package.json` (+2 -0) ➕ `scripts/generate-firmware-fixtures.mjs` (+42 -0) 📝 `scripts/lib/catalog-builders.mjs` (+17 -0) 📝 `scripts/lib/catalog-core.mjs` (+2 -0) ➕ `scripts/lib/generate-firmware-fixture.py` (+43 -0) ➕ `scripts/verify-real-firmware.mjs` (+254 -0) _...and 38 more files_ </details> ### 📄 Description ## Changes First firmware integration milestone for #718: - Add `inspect_firmware_regions` (Binwalk 3.1.0) and `extract_firmware` (Unblob 26.6.4), sharing application workflows across CLI and MCP. - Preserve unmodified MIT upstream sources as optional submodules at fixed release commits. Engines/extractors remain bring-your-own; npm packages include neither upstream source nor binaries. - Bind observations to private source snapshots and digests. Retain Binwalk's reported lengths with explicit unknown validation; retain unordered Unblob task lineage, root byte ranges, unknown/encrypted chunks, depth stops and extractor diagnostics. Decompressed offsets and runtime addresses remain unknown. - Publish regular files into an exclusively created output directory with SHA-256 readback. Preserve link/special-file metadata without publishing them; never overwrite existing output or execute extracted programs. - Preserve stderr/exit status when failed extraction produces no usable report, and retain cleanup process coordinates. Keep typed output constraints in shared CLI/MCP error details without exposing internal causes. - Queue expensive operations; use one engine worker, Linux `prlimit`, a deadline and checked staging/report budgets. Keep Unblob's default sandbox enabled. Cancellation verifies owned process cleanup; uncertain cleanup retains resources and blocks later launches. - Update exact named contracts, capability availability, generated catalog, localized inventories, architecture and firmware/testing/provenance guides. This milestone is Linux-only. Memory/file limits are per process/file; staging budgets are polled and may transiently overshoot, rather than aggregate quotas. Generated gzip/USTAR/ext4 and a host ELF are verified; SquashFS/UBIFS, MIPS/PPC/RISC-V, raw flash loading and emulation remain follow-ups. #718 stays open. ## MCP examples ```json {"name":"inspect_firmware_regions","arguments":{"path":"/tmp/firmware.bin"}} ``` ```json {"name":"extract_firmware","arguments":{"path":"/tmp/firmware.bin","output_directory":"/tmp/new-output","range":{"offset":64,"length":4096},"max_depth":3}} ``` Results include inline Evidence and child paths/digests. Select a supported native child separately with `open_binary` and `provider_id:"ghidra"`. ## Verification - `npm run check:fast`, generated-document checks, formatting and Knip. - 85 distinct focused regressions across firmware provider/lifecycle, report normalization, MCP, capability, canonical catalog and SDK-converted JSON Schema validity. Includes nonexistent reported output, reserved filename `$input`, cancellation while queued, uncertain cleanup blocking and batch error projection. - `npm run fixtures:firmware`; real CLI/MCP Binwalk/Unblob verification with independent offset/hash oracle, whole-file and selected-range extraction, unknown chunks and depth limits. - Optional real ext4/debugfs lane, including useful partial diagnostics when debugfs is unavailable (Unblob exit 1). - Real Ghidra 12.1.4 handoff: extracted ELF digest, function dossier and direct pseudocode provider identity. No real Hopper verification: no Hopper integration changes. - Isolated npm pack/install (754 shipped files; no firmware/upstream/tool binaries) and the same real firmware + ext4 + Ghidra workflow against its installed CLI/MCP entrypoint. No firmware, private game data, generated binaries, toolchain installation or host configuration changes are committed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-16 21:15:41 +00:00