[GH-ISSUE #4245] [Feature] Propose CPU architecture and endianness candidates for headerless firmware blobs #1671

New issue

Open

opened 2026-10-16 21:08:49 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:08:49 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4245

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

An analyst who extracts a headerless image (a kernel or bootloader blob from a router dump, an RTOS flash image) must already know its CPU family and byte order before REA can analyze it. open_binary format=raw-binary requires a caller-supplied architecture and entry_point, and docs/raw-binary.md states that the interpretation "is not inferred from a .bin suffix or from unrecognized bytes". inspect_firmware_regions passes Binwalk's free-text description through without parsing any CPU facts (src/firmware/FirmwareReports.ts:150-164), and Binwalk 3.x has no opcode-based architecture scan. inspect_binary_layout covers ELF64 x86-64 only, and search_bytes needs an open provider session, which already needs an architecture. Today the analyst leaves REA to guess the CPU with other tools or trial imports.

Use cases

A U-Boot uImage inside a vendor firmware container: its 64-byte header declares the CPU in ih_arch, but REA does not surface it as a structured fact.
A squashfs or initramfs carved from a flash dump contains many ELF executables: their e_machine values agree on the device's CPU, which also identifies the CPU of a sibling headerless kernel or bootloader blob.
A bare-metal blob with no header at all: frequent return or prologue encodings (ARM bx lr, AArch64 ret, MIPS jr ra, PowerPC blr) indicate the instruction set and byte order with useful, but weaker, confidence.

Desired outcome and success criteria

A read-only, path-based operation that accepts one local file and returns ranked CPU candidates without opening a provider session or changing open_binary's explicit-choice contract. Success criteria:

Each candidate reports a CPU family, word size, byte order (or unknown when the source does not declare it), the detector that produced it, its confidence class, and Evidence locations as file-offset ranges.
A U-Boot uImage header found at any offset is parsed directly: magic 27 05 19 56, header CRC, ih_arch, ih_comp, ih_load, and ih_ep, keeping the raw ih_arch byte beside its name. The mapping follows U-Boot's include/image.h numbering, not Binwalk's description text.
Embedded ELF headers are validated (class, data, version) and their e_machine values are tallied through REA's existing ELF machine mapping. The result reports the counts, and reports "mixed" rather than a winner when no clear majority exists.
An instruction-signature census reports per-signature hit counts, alignment, and the runner-up, so a reader can judge the margin.
When no detector produces a candidate, the result is an explicit unknown with the detectors run and their coverage, never "none".
Each candidate states whether it maps to an open_binary format=raw-binary architecture accepted today (currently x86, x86_64, arm, arm64, 6502) and suggests those arguments without applying them.
The uImage load address and entry point are reported as declared header facts, and the result states that they describe the payload after the 64-byte header (possibly compressed per ih_comp), not a file-offset-0 mapping of the input.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Static byte inspection of one caller-selected local file. No execution, no extraction, no network, no writes beyond REA's normal Evidence retention. Non-goals: automatically choosing an architecture inside open_binary, guessing base addresses, vector tables, or memory maps, and adding new raw-binary architectures, which is a separate decision.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Self-built samples are sufficient: mkimage -A arm64 and mkimage -A mips uImages wrapping a small payload; a cpio archive holding several cross-compiled ELF executables for one CPU plus one outlier; objcopy -O binary text sections from cross-compiled ARM, AArch64, MIPS LE/BE, and PowerPC objects; and random bytes as a negative control that must produce an explicit unknown. The census thresholds and the scan cap should be justified by measurement on those samples, per AGENTS.md's performance rules.

Alternatives considered

Extending inspect_firmware_regions: it would tie a pure-byte detector to Binwalk on Linux, and its output has a literal coverage: "signature_scan".
Parsing Binwalk's uImage description: the description is free text, and Binwalk's uImage CPU table appears to be offset from U-Boot's numbering for values 9 and above (src/structures/uimage.rs in Binwalk compared with U-Boot include/image.h; source reading only, not observed by running Binwalk).
Auto-detection inside open_binary: contradicts the documented caller-supplied contract and its language_selection: caller-supplied-v1 profile.
External tools such as cpu_rec or Binwalk v2 -A: these would add an unpinned dependency, and Binwalk v2's opcode signatures were not carried into Binwalk 3.

Possible direction

A new session-free tool (for example inspect_raw_architecture_candidates) in pure TypeScript, reusing the ELF e_machine mapping in src/domain/binaryTarget.ts rather than adding a second table. Classify uImage and ELF facts as observed or derived, and the instruction census as inferred. A suggested detector order is a uImage header check, then an embedded-ELF majority vote (for example at least 75% agreement), then an opcode census (for example at least 10 hits and twice the runner-up) over a bounded prefix of the file, with the thresholds and the cap set by measurement. Validate the uImage header CRC, count only aligned instruction matches, and include Thumb, little-endian PowerPC, RISC-V, and Xtensa signatures alongside the ARM, AArch64, MIPS, big-endian PowerPC, and x86 ones.

References

REA baseline: main at 0cc56493be (package 6.5.0): docs/raw-binary.md, src/domain/rawBinary.ts:11-17, src/firmware/FirmwareProvider.ts:297-306, src/domain/binaryTarget.ts:532-566.
Related, not duplicates: #3714 (artifact discovery RFC, no ISA detection), #3621 (file-format inference), #3615 (ESP32 flash mapping), #3532 (SH-4 raw lane), #4110 (Binwalk-NG scope), #1836 and #1868 (caller-supplied raw-binary design).
Search: all 1,667 issues and 2,501 pull requests were listed and searched locally for uImage, U-Boot, opcode, endianness, ih_arch, headerless, "detect/infer architecture/cpu/processor", "raw firmware", and "binwalk -A"; no existing proposal found.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4245 ### Before submitting - [X] I searched the open and closed issues and pull requests for a related proposal. - [X] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [X] This request does not include credentials, private samples, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation An analyst who extracts a headerless image (a kernel or bootloader blob from a router dump, an RTOS flash image) must already know its CPU family and byte order before REA can analyze it. `open_binary format=raw-binary` requires a caller-supplied `architecture` and `entry_point`, and `docs/raw-binary.md` states that the interpretation "is not inferred from a `.bin` suffix or from unrecognized bytes". `inspect_firmware_regions` passes Binwalk's free-text `description` through without parsing any CPU facts (`src/firmware/FirmwareReports.ts:150-164`), and Binwalk 3.x has no opcode-based architecture scan. `inspect_binary_layout` covers ELF64 x86-64 only, and `search_bytes` needs an open provider session, which already needs an architecture. Today the analyst leaves REA to guess the CPU with other tools or trial imports. ### Use cases - A U-Boot uImage inside a vendor firmware container: its 64-byte header declares the CPU in `ih_arch`, but REA does not surface it as a structured fact. - A squashfs or initramfs carved from a flash dump contains many ELF executables: their `e_machine` values agree on the device's CPU, which also identifies the CPU of a sibling headerless kernel or bootloader blob. - A bare-metal blob with no header at all: frequent return or prologue encodings (ARM `bx lr`, AArch64 `ret`, MIPS `jr ra`, PowerPC `blr`) indicate the instruction set and byte order with useful, but weaker, confidence. ### Desired outcome and success criteria A read-only, path-based operation that accepts one local file and returns ranked CPU candidates without opening a provider session or changing `open_binary`'s explicit-choice contract. Success criteria: - Each candidate reports a CPU family, word size, byte order (or `unknown` when the source does not declare it), the detector that produced it, its confidence class, and Evidence locations as file-offset ranges. - A U-Boot uImage header found at any offset is parsed directly: magic `27 05 19 56`, header CRC, `ih_arch`, `ih_comp`, `ih_load`, and `ih_ep`, keeping the raw `ih_arch` byte beside its name. The mapping follows U-Boot's `include/image.h` numbering, not Binwalk's description text. - Embedded ELF headers are validated (class, data, version) and their `e_machine` values are tallied through REA's existing ELF machine mapping. The result reports the counts, and reports "mixed" rather than a winner when no clear majority exists. - An instruction-signature census reports per-signature hit counts, alignment, and the runner-up, so a reader can judge the margin. - When no detector produces a candidate, the result is an explicit unknown with the detectors run and their coverage, never "none". - Each candidate states whether it maps to an `open_binary format=raw-binary` architecture accepted today (currently `x86`, `x86_64`, `arm`, `arm64`, `6502`) and suggests those arguments without applying them. - The uImage load address and entry point are reported as declared header facts, and the result states that they describe the payload after the 64-byte header (possibly compressed per `ih_comp`), not a file-offset-0 mapping of the input. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) Static byte inspection of one caller-selected local file. No execution, no extraction, no network, no writes beyond REA's normal Evidence retention. Non-goals: automatically choosing an architecture inside `open_binary`, guessing base addresses, vector tables, or memory maps, and adding new raw-binary architectures, which is a separate decision. ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Self-built samples are sufficient: `mkimage -A arm64` and `mkimage -A mips` uImages wrapping a small payload; a cpio archive holding several cross-compiled ELF executables for one CPU plus one outlier; `objcopy -O binary` text sections from cross-compiled ARM, AArch64, MIPS LE/BE, and PowerPC objects; and random bytes as a negative control that must produce an explicit unknown. The census thresholds and the scan cap should be justified by measurement on those samples, per `AGENTS.md`'s performance rules. ### Alternatives considered - Extending `inspect_firmware_regions`: it would tie a pure-byte detector to Binwalk on Linux, and its output has a literal `coverage: "signature_scan"`. - Parsing Binwalk's uImage description: the description is free text, and Binwalk's uImage CPU table appears to be offset from U-Boot's numbering for values 9 and above (`src/structures/uimage.rs` in Binwalk compared with U-Boot `include/image.h`; source reading only, not observed by running Binwalk). - Auto-detection inside `open_binary`: contradicts the documented caller-supplied contract and its `language_selection: caller-supplied-v1` profile. - External tools such as `cpu_rec` or Binwalk v2 `-A`: these would add an unpinned dependency, and Binwalk v2's opcode signatures were not carried into Binwalk 3. ### Possible direction A new session-free tool (for example `inspect_raw_architecture_candidates`) in pure TypeScript, reusing the ELF `e_machine` mapping in `src/domain/binaryTarget.ts` rather than adding a second table. Classify uImage and ELF facts as observed or derived, and the instruction census as inferred. A suggested detector order is a uImage header check, then an embedded-ELF majority vote (for example at least 75% agreement), then an opcode census (for example at least 10 hits and twice the runner-up) over a bounded prefix of the file, with the thresholds and the cap set by measurement. Validate the uImage header CRC, count only aligned instruction matches, and include Thumb, little-endian PowerPC, RISC-V, and Xtensa signatures alongside the ARM, AArch64, MIPS, big-endian PowerPC, and x86 ones. ### References - REA baseline: `main` at 0cc56493be167309bd72040273de603ad75ad9e0 (package 6.5.0): `docs/raw-binary.md`, `src/domain/rawBinary.ts:11-17`, `src/firmware/FirmwareProvider.ts:297-306`, `src/domain/binaryTarget.ts:532-566`. - Related, not duplicates: #3714 (artifact discovery RFC, no ISA detection), #3621 (file-format inference), #3615 (ESP32 flash mapping), #3532 (SH-4 raw lane), #4110 (Binwalk-NG scope), #1836 and #1868 (caller-supplied raw-binary design). - Search: all 1,667 issues and 2,501 pull requests were listed and searched locally for uImage, U-Boot, opcode, endianness, `ih_arch`, headerless, "detect/infer architecture/cpu/processor", "raw firmware", and "binwalk -A"; no existing proposal found.

bsakaguchi commented

2026-10-16 21:08:51 +00:00

Author

Owner

@dandgabr commented on GitHub (Oct 16, 2026):

Static triage: this proposal is a separate, session-free byte-analysis operation. The requested U-Boot uImage and embedded ELF facts can be reported as observed/derived evidence with original file offsets; an opcode census remains an inference and should expose its counts, alignment, confidence class, runner-up and scan coverage. A negative result should be explicit unknown with the detectors and coverage recorded.

This should remain separate from the firmware-provider integrations in #4110. inspect_firmware_regions carries provider description text, and Binwalk-NG signature descriptions do not establish a structured CPU/endianness fact. open_binary format=raw-binary should continue to require caller-supplied architecture and entry point; candidate detection must not silently choose or open a target.

Before implementation, the design should pin the U-Boot architecture mapping and ELF machine mapping to source, validate uImage header CRC and offsets, define measured thresholds/scan bounds with self-built positive and negative fixtures, and preserve the distinction between observed headers, derived tallies and inferred opcode candidates. No part of this capability is included in the current FACT, Unblob or Binwalk-NG work. Please treat it as its own scope/implementation decision under the contribution-scope gate.

@dandgabr commented on GitHub (Oct 16, 2026): Static triage: this proposal is a separate, session-free byte-analysis operation. The requested U-Boot `uImage` and embedded ELF facts can be reported as observed/derived evidence with original file offsets; an opcode census remains an inference and should expose its counts, alignment, confidence class, runner-up and scan coverage. A negative result should be explicit `unknown` with the detectors and coverage recorded. This should remain separate from the firmware-provider integrations in #4110. `inspect_firmware_regions` carries provider description text, and Binwalk-NG signature descriptions do not establish a structured CPU/endianness fact. `open_binary format=raw-binary` should continue to require caller-supplied architecture and entry point; candidate detection must not silently choose or open a target. Before implementation, the design should pin the U-Boot architecture mapping and ELF machine mapping to source, validate uImage header CRC and offsets, define measured thresholds/scan bounds with self-built positive and negative fixtures, and preserve the distinction between observed headers, derived tallies and inferred opcode candidates. No part of this capability is included in the current FACT, Unblob or Binwalk-NG work. Please treat it as its own scope/implementation decision under the contribution-scope gate.

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-31 00:54:35 +00:00

[PR #1671] [CLOSED] fix(javascript): respect static path helper ownership #4742