morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3615] [Feature] Hardware and Embedded Reverse Engineering: ESP32 firmware extraction, NVS decoding, raw flash mapping, and bus capture analysis #1344

New issue

Open

opened 2026-10-16 21:04:30 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:04:30 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3615

Summary

Hardware and IoT reverse engineering begins with extracting and analyzing embedded firmware from microcontroller platforms and physical hardware buses. Among IoT devices, Espressif's ESP32 family (spanning Xtensa dual-core LX6/LX7 on ESP32/S2/S3 and 32-bit RISC-V on ESP32-C3/C6/H2) represents one of the most widely deployed architectures in smart home devices, automotive telematics, industrial controllers, and connected consumer electronics.

While REA currently supports Linux firmware extraction using Binwalk and Unblob (docs/firmware-analysis.md), it explicitly notes:

"This milestone does not add raw flash load profiles, bootloader/device tree semantics, emulation or hardware runtime capture."

Raw flash dumps from embedded microcontrollers cannot be decompiled by generic tools without reconstructing their flash partition layout, virtual memory mapping (MMU), Non-Volatile Storage (NVS) configurations, and hardware peripheral registers.

This feature proposal introduces dedicated Hardware & Embedded Reverse Engineering primitives to REA, focusing on:

ESP32 / ESP-IDF raw flash image parsing and partition carving;
Non-Volatile Storage (NVS) key-value decoding (extracting Wi-Fi credentials, cloud tokens, device keys);
Embedded filesystem unpacking (SPIFFS, LittleFS, FATFS);
Flash MMU virtual address mapping and peripheral register hydration in Ghidra/Rizin;
Hardware bus forensics (UART serial boot logs, logic analyzer VCD captures for I2C/SPI/CAN, and flash chip programmer images).

Technical Capabilities

1. ESP32 / ESP-IDF Flash Image & Partition Carving

Image Header Parser: Parse ESP-IDF image headers (0xE9 magic byte) across Xtensa and RISC-V chip series. Extract entry point addresses, SPI flash mode (QIO, QOUT, DIO, DOUT), flash frequency (40MHz, 80MHz), flash size, and segment counts.
Partition Table Analysis (0x8000):
- Parse partition records at default and custom offsets: type (app, data), subtype (factory, ota_0, ota_1, nvs, nvs_keys, phy_init, spiffs, littlefs), offset, length, flags (encrypted flag).
- Verify partition table MD5/SHA-256 and cryptographic integrity.
App Image Slicing: Automatically carve bootloader (offset 0x1000 / 0x0000) and active application partitions for disassembler ingestion.

2. NVS (Non-Volatile Storage) Key-Value Extraction

NVS Page & Entry Decoding:
- Parse NVS page headers (active, full, free states) and 32-byte entry structures.
- Reconstruct namespaces, string keys, primitive integers, null-terminated strings, and multi-entry BLOBs.
- Extract high-value IoT forensic artifacts: stored Wi-Fi SSIDs/passwords, MQTT broker endpoints, AWS IoT / Azure device credentials, auth tokens, device certificates, and Bluetooth pairing bonds.
Encrypted NVS (nvs_keys): Support decryption of NVS partitions when supplied with an extracted or recovered 64-byte XTS-AES encryption key.

3. Embedded Filesystem Unpacking (SPIFFS & LittleFS)

SPIFFS (SPI Flash File System): Parse page headers, object indices, and lookup tables; carve embedded HTML web interfaces, JavaScript bundles, root CA certificates, and static configuration files.
LittleFS & FATFS: Unpack LittleFS blocks, directory revisions, and inline files used in modern ESP-IDF / Arduino firmware.

4. Memory Map Synthesis & Headless Ghidra / Rizin Loader

Flash MMU Address Re-mapping:
- Embedded microcontrollers execute via cache/MMU mapping: physical flash offsets correspond to distinct virtual memory addresses (e.g. ESP32 DROM data mapped to 0x3F400000, IROM code mapped to 0x400D0000, IRAM at 0x40080000, DRAM at 0x3FFB0000).
- Automatically configure Ghidra/Rizin memory maps to load segments at their respective virtual execution bases rather than physical file offsets.
Peripheral Register Map (SVD) Hydration:
- Hydrate hardware memory-mapped I/O (MMIO) registers (GPIO, UART, SPI, I2C, Timers, Watchdog, Wi-Fi baseband) using SVD / header definitions, converting anonymous memory reads (e.g. *(0x3FF44004)) into meaningful peripheral labels (GPIO_OUT_W1TS_REG).
FreeRTOS & ESP-IDF Signature Annotation:
- Apply FLIRT / symbol signatures for core ESP-IDF functions: app_main, xTaskCreatePinnedToCore, esp_wifi_init, nvs_get_str, mbedtls_ssl_*.

5. Hardware Bus Forensics & Acquisition Analysis

UART Serial Console Log Parsing:
- Parse raw serial terminal logs from boot sequences, auto-detecting baud rate, extracting chip revision, MAC address, eFuse security states (Flash Encryption enabled, Secure Boot enabled), and interactive command prompt sessions (U-Boot, ESP console).
Logic Analyzer & Protocol Decodes (VCD / Sigrok):
- Ingest Value Change Dump (.vcd) and logic analyzer session decodes for hardware buses:
  - I2C: Address scanner, read/write payload carving (EEPROM readouts, sensor telemetry).
  - SPI / QSPI: Flash memory read command sequences (0x03, 0x0B), transaction payload extraction.
  - CAN Bus: Ingest vehicle/industrial CAN frames (IDs, payloads) for automotive hardware reverse engineering.
Raw SPI Flash Programmer Images:
- Accept full flash dumps from hardware programmers (CH341A, Bus Pirate, Tigard, Raspberry Pi Pico) and detect chip profiles (Winbond W25Qxx, Macronix, GigaDevice).

Acceptance Criteria

Introduce analyze_esp32_firmware in REA CLI and MCP contracts, accepting raw flash dumps and individual partition binaries.
Parse valid ESP-IDF partition tables and automatically carve application and data segments.
Extract all valid namespaces and key-value entries from synthetic unencrypted NVS partitions into structured JSON evidence.
Unpack synthetic SPIFFS and LittleFS image fixtures into regular files with verified SHA-256 digests.
Correctly synthesize virtual MMU memory maps for Xtensa and RISC-V ESP32 targets in headless Ghidra analysis.
Ingest UART boot logs and logic analyzer VCD files, surfacing decoded transactions as structured analysis observations.

#743 [Feature] Recover .NET NativeAOT metadata in Ghidra workflows
#3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives
#3607 [Feature] Comprehensive Unity extraction and reverse engineering
docs/firmware-analysis.md

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3615 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Reverse Engineering](https://img.shields.io/badge/domain-reverse_engineering-blueviolet.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Hardware and IoT reverse engineering begins with extracting and analyzing embedded firmware from microcontroller platforms and physical hardware buses. Among IoT devices, Espressif's **ESP32** family (spanning Xtensa dual-core LX6/LX7 on ESP32/S2/S3 and 32-bit RISC-V on ESP32-C3/C6/H2) represents one of the most widely deployed architectures in smart home devices, automotive telematics, industrial controllers, and connected consumer electronics. While REA currently supports Linux firmware extraction using Binwalk and Unblob (`docs/firmware-analysis.md`), it explicitly notes: > *"This milestone does not add raw flash load profiles, bootloader/device tree semantics, emulation or hardware runtime capture."* Raw flash dumps from embedded microcontrollers cannot be decompiled by generic tools without reconstructing their flash partition layout, virtual memory mapping (MMU), Non-Volatile Storage (NVS) configurations, and hardware peripheral registers. This feature proposal introduces dedicated **Hardware & Embedded Reverse Engineering** primitives to REA, focusing on: 1. ESP32 / ESP-IDF raw flash image parsing and partition carving; 2. Non-Volatile Storage (NVS) key-value decoding (extracting Wi-Fi credentials, cloud tokens, device keys); 3. Embedded filesystem unpacking (SPIFFS, LittleFS, FATFS); 4. Flash MMU virtual address mapping and peripheral register hydration in Ghidra/Rizin; 5. Hardware bus forensics (UART serial boot logs, logic analyzer VCD captures for I2C/SPI/CAN, and flash chip programmer images). --- ## Technical Capabilities ### 1. ESP32 / ESP-IDF Flash Image & Partition Carving - **Image Header Parser:** Parse ESP-IDF image headers (`0xE9` magic byte) across Xtensa and RISC-V chip series. Extract entry point addresses, SPI flash mode (QIO, QOUT, DIO, DOUT), flash frequency (40MHz, 80MHz), flash size, and segment counts. - **Partition Table Analysis (`0x8000`):** - Parse partition records at default and custom offsets: type (`app`, `data`), subtype (`factory`, `ota_0`, `ota_1`, `nvs`, `nvs_keys`, `phy_init`, `spiffs`, `littlefs`), offset, length, flags (encrypted flag). - Verify partition table MD5/SHA-256 and cryptographic integrity. - **App Image Slicing:** Automatically carve bootloader (offset `0x1000` / `0x0000`) and active application partitions for disassembler ingestion. ### 2. NVS (Non-Volatile Storage) Key-Value Extraction - **NVS Page & Entry Decoding:** - Parse NVS page headers (active, full, free states) and 32-byte entry structures. - Reconstruct namespaces, string keys, primitive integers, null-terminated strings, and multi-entry BLOBs. - Extract high-value IoT forensic artifacts: stored Wi-Fi SSIDs/passwords, MQTT broker endpoints, AWS IoT / Azure device credentials, auth tokens, device certificates, and Bluetooth pairing bonds. - **Encrypted NVS (`nvs_keys`):** Support decryption of NVS partitions when supplied with an extracted or recovered 64-byte XTS-AES encryption key. ### 3. Embedded Filesystem Unpacking (SPIFFS & LittleFS) - **SPIFFS (SPI Flash File System):** Parse page headers, object indices, and lookup tables; carve embedded HTML web interfaces, JavaScript bundles, root CA certificates, and static configuration files. - **LittleFS & FATFS:** Unpack LittleFS blocks, directory revisions, and inline files used in modern ESP-IDF / Arduino firmware. ### 4. Memory Map Synthesis & Headless Ghidra / Rizin Loader - **Flash MMU Address Re-mapping:** - Embedded microcontrollers execute via cache/MMU mapping: physical flash offsets correspond to distinct virtual memory addresses (e.g. ESP32 DROM data mapped to `0x3F400000`, IROM code mapped to `0x400D0000`, IRAM at `0x40080000`, DRAM at `0x3FFB0000`). - Automatically configure Ghidra/Rizin memory maps to load segments at their respective virtual execution bases rather than physical file offsets. - **Peripheral Register Map (SVD) Hydration:** - Hydrate hardware memory-mapped I/O (MMIO) registers (GPIO, UART, SPI, I2C, Timers, Watchdog, Wi-Fi baseband) using SVD / header definitions, converting anonymous memory reads (e.g. `*(0x3FF44004)`) into meaningful peripheral labels (`GPIO_OUT_W1TS_REG`). - **FreeRTOS & ESP-IDF Signature Annotation:** - Apply FLIRT / symbol signatures for core ESP-IDF functions: `app_main`, `xTaskCreatePinnedToCore`, `esp_wifi_init`, `nvs_get_str`, `mbedtls_ssl_*`. ### 5. Hardware Bus Forensics & Acquisition Analysis - **UART Serial Console Log Parsing:** - Parse raw serial terminal logs from boot sequences, auto-detecting baud rate, extracting chip revision, MAC address, eFuse security states (Flash Encryption enabled, Secure Boot enabled), and interactive command prompt sessions (U-Boot, ESP console). - **Logic Analyzer & Protocol Decodes (VCD / Sigrok):** - Ingest Value Change Dump (`.vcd`) and logic analyzer session decodes for hardware buses: - **I2C:** Address scanner, read/write payload carving (EEPROM readouts, sensor telemetry). - **SPI / QSPI:** Flash memory read command sequences (`0x03`, `0x0B`), transaction payload extraction. - **CAN Bus:** Ingest vehicle/industrial CAN frames (IDs, payloads) for automotive hardware reverse engineering. - **Raw SPI Flash Programmer Images:** - Accept full flash dumps from hardware programmers (CH341A, Bus Pirate, Tigard, Raspberry Pi Pico) and detect chip profiles (Winbond W25Qxx, Macronix, GigaDevice). --- ## Acceptance Criteria 1. Introduce `analyze_esp32_firmware` in REA CLI and MCP contracts, accepting raw flash dumps and individual partition binaries. 2. Parse valid ESP-IDF partition tables and automatically carve application and data segments. 3. Extract all valid namespaces and key-value entries from synthetic unencrypted NVS partitions into structured JSON evidence. 4. Unpack synthetic SPIFFS and LittleFS image fixtures into regular files with verified SHA-256 digests. 5. Correctly synthesize virtual MMU memory maps for Xtensa and RISC-V ESP32 targets in headless Ghidra analysis. 6. Ingest UART boot logs and logic analyzer VCD files, surfacing decoded transactions as structured analysis observations. --- ## Related - #743 [Feature] Recover .NET NativeAOT metadata in Ghidra workflows - #3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives - #3607 [Feature] Comprehensive Unity extraction and reverse engineering - docs/firmware-analysis.md

bsakaguchi referenced this issue

2026-10-17 21:10:33 +00:00

[PR #1344] [MERGED] fix(ci): scope verification and reuse native runners #2810

bsakaguchi referenced this issue

2026-10-17 21:10:37 +00:00

[PR #1348] [MERGED] docs: explain CI scope and native runner reuse #2816

bsakaguchi commented

2026-10-31 00:30:42 +00:00

Author

Owner

@Entrpi commented on GitHub (Oct 27, 2026):

A scope question before I propose an NVS reader. It would be a general reader through inspect_artifact:

physical records (pages, entries, CRC checks) kept separate from reconstructed values;
a default summary with keys, types, sizes and numeric values, plus digests for strings and blobs, whose values come only with detail: "complete";
encryption reported as caller-declared, indicated by an nvs_keys partition, or unknown;
no key recovery, decryption or credential-specific features;
no redaction by key name, per AGENTS.md.

A flash-layout reader would extract nvs_keys partitions like any other partition. NVS often holds Wi-Fi credentials, and this issue also asks for decryption with recovered keys, which I'm not proposing.

Is the general reader within scope? Any NVS PR will stay a draft until there's a decision.

@Entrpi commented on GitHub (Oct 27, 2026): A scope question before I propose an NVS reader. It would be a general reader through `inspect_artifact`: - physical records (pages, entries, CRC checks) kept separate from reconstructed values; - a default summary with keys, types, sizes and numeric values, plus digests for strings and blobs, whose values come only with `detail: "complete"`; - encryption reported as caller-declared, indicated by an `nvs_keys` partition, or unknown; - no key recovery, decryption or credential-specific features; - no redaction by key name, per AGENTS.md. A flash-layout reader would extract `nvs_keys` partitions like any other partition. NVS often holds Wi-Fi credentials, and this issue also asks for decryption with recovered keys, which I'm not proposing. Is the general reader within scope? Any NVS PR will stay a draft until there's a decision.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1344

No description provided.

Rows
Columns