morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3641] [Feature] USB protocol reverse engineering: descriptors, HID report schemas, and recorded transaction reconstruction #1363

New issue

Open

opened 2026-10-16 21:04:42 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:04:42 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3641

Summary

The Universal Serial Bus (USB) is the ubiquitous hardware bus connecting peripherals, embedded microcontroller units (such as ESP32 and ARM Cortex boards), medical instruments, security dongles, biometric scanners, automotive diagnostic tools, and custom hardware. In hardware reverse engineering and firmware security audits, analysts frequently record USB communication using USBPcap (on Windows) or usbmon / Wireshark (on Linux) to analyze device behaviors and understand proprietary protocols.

While REA possesses network PCAP capture and passive DNS analysis capabilities (#3599), it currently has no parser for USB link-layer captures (LINKTYPE_USBPCAP, LINKTYPE_USB_LINUX). Reversing an unknown USB device from packet captures requires tedious manual effort: reconstructing descriptor trees, parsing bitfield-level HID report items, decoding control transfer setup packets, and inferring command-response framing over bulk/interrupt endpoints.

This feature proposal introduces dedicated USB Protocol Reverse Engineering to REA, enabling automated ingestion of USB PCAP/PCAPNG captures, complete descriptor hierarchy extraction, HID report schema de-serialization, control transfer classification, and proprietary transaction inference.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: USB Capture (.pcap, .pcapng)     |
                                    |        (USBPcap / usbmon link layers)   |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | USB Link-Layer Header     |                                   | Descriptor Hierarchy      |
           | Parser (249 / 189)        |                                   | Reconstructor (Dev, Config|
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Control Transfer Setup Decoder        |
                                     | (bmRequestType, bRequest, wValue)     |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | HID Report Descriptor     |                                   | Bulk/Interrupt Endpoint   |
           | Bitfield Parser (0x22)    |                                   | Transaction Pairing       |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Proprietary Protocol Inference        |
                                     | & pyusb / WebUSB Script Synthesis     |
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: USB Device Dossier,     |
                                     | HID Schema & Transaction Sequence     |
                                     +---------------------------------------+

1. USB Packet Capture Ingestion (USBPcap & usbmon)

Link-Layer Ingestion: Support PCAP and PCAPNG files containing:
- LINKTYPE_USBPCAP (249): Windows USBPcap capture headers.
- LINKTYPE_USB_LINUX (189): Linux usbmon capture headers.
Packet Header Extraction: Extract bus number, device address, endpoint address (0x00..0x0F OUT, 0x80..0x8F IN), transfer type (Control = 0, Isochronous = 1, Bulk = 2, Interrupt = 3), transfer status, and payload length.

2. Standard Descriptor Hierarchy Reconstruction

Descriptor Reassembly: Reconstruct the complete USB descriptor hierarchy from GET_DESCRIPTOR control transfers:
- Device Descriptor: USB specification version (bcdUSB), Vendor ID (idVendor), Product ID (idProduct), device revision (bcdDevice), class/subclass/protocol, and string descriptor indices.
- Configuration Descriptor: Power attributes (bus-powered vs. self-powered, remote wakeup), max power consumption (bMaxPower in 2mA units), and interface counts.
- Interface Descriptors: Class definitions (e.g. 0x03 HID, 0x08 Mass Storage, 0x0A CDC Data, 0xFF Vendor-Specific), alternate settings, and endpoint counts.
- Endpoint Descriptors: Direction, transfer type, max packet size, and polling interval (bInterval).
- String Descriptors: Extract UTF-16LE text for Manufacturer, Product, Serial Number, and Interface names.

3. HID (Human Interface Device) Report Descriptor Parser

Report Descriptor Parser (0x22): Implement a bitfield-level parser for HID item structures:
- Main Items: Input, Output, Feature, Collection, End Collection.
- Global Items: Usage Page, Logical Minimum/Maximum, Physical Minimum/Maximum, Report Size (in bits), Report Count, Report ID.
- Local Items: Usage, Usage Minimum/Maximum.
Bitfield Schema Synthesis: Invert binary descriptor byte streams into structured, human-readable packet layout definitions (mapping bit offsets to specific axes, buttons, sliders, battery status, or vendor-defined telemetry fields).

4. Control Transfer Setup Inversion

Setup Packet Decoding: Deconstruct 8-byte SETUP packets:
- bmRequestType: direction (Host-to-Device / Device-to-Host), request type (Standard, Class, Vendor), and recipient (Device, Interface, Endpoint, Other).
- bRequest: standard requests (GET_STATUS, CLEAR_FEATURE, SET_ADDRESS, GET_DESCRIPTOR, SET_CONFIGURATION, GET_INTERFACE) and class/vendor requests.
- wValue, wIndex, wLength: parameter decoding based on request context.
Vendor Request Cataloging: Flag and catalog proprietary vendor requests (0x40 / 0xC0), documenting observed parameter values.

5. Transaction Pairing & Protocol Inference

Endpoint Stream Tracking: Group packets by (Bus, Device, Endpoint).
Request-Response Pairing: Correlate outgoing bulk/interrupt packets with incoming responses based on timing and sequence markers.
Framing & Protocol Inference:
- Detect recurring packet structures: header magic bytes, command opcodes, sequence numbers, payload lengths, and trailing checksums (CRC16, CRC32, XOR sums).
Replay Script Generation: Generate clean Python (pyusb) and WebUSB boilerplate scripts replicating the observed command-and-control sequences for rapid hardware interaction testing.

Acceptance Criteria

Introduce analyze_usb_capture in REA CLI and MCP contracts, accepting .pcap and .pcapng USB capture files.
Ingest synthetic USBPcap (249) and Linux usbmon (189) captures without format rejection.
Reconstruct complete USB Device, Configuration, and Interface descriptor hierarchies into structured JSON.
Parse synthetic HID Report Descriptors and output verified bit-level report schemas.
Decode Control Transfer SETUP packets and catalog vendor-specific requests.
Pair Bulk/Interrupt request-response transactions and output inferred framing parameters.

#3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives
#3615 [Feature] Hardware and Embedded Reverse Engineering: ESP32 firmware
#3621 [Feature] Unknown binary file-format inference

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3641 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary The Universal Serial Bus (USB) is the ubiquitous hardware bus connecting peripherals, embedded microcontroller units (such as ESP32 and ARM Cortex boards), medical instruments, security dongles, biometric scanners, automotive diagnostic tools, and custom hardware. In hardware reverse engineering and firmware security audits, analysts frequently record USB communication using **USBPcap** (on Windows) or **usbmon** / Wireshark (on Linux) to analyze device behaviors and understand proprietary protocols. While REA possesses network PCAP capture and passive DNS analysis capabilities (#3599), it currently has no parser for USB link-layer captures (`LINKTYPE_USBPCAP`, `LINKTYPE_USB_LINUX`). Reversing an unknown USB device from packet captures requires tedious manual effort: reconstructing descriptor trees, parsing bitfield-level HID report items, decoding control transfer setup packets, and inferring command-response framing over bulk/interrupt endpoints. This feature proposal introduces dedicated **USB Protocol Reverse Engineering** to REA, enabling automated ingestion of USB PCAP/PCAPNG captures, complete descriptor hierarchy extraction, HID report schema de-serialization, control transfer classification, and proprietary transaction inference. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: USB Capture (.pcap, .pcapng) | | (USBPcap / usbmon link layers) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | USB Link-Layer Header | | Descriptor Hierarchy | | Parser (249 / 189) | | Reconstructor (Dev, Config| +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Control Transfer Setup Decoder | | (bmRequestType, bRequest, wValue) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | HID Report Descriptor | | Bulk/Interrupt Endpoint | | Bitfield Parser (0x22) | | Transaction Pairing | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | Proprietary Protocol Inference | | & pyusb / WebUSB Script Synthesis | +-------------------+-------------------+ | v +---------------------------------------+ | REA Evidence: USB Device Dossier, | | HID Schema & Transaction Sequence | +---------------------------------------+ ``` ### 1. USB Packet Capture Ingestion (USBPcap & usbmon) - **Link-Layer Ingestion:** Support PCAP and PCAPNG files containing: - `LINKTYPE_USBPCAP` (249): Windows USBPcap capture headers. - `LINKTYPE_USB_LINUX` (189): Linux usbmon capture headers. - **Packet Header Extraction:** Extract bus number, device address, endpoint address (`0x00`..`0x0F` OUT, `0x80`..`0x8F` IN), transfer type (Control = 0, Isochronous = 1, Bulk = 2, Interrupt = 3), transfer status, and payload length. ### 2. Standard Descriptor Hierarchy Reconstruction - **Descriptor Reassembly:** Reconstruct the complete USB descriptor hierarchy from `GET_DESCRIPTOR` control transfers: - **Device Descriptor:** USB specification version (`bcdUSB`), Vendor ID (`idVendor`), Product ID (`idProduct`), device revision (`bcdDevice`), class/subclass/protocol, and string descriptor indices. - **Configuration Descriptor:** Power attributes (bus-powered vs. self-powered, remote wakeup), max power consumption (`bMaxPower` in 2mA units), and interface counts. - **Interface Descriptors:** Class definitions (e.g. `0x03` HID, `0x08` Mass Storage, `0x0A` CDC Data, `0xFF` Vendor-Specific), alternate settings, and endpoint counts. - **Endpoint Descriptors:** Direction, transfer type, max packet size, and polling interval (`bInterval`). - **String Descriptors:** Extract UTF-16LE text for Manufacturer, Product, Serial Number, and Interface names. ### 3. HID (Human Interface Device) Report Descriptor Parser - **Report Descriptor Parser (`0x22`):** Implement a bitfield-level parser for HID item structures: - **Main Items:** `Input`, `Output`, `Feature`, `Collection`, `End Collection`. - **Global Items:** `Usage Page`, `Logical Minimum/Maximum`, `Physical Minimum/Maximum`, `Report Size` (in bits), `Report Count`, `Report ID`. - **Local Items:** `Usage`, `Usage Minimum/Maximum`. - **Bitfield Schema Synthesis:** Invert binary descriptor byte streams into structured, human-readable packet layout definitions (mapping bit offsets to specific axes, buttons, sliders, battery status, or vendor-defined telemetry fields). ### 4. Control Transfer Setup Inversion - **Setup Packet Decoding:** Deconstruct 8-byte `SETUP` packets: - `bmRequestType`: direction (Host-to-Device / Device-to-Host), request type (Standard, Class, Vendor), and recipient (Device, Interface, Endpoint, Other). - `bRequest`: standard requests (`GET_STATUS`, `CLEAR_FEATURE`, `SET_ADDRESS`, `GET_DESCRIPTOR`, `SET_CONFIGURATION`, `GET_INTERFACE`) and class/vendor requests. - `wValue`, `wIndex`, `wLength`: parameter decoding based on request context. - **Vendor Request Cataloging:** Flag and catalog proprietary vendor requests (`0x40` / `0xC0`), documenting observed parameter values. ### 5. Transaction Pairing & Protocol Inference - **Endpoint Stream Tracking:** Group packets by `(Bus, Device, Endpoint)`. - **Request-Response Pairing:** Correlate outgoing bulk/interrupt packets with incoming responses based on timing and sequence markers. - **Framing & Protocol Inference:** - Detect recurring packet structures: header magic bytes, command opcodes, sequence numbers, payload lengths, and trailing checksums (CRC16, CRC32, XOR sums). - **Replay Script Generation:** Generate clean Python (`pyusb`) and WebUSB boilerplate scripts replicating the observed command-and-control sequences for rapid hardware interaction testing. --- ## Acceptance Criteria 1. Introduce `analyze_usb_capture` in REA CLI and MCP contracts, accepting `.pcap` and `.pcapng` USB capture files. 2. Ingest synthetic USBPcap (249) and Linux usbmon (189) captures without format rejection. 3. Reconstruct complete USB Device, Configuration, and Interface descriptor hierarchies into structured JSON. 4. Parse synthetic HID Report Descriptors and output verified bit-level report schemas. 5. Decode Control Transfer `SETUP` packets and catalog vendor-specific requests. 6. Pair Bulk/Interrupt request-response transactions and output inferred framing parameters. --- ## Related - #3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives - #3615 [Feature] Hardware and Embedded Reverse Engineering: ESP32 firmware - #3621 [Feature] Unknown binary file-format inference

bsakaguchi referenced this issue

2026-10-16 21:05:45 +00:00

[GH-ISSUE #3784] [Feature] Read-only inventory of Unity IL2CPP global-metadata.dat (first slice of #3607) #1469

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:45 +00:00

[PR #1363] [MERGED] feat(go): inspect embedded binary build metadata #2828

bsakaguchi referenced this issue

2026-10-31 00:54:32 +00:00

[PR #1668] [MERGED] test(ci): stabilize JADX recovery and ignore generated reports #4736

bsakaguchi referenced this issue

2026-10-31 01:04:48 +00:00

[PR #2406] feat(wasm): decode WebAssembly components offline #5206

bsakaguchi commented

2026-11-01 21:10:55 +00:00

Author

Owner

@emirmrkaljevic commented on GitHub (Oct 31, 2026):

In case it helps as a reference for this: I built usb-rea (MIT, Python), which does a lot of what's listed here for USBPcap and usbmon captures. It reads both link types, finds VID:PID from the GET_DESCRIPTOR replies, decodes control setup packets, and lines the host->device packets up with the actions done in the vendor app to infer the command fields (constants, values, multi-packet transactions, request/response pairs and background polling). It also sets aside continuous image streams to device LCDs (JPEG and raw RGB565/RGB888) so they don't drown out the commands.

Two things I learned that might be worth planning for: USBPcap often puts the control IN data in the "complete" record rather than the submit one, and per-frame acks in streams are best counted per frame, not over the whole capture. There's a trimmed real USBPcap capture (ASUS LC III AIO LCD) in tests/data if you want a test input. HID report descriptor parsing is the part it doesn't do.

@emirmrkaljevic commented on GitHub (Oct 31, 2026): In case it helps as a reference for this: I built [usb-rea](https://github.com/emirmrkaljevic/usb-rea) (MIT, Python), which does a lot of what's listed here for USBPcap and usbmon captures. It reads both link types, finds VID:PID from the GET_DESCRIPTOR replies, decodes control setup packets, and lines the host->device packets up with the actions done in the vendor app to infer the command fields (constants, values, multi-packet transactions, request/response pairs and background polling). It also sets aside continuous image streams to device LCDs (JPEG and raw RGB565/RGB888) so they don't drown out the commands. Two things I learned that might be worth planning for: USBPcap often puts the control IN data in the "complete" record rather than the submit one, and per-frame acks in streams are best counted per frame, not over the whole capture. There's a trimmed real USBPcap capture (ASUS LC III AIO LCD) in `tests/data` if you want a test input. HID report descriptor parsing is the part it doesn't do.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1363

No description provided.

Rows
Columns