[GH-ISSUE #3717] [RFC/Epic] Universal Protocol Inference: stream framing, opcode discovery, request-response pairing and state machine recovery #1427

New issue

Open

opened 2026-10-16 21:05:21 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:21 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3717

Problem Statement & Architectural Context

Network captures (PCAP/pcapng), IPC streams, USB transactions (#3641), Bluetooth HCI traces (#3644), and serial bus communications frequently carry proprietary binary protocols. While Wireshark and REA dissect standardized protocols (HTTP, TLS, DNS, WebSocket), proprietary application protocols remain completely opaque: streams of raw hex bytes.

Reverse engineers currently dissect these protocols by hand in hex editors or by writing custom Wireshark dissectors in Lua. REA needs an automated, statistic-driven and heuristic Universal Protocol Inference Engine capable of discovering protocol grammar, framing delimiters, message boundaries, opcode tables, field types, and session state machines directly from recorded message traffic.

Proposed Architecture

flowchart TD
    A["Recorded Traffic Stream (PCAP / Serial / IPC / Sockets)"] --> B["Protocol Inference Engine"]
    B --> C["Stream Alignment & Framing Slicer"]
    B --> D["Field Boundary & Entropy Clusterer"]
    B --> E["Opcode & Message Type Classifier"]
    B --> F["Session State Machine Synthesizer"]
    
    C --> C1["Length Prefix Detection (Big/Little Endian)"]
    C --> C2["Delimiter & Sync Byte Discovery (e.g. 0x55AA)"]
    
    D --> D1["Columnar Entropy & Invariance Analysis"]
    D --> D2["Field Types (Integers, Timestamps, Flags, Payload Slices)"]
    
    E --> E1["Candidate Opcode Identifier"]
    E --> E2["Request-Response Pair Correlation"]
    
    F --> F1["State Transitions (Handshake -> Auth -> Stream -> Teardown)"]
    F --> F2["Wireshark Lua Dissector & Kaitai Struct Protocol Schema"]
    
    C & D & E & F --> G["Synthesized Protocol Specification & Interactive Flow Graph"]

Technical Capabilities & Modules

Stream Framing & Message Boundary Discovery:
- Detect packet framing styles:
  - Fixed length frames.
  - Length-prefixed frames: detect 16-bit or 32-bit length fields, verify consistency against packet boundary offsets.
  - Delimiter-based framing: identify repeating synchronization bytes (e.g. 0xAA 0x55, 0x7E, carriage returns).
- Dissect raw TCP byte streams into discrete application-layer protocol data units (APDUs).
Columnar Invariance & Field Slicing:
- Align multiple recorded messages of the same candidate type into a matrix.
- Compute bitwise variance and Shannon entropy per column:
  - Zero-entropy columns -> Magic constants, protocol versions, fixed flags.
  - Low-entropy discrete values -> Opcodes, status codes, enumeration types.
  - Monotonically increasing values -> Sequence numbers, timestamps, transaction IDs.
  - High-entropy variable-length fields -> Payloads, encrypted blocks, compressed data.
Opcode Dispatch & Message Hierarchy:
- Automatically cluster messages by candidate opcode positions.
- Isolate directionality (client-to-server vs server-to-client).
- Match request-response pairs based on transaction ID correlation and temporal proximity.
Session State Machine Recovery:
- Model the session lifecycle as a Finite State Automaton (FSA):
  - Identify initial handshake and authentication states.
  - Identify steady-state data exchange and keepalive pulses.
  - Identify termination and error teardowns.
- Generate Mermaid state diagrams mapping valid message sequences.
Declarative Dissector Export:
- Synthesize formal protocol specifications exportable as:
  - Wireshark Lua dissector scripts.
  - Kaitai Struct protocol specs (.ksy).
  - JSON protocol schema definitions.

Acceptance Criteria

Automated message boundary extraction from length-prefixed and delimited raw packet streams.
Identification of opcode fields and classification of distinct message types.
Accurate pairing of request and response transactions.
Reconstitution of session state machines from sequential conversation captures.
Export of working Wireshark Lua dissectors reflecting the inferred protocol grammar.

Complements #3621 (Unknown binary file-format inference)
Complements #3641 (USB protocol reverse engineering)
Complements #3644 (Bluetooth/BLE reverse engineering)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3717 [![Type: Architecture / RFC](https://img.shields.io/badge/type-architecture_%2F_RFC-darkred.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Architectural Context Network captures (PCAP/pcapng), IPC streams, USB transactions (#3641), Bluetooth HCI traces (#3644), and serial bus communications frequently carry proprietary binary protocols. While Wireshark and REA dissect standardized protocols (HTTP, TLS, DNS, WebSocket), proprietary application protocols remain completely opaque: streams of raw hex bytes. Reverse engineers currently dissect these protocols by hand in hex editors or by writing custom Wireshark dissectors in Lua. REA needs an automated, statistic-driven and heuristic **Universal Protocol Inference Engine** capable of discovering protocol grammar, framing delimiters, message boundaries, opcode tables, field types, and session state machines directly from recorded message traffic. ### Proposed Architecture ```mermaid flowchart TD A["Recorded Traffic Stream (PCAP / Serial / IPC / Sockets)"] --> B["Protocol Inference Engine"] B --> C["Stream Alignment & Framing Slicer"] B --> D["Field Boundary & Entropy Clusterer"] B --> E["Opcode & Message Type Classifier"] B --> F["Session State Machine Synthesizer"] C --> C1["Length Prefix Detection (Big/Little Endian)"] C --> C2["Delimiter & Sync Byte Discovery (e.g. 0x55AA)"] D --> D1["Columnar Entropy & Invariance Analysis"] D --> D2["Field Types (Integers, Timestamps, Flags, Payload Slices)"] E --> E1["Candidate Opcode Identifier"] E --> E2["Request-Response Pair Correlation"] F --> F1["State Transitions (Handshake -> Auth -> Stream -> Teardown)"] F --> F2["Wireshark Lua Dissector & Kaitai Struct Protocol Schema"] C & D & E & F --> G["Synthesized Protocol Specification & Interactive Flow Graph"] ``` ### Technical Capabilities & Modules 1. **Stream Framing & Message Boundary Discovery:** - Detect packet framing styles: - Fixed length frames. - Length-prefixed frames: detect 16-bit or 32-bit length fields, verify consistency against packet boundary offsets. - Delimiter-based framing: identify repeating synchronization bytes (e.g. `0xAA 0x55`, `0x7E`, carriage returns). - Dissect raw TCP byte streams into discrete application-layer protocol data units (APDUs). 2. **Columnar Invariance & Field Slicing:** - Align multiple recorded messages of the same candidate type into a matrix. - Compute bitwise variance and Shannon entropy per column: - Zero-entropy columns -> Magic constants, protocol versions, fixed flags. - Low-entropy discrete values -> Opcodes, status codes, enumeration types. - Monotonically increasing values -> Sequence numbers, timestamps, transaction IDs. - High-entropy variable-length fields -> Payloads, encrypted blocks, compressed data. 3. **Opcode Dispatch & Message Hierarchy:** - Automatically cluster messages by candidate opcode positions. - Isolate directionality (client-to-server vs server-to-client). - Match request-response pairs based on transaction ID correlation and temporal proximity. 4. **Session State Machine Recovery:** - Model the session lifecycle as a Finite State Automaton (FSA): - Identify initial handshake and authentication states. - Identify steady-state data exchange and keepalive pulses. - Identify termination and error teardowns. - Generate Mermaid state diagrams mapping valid message sequences. 5. **Declarative Dissector Export:** - Synthesize formal protocol specifications exportable as: - Wireshark Lua dissector scripts. - Kaitai Struct protocol specs (`.ksy`). - JSON protocol schema definitions. ### Acceptance Criteria - [ ] Automated message boundary extraction from length-prefixed and delimited raw packet streams. - [ ] Identification of opcode fields and classification of distinct message types. - [ ] Accurate pairing of request and response transactions. - [ ] Reconstitution of session state machines from sequential conversation captures. - [ ] Export of working Wireshark Lua dissectors reflecting the inferred protocol grammar. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3641 (USB protocol reverse engineering) - Complements #3644 (Bluetooth/BLE reverse engineering)

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:11:20 +00:00

[PR #1427] [MERGED] fix(javascript)!: avoid event payload copies in graph metadata #2879