[GH-ISSUE #3714] [RFC/Epic] Universal Artifact Discovery: recursive inspection, entropy profiling, file signatures and candidate analyzer routing #1425

New issue

Open

opened 2026-10-16 21:05:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:19 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3714

Problem Statement & Architectural Context

Currently in REA, users and autonomous agents typically invoke specialized entrypoints based on prior knowledge of the target (e.g. analyze-pe, analyze-javascript, analyze-apk, analyze-firmware). However, real-world investigations frequently begin with ambiguous, unidentified, or nested inputs: an unlabelled .dat file, a memory dump slice, an encrypted firmware partition, an embedded micro-controller blob, or a proprietary multi-layer package.

To fulfill the vision of Reverse Engineer Anything, REA requires a unified discovery primitive:
inspect_anything(path | bytes | stream) -> ArtifactInspectionReport

This engine must inspect arbitrary inputs without preconceptions, perform multi-spectral classification (magic signatures, entropy distribution, structure heuristics, string densities), unpack recognized containers recursively, and emit a confidence-weighted routing plan for candidate domain analyzers.

Proposed Architecture

flowchart TD
    A["Raw Input (Arbitrary File / Buffer / Stream)"] --> B["Universal Artifact Inspector"]
    B --> C["Byte-Level Profile & Entropy Analyzer"]
    B --> D["Signature & Format Detection Matrix"]
    B --> E["Structural Hypothesis Engine (#3621)"]
    
    C --> C1["Shannon Entropy by Chunk / Sliding Window"]
    C --> C2["Executable Code vs Compressed vs Encrypted vs Data Slices"]
    
    D --> D1["Magic Bytes (PE, ELF, Mach-O, ZIP, Tar, Pcap, Bytecodes)"]
    D --> D2["Heuristic Classifiers (ASCII/UTF-8 Text, JSON, Protobuf, SQLite)"]
    
    E --> E1["Nested Container & Member Detection"]
    E --> E2["Pointer Tables, Length Prefixes & Structured Records"]
    
    C & D & E --> F["Capability Router & Ranked Analyzer Recommendations"]

Technical Capabilities & Modules

Multi-Spectral Bytecode & Entropy Profiling:
- Compute sliding-window Shannon entropy and chi-squared distribution to visually and statistically classify regions:
  - High entropy (>7.8 bits/byte): Compressed archives or encrypted streams.
  - Medium-high entropy (6.0 - 7.5): Executable machine code / bytecode.
  - Medium entropy (4.0 - 6.0): Structured binary data tables, vectors, serializations.
  - Low entropy (<4.0): Text, sparse matrices, ASCII/Unicode tables.
Hierarchical Signature & Magic Matrix:
- Unified catalog of signatures spanning executables (PE, ELF, Mach-O, WASM, DEX), containers (ZIP, TAR, GZ, 7z, CPIO, SquashFS), archives (RPF, VPK, MPQ, CAB), documents (PDF, OLE, OOXML), and models (Safetensors, GGUF, ONNX).
- Fallback statistical analyzers for signatureless or stripped binary blobs.
Recursive Container Traversal:
- Seamlessly recurse into detected containers, yielding a hierarchical member manifest.
- Guard against decompression bombs and circular archive links through bounded recursion limits and memory budgets.
Structured Format Hypothesis Generation:
- Correlate with #3621 to identify candidate pointer arrays, variable-length strings, and repeating record boundaries when no known format matches.
Capability Router & Analyzer Recommendation:
- Emit an ArtifactInspectionReport declaring candidate analyzers with explicit confidence tiers:
  - high: Verified signature and structural validation passed.
  - medium: Partial format match or nested member identified.
  - speculative: Heuristic or entropy-based hypothesis.
- Estimate execution complexity and resource cost for recommended pipelines.

Acceptance Criteria

Single entrypoint inspect_anything accepting raw files, byte streams, and directory roots.
Accurate entropy segmentation distinguishing code, compressed, encrypted, and tabular regions.
Recursive classification of nested containers up to configurable depth.
Output of ranked candidate analyzer recommendations with clear confidence ratings.
Zero unhandled exceptions or crashes when ingesting completely corrupted, malformed, or random byte buffers.

Complements #3621 (Unknown binary file-format inference)
Complements #740 (Roadmap: clarify and formalize capability-module declarations)
Complements #117 (Full-stack reconstruction)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3714 [![Type: Architecture / RFC](https://img.shields.io/badge/type-architecture_%2F_RFC-darkred.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Architectural Context Currently in REA, users and autonomous agents typically invoke specialized entrypoints based on prior knowledge of the target (e.g. `analyze-pe`, `analyze-javascript`, `analyze-apk`, `analyze-firmware`). However, real-world investigations frequently begin with ambiguous, unidentified, or nested inputs: an unlabelled `.dat` file, a memory dump slice, an encrypted firmware partition, an embedded micro-controller blob, or a proprietary multi-layer package. To fulfill the vision of *Reverse Engineer Anything*, REA requires a unified discovery primitive: `inspect_anything(path | bytes | stream) -> ArtifactInspectionReport` This engine must inspect arbitrary inputs without preconceptions, perform multi-spectral classification (magic signatures, entropy distribution, structure heuristics, string densities), unpack recognized containers recursively, and emit a confidence-weighted routing plan for candidate domain analyzers. ### Proposed Architecture ```mermaid flowchart TD A["Raw Input (Arbitrary File / Buffer / Stream)"] --> B["Universal Artifact Inspector"] B --> C["Byte-Level Profile & Entropy Analyzer"] B --> D["Signature & Format Detection Matrix"] B --> E["Structural Hypothesis Engine (#3621)"] C --> C1["Shannon Entropy by Chunk / Sliding Window"] C --> C2["Executable Code vs Compressed vs Encrypted vs Data Slices"] D --> D1["Magic Bytes (PE, ELF, Mach-O, ZIP, Tar, Pcap, Bytecodes)"] D --> D2["Heuristic Classifiers (ASCII/UTF-8 Text, JSON, Protobuf, SQLite)"] E --> E1["Nested Container & Member Detection"] E --> E2["Pointer Tables, Length Prefixes & Structured Records"] C & D & E --> F["Capability Router & Ranked Analyzer Recommendations"] ``` ### Technical Capabilities & Modules 1. **Multi-Spectral Bytecode & Entropy Profiling:** - Compute sliding-window Shannon entropy and chi-squared distribution to visually and statistically classify regions: - High entropy (>7.8 bits/byte): Compressed archives or encrypted streams. - Medium-high entropy (6.0 - 7.5): Executable machine code / bytecode. - Medium entropy (4.0 - 6.0): Structured binary data tables, vectors, serializations. - Low entropy (<4.0): Text, sparse matrices, ASCII/Unicode tables. 2. **Hierarchical Signature & Magic Matrix:** - Unified catalog of signatures spanning executables (PE, ELF, Mach-O, WASM, DEX), containers (ZIP, TAR, GZ, 7z, CPIO, SquashFS), archives (RPF, VPK, MPQ, CAB), documents (PDF, OLE, OOXML), and models (Safetensors, GGUF, ONNX). - Fallback statistical analyzers for signatureless or stripped binary blobs. 3. **Recursive Container Traversal:** - Seamlessly recurse into detected containers, yielding a hierarchical member manifest. - Guard against decompression bombs and circular archive links through bounded recursion limits and memory budgets. 4. **Structured Format Hypothesis Generation:** - Correlate with #3621 to identify candidate pointer arrays, variable-length strings, and repeating record boundaries when no known format matches. 5. **Capability Router & Analyzer Recommendation:** - Emit an `ArtifactInspectionReport` declaring candidate analyzers with explicit confidence tiers: - `high`: Verified signature and structural validation passed. - `medium`: Partial format match or nested member identified. - `speculative`: Heuristic or entropy-based hypothesis. - Estimate execution complexity and resource cost for recommended pipelines. ### Acceptance Criteria - [ ] Single entrypoint `inspect_anything` accepting raw files, byte streams, and directory roots. - [ ] Accurate entropy segmentation distinguishing code, compressed, encrypted, and tabular regions. - [ ] Recursive classification of nested containers up to configurable depth. - [ ] Output of ranked candidate analyzer recommendations with clear confidence ratings. - [ ] Zero unhandled exceptions or crashes when ingesting completely corrupted, malformed, or random byte buffers. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #740 (Roadmap: clarify and formalize capability-module declarations) - Complements #117 (Full-stack reconstruction)

bsakaguchi referenced this issue

2026-10-17 21:11:19 +00:00

[PR #1425] [MERGED] fix(validation): retain unmatched union correction details #2877

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1425

No description provided.

Rows
Columns