morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3637] [Feature] ML model graph reverse engineering: ONNX, TFLite, CoreML, and TensorRT topology #1361

New issue

Open

opened 2026-10-16 21:04:40 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:04:40 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3637

Summary

Modern desktop applications (Electron, native Windows/macOS/Linux), mobile apps (Android, iOS), and IoT edge devices increasingly bundle deep learning models for on-device inference: computer vision (object detection, biometric facial recognition), speech recognition, noise cancellation, natural language embeddings, optical character recognition (OCR), and autonomous agent logic.

While issue #3323 (inspect_model_artifact) introduced read-only header inspection for LLM weight containers (GGUF and safetensors), it explicitly identified the follow-up milestone:

"Later: ONNX (protobuf graph and initializer inventory), as a separate follow-up once this contract is settled."

Beyond passive weight storage, on-device AI features depend on executable computation graphs. When reverse engineering an AI-powered application, an analyst needs to:

Invert the computational Directed Acyclic Graph (DAG) without running Python, PyTorch, or GPU runtimes;
Identify input/output tensor shapes, data types, and dynamic dimensions;
Recover image/audio input preprocessing parameters (scaling, normalization mean/std, color channels);
Inspect quantization parameters (INT8 scale and zero-point);
Detect custom operators, proprietary layers, and hardware delegate acceleration targets (Apple Neural Engine, NNAPI, TensorRT).

This feature proposal introduces dedicated ML Model Graph Reverse Engineering to REA, covering ONNX, TensorFlow Lite (TFLite), Apple CoreML, and NVIDIA TensorRT.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Model File (.onnx, .tflite,      |
                                    |        .mlmodel, .mlpackage, .plan)     |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                               |                               |
                         v                               v                               v
           +---------------------------+   +---------------------------+   +---------------------------+
           | ONNX Protobuf Graph       |   | TFLite FlatBuffers        |   | CoreML & TensorRT         |
           | (Nodes, Initializers, Ops)|   | (Subgraphs, Quant, Ops)   |   | (MIL, ANE, Preprocessing) |
           +-------------+-------------+   +-------------+-------------+   +-------------+-------------+
                         |                               |                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Graph Topology & DAG Inversion        |
                                     | (Inputs, Outputs, Layer Connections)  |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Quantization & Precision  |                                   | Input Preprocessing       |
           | (INT8, FP16, Scales, ZP)  |                                   | (Normalization, Mean/Std) |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Graph Topology Dossier, |
                                     | Operators, Shapes, and Backbones      |
                                     +---------------------------------------+

1. ONNX (Open Neural Network Exchange) Graph Inversion

Protobuf Parser: Parse binary ONNX .onnx files via inert Protocol Buffers schemas (ModelProto, GraphProto, NodeProto, TensorProto) without external Python/C++ ONNX runtime dependencies.
Graph & Node Extraction:
- Extract Graph Inputs and Outputs: tensor names, element types (FLOAT, FLOAT16, INT8, BFLOAT16), and dimension shapes (including symbolic/dynamic batch and sequence dimensions ?).
- Map Node Sequence: operator types (e.g. Conv, BatchNormalization, Relu, MatMul, Softmax, LayerNormalization, Attention), operator domains, node attributes, and input/output edge names.
- Initializers & Weights Inventory: report parameter tensor shapes, data types, and byte offsets without materializing multi-gigabyte weight arrays into memory.
- Subgraph Detection: extract nested control flow subgraphs within If, Loop, and Scan nodes.

2. TensorFlow Lite (TFLite) FlatBuffers Inversion

FlatBuffers Parser: Parse .tflite models (magic TFL3) across schema revisions:
- SubGraphs, OperatorCodes, Operators, Tensors, and Buffer tables.
Quantization Analysis:
- Extract QuantizationParameters: scale factors, zero points, and quantization dimensions (per-tensor vs. per-channel quantization).
Hardware Delegate & Custom Op Discovery:
- Detect custom operator strings (custom_code) representing proprietary or OEM native kernels.
- Identify delegate compatibility (GPU, NNAPI, Hexagon DSP, Edge TPU).

3. Apple CoreML Model & Preprocessing Inversion

Format Ingestion: Parse legacy .mlmodel Protobuf descriptors and modern .mlpackage directory bundles.
Neural Network & MIL Layer Recovery: Parse Model Intermediate Language (MIL) operations and neural network layer specifications.
Target Hardware Mapping: Detect designated execution targets (CPU, GPU, Apple Neural Engine / ANE).
Input Preprocessing Extraction: Recover exact image preprocessing pipelines embedded in the model:
- Color space formats (RGB, BGR, Grayscale).
- Channel normalization offsets (blue/green/red biases).
- Scaling factors (e.g. 1/255.0 or 0.5).
- Expected resolution and aspect ratio constraints.

4. NVIDIA TensorRT Plan Inspection

Plan Header Inversion: Parse serialized .plan / .engine binaries, extracting CUDA compute capability requirements (sm_75, sm_80, sm_89, sm_90), precision profiles (FP32, FP16, INT8), layer bindings, and GPU memory workspace budgets.

5. Architectural Backbone & Topology Recognition

Structural Heuristics: Identify common vision and language architectures based on graph motifs:
- Vision: ResNet residual blocks, MobileNet depthwise-separable convolutions, YOLO detection heads, Vision Transformers (ViT).
- Audio / NLP: Mel-spectrogram frontends, Multi-Head Self-Attention blocks, RMSNorm / LayerNorm sequences.
Graph Summary Export: Output structured JSON adjacency lists connecting nodes, layer types, and tensor dimensions.

Acceptance Criteria

Introduce analyze_model_graph in REA CLI and MCP contracts, accepting .onnx, .tflite, .mlmodel, and .plan files.
Invert synthetic ONNX test models, outputting full node sequences, operator names, and input/output shapes.
Extract INT8 quantization parameters (scale, zero-point) from synthetic TFLite models into structured evidence.
Extract input image normalization parameters (bias, scale, color space) from synthetic CoreML models.
Enforce strict memory bounds during parsing: weight payloads are referenced by offset and digest rather than read into RAM.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives
#3620 [Feature] Shader and GPU Bytecode Reverse Engineering
#3621 [Feature] Unknown binary file-format inference

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3637 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: AI & Runtimes](https://img.shields.io/badge/domain-AI_&_runtimes-blueviolet.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Modern desktop applications (Electron, native Windows/macOS/Linux), mobile apps (Android, iOS), and IoT edge devices increasingly bundle deep learning models for on-device inference: computer vision (object detection, biometric facial recognition), speech recognition, noise cancellation, natural language embeddings, optical character recognition (OCR), and autonomous agent logic. While issue #3323 (`inspect_model_artifact`) introduced read-only header inspection for LLM weight containers (GGUF and safetensors), it explicitly identified the follow-up milestone: > *"Later: ONNX (protobuf graph and initializer inventory), as a separate follow-up once this contract is settled."* Beyond passive weight storage, on-device AI features depend on executable computation graphs. When reverse engineering an AI-powered application, an analyst needs to: - Invert the computational Directed Acyclic Graph (DAG) without running Python, PyTorch, or GPU runtimes; - Identify input/output tensor shapes, data types, and dynamic dimensions; - Recover image/audio input preprocessing parameters (scaling, normalization mean/std, color channels); - Inspect quantization parameters (INT8 scale and zero-point); - Detect custom operators, proprietary layers, and hardware delegate acceleration targets (Apple Neural Engine, NNAPI, TensorRT). This feature proposal introduces dedicated **ML Model Graph Reverse Engineering** to REA, covering **ONNX**, **TensorFlow Lite (TFLite)**, **Apple CoreML**, and **NVIDIA TensorRT**. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Model File (.onnx, .tflite, | | .mlmodel, .mlpackage, .plan) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | | v v v +---------------------------+ +---------------------------+ +---------------------------+ | ONNX Protobuf Graph | | TFLite FlatBuffers | | CoreML & TensorRT | | (Nodes, Initializers, Ops)| | (Subgraphs, Quant, Ops) | | (MIL, ANE, Preprocessing) | +-------------+-------------+ +-------------+-------------+ +-------------+-------------+ | | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Graph Topology & DAG Inversion | | (Inputs, Outputs, Layer Connections) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Quantization & Precision | | Input Preprocessing | | (INT8, FP16, Scales, ZP) | | (Normalization, Mean/Std) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Graph Topology Dossier, | | Operators, Shapes, and Backbones | +---------------------------------------+ ``` ### 1. ONNX (Open Neural Network Exchange) Graph Inversion - **Protobuf Parser:** Parse binary ONNX `.onnx` files via inert Protocol Buffers schemas (`ModelProto`, `GraphProto`, `NodeProto`, `TensorProto`) without external Python/C++ ONNX runtime dependencies. - **Graph & Node Extraction:** - Extract Graph Inputs and Outputs: tensor names, element types (`FLOAT`, `FLOAT16`, `INT8`, `BFLOAT16`), and dimension shapes (including symbolic/dynamic batch and sequence dimensions `?`). - Map Node Sequence: operator types (e.g. `Conv`, `BatchNormalization`, `Relu`, `MatMul`, `Softmax`, `LayerNormalization`, `Attention`), operator domains, node attributes, and input/output edge names. - Initializers & Weights Inventory: report parameter tensor shapes, data types, and byte offsets without materializing multi-gigabyte weight arrays into memory. - Subgraph Detection: extract nested control flow subgraphs within `If`, `Loop`, and `Scan` nodes. ### 2. TensorFlow Lite (TFLite) FlatBuffers Inversion - **FlatBuffers Parser:** Parse `.tflite` models (magic `TFL3`) across schema revisions: - SubGraphs, OperatorCodes, Operators, Tensors, and Buffer tables. - **Quantization Analysis:** - Extract `QuantizationParameters`: scale factors, zero points, and quantization dimensions (per-tensor vs. per-channel quantization). - **Hardware Delegate & Custom Op Discovery:** - Detect custom operator strings (`custom_code`) representing proprietary or OEM native kernels. - Identify delegate compatibility (GPU, NNAPI, Hexagon DSP, Edge TPU). ### 3. Apple CoreML Model & Preprocessing Inversion - **Format Ingestion:** Parse legacy `.mlmodel` Protobuf descriptors and modern `.mlpackage` directory bundles. - **Neural Network & MIL Layer Recovery:** Parse Model Intermediate Language (MIL) operations and neural network layer specifications. - **Target Hardware Mapping:** Detect designated execution targets (CPU, GPU, Apple Neural Engine / ANE). - **Input Preprocessing Extraction:** Recover exact image preprocessing pipelines embedded in the model: - Color space formats (RGB, BGR, Grayscale). - Channel normalization offsets (blue/green/red biases). - Scaling factors (e.g. $1/255.0$ or $0.5$). - Expected resolution and aspect ratio constraints. ### 4. NVIDIA TensorRT Plan Inspection - **Plan Header Inversion:** Parse serialized `.plan` / `.engine` binaries, extracting CUDA compute capability requirements (sm_75, sm_80, sm_89, sm_90), precision profiles (FP32, FP16, INT8), layer bindings, and GPU memory workspace budgets. ### 5. Architectural Backbone & Topology Recognition - **Structural Heuristics:** Identify common vision and language architectures based on graph motifs: - Vision: ResNet residual blocks, MobileNet depthwise-separable convolutions, YOLO detection heads, Vision Transformers (ViT). - Audio / NLP: Mel-spectrogram frontends, Multi-Head Self-Attention blocks, RMSNorm / LayerNorm sequences. - **Graph Summary Export:** Output structured JSON adjacency lists connecting nodes, layer types, and tensor dimensions. --- ## Acceptance Criteria 1. Introduce `analyze_model_graph` in REA CLI and MCP contracts, accepting `.onnx`, `.tflite`, `.mlmodel`, and `.plan` files. 2. Invert synthetic ONNX test models, outputting full node sequences, operator names, and input/output shapes. 3. Extract INT8 quantization parameters (scale, zero-point) from synthetic TFLite models into structured evidence. 4. Extract input image normalization parameters (bias, scale, color space) from synthetic CoreML models. 5. Enforce strict memory bounds during parsing: weight payloads are referenced by offset and digest rather than read into RAM. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives - #3620 [Feature] Shader and GPU Bytecode Reverse Engineering - #3621 [Feature] Unknown binary file-format inference

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:44 +00:00

[PR #1361] [MERGED] fix(javascript): honor module-sync package exports #2827

bsakaguchi commented

2026-10-31 00:30:46 +00:00

Author

Owner

@zdmilot commented on GitHub (Oct 30, 2026):

Cross-reference: #8097 requests a reader for the ONNX Runtime .ort format, which I think is a sibling of this issue rather than part of it.

The distinction is the serialization. This issue covers the .onnx protobuf graph (alongside TFLite, CoreML, and TensorRT). .ort is ONNX Runtime's own FlatBuffers format, identified by the ORTM file identifier at bytes 4..8 and defined by the upstream ort.fbs schema, so it needs a different reader even though the recovered facts — graph I/O, operators, initializers, quantization — are the same ones this issue describes.

It seemed worth flagging because .ort is ORT's deployment format, so it is a likely encounter in shipped products. The concrete case in #8097 is a commercial Windows barcode SDK whose nine bundled models are all .ort; I extracted them and confirmed all nine load in onnxruntime 1.27, with symbolic input dimensions (['batch', 3, 'height', 'width']) and a com.microsoft:QuickGelu:1 contrib operator among the op entries.

No action implied for this issue. If maintainers would rather treat .ort as one more format under this umbrella than as its own request, #8097 can be folded in or closed as a duplicate — I filed it separately only because the protobuf reader proposed here would not cover it.

@zdmilot commented on GitHub (Oct 30, 2026): Cross-reference: #8097 requests a reader for the **ONNX Runtime `.ort` format**, which I think is a sibling of this issue rather than part of it. The distinction is the serialization. This issue covers the `.onnx` **protobuf** graph (alongside TFLite, CoreML, and TensorRT). `.ort` is ONNX Runtime's own **FlatBuffers** format, identified by the `ORTM` file identifier at bytes 4..8 and defined by the upstream `ort.fbs` schema, so it needs a different reader even though the recovered facts — graph I/O, operators, initializers, quantization — are the same ones this issue describes. It seemed worth flagging because `.ort` is ORT's deployment format, so it is a likely encounter in shipped products. The concrete case in #8097 is a commercial Windows barcode SDK whose nine bundled models are all `.ort`; I extracted them and confirmed all nine load in `onnxruntime` 1.27, with symbolic input dimensions (`['batch', 3, 'height', 'width']`) and a `com.microsoft:QuickGelu:1` contrib operator among the op entries. No action implied for this issue. If maintainers would rather treat `.ort` as one more format under this umbrella than as its own request, #8097 can be folded in or closed as a duplicate — I filed it separately only because the protobuf reader proposed here would not cover it.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1361

No description provided.

Rows
Columns