[GH-ISSUE #3652] [Feature] PyTorch checkpoint reverse engineering: ZIP/pickle opcode graph, tensor storages, state_dict reconstruction and unsafe-global detection #1372

New issue

Open

opened 2026-10-16 21:04:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:47 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3652

Summary

PyTorch model checkpoints (.pt, .pth, .bin, .ckpt) are the standard serialization format for training checkpoints, fine-tuned weights, and model weights across modern machine learning. Under the hood, PyTorch checkpoints are ZIP archives containing serialized Python pickle streams and binary tensor storage files (archive/data.pkl, archive/data/0, archive/data/1).

Issue #3323 (inspect_model_artifact) specifically noted:

"Out of scope: PyTorch pickle checkpoints (.pt, .bin, .ckpt), because a faithful reading needs unpickling semantics. They stay unsupported, with a remediation message."

Relying on standard Python pickle.load or torch.load to inspect a checkpoint is a severe security risk, as unpickling arbitrary pickle byte streams can execute arbitrary code (RCE) via __reduce__ exploit payloads.

This feature proposal introduces dedicated PyTorch Checkpoint Reverse Engineering to REA, implementing an inert, safe static pickle decompiler that reconstructs the full state_dict, tensor shapes, dtypes, and storage offsets, while detecting and flagging unsafe global imports and exploit payloads—without executing code or invoking Python!

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: PyTorch Checkpoint (.pt / .ckpt) |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | ZIP Archive Container     |                                   | Unsafe Global / RCE       |
           | Reader (archive/data.pkl) |                                   | Detector (GLOBAL, REDUCE) |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Inert Static Pickle Opcode Engine     |
                                     | (No execution, stack & memo emulation)|
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Tensor Storage Resolver   |                                   | state_dict Key-Value Tree |
           | (Shapes, dtypes, offsets) |                                   | Reconstruction            |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Checkpoint Dossier,     |
                                     | Architecture Schemas & Security Flags |
                                     +---------------------------------------+

1. Safe Container Ingestion & Extraction

ZIP Packaging: Parse PyTorch's standard ZIP container (v0.1.1+ format), extracting archive member names (archive/version, archive/data.pkl, and individual tensor storage files archive/data/*).
Legacy Tarball Support: Parse legacy PyTorch tarball checkpoints.

2. Inert Static Pickle Opcode Emulation

Pure Static Pickle Engine: Implement an inert pickle virtual machine emulating pickle opcodes (protocols 0 through 5) without executing arbitrary Python:
- Stack and memo operations: PROTO, GLOBAL, STACK_GLOBAL, BINGET, BINPUT, MEMOIZE, MARK, TUPLE, LIST, DICT, SETITEMS.
- Object instantiation: Emulate _rebuild_tensor_v2, _rebuild_parameter, OrderedDict, and torch.storage._load_from_bytes.
Zero-Code Execution Guarantee: No native dynamic imports or Python eval are ever executed.

3. Unsafe Global & Malicious Payload Detection

RCE Payload Audit: Inspect all GLOBAL, STACK_GLOBAL, and REDUCE opcodes:
- Flag any non-standard or dangerous global call (e.g. posix.system, subprocess.Popen, os.system, builtins.eval, builtins.exec, pty.spawn, socket.socket).
- Surface exact byte offsets and arguments of malicious exploit payloads.
Allowlist Verification: Verify whether all referenced globals belong strictly to standard PyTorch and NumPy namespaces (torch._utils, torch, collections.OrderedDict).

4. `state_dict` & Tensor Storage Reconstruction

Tensor Reconstruction: Map _rebuild_tensor_v2 declarations to their corresponding storage files:
- Parameter / buffer name (e.g. model.layers.0.self_attn.q_proj.weight).
- Tensor dimensions / shape (e.g. [4096, 4096]).
- Data type (torch.float32, torch.float16, torch.bfloat16, torch.int8).
- Storage offset, stride, and byte length.
Architecture Parameter Summary: Derive total parameter count, memory footprint by dtype, and model layer hierarchy directly from the reconstructed state_dict.

Acceptance Criteria

Introduce analyze_pytorch_checkpoint in REA CLI and MCP contracts, accepting .pt, .pth, .bin, and .ckpt files.
Safely parse synthetic PyTorch checkpoints without invoking Python pickle.load or torch.load.
Output the complete state_dict hierarchy, tensor shapes, dtypes, and total parameter counts in structured JSON.
Detect synthetic posix.system RCE exploit payloads and report typed security violations with byte locations.
Enforce bounded parsing budgets, preventing memory exhaustion from malicious recursive pickle data.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3637 [Feature] ML model graph reverse engineering: ONNX, TFLite, CoreML, and TensorRT topology
#3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3652 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary PyTorch model checkpoints (`.pt`, `.pth`, `.bin`, `.ckpt`) are the standard serialization format for training checkpoints, fine-tuned weights, and model weights across modern machine learning. Under the hood, PyTorch checkpoints are ZIP archives containing serialized Python pickle streams and binary tensor storage files (`archive/data.pkl`, `archive/data/0`, `archive/data/1`). Issue #3323 (`inspect_model_artifact`) specifically noted: > *"Out of scope: PyTorch pickle checkpoints (.pt, .bin, .ckpt), because a faithful reading needs unpickling semantics. They stay unsupported, with a remediation message."* Relying on standard Python `pickle.load` or `torch.load` to inspect a checkpoint is a **severe security risk**, as unpickling arbitrary pickle byte streams can execute arbitrary code (RCE) via `__reduce__` exploit payloads. This feature proposal introduces dedicated **PyTorch Checkpoint Reverse Engineering** to REA, implementing an **inert, safe static pickle decompiler** that reconstructs the full `state_dict`, tensor shapes, dtypes, and storage offsets, while detecting and flagging unsafe global imports and exploit payloads—without executing code or invoking Python! --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: PyTorch Checkpoint (.pt / .ckpt) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | ZIP Archive Container | | Unsafe Global / RCE | | Reader (archive/data.pkl) | | Detector (GLOBAL, REDUCE) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Inert Static Pickle Opcode Engine | | (No execution, stack & memo emulation)| +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Tensor Storage Resolver | | state_dict Key-Value Tree | | (Shapes, dtypes, offsets) | | Reconstruction | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Checkpoint Dossier, | | Architecture Schemas & Security Flags | +---------------------------------------+ ``` ### 1. Safe Container Ingestion & Extraction - **ZIP Packaging:** Parse PyTorch's standard ZIP container (v0.1.1+ format), extracting archive member names (`archive/version`, `archive/data.pkl`, and individual tensor storage files `archive/data/*`). - **Legacy Tarball Support:** Parse legacy PyTorch tarball checkpoints. ### 2. Inert Static Pickle Opcode Emulation - **Pure Static Pickle Engine:** Implement an inert pickle virtual machine emulating pickle opcodes (protocols 0 through 5) without executing arbitrary Python: - Stack and memo operations: `PROTO`, `GLOBAL`, `STACK_GLOBAL`, `BINGET`, `BINPUT`, `MEMOIZE`, `MARK`, `TUPLE`, `LIST`, `DICT`, `SETITEMS`. - Object instantiation: Emulate `_rebuild_tensor_v2`, `_rebuild_parameter`, `OrderedDict`, and `torch.storage._load_from_bytes`. - **Zero-Code Execution Guarantee:** No native dynamic imports or Python `eval` are ever executed. ### 3. Unsafe Global & Malicious Payload Detection - **RCE Payload Audit:** Inspect all `GLOBAL`, `STACK_GLOBAL`, and `REDUCE` opcodes: - Flag any non-standard or dangerous global call (e.g. `posix.system`, `subprocess.Popen`, `os.system`, `builtins.eval`, `builtins.exec`, `pty.spawn`, `socket.socket`). - Surface exact byte offsets and arguments of malicious exploit payloads. - **Allowlist Verification:** Verify whether all referenced globals belong strictly to standard PyTorch and NumPy namespaces (`torch._utils`, `torch`, `collections.OrderedDict`). ### 4. `state_dict` & Tensor Storage Reconstruction - **Tensor Reconstruction:** Map `_rebuild_tensor_v2` declarations to their corresponding storage files: - Parameter / buffer name (e.g. `model.layers.0.self_attn.q_proj.weight`). - Tensor dimensions / shape (e.g. `[4096, 4096]`). - Data type (`torch.float32`, `torch.float16`, `torch.bfloat16`, `torch.int8`). - Storage offset, stride, and byte length. - **Architecture Parameter Summary:** Derive total parameter count, memory footprint by dtype, and model layer hierarchy directly from the reconstructed `state_dict`. --- ## Acceptance Criteria 1. Introduce `analyze_pytorch_checkpoint` in REA CLI and MCP contracts, accepting `.pt`, `.pth`, `.bin`, and `.ckpt` files. 2. Safely parse synthetic PyTorch checkpoints without invoking Python `pickle.load` or `torch.load`. 3. Output the complete `state_dict` hierarchy, tensor shapes, dtypes, and total parameter counts in structured JSON. 4. Detect synthetic `posix.system` RCE exploit payloads and report typed security violations with byte locations. 5. Enforce bounded parsing budgets, preventing memory exhaustion from malicious recursive pickle data. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3637 [Feature] ML model graph reverse engineering: ONNX, TFLite, CoreML, and TensorRT topology - #3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:49 +00:00

[PR #1372] [MERGED] fix(javascript): decline source-map UNC references with an unusable host #2834