[GH-ISSUE #3678] [Feature] Python packaged application reverse engineering: PyInstaller, PYZ, zipapp and frozen-module reconstruction #1391

New issue

Open

opened 2026-10-16 21:05:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:02 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3678

Summary

Python applications across security utilities, desktop GUI tools, malware payloads, CLI tools, and enterprise scripts are frequently packaged and distributed as standalone native executables using freezing tools:

PyInstaller: The most popular Python freezer, bundling a native bootloader (ELF, PE, Mach-O), compressed CArchive cookie, and PYZ archive containing marshalled Python bytecode (.pyc);
Py2exe / cx_Freeze / Py2app: Windows, Linux, and macOS standalone Python packages;
Zipapp / PEX (Python Executable): Self-contained executable ZIP archives with a __main__.py entrypoint.

While issues #2296 and #2383 examine multi-process Python runtimes, issue #2296 explicitly left PyInstaller executable unpacking as a separate follow-up. Currently, REA analyzes a PyInstaller executable as an ordinary native binary in Ghidra, failing to reveal the underlying Python application code.

This feature proposal introduces dedicated Python Packaged Application Reverse Engineering to REA, enabling automated PyInstaller bootloader extraction, PYZ archive decompression, Python bytecode (.pyc) magic validation, module unbundling, and Python AST reconstruction.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: PyInstaller Executable (PE/ELF)  |
                                    |        or Zipapp / PEX Archive          |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | PyInstaller CArchive      |                                   | Zipapp / PEX Container    |
           | Cookie Scanner (MEI\012) |                                   | Extractor (ZIP manifest)  |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Table of Contents & PYZ Archive Parser|
                                     | (Compressed Zlib Bytecode Streams)    |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Python Bytecode (.pyc)    |                                   | Native C-Extension &      |
           | Header Synthesizer (Magic)|                                   | Dependency Mapper (.pyd)  |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Python AST & Source Decompilation     |
                                     | (Decompile pyc -> clean .py scripts)  |
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Extracted Application   |
                                     | Tree, Python Entrypoint & Dependencies|
                                     +---------------------------------------+

Trailer & Overlay Scanner: Scan the tail of PE, ELF, or Mach-O binaries for the PyInstaller magic cookies:
- MEI\x0c\x0b\x0a\x0b\x0e (PyInstaller 2.1+) or MEI\x01\x02\x03\x04\x05\x06.
CArchive Header: Parse package length, table of contents (TOC) offset, TOC length, and Python version string (e.g. python311).

2. Table of Contents & PYZ Archive Parsing

TOC Records: Parse TOC entries: entry name, compression flag, compressed length, uncompressed length, data offset, and type code:
- s (Python script / entrypoint).
- m (Python module).
- b (Binary / C-extension .pyd / .so).
- z (PYZ archive containing pure Python modules).
- x (Data file / embedded asset).
PYZ Decompressor: Parse the embedded PYZ archive, extracting all zlib-compressed compiled module bytecode streams.

3. Python Bytecode (`.pyc`) Reconstruction

Magic Header Synthesis: PyInstaller strips the standard 16-byte .pyc header to save space. Reconstruct the valid .pyc header for the detected Python version:
- 4-byte magic number (matching Python 2.7, 3.6, 3.8, 3.10, 3.11, 3.12, 3.13+).
- 4-byte bitmask flags (e.g. hash-based vs. timestamp-based).
- 4-byte modification timestamp.
- 4-byte source size.
Decompilation to Source: Decompile recovered .pyc files into clean, readable Python source code (.py).

4. Application Graph & Dependency Wiring

Entrypoint Identification: Detect the primary main execution script (e.g. pyi-windows-launcher, app.py).
Native Extension Mapping: Inventory all bundled native .pyd / .so libraries (e.g. numpy, cryptography, ctypes), linking them to REA's native Ghidra analysis.

Acceptance Criteria

Introduce analyze_pyinstaller_executable in REA CLI and MCP contracts, accepting PE, ELF, and Mach-O executables.
Locate PyInstaller cookies and unpack TOC records across synthetic PyInstaller 3.x/4.x/5.x/6.x test fixtures.
Extract embedded PYZ archives and reconstruct valid .pyc files with appropriate Python version magic bytes.
Decompile synthetic Python bytecode files into valid Python source code.
Identify the application entrypoint and export the complete reconstructed project tree in structured evidence.

#2296 [Feature] Multi-process Python application evidence
#2383 [Tracker] Audit and subsystem index of active bug reports
#3624 [Feature] Native C++ semantic reconstruction

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3678 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Python applications across security utilities, desktop GUI tools, malware payloads, CLI tools, and enterprise scripts are frequently packaged and distributed as standalone native executables using freezing tools: - **PyInstaller:** The most popular Python freezer, bundling a native bootloader (ELF, PE, Mach-O), compressed CArchive cookie, and `PYZ` archive containing marshalled Python bytecode (`.pyc`); - **Py2exe / cx_Freeze / Py2app:** Windows, Linux, and macOS standalone Python packages; - **Zipapp / PEX (Python Executable):** Self-contained executable ZIP archives with a `__main__.py` entrypoint. While issues #2296 and #2383 examine multi-process Python runtimes, issue #2296 explicitly left PyInstaller executable unpacking as a separate follow-up. Currently, REA analyzes a PyInstaller executable as an ordinary native binary in Ghidra, failing to reveal the underlying Python application code. This feature proposal introduces dedicated **Python Packaged Application Reverse Engineering** to REA, enabling automated PyInstaller bootloader extraction, PYZ archive decompression, Python bytecode (`.pyc`) magic validation, module unbundling, and Python AST reconstruction. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: PyInstaller Executable (PE/ELF) | | or Zipapp / PEX Archive | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | PyInstaller CArchive | | Zipapp / PEX Container | | Cookie Scanner (MEI\012) | | Extractor (ZIP manifest) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Table of Contents & PYZ Archive Parser| | (Compressed Zlib Bytecode Streams) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Python Bytecode (.pyc) | | Native C-Extension & | | Header Synthesizer (Magic)| | Dependency Mapper (.pyd) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | Python AST & Source Decompilation | | (Decompile pyc -> clean .py scripts) | +-------------------+-------------------+ | v +---------------------------------------+ | REA Evidence: Extracted Application | | Tree, Python Entrypoint & Dependencies| +---------------------------------------+ ``` ### 1. PyInstaller CArchive Cookie Detection & Extraction - **Trailer & Overlay Scanner:** Scan the tail of PE, ELF, or Mach-O binaries for the PyInstaller magic cookies: - `MEI\x0c\x0b\x0a\x0b\x0e` (PyInstaller 2.1+) or `MEI\x01\x02\x03\x04\x05\x06`. - **CArchive Header:** Parse package length, table of contents (TOC) offset, TOC length, and Python version string (e.g. `python311`). ### 2. Table of Contents & PYZ Archive Parsing - **TOC Records:** Parse TOC entries: entry name, compression flag, compressed length, uncompressed length, data offset, and type code: - `s` (Python script / entrypoint). - `m` (Python module). - `b` (Binary / C-extension `.pyd` / `.so`). - `z` (PYZ archive containing pure Python modules). - `x` (Data file / embedded asset). - **PYZ Decompressor:** Parse the embedded `PYZ` archive, extracting all zlib-compressed compiled module bytecode streams. ### 3. Python Bytecode (`.pyc`) Reconstruction - **Magic Header Synthesis:** PyInstaller strips the standard 16-byte `.pyc` header to save space. Reconstruct the valid `.pyc` header for the detected Python version: - 4-byte magic number (matching Python 2.7, 3.6, 3.8, 3.10, 3.11, 3.12, 3.13+). - 4-byte bitmask flags (e.g. hash-based vs. timestamp-based). - 4-byte modification timestamp. - 4-byte source size. - **Decompilation to Source:** Decompile recovered `.pyc` files into clean, readable Python source code (`.py`). ### 4. Application Graph & Dependency Wiring - **Entrypoint Identification:** Detect the primary main execution script (e.g. `pyi-windows-launcher`, `app.py`). - **Native Extension Mapping:** Inventory all bundled native `.pyd` / `.so` libraries (e.g. `numpy`, `cryptography`, `ctypes`), linking them to REA's native Ghidra analysis. --- ## Acceptance Criteria 1. Introduce `analyze_pyinstaller_executable` in REA CLI and MCP contracts, accepting PE, ELF, and Mach-O executables. 2. Locate PyInstaller cookies and unpack TOC records across synthetic PyInstaller 3.x/4.x/5.x/6.x test fixtures. 3. Extract embedded PYZ archives and reconstruct valid `.pyc` files with appropriate Python version magic bytes. 4. Decompile synthetic Python bytecode files into valid Python source code. 5. Identify the application entrypoint and export the complete reconstructed project tree in structured evidence. --- ## Related - #2296 [Feature] Multi-process Python application evidence - #2383 [Tracker] Audit and subsystem index of active bug reports - #3624 [Feature] Native C++ semantic reconstruction

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:59 +00:00

[PR #1391] [MERGED] fix(javascript): recognize Electron APIs through renamed bindings #2848

bsakaguchi referenced this issue

2026-10-31 00:43:58 +00:00

[GH-ISSUE #6208] [Bug] Second-argument utilityProcess.fork options lose the service name #3778