[PR #6147] feat: extended artifact formats, Gzip reader, Python/PyInstaller, PE inspector, JVM reader, Minidump parser, and platform portability #7162

New issue

Open

opened 2026-11-01 21:45:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:45:50 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6147
Author: @ahmetozer27
Created: 10/21/2026
Status: 🔄 Open

Base: main ← Head: feat/extended-artifact-formats-and-gzip-reader

📝 Commits (10+)

f144571 feat(artifacts): add gzip reader and extend artifact format classifications
981a28e feat(python): add Python bytecode parser and PyInstaller extractor
6d4667e feat(native,jvm,windows): add PE inspector, Rich header, JVM reader, Minidump parser, and platform portability
06f4c69 chore: merge upstream main and resolve conflicts
b0f0b7c chore: merge origin/main and restore asar reader routing
f063b15 fix(artifacts,native,python,jvm,windows): address review feedback on bounds, layouts, and limits
78fdca3 fix(artifacts,native,python,windows): address review findings on stream handle, section scanning, minidump bounds, and pyz types
83210b2 fix(pe): mark section entropy as undefined when scan is skipped due to budget
5e10030 fix(artifacts,python,jvm): address review feedback on gzip admission, pyz parsing, inflation bounds, and streaming class probe
e76de29 fix(jvm,python,ci): remove unused barrel files and dead JVM code to satisfy knip and update jvm-class acceptance expectations

📊 Changes

35 files changed (+4425 additions, -26 deletions)

View changed files

📝 scripts/lib/test-projects.mjs (+1 -1)
📝 src/application/BinaryTargetResolver.ts (+3 -1)
📝 src/artifacts/ArtifactClassificationProbe.ts (+12 -3)
📝 src/artifacts/ArtifactReader.ts (+1 -0)
➕ src/artifacts/GzipArtifactReader.test.ts (+168 -0)
➕ src/artifacts/GzipArtifactReader.ts (+327 -0)
📝 src/artifacts/extraction/ArtifactExtraction.test.ts (+28 -0)
📝 src/artifacts/extraction/ArtifactExtraction.ts (+4 -0)
📝 src/artifacts/inventory/ArtifactGraphConstruction.test.ts (+169 -0)
📝 src/artifacts/inventory/ArtifactGraphConstruction.ts (+69 -0)
📝 src/artifacts/inventory/classify.ts (+1 -0)
📝 src/artifacts/inventory/reader.ts (+9 -1)
📝 src/artifacts/pyinstaller/CArchiveLayout.ts (+15 -7)
📝 src/contracts/toolOutputSchemaPrimitives.ts (+1 -0)
📝 src/domain/adb/adbDeviceAnalysis.ts (+1 -1)
📝 src/domain/analysisSnapshot.ts (+10 -0)
📝 src/domain/apple/jvmClassFile.ts (+165 -1)
📝 src/domain/artifactGraph.ts (+20 -0)
📝 src/domain/binaryTargetTypes.ts (+2 -1)
📝 src/domain/evidence.ts (+10 -0)

...and 15 more files

📄 Description

Overview

This PR introduces major architectural enhancements to REA, covering new archive formats, Python/PyInstaller extraction, PE binary forensics, JVM bytecode parsing, Windows Minidump crash analysis, Shannon entropy calculation, and cross-platform compatibility.

Key Additions

Extended Artifact Classifications: Added gzip, 7z, tar, bzip2, xz, wasm, sqlite, dex, jvm-class, and python-bytecode to artifact graph and evidence schemas.
GZIP Archive Reader (GzipArtifactReader): Streaming decompressor via node:zlib with RFC 1952 FNAME extraction and ISIZE size probing. Wired into container extraction and root inventory.
Python Bytecode & PEP 552 (PythonBytecode): Header parsing and generation supporting Python 2.7 to 3.13 bytecode specifications and validation.
PyInstaller Binary Extraction (PyInstallerExtractor): Cookie locator, CArchive TOC traversal, embedded PYZ parsing with standalone safe marshal deserializer, and automatic .pyc reconstruction.
Shannon Entropy & Packer Heuristics (ShannonEntropy): Byte entropy calculation (0.0 to 8.0), packer detection heuristics (threshold: 7.2), and chunked entropy heatmap generation.
PE Format & Rich Header Forensics (PeFormatInspector, PeRichHeaderParser): PE32/PE32+ parsing, section entropy calculation, Authenticode certificate extraction, DanS/Rich header XOR decryption, and compiler/toolset telemetry mapping to Visual Studio releases.
Pure TypeScript JVM Class Reader (JvmClassReader): Inspects Java .class files without external JVM dependencies, parsing constant pools, class hierarchies, interfaces, fields, methods, descriptors, and code lengths.
Windows Minidump Forensics (MinidumpReader): Native parsing of .dmp (MDMP) files: SystemInfo (architecture, OS build), Exception records (status codes, crash address, thread ID), and loaded module lists.
Platform Portability: Relaxed rigid Linux-only checks across 5 core adapters (FirmwareCommand, PwntoolsDecoder, EvmoleInterfaceProvider, WakaruCommand, MitmproxyCaptureAdapter), supporting REA_ALLOW_NON_LINUX=1 on Windows/macOS.
Test Coverage: Comprehensive unit test suites (443 tests across 30 suites) all passing in Vitest, 0 TypeScript errors, and 0 Oxlint warnings.

Author readiness

Human owner: @ahmetozer27

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6147 **Author:** [@ahmetozer27](https://github.com/ahmetozer27) **Created:** 10/21/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/extended-artifact-formats-and-gzip-reader` --- ### 📝 Commits (10+) - [`f144571`](https://github.com/morluto/rea/commit/f1445713b83e2c069fe48647399e0cbde9f4ba5a) feat(artifacts): add gzip reader and extend artifact format classifications - [`981a28e`](https://github.com/morluto/rea/commit/981a28e42f70fb1e7578690cd32e82f4852f5b04) feat(python): add Python bytecode parser and PyInstaller extractor - [`6d4667e`](https://github.com/morluto/rea/commit/6d4667e6dab296d78baa0a844398309379740cd3) feat(native,jvm,windows): add PE inspector, Rich header, JVM reader, Minidump parser, and platform portability - [`06f4c69`](https://github.com/morluto/rea/commit/06f4c6969a1e294d5ea6f4eec1484324d3f12fe9) chore: merge upstream main and resolve conflicts - [`b0f0b7c`](https://github.com/morluto/rea/commit/b0f0b7ce86bab21db75aec4d991455fc0cd98b6e) chore: merge origin/main and restore asar reader routing - [`f063b15`](https://github.com/morluto/rea/commit/f063b15f4ee2d77c75daa6ac1f54676ea1e55fb9) fix(artifacts,native,python,jvm,windows): address review feedback on bounds, layouts, and limits - [`78fdca3`](https://github.com/morluto/rea/commit/78fdca3184824ceded96e92c9b184b07ea87b9fc) fix(artifacts,native,python,windows): address review findings on stream handle, section scanning, minidump bounds, and pyz types - [`83210b2`](https://github.com/morluto/rea/commit/83210b2c46662288fa7b9b2b0d4245bed8ebb2e0) fix(pe): mark section entropy as undefined when scan is skipped due to budget - [`5e10030`](https://github.com/morluto/rea/commit/5e100302bb67a584e218a48e2c024f4bbd34a29c) fix(artifacts,python,jvm): address review feedback on gzip admission, pyz parsing, inflation bounds, and streaming class probe - [`e76de29`](https://github.com/morluto/rea/commit/e76de2985b8d126709f7052680e6968030a4fdc8) fix(jvm,python,ci): remove unused barrel files and dead JVM code to satisfy knip and update jvm-class acceptance expectations ### 📊 Changes **35 files changed** (+4425 additions, -26 deletions) <details> <summary>View changed files</summary> 📝 `scripts/lib/test-projects.mjs` (+1 -1) 📝 `src/application/BinaryTargetResolver.ts` (+3 -1) 📝 `src/artifacts/ArtifactClassificationProbe.ts` (+12 -3) 📝 `src/artifacts/ArtifactReader.ts` (+1 -0) ➕ `src/artifacts/GzipArtifactReader.test.ts` (+168 -0) ➕ `src/artifacts/GzipArtifactReader.ts` (+327 -0) 📝 `src/artifacts/extraction/ArtifactExtraction.test.ts` (+28 -0) 📝 `src/artifacts/extraction/ArtifactExtraction.ts` (+4 -0) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.test.ts` (+169 -0) 📝 `src/artifacts/inventory/ArtifactGraphConstruction.ts` (+69 -0) 📝 `src/artifacts/inventory/classify.ts` (+1 -0) 📝 `src/artifacts/inventory/reader.ts` (+9 -1) 📝 `src/artifacts/pyinstaller/CArchiveLayout.ts` (+15 -7) 📝 `src/contracts/toolOutputSchemaPrimitives.ts` (+1 -0) 📝 `src/domain/adb/adbDeviceAnalysis.ts` (+1 -1) 📝 `src/domain/analysisSnapshot.ts` (+10 -0) 📝 `src/domain/apple/jvmClassFile.ts` (+165 -1) 📝 `src/domain/artifactGraph.ts` (+20 -0) 📝 `src/domain/binaryTargetTypes.ts` (+2 -1) 📝 `src/domain/evidence.ts` (+10 -0) _...and 15 more files_ </details> ### 📄 Description ## Overview This PR introduces major architectural enhancements to REA, covering new archive formats, Python/PyInstaller extraction, PE binary forensics, JVM bytecode parsing, Windows Minidump crash analysis, Shannon entropy calculation, and cross-platform compatibility. ### Key Additions 1. **Extended Artifact Classifications**: Added `gzip`, `7z`, `tar`, `bzip2`, `xz`, `wasm`, `sqlite`, `dex`, `jvm-class`, and `python-bytecode` to artifact graph and evidence schemas. 2. **GZIP Archive Reader (`GzipArtifactReader`)**: Streaming decompressor via `node:zlib` with RFC 1952 `FNAME` extraction and `ISIZE` size probing. Wired into container extraction and root inventory. 3. **Python Bytecode & PEP 552 (`PythonBytecode`)**: Header parsing and generation supporting Python 2.7 to 3.13 bytecode specifications and validation. 4. **PyInstaller Binary Extraction (`PyInstallerExtractor`)**: Cookie locator, CArchive TOC traversal, embedded PYZ parsing with standalone safe marshal deserializer, and automatic .pyc reconstruction. 5. **Shannon Entropy & Packer Heuristics (`ShannonEntropy`)**: Byte entropy calculation (0.0 to 8.0), packer detection heuristics (threshold: 7.2), and chunked entropy heatmap generation. 6. **PE Format & Rich Header Forensics (`PeFormatInspector`, `PeRichHeaderParser`)**: PE32/PE32+ parsing, section entropy calculation, Authenticode certificate extraction, DanS/Rich header XOR decryption, and compiler/toolset telemetry mapping to Visual Studio releases. 7. **Pure TypeScript JVM Class Reader (`JvmClassReader`)**: Inspects Java `.class` files without external JVM dependencies, parsing constant pools, class hierarchies, interfaces, fields, methods, descriptors, and code lengths. 8. **Windows Minidump Forensics (`MinidumpReader`)**: Native parsing of `.dmp` (MDMP) files: SystemInfo (architecture, OS build), Exception records (status codes, crash address, thread ID), and loaded module lists. 9. **Platform Portability**: Relaxed rigid Linux-only checks across 5 core adapters (`FirmwareCommand`, `PwntoolsDecoder`, `EvmoleInterfaceProvider`, `WakaruCommand`, `MitmproxyCaptureAdapter`), supporting `REA_ALLOW_NON_LINUX=1` on Windows/macOS. 10. **Test Coverage**: Comprehensive unit test suites (443 tests across 30 suites) all passing in Vitest, 0 TypeScript errors, and 0 Oxlint warnings. ## Author readiness Human owner: @ahmetozer27 - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-11-01 21:45:50 +00:00

bsakaguchi referenced this issue

2026-11-03 13:39:39 +00:00

[PR #7162] [CLOSED] feat(browser): save values in browser scenarios and send them in later steps #7687

bsakaguchi referenced this issue

2026-11-03 13:40:08 +00:00

[PR #7256] [CLOSED] test(android): cover native libraries below runtime-named ancestors #7751

bsakaguchi referenced this issue

2026-11-03 13:40:10 +00:00