[PR #2433] [MERGED] feat(artifacts): inventory and extract PyInstaller archives #5224

New issue

Closed

opened 2026-10-31 01:04:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:04:56 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2433
Author: @mvanhorn
Created: 10/13/2026
Status: ✅ Merged
Merged: 10/21/2026
Merged by: @gensokyo-dev

Base: main ← Head: fix/pyinstaller-archive-reader

📝 Commits (10+)

6af53a2 feat(artifacts): inventory and extract PyInstaller archives
8aba417 fix(artifacts): keep ZIP and empty members readable and retain the archive owner
b5b0152 fix(artifacts): normalize Windows member names and report physical spans
8728061 fix(artifacts): project PyInstaller symlinks and retain admission owners
2163e03 fix(artifacts): use each PYZ image's magic and scope detection cleanup
e6be538 test(artifacts): build PyInstaller fixtures on a well-formed ELF header
1711796 fix(artifacts): reconcile PyInstaller extraction with main
0a9b47e refactor(artifacts): centralize lazy PyInstaller reads
43a65e7 fix(artifacts): bind PyInstaller reads to stable source identity
9c32198 chore: reconcile captured main with PR #2433

📊 Changes

63 files changed (+7254 additions, -114 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+9 -3)
📝 README.md (+30 -29)
📝 docs/.vitepress/config.ts (+4 -0)
📝 docs/index.md (+13 -13)
➕ docs/python-packaged-applications.md (+128 -0)
📝 docs/readme/README_ar.md (+1 -0)
📝 docs/readme/README_de.md (+1 -0)
📝 docs/readme/README_el.md (+1 -0)
📝 docs/readme/README_es.md (+1 -0)
📝 docs/readme/README_fa.md (+1 -0)
📝 docs/readme/README_fr.md (+1 -0)
📝 docs/readme/README_id.md (+1 -0)
📝 docs/readme/README_it.md (+1 -0)
📝 docs/readme/README_ja.md (+1 -0)
📝 docs/readme/README_ko.md (+1 -0)
📝 docs/readme/README_ml.md (+1 -0)
📝 docs/readme/README_pl.md (+1 -0)
📝 docs/readme/README_pt-BR.md (+1 -0)
📝 docs/readme/README_ru.md (+1 -0)
📝 docs/readme/README_th.md (+1 -0)

...and 43 more files

📄 Description

Summary

Inventory and extract the Python application inside a PyInstaller bootloader through the shared CLI/MCP artifact workflow. Continues @mvanhorn’s original implementation and history. Part of #3678: CArchive/PYZ inventory and extraction; source decompilation, zipapp/PEX and encrypted recovery remain outside this PR.

Behavior

Static parsing and ordinary decompression; no target execution, injection, decryption or external Python dependency. Extraction creates a fresh directory without changing the carrier or opening referenced external executables.
Stable facts preserve original names, cookie/version/library, indices, spans, compression, PYZ state, hashes and derived-header provenance. Runtime options and multipackage references are non-materializable metadata, with unresolved owners explicit. Metadata query names do not occupy filesystem/source-map namespaces.
Shared retained views provide exact batches, filters, selected fields and pages, with complete Evidence accessible. Mixed-kind duplicate identities survive authenticated-page merge order. Opaque tables preserve observed magic, unknown traversal and limitations through compact CLI/MCP responses.
Streamed reads bound decompression and representation growth, validate compressed-input consumption, and preserve short reads, native I/O failures and cancellation reasons. Outer-compressed PYZ images reserve their 128-file resource budget before writing. Diagnostics use bounded table indices.
Encryption markers describe nonempty inner payloads, including legacy namespaces; empty namespaces have no encryption source. Opaque raw PYZ containers remain readable/hashable while retaining marker metadata and unknown child coverage. No decryption is attempted.
Cleanup precedes successful member completion. The reader and private store reuse ArtifactResourceScope to settle every independent owner, aggregate exact cleanup observations and retain only failed owners for retry. Borrowed descriptors remain caller-owned.

Executed validation

Head f11b1fa6ca2e57d582bb16669ccfb23df9c14cc9 integrates main 08f4ba99c. The latest follow-up bounds shared path diagnostics before escaping and reuses verified opaque private images; their regression cases fail before and pass after. Fresh CI 39541970549 completed successfully, including all six shards and platform lanes. Greptile passed and all published actionable threads are resolved. Codex remains running for this head; it is not claimed complete or clean. The superseded 639f120a0 CI passed five shards but failed the unchanged unqualified-doctor test during Java cleanup: EACCES reading another live process environment. This remains CI-only and is not claimed baseline-reproduced; the same test passed locally on the newer-main integration. That workflow completed, so no active obsolete run remained to cancel. The preceding a6f544310 CI also completed successfully. The human-declaration workflow remains deliberately unasserted; it is not a required merge check.

The final merge candidate includes main 044f18022. Its production code passed typecheck/lint, docs and the five CLI/MCP cases after the latest Apple increment; 92 Apple-domain cases and 41 isolated WABT source-boundary cases passed. The final main increment changes only the doctor test: all six cancellation/exit cases passed against the identical compiled production code. The real-data candidate at main 3a0e91dd9 independently verifies the same 1,801-file graph; later main changes touch unrelated Apple/WABT behavior and that test only. Real WABT verification is not claimed.
680 source tests / 54 files and 71 public/boundary tests / nine files: all passed. npm run check passed typecheck, lint, formatting, knip, metadata and architecture guards; docs and five README consistency tests passed. Discovery found all 1,887 files exactly once. Artifact-lock preflight refusals ran no tests; the affected commands were subsequently completed. No local resource guard stopped a task.
Seven latest review regressions fail before and pass after the fixes, including the admitted 10 MiB drive-prefixed path and both opaque-table modes with carrier reads forbidden during repeated opens. Escaped previews and a real 11-byte compressed payload with equal encoded/decoded lengths have separate coverage. Direct source spans now derive from compression flags, never coincident sizes. Shared normalization/collision errors are bounded without limiting accepted paths or shortening retained names.
The opaque-container regression verifies available content identity and exact extracted bytes despite an encryption marker. Two real-descriptor cleanup cases preserve two private failures while allowing the independent carrier to succeed or fail, assert every close and aggregated remaining resources, then verify retries and idempotence. Existing malformed-input, capacity, cancellation, failed admission and borrowed-owner coverage passes.
Latest f11b1fa6c stdio MCP and independent raw-reader verification pass all 1,801 Linux files; the complete canonical graph matches the preceding head. An actual official-writer encrypted opaque image of 33,564,857 bytes passes before/after physical extraction with identical raw digest and unknown coverage. The repeated measurements below preserve identical normalized graphs and extracted bytes; no general speedup is claimed.
Fresh publisher-digest-verified yt-dlp 2026.08.19 Linux (58162f9bfdc27458ea47bfcb311cf47028f17d8154a8bf7d689861d46399230a): actual stdio MCP inspects the app, queries two module identities/hashes, exports full Evidence and performs three warm retained queries. Every one of 1,801 files matches official PyInstaller 6.16 raw-reader hashes/sizes. Complete canonical graphs match before/after. Input/output schemas were checked with the actual SDK/Ajv2020. The latest malformed-path public CLI oracle uses an official writer with a 10 MiB drive-prefixed member: failure output falls from 31,457,633 to 827 bytes, retaining the path refusal and original length. This is measured response size, not actual model-token usage. No native analysis provider or target was launched.
Official PyInstaller 5.13.2 ZlibArchiveWriter, CArchiveWriter and PyiBlockCipher/tinyaes generated encrypted payloads; fixture-only AES verification confirmed them without executing recovered code. After making their magic/table unsupported, the old CLI reports an unavailable container identity and refuses extraction. The current CLI verifies and extracts the exact raw image for both cases, retaining the marker and unknown child coverage.
Earlier pinned real validation remains: 46c2eaaed inventories of Linux/Windows/macOS yt-dlp match 5,251 files; physical extraction at ffba2902e independently rehashed 5,251 files / 245,109,764 bytes, with four macOS symlinks unmaterialized. Official 5.13.2/6.16 namespace and two actual MERGE applications were checked in earlier follow-ups; missing external references stay unresolved. Windows/macOS artifacts were parsed on Linux; native hosts/disassemblers were not exercised. Those physical extraction runs were not repeated on this head.

Performance and limits

Equivalent completed stdio MCP tasks, Node 24.18.0/Linux, two pinned CPUs per task, fresh processes, three retained warm queries. Tokenx estimates include advertised declarations and argument/reply payloads; whole-catalog counts include output schemas. Initialize/JSON-RPC envelopes and actual model usage are unmeasured. Process-family RSS sampled every 20 ms with an 8 GiB guard; neither run stopped. Host load and OS caches are uncontrolled, so these single paired observations establish no causal speedup.

Revision	Used input declarations + primary estimated tokens	All input declarations + primary	Catalog + all calls	Wall / CPU seconds	Peak family GiB
`639f120a0` + main `df8b4d8a5`	26,786	254,148	1,083,059	20.68 / 24.89	1.183
`f11b1fa6c`	26,786	254,148	1,083,059	19.80 / 24.03	1.117

Opaque-image CLI inspection plus physical extraction, 33,564,857 bytes from the official legacy writer, alternated before/after/after/before on the same two CPUs:

Revision	Wall seconds (two runs)	CPU seconds (two runs)	Sampled peak family GiB
Before	8.26 / 8.77	12.01 / 12.54	0.496 / 0.478
After	7.41 / 7.77	10.66 / 11.38	0.480 / 0.486

Both variants returned identical canonical graphs, raw-image digests, extracted bytes and unknown coverage. Fresh processes; OS caches were not cold-controlled. The regression also forbids carrier reads during repeated opaque opens, directly establishing reuse rather than inferring it from timing. Earlier individual runs varied (including 36.30 s for the real MCP task); their causes remain unattributed. The final CPU measurements subtract initial wait-rusage counters to exclude usage inherited from the launching shell; earlier alternating samples affected by that accounting error are retained but not used here.

Five primary calls include complete Evidence export; three extra calls reuse immutable retained facts, not live-file state. Discovery dominates transport. Existing selection already avoided whole-report delivery; small complete reports can be cheaper. No universal latency, memory or agent-token improvement is claimed. Earlier complete/small workflows were measured at d2aeb15bc and were not remeasured here.

Author readiness

Human owner: @gensokyo-dev

The user explicitly delegated follow-up ownership, ready-for-review publication and necessary merge bypass to this account. Review and validation above were performed by the AI agent; independent human review is not claimed. Human-only attestations remain unasserted.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2433 **Author:** [@mvanhorn](https://github.com/mvanhorn) **Created:** 10/13/2026 **Status:** ✅ Merged **Merged:** 10/21/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `fix/pyinstaller-archive-reader` --- ### 📝 Commits (10+) - [`6af53a2`](https://github.com/morluto/rea/commit/6af53a22a95c86f31041acf580bc3e840c7155d2) feat(artifacts): inventory and extract PyInstaller archives - [`8aba417`](https://github.com/morluto/rea/commit/8aba4170d2ba36704ad12d0edebe5489dd052c27) fix(artifacts): keep ZIP and empty members readable and retain the archive owner - [`b5b0152`](https://github.com/morluto/rea/commit/b5b01522c6e2dcb7c0a23911c42fe8417c979633) fix(artifacts): normalize Windows member names and report physical spans - [`8728061`](https://github.com/morluto/rea/commit/87280615ce24f90f937a686d3168bdaa568575d3) fix(artifacts): project PyInstaller symlinks and retain admission owners - [`2163e03`](https://github.com/morluto/rea/commit/2163e03f77671f3ef714e4811e6607e0c7d46296) fix(artifacts): use each PYZ image's magic and scope detection cleanup - [`e6be538`](https://github.com/morluto/rea/commit/e6be538124a9ee61292f9d578fad64550157b6f0) test(artifacts): build PyInstaller fixtures on a well-formed ELF header - [`1711796`](https://github.com/morluto/rea/commit/17117968948a8a8bb77630a6807b293367076145) fix(artifacts): reconcile PyInstaller extraction with main - [`0a9b47e`](https://github.com/morluto/rea/commit/0a9b47eb45a19890298507409808772bdadd56f7) refactor(artifacts): centralize lazy PyInstaller reads - [`43a65e7`](https://github.com/morluto/rea/commit/43a65e75d5d3cc1ca8988ad5d6c2569b76c084df) fix(artifacts): bind PyInstaller reads to stable source identity - [`9c32198`](https://github.com/morluto/rea/commit/9c321985812e218d3308898fea9f9991fbdf33da) chore: reconcile captured main with PR #2433 ### 📊 Changes **63 files changed** (+7254 additions, -114 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+9 -3) 📝 `README.md` (+30 -29) 📝 `docs/.vitepress/config.ts` (+4 -0) 📝 `docs/index.md` (+13 -13) ➕ `docs/python-packaged-applications.md` (+128 -0) 📝 `docs/readme/README_ar.md` (+1 -0) 📝 `docs/readme/README_de.md` (+1 -0) 📝 `docs/readme/README_el.md` (+1 -0) 📝 `docs/readme/README_es.md` (+1 -0) 📝 `docs/readme/README_fa.md` (+1 -0) 📝 `docs/readme/README_fr.md` (+1 -0) 📝 `docs/readme/README_id.md` (+1 -0) 📝 `docs/readme/README_it.md` (+1 -0) 📝 `docs/readme/README_ja.md` (+1 -0) 📝 `docs/readme/README_ko.md` (+1 -0) 📝 `docs/readme/README_ml.md` (+1 -0) 📝 `docs/readme/README_pl.md` (+1 -0) 📝 `docs/readme/README_pt-BR.md` (+1 -0) 📝 `docs/readme/README_ru.md` (+1 -0) 📝 `docs/readme/README_th.md` (+1 -0) _...and 43 more files_ </details> ### 📄 Description ## Summary Inventory and extract the Python application inside a PyInstaller bootloader through the shared CLI/MCP artifact workflow. Continues @mvanhorn’s original implementation and history. Part of #3678: CArchive/PYZ inventory and extraction; source decompilation, zipapp/PEX and encrypted recovery remain outside this PR. ## Behavior - Static parsing and ordinary decompression; no target execution, injection, decryption or external Python dependency. Extraction creates a fresh directory without changing the carrier or opening referenced external executables. - Stable facts preserve original names, cookie/version/library, indices, spans, compression, PYZ state, hashes and derived-header provenance. Runtime options and multipackage references are non-materializable metadata, with unresolved owners explicit. Metadata query names do not occupy filesystem/source-map namespaces. - Shared retained views provide exact batches, filters, selected fields and pages, with complete Evidence accessible. Mixed-kind duplicate identities survive authenticated-page merge order. Opaque tables preserve observed magic, unknown traversal and limitations through compact CLI/MCP responses. - Streamed reads bound decompression and representation growth, validate compressed-input consumption, and preserve short reads, native I/O failures and cancellation reasons. Outer-compressed PYZ images reserve their 128-file resource budget before writing. Diagnostics use bounded table indices. - Encryption markers describe nonempty inner payloads, including legacy namespaces; empty namespaces have no encryption source. Opaque raw PYZ containers remain readable/hashable while retaining marker metadata and unknown child coverage. No decryption is attempted. - Cleanup precedes successful member completion. The reader and private store reuse ArtifactResourceScope to settle every independent owner, aggregate exact cleanup observations and retain only failed owners for retry. Borrowed descriptors remain caller-owned. ## Executed validation Head `f11b1fa6ca2e57d582bb16669ccfb23df9c14cc9` integrates main `08f4ba99c`. The latest follow-up bounds shared path diagnostics before escaping and reuses verified opaque private images; their regression cases fail before and pass after. Fresh [CI 39541970549](https://github.com/morluto/rea/actions/runs/39541970549) completed successfully, including all six shards and platform lanes. Greptile passed and all published actionable threads are resolved. Codex remains running for this head; it is not claimed complete or clean. The superseded `639f120a0` CI passed five shards but failed the unchanged unqualified-doctor test during Java cleanup: EACCES reading another live process environment. This remains CI-only and is not claimed baseline-reproduced; the same test passed locally on the newer-main integration. That workflow completed, so no active obsolete run remained to cancel. The preceding `a6f544310` CI also completed successfully. The human-declaration workflow remains deliberately unasserted; it is not a required merge check. - The final merge candidate includes main `044f18022`. Its production code passed typecheck/lint, docs and the five CLI/MCP cases after the latest Apple increment; 92 Apple-domain cases and 41 isolated WABT source-boundary cases passed. The final main increment changes only the doctor test: all six cancellation/exit cases passed against the identical compiled production code. The real-data candidate at main `3a0e91dd9` independently verifies the same 1,801-file graph; later main changes touch unrelated Apple/WABT behavior and that test only. Real WABT verification is not claimed. - **680 source tests / 54 files** and **71 public/boundary tests / nine files**: all passed. `npm run check` passed typecheck, lint, formatting, knip, metadata and architecture guards; docs and five README consistency tests passed. Discovery found all 1,887 files exactly once. Artifact-lock preflight refusals ran no tests; the affected commands were subsequently completed. No local resource guard stopped a task. - Seven latest review regressions fail before and pass after the fixes, including the admitted 10 MiB drive-prefixed path and both opaque-table modes with carrier reads forbidden during repeated opens. Escaped previews and a real 11-byte compressed payload with equal encoded/decoded lengths have separate coverage. Direct source spans now derive from compression flags, never coincident sizes. Shared normalization/collision errors are bounded without limiting accepted paths or shortening retained names. - The opaque-container regression verifies available content identity and exact extracted bytes despite an encryption marker. Two real-descriptor cleanup cases preserve two private failures while allowing the independent carrier to succeed or fail, assert every close and aggregated remaining resources, then verify retries and idempotence. Existing malformed-input, capacity, cancellation, failed admission and borrowed-owner coverage passes. - Latest `f11b1fa6c` stdio MCP and independent raw-reader verification pass all 1,801 Linux files; the complete canonical graph matches the preceding head. An actual official-writer encrypted opaque image of **33,564,857 bytes** passes before/after physical extraction with identical raw digest and unknown coverage. The repeated measurements below preserve identical normalized graphs and extracted bytes; no general speedup is claimed. - Fresh publisher-digest-verified **yt-dlp 2026.08.19 Linux** (`58162f9bfdc27458ea47bfcb311cf47028f17d8154a8bf7d689861d46399230a`): actual stdio MCP inspects the app, queries two module identities/hashes, exports full Evidence and performs three warm retained queries. Every one of **1,801 files** matches official PyInstaller 6.16 raw-reader hashes/sizes. Complete canonical graphs match before/after. Input/output schemas were checked with the actual SDK/Ajv2020. The latest malformed-path public CLI oracle uses an official writer with a 10 MiB drive-prefixed member: failure output falls from **31,457,633 to 827 bytes**, retaining the path refusal and original length. This is measured response size, not actual model-token usage. No native analysis provider or target was launched. - Official **PyInstaller 5.13.2** ZlibArchiveWriter, CArchiveWriter and PyiBlockCipher/tinyaes generated encrypted payloads; fixture-only AES verification confirmed them without executing recovered code. After making their magic/table unsupported, the old CLI reports an unavailable container identity and refuses extraction. The current CLI verifies and extracts the exact raw image for both cases, retaining the marker and unknown child coverage. - Earlier pinned real validation remains: `46c2eaaed` inventories of Linux/Windows/macOS yt-dlp match **5,251 files**; physical extraction at `ffba2902e` independently rehashed **5,251 files / 245,109,764 bytes**, with four macOS symlinks unmaterialized. Official 5.13.2/6.16 namespace and two actual MERGE applications were checked in earlier follow-ups; missing external references stay unresolved. Windows/macOS artifacts were parsed on Linux; native hosts/disassemblers were not exercised. Those physical extraction runs were not repeated on this head. ## Performance and limits Equivalent completed stdio MCP tasks, Node 24.18.0/Linux, two pinned CPUs per task, fresh processes, three retained warm queries. Tokenx estimates include advertised declarations and argument/reply payloads; whole-catalog counts include output schemas. Initialize/JSON-RPC envelopes and actual model usage are unmeasured. Process-family RSS sampled every 20 ms with an 8 GiB guard; neither run stopped. Host load and OS caches are uncontrolled, so these single paired observations establish no causal speedup. | Revision | Used input declarations + primary estimated tokens | All input declarations + primary | Catalog + all calls | Wall / CPU seconds | Peak family GiB | | --- | ---: | ---: | ---: | --- | ---: | | `639f120a0` + main `df8b4d8a5` | 26,786 | 254,148 | 1,083,059 | 20.68 / 24.89 | 1.183 | | `f11b1fa6c` | 26,786 | 254,148 | 1,083,059 | 19.80 / 24.03 | 1.117 | Opaque-image CLI inspection plus physical extraction, 33,564,857 bytes from the official legacy writer, alternated before/after/after/before on the same two CPUs: | Revision | Wall seconds (two runs) | CPU seconds (two runs) | Sampled peak family GiB | | --- | --- | --- | --- | | Before | 8.26 / 8.77 | 12.01 / 12.54 | 0.496 / 0.478 | | After | 7.41 / 7.77 | 10.66 / 11.38 | 0.480 / 0.486 | Both variants returned identical canonical graphs, raw-image digests, extracted bytes and unknown coverage. Fresh processes; OS caches were not cold-controlled. The regression also forbids carrier reads during repeated opaque opens, directly establishing reuse rather than inferring it from timing. Earlier individual runs varied (including 36.30 s for the real MCP task); their causes remain unattributed. The final CPU measurements subtract initial wait-rusage counters to exclude usage inherited from the launching shell; earlier alternating samples affected by that accounting error are retained but not used here. Five primary calls include complete Evidence export; three extra calls reuse immutable retained facts, not live-file state. Discovery dominates transport. Existing selection already avoided whole-report delivery; small complete reports can be cheaper. No universal latency, memory or agent-token improvement is claimed. Earlier complete/small workflows were measured at `d2aeb15bc` and were not remeasured here. ## Author readiness Human owner: @gensokyo-dev The user explicitly delegated follow-up ownership, ready-for-review publication and necessary merge bypass to this account. Review and validation above were performed by the AI agent; independent human review is not claimed. Human-only attestations remain unasserted. - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:04:56 +00:00