morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #2296] [Feature] Decompile static Python bytecode (.pyc) files into labeled source inference #758

New issue

Open

opened 2026-10-15 18:28:03 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:28:03 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2296

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

REA main has no Python bytecode support. parseExecutableHeader in src/domain/binaryTarget.ts recognizes only ELF, MZ, and Mach-O magic, so open_binary on a .pyc fails with the generic unsupported binary format (observed with rea-agents@6.1.0 on a .pyc compiled by CPython 3.14.7; the same code path is present at main aeee5074d1). There is no hint of another route.

#367 ("Add JVM and Python bytecode providers") is closed as completed by PR #449, but that PR only added src/domain/bytecodeProvider.ts, which held schemas, extension-based family detection, and path-based provenance classification. It never read bytecode content and was never wired to a tool. Commit c93422a898 removed it on 2026-10-07 as abandoned test-only scaffolding.

Version coverage is the hard constraint. pycdc's opcode tables stop at Python 3.13, decompyle3 covers 3.7 to 3.8, and uncompyle6 stops at 3.8, so a 3.14 .pyc (magic 2b0e0d0a) is outside all three.

Use cases

An agent receives a .pyc from a distributed tool, malware sample, or CTF challenge and needs the most plausible source, with an explicit statement of which parts the decompiler could not reconstruct.
An analyst needs to know which CPython version produced a bytecode file and whether any configured engine supports that version before trusting the output.
An agent wants decompiled Python with the same evidence semantics REA gives other static artifacts: digests, engine provenance, inference labeling, declared effects, and limits.

Desired outcome and success criteria

A target-free CLI command and MCP tool, similar in shape to inspect_wasm_artifact, that takes one absolute .pyc path and returns header evidence parsed by REA, plus decompiled source from a caller-configured local engine labeled as inference.

Returns the file SHA-256, magic number, resolved CPython version, header flags and timestamp or source-hash fields, the engine name and version (with model identifiers and exact resolved revisions for model-based engines), decompiled source as inference, and per-code-object outcomes that keep successful, partial, and failed code objects distinct.
Fixtures compiled from known source cover at least one version at or below 3.8, one in 3.10 through 3.12, and 3.14. Each fixture either returns source inference or fails with unsupported_target naming its magic and version, and none returns empty or partial source as complete success.
With no engine configured, the tool and capability discovery report an unavailable capability, distinguishable from unsupported_target and from malformed input.
open_binary on a .pyc returns a constraint naming the new tool, as WASM does with wasm_requires_static_inspection.
The tool is offline by default: REA does not install engines, download models, or provision interpreters. A missing local asset fails with an actionable diagnostic instead of triggering a fetch, and a test with network access denied proves this.
Target bytecode never runs. A fixture whose module body would write a marker file leaves no marker.
Size, time, and cancellation limits clean up owned processes and private files and return no partial success.
A real-toolchain verification lane and a docs page record the engine version, the verified version range, and the host profile.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Input is one explicitly selected local .pyc file. REA parses the header itself and runs a caller-supplied local decompiler as an owned subprocess on a private read-only copy. The decompiler may run its own helper subprocesses (for example, compiling candidate source with a matching interpreter to check bytecode equivalence), but the target's bytecode is never imported or executed and no network access is needed.

Out of scope: extracting bytecode from PyInstaller-frozen executables or other packagers (a separate follow-up, because it adds a container format, marshalled archive parsing, and encrypted-archive scope questions), wheel and zipapp inventory, dynamic Python analysis, hosted decompilation services, bundling or installing engines, and defeating obfuscation or encryption.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Compile a small module with CPython 3.14 (python3 -I -c 'import py_compile; py_compile.compile("sample.py", cfile="sample.pyc")') and call open_binary on the result: REA 6.1.0 returns target_unavailable / unsupported binary format. The same file is outside pycdc's tables, which end at bytes/python_3_13.cpp and MAGIC_3_13 in pyc_module.h. Fixtures compiled with 3.8, 3.11, and 3.14 from the same source give a version matrix with known ground truth.

Alternatives considered

PyLingual (syssec-utd/pylingual, GPL-3.0-only, v0.0.1 released 2026-09-21) claims coverage of 3.6 through 3.15 and has a CLI, pylingual [-o OUT_DIR] [-c CONFIG] [-v VERSION] [--timeout S] FILES, that writes .py files whose header comments record per-code-object equivalence results; it has no machine-readable output. It needs Python 3.12 or later with torch and transformers. By default it downloads models from the Hugging Face Hub at the unpinned revision main, and its equivalence check can make uvx --python download interpreters on demand (or use pyenv for 3.6 and 3.7). It has the broadest coverage, but it conflicts with an offline default unless REA can require a pinned config, cached models, and pre-installed interpreters.
pycdc (zrax/pycdc, GPL-3.0, C++, no releases) prints source to stdout and needs no network or models, but stops at 3.13.
decompyle3 and uncompyle6 (GPL-3.0) are offline but stop at 3.8.
rocky/python-xdis (GPL-2.0) disassembles bytecode across versions (README: 1.0 to 3.13) but does not decompile; it could supply disassembly evidence or a version check.
Reviving the removed bytecodeProvider.ts scaffold is not useful, because it never read bytecode content.

Possible direction

Have REA own .pyc header parsing and version routing, and keep the engine caller-supplied through an absolute-path setting, as WABT (REA_WABT_BIN_DIRECTORY) and ILSpy (REA_ILSPY_CMD_PATH) already are. This keeps GPL engines out of the MIT package. A later step could route by version across several engines, for example pycdc up to 3.13 and a model-based engine for 3.14 and later. Open questions for maintainers: whether PyLingual can run fully offline from a pinned local config with pre-cached models (for example with HF_HUB_OFFLINE=1) and pre-installed interpreters, which is unverified; whether its equivalence-check subprocesses fit REA's subprocess ownership and limit model; and whether the first slice should support one engine or several.

References

#2258 and the follow-up gap analysis: https://github.com/morluto/rea/issues/2258#issuecomment-6130226337
#367, PR #449, and #42 (closed managed-runtime and bytecode tracker)
Scaffold removal: github.com/morluto/rea@c93422a898
Contribution scope: https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope
Precedents: https://github.com/morluto/rea/blob/main/docs/wasm-artifacts.md, https://github.com/morluto/rea/blob/main/docs/evm-bytecode.md, https://github.com/morluto/rea/blob/main/docs/managed-code-analysis.md
PyLingual: https://github.com/syssec-utd/pylingual (model config: https://github.com/syssec-utd/pylingual/blob/main/pylingual/decompiler_config.yaml; interpreter provisioning: https://github.com/syssec-utd/pylingual/blob/main/pylingual/utils/generate_bytecode.py)
pycdc version tables: https://github.com/zrax/pycdc/tree/master/bytes
decompyle3: https://github.com/rocky/python-decompile3; uncompyle6: https://github.com/rocky/python-uncompyle6; xdis: https://github.com/rocky/python-xdis
PyInstaller extraction (follow-up candidate): https://github.com/extremecoders-re/pyinstxtractor

Originally created by @coygeek on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2296 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation REA `main` has no Python bytecode support. `parseExecutableHeader` in `src/domain/binaryTarget.ts` recognizes only ELF, MZ, and Mach-O magic, so `open_binary` on a `.pyc` fails with the generic `unsupported binary format` (observed with `rea-agents@6.1.0` on a `.pyc` compiled by CPython 3.14.7; the same code path is present at `main` aeee5074d124414239b6dd440dc79c26bcbd58a4). There is no hint of another route. #367 ("Add JVM and Python bytecode providers") is closed as completed by PR #449, but that PR only added `src/domain/bytecodeProvider.ts`, which held schemas, extension-based family detection, and path-based provenance classification. It never read bytecode content and was never wired to a tool. Commit c93422a898fdbedb7d28709144b4337a2669440b removed it on 2026-10-07 as abandoned test-only scaffolding. Version coverage is the hard constraint. pycdc's opcode tables stop at Python 3.13, decompyle3 covers 3.7 to 3.8, and uncompyle6 stops at 3.8, so a 3.14 `.pyc` (magic `2b0e0d0a`) is outside all three. ### Use cases - An agent receives a `.pyc` from a distributed tool, malware sample, or CTF challenge and needs the most plausible source, with an explicit statement of which parts the decompiler could not reconstruct. - An analyst needs to know which CPython version produced a bytecode file and whether any configured engine supports that version before trusting the output. - An agent wants decompiled Python with the same evidence semantics REA gives other static artifacts: digests, engine provenance, inference labeling, declared effects, and limits. ### Desired outcome and success criteria A target-free CLI command and MCP tool, similar in shape to `inspect_wasm_artifact`, that takes one absolute `.pyc` path and returns header evidence parsed by REA, plus decompiled source from a caller-configured local engine labeled as inference. - Returns the file SHA-256, magic number, resolved CPython version, header flags and timestamp or source-hash fields, the engine name and version (with model identifiers and exact resolved revisions for model-based engines), decompiled source as inference, and per-code-object outcomes that keep successful, partial, and failed code objects distinct. - Fixtures compiled from known source cover at least one version at or below 3.8, one in 3.10 through 3.12, and 3.14. Each fixture either returns source inference or fails with `unsupported_target` naming its magic and version, and none returns empty or partial source as complete success. - With no engine configured, the tool and capability discovery report an unavailable capability, distinguishable from `unsupported_target` and from malformed input. - `open_binary` on a `.pyc` returns a constraint naming the new tool, as WASM does with `wasm_requires_static_inspection`. - The tool is offline by default: REA does not install engines, download models, or provision interpreters. A missing local asset fails with an actionable diagnostic instead of triggering a fetch, and a test with network access denied proves this. - Target bytecode never runs. A fixture whose module body would write a marker file leaves no marker. - Size, time, and cancellation limits clean up owned processes and private files and return no partial success. - A real-toolchain verification lane and a docs page record the engine version, the verified version range, and the host profile. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) Input is one explicitly selected local `.pyc` file. REA parses the header itself and runs a caller-supplied local decompiler as an owned subprocess on a private read-only copy. The decompiler may run its own helper subprocesses (for example, compiling candidate source with a matching interpreter to check bytecode equivalence), but the target's bytecode is never imported or executed and no network access is needed. Out of scope: extracting bytecode from PyInstaller-frozen executables or other packagers (a separate follow-up, because it adds a container format, marshalled archive parsing, and encrypted-archive scope questions), wheel and zipapp inventory, dynamic Python analysis, hosted decompilation services, bundling or installing engines, and defeating obfuscation or encryption. ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Compile a small module with CPython 3.14 (`python3 -I -c 'import py_compile; py_compile.compile("sample.py", cfile="sample.pyc")'`) and call `open_binary` on the result: REA 6.1.0 returns `target_unavailable` / `unsupported binary format`. The same file is outside pycdc's tables, which end at `bytes/python_3_13.cpp` and `MAGIC_3_13` in `pyc_module.h`. Fixtures compiled with 3.8, 3.11, and 3.14 from the same source give a version matrix with known ground truth. ### Alternatives considered - PyLingual (`syssec-utd/pylingual`, GPL-3.0-only, v0.0.1 released 2026-09-21) claims coverage of 3.6 through 3.15 and has a CLI, `pylingual [-o OUT_DIR] [-c CONFIG] [-v VERSION] [--timeout S] FILES`, that writes `.py` files whose header comments record per-code-object equivalence results; it has no machine-readable output. It needs Python 3.12 or later with `torch` and `transformers`. By default it downloads models from the Hugging Face Hub at the unpinned revision `main`, and its equivalence check can make `uvx --python` download interpreters on demand (or use `pyenv` for 3.6 and 3.7). It has the broadest coverage, but it conflicts with an offline default unless REA can require a pinned config, cached models, and pre-installed interpreters. - pycdc (`zrax/pycdc`, GPL-3.0, C++, no releases) prints source to stdout and needs no network or models, but stops at 3.13. - decompyle3 and uncompyle6 (GPL-3.0) are offline but stop at 3.8. - `rocky/python-xdis` (GPL-2.0) disassembles bytecode across versions (README: 1.0 to 3.13) but does not decompile; it could supply disassembly evidence or a version check. - Reviving the removed `bytecodeProvider.ts` scaffold is not useful, because it never read bytecode content. ### Possible direction Have REA own `.pyc` header parsing and version routing, and keep the engine caller-supplied through an absolute-path setting, as WABT (`REA_WABT_BIN_DIRECTORY`) and ILSpy (`REA_ILSPY_CMD_PATH`) already are. This keeps GPL engines out of the MIT package. A later step could route by version across several engines, for example pycdc up to 3.13 and a model-based engine for 3.14 and later. Open questions for maintainers: whether PyLingual can run fully offline from a pinned local config with pre-cached models (for example with `HF_HUB_OFFLINE=1`) and pre-installed interpreters, which is unverified; whether its equivalence-check subprocesses fit REA's subprocess ownership and limit model; and whether the first slice should support one engine or several. ### References - #2258 and the follow-up gap analysis: https://github.com/morluto/rea/issues/2258#issuecomment-6130226337 - #367, PR #449, and #42 (closed managed-runtime and bytecode tracker) - Scaffold removal: https://github.com/morluto/rea/commit/c93422a898fdbedb7d28709144b4337a2669440b - Contribution scope: https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope - Precedents: https://github.com/morluto/rea/blob/main/docs/wasm-artifacts.md, https://github.com/morluto/rea/blob/main/docs/evm-bytecode.md, https://github.com/morluto/rea/blob/main/docs/managed-code-analysis.md - PyLingual: https://github.com/syssec-utd/pylingual (model config: https://github.com/syssec-utd/pylingual/blob/main/pylingual/decompiler_config.yaml; interpreter provisioning: https://github.com/syssec-utd/pylingual/blob/main/pylingual/utils/generate_bytecode.py) - pycdc version tables: https://github.com/zrax/pycdc/tree/master/bytes - decompyle3: https://github.com/rocky/python-decompile3; uncompyle6: https://github.com/rocky/python-uncompyle6; xdis: https://github.com/rocky/python-xdis - PyInstaller extraction (follow-up candidate): https://github.com/extremecoders-re/pyinstxtractor

bsakaguchi added the

P3: low

area: domain

enhancement

labels

2026-10-15 18:28:03 +00:00

bsakaguchi commented

2026-10-15 18:28:04 +00:00

Author

Owner

@Joel-Flores123 commented on GitHub (Oct 12, 2026):

For pyinstaller you should look at pyinstextractor-ng, since the normal pyinstxtractor requires you to be running the same version of python as the binary you are extracting from. The ng fork allows you to extract using any supported python versions.

@Joel-Flores123 commented on GitHub (Oct 12, 2026): For pyinstaller you should look at [pyinstextractor-ng](https://github.com/pyinstxtractor/pyinstxtractor-ng), since the normal pyinstxtractor requires you to be running the same version of python as the binary you are extracting from. The ng fork allows you to extract using any supported python versions.

bsakaguchi commented

2026-10-15 18:28:04 +00:00

Author

Owner

@Joel-Flores123 commented on GitHub (Oct 12, 2026):

Additionally thank you, I did not realize that PyLingual wasn't able to be used offline after downloading the models. I'll see to adding that feature.

@Joel-Flores123 commented on GitHub (Oct 12, 2026): Additionally thank you, I did not realize that PyLingual wasn't able to be used offline after downloading the models. I'll see to adding that feature.

bsakaguchi commented

2026-10-16 21:02:30 +00:00

Author

Owner

@noorranktools commented on GitHub (Oct 15, 2026):

Concrete verified result, not just a survey finding: I tried pycdc (zrax/pycdc, the one tool in the original survey whose opcode tables claim 3.7–3.13 coverage) against real targets this session and it actually works — with two caveats worth recording here for anyone else evaluating it as a REA backend.

Setup: no C++ toolchain was present, so built from scratch: winget install BrechtSanders.WinLibs.POSIX.UCRT (MinGW-w64, gcc 16.2), cmake -G "MinGW Makefiles", mingw32-make.

Caveat 1 — the Release build is broken on Windows/MinGW: -DCMAKE_BUILD_TYPE=Release produces a binary that STATUS_ACCESS_VIOLATION-crashes on every input, including a trivially simple freshly-compiled "hello world" .pyc — not a data problem, a real build-config bug in this toolchain combination. -DCMAKE_BUILD_TYPE=Debug works correctly for the large majority of inputs. (First debugging step wasted time on a false lead: Git-Bash reported the Debug build as "segfaulting" on --help too, which turned out to be a console-handling artifact of that shell — gdb -batch -ex run and plain PowerShell both showed it running --help cleanly. Worth knowing if anyone else hits what looks like a universal crash.)

Caveat 2 — real, version-specific opcode gaps, as flagged in the original survey: hit Unsupported opcode: LOAD_FAST_CHECK (237) and Unsupported opcode: MAKE_CELL (225) on two separate Python 3.12 targets — both Python 3.11+ features (stricter unbound-local checking, closure-cell creation) that this build's 3.12 table doesn't implement. One Python 3.10 target also hit an internal assert(!this->empty()) in the AST block-handling partway through one function (a real decompiler limitation on some control-flow shape, not the build) — though pycdas (the disassembler half) still produced full output for that same file with no issue.

Where it actually worked — real decompiled Python source, not just disassembly, for:

A 3.12 target (AESCipher class — hashlib/Crypto.Random/AES-CBC implementation, full encrypt/decrypt/_pad/_unpad logic, near-perfect reconstruction with only minor artifacts around with-statement teardown)
Two more 3.12 targets (small ONNX-model-loading wrapper classes, full __init__/model-loading logic, confirmed a hardcoded AES key literal in source form)
A 3.10 target, raw marshalled code object (not even a .pyc, via -c -v 3.10) that locally-mismatched marshal.loads() couldn't even parse — pycdc doesn't use the running interpreter's own marshal module, so cross-version input was never the obstacle for it that it is for a stdlib-marshal-based approach. Recovered full source including a literal embedded RSA public key in PEM form, before hitting the assertion above partway through one function.
A second 3.10 target, full clean decompile, zero issues (a short image-processing script — PIL-based image→PDF conversion).

This matches my understanding of where #2433 (pure extraction) and this issue (decompilation) currently sit: acquisition is solved, decompilation has a real, working option for a meaningful version range even though it's imperfect (opcode gaps, occasional internal assertion failures) — closer to "usable with explicit per-failure limitations recorded" than "doesn't exist yet." No code to share (ad hoc build/run, not written for upstreaming), but happy to share exact repro steps/inputs if useful for picking an engine.

@noorranktools commented on GitHub (Oct 15, 2026): Concrete verified result, not just a survey finding: I tried `pycdc` (zrax/pycdc, the one tool in the original survey whose opcode tables claim 3.7–3.13 coverage) against real targets this session and it actually works — with two caveats worth recording here for anyone else evaluating it as a REA backend. **Setup:** no C++ toolchain was present, so built from scratch: `winget install BrechtSanders.WinLibs.POSIX.UCRT` (MinGW-w64, gcc 16.2), `cmake -G "MinGW Makefiles"`, `mingw32-make`. **Caveat 1 — the Release build is broken on Windows/MinGW:** `-DCMAKE_BUILD_TYPE=Release` produces a binary that `STATUS_ACCESS_VIOLATION`-crashes on *every* input, including a trivially simple freshly-compiled "hello world" `.pyc` — not a data problem, a real build-config bug in this toolchain combination. `-DCMAKE_BUILD_TYPE=Debug` works correctly for the large majority of inputs. (First debugging step wasted time on a false lead: Git-Bash reported the Debug build as "segfaulting" on `--help` too, which turned out to be a console-handling artifact of that shell — `gdb -batch -ex run` and plain PowerShell both showed it running `--help` cleanly. Worth knowing if anyone else hits what looks like a universal crash.) **Caveat 2 — real, version-specific opcode gaps, as flagged in the original survey:** hit `Unsupported opcode: LOAD_FAST_CHECK (237)` and `Unsupported opcode: MAKE_CELL (225)` on two separate Python 3.12 targets — both Python 3.11+ features (stricter unbound-local checking, closure-cell creation) that this build's 3.12 table doesn't implement. One Python 3.10 target also hit an internal `assert(!this->empty())` in the AST block-handling partway through one function (a real decompiler limitation on some control-flow shape, not the build) — though `pycdas` (the disassembler half) still produced full output for that same file with no issue. **Where it actually worked — real decompiled Python source, not just disassembly, for:** - A 3.12 target (`AESCipher` class — hashlib/Crypto.Random/AES-CBC implementation, full `encrypt`/`decrypt`/`_pad`/`_unpad` logic, near-perfect reconstruction with only minor artifacts around `with`-statement teardown) - Two more 3.12 targets (small ONNX-model-loading wrapper classes, full `__init__`/model-loading logic, confirmed a hardcoded AES key literal in source form) - A 3.10 target, raw marshalled code object (not even a `.pyc`, via `-c -v 3.10`) that **locally-mismatched `marshal.loads()` couldn't even parse** — `pycdc` doesn't use the running interpreter's own marshal module, so cross-version input was never the obstacle for it that it is for a stdlib-`marshal`-based approach. Recovered full source including a literal embedded RSA public key in PEM form, before hitting the assertion above partway through one function. - A second 3.10 target, full clean decompile, zero issues (a short image-processing script — PIL-based image→PDF conversion). This matches my understanding of where #2433 (pure extraction) and this issue (decompilation) currently sit: acquisition is solved, decompilation has a real, working option for a meaningful version range even though it's imperfect (opcode gaps, occasional internal assertion failures) — closer to "usable with explicit per-failure limitations recorded" than "doesn't exist yet." No code to share (ad hoc build/run, not written for upstreaming), but happy to share exact repro steps/inputs if useful for picking an engine.

bsakaguchi referenced this issue

2026-10-16 21:15:53 +00:00

[PR #758] [MERGED] fix(semantics): preserve direct promise ownership boundaries #2221

bsakaguchi referenced this issue

2026-10-16 21:16:10 +00:00

[PR #794] [CLOSED] fix(semantics): link wrapped arrow Promise returns #2243

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-17 21:01:26 +00:00