[PR #4826] [MERGED] feat(python): inspect CPython pyc headers and marshal code objects #6428

Closed
opened 2026-10-31 01:14:23 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4826
Author: @mvanhorn
Created: 10/17/2026
Status: ✅ Merged
Merged: 10/22/2026
Merged by: @morluto

Base: main ← Head: fix/python-pyc


📝 Commits (10+)

  • e99bf23 feat(python): inspect CPython pyc headers and marshal code objects
  • f0ca35b chore(python): drop unused PythonPycHeader export
  • 5543a90 fix(python): bound pyc expansion and decode marshal like CPython
  • c6c6682 fix(python): list pyc code objects once instead of a recursive tree
  • 0468d7d fix(python): bound shared pyc constants and reject malformed tables
  • a08831d fix(python): bound shared pyc names and read CPython marshal graphs
  • 2214efc fix(python): bound pyc key expansion and shared nested code
  • a8ebcd1 fix(python): correct pyc equality, counts, and nested walks
  • 168fe56 fix(python): bound shared pyc names and keep the first dict key
  • 2689744 merge: integrate main into CPython pyc inspection

📊 Changes

36 files changed (+4457 additions, -2 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/SKILL.md (+7 -0)
📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+10 -0)
📝 docs/.vitepress/config.ts (+1 -0)
📝 docs/cli.md (+11 -0)
📝 docs/index.md (+1 -1)
➕ docs/python-pyc.md (+92 -0)
📝 scripts/lib/catalog-builders.mjs (+5 -0)
📝 scripts/lib/catalog-core.mjs (+2 -0)
📝 scripts/lib/module-boundaries.mjs (+1 -0)
📝 scripts/lib/test-projects.mjs (+1 -1)
📝 src/application/CapabilityInventory.ts (+1 -0)
➕ src/application/python/PythonPycPort.ts (+20 -0)
➕ src/application/python/PythonPycService.test.ts (+123 -0)
➕ src/application/python/PythonPycService.ts (+102 -0)
📝 src/cli.ts (+2 -0)
➕ src/cli/pythonPycCommands.ts (+56 -0)
📝 src/cliCommandNames.ts (+1 -0)
➕ src/composition/pythonPyc.ts (+6 -0)
➕ src/contracts/python/pythonPycToolContracts.ts (+26 -0)
📝 src/contracts/toolContracts.ts (+2 -0)

...and 16 more files

📄 Description

Summary

Add inspect_python_pyc / inspect-python-pyc so an analyst can read one explicit local CPython 3.7–3.13 .pyc header and nested marshal code objects without executing bytecode or recovering source. No linked issue: #2296 asks for decompilation; this change does not decompile.

Problem and expected behavior

Packaged Python artifacts often ship only .pyc files. Analysts need the pinned CPython magic/version, PEP 552 hash or mtime/size, nested function names, and bytecode SHA-256 from one selected file. Magics outside 3.7–3.13, truncated streams, and exhausted budgets fail with no partial success.

Change and scope

A portable in-process parser reads the 16-byte header, walks marshal with FLAG_REF intern handling, and projects TYPE_CODE layouts py37, py38, and py311. Intentionally not included: decompilation, disassembly, invoking CPython, or reading the matching .py.

Follow the contribution scope.
Unresolved scope, authority, side effects, or required verification block
integration. Link the required maintainer scope decision for new providers or
capabilities involving target execution, injection, mutation, or protection
recovery; ordinary fixes within existing boundaries may state "not applicable".
Draft PRs may leave the scope decision pending and the final review checklist
unchecked. Maintainers can help resolve open questions.

Contract and boundary impact

  • Semantic owner and earliest changed stage: domain pythonPyc schema; src/python/PythonPycParser.ts owns header and marshal decoding; CLI/MCP share PythonPycService.
  • Contribution scope fit and linked maintainer decision: format inspection of a caller-selected .pyc; not applicable (no execution, injection, mutation, or protection recovery).
  • CLI and MCP/tool-catalog contract: new inspect_python_pyc / inspect-python-pyc; MCP requires an absolute path; CLI resolves relative paths.
  • Provider, bridge, target-format, or platform compatibility: bundled rea-python-pyc reader; CPython 3.7–3.13 pycs on supported Node hosts; no Python install required at runtime.
  • Evidence, artifact, provenance, or reconstruction contract: observed Evidence bound to the selected path and SHA-256; bytecode retained as length and SHA-256 only.
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): explicit local regular file only; no process launch; owned snapshot cleanup retained on failure.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact (CI generates and validates ignored catalogs and skills/; edit authored instructions in .agents/skills/reverse-engineer-anything/ and never commit build outputs): authored skill and docs updated; catalogs remain generated/uncommitted.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: magic table, parser layouts including nested code and hash-based headers, service input/Evidence locations, filesystem admission, CLI relative-path and malformed input, MCP advertisement, absolute-path rejection, and unknown-magic unsupported_target. Shared empty name tables are charged per copied entry. Equal code-object dictionary keys keep the first key's filename. A wide dictionary is walked without copying its child list on every step.
  • Base reproduction or other evidence: source-owned marshal fixtures plus a real CPython 3.12 hash-based compile in the demo below. A CPython 3.12 dict of equal code keys kept first.py and the replacement value new.py.
  • User-visible CLI/MCP output (if applicable): JSON Evidence with header.python_version and code.nested_code[].name.
  • Remaining proof gaps: live 3.7/3.8/3.11/3.13 interpreters were not installed here; those magics are covered by pinned-table fixtures rather than host-produced pycs.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Recorded before this restack:

  • npm run test:local -- src/domain/python/pythonPyc.test.ts src/python/PythonPycParser.test.ts src/application/python/PythonPycService.test.ts src/contracts/toolEffects.test.ts — pass
  • npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts — pass
  • npm run check — pass (typecheck, lint, format, knip, architecture guards)
  • npm run docs:check — pass
  • npm run verify:test-discovery — pass (1238 files)
  • node scripts/rea.mjs inspect-python-pyc /tmp/rea-python-pyc-demo/sample.pyc --json — CPython 3.12 magic 3531, hash-based header, nested greet

Restack onto main 0e5c511c0, tip 87030cc43:

  • npm run typecheck — pass
  • npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts — 9 passed
  • Do not merge tip 3376d438 (or any commit that keeps fork ci.yml blob 235092e9). Parent f1daf155 matches main's ci.yml (21f317fd) and needs a workflows-capable force-with-lease onto mvanhorn:fix/python-pyc before merge.

Head e1af03d7ba6510cbec59fc9f237c2903fa49ca35 merges main 46cc5ce05 and includes 168fe56ce and e1af03d7b:

  • Shared name-table copies, including empty strings, are charged against max_objects before the report array is built.
  • Equal dictionary keys keep the first key object and replace only its value.
  • Each nested-code frame keeps its child list, so a wide dictionary is not copied on every step.
  • The merge keeps both this pyc workflow and the Erlang BEAM inspection from main. Workflow files were not edited.
  • npm run typecheck — pass
  • npm run verify:module-boundaries — pass
  • npm run test:local -- src/python/PythonPycParser.test.ts src/domain/python/pythonPyc.test.ts src/application/python/PythonPycService.test.ts src/contracts/toolEffects.test.ts src/application/CapabilityInventory.test.ts — 72 passed
  • npm run test:local -- src/domain/erlang/beam.test.ts src/erlang/BeamParser.test.ts src/application/erlang/BeamService.test.ts src/python/PythonPycParser.test.ts — 47 passed
  • npm run test:local -- src/python/PythonPycParser.test.ts — 34 passed, including the 8,000-entry dictionary
  • npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts tests/acceptance/erlangBeamCli.test.ts tests/boundary/filesystem/erlangBeam.test.ts tests/boundary/mcp/erlangBeamMcp.test.ts — 18 passed

Compatibility, safety, and release

  • Breaking changes or migration steps: none; additive tool and CLI command.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable; no engine or GUI.
  • Package or release metadata impact: new public tool name in the generated catalog after CI.
  • Security, privacy, process, or containment review: read-only regular-file snapshot; bytecode is not executed; CPython is not invoked by the tool.

Demo

demo

Author readiness

Human owner: @mvanhorn

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.
Open in Web Open in Cursor 

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4826 **Author:** [@mvanhorn](https://github.com/mvanhorn) **Created:** 10/17/2026 **Status:** ✅ Merged **Merged:** 10/22/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/python-pyc` --- ### 📝 Commits (10+) - [`e99bf23`](https://github.com/morluto/rea/commit/e99bf2304c436f99b4dc2b83c886477223a13c4b) feat(python): inspect CPython pyc headers and marshal code objects - [`f0ca35b`](https://github.com/morluto/rea/commit/f0ca35b7938bbe012c993e9c11f68027721a490c) chore(python): drop unused PythonPycHeader export - [`5543a90`](https://github.com/morluto/rea/commit/5543a90ff7d9d096a9f836bfc9d71fb8a6d467a7) fix(python): bound pyc expansion and decode marshal like CPython - [`c6c6682`](https://github.com/morluto/rea/commit/c6c6682c745d684187db0635254b44b8563c9ab1) fix(python): list pyc code objects once instead of a recursive tree - [`0468d7d`](https://github.com/morluto/rea/commit/0468d7df5cf8815f1035a3952be390776336c228) fix(python): bound shared pyc constants and reject malformed tables - [`a08831d`](https://github.com/morluto/rea/commit/a08831d222f5dd591982f80f95a045b2a3d63ddc) fix(python): bound shared pyc names and read CPython marshal graphs - [`2214efc`](https://github.com/morluto/rea/commit/2214efce256a54b2641fb2fe881c4a5fe5e79cbd) fix(python): bound pyc key expansion and shared nested code - [`a8ebcd1`](https://github.com/morluto/rea/commit/a8ebcd133aaff8e174874f23d435ed6cd81e9009) fix(python): correct pyc equality, counts, and nested walks - [`168fe56`](https://github.com/morluto/rea/commit/168fe56ce24d5a770948265705abc43cda535a85) fix(python): bound shared pyc names and keep the first dict key - [`2689744`](https://github.com/morluto/rea/commit/2689744d57e571f0c62d08c52db8c4eb320c1a46) merge: integrate main into CPython pyc inspection ### 📊 Changes **36 files changed** (+4457 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/SKILL.md` (+7 -0) 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+10 -0) 📝 `docs/.vitepress/config.ts` (+1 -0) 📝 `docs/cli.md` (+11 -0) 📝 `docs/index.md` (+1 -1) ➕ `docs/python-pyc.md` (+92 -0) 📝 `scripts/lib/catalog-builders.mjs` (+5 -0) 📝 `scripts/lib/catalog-core.mjs` (+2 -0) 📝 `scripts/lib/module-boundaries.mjs` (+1 -0) 📝 `scripts/lib/test-projects.mjs` (+1 -1) 📝 `src/application/CapabilityInventory.ts` (+1 -0) ➕ `src/application/python/PythonPycPort.ts` (+20 -0) ➕ `src/application/python/PythonPycService.test.ts` (+123 -0) ➕ `src/application/python/PythonPycService.ts` (+102 -0) 📝 `src/cli.ts` (+2 -0) ➕ `src/cli/pythonPycCommands.ts` (+56 -0) 📝 `src/cliCommandNames.ts` (+1 -0) ➕ `src/composition/pythonPyc.ts` (+6 -0) ➕ `src/contracts/python/pythonPycToolContracts.ts` (+26 -0) 📝 `src/contracts/toolContracts.ts` (+2 -0) _...and 16 more files_ </details> ### 📄 Description <!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Summary Add `inspect_python_pyc` / `inspect-python-pyc` so an analyst can read one explicit local CPython 3.7–3.13 `.pyc` header and nested marshal code objects without executing bytecode or recovering source. No linked issue: #2296 asks for decompilation; this change does not decompile. ## Problem and expected behavior Packaged Python artifacts often ship only `.pyc` files. Analysts need the pinned CPython magic/version, PEP 552 hash or mtime/size, nested function names, and bytecode SHA-256 from one selected file. Magics outside 3.7–3.13, truncated streams, and exhausted budgets fail with no partial success. ## Change and scope A portable in-process parser reads the 16-byte header, walks marshal with FLAG_REF intern handling, and projects TYPE_CODE layouts `py37`, `py38`, and `py311`. Intentionally not included: decompilation, disassembly, invoking CPython, or reading the matching `.py`. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Unresolved scope, authority, side effects, or required verification block integration. Link the required maintainer scope decision for new providers or capabilities involving target execution, injection, mutation, or protection recovery; ordinary fixes within existing boundaries may state "not applicable". Draft PRs may leave the scope decision pending and the final review checklist unchecked. Maintainers can help resolve open questions. ## Contract and boundary impact - Semantic owner and earliest changed stage: domain `pythonPyc` schema; `src/python/PythonPycParser.ts` owns header and marshal decoding; CLI/MCP share `PythonPycService`. - Contribution scope fit and linked maintainer decision: format inspection of a caller-selected `.pyc`; not applicable (no execution, injection, mutation, or protection recovery). - CLI and MCP/tool-catalog contract: new `inspect_python_pyc` / `inspect-python-pyc`; MCP requires an absolute path; CLI resolves relative paths. - Provider, bridge, target-format, or platform compatibility: bundled `rea-python-pyc` reader; CPython 3.7–3.13 pycs on supported Node hosts; no Python install required at runtime. - Evidence, artifact, provenance, or reconstruction contract: observed Evidence bound to the selected path and SHA-256; bytecode retained as length and SHA-256 only. - Target selection, authority, upstream behavior, and cleanup details (if applicable): explicit local regular file only; no process launch; owned snapshot cleanup retained on failure. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact (CI generates and validates ignored catalogs and `skills/`; edit authored instructions in `.agents/skills/reverse-engineer-anything/` and never commit build outputs): authored skill and docs updated; catalogs remain generated/uncommitted. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: magic table, parser layouts including nested code and hash-based headers, service input/Evidence locations, filesystem admission, CLI relative-path and malformed input, MCP advertisement, absolute-path rejection, and unknown-magic `unsupported_target`. Shared empty name tables are charged per copied entry. Equal code-object dictionary keys keep the first key's filename. A wide dictionary is walked without copying its child list on every step. - Base reproduction or other evidence: source-owned marshal fixtures plus a real CPython 3.12 hash-based compile in the demo below. A CPython 3.12 dict of equal code keys kept `first.py` and the replacement value `new.py`. - User-visible CLI/MCP output (if applicable): JSON Evidence with `header.python_version` and `code.nested_code[].name`. - Remaining proof gaps: live 3.7/3.8/3.11/3.13 interpreters were not installed here; those magics are covered by pinned-table fixtures rather than host-produced pycs. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Recorded before this restack: - `npm run test:local -- src/domain/python/pythonPyc.test.ts src/python/PythonPycParser.test.ts src/application/python/PythonPycService.test.ts src/contracts/toolEffects.test.ts` — pass - `npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts` — pass - `npm run check` — pass (typecheck, lint, format, knip, architecture guards) - `npm run docs:check` — pass - `npm run verify:test-discovery` — pass (1238 files) - `node scripts/rea.mjs inspect-python-pyc /tmp/rea-python-pyc-demo/sample.pyc --json` — CPython 3.12 magic 3531, hash-based header, nested `greet` Restack onto main `0e5c511c0`, tip `87030cc43`: - `npm run typecheck` — pass - `npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts` — 9 passed - Do not merge tip `3376d438` (or any commit that keeps fork `ci.yml` blob `235092e9`). Parent `f1daf155` matches main's `ci.yml` (`21f317fd`) and needs a workflows-capable force-with-lease onto `mvanhorn:fix/python-pyc` before merge. Head `e1af03d7ba6510cbec59fc9f237c2903fa49ca35` merges main `46cc5ce05` and includes `168fe56ce` and `e1af03d7b`: - Shared name-table copies, including empty strings, are charged against `max_objects` before the report array is built. - Equal dictionary keys keep the first key object and replace only its value. - Each nested-code frame keeps its child list, so a wide dictionary is not copied on every step. - The merge keeps both this pyc workflow and the Erlang BEAM inspection from main. Workflow files were not edited. - `npm run typecheck` — pass - `npm run verify:module-boundaries` — pass - `npm run test:local -- src/python/PythonPycParser.test.ts src/domain/python/pythonPyc.test.ts src/application/python/PythonPycService.test.ts src/contracts/toolEffects.test.ts src/application/CapabilityInventory.test.ts` — 72 passed - `npm run test:local -- src/domain/erlang/beam.test.ts src/erlang/BeamParser.test.ts src/application/erlang/BeamService.test.ts src/python/PythonPycParser.test.ts` — 47 passed - `npm run test:local -- src/python/PythonPycParser.test.ts` — 34 passed, including the 8,000-entry dictionary - `npm run test:focused -- tests/acceptance/pythonPycCli.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/boundary/mcp/pythonPycMcp.test.ts tests/acceptance/erlangBeamCli.test.ts tests/boundary/filesystem/erlangBeam.test.ts tests/boundary/mcp/erlangBeamMcp.test.ts` — 18 passed ## Compatibility, safety, and release - Breaking changes or migration steps: none; additive tool and CLI command. - Real Hopper/Ghidra, browser, or OS coverage: not applicable; no engine or GUI. - Package or release metadata impact: new public tool name in the generated catalog after CI. - Security, privacy, process, or containment review: read-only regular-file snapshot; bytecode is not executed; CPython is not invoked by the tool. ## Demo ![demo](https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/python-pyc/demo.gif) <video src="https://raw.githubusercontent.com/mvanhorn/pr-demo-assets/main/rea/python-pyc/demo.mp4" controls width="720"></video> ## Author readiness Human owner: @mvanhorn - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-b754cd7f-f9b1-5590-a1a6-78b5b1def10d?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-b754cd7f-f9b1-5590-a1a6-78b5b1def10d&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:14:23 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#6428
No description provided.