[PR #6424] [MERGED] fix(python): reject code keys and set members with unhashable consts #7352

New issue

Closed

opened 2026-11-01 21:47:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:47:15 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6424
Author: @Sriram-PR
Created: 10/23/2026
Status: ✅ Merged
Merged: 10/23/2026
Merged by: @morluto

Base: main ← Head: fix/pyc-unhashable-code-keys

📝 Commits (4)

6f16303 fix(python): reject code keys and set members with unhashable consts
6ae3927 fix(python): refuse to hash a tuple that is still being read
fe5fb7f fix(python): hash a code object's co_consts tuple, not its item list
498e694 chore(python): resolve marshal parser merge conflicts

📊 Changes

3 files changed (+107 additions, -3 deletions)

View changed files

📝 docs/python-pyc.md (+2 -2)
📝 src/python/PythonPycParser.test.ts (+73 -0)
📝 src/python/PythonPycParser.ts (+32 -1)

📄 Description

Summary

inspect_python_pyc accepted a code object as a set member or dictionary key even when one of its consts was a list, and reported the file with complete coverage. CPython refuses the same bytes with TypeError: unhashable type: 'list', because hashing a code object hashes its co_consts tuple. docs/python-pyc.md already says unhashable keys and members fail the walk.

Fixes #6415

Problem and expected behavior

Smallest trigger: a 3.11 pyc whose module consts hold a set (or a dict key) containing a code object whose own consts hold an empty list. On main, inspect-python-pyc exits 0 with coverage.status: "complete" and a {"kind":"set","length":1} summary. Expected: the existing malformed-input outcome, the same one a bare list in a set already gets.

Change and scope

In hash mode, MarshalReader.#leafDescriptor returned #codeKey for a code object. That key compares consts with the constant-key rules, where a list just gets an identity token, so a code object never looked unhashable. Now hash mode first checks that every const of the code object has a hash-mode key (#isCodeHashable, cached per code object), and returns null otherwise, which the existing set/dict paths turn into Unhashable set member. / Unhashable dictionary key..
A const that leads back to the same code object counts as unhashable instead of recursing. CPython 3.11 and 3.14 both segfault loading that shape, so no valid file has it.
The check hashes the co_consts tuple value itself (kept on the code node), as CPython's code_hash does, so a co_consts that is a reference to an unfinished tuple is caught too.
Hashing a tuple that is still being read now counts as unhashable too (follow-up from review). A set inside a tuple can TYPE_REF that tuple, directly or through a code object's consts, before the tuple's later items exist; main hashed it with the items read so far and cached the key, so ({ref 0}, []) was reported complete. CPython 3.11 and 3.14 segfault on both shapes.
The equality key itself is unchanged, so dedup of equal code members and keys works as before.
docs/python-pyc.md: code objects are valid keys and members only when all of their consts are hashable.

Contribution scope: ordinary fix within existing boundaries; no scope decision needed.

Contract and boundary impact

Semantic owner and earliest changed stage: MarshalReader hash-mode keys in src/python/PythonPycParser.ts, during the marshal walk.
Contribution scope fit and linked maintainer decision: not applicable (bug fix, #6415).
CLI and MCP/tool-catalog contract: no schema change. These files now fail as invalid_request with an invalid_format issue at path, instead of complete Evidence.
Provider, bridge, target-format, or platform compatibility: none. CPython hashes co_consts on every version in the pinned 3.7 to 3.13 table.
Evidence, artifact, provenance, or reconstruction contract: malformed marshal containers no longer get complete Evidence.
Target selection, authority, upstream behavior, and cleanup details: not applicable.
Generated metadata, package, or installation impact: none.

Expected effects (check known effects; use documentation-only on its own).
Unchecked items do not establish absence; leave uncertainties visible.

Read existing files or captured data.
Launch or attach to a process.
Inject instrumentation or run scripts/code.
Change process memory, registers, files, or configuration.
Contact network services.
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Tests added or updated: src/python/PythonPycParser.test.ts adds rejects code keys and members whose consts are unhashable. A code object with a list const, with a tuple holding a list, and with a nested code object holding a list must each fail as a set member, a frozenset member and a dict key. A code object whose consts lead back to it through a tuple must fail too, as must a set inside a tuple that references that unfinished tuple directly, inside a code object's consts, or as a code object's whole co_consts. Controls: the same code object inside a tuple still parses with complete coverage, and a code object with only hashable consts is still a valid set member.
Base reproduction: the issue's script on main 74ed67da gives cli_exit: 0, coverage: complete for the set and dict cases. The new test fails on main's source (expected function to throw an error, but it didn't); the other 37 tests in the file pass on both. I wrote the test fixtures out and loaded them with marshal.loads in CPython 3.11.17: all 9 rejected cases raise TypeError: unhashable type: 'list', both controls load, and the cyclic one segfaults.

User-visible CLI output for the issue's set case on this branch (remediation omitted):

{"code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":false,"details":{"operation":"inspect_python_pyc","issues":[{"path":["path"],"reason":"invalid_format","message":"Selected pyc .../set.pyc: Unhashable set member."}]}}

With the issue's script on this branch, the set and dict cases exit 1 with no coverage, and the tuple and hashable-set controls still exit 0 with complete coverage.

Remaining proof gaps: none known.

For evidence-bearing changes:

Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

Current conflict-resolution head 498e6945d716bf8da4612a90c0e7fc3fb58b79b2, merging main 630e6becdf3114ca57818c17b36700355ad9696e (macOS, Node 24.22.1/npm 11.16.0):

Retained main's version-specific code equality and constructor-normalized flags alongside this PR's const-tuple hashability checks. Both sets of parser regressions remain. The diff against the merged main remains limited to the original three files.
npm run test:focused -- src/artifacts/pyinstaller/PyInstallerBoundaries.test.ts src/artifacts/pyinstaller/MarshalSubset.test.ts src/artifacts/pyinstaller/PyzArchive.test.ts src/artifacts/pyinstaller/PyInstallerAdmissionBoundaries.test.ts src/artifacts/pyinstaller/PyInstallerFileAdmission.test.ts src/artifacts/pyinstaller/PyInstallerArtifactReader.test.ts src/artifacts/pyinstaller/PyInstallerFormatBoundaries.test.ts src/python/PythonPycParser.test.ts src/application/python/PythonPycService.test.ts tests/boundary/mcp/pythonPycMcp.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/acceptance/pythonPycCli.test.ts tests/acceptance/pyinstallerArchive.test.ts tests/boundary/mcp/pyinstallerArtifactMcp.test.ts: 208 tests passed in 14 files (31.03s).
npm run check, npm run docs:check (including 9 README consistency tests), and formatting checks on the three changed files passed. Normal pre-commit and pre-push (check:fast) checks passed.
No fresh CPython oracle replay, full suite, Linux or Windows run on this merge head. Earlier CPython reproduction and validation below are historical.
npx vitest run src/python/ src/application/python/ src/artifacts/pyinstaller/: 164 passed (9 files)
npm run test:focused -- tests/boundary/mcp/pythonPycMcp.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/acceptance/pythonPycCli.test.ts tests/acceptance/pyinstallerArchive.test.ts tests/boundary/mcp/pyinstallerArtifactMcp.test.ts: 14 passed
npm run check:fast: typecheck and lint passed
issue's reproduce.py with CPython 3.11.17 against the built CLI: set/dict now rejected, controls unchanged

Compatibility, safety, and release

Breaking changes or migration steps: none. Files that were wrongly reported complete now fail as malformed input.
Real Hopper/Ghidra, browser, or OS coverage: Linux x64, Node 24.18.0; no provider involved.
Package or release metadata impact: none.
Security, privacy, process, or containment review: none; no bytecode is executed.

Author readiness

Human owner: @Sriram-PR

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6424 **Author:** [@Sriram-PR](https://github.com/Sriram-PR) **Created:** 10/23/2026 **Status:** ✅ Merged **Merged:** 10/23/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/pyc-unhashable-code-keys` --- ### 📝 Commits (4) - [`6f16303`](https://github.com/morluto/rea/commit/6f163035832f31943253c6e4dc50853a88476970) fix(python): reject code keys and set members with unhashable consts - [`6ae3927`](https://github.com/morluto/rea/commit/6ae3927277a24dffe29d82c43fbfd5429a1cefb2) fix(python): refuse to hash a tuple that is still being read - [`fe5fb7f`](https://github.com/morluto/rea/commit/fe5fb7fd6133d587a194680df400f111f4daca16) fix(python): hash a code object's co_consts tuple, not its item list - [`498e694`](https://github.com/morluto/rea/commit/498e6945d716bf8da4612a90c0e7fc3fb58b79b2) chore(python): resolve marshal parser merge conflicts ### 📊 Changes **3 files changed** (+107 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `docs/python-pyc.md` (+2 -2) 📝 `src/python/PythonPycParser.test.ts` (+73 -0) 📝 `src/python/PythonPycParser.ts` (+32 -1) </details> ### 📄 Description ## Summary `inspect_python_pyc` accepted a code object as a set member or dictionary key even when one of its consts was a list, and reported the file with complete coverage. CPython refuses the same bytes with `TypeError: unhashable type: 'list'`, because hashing a code object hashes its `co_consts` tuple. docs/python-pyc.md already says unhashable keys and members fail the walk. Fixes #6415 ## Problem and expected behavior Smallest trigger: a 3.11 pyc whose module consts hold a set (or a dict key) containing a code object whose own consts hold an empty list. On main, `inspect-python-pyc` exits 0 with `coverage.status: "complete"` and a `{"kind":"set","length":1}` summary. Expected: the existing malformed-input outcome, the same one a bare list in a set already gets. ## Change and scope - In hash mode, `MarshalReader.#leafDescriptor` returned `#codeKey` for a code object. That key compares consts with the constant-key rules, where a list just gets an identity token, so a code object never looked unhashable. Now hash mode first checks that every const of the code object has a hash-mode key (`#isCodeHashable`, cached per code object), and returns null otherwise, which the existing set/dict paths turn into `Unhashable set member.` / `Unhashable dictionary key.`. - A const that leads back to the same code object counts as unhashable instead of recursing. CPython 3.11 and 3.14 both segfault loading that shape, so no valid file has it. - The check hashes the `co_consts` tuple value itself (kept on the code node), as CPython's `code_hash` does, so a `co_consts` that is a reference to an unfinished tuple is caught too. - Hashing a tuple that is still being read now counts as unhashable too (follow-up from review). A set inside a tuple can `TYPE_REF` that tuple, directly or through a code object's consts, before the tuple's later items exist; main hashed it with the items read so far and cached the key, so `({ref 0}, [])` was reported complete. CPython 3.11 and 3.14 segfault on both shapes. - The equality key itself is unchanged, so dedup of equal code members and keys works as before. - docs/python-pyc.md: code objects are valid keys and members only when all of their consts are hashable. Contribution scope: ordinary fix within existing boundaries; no scope decision needed. ## Contract and boundary impact - Semantic owner and earliest changed stage: `MarshalReader` hash-mode keys in `src/python/PythonPycParser.ts`, during the marshal walk. - Contribution scope fit and linked maintainer decision: not applicable (bug fix, #6415). - CLI and MCP/tool-catalog contract: no schema change. These files now fail as `invalid_request` with an `invalid_format` issue at `path`, instead of complete Evidence. - Provider, bridge, target-format, or platform compatibility: none. CPython hashes `co_consts` on every version in the pinned 3.7 to 3.13 table. - Evidence, artifact, provenance, or reconstruction contract: malformed marshal containers no longer get complete Evidence. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata, package, or installation impact: none. Expected effects (check known effects; use documentation-only on its own). Unchecked items do not establish absence; leave uncertainties visible. - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: `src/python/PythonPycParser.test.ts` adds `rejects code keys and members whose consts are unhashable`. A code object with a list const, with a tuple holding a list, and with a nested code object holding a list must each fail as a set member, a frozenset member and a dict key. A code object whose consts lead back to it through a tuple must fail too, as must a set inside a tuple that references that unfinished tuple directly, inside a code object's consts, or as a code object's whole `co_consts`. Controls: the same code object inside a tuple still parses with complete coverage, and a code object with only hashable consts is still a valid set member. - Base reproduction: the issue's script on main 74ed67da gives `cli_exit: 0, coverage: complete` for the set and dict cases. The new test fails on main's source (`expected function to throw an error, but it didn't`); the other 37 tests in the file pass on both. I wrote the test fixtures out and loaded them with `marshal.loads` in CPython 3.11.17: all 9 rejected cases raise `TypeError: unhashable type: 'list'`, both controls load, and the cyclic one segfaults. - User-visible CLI output for the issue's set case on this branch (remediation omitted): ``` {"code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":false,"details":{"operation":"inspect_python_pyc","issues":[{"path":["path"],"reason":"invalid_format","message":"Selected pyc .../set.pyc: Unhashable set member."}]}} ``` With the issue's script on this branch, the set and dict cases exit 1 with no coverage, and the tuple and hashable-set controls still exit 0 with complete coverage. - Remaining proof gaps: none known. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed Current conflict-resolution head `498e6945d716bf8da4612a90c0e7fc3fb58b79b2`, merging main `630e6becdf3114ca57818c17b36700355ad9696e` (macOS, Node 24.22.1/npm 11.16.0): - Retained main's version-specific code equality and constructor-normalized flags alongside this PR's const-tuple hashability checks. Both sets of parser regressions remain. The diff against the merged main remains limited to the original three files. - `npm run test:focused -- src/artifacts/pyinstaller/PyInstallerBoundaries.test.ts src/artifacts/pyinstaller/MarshalSubset.test.ts src/artifacts/pyinstaller/PyzArchive.test.ts src/artifacts/pyinstaller/PyInstallerAdmissionBoundaries.test.ts src/artifacts/pyinstaller/PyInstallerFileAdmission.test.ts src/artifacts/pyinstaller/PyInstallerArtifactReader.test.ts src/artifacts/pyinstaller/PyInstallerFormatBoundaries.test.ts src/python/PythonPycParser.test.ts src/application/python/PythonPycService.test.ts tests/boundary/mcp/pythonPycMcp.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/acceptance/pythonPycCli.test.ts tests/acceptance/pyinstallerArchive.test.ts tests/boundary/mcp/pyinstallerArtifactMcp.test.ts`: 208 tests passed in 14 files (31.03s). - `npm run check`, `npm run docs:check` (including 9 README consistency tests), and formatting checks on the three changed files passed. Normal pre-commit and pre-push (`check:fast`) checks passed. - No fresh CPython oracle replay, full suite, Linux or Windows run on this merge head. Earlier CPython reproduction and validation below are historical. - `npx vitest run src/python/ src/application/python/ src/artifacts/pyinstaller/`: 164 passed (9 files) - `npm run test:focused -- tests/boundary/mcp/pythonPycMcp.test.ts tests/boundary/filesystem/pythonPyc.test.ts tests/acceptance/pythonPycCli.test.ts tests/acceptance/pyinstallerArchive.test.ts tests/boundary/mcp/pyinstallerArtifactMcp.test.ts`: 14 passed - `npm run check:fast`: typecheck and lint passed - issue's `reproduce.py` with CPython 3.11.17 against the built CLI: set/dict now rejected, controls unchanged ## Compatibility, safety, and release - Breaking changes or migration steps: none. Files that were wrongly reported complete now fail as malformed input. - Real Hopper/Ghidra, browser, or OS coverage: Linux x64, Node 24.18.0; no provider involved. - Package or release metadata impact: none. - Security, privacy, process, or containment review: none; no bytecode is executed. ## Author readiness Human owner: @Sriram-PR - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:47:15 +00:00