morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6415] [Bug] Pyc inspection accepts unhashable code objects as set members and dictionary keys #3841

New issue

Closed

opened 2026-10-31 00:44:33 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:44:33 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6415

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The original 196-byte set stream and its 198-byte dictionary-key variant both exit 0 through the CLI and succeed through registered stdio MCP with coverage.status: "complete", two code objects named <module> and u, and nested_code: [1]. Their first constant summaries are {"kind":"set","length":1} and {"kind":"dict","length":1}. Matching CPython refuses both with TypeError: unhashable type: 'list'.

The 193-byte tuple control fully loads in CPython, with tuple-shaped co_consts on both code objects and a list as the inner object's first constant. hash(u) fails with that same TypeError. Changing that list constant to a tuple creates a hashable-code set control that both CPython and REA accept. Across the verifier checks and coordinator rerun, the original set and dict variants reproduced three times each through the compiled CLI and once each through registered stdio MCP. The verifier checked successful CLI/MCP normalized results for exact agreement.

Reproduction

From the already built baseline, save the following script as reproduce.py outside the checkout. With Node from the pinned toolchain on PATH, run python3.11 -I -S reproduce.py "$PWD" /absolute/path/to/writable-fixtures from the checkout. The script writes four tiny files and isolates REA configuration under the supplied writable directory. It parses and hashes code objects without executing their bytecode.

import io, json, marshal, os, pathlib, struct, subprocess, sys
# Run with CPython 3.11. Arguments: REA checkout, writable fixture directory.
repo, root = map(pathlib.Path, sys.argv[1:3])
root.mkdir(parents=True, exist_ok=True)
b = bytes.fromhex('a70d0d0a0000000000000000000000006300000000000000000000000001000000000000007302000000530029013c010000006300000000000000000000000001000000000000007302000000530029015b0100000069010000002900290073000000007a0973616d706c652e70797a01757a083c6d6f64756c653e010000007302000000800073000000002900290073000000007a0973616d706c652e70797a083c6d6f64756c653e7a083c6d6f64756c653e01000000730200000080007300000000')
assert len(b) == 196
assert b[46:51] == b'<' + struct.pack('<i', 1)
f = io.BytesIO(b[51:]); marshal.load(f); end = 51 + f.tell()
variants = {
    'set': b,
    'dict': b[:46] + b'{' + b[51:end] + marshal.dumps(1, 2) + b'0' + b[end:],
    'tuple': b[:46] + b')\x01' + b[51:],
    'hashable-set': b.replace(bytes.fromhex('5b010000006901000000'), bytes.fromhex('29016901000000')),
}
u = marshal.loads(variants['tuple'][16:]).co_consts[0][0]
assert isinstance(u.co_consts, tuple) and isinstance(u.co_consts[0], list)
try: hash(u)
except TypeError as e: print('hash(u):', e)
else: raise AssertionError('expected unhashable code')
env = dict(os.environ, REA_LOG_LEVEL='silent', NODE_DISABLE_COMPILE_CACHE='1')
for key in ['HOME','DSH_HOME','XDG_CONFIG_HOME','XDG_CACHE_HOME','XDG_DATA_HOME','XDG_STATE_HOME','TMPDIR']:
    p = root / key.lower(); p.mkdir(exist_ok=True); env[key] = str(p.resolve())
for name, data in variants.items():
    p = root / (name + '.pyc'); p.write_bytes(data)
    try: marshal.loads(data[16:]); oracle = 'loads'
    except TypeError as e: oracle = str(e)
    r = subprocess.run(['node', str(repo / 'scripts/rea.mjs'), 'inspect-python-pyc', str(p.resolve()), '--json'], cwd=root, env=env, text=True, capture_output=True, timeout=30)
    value = json.loads(r.stdout)
    report = value.get('normalized_result')
    print(json.dumps(dict(case=name, bytes=len(data), cpython=oracle, cli_exit=r.returncode, coverage=report['coverage']['status'] if report else None, const=report['code_objects'][0]['consts'][0] if report else None)))

Expected behavior

Unhashable code objects in set-member or dictionary-key positions should fail the marshal walk and return the existing malformed-input outcome without complete Evidence. Code objects with mutable constants remain readable in non-hashed positions, and hashable code objects remain valid keys/members. A valid PEP 552 header does not make a malformed marshal body loadable.

The documented dictionary/set contract promises rejection of unhashable keys and members. CPython code hashing hashes the actual constants tuple, and its marshal container reader propagates insertion failures. PEP 552 supplies the timestamp-header interpretation, not code hashability.

Impact and scope

P2: a bounded input-validation defect lets malformed marshal containers receive complete successful Evidence. The bytes contain a nested code record, so the observation does not establish that the structural edge itself was invented. Ordinary compiler-produced pycs were not shown to fail; real-world prevalence and other interpreter versions are unknown. No bytecode execution, input mutation, process denial, or security-boundary crossing was observed.

A matching CPython parse detects these bounded acyclic fixtures, but it requires an interpreter and does not establish a safe general workaround for arbitrary untrusted marshal streams. No practical general replacement for REA's Python-free validation was verified.

REA version

Source commit 42e1a17d6e (main)

Runtime and platform

Node v24.18.0 (repository pin), macOS 27.0.1 arm64; CPython 3.11.17 only for fixture generation and marshal/hash comparison. No native analysis engine was used.

Target and artifact context

Tiny acyclic CPython 3.11 pyc files; magic 3495, flags 0, timestamp and source size 0. The top-level co_consts is a tuple containing a set/dict/tuple; the nested code object's co_consts is a tuple containing a list. The set file is 196 bytes with SHA-256 1b542ac809cdae524e634540119986122d98a2229c8624147f871f8567192486; dict is 198 bytes and tuple control is 193 bytes.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The executed standalone script prints:

hash(u): unhashable type: 'list'
{"case": "set", "bytes": 196, "cpython": "unhashable type: 'list'", "cli_exit": 0, "coverage": "complete", "const": {"kind": "set", "length": 1}}
{"case": "dict", "bytes": 198, "cpython": "unhashable type: 'list'", "cli_exit": 0, "coverage": "complete", "const": {"kind": "dict", "length": 1}}
{"case": "tuple", "bytes": 193, "cpython": "loads", "cli_exit": 0, "coverage": "complete", "const": {"kind": "tuple", "length": 1}}
{"case": "hashable-set", "bytes": 193, "cpython": "loads", "cli_exit": 0, "coverage": "complete", "const": {"kind": "set", "length": 1}}

The MCP tool was discovered from actual tools/list and called as inspect_python_pyc with the absolute generated fixture path. Original set and dict calls had isError absent/false and the same normalized results as CLI. Positive results were checked through the SDK's advertised tool definition, including artifact length and digest. Direct-list set/dictionary controls returned typed malformed-input errors on both adapters. MCP processes were confirmed exited after closure.

Typed outcome or recovery expected

Return invalid_request, category invalid_input, with an invalid_format issue identifying the unhashable set member or dictionary key, consistent with the direct-list controls. Return no successful complete report for the invalid body.

Analysis or suspected cause

Source-derived: the code descriptor branch returns a code equality identity even in hash mode. Its constant-key representation permits mutable constants by identity, so the non-null key does not establish CPython hashability. This is an explanation from source, not a tested repair.

Additional context

Related #4826 introduced this workflow and discussed ordinary unhashable-member checks; it does not report this code-constant case. #6147 concerns other artifact/Python support; #6120 concerns negative PYZ sizes in a different reader. Searches covered open and closed issues and PRs with multiple pyc/marshal/hash/unhashable keyword sets; no matching duplicate found. Cited parser/docs/tests/adapters are unchanged in fetched main 0281d101307c8884f64d8d31f48e6ca982e3c42a; that revision was source-inspected only. No restoration patch was tested.

Originally created by @coygeek on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6415 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The original 196-byte set stream and its 198-byte dictionary-key variant both exit 0 through the CLI and succeed through registered stdio MCP with `coverage.status: "complete"`, two code objects named `<module>` and `u`, and `nested_code: [1]`. Their first constant summaries are `{"kind":"set","length":1}` and `{"kind":"dict","length":1}`. Matching CPython refuses both with `TypeError: unhashable type: 'list'`. The 193-byte tuple control fully loads in CPython, with tuple-shaped `co_consts` on both code objects and a list as the inner object's first constant. `hash(u)` fails with that same TypeError. Changing that list constant to a tuple creates a hashable-code set control that both CPython and REA accept. Across the verifier checks and coordinator rerun, the original set and dict variants reproduced three times each through the compiled CLI and once each through registered stdio MCP. The verifier checked successful CLI/MCP normalized results for exact agreement. ### Reproduction From the already built baseline, save the following script as `reproduce.py` outside the checkout. With Node from the pinned toolchain on PATH, run `python3.11 -I -S reproduce.py "$PWD" /absolute/path/to/writable-fixtures` from the checkout. The script writes four tiny files and isolates REA configuration under the supplied writable directory. It parses and hashes code objects without executing their bytecode. ```python import io, json, marshal, os, pathlib, struct, subprocess, sys # Run with CPython 3.11. Arguments: REA checkout, writable fixture directory. repo, root = map(pathlib.Path, sys.argv[1:3]) root.mkdir(parents=True, exist_ok=True) b = bytes.fromhex('a70d0d0a0000000000000000000000006300000000000000000000000001000000000000007302000000530029013c010000006300000000000000000000000001000000000000007302000000530029015b0100000069010000002900290073000000007a0973616d706c652e70797a01757a083c6d6f64756c653e010000007302000000800073000000002900290073000000007a0973616d706c652e70797a083c6d6f64756c653e7a083c6d6f64756c653e01000000730200000080007300000000') assert len(b) == 196 assert b[46:51] == b'<' + struct.pack('<i', 1) f = io.BytesIO(b[51:]); marshal.load(f); end = 51 + f.tell() variants = { 'set': b, 'dict': b[:46] + b'{' + b[51:end] + marshal.dumps(1, 2) + b'0' + b[end:], 'tuple': b[:46] + b')\x01' + b[51:], 'hashable-set': b.replace(bytes.fromhex('5b010000006901000000'), bytes.fromhex('29016901000000')), } u = marshal.loads(variants['tuple'][16:]).co_consts[0][0] assert isinstance(u.co_consts, tuple) and isinstance(u.co_consts[0], list) try: hash(u) except TypeError as e: print('hash(u):', e) else: raise AssertionError('expected unhashable code') env = dict(os.environ, REA_LOG_LEVEL='silent', NODE_DISABLE_COMPILE_CACHE='1') for key in ['HOME','DSH_HOME','XDG_CONFIG_HOME','XDG_CACHE_HOME','XDG_DATA_HOME','XDG_STATE_HOME','TMPDIR']: p = root / key.lower(); p.mkdir(exist_ok=True); env[key] = str(p.resolve()) for name, data in variants.items(): p = root / (name + '.pyc'); p.write_bytes(data) try: marshal.loads(data[16:]); oracle = 'loads' except TypeError as e: oracle = str(e) r = subprocess.run(['node', str(repo / 'scripts/rea.mjs'), 'inspect-python-pyc', str(p.resolve()), '--json'], cwd=root, env=env, text=True, capture_output=True, timeout=30) value = json.loads(r.stdout) report = value.get('normalized_result') print(json.dumps(dict(case=name, bytes=len(data), cpython=oracle, cli_exit=r.returncode, coverage=report['coverage']['status'] if report else None, const=report['code_objects'][0]['consts'][0] if report else None))) ``` ### Expected behavior Unhashable code objects in set-member or dictionary-key positions should fail the marshal walk and return the existing malformed-input outcome without complete Evidence. Code objects with mutable constants remain readable in non-hashed positions, and hashable code objects remain valid keys/members. A valid PEP 552 header does not make a malformed marshal body loadable. The [documented dictionary/set contract](https://github.com/morluto/rea/blob/42e1a17d6e6c0487f17be0bae248553c4caba7f7/docs/python-pyc.md#L53) promises rejection of unhashable keys and members. [CPython code hashing](https://github.com/python/cpython/blob/v3.11.17/Objects/codeobject.c#L1668) hashes the actual constants tuple, and its [marshal container reader](https://github.com/python/cpython/blob/v3.11.17/Python/marshal.c#L1205) propagates insertion failures. [PEP 552](https://peps.python.org/pep-0552/#specification) supplies the timestamp-header interpretation, not code hashability. ### Impact and scope P2: a bounded input-validation defect lets malformed marshal containers receive complete successful Evidence. The bytes contain a nested code record, so the observation does not establish that the structural edge itself was invented. Ordinary compiler-produced pycs were not shown to fail; real-world prevalence and other interpreter versions are unknown. No bytecode execution, input mutation, process denial, or security-boundary crossing was observed. A matching CPython parse detects these bounded acyclic fixtures, but it requires an interpreter and does not establish a safe general workaround for arbitrary untrusted marshal streams. No practical general replacement for REA's Python-free validation was verified. ### REA version Source commit 42e1a17d6e6c0487f17be0bae248553c4caba7f7 (main) ### Runtime and platform Node v24.18.0 (repository pin), macOS 27.0.1 arm64; CPython 3.11.17 only for fixture generation and marshal/hash comparison. No native analysis engine was used. ### Target and artifact context Tiny acyclic CPython 3.11 pyc files; magic 3495, flags 0, timestamp and source size 0. The top-level co_consts is a tuple containing a set/dict/tuple; the nested code object's co_consts is a tuple containing a list. The set file is 196 bytes with SHA-256 `1b542ac809cdae524e634540119986122d98a2229c8624147f871f8567192486`; dict is 198 bytes and tuple control is 193 bytes. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The executed standalone script prints: ```text hash(u): unhashable type: 'list' {"case": "set", "bytes": 196, "cpython": "unhashable type: 'list'", "cli_exit": 0, "coverage": "complete", "const": {"kind": "set", "length": 1}} {"case": "dict", "bytes": 198, "cpython": "unhashable type: 'list'", "cli_exit": 0, "coverage": "complete", "const": {"kind": "dict", "length": 1}} {"case": "tuple", "bytes": 193, "cpython": "loads", "cli_exit": 0, "coverage": "complete", "const": {"kind": "tuple", "length": 1}} {"case": "hashable-set", "bytes": 193, "cpython": "loads", "cli_exit": 0, "coverage": "complete", "const": {"kind": "set", "length": 1}} ``` The MCP tool was discovered from actual tools/list and called as `inspect_python_pyc` with the absolute generated fixture path. Original set and dict calls had `isError` absent/false and the same normalized results as CLI. Positive results were checked through the SDK's advertised tool definition, including artifact length and digest. Direct-list set/dictionary controls returned typed malformed-input errors on both adapters. MCP processes were confirmed exited after closure. ### Typed outcome or recovery expected Return `invalid_request`, category `invalid_input`, with an `invalid_format` issue identifying the unhashable set member or dictionary key, consistent with the direct-list controls. Return no successful complete report for the invalid body. ### Analysis or suspected cause Source-derived: the [code descriptor branch](https://github.com/morluto/rea/blob/42e1a17d6e6c0487f17be0bae248553c4caba7f7/src/python/PythonPycParser.ts#L1074) returns a code equality identity even in hash mode. Its constant-key representation permits mutable constants by identity, so the non-null key does not establish CPython hashability. This is an explanation from source, not a tested repair. ### Additional context Related #4826 introduced this workflow and discussed ordinary unhashable-member checks; it does not report this code-constant case. #6147 concerns other artifact/Python support; #6120 concerns negative PYZ sizes in a different reader. Searches covered open and closed issues and PRs with multiple pyc/marshal/hash/unhashable keyword sets; no matching duplicate found. Cited parser/docs/tests/adapters are unchanged in fetched main `0281d101307c8884f64d8d31f48e6ca982e3c42a`; that revision was source-inspected only. No restoration patch was tested.

bsakaguchi closed this issue

2026-10-31 00:44:33 +00:00

bsakaguchi commented

2026-10-31 00:44:34 +00:00

Author

Owner

@Sriram-PR commented on GitHub (Oct 23, 2026):

I'll take this one. Reproduced with your script on current main (74ed67da): the set and dict variants both exit 0 with complete coverage while CPython 3.11 refuses them, and the tuple and hashable-set controls load as expected. The cause is in PythonPycParser.ts: in hash mode #leafDescriptor returns #codeKey for a code object, and that key is built from the consts in const mode, where a list just gets an identity token. So the code object always looks hashable. CPython's code_hash hashes the actual co_consts tuple (same on 3.14, nested tuples too). Plan: in hash mode, a code object is unhashable when any of its consts is, checked with the existing hash-mode walk and cached per code object, plus parser tests for the set member, dict key, nested code and hashable control cases. PR shortly.

@Sriram-PR commented on GitHub (Oct 23, 2026): I'll take this one. Reproduced with your script on current main (74ed67da): the set and dict variants both exit 0 with complete coverage while CPython 3.11 refuses them, and the tuple and hashable-set controls load as expected. The cause is in `PythonPycParser.ts`: in hash mode `#leafDescriptor` returns `#codeKey` for a code object, and that key is built from the consts in const mode, where a list just gets an identity token. So the code object always looks hashable. CPython's `code_hash` hashes the actual `co_consts` tuple (same on 3.14, nested tuples too). Plan: in hash mode, a code object is unhashable when any of its consts is, checked with the existing hash-mode walk and cached per code object, plus parser tests for the set member, dict key, nested code and hashable control cases. PR shortly.

bsakaguchi commented

2026-10-31 00:44:34 +00:00

Author

Owner

@akarsh11111111 commented on GitHub (Oct 23, 2026):

awesome Thanks for the detailed report and reproducible test cases! The comparison with CPython and the tuple/hashable-set controls makes the issue clear. Ensuring that code objects are validated for hashability when used as set members or dictionary keys, while preserving valid cases, sounds like the right approach. Looking forward to the fix and regression tests.

@akarsh11111111 commented on GitHub (Oct 23, 2026): awesome Thanks for the detailed report and reproducible test cases! The comparison with CPython and the tuple/hashable-set controls makes the issue clear. Ensuring that code objects are validated for hashability when used as set members or dictionary keys, while preserving valid cases, sounds like the right approach. Looking forward to the fix and regression tests.

bsakaguchi referenced this issue

2026-10-31 01:10:35 +00:00

[PR #3841] [CLOSED] fix(javascript): attribute service workers only to the navigator global #5940

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425