[GH-ISSUE #6568] [Bug] inspect-python-pyc misclassifies valid 12-byte-header alpha pycs as invalid input #3959

New issue

Closed

opened 2026-10-31 00:45:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:35 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6568

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-python-pyc FILE --json admits CPython 3.7.0a1/a2 magics 3390/3391, then starts marshal decoding at offset 16 even though those releases emitted 12-byte headers. Two unchanged producer files return exit 1, code: "invalid_request", category: "invalid_input", an invalid_format issue ending Unsupported marshal type 0x0., and recovery.kind: "inspect_input_contract". Their valid historical marshal bodies are misclassified as malformed input; the response asks the caller to correct request fields.

Reproduction

Use an already built checkout at the source revision below, with the pinned Node/npm toolchain on PATH. For a fresh checkout, the prerequisites are npm ci --ignore-scripts and npm run build:cached; both were run for this verification in an isolated environment. Save this script outside the checkout as reproduce.py and run python3 reproduce.py "$PWD" from the checkout root. It reconstructs the exact producer files, verifies their digests, and uses the identical marshal body for two synthetic 16-byte-header alignment controls and an unlisted-magic control. It isolates HOME/XDG/temp directories and never loads or executes target bytecode.

import hashlib, json, os, pathlib, struct, subprocess, sys, tempfile
repo = pathlib.Path(sys.argv[1]).resolve()
# co = compile("", "mod.py", "exec", dont_inherit=True); marshal.dumps(co)
# emitted identically by isolated CPython 3.7.0a1 and 3.7.0a2 builds.
body = bytes.fromhex(
    "e30000000000000000000000000100000040000000730400000064005300"
    "29014ea900720100000072010000007201000000fa066d6f642e7079da08"
    "3c6d6f64756c653e010000007300000000")
assert len(body) == 77
with tempfile.TemporaryDirectory() as directory:
    root = pathlib.Path(directory)
    env = {"PATH": os.environ["PATH"]}
    for key in ["HOME", "XDG_CONFIG_HOME", "XDG_CACHE_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", "TMPDIR"]:
        location = root / key.lower(); location.mkdir(); env[key] = str(location)
    env.update(REA_LOG_LEVEL="silent", NODE_DISABLE_COMPILE_CACHE="1")
    for magic, size in [(3390,12),(3391,12),(3392,16),(3394,16),(3389,12)]:
        # 12 bytes: magic, mtime, source size. 16 bytes: magic, flags, mtime, size.
        header = struct.pack("<H", magic) + b"\r\n" + bytes(size-4)
        path = root / (str(magic)+".pyc"); path.write_bytes(header+body)
        assert int.from_bytes(path.read_bytes()[:2], "little") == magic
        if magic in [3390,3391]:
            expected = {3390:"7d9e812ccc96bda12b215c23cd2725ccbb86f055bf3096b9e7599f020b091e49", 3391:"b27e2b775fc6b195d8ff6501158c3e9c434a963d43dac67da2b1762400ef46c4"}
            assert hashlib.sha256(path.read_bytes()).hexdigest() == expected[magic]
        result = subprocess.run(["node", str(repo/"scripts/rea.mjs"), "inspect-python-pyc", str(path), "--json"], env=env, text=True, capture_output=True, timeout=30)
        value = json.loads(result.stdout); report = value.get("normalized_result")
        out = {"magic":magic, "exit":result.returncode}
        if report:
            out.update(coverage=report["coverage"]["status"], bytecode_bytes=report["code_objects"][0]["bytecode"]["bytes"], consts=report["code_objects"][0]["consts"])
        else:
            out.update(code=value.get("code"), category=value.get("category"), issues=[{"reason":i["reason"], "message":i["message"].replace(str(path),"<fixture>")} for i in value.get("details",{}).get("issues",[])], recovery=value.get("recovery"))
        print(json.dumps(out))

Observed output from the freshly built main checkout:

{"magic": 3390, "exit": 1, "code": "invalid_request", "category": "invalid_input", "issues": [{"reason": "invalid_format", "message": "Selected pyc <fixture>: Unsupported marshal type 0x0."}], "recovery": {"kind": "inspect_input_contract", "operation": "inspect_python_pyc"}}
{"magic": 3391, "exit": 1, "code": "invalid_request", "category": "invalid_input", "issues": [{"reason": "invalid_format", "message": "Selected pyc <fixture>: Unsupported marshal type 0x0."}], "recovery": {"kind": "inspect_input_contract", "operation": "inspect_python_pyc"}}
{"magic": 3392, "exit": 0, "coverage": "complete", "bytecode_bytes": 4, "consts": [{"kind": "none"}]}
{"magic": 3394, "exit": 0, "coverage": "complete", "bytecode_bytes": 4, "consts": [{"kind": "none"}]}
{"magic": 3389, "exit": 1, "code": "unsupported_target", "category": "unsupported_target", "issues": [], "recovery": null}

The two unchanged producer files were separately passed through the public CLI and returned the same errors. The 16-byte fixtures are consumer alignment controls, not independent producer-validity evidence.

Expected behavior

Correctly classify an unsupported historical header as unsupported_target, without alleging corrupted marshal data or directing request-field repair. Accurate 12-byte-header decoding would also resolve this mismatch, but is optional: the documented 16-byte scope does not require supporting every prerelease format. Explicitly admitting the magic and then consuming the wrong header length must not turn a format limitation into invalid-input recovery.

Impact and scope

P2: callers cannot distinguish the observed producer-format limitation from malformed data using the returned category/recovery. The measured scope is these two historical alpha formats; real-world prevalence is NOT_ENOUGH_INFO. MCP and other platforms were not exercised. No input mutation, target execution, crash, or data loss was observed.

Impact level: P2
Assessment status: supported
Impact basis: The public CLI returns a consequentially wrong machine-readable category and recovery action for a bounded pair of valid artifacts; severe harm was not observed.
Workaround status: unverified
Workaround basis: No workaround preserving REA inspection was demonstrated. Preclassifying these magics can avoid the misleading call but supplies no inspection result.

REA version

Unreleased main source 6bb4f4bf7d64e5d2f47479d46225a5383b57cdad, freshly built with locked dependencies; package and CLI identify as 6.7.0. The separately installed latest npm release rea-agents@6.7.0 returns exit 1 with code: "COMMAND_NOT_FOUND" for this exact command and every fixture; this is not a published-release reproduction of the parser defect.

Runtime and platform

Node.js v24.22.1; npm 11.16.0; macOS 27.0.1 arm64. Python 3.15.0 only writes bytes and launches the CLI in the replay. The supplied independent producer evidence used core source builds of CPython 3.7.0a1 and 3.7.0a2 with Apple Clang 21.0.0, no installation: configure needed arm64 recognition and compilation included the system socket header. Compiler, marshal, code-object, opcode, and importlib files were verified byte-identical to the release-tag sources. Optional extensions and general historical interpreter compatibility were not tested.

Target and artifact context

Each historical producer compiled an empty module with compile("", "mod.py", "exec", dont_inherit=True) and serialized it with its actual _code_to_bytecode writer using zero mtime/source size. Both emitted 89-byte pycs: a 12-byte header plus the identical 77-byte body embedded above. The compiled code has stack size 1, flags 64 (CO_NOFREE), bytecode 64 00 53 00 (LOAD_CONST 0; RETURN_VALUE), constants (None,), zero argument/local counts, empty name/local/free/cell tables, first line 1, and an empty line table.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Supplied producer logs record each historical runtime's version/magic, emitted bytes, and successful marshal.loads(pyc[12:]) == code round trip without executing the code object. Their headers are 3e0d0d0a0000000000000000 and 3f0d0d0a0000000000000000. The inline digest assertions match those unchanged artifacts exactly; current CLI reproduction uses both the original files and reconstructions. Producer compilation/round trips were performed in prior independent verification, not repeated for this submission.

Expected source: the inspection guide describes 16-byte headers; the magic table admits 3390/3391 while excluding incompatible early 3.11 layouts; the provider distinguishes unsupported targets from malformed input. Exact CPython a1 and a2 writers emit magic, mtime, size, then marshal, without a flags word. The a4 writer adds flags before mtime/size.
Failure source: the executed public CLI output above and separate unchanged-producer-file calls. The parser unconditionally starts marshal at 16 after magic admission.
Evidence provenance: observed current CLI results; supplied independently executed historical producer compilation/serialization and round trips; source-derived header history and explanation. The alignment controls are synthetic.
Local verification: reproduced.
Reproduction completeness: complete.

Typed outcome or recovery expected

For the valid historical files, return unsupported_target in both code and category for an unsupported layout, or accurately inspect their unchanged bytes. Do not return the current invalid_format/inspect_input_contract result. Preserve complete successful coverage for the aligned controls and unsupported-target rejection for the fabricated unlisted magic.

Analysis or suspected cause

Source-derived and consistent with execution: skipping 16 bytes removes the historical 12-byte header plus the marshal code-object tag and first three argument-count bytes. The reader sees the remaining zero argument-count byte as an object tag, producing Unsupported marshal type 0x0. No repair was applied or tested.

Additional context

Open/closed issue and PR searches covered both magics, PEP 552, pyc/header, 3.7a1/a2, 12-byte, pre-PEP, alpha/pyc, and the exact error. The original inspector PR #4826 introduced this 16-byte path; PR #6147 concerns a separate PythonBytecode/PyInstaller path. Reports #6415, #6487, #6490, and #6547 concern hashability, code flags, first-line values, and line-table equality, with distinct triggers and restoration conditions. No matching report or correction was found.

Restoration check: rerun the exact script. Both historical files must receive the typed unsupported-layout outcome or accurate historical-header evidence. The 3392/3394 controls must still return complete coverage, four bytecode bytes and a None constant; the 3389 control must remain unsupported. Rewriting the selected artifact into a synthetic 16-byte file would not restore this contract.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6568 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `inspect-python-pyc FILE --json` admits CPython 3.7.0a1/a2 magics 3390/3391, then starts marshal decoding at offset 16 even though those releases emitted 12-byte headers. Two unchanged producer files return exit 1, `code: "invalid_request"`, `category: "invalid_input"`, an `invalid_format` issue ending `Unsupported marshal type 0x0.`, and `recovery.kind: "inspect_input_contract"`. Their valid historical marshal bodies are misclassified as malformed input; the response asks the caller to correct request fields. ### Reproduction Use an already built checkout at the source revision below, with the pinned Node/npm toolchain on PATH. For a fresh checkout, the prerequisites are `npm ci --ignore-scripts` and `npm run build:cached`; both were run for this verification in an isolated environment. Save this script outside the checkout as `reproduce.py` and run `python3 reproduce.py "$PWD"` from the checkout root. It reconstructs the exact producer files, verifies their digests, and uses the identical marshal body for two synthetic 16-byte-header alignment controls and an unlisted-magic control. It isolates HOME/XDG/temp directories and never loads or executes target bytecode. ```python import hashlib, json, os, pathlib, struct, subprocess, sys, tempfile repo = pathlib.Path(sys.argv[1]).resolve() # co = compile("", "mod.py", "exec", dont_inherit=True); marshal.dumps(co) # emitted identically by isolated CPython 3.7.0a1 and 3.7.0a2 builds. body = bytes.fromhex( "e30000000000000000000000000100000040000000730400000064005300" "29014ea900720100000072010000007201000000fa066d6f642e7079da08" "3c6d6f64756c653e010000007300000000") assert len(body) == 77 with tempfile.TemporaryDirectory() as directory: root = pathlib.Path(directory) env = {"PATH": os.environ["PATH"]} for key in ["HOME", "XDG_CONFIG_HOME", "XDG_CACHE_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", "TMPDIR"]: location = root / key.lower(); location.mkdir(); env[key] = str(location) env.update(REA_LOG_LEVEL="silent", NODE_DISABLE_COMPILE_CACHE="1") for magic, size in [(3390,12),(3391,12),(3392,16),(3394,16),(3389,12)]: # 12 bytes: magic, mtime, source size. 16 bytes: magic, flags, mtime, size. header = struct.pack("<H", magic) + b"\r\n" + bytes(size-4) path = root / (str(magic)+".pyc"); path.write_bytes(header+body) assert int.from_bytes(path.read_bytes()[:2], "little") == magic if magic in [3390,3391]: expected = {3390:"7d9e812ccc96bda12b215c23cd2725ccbb86f055bf3096b9e7599f020b091e49", 3391:"b27e2b775fc6b195d8ff6501158c3e9c434a963d43dac67da2b1762400ef46c4"} assert hashlib.sha256(path.read_bytes()).hexdigest() == expected[magic] result = subprocess.run(["node", str(repo/"scripts/rea.mjs"), "inspect-python-pyc", str(path), "--json"], env=env, text=True, capture_output=True, timeout=30) value = json.loads(result.stdout); report = value.get("normalized_result") out = {"magic":magic, "exit":result.returncode} if report: out.update(coverage=report["coverage"]["status"], bytecode_bytes=report["code_objects"][0]["bytecode"]["bytes"], consts=report["code_objects"][0]["consts"]) else: out.update(code=value.get("code"), category=value.get("category"), issues=[{"reason":i["reason"], "message":i["message"].replace(str(path),"<fixture>")} for i in value.get("details",{}).get("issues",[])], recovery=value.get("recovery")) print(json.dumps(out)) ``` Observed output from the freshly built main checkout: ```text {"magic": 3390, "exit": 1, "code": "invalid_request", "category": "invalid_input", "issues": [{"reason": "invalid_format", "message": "Selected pyc <fixture>: Unsupported marshal type 0x0."}], "recovery": {"kind": "inspect_input_contract", "operation": "inspect_python_pyc"}} {"magic": 3391, "exit": 1, "code": "invalid_request", "category": "invalid_input", "issues": [{"reason": "invalid_format", "message": "Selected pyc <fixture>: Unsupported marshal type 0x0."}], "recovery": {"kind": "inspect_input_contract", "operation": "inspect_python_pyc"}} {"magic": 3392, "exit": 0, "coverage": "complete", "bytecode_bytes": 4, "consts": [{"kind": "none"}]} {"magic": 3394, "exit": 0, "coverage": "complete", "bytecode_bytes": 4, "consts": [{"kind": "none"}]} {"magic": 3389, "exit": 1, "code": "unsupported_target", "category": "unsupported_target", "issues": [], "recovery": null} ``` The two unchanged producer files were separately passed through the public CLI and returned the same errors. The 16-byte fixtures are consumer alignment controls, not independent producer-validity evidence. ### Expected behavior Correctly classify an unsupported historical header as `unsupported_target`, without alleging corrupted marshal data or directing request-field repair. Accurate 12-byte-header decoding would also resolve this mismatch, but is optional: the documented 16-byte scope does not require supporting every prerelease format. Explicitly admitting the magic and then consuming the wrong header length must not turn a format limitation into invalid-input recovery. ### Impact and scope P2: callers cannot distinguish the observed producer-format limitation from malformed data using the returned category/recovery. The measured scope is these two historical alpha formats; real-world prevalence is NOT_ENOUGH_INFO. MCP and other platforms were not exercised. No input mutation, target execution, crash, or data loss was observed. - Impact level: P2 - Assessment status: supported - Impact basis: The public CLI returns a consequentially wrong machine-readable category and recovery action for a bounded pair of valid artifacts; severe harm was not observed. - Workaround status: unverified - Workaround basis: No workaround preserving REA inspection was demonstrated. Preclassifying these magics can avoid the misleading call but supplies no inspection result. ### REA version Unreleased main source `6bb4f4bf7d64e5d2f47479d46225a5383b57cdad`, freshly built with locked dependencies; package and CLI identify as 6.7.0. The separately installed latest npm release `rea-agents@6.7.0` returns exit 1 with `code: "COMMAND_NOT_FOUND"` for this exact command and every fixture; this is not a published-release reproduction of the parser defect. ### Runtime and platform Node.js v24.22.1; npm 11.16.0; macOS 27.0.1 arm64. Python 3.15.0 only writes bytes and launches the CLI in the replay. The supplied independent producer evidence used core source builds of CPython 3.7.0a1 and 3.7.0a2 with Apple Clang 21.0.0, no installation: configure needed arm64 recognition and compilation included the system socket header. Compiler, marshal, code-object, opcode, and importlib files were verified byte-identical to the release-tag sources. Optional extensions and general historical interpreter compatibility were not tested. ### Target and artifact context Each historical producer compiled an empty module with `compile("", "mod.py", "exec", dont_inherit=True)` and serialized it with its actual `_code_to_bytecode` writer using zero mtime/source size. Both emitted 89-byte pycs: a 12-byte header plus the identical 77-byte body embedded above. The compiled code has stack size 1, flags 64 (`CO_NOFREE`), bytecode `64 00 53 00` (`LOAD_CONST 0; RETURN_VALUE`), constants `(None,)`, zero argument/local counts, empty name/local/free/cell tables, first line 1, and an empty line table. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Supplied producer logs record each historical runtime's version/magic, emitted bytes, and successful `marshal.loads(pyc[12:]) == code` round trip without executing the code object. Their headers are `3e0d0d0a0000000000000000` and `3f0d0d0a0000000000000000`. The inline digest assertions match those unchanged artifacts exactly; current CLI reproduction uses both the original files and reconstructions. Producer compilation/round trips were performed in prior independent verification, not repeated for this submission. - Expected source: the [inspection guide](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/docs/python-pyc.md#L3-L7) describes 16-byte headers; the [magic table](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/domain/python/pythonPyc.ts#L43-L67) admits 3390/3391 while excluding incompatible early 3.11 layouts; the [provider](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/python/PythonPycProvider.ts#L61-L75) distinguishes unsupported targets from malformed input. Exact CPython [a1](https://github.com/python/cpython/blob/v3.7.0a1/Lib/importlib/_bootstrap_external.py#L498-L506) and [a2](https://github.com/python/cpython/blob/v3.7.0a2/Lib/importlib/_bootstrap_external.py#L499-L507) writers emit magic, mtime, size, then marshal, without a flags word. The [a4 writer](https://github.com/python/cpython/blob/v3.7.0a4/Lib/importlib/_bootstrap_external.py#L531-L539) adds flags before mtime/size. - Failure source: the executed public CLI output above and separate unchanged-producer-file calls. The [parser](https://github.com/morluto/rea/blob/6bb4f4bf7d64e5d2f47479d46225a5383b57cdad/src/python/PythonPycParser.ts#L183-L210) unconditionally starts marshal at 16 after magic admission. - Evidence provenance: observed current CLI results; supplied independently executed historical producer compilation/serialization and round trips; source-derived header history and explanation. The alignment controls are synthetic. - Local verification: reproduced. - Reproduction completeness: complete. ### Typed outcome or recovery expected For the valid historical files, return `unsupported_target` in both code and category for an unsupported layout, or accurately inspect their unchanged bytes. Do not return the current `invalid_format`/`inspect_input_contract` result. Preserve complete successful coverage for the aligned controls and unsupported-target rejection for the fabricated unlisted magic. ### Analysis or suspected cause Source-derived and consistent with execution: skipping 16 bytes removes the historical 12-byte header plus the marshal code-object tag and first three argument-count bytes. The reader sees the remaining zero argument-count byte as an object tag, producing `Unsupported marshal type 0x0.` No repair was applied or tested. ### Additional context Open/closed issue and PR searches covered both magics, PEP 552, pyc/header, 3.7a1/a2, 12-byte, pre-PEP, alpha/pyc, and the exact error. The original inspector PR #4826 introduced this 16-byte path; PR #6147 concerns a separate PythonBytecode/PyInstaller path. Reports #6415, #6487, #6490, and #6547 concern hashability, code flags, first-line values, and line-table equality, with distinct triggers and restoration conditions. No matching report or correction was found. Restoration check: rerun the exact script. Both historical files must receive the typed unsupported-layout outcome or accurate historical-header evidence. The 3392/3394 controls must still return complete coverage, four bytecode bytes and a None constant; the 3389 control must remain unsupported. Rewriting the selected artifact into a synthetic 16-byte file would not restore this contract.

bsakaguchi closed this issue

2026-10-31 00:45:36 +00:00

bsakaguchi referenced this issue

2026-10-31 01:10:54 +00:00

[PR #3994] [CLOSED] fix: exclude credential header values from header allocation budget preflight #5980