[GH-ISSUE #7313] [Bug] Python 3.13 pyc inspection merges distinct code objects after STORE_SLICE #4347

New issue

Closed

opened 2026-10-31 00:49:05 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:05 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7313

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-python-pyc silently drops one of two unequal code objects from a CPython-produced frozenset under final Python 3.13 magic 3571. It exits 0 with confidence: "observed" and coverage.status: "complete", but reports frozenset length 1, two total code objects, and nested_code: [1]. Matching CPython reads the same body as two distinct members. Two consecutive isolated CLI calls reproduced this result.

Reproduction

Use the prepared source checkout and its supported Node.js runtime. Set REA_ROOT to that checkout's absolute path, then run this block from a separate writable directory with CPython 3.13. The failing file is entirely written by CPython's marshal serializer; its header uses that interpreter's genuine magic. The module retains its original None constant and return instruction. No generated bytecode is executed.

python3.13 -I -B - <<'PY'
import importlib.util, marshal, pathlib, sys
assert sys.version_info[:2] == (3, 13)
assert importlib.util.MAGIC_NUMBER == bytes.fromhex('f30d0d0a')
functions = [compile('def f(a, b):\n    a[1:2] = b\n    return ' + name + '\n',
                     'fixture.py', 'exec').co_consts[0] for name in ('a', 'b')]
a, b = functions
assert a != b and len(frozenset(functions)) == 2
base = compile('pass\n', 'fixture.py', 'exec')
assert base.co_consts == (None,)
root = base.replace(co_consts=(None, frozenset(functions)))
data = importlib.util.MAGIC_NUMBER + bytes(12) + marshal.dumps(root)
pathlib.Path('code-set.pyc').write_bytes(data)
loaded = marshal.loads(data[16:])
assert len(loaded.co_consts[1]) == 2
assert a in loaded.co_consts[1] and b in loaded.co_consts[1]
print('magic', int.from_bytes(data[:2], 'little'), 'members', len(loaded.co_consts[1]))
PY
node "$REA_ROOT/scripts/rea.mjs" inspect-python-pyc code-set.pyc --json

The Python check prints magic 3571 members 2. Inspect normalized_result.code_objects[0].consts[1], normalized_result.code_objects.length, and the root's nested_code in the CLI result.

Expected behavior

Report frozenset length 2, three total code objects including the module, and root nested_code: [1,2]. docs/python-pyc.md:94-110,130-134 explicitly promises every distinct reachable code object, CPython container equality, and inline-cache skipping for the exact magic. Its specialized/instrumented-opcode caveat does not apply to these unexecuted compiler-produced function instructions.

Impact and scope

The selected artifact remains unchanged, but the successful inspection omits a reachable code object and understates its container. The reproduced input uses an uncommon synthetic constant container created through CodeType.replace; this is not evidence that ordinary py_compile output loses code objects. Both functions themselves are ordinary compiler output. Other containers, MCP, other platforms and naturally occurring affected artifacts were not tested. A matching interpreter confirms the correct graph, but no equivalent REA workaround preserving the selected artifact and Evidence result was demonstrated.

REA version

CLI/package 6.8.0; source e891926534, using the existing compiled runtime without rebuilding. Independently fetched current-main parser/equality source and latest-release equality source retain the relevant behavior; those separate runtimes were not executed.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1 arm64; CPython 3.13.13 for fixture generation, equality checks and marshal round-trip. No external analysis provider.

Target and artifact context

A 353-byte timestamp-header Python 3.13 pyc with magic 3571 and flags 0. Root constants are None and a frozenset of two hashable function code objects. Their bytecode differs only at byte offset 11: the operand of LOAD_FAST at offset 10, immediately after STORE_SLICE at offset 8. All fixtures remain below default budgets.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed results from the actual CLI and matching interpreter:

CPython: a != b; marshal round-trip frozenset members = 2
final magic, two CLI calls: exit 0; stderr empty
  consts = [{"kind":"none"},{"kind":"frozenset","length":1}]
  code_objects = 2; nested_code = [1]; coverage = complete; confidence = observed
same final magic with tuple(functions): exit 0
  tuple length = 2; code_objects = 3; nested_code = [1,2]; coverage = complete
same marshal body with magic changed to 3570: exit 0
  frozenset length = 2; code_objects = 3; nested_code = [1,2]; coverage = complete

The tuple control retains both code objects without set deduplication. The altered-magic control is diagnostic only; it does not establish a valid final-format file or an equivalent workaround.

Expected source: The documented inspection contract above and matching CPython's code comparison and hash, which compare instruction operands and skip actual cache slots. Its opcode IDs and cache table identify STORE_SLICE as opcode 38 with zero cache units. The marshal writer explicitly serializes frozenset members; the failing fixture uses that writer without patching records.
Failure source: The repeated production CLI results above; src/python/PythonPycBytecodeEquality.ts:417-440,444-469 assigns opcode 38 one cache unit for final magic and skips the following instruction. src/python/PythonPycParser.ts:952,987-993,1254-1289 uses that identity for frozenset deduplication before reachable-code collection.
Evidence provenance: observed CLI output and matching-interpreter round-trip; the causal explanation is source-derived.
Local verification: reproduced.
Reproduction completeness: complete for the synthetic fixture shown.

Typed outcome or recovery expected

Successful complete inspection retaining both unequal members and all three reachable code objects. Rerunning the unchanged failing fixture should produce the expected counts while preserving original artifact and bytecode digests.

Analysis or suspected cause

Source-derived: the final-magic table skips bytes 10–11 as a nonexistent STORE_SLICE cache. The differing LOAD_FAST operand is therefore absent from the equality identity, although CPython compares it. The existing CLI acceptance case at tests/acceptance/pythonPycCli.test.ts:129-141 assumes opcode 81 has nine caches for this magic; it does not validate the released opcode mapping independently.

Additional context

Closed #6597 fixed the opposite failure for Python 3.12: cache-payload differences were counted as distinct members. Its closing commit introduced the table used here; the remaining final-version case loses an ordinary instruction operand. Closed #6593 fixed Unicode code-name collisions; these names are identical ASCII. Both complete discussions and closure commits were checked, and current issue/PR searches found no exact open duplicate. The withdrawn synthetic-record acceptance report #6677 does not supply this report's premise: this fixture uses a genuine matching marshal writer and the explicitly documented code-object set-member contract.

NOT_ENOUGH_INFO: naturally occurring affected artifacts and other-platform behavior were not established.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The public CLI repeatedly returns incomplete code-object facts as complete observed Evidence for a valid matching-producer fixture. The demonstrated scope is a synthetic constant-container input, with no established severe operational loss or ordinary-compiler prevalence.
Workaround status: unverified
Workaround basis: Matching CPython independently recovers both members, but no demonstrated REA option restores the original artifact's complete Evidence result. Replacing the container or header changes the selected artifact and is only a diagnostic control.

Originally created by @coygeek on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7313 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-python-pyc` silently drops one of two unequal code objects from a CPython-produced frozenset under final Python 3.13 magic 3571. It exits 0 with `confidence: "observed"` and `coverage.status: "complete"`, but reports frozenset length 1, two total code objects, and `nested_code: [1]`. Matching CPython reads the same body as two distinct members. Two consecutive isolated CLI calls reproduced this result. ## Reproduction Use the prepared source checkout and its supported Node.js runtime. Set `REA_ROOT` to that checkout's absolute path, then run this block from a separate writable directory with CPython 3.13. The failing file is entirely written by CPython's marshal serializer; its header uses that interpreter's genuine magic. The module retains its original `None` constant and return instruction. No generated bytecode is executed. ```sh python3.13 -I -B - <<'PY' import importlib.util, marshal, pathlib, sys assert sys.version_info[:2] == (3, 13) assert importlib.util.MAGIC_NUMBER == bytes.fromhex('f30d0d0a') functions = [compile('def f(a, b):\n a[1:2] = b\n return ' + name + '\n', 'fixture.py', 'exec').co_consts[0] for name in ('a', 'b')] a, b = functions assert a != b and len(frozenset(functions)) == 2 base = compile('pass\n', 'fixture.py', 'exec') assert base.co_consts == (None,) root = base.replace(co_consts=(None, frozenset(functions))) data = importlib.util.MAGIC_NUMBER + bytes(12) + marshal.dumps(root) pathlib.Path('code-set.pyc').write_bytes(data) loaded = marshal.loads(data[16:]) assert len(loaded.co_consts[1]) == 2 assert a in loaded.co_consts[1] and b in loaded.co_consts[1] print('magic', int.from_bytes(data[:2], 'little'), 'members', len(loaded.co_consts[1])) PY node "$REA_ROOT/scripts/rea.mjs" inspect-python-pyc code-set.pyc --json ``` The Python check prints `magic 3571 members 2`. Inspect `normalized_result.code_objects[0].consts[1]`, `normalized_result.code_objects.length`, and the root's `nested_code` in the CLI result. ## Expected behavior Report frozenset length 2, three total code objects including the module, and root `nested_code: [1,2]`. `docs/python-pyc.md:94-110,130-134` explicitly promises every distinct reachable code object, CPython container equality, and inline-cache skipping for the exact magic. Its specialized/instrumented-opcode caveat does not apply to these unexecuted compiler-produced function instructions. ## Impact and scope The selected artifact remains unchanged, but the successful inspection omits a reachable code object and understates its container. The reproduced input uses an uncommon synthetic constant container created through `CodeType.replace`; this is not evidence that ordinary `py_compile` output loses code objects. Both functions themselves are ordinary compiler output. Other containers, MCP, other platforms and naturally occurring affected artifacts were not tested. A matching interpreter confirms the correct graph, but no equivalent REA workaround preserving the selected artifact and Evidence result was demonstrated. ## REA version CLI/package 6.8.0; source e89192653402222c5b099216f0d83a2d1bceace0, using the existing compiled runtime without rebuilding. Independently fetched current-main parser/equality source and latest-release equality source retain the relevant behavior; those separate runtimes were not executed. ## Runtime and platform Node.js v24.22.1; macOS 27.0.1 arm64; CPython 3.13.13 for fixture generation, equality checks and marshal round-trip. No external analysis provider. ## Target and artifact context A 353-byte timestamp-header Python 3.13 pyc with magic 3571 and flags 0. Root constants are `None` and a frozenset of two hashable function code objects. Their bytecode differs only at byte offset 11: the operand of `LOAD_FAST` at offset 10, immediately after `STORE_SLICE` at offset 8. All fixtures remain below default budgets. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Observed results from the actual CLI and matching interpreter: ```text CPython: a != b; marshal round-trip frozenset members = 2 final magic, two CLI calls: exit 0; stderr empty consts = [{"kind":"none"},{"kind":"frozenset","length":1}] code_objects = 2; nested_code = [1]; coverage = complete; confidence = observed same final magic with tuple(functions): exit 0 tuple length = 2; code_objects = 3; nested_code = [1,2]; coverage = complete same marshal body with magic changed to 3570: exit 0 frozenset length = 2; code_objects = 3; nested_code = [1,2]; coverage = complete ``` The tuple control retains both code objects without set deduplication. The altered-magic control is diagnostic only; it does not establish a valid final-format file or an equivalent workaround. - Expected source: The documented inspection contract above and matching CPython's [code comparison and hash](https://github.com/python/cpython/blob/v3.13.13/Objects/codeobject.c#L1982), which compare instruction operands and skip actual cache slots. Its [opcode IDs](https://github.com/python/cpython/blob/v3.13.13/Include/opcode_ids.h#L49) and [cache table](https://github.com/python/cpython/blob/v3.13.13/Include/internal/pycore_opcode_metadata.h#L1598) identify STORE_SLICE as opcode 38 with zero cache units. The [marshal writer](https://github.com/python/cpython/blob/v3.13.13/Python/marshal.c#L518) explicitly serializes frozenset members; the failing fixture uses that writer without patching records. - Failure source: The repeated production CLI results above; `src/python/PythonPycBytecodeEquality.ts:417-440,444-469` assigns opcode 38 one cache unit for final magic and skips the following instruction. `src/python/PythonPycParser.ts:952,987-993,1254-1289` uses that identity for frozenset deduplication before reachable-code collection. - Evidence provenance: observed CLI output and matching-interpreter round-trip; the causal explanation is source-derived. - Local verification: reproduced. - Reproduction completeness: complete for the synthetic fixture shown. ## Typed outcome or recovery expected Successful complete inspection retaining both unequal members and all three reachable code objects. Rerunning the unchanged failing fixture should produce the expected counts while preserving original artifact and bytecode digests. ## Analysis or suspected cause Source-derived: the final-magic table skips bytes 10–11 as a nonexistent STORE_SLICE cache. The differing LOAD_FAST operand is therefore absent from the equality identity, although CPython compares it. The existing CLI acceptance case at `tests/acceptance/pythonPycCli.test.ts:129-141` assumes opcode 81 has nine caches for this magic; it does not validate the released opcode mapping independently. ## Additional context Closed [#6597](https://github.com/morluto/rea/issues/6597) fixed the opposite failure for Python 3.12: cache-payload differences were counted as distinct members. Its closing commit introduced the table used here; the remaining final-version case loses an ordinary instruction operand. Closed [#6593](https://github.com/morluto/rea/issues/6593) fixed Unicode code-name collisions; these names are identical ASCII. Both complete discussions and closure commits were checked, and current issue/PR searches found no exact open duplicate. The withdrawn synthetic-record acceptance report [#6677](https://github.com/morluto/rea/issues/6677) does not supply this report's premise: this fixture uses a genuine matching marshal writer and the explicitly documented code-object set-member contract. NOT_ENOUGH_INFO: naturally occurring affected artifacts and other-platform behavior were not established. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The public CLI repeatedly returns incomplete code-object facts as complete observed Evidence for a valid matching-producer fixture. The demonstrated scope is a synthetic constant-container input, with no established severe operational loss or ordinary-compiler prevalence. - Workaround status: unverified - Workaround basis: Matching CPython independently recovers both members, but no demonstrated REA option restores the original artifact's complete Evidence result. Replacing the container or header changes the selected artifact and is only a diagnostic control.

bsakaguchi closed this issue

2026-10-31 00:49:05 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:34 +00:00

[PR #4347] [CLOSED] fix(javascript): escape repeated prefixes in front of copied-slot selections #6192