[GH-ISSUE #6597] [Bug] Pyc code equality counts inline-cache differences as distinct frozenset members #3983

New issue

Closed

opened 2026-10-31 00:45:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:47 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6597

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-python-pyc reports two frozenset members and three reachable code objects for a valid CPython 3.12 marshal stream whose frozenset contains two code records differing only in one inline-cache byte. Matching CPython 3.12.15 loads the unchanged body, considers those code objects equal with equal hashes, and retains one frozenset member. REA returns exit 0, confidence: "observed", and complete coverage while overcounting members and reporting a code object CPython discards.

The same fixture with identical cache bytes produces one member in both implementations. The records' ordinary opcodes and all other code fields are identical. The fixture is synthetic; this is not a claim that the normal compiler writes nonzero cache bytes.

Reproduction

From the source revision below with dependencies and runtime already built, put its pinned Node.js on PATH. Save as reproduce.py outside the checkout and run python3.12 reproduce.py "$PWD". CPython only compiles and unmarshals these fixtures; target code never executes.

import dis, importlib.util, json, marshal, pathlib, struct, subprocess, sys, tempfile
repo = pathlib.Path(sys.argv[1]).resolve()
u32 = lambda n: struct.pack('<I', n)
text = lambda s: b'z' + bytes([len(s)]) + s.encode('ascii')
blob = lambda b: b's' + u32(len(b)) + b
def outer(constant):
    return (b'c' + bytes(12) + u32(1) + bytes(4) + blob(bytes.fromhex('970064005300')) +
        b')\1' + constant + b')\0' + b')\0' + blob(b'') + text('outer.py') +
        text('<module>') * 2 + u32(1) + blob(b'') + blob(b''))
base = compile('obj.attr', 'fixture.py', 'eval', dont_inherit=True)
original = marshal.dumps(base, 2)
needle = blob(base.co_code)
assert original.count(needle) == 1
position = original.index(needle) + 5
cache = next(i.offset for i in dis.get_instructions(base, show_caches=True)
             if i.opname == 'CACHE')
with tempfile.TemporaryDirectory() as directory:
    for label, cache_byte in [('cache-control', 0), ('changed-cache', 1)]:
        changed = bytearray(original)
        changed[position + cache] = cache_byte
        changed = bytes(changed)
        a, b = marshal.loads(original), marshal.loads(changed)
        body = outer(b'>' + u32(2) + original + changed)
        path = pathlib.Path(directory) / (label + '.pyc')
        path.write_bytes(importlib.util.MAGIC_NUMBER + bytes(12) + body)
        loaded = marshal.loads(body)
        result = subprocess.run(
            ['node', str(repo/'scripts/rea.mjs'), 'inspect-python-pyc', str(path), '--json'],
            capture_output=True, text=True, timeout=30)
        report = json.loads(result.stdout)['normalized_result']
        print(json.dumps({'case': label, 'codes_equal': a == b,
            'hashes_equal': hash(a) == hash(b),
            'cpython_member_count': len(loaded.co_consts[0]),
            'exit': result.returncode, 'consts': report['code_objects'][0]['consts'],
            'code_objects': len(report['code_objects']),
            'coverage': report['coverage']['status']}))

The executed verifier used this fixture construction and the same public CLI calls, retaining artifacts and full JSON output. Each case was run once.

Expected behavior

Keep the original artifact/bytecode digests as exact byte identities, but use CPython code-object semantics when deciding set membership: inline-cache payload bytes do not distinguish code objects. Report one frozenset member and two reachable code objects, as the matching interpreter does.

The REA inspection contract states that set lengths follow CPython and that the result describes the decoded graph rather than every serialized code record. CPython 3.12.15 skips inline-cache slots in equality and hashing.

Impact and scope

Bounded, consequential evidence mismatch for admitted 3.12 code records with differing cache payloads: REA overstates a container and reports a code object absent from the actual decoded graph. Both inputs remain below every default resource limit. Normal compiler output prevalence, other CPython versions, dictionary behavior, and MCP were not tested. No target execution, crash, or mutation of source occurred.

REA version

Source 642f6571ea9d333ea50114c25e648dbb5b8a3c46, package 6.7.0, freshly built from locked dependencies. This establishes current-main behavior, not availability of this newer command in the published npm package.

Runtime and platform

Node.js v24.22.1; npm 11.16.0 for the dependency installation/build; CPython 3.12.15; Omarchy 4.0.6, Linux 7.2.8-9-omarchy x86_64, glibc 2.44. No external analysis provider.

Target and artifact context

Both files are 420-byte CPython 3.12 pycs, magic cb0d0d0a, timestamp headers, containing a module with a frozenset of two code records compiled from obj.attr. Marshal version 2 avoids reference-slot relocation when composing the synthetic wrapper. changed-cache.pyc changes bytecode offset 6, the first CACHE slot, from 0 to 1 in the second record only. Its SHA-256 is c832a009e4555f23256d1928486123132646b71d5be2d81dc905d715d1ecd2ec; the control is 8adc928dd8e980d0e832dbf5c06e96fe043ac4601b699fb6ab38753958c76c09.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed verifier results:

bytecode 970065006a020000000000000000000000000000000000005300; cache_offset 6
cache-control: CPython codes_equal true, hashes_equal true, member_count 1
REA exit 0, stderr empty, consts [{"kind":"frozenset","length":1}], code_objects 2, complete
changed-cache: CPython codes_equal true, hashes_equal true, member_count 1
REA exit 0, stderr empty, consts [{"kind":"frozenset","length":2}], code_objects 3
coverage {"status":"complete","objects":38,"code_objects":3,"marshal_bytes":404,"marshal_depth":5}

The runtime observation is a freshly built public CLI plus unchanged-body matching-interpreter decoding. Source-derived explanation: the reader records SHA-256 of every original bytecode byte, then reuses that exact digest in code-object equality. Set deduplication therefore keeps both before reachable-code collection. The shared service accepts this as observed Evidence.

Typed outcome or recovery expected

Successful complete inspection with one member and two reachable code objects, while retaining the exact original artifact digest. This is a valid-input interpretation defect, not a request-validation failure.

Analysis or suspected cause

Source-derived and consistent with execution: exact stored-bytecode identity and CPython semantic code identity are assigned the same hash in #codeKey. A byte difference inside a CACHE payload changes that hash even though CPython skips the cache slot. A separate comparison representation is needed without changing the exact bytecode evidence.

Additional context

Related closed #6547 concerns 3.8–3.10 line-table equality; it does not cover 3.11+ cache-aware instruction comparison. Open #6415 concerns unhashable constants and rejection rather than equal, hashable, admitted code records. Searches for pyc inline cache and marshal cache bytes found no issue. A fresh pre-submission search also found no duplicate. No regression revision was established.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Matching CPython decoding and the public CLI disagree on a valid fixture's set length and reachable code graph while REA reports complete observed evidence.
Workaround status: unverified
Workaround basis: No REA option that selects CPython cache-aware equality was demonstrated. Rewriting the fixture would change the selected artifact and does not preserve the inspection input.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6597 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `inspect-python-pyc` reports two frozenset members and three reachable code objects for a valid CPython 3.12 marshal stream whose frozenset contains two code records differing only in one inline-cache byte. Matching CPython 3.12.15 loads the unchanged body, considers those code objects equal with equal hashes, and retains one frozenset member. REA returns exit 0, `confidence: "observed"`, and complete coverage while overcounting members and reporting a code object CPython discards. The same fixture with identical cache bytes produces one member in both implementations. The records' ordinary opcodes and all other code fields are identical. The fixture is synthetic; this is not a claim that the normal compiler writes nonzero cache bytes. ### Reproduction From the source revision below with dependencies and runtime already built, put its pinned Node.js on PATH. Save as `reproduce.py` outside the checkout and run `python3.12 reproduce.py "$PWD"`. CPython only compiles and unmarshals these fixtures; target code never executes. ```python import dis, importlib.util, json, marshal, pathlib, struct, subprocess, sys, tempfile repo = pathlib.Path(sys.argv[1]).resolve() u32 = lambda n: struct.pack('<I', n) text = lambda s: b'z' + bytes([len(s)]) + s.encode('ascii') blob = lambda b: b's' + u32(len(b)) + b def outer(constant): return (b'c' + bytes(12) + u32(1) + bytes(4) + blob(bytes.fromhex('970064005300')) + b')\1' + constant + b')\0' + b')\0' + blob(b'') + text('outer.py') + text('<module>') * 2 + u32(1) + blob(b'') + blob(b'')) base = compile('obj.attr', 'fixture.py', 'eval', dont_inherit=True) original = marshal.dumps(base, 2) needle = blob(base.co_code) assert original.count(needle) == 1 position = original.index(needle) + 5 cache = next(i.offset for i in dis.get_instructions(base, show_caches=True) if i.opname == 'CACHE') with tempfile.TemporaryDirectory() as directory: for label, cache_byte in [('cache-control', 0), ('changed-cache', 1)]: changed = bytearray(original) changed[position + cache] = cache_byte changed = bytes(changed) a, b = marshal.loads(original), marshal.loads(changed) body = outer(b'>' + u32(2) + original + changed) path = pathlib.Path(directory) / (label + '.pyc') path.write_bytes(importlib.util.MAGIC_NUMBER + bytes(12) + body) loaded = marshal.loads(body) result = subprocess.run( ['node', str(repo/'scripts/rea.mjs'), 'inspect-python-pyc', str(path), '--json'], capture_output=True, text=True, timeout=30) report = json.loads(result.stdout)['normalized_result'] print(json.dumps({'case': label, 'codes_equal': a == b, 'hashes_equal': hash(a) == hash(b), 'cpython_member_count': len(loaded.co_consts[0]), 'exit': result.returncode, 'consts': report['code_objects'][0]['consts'], 'code_objects': len(report['code_objects']), 'coverage': report['coverage']['status']})) ``` The executed verifier used this fixture construction and the same public CLI calls, retaining artifacts and full JSON output. Each case was run once. ### Expected behavior Keep the original artifact/bytecode digests as exact byte identities, but use CPython code-object semantics when deciding set membership: inline-cache payload bytes do not distinguish code objects. Report one frozenset member and two reachable code objects, as the matching interpreter does. The [REA inspection contract](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/docs/python-pyc.md#L51) states that set lengths follow CPython and that the result describes the decoded graph rather than every serialized code record. CPython 3.12.15 [skips inline-cache slots in equality](https://github.com/python/cpython/blob/v3.12.15/Objects/codeobject.c#L1706) and [hashing](https://github.com/python/cpython/blob/v3.12.15/Objects/codeobject.c#L1788). ### Impact and scope Bounded, consequential evidence mismatch for admitted 3.12 code records with differing cache payloads: REA overstates a container and reports a code object absent from the actual decoded graph. Both inputs remain below every default resource limit. Normal compiler output prevalence, other CPython versions, dictionary behavior, and MCP were not tested. No target execution, crash, or mutation of source occurred. ### REA version Source `642f6571ea9d333ea50114c25e648dbb5b8a3c46`, package 6.7.0, freshly built from locked dependencies. This establishes current-main behavior, not availability of this newer command in the published npm package. ### Runtime and platform Node.js v24.22.1; npm 11.16.0 for the dependency installation/build; CPython 3.12.15; Omarchy 4.0.6, Linux 7.2.8-9-omarchy x86_64, glibc 2.44. No external analysis provider. ### Target and artifact context Both files are 420-byte CPython 3.12 pycs, magic `cb0d0d0a`, timestamp headers, containing a module with a frozenset of two code records compiled from `obj.attr`. Marshal version 2 avoids reference-slot relocation when composing the synthetic wrapper. `changed-cache.pyc` changes bytecode offset 6, the first CACHE slot, from 0 to 1 in the second record only. Its SHA-256 is `c832a009e4555f23256d1928486123132646b71d5be2d81dc905d715d1ecd2ec`; the control is `8adc928dd8e980d0e832dbf5c06e96fe043ac4601b699fb6ab38753958c76c09`. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Observed verifier results: ```text bytecode 970065006a020000000000000000000000000000000000005300; cache_offset 6 cache-control: CPython codes_equal true, hashes_equal true, member_count 1 REA exit 0, stderr empty, consts [{"kind":"frozenset","length":1}], code_objects 2, complete changed-cache: CPython codes_equal true, hashes_equal true, member_count 1 REA exit 0, stderr empty, consts [{"kind":"frozenset","length":2}], code_objects 3 coverage {"status":"complete","objects":38,"code_objects":3,"marshal_bytes":404,"marshal_depth":5} ``` The runtime observation is a freshly built public CLI plus unchanged-body matching-interpreter decoding. Source-derived explanation: [the reader records SHA-256 of every original bytecode byte](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L504), then [reuses that exact digest in code-object equality](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L1158). [Set deduplication](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L874) therefore keeps both before [reachable-code collection](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L522). The [shared service](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/application/python/PythonPycService.ts#L68) accepts this as observed Evidence. ### Typed outcome or recovery expected Successful complete inspection with one member and two reachable code objects, while retaining the exact original artifact digest. This is a valid-input interpretation defect, not a request-validation failure. ### Analysis or suspected cause Source-derived and consistent with execution: exact stored-bytecode identity and CPython semantic code identity are assigned the same hash in `#codeKey`. A byte difference inside a CACHE payload changes that hash even though CPython skips the cache slot. A separate comparison representation is needed without changing the exact bytecode evidence. ### Additional context Related closed #6547 concerns 3.8–3.10 line-table equality; it does not cover 3.11+ cache-aware instruction comparison. Open #6415 concerns unhashable constants and rejection rather than equal, hashable, admitted code records. Searches for `pyc inline cache` and `marshal cache bytes` found no issue. A fresh pre-submission search also found no duplicate. No regression revision was established. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Matching CPython decoding and the public CLI disagree on a valid fixture's set length and reachable code graph while REA reports complete observed evidence. - Workaround status: unverified - Workaround basis: No REA option that selects CPython cache-aware equality was demonstrated. Rewriting the fixture would change the selected artifact and does not preserve the inspection input.

bsakaguchi closed this issue

2026-10-31 00:45:47 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:11 +00:00

[PR #4238] [MERGED] fix(lifecycle): preserve ownership, cancellation, and cleanup outcomes #6141

bsakaguchi referenced this issue

2026-10-31 01:12:16 +00:00

[PR #4274] [MERGED] fix(lifecycle): preserve observations and cleanup ownership #6153