morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #6593] [Bug] Pyc code-name equality merges distinct Unicode names and drops a code object #3979

New issue

Closed

opened 2026-10-31 00:45:45 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:45 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6593

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public inspect-python-pyc CLI reports a two-member CPython-produced frozenset as length 1 and omits one reachable code object while returning exit 0, confidence: "observed", and coverage.status: "complete".

The two code objects differ in co_name: one contains U+1F600, and the other contains the two code points U+D83D U+DE00. CPython 3.12.15 considers the names and code objects distinct and preserves both through a marshal round trip. REA merges their names into the same JavaScript string before comparing code-object keys. An ASCII-name control preserves both members.

Reproduction

From the source revision below with dependencies and runtime already built, put its pinned Node.js on PATH. Save this script as reproduce.py outside the checkout, then run python3.12 reproduce.py "$PWD". It creates valid pycs with the matching interpreter, reads them back without executing their code, and invokes the public CLI once per case.

import hashlib, importlib.util, json, marshal, pathlib, subprocess, sys, tempfile
repo = pathlib.Path(sys.argv[1]).resolve()
base = compile('pass', 'fixture.py', 'exec', dont_inherit=True)
with tempfile.TemporaryDirectory() as directory:
    for label, names in {
        'unicode-name': ('\U0001f600', '\ud83d\ude00'),
        'ascii-control': ('first', 'second'),
    }.items():
        codes = [base.replace(co_name=n, co_qualname=n) for n in names]
        root = base.replace(co_consts=(frozenset(codes),))
        data = importlib.util.MAGIC_NUMBER + bytes(12) + marshal.dumps(root)
        path = pathlib.Path(directory) / (label + '.pyc')
        path.write_bytes(data)
        loaded = marshal.loads(data[16:])
        result = subprocess.run(
            ['node', str(repo/'scripts/rea.mjs'), 'inspect-python-pyc', str(path), '--json'],
            capture_output=True, text=True, timeout=30)
        report = json.loads(result.stdout)['normalized_result']
        print(json.dumps({'case': label, 'exit': result.returncode,
            'cpython_member_count': len(loaded.co_consts[0]),
            'consts': report['code_objects'][0]['consts'],
            'code_objects': len(report['code_objects']),
            'coverage': report['coverage']['status']}))

The executed verifier used the same fixture construction and CLI calls, retaining fixtures and full output instead of deleting them. Each case was run once.

Expected behavior

Keep both distinct code objects and report frozenset length 2, three code objects including the root, and two nested relationships. The inspection contract states that set lengths follow CPython and every reachable distinct code object is listed. String display may need JavaScript encoding, but that must not become the identity used for deduplication.

Impact and scope

This is a bounded evidence-integrity defect for valid code names that become identical UTF-16 strings. It silently loses an entire reachable code object and understates a container count. The reproduced case uses a valid code-object constructor and actual CPython marshal serialization; ordinary compiler-produced identifiers and other platforms were not tested. No target code was executed. Dictionary keys and other code-name tables share relevant source logic but were not exercised.

REA version

Source 642f6571ea9d333ea50114c25e648dbb5b8a3c46, package 6.7.0, freshly built from locked dependencies. This establishes current-main behavior, not that the published npm package exposes this newer command.

Runtime and platform

Node.js v24.22.1; npm 11.16.0 for the parent-owned dependency installation/build; CPython 3.12.15, magic cb0d0d0a; Omarchy 4.0.6, Linux 7.2.8-9-omarchy x86_64, glibc 2.44. No external analysis provider.

Target and artifact context

280-byte unicode-name.pyc, SHA-256 e4b31775e91fc8a929e02e0b3b2481f802380d04c946a14bfe8d5927b66ac7de. Root constants contain a frozenset of two code objects derived from compile('pass', 'fixture.py', 'exec'). Only names/qualified names differ. The 275-byte ASCII control uses the same construction.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Actual retained verifier output:

unicode-name: CPython member count 2; name code points [0x1f600] and [0xd83d,0xde00]
REA exit 0; stderr empty; consts [{"kind":"frozenset","length":1}]; code_objects 2
coverage {"status":"complete","objects":35,"code_objects":2,"marshal_bytes":264,"marshal_depth":5}
ascii-control: CPython member count 2
REA exit 0; stderr empty; consts [{"kind":"frozenset","length":2}]; code_objects 3; complete

Observed runtime evidence comes from the matching CPython round trip and freshly built public CLI. Source-derived explanation: code-object keys use decoded names and name tables, while ordinary string keys deliberately retain raw marshal bytes. The shared service accepts the result as observed Evidence. No MCP execution was performed.

Typed outcome or recovery expected

A successful complete result that retains both distinct members and code objects. No error or recovery action is needed for this valid file.

Analysis or suspected cause

Source-derived: CPython surrogatepass encodes the scalar and the surrogate pair differently. decodeSurrogatePass correctly permits both, but both become the same JavaScript UTF-16 string. #codeKey uses node.name rather than the raw string identity used by #stringKey; the resulting equality token is shared, and #unique discards one member before reachable-code collection.

Additional context

Related but distinct: closed #6547 concerns Python 3.8–3.10 line-table equality; this report concerns Unicode-name collisions with matching CPython 3.12 execution. #6476 concerns truncating an astral constant preview and invalid JSON. This input is below all preview limits, and the observed failure loses a code object through equality, rather than truncating a preview. No last-known-good revision was established. Open/full-body and closed-issue searches found no exact related report; the final filing check also found no duplicate.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed public CLI silently drops one reachable code object and reports an incorrect frozenset length as complete observed evidence for a valid matching-interpreter fixture.
Workaround status: unverified
Workaround basis: No REA option preserving both original code names and the complete object graph was demonstrated; independently inspecting the artifact in CPython does not restore REA's result.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6593 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The public `inspect-python-pyc` CLI reports a two-member CPython-produced frozenset as length 1 and omits one reachable code object while returning exit 0, `confidence: "observed"`, and `coverage.status: "complete"`. The two code objects differ in `co_name`: one contains U+1F600, and the other contains the two code points U+D83D U+DE00. CPython 3.12.15 considers the names and code objects distinct and preserves both through a marshal round trip. REA merges their names into the same JavaScript string before comparing code-object keys. An ASCII-name control preserves both members. ### Reproduction From the source revision below with dependencies and runtime already built, put its pinned Node.js on PATH. Save this script as `reproduce.py` outside the checkout, then run `python3.12 reproduce.py "$PWD"`. It creates valid pycs with the matching interpreter, reads them back without executing their code, and invokes the public CLI once per case. ```python import hashlib, importlib.util, json, marshal, pathlib, subprocess, sys, tempfile repo = pathlib.Path(sys.argv[1]).resolve() base = compile('pass', 'fixture.py', 'exec', dont_inherit=True) with tempfile.TemporaryDirectory() as directory: for label, names in { 'unicode-name': ('\U0001f600', '\ud83d\ude00'), 'ascii-control': ('first', 'second'), }.items(): codes = [base.replace(co_name=n, co_qualname=n) for n in names] root = base.replace(co_consts=(frozenset(codes),)) data = importlib.util.MAGIC_NUMBER + bytes(12) + marshal.dumps(root) path = pathlib.Path(directory) / (label + '.pyc') path.write_bytes(data) loaded = marshal.loads(data[16:]) result = subprocess.run( ['node', str(repo/'scripts/rea.mjs'), 'inspect-python-pyc', str(path), '--json'], capture_output=True, text=True, timeout=30) report = json.loads(result.stdout)['normalized_result'] print(json.dumps({'case': label, 'exit': result.returncode, 'cpython_member_count': len(loaded.co_consts[0]), 'consts': report['code_objects'][0]['consts'], 'code_objects': len(report['code_objects']), 'coverage': report['coverage']['status']})) ``` The executed verifier used the same fixture construction and CLI calls, retaining fixtures and full output instead of deleting them. Each case was run once. ### Expected behavior Keep both distinct code objects and report frozenset length 2, three code objects including the root, and two nested relationships. The [inspection contract](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/docs/python-pyc.md) states that set lengths follow CPython and every reachable distinct code object is listed. String display may need JavaScript encoding, but that must not become the identity used for deduplication. ### Impact and scope This is a bounded evidence-integrity defect for valid code names that become identical UTF-16 strings. It silently loses an entire reachable code object and understates a container count. The reproduced case uses a valid code-object constructor and actual CPython marshal serialization; ordinary compiler-produced identifiers and other platforms were not tested. No target code was executed. Dictionary keys and other code-name tables share relevant source logic but were not exercised. ### REA version Source `642f6571ea9d333ea50114c25e648dbb5b8a3c46`, package 6.7.0, freshly built from locked dependencies. This establishes current-main behavior, not that the published npm package exposes this newer command. ### Runtime and platform Node.js v24.22.1; npm 11.16.0 for the parent-owned dependency installation/build; CPython 3.12.15, magic `cb0d0d0a`; Omarchy 4.0.6, Linux 7.2.8-9-omarchy x86_64, glibc 2.44. No external analysis provider. ### Target and artifact context 280-byte `unicode-name.pyc`, SHA-256 `e4b31775e91fc8a929e02e0b3b2481f802380d04c946a14bfe8d5927b66ac7de`. Root constants contain a frozenset of two code objects derived from `compile('pass', 'fixture.py', 'exec')`. Only names/qualified names differ. The 275-byte ASCII control uses the same construction. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Actual retained verifier output: ```text unicode-name: CPython member count 2; name code points [0x1f600] and [0xd83d,0xde00] REA exit 0; stderr empty; consts [{"kind":"frozenset","length":1}]; code_objects 2 coverage {"status":"complete","objects":35,"code_objects":2,"marshal_bytes":264,"marshal_depth":5} ascii-control: CPython member count 2 REA exit 0; stderr empty; consts [{"kind":"frozenset","length":2}]; code_objects 3; complete ``` Observed runtime evidence comes from the matching CPython round trip and freshly built public CLI. Source-derived explanation: [code-object keys use decoded names and name tables](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L1140), while [ordinary string keys deliberately retain raw marshal bytes](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/python/PythonPycParser.ts#L1215). The [shared service](https://github.com/morluto/rea/blob/642f6571ea9d333ea50114c25e648dbb5b8a3c46/src/application/python/PythonPycService.ts#L68) accepts the result as observed Evidence. No MCP execution was performed. ### Typed outcome or recovery expected A successful complete result that retains both distinct members and code objects. No error or recovery action is needed for this valid file. ### Analysis or suspected cause Source-derived: CPython surrogatepass encodes the scalar and the surrogate pair differently. `decodeSurrogatePass` correctly permits both, but both become the same JavaScript UTF-16 string. `#codeKey` uses `node.name` rather than the raw string identity used by `#stringKey`; the resulting equality token is shared, and `#unique` discards one member before reachable-code collection. ### Additional context Related but distinct: closed #6547 concerns Python 3.8–3.10 line-table equality; this report concerns Unicode-name collisions with matching CPython 3.12 execution. #6476 concerns truncating an astral constant preview and invalid JSON. This input is below all preview limits, and the observed failure loses a code object through equality, rather than truncating a preview. No last-known-good revision was established. Open/full-body and closed-issue searches found no exact related report; the final filing check also found no duplicate. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed public CLI silently drops one reachable code object and reports an incorrect frozenset length as complete observed evidence for a valid matching-interpreter fixture. - Workaround status: unverified - Workaround basis: No REA option preserving both original code names and the complete object graph was demonstrated; independently inspecting the artifact in CPython does not restore REA's result.

bsakaguchi closed this issue

2026-10-31 00:45:45 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3979

No description provided.

Rows
Columns