morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6476] [Bug] inspect-python-pyc splits a valid Unicode scalar in a constant preview #3884

New issue

Closed

opened 2026-10-31 00:44:55 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:55 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6476

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A compiler-produced CPython 3.13 file containing a followed by 40 U+1F600 returns success and complete coverage, but its string preview ends with an unpaired U+D83D after 80 UTF-16 units. jq rejects the JSON result with exit 5, while TOON contains one introduced U+FFFD. The same incomplete preview is present in the real stdio MCP structured result. The shorter-astral, exact-boundary-astral and long-ASCII controls succeed without an unpaired surrogate.

Reproduction

Run from the repository root with its prepared compiled runtime, a supported Node.js runtime, CPython 3.13, and jq on PATH. CPython only compiles the inert assignment and reads its code object; neither CPython nor REA executes the generated bytecode. The block creates and removes a fresh temporary root, gives REA a minimal environment with isolated home/config/cache/data/state/temp directories, and writes no fixture or state into the checkout. This exact block was replayed successfully. Dependency installation and runtime building are prerequisites for an unprepared checkout and were not performed during verification.

python3.13 -I -S - <<'PYTHON'
import hashlib, importlib.util, json, marshal, os, pathlib, py_compile, subprocess, sys, tempfile
assert sys.version_info[:2] == (3, 13)
repo = pathlib.Path.cwd()
assert (repo / 'scripts/rea.mjs').is_file()
values = {'split': 'a' + '\U0001f600' * 40,
          'astral-short': 'a' + '\U0001f600' * 39,
          'astral-boundary': '\U0001f600' * 40,
          'ascii-long': 'z' * 81}
with tempfile.TemporaryDirectory(prefix='pyc-preview-') as directory:
    root = pathlib.Path(directory).resolve()
    env = {'PATH': os.environ['PATH'], 'REA_LOG_LEVEL': 'silent',
           'NODE_DISABLE_COMPILE_CACHE': '1', 'NO_COLOR': '1'}
    for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME',
                'XDG_DATA_HOME', 'XDG_STATE_HOME', 'TMPDIR', 'DSH_HOME'):
        location = root / key.lower(); location.mkdir(); env[key] = str(location)
    def run(path, *flags):
        r = subprocess.run(['node', str(repo / 'scripts/rea.mjs'),
                            'inspect-python-pyc', str(path), *flags],
                           cwd=root, env=env, capture_output=True, timeout=30)
        assert r.returncode == 0 and not r.stderr, (r.returncode, r.stderr)
        return r.stdout
    for name, value in values.items():
        source, artifact = root / (name + '.py'), root / (name + '.pyc')
        source.write_text('X = ' + repr(value) + '\n', encoding='utf8')
        py_compile.compile(str(source), cfile=str(artifact), dfile=name + '.py',
                           doraise=True, invalidation_mode=py_compile.PycInvalidationMode.CHECKED_HASH)
        raw = artifact.read_bytes(); code = marshal.loads(raw[16:])
        assert raw[:4] == importlib.util.MAGIC_NUMBER
        assert code.co_consts == (value, None)
        assert not any(0xd800 <= ord(c) <= 0xdfff for c in value)
        output = run(artifact, '--json'); report = json.loads(output)['normalized_result']
        assert report['coverage']['status'] == 'complete'
        assert report['artifact']['sha256'] == hashlib.sha256(raw).hexdigest()
        assert report['artifact']['bytes'] == len(raw)
        assert report['code_objects'][0]['bytecode']['sha256'] == hashlib.sha256(code.co_code).hexdigest()
        assert report['code_objects'][0]['bytecode']['bytes'] == len(code.co_code)
        const = report['code_objects'][0]['consts'][0]; s = const['value']
        jq = subprocess.run(['jq', '-e', '.'], input=output, env=env, capture_output=True)
        print(json.dumps({'case': name, 'cli_exit': 0, 'bytes': const['bytes'],
            'truncated': const['truncated'], 'utf16_units': len(s.encode('utf-16-le', 'surrogatepass')) // 2,
            'unpaired_surrogate': any(0xd800 <= ord(c) <= 0xdfff for c in s),
            'jq_exit': jq.returncode}))
        if jq.returncode:
            print(jq.stderr.decode().strip().split(' at line ', 1)[0])
        if name == 'split':
            print('TOON replacement characters:', run(artifact, '--format', 'toon').count('\ufffd'.encode()))
            # A derived display only; keep the original Evidence unchanged.
            display = {'value': s[:-1], 'truncated': const['truncated'], 'bytes': const['bytes']}
            assert display['value'] == 'a' + '\U0001f600' * 39
            recovered = subprocess.run(['jq', '-e', '.'], input=json.dumps(display).encode(),
                                       env=env, capture_output=True)
            assert recovered.returncode == 0
            print('Derived whole-scalar display jq exit:', recovered.returncode)
PYTHON

Expected behavior

For these compiler-produced constants containing only Unicode scalar values, return a bounded preview that ends between complete scalars, with the existing original marshal byte count and an accurate truncation flag. A successful JSON result should work with the documented jq consumer, and TOON should not acquire a replacement character from the preview cut. This does not require a universal code-point-counted limit: an 80-UTF-16-unit budget can stop before the final complete scalar. Preserving source-encoded lone surrogates under CPython surrogatepass is a separate existing behavior, outside this report.

Impact and scope

P2: a bounded constant-preview defect blocks the documented jq workflow for the successful result and corrupts the displayed final character. The original artifact is unchanged. Artifact byte count/SHA-256, bytecode byte count/SHA-256, original constant marshal byte count, and complete coverage matched independent checks in all five fixture cases. Only synthetic fixtures were measured; natural frequency and unrelated consumers/platforms are unknown. No security boundary was crossed.

A verified workaround for displaying or processing this preview is to parse the original output with a parser accepting lone-surrogate escapes, then make a separate display value that omits the final incomplete high surrogate while retaining truncated: true and bytes: 161. The inline block demonstrates a 40-scalar prefix accepted by jq. Retain the original Evidence unchanged; the derived display is not replacement authenticated Evidence and does not make jq accept the original document.

REA version

Unreleased main e32a0494d4, package/CLI version 6.7.0, using its prepared compiled runtime. The latest published release tag, rea-agents-6.7.0, lacks this pyc command/parser; no stable-release runtime reproduction is claimed.

Runtime and platform

Node.js v24.18.0; npm 11.16.0; CPython 3.13.13 for compilation and code-object inspection only; jq 1.8.2; macOS 27.0.1 arm64. No install, build, native engine, or target-bytecode execution ran.

Target and artifact context

Four small hash-based CPython 3.13 pycs compiled from inert constant assignments, with runtime magic 3571 and exact stored constants verified by marshal inspection. Input strings contain no lone surrogates. A fifth case with a trailing ASCII character reproduced the same cut. Fixtures and child state lived under disposable temporary roots.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

{"case": "split", "cli_exit": 0, "bytes": 161, "truncated": true, "utf16_units": 80, "unpaired_surrogate": true, "jq_exit": 5}
jq: parse error: Invalid \uXXXX\uXXXX surrogate pair escape
TOON replacement characters: 1
Derived whole-scalar display jq exit: 0
{"case": "astral-short", "cli_exit": 0, "bytes": 157, "truncated": false, "utf16_units": 79, "unpaired_surrogate": false, "jq_exit": 0}
{"case": "astral-boundary", "cli_exit": 0, "bytes": 160, "truncated": false, "utf16_units": 80, "unpaired_surrogate": false, "jq_exit": 0}
{"case": "ascii-long", "cli_exit": 0, "bytes": 81, "truncated": true, "utf16_units": 80, "unpaired_surrogate": false, "jq_exit": 0}

The four primary JSON calls reproduced on three CLI passes and one MCP pass; normalized MCP results matched CLI exactly. Three split-case TOON calls each contained one replacement character. Node JSON.parse also accepted the affected JSON: its string was 80 UTF-16 units, ended in 0xd83d, and failed isWellFormed(). An escaped lone surrogate is not necessarily invalid JSON grammar; the observed incompatibility is with jq and the preview text itself.

Contract breadcrumbs: docs/python-pyc.md:3-14,44-51 documents the read-only pyc operation and truncated const summaries; docs/cli.md:543-548 documents JSON consumers and jq. The shared preview is produced at src/python/PythonPycParser.ts:1559-1568, and src/domain/python/pythonPyc.ts:126-131 validates length without detecting a split pair. Existing parser/service tests passed, 42 tests in two files with one worker.

Typed outcome or recovery expected

Return successful Evidence with a bounded, whole-scalar source-prefix preview for these scalar-only inputs. Preserve the original constant byte count, artifact/bytecode identity and applicable truncation flag. Either a shorter whole-scalar prefix within the current UTF-16 budget or another compatible bounded policy can satisfy this; a universal code-point-counted limit is not required.

Analysis or suspected cause

Source-derived and consistent with execution: the preview uses JavaScript string length and slice, so its cut can separate a valid surrogate pair. The shared service accepts the length-bounded result, which explains why CLI and MCP agree. This is a shared preview-generation issue, not a general JSON-serializer failure. No repair was applied.

Additional context

Live open-and-closed searches and the supplied issue-title snapshot found no matching report. Related independently closable cases are #6437 (negative-zero float projection), #6438 (decode-limit recovery), and #6415 (code-object hashability). #5530 concerns CLI token windows; this case occurs without either token flag and is already present in MCP structuredContent. #5097 concerns Frida JSON-byte accounting, and #3177 concerns Apple path ordering. These reports have different contracts and closure signals from this shared pyc preview.

Proposed restoration check: Rerun the same compiler-produced fixtures through the CLI and inspect_python_pyc MCP tool. The split case must return a source-prefix preview containing only whole scalars, accurate truncation metadata, unchanged original byte counts/digests, JSON accepted by jq, and TOON with no introduced U+FFFD. Keep the shorter astral and exact-boundary astral controls untruncated and intact, and the 81-ASCII control truncated to 80 ASCII characters. A related executed case with one extra trailing ASCII character must also avoid splitting the scalar. These are restoration criteria; no source fix was made or tested.

NOT_ENOUGH_INFO: natural-workload frequency, first affected source commit, last known good behavior, and other-platform behavior were not established. Existing support for source-encoded lone surrogates is outside this scalar-only input case.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6476 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior A compiler-produced CPython 3.13 file containing `a` followed by 40 U+1F600 returns success and complete coverage, but its string preview ends with an unpaired U+D83D after 80 UTF-16 units. jq rejects the JSON result with exit 5, while TOON contains one introduced U+FFFD. The same incomplete preview is present in the real stdio MCP structured result. The shorter-astral, exact-boundary-astral and long-ASCII controls succeed without an unpaired surrogate. ## Reproduction Run from the repository root with its prepared compiled runtime, a supported Node.js runtime, CPython 3.13, and jq on PATH. CPython only compiles the inert assignment and reads its code object; neither CPython nor REA executes the generated bytecode. The block creates and removes a fresh temporary root, gives REA a minimal environment with isolated home/config/cache/data/state/temp directories, and writes no fixture or state into the checkout. This exact block was replayed successfully. Dependency installation and runtime building are prerequisites for an unprepared checkout and were not performed during verification. ```sh python3.13 -I -S - <<'PYTHON' import hashlib, importlib.util, json, marshal, os, pathlib, py_compile, subprocess, sys, tempfile assert sys.version_info[:2] == (3, 13) repo = pathlib.Path.cwd() assert (repo / 'scripts/rea.mjs').is_file() values = {'split': 'a' + '\U0001f600' * 40, 'astral-short': 'a' + '\U0001f600' * 39, 'astral-boundary': '\U0001f600' * 40, 'ascii-long': 'z' * 81} with tempfile.TemporaryDirectory(prefix='pyc-preview-') as directory: root = pathlib.Path(directory).resolve() env = {'PATH': os.environ['PATH'], 'REA_LOG_LEVEL': 'silent', 'NODE_DISABLE_COMPILE_CACHE': '1', 'NO_COLOR': '1'} for key in ('HOME', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME', 'XDG_DATA_HOME', 'XDG_STATE_HOME', 'TMPDIR', 'DSH_HOME'): location = root / key.lower(); location.mkdir(); env[key] = str(location) def run(path, *flags): r = subprocess.run(['node', str(repo / 'scripts/rea.mjs'), 'inspect-python-pyc', str(path), *flags], cwd=root, env=env, capture_output=True, timeout=30) assert r.returncode == 0 and not r.stderr, (r.returncode, r.stderr) return r.stdout for name, value in values.items(): source, artifact = root / (name + '.py'), root / (name + '.pyc') source.write_text('X = ' + repr(value) + '\n', encoding='utf8') py_compile.compile(str(source), cfile=str(artifact), dfile=name + '.py', doraise=True, invalidation_mode=py_compile.PycInvalidationMode.CHECKED_HASH) raw = artifact.read_bytes(); code = marshal.loads(raw[16:]) assert raw[:4] == importlib.util.MAGIC_NUMBER assert code.co_consts == (value, None) assert not any(0xd800 <= ord(c) <= 0xdfff for c in value) output = run(artifact, '--json'); report = json.loads(output)['normalized_result'] assert report['coverage']['status'] == 'complete' assert report['artifact']['sha256'] == hashlib.sha256(raw).hexdigest() assert report['artifact']['bytes'] == len(raw) assert report['code_objects'][0]['bytecode']['sha256'] == hashlib.sha256(code.co_code).hexdigest() assert report['code_objects'][0]['bytecode']['bytes'] == len(code.co_code) const = report['code_objects'][0]['consts'][0]; s = const['value'] jq = subprocess.run(['jq', '-e', '.'], input=output, env=env, capture_output=True) print(json.dumps({'case': name, 'cli_exit': 0, 'bytes': const['bytes'], 'truncated': const['truncated'], 'utf16_units': len(s.encode('utf-16-le', 'surrogatepass')) // 2, 'unpaired_surrogate': any(0xd800 <= ord(c) <= 0xdfff for c in s), 'jq_exit': jq.returncode})) if jq.returncode: print(jq.stderr.decode().strip().split(' at line ', 1)[0]) if name == 'split': print('TOON replacement characters:', run(artifact, '--format', 'toon').count('\ufffd'.encode())) # A derived display only; keep the original Evidence unchanged. display = {'value': s[:-1], 'truncated': const['truncated'], 'bytes': const['bytes']} assert display['value'] == 'a' + '\U0001f600' * 39 recovered = subprocess.run(['jq', '-e', '.'], input=json.dumps(display).encode(), env=env, capture_output=True) assert recovered.returncode == 0 print('Derived whole-scalar display jq exit:', recovered.returncode) PYTHON ``` ## Expected behavior For these compiler-produced constants containing only Unicode scalar values, return a bounded preview that ends between complete scalars, with the existing original marshal byte count and an accurate truncation flag. A successful JSON result should work with the documented jq consumer, and TOON should not acquire a replacement character from the preview cut. This does not require a universal code-point-counted limit: an 80-UTF-16-unit budget can stop before the final complete scalar. Preserving source-encoded lone surrogates under CPython surrogatepass is a separate existing behavior, outside this report. ## Impact and scope P2: a bounded constant-preview defect blocks the documented jq workflow for the successful result and corrupts the displayed final character. The original artifact is unchanged. Artifact byte count/SHA-256, bytecode byte count/SHA-256, original constant marshal byte count, and complete coverage matched independent checks in all five fixture cases. Only synthetic fixtures were measured; natural frequency and unrelated consumers/platforms are unknown. No security boundary was crossed. A verified workaround for displaying or processing this preview is to parse the original output with a parser accepting lone-surrogate escapes, then make a separate display value that omits the final incomplete high surrogate while retaining `truncated: true` and `bytes: 161`. The inline block demonstrates a 40-scalar prefix accepted by jq. Retain the original Evidence unchanged; the derived display is not replacement authenticated Evidence and does not make jq accept the original document. ## REA version Unreleased main e32a0494d42dedd5e79ca341d20c53f07479d9bb, package/CLI version 6.7.0, using its prepared compiled runtime. The latest published release tag, rea-agents-6.7.0, lacks this pyc command/parser; no stable-release runtime reproduction is claimed. ## Runtime and platform Node.js v24.18.0; npm 11.16.0; CPython 3.13.13 for compilation and code-object inspection only; jq 1.8.2; macOS 27.0.1 arm64. No install, build, native engine, or target-bytecode execution ran. ## Target and artifact context Four small hash-based CPython 3.13 pycs compiled from inert constant assignments, with runtime magic 3571 and exact stored constants verified by marshal inspection. Input strings contain no lone surrogates. A fifth case with a trailing ASCII character reproduced the same cut. Fixtures and child state lived under disposable temporary roots. ## Interface CLI ## Earliest failure stage Transport or result projection ## Relevant output or evidence ```text {"case": "split", "cli_exit": 0, "bytes": 161, "truncated": true, "utf16_units": 80, "unpaired_surrogate": true, "jq_exit": 5} jq: parse error: Invalid \uXXXX\uXXXX surrogate pair escape TOON replacement characters: 1 Derived whole-scalar display jq exit: 0 {"case": "astral-short", "cli_exit": 0, "bytes": 157, "truncated": false, "utf16_units": 79, "unpaired_surrogate": false, "jq_exit": 0} {"case": "astral-boundary", "cli_exit": 0, "bytes": 160, "truncated": false, "utf16_units": 80, "unpaired_surrogate": false, "jq_exit": 0} {"case": "ascii-long", "cli_exit": 0, "bytes": 81, "truncated": true, "utf16_units": 80, "unpaired_surrogate": false, "jq_exit": 0} ``` The four primary JSON calls reproduced on three CLI passes and one MCP pass; normalized MCP results matched CLI exactly. Three split-case TOON calls each contained one replacement character. Node JSON.parse also accepted the affected JSON: its string was 80 UTF-16 units, ended in `0xd83d`, and failed `isWellFormed()`. An escaped lone surrogate is not necessarily invalid JSON grammar; the observed incompatibility is with jq and the preview text itself. Contract breadcrumbs: `docs/python-pyc.md:3-14,44-51` documents the read-only pyc operation and truncated const summaries; `docs/cli.md:543-548` documents JSON consumers and jq. The shared preview is produced at `src/python/PythonPycParser.ts:1559-1568`, and `src/domain/python/pythonPyc.ts:126-131` validates length without detecting a split pair. Existing parser/service tests passed, 42 tests in two files with one worker. ## Typed outcome or recovery expected Return successful Evidence with a bounded, whole-scalar source-prefix preview for these scalar-only inputs. Preserve the original constant byte count, artifact/bytecode identity and applicable truncation flag. Either a shorter whole-scalar prefix within the current UTF-16 budget or another compatible bounded policy can satisfy this; a universal code-point-counted limit is not required. ## Analysis or suspected cause Source-derived and consistent with execution: the preview uses JavaScript string length and slice, so its cut can separate a valid surrogate pair. The shared service accepts the length-bounded result, which explains why CLI and MCP agree. This is a shared preview-generation issue, not a general JSON-serializer failure. No repair was applied. ## Additional context Live open-and-closed searches and the supplied issue-title snapshot found no matching report. Related independently closable cases are [#6437](https://github.com/morluto/rea/issues/6437) (negative-zero float projection), [#6438](https://github.com/morluto/rea/issues/6438) (decode-limit recovery), and [#6415](https://github.com/morluto/rea/issues/6415) (code-object hashability). [#5530](https://github.com/morluto/rea/issues/5530) concerns CLI token windows; this case occurs without either token flag and is already present in MCP structuredContent. [#5097](https://github.com/morluto/rea/issues/5097) concerns Frida JSON-byte accounting, and [#3177](https://github.com/morluto/rea/issues/3177) concerns Apple path ordering. These reports have different contracts and closure signals from this shared pyc preview. Proposed restoration check: Rerun the same compiler-produced fixtures through the CLI and `inspect_python_pyc` MCP tool. The split case must return a source-prefix preview containing only whole scalars, accurate truncation metadata, unchanged original byte counts/digests, JSON accepted by jq, and TOON with no introduced U+FFFD. Keep the shorter astral and exact-boundary astral controls untruncated and intact, and the 81-ASCII control truncated to 80 ASCII characters. A related executed case with one extra trailing ASCII character must also avoid splitting the scalar. These are restoration criteria; no source fix was made or tested. NOT_ENOUGH_INFO: natural-workload frequency, first affected source commit, last known good behavior, and other-platform behavior were not established. Existing support for source-encoded lone surrogates is outside this scalar-only input case.

bsakaguchi closed this issue

2026-10-31 00:44:55 +00:00

bsakaguchi commented

2026-10-31 00:44:56 +00:00

Author

Owner

@morluto commented on GitHub (Oct 23, 2026):

Resolved on main in 06032237c. The preview owner retreats only when its UTF-16 cut splits a valid surrogate pair. Executed controls preserve original byte count/truncation and deliberately preserve source lone-surrogate identity. Validation: src/python/PythonPycParser.test.ts — truncates string previews without introducing a lone surrogate; recorded execution at e1d9ebd3c. Parser suite52 passed; jq, TOON and stdio MCP were not replayed. The corrected preview is shared by those projections.

@morluto commented on GitHub (Oct 23, 2026): Resolved on main in [06032237c](https://github.com/morluto/rea/commit/06032237c6755e3dd5f9a8db732c89958b238cc7). The preview owner retreats only when its UTF-16 cut splits a valid surrogate pair. Executed controls preserve original byte count/truncation and deliberately preserve source lone-surrogate identity. Validation: `src/python/PythonPycParser.test.ts` — truncates string previews without introducing a lone surrogate; recorded execution at `e1d9ebd3c`. Parser suite52 passed; jq, TOON and stdio MCP were not replayed. The corrected preview is shared by those projections.

bsakaguchi referenced this issue

2026-10-31 01:12:30 +00:00

[PR #4338] [CLOSED] fix(javascript): verify loader origins, assignment operators, and bootstrap integrity in static analysis #6182

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3884

No description provided.

Rows
Columns