morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6677] [Bug] PYC inspection rejects one-byte strings accepted by matching CPython #4056

New issue

Closed

opened 2026-10-31 00:46:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:25 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6677

Withdrawn as an unestablished defect. The observed CPython/REA acceptance difference is real, but these are patched synthetic marshal records. Producer/spec validity and REA's obligation to accept them were not established. This report was submitted before the research handoff was complete and is now closed; it does not establish a bug in REA. No further bug checks were run after drafting.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-python-pyc rejects a CPython 3.12 pyc whose string constant uses a one-byte marshal tag with payload e9. Matching CPython 3.12.15 decodes that constant as "é" (U+00E9). REA exits 1 with invalid_request / invalid_input, issue reason invalid_format, and Marshal ASCII string contains a non-ASCII byte.

All four tags (a, A, z, Z) reproduced once through the production CLI and once through the shared production service. A Unicode-tag control encoding the same constant succeeds with complete coverage. No target bytecode was executed.

Reproduction

With the built source checkout and Node.js 24.22.1, create the inert fixtures using CPython 3.12.15:

WORK="$(mktemp -d)"
python3.12 - "$WORK" <<'PY'
import importlib.util, marshal, pathlib, struct, sys
root = pathlib.Path(sys.argv[1])
code = compile("x = 'é'\n", 'sample.py', 'exec', dont_inherit=True)
body = marshal.dumps(code, 2)
old = b'u' + struct.pack('<I', 2) + 'é'.encode('utf8')
assert body.count(old) == 1
header = importlib.util.MAGIC_NUMBER + struct.pack('<III', 0, 0, 0)
for label, payload in [('unicode-control', body), ('one-byte-z',
        body.replace(old, b'z\x01\xe9', 1))]:
    decoded = marshal.loads(payload)
    print(label, repr(decoded.co_consts))
    assert decoded.co_consts == ('é', None)
    (root / (label + '.pyc')).write_bytes(header + payload)
PY
node scripts/rea.mjs inspect-python-pyc "$WORK/unicode-control.pyc" --json
node scripts/rea.mjs inspect-python-pyc "$WORK/one-byte-z.pyc" --json

These are the fixture construction and CLI operations actually run, reduced to one failing tag and its control. The executed matrix also replaced the same record with a 01 00 00 00 e9, A 01 00 00 00 e9, and Z 01 e9. CLI subprocesses used isolated home/config/cache/data/state/temp directories and a clean environment. Neither compilation nor marshal.loads executes the target code.

Expected behavior

Contract limitation: matching-interpreter acceptance is established, but producer/spec validity for the patched one-byte records and an explicit REA promise to accept them were not established before this draft. The expected behavior below is conditional on that compatibility being in scope; do not submit this as a confirmed defect on oracle acceptance alone. No subsequent checks were performed.

Decode these reference-reader-compatible one-byte strings as Latin-1 and report the U+00E9 constant, preserving its one-byte marshal length. CPython's matching reader constructs all four tags with PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, ptr, n), without an ASCII-only check. Its executed decoding confirms applicability to these exact fixtures.

REA documents inspection of the supported CPython marshal graph. This is decoding compatibility, not a request to execute bytecode or add another Python version.

Impact and scope

P2: an entire supported-version inspection fails on a string representation the corresponding interpreter reads successfully. The fixtures are synthetic, 144–148 bytes, and below all default budgets. Normal compiler-output prevalence is unknown; this report does not claim that CPython's writer emits high bytes under these tags. No input mutation or security-boundary failure was observed.

Matching CPython decoding was demonstrated as an external inspection workaround. Changing the selected file to a Unicode encoding also succeeds but changes artifact identity and is not an equivalent REA inspection.

REA version

Source a7f967cf240eba8309cbf4526b2fc6338a52a617, package/CLI 6.7.0, using the centrally prepared compiled runtime. This source is newer than published release rea-agents-6.7.0; released-package behavior is not claimed.

Runtime and platform

Node.js v24.22.1; CPython 3.12.15 for fixture construction and independent decoding; Linux x86_64. No native analysis provider.

Target and artifact context

CPython 3.12 timestamp-header pycs, magic 3531, containing a module compiled from x = 'é' and constant tuple ('é', None). Marshal version 2 avoids reference relocation when replacing the string record. The Unicode control stores 75 02 00 00 00 c3 a9; the shortest failing form stores 7a 01 e9.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed on the same selected files:

CPython, all five files: ('é', None), first constant code points [233]
CLI unicode-control: exit 0, coverage complete
  {"kind":"str","bytes":2,"truncated":false,"value":"é"}
CLI a/A/z/Z: exit 1, empty stderr
  code: invalid_request; category: invalid_input
  issues[0].path: ["path"]; reason: invalid_format
  message suffix: Marshal ASCII string contains a non-ASCII byte.

Source-derived explanation: all four tags enter the ASCII branch, which rejects every byte above 0x7f before its Latin-1 conversion. The provider maps that rejection to invalid input.

Evidence provenance: observed production CLI/service results and matching-interpreter decoding. Local verification: reproduced. Reproduction completeness: complete for the reduced fixtures. Missing coverage: live MCP, other CPython versions/platforms, and an affected naturally generated artifact. No repaired implementation was tested.

Typed outcome or recovery expected

Successful inspection with the decoded U+00E9 constant and correct stored byte count, rather than a malformed-input error. Existing size, UTF-8, and reference constraints should remain effective; their restoration tests were not run or proposed as passing.

Analysis or suspected cause

The parser treats the tag's historical ASCII name as a stricter acceptance rule than the matching CPython reader applies. Removing the refusal alone also needs care at the string-equality boundary: Latin-1 and UTF-8 records can encode the same Python string, while raw byte encodings differ. That equality concern is source-derived and was not separately reproduced or presented as another finding.

Additional context

Live open/closed searches covered marshal ASCII, pyc Latin-1, PythonPycParser ASCII, and marshal non-ASCII. Related open #6476 concerns slicing a successfully decoded supplementary Unicode scalar in a preview; closed #6593 concerns Unicode code-name equality. This report fails earlier at one-byte tag admission. No matching report was found.

Prepared with AI assistance.

Triage assessment

Impact level: P2
Assessment status: provisional
Impact basis: The public CLI refuses four bounded inputs accepted by matching CPython. P2 applies if these patched one-byte records belong to REA's supported profile; producer/spec validity and that compatibility obligation remain unestablished. Natural prevalence and broader platform reach are unknown.
Workaround status: available
Workaround basis: Matching CPython 3.12.15 independently decoded each exact selected marshal body and recovered its constants without executing target bytecode. This requires a matching interpreter and does not repair REA's result.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6677 > **Withdrawn as an unestablished defect.** The observed CPython/REA acceptance difference is real, but these are patched synthetic marshal records. Producer/spec validity and REA's obligation to accept them were not established. This report was submitted before the research handoff was complete and is now closed; it does not establish a bug in REA. No further bug checks were run after drafting. ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-python-pyc` rejects a CPython 3.12 pyc whose string constant uses a one-byte marshal tag with payload `e9`. Matching CPython 3.12.15 decodes that constant as `"é"` (U+00E9). REA exits 1 with `invalid_request` / `invalid_input`, issue reason `invalid_format`, and `Marshal ASCII string contains a non-ASCII byte.` All four tags (`a`, `A`, `z`, `Z`) reproduced once through the production CLI and once through the shared production service. A Unicode-tag control encoding the same constant succeeds with complete coverage. No target bytecode was executed. ## Reproduction With the built source checkout and Node.js 24.22.1, create the inert fixtures using CPython 3.12.15: ```sh WORK="$(mktemp -d)" python3.12 - "$WORK" <<'PY' import importlib.util, marshal, pathlib, struct, sys root = pathlib.Path(sys.argv[1]) code = compile("x = 'é'\n", 'sample.py', 'exec', dont_inherit=True) body = marshal.dumps(code, 2) old = b'u' + struct.pack('<I', 2) + 'é'.encode('utf8') assert body.count(old) == 1 header = importlib.util.MAGIC_NUMBER + struct.pack('<III', 0, 0, 0) for label, payload in [('unicode-control', body), ('one-byte-z', body.replace(old, b'z\x01\xe9', 1))]: decoded = marshal.loads(payload) print(label, repr(decoded.co_consts)) assert decoded.co_consts == ('é', None) (root / (label + '.pyc')).write_bytes(header + payload) PY node scripts/rea.mjs inspect-python-pyc "$WORK/unicode-control.pyc" --json node scripts/rea.mjs inspect-python-pyc "$WORK/one-byte-z.pyc" --json ``` These are the fixture construction and CLI operations actually run, reduced to one failing tag and its control. The executed matrix also replaced the same record with `a 01 00 00 00 e9`, `A 01 00 00 00 e9`, and `Z 01 e9`. CLI subprocesses used isolated home/config/cache/data/state/temp directories and a clean environment. Neither compilation nor `marshal.loads` executes the target code. ## Expected behavior Contract limitation: matching-interpreter acceptance is established, but producer/spec validity for the patched one-byte records and an explicit REA promise to accept them were not established before this draft. The expected behavior below is conditional on that compatibility being in scope; do not submit this as a confirmed defect on oracle acceptance alone. No subsequent checks were performed. Decode these reference-reader-compatible one-byte strings as Latin-1 and report the U+00E9 constant, preserving its one-byte marshal length. CPython's [matching reader](https://github.com/python/cpython/blob/v3.12.15/Python/marshal.c#L1144-L1170) constructs all four tags with `PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, ptr, n)`, without an ASCII-only check. Its executed decoding confirms applicability to these exact fixtures. REA [documents inspection of the supported CPython marshal graph](https://github.com/morluto/rea/blob/a7f967cf240eba8309cbf4526b2fc6338a52a617/docs/python-pyc.md#L25-L48). This is decoding compatibility, not a request to execute bytecode or add another Python version. ## Impact and scope P2: an entire supported-version inspection fails on a string representation the corresponding interpreter reads successfully. The fixtures are synthetic, 144–148 bytes, and below all default budgets. Normal compiler-output prevalence is unknown; this report does not claim that CPython's writer emits high bytes under these tags. No input mutation or security-boundary failure was observed. Matching CPython decoding was demonstrated as an external inspection workaround. Changing the selected file to a Unicode encoding also succeeds but changes artifact identity and is not an equivalent REA inspection. ## REA version Source `a7f967cf240eba8309cbf4526b2fc6338a52a617`, package/CLI 6.7.0, using the centrally prepared compiled runtime. This source is newer than published release `rea-agents-6.7.0`; released-package behavior is not claimed. ## Runtime and platform Node.js v24.22.1; CPython 3.12.15 for fixture construction and independent decoding; Linux x86_64. No native analysis provider. ## Target and artifact context CPython 3.12 timestamp-header pycs, magic 3531, containing a module compiled from `x = 'é'` and constant tuple `('é', None)`. Marshal version 2 avoids reference relocation when replacing the string record. The Unicode control stores `75 02 00 00 00 c3 a9`; the shortest failing form stores `7a 01 e9`. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Observed on the same selected files: ```text CPython, all five files: ('é', None), first constant code points [233] CLI unicode-control: exit 0, coverage complete {"kind":"str","bytes":2,"truncated":false,"value":"é"} CLI a/A/z/Z: exit 1, empty stderr code: invalid_request; category: invalid_input issues[0].path: ["path"]; reason: invalid_format message suffix: Marshal ASCII string contains a non-ASCII byte. ``` Source-derived explanation: [all four tags enter the ASCII branch](https://github.com/morluto/rea/blob/a7f967cf240eba8309cbf4526b2fc6338a52a617/src/python/PythonPycParser.ts#L419-L424), which [rejects every byte above 0x7f](https://github.com/morluto/rea/blob/a7f967cf240eba8309cbf4526b2fc6338a52a617/src/python/PythonPycParser.ts#L1454-L1464) before its Latin-1 conversion. The [provider maps that rejection to invalid input](https://github.com/morluto/rea/blob/a7f967cf240eba8309cbf4526b2fc6338a52a617/src/python/PythonPycProvider.ts#L61-L75). Evidence provenance: observed production CLI/service results and matching-interpreter decoding. Local verification: reproduced. Reproduction completeness: complete for the reduced fixtures. Missing coverage: live MCP, other CPython versions/platforms, and an affected naturally generated artifact. No repaired implementation was tested. ## Typed outcome or recovery expected Successful inspection with the decoded U+00E9 constant and correct stored byte count, rather than a malformed-input error. Existing size, UTF-8, and reference constraints should remain effective; their restoration tests were not run or proposed as passing. ## Analysis or suspected cause The parser treats the tag's historical ASCII name as a stricter acceptance rule than the matching CPython reader applies. Removing the refusal alone also needs care at the string-equality boundary: Latin-1 and UTF-8 records can encode the same Python string, while raw byte encodings differ. That equality concern is source-derived and was not separately reproduced or presented as another finding. ## Additional context Live open/closed searches covered `marshal ASCII`, `pyc Latin-1`, `PythonPycParser ASCII`, and `marshal non-ASCII`. Related open [#6476](https://github.com/morluto/rea/issues/6476) concerns slicing a successfully decoded supplementary Unicode scalar in a preview; closed [#6593](https://github.com/morluto/rea/issues/6593) concerns Unicode code-name equality. This report fails earlier at one-byte tag admission. No matching report was found. Prepared with AI assistance. ## Triage assessment - Impact level: P2 - Assessment status: provisional - Impact basis: The public CLI refuses four bounded inputs accepted by matching CPython. P2 applies if these patched one-byte records belong to REA's supported profile; producer/spec validity and that compatibility obligation remain unestablished. Natural prevalence and broader platform reach are unknown. - Workaround status: available - Workaround basis: Matching CPython 3.12.15 independently decoded each exact selected marshal body and recovered its constants without executing target bytecode. This requires a matching interpreter and does not repair REA's result.

bsakaguchi closed this issue

2026-10-31 00:46:25 +00:00

bsakaguchi referenced this issue

2026-11-01 21:46:23 +00:00

[PR #6260] [MERGED] feat(managed): project call and field-access edges into the application graph #7236