morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6676] [Bug] PYZ inventory leaves valid interned Unicode module names opaque #4055

New issue

Closed

opened 2026-10-31 00:46:25 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:25 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6676

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-artifact leaves a valid producer-generated PYZ opaque when its module name is an interned Unicode string. A module named café is absent from the inventory, the PYZ reports pyz_status: "opaque", and container-entry coverage becomes unknown. The same writer and ordinary module body with the ASCII name cafe produce a decoded PYZ and a PYZ.pyz/cafe.pyc occurrence.

The independent producer reader lists café and extracts its 110 raw marshal bytes successfully. The Unicode and ASCII carriers each passed through the source CLI twice: both Unicode inspections were opaque (2/2), and both ASCII controls decoded (2/2). All four CLI calls exited 0. No packaged target or inspected bytecode was executed.

Reproduction

Use the source checkout and locked dependencies identified below, with the fixture producer available on the selected Python interpreter's import path. The executed setup placed the producer and its dependencies in a disposable directory; no global provider installation was required.

Save the following as reproduce.py. It uses the real archive writers and their ordinary defaults. Compiling an import statement supplies its normal code-name string; there is no manual sys.intern, edited marshal stream, or target execution. The native carrier is a copy of the distributed Linux bootloader with the generated CArchive appended. This is an inspection fixture, not a complete runnable application with bundled runtime dependencies.

import hashlib, json, pathlib, struct, sys
import PyInstaller
from PyInstaller.archive.writers import CArchiveWriter, ZlibArchiveWriter
from PyInstaller.loader.pyimod01_archive import ZlibArchiveReader

out = pathlib.Path(sys.argv[1])
out.mkdir(parents=True, exist_ok=True)
bootloader = pathlib.Path(PyInstaller.__file__).parent / 'bootloader/Linux-64bit-intel/run'
for label, spelling in [('ascii', 'cafe'), ('unicode', 'café')]:
    name = compile('import ' + spelling, 'entry.py', 'exec').co_names[0]
    code = compile('value = 42\n', spelling + '.py', 'exec')
    pyz = out / (label + '.pyz')
    ZlibArchiveWriter(str(pyz), [(name, spelling + '.py', 'PYMODULE')], {name: code})
    pkg = out / (label + '.pkg')
    CArchiveWriter(str(pkg), [('PYZ.pyz', str(pyz), False, 'z')], 'libpython3.13.so.1.0')
    app = out / (label + '.app')
    app.write_bytes(bootloader.read_bytes() + pkg.read_bytes())
    image = pyz.read_bytes()
    offset = struct.unpack('!i', image[8:12])[0]
    reader = ZlibArchiveReader(str(pyz))
    raw = reader.extract(name, raw=True)
    print(json.dumps({'case': label, 'toc': reader.toc, 'raw_module_bytes': len(raw),
      'toc_hex': image[offset:].hex(), 'carrier_sha256': hashlib.sha256(app.read_bytes()).hexdigest()}, ensure_ascii=False))

Save this source CLI runner as run-cli.mts:

import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';
const [repo, ...args] = process.argv.slice(2);
const { createCli, closeCliJavaScriptResources } =
  await import(pathToFileURL(resolve(repo, 'src/cli.ts')).href);
try {
  await createCli({ REA_LOG_LEVEL: 'silent' }).serve(args, {
    stdout: (output: string) => process.stdout.write(output),
    exit: (code: number) => { process.exitCode = code; },
  });
} finally { await closeCliJavaScriptResources(); }

From the repository root, run the following with python and node selecting the recorded fixture and REA runtimes. The generator and source CLI logic above were executed; the temporary directory variable below substitutes for investigator-specific locations.

REA_REPRO="$(mktemp -d)"
python /path/to/reproduce.py "$REA_REPRO"
node --import ./node_modules/tsx/dist/loader.mjs /path/to/run-cli.mts "$PWD" inspect-artifact "$REA_REPRO/ascii.app" --json
node --import ./node_modules/tsx/dist/loader.mjs /path/to/run-cli.mts "$PWD" inspect-artifact "$REA_REPRO/unicode.app" --json

Compare the first inventory substep's normalized_result.coverage and occurrences. The fixture, its upstream reader check, and both source CLI cases were run before this draft was written.

Expected behavior

Both names should be decoded as strings, preserving the module's source name and projected .pyc occurrence with complete container-entry coverage for these fully supported one-entry tables.

REA's pinned Python packaged application contract supports the selected magic and marshal lists, tuples, strings and integers. Both directories contain only those types. The producer writes its directory with marshal.dumps(toc). CPython's pinned Unicode marshal writer uses TYPE_INTERNED for an interned non-ASCII string. This is a supported string value's normal encoding, not an additional directory value type.

Impact and scope

Confirmed compatibility failure for a valid non-ASCII module name in an otherwise ordinary one-module PYZ. The module remains inaccessible through REA's decoded module inventory; the outer PYZ and its digest remain available with an explicit limitation. The observed result is incomplete coverage, not a false claim that the module is absent.

Real-application frequency, multi-module tables, other Python releases, MCP transport, and other platforms were not tested. The standalone producer reader demonstrates that the module is recoverable, but a complete REA workaround for an unchanged carrier was not demonstrated.

REA version

Package 6.7.0, source revision 99a5ffae3bf3d73f11bca3d9b8893347964b10b9. Locked dependencies were installed. The production source CLI was loaded through tsx; the published npm binary was not tested.

Runtime and platform

Node.js v24.22.1; Linux x86_64. Fixture generation and the independent reader used CPython 3.13.17 and PyInstaller 6.22.3, obtained as disposable local tooling. No native analysis provider was involved.

Target and artifact context

Two small CArchives with the 2.1 cookie layout, each containing one stored PYZ.pyz member and no encryption marker. Each PYZ has one compressed module with source value = 42\n. The selected magic is f30d0d0a. Only the module name and the resulting filename metadata differ between the Unicode and ASCII cases.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed producer-reader output, shortened to the relevant fields:

{"case":"ascii","toc":{"cafe":[0,17,88]},"raw_module_bytes":106}
{"case":"unicode","toc":{"café":[0,17,91]},"raw_module_bytes":110}

The exact producer-generated directory bytes are:

ascii:   db01000000a902da04636166652903e900000000e911000000e958000000
unicode: db01000000a902f405000000636166c3a92903e900000000e911000000e95b000000

In the Unicode directory, f4 is FLAG_REF | TYPE_INTERNED; its five UTF-8 bytes encode café.

The source CLI returned these relevant facts:

ASCII:
  inspection coverage: complete-within-substeps
  inventory coverage: {"scope":"container-entries","status":"complete"}
  PYZ.pyz: entry_kind=directory, pyz_status=decoded, pyz_entry_count=1
  PYZ.pyz/cafe.pyc: entry_kind=file

Unicode:
  inspection coverage: partial
  inventory coverage: {"scope":"container-entries","status":"unknown"}
  PYZ.pyz: entry_kind=file, pyz_status=opaque, pyz_entry_count=null
  no module occurrence
  limitation: PYZ table of contents could not be parsed (PYZ marshal value uses a type outside the supported subset); the member was left opaque.

The tested carrier SHA-256 values were:

ascii:   287a66021457d3c43ae8e3d6dec822eed785f1a738518ea8b6b63321cf0d41ab
unicode: 5ccbbbce25799a8781e44f631ab068820220112b5f833e337b31e1acea749614

Evidence provenance: observed producer-reader and source CLI results, with source-derived encoding analysis. Local verification: reproduced. Reproduction completeness: complete for the supplied writer-generated inspection fixtures.

Typed outcome or recovery expected

For the unchanged Unicode fixture, return a decoded PYZ with its café source name, one PYZ.pyz/café.pyc occurrence and complete container-entry coverage. Keep the ASCII control decoded. Merely removing the limitation while still omitting the module would not restore the contract. This restoration check is proposed; no fix was tested.

Analysis or suspected cause

Source-derived: readObject masks FLAG_REF and recognizes the Unicode string tag 0x75, plus the ASCII and ASCII-interned forms, but has no branch for TYPE_INTERNED (0x74). parsePyzParts then turns that parser failure into an opaque PYZ. The independently inspected table contains a valid string at that position.

Additional context

All-state GitHub issue searches included repo:morluto/rea is:issue PYZ interned and repo:morluto/rea is:issue "marshal" "interned"; both returned zero matches. A broader PYZ unicode search was also reviewed, alongside the initial open-issue corpus. No regression range is established. No bug reproduction was rerun after this draft was written.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A valid producer-generated table loses its module inventory because of a normal string encoding, while the ASCII control works. The loss is explicitly marked unknown and the original PYZ remains retained, so the observed consequence is a bounded compatibility failure.
Workaround status: unverified
Workaround basis: The independent producer reader recovered the standalone PYZ module bytes, but a complete alternative REA inventory or extraction workflow for the unchanged carrier was not demonstrated.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6676 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-artifact` leaves a valid producer-generated PYZ opaque when its module name is an interned Unicode string. A module named `café` is absent from the inventory, the PYZ reports `pyz_status: "opaque"`, and container-entry coverage becomes `unknown`. The same writer and ordinary module body with the ASCII name `cafe` produce a decoded PYZ and a `PYZ.pyz/cafe.pyc` occurrence. The independent producer reader lists `café` and extracts its 110 raw marshal bytes successfully. The Unicode and ASCII carriers each passed through the source CLI twice: both Unicode inspections were opaque (2/2), and both ASCII controls decoded (2/2). All four CLI calls exited 0. No packaged target or inspected bytecode was executed. ## Reproduction Use the source checkout and locked dependencies identified below, with the fixture producer available on the selected Python interpreter's import path. The executed setup placed the producer and its dependencies in a disposable directory; no global provider installation was required. Save the following as `reproduce.py`. It uses the real archive writers and their ordinary defaults. Compiling an `import` statement supplies its normal code-name string; there is no manual `sys.intern`, edited marshal stream, or target execution. The native carrier is a copy of the distributed Linux bootloader with the generated CArchive appended. This is an inspection fixture, not a complete runnable application with bundled runtime dependencies. ```python import hashlib, json, pathlib, struct, sys import PyInstaller from PyInstaller.archive.writers import CArchiveWriter, ZlibArchiveWriter from PyInstaller.loader.pyimod01_archive import ZlibArchiveReader out = pathlib.Path(sys.argv[1]) out.mkdir(parents=True, exist_ok=True) bootloader = pathlib.Path(PyInstaller.__file__).parent / 'bootloader/Linux-64bit-intel/run' for label, spelling in [('ascii', 'cafe'), ('unicode', 'café')]: name = compile('import ' + spelling, 'entry.py', 'exec').co_names[0] code = compile('value = 42\n', spelling + '.py', 'exec') pyz = out / (label + '.pyz') ZlibArchiveWriter(str(pyz), [(name, spelling + '.py', 'PYMODULE')], {name: code}) pkg = out / (label + '.pkg') CArchiveWriter(str(pkg), [('PYZ.pyz', str(pyz), False, 'z')], 'libpython3.13.so.1.0') app = out / (label + '.app') app.write_bytes(bootloader.read_bytes() + pkg.read_bytes()) image = pyz.read_bytes() offset = struct.unpack('!i', image[8:12])[0] reader = ZlibArchiveReader(str(pyz)) raw = reader.extract(name, raw=True) print(json.dumps({'case': label, 'toc': reader.toc, 'raw_module_bytes': len(raw), 'toc_hex': image[offset:].hex(), 'carrier_sha256': hashlib.sha256(app.read_bytes()).hexdigest()}, ensure_ascii=False)) ``` Save this source CLI runner as `run-cli.mts`: ```ts import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; const [repo, ...args] = process.argv.slice(2); const { createCli, closeCliJavaScriptResources } = await import(pathToFileURL(resolve(repo, 'src/cli.ts')).href); try { await createCli({ REA_LOG_LEVEL: 'silent' }).serve(args, { stdout: (output: string) => process.stdout.write(output), exit: (code: number) => { process.exitCode = code; }, }); } finally { await closeCliJavaScriptResources(); } ``` From the repository root, run the following with `python` and `node` selecting the recorded fixture and REA runtimes. The generator and source CLI logic above were executed; the temporary directory variable below substitutes for investigator-specific locations. ```sh REA_REPRO="$(mktemp -d)" python /path/to/reproduce.py "$REA_REPRO" node --import ./node_modules/tsx/dist/loader.mjs /path/to/run-cli.mts "$PWD" inspect-artifact "$REA_REPRO/ascii.app" --json node --import ./node_modules/tsx/dist/loader.mjs /path/to/run-cli.mts "$PWD" inspect-artifact "$REA_REPRO/unicode.app" --json ``` Compare the first inventory substep's `normalized_result.coverage` and `occurrences`. The fixture, its upstream reader check, and both source CLI cases were run before this draft was written. ## Expected behavior Both names should be decoded as strings, preserving the module's source name and projected `.pyc` occurrence with complete container-entry coverage for these fully supported one-entry tables. REA's pinned [Python packaged application contract](https://github.com/morluto/rea/blob/99a5ffae3bf3d73f11bca3d9b8893347964b10b9/docs/python-packaged-applications.md#L106-L118) supports the selected magic and marshal lists, tuples, strings and integers. Both directories contain only those types. The producer [writes its directory with `marshal.dumps(toc)`](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/PyInstaller/archive/writers.py#L49-L67). CPython's pinned [Unicode marshal writer](https://github.com/python/cpython/blob/v3.13.17/Python/marshal.c#L470-L482) uses `TYPE_INTERNED` for an interned non-ASCII string. This is a supported string value's normal encoding, not an additional directory value type. ## Impact and scope Confirmed compatibility failure for a valid non-ASCII module name in an otherwise ordinary one-module PYZ. The module remains inaccessible through REA's decoded module inventory; the outer PYZ and its digest remain available with an explicit limitation. The observed result is incomplete coverage, not a false claim that the module is absent. Real-application frequency, multi-module tables, other Python releases, MCP transport, and other platforms were not tested. The standalone producer reader demonstrates that the module is recoverable, but a complete REA workaround for an unchanged carrier was not demonstrated. ## REA version Package `6.7.0`, source revision `99a5ffae3bf3d73f11bca3d9b8893347964b10b9`. Locked dependencies were installed. The production source CLI was loaded through `tsx`; the published npm binary was not tested. ## Runtime and platform Node.js `v24.22.1`; Linux x86_64. Fixture generation and the independent reader used CPython `3.13.17` and PyInstaller `6.22.3`, obtained as disposable local tooling. No native analysis provider was involved. ## Target and artifact context Two small CArchives with the 2.1 cookie layout, each containing one stored `PYZ.pyz` member and no encryption marker. Each PYZ has one compressed module with source `value = 42\n`. The selected magic is `f30d0d0a`. Only the module name and the resulting filename metadata differ between the Unicode and ASCII cases. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Observed producer-reader output, shortened to the relevant fields: ```json {"case":"ascii","toc":{"cafe":[0,17,88]},"raw_module_bytes":106} {"case":"unicode","toc":{"café":[0,17,91]},"raw_module_bytes":110} ``` The exact producer-generated directory bytes are: ```text ascii: db01000000a902da04636166652903e900000000e911000000e958000000 unicode: db01000000a902f405000000636166c3a92903e900000000e911000000e95b000000 ``` In the Unicode directory, `f4` is `FLAG_REF | TYPE_INTERNED`; its five UTF-8 bytes encode `café`. The source CLI returned these relevant facts: ```text ASCII: inspection coverage: complete-within-substeps inventory coverage: {"scope":"container-entries","status":"complete"} PYZ.pyz: entry_kind=directory, pyz_status=decoded, pyz_entry_count=1 PYZ.pyz/cafe.pyc: entry_kind=file Unicode: inspection coverage: partial inventory coverage: {"scope":"container-entries","status":"unknown"} PYZ.pyz: entry_kind=file, pyz_status=opaque, pyz_entry_count=null no module occurrence limitation: PYZ table of contents could not be parsed (PYZ marshal value uses a type outside the supported subset); the member was left opaque. ``` The tested carrier SHA-256 values were: ```text ascii: 287a66021457d3c43ae8e3d6dec822eed785f1a738518ea8b6b63321cf0d41ab unicode: 5ccbbbce25799a8781e44f631ab068820220112b5f833e337b31e1acea749614 ``` Evidence provenance: observed producer-reader and source CLI results, with source-derived encoding analysis. Local verification: reproduced. Reproduction completeness: complete for the supplied writer-generated inspection fixtures. ## Typed outcome or recovery expected For the unchanged Unicode fixture, return a decoded PYZ with its `café` source name, one `PYZ.pyz/café.pyc` occurrence and complete container-entry coverage. Keep the ASCII control decoded. Merely removing the limitation while still omitting the module would not restore the contract. This restoration check is proposed; no fix was tested. ## Analysis or suspected cause Source-derived: [`readObject`](https://github.com/morluto/rea/blob/99a5ffae3bf3d73f11bca3d9b8893347964b10b9/src/artifacts/pyinstaller/MarshalSubset.ts#L68-L85) masks `FLAG_REF` and recognizes the Unicode string tag `0x75`, plus the ASCII and ASCII-interned forms, but has no branch for `TYPE_INTERNED` (`0x74`). [`parsePyzParts`](https://github.com/morluto/rea/blob/99a5ffae3bf3d73f11bca3d9b8893347964b10b9/src/artifacts/pyinstaller/PyzArchive.ts#L132-L140) then turns that parser failure into an opaque PYZ. The independently inspected table contains a valid string at that position. ## Additional context All-state GitHub issue searches included `repo:morluto/rea is:issue PYZ interned` and `repo:morluto/rea is:issue "marshal" "interned"`; both returned zero matches. A broader `PYZ unicode` search was also reviewed, alongside the initial open-issue corpus. No regression range is established. No bug reproduction was rerun after this draft was written. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A valid producer-generated table loses its module inventory because of a normal string encoding, while the ASCII control works. The loss is explicitly marked unknown and the original PYZ remains retained, so the observed consequence is a bounded compatibility failure. - Workaround status: unverified - Workaround basis: The independent producer reader recovered the standalone PYZ module bytes, but a complete alternative REA inventory or extraction workflow for the unchanged carrier was not demonstrated.

bsakaguchi closed this issue

2026-10-31 00:46:25 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:23 +00:00

[PR #4076] [CLOSED] fix(managed): keep runtime-provided methods out of unresolved native bodies #6040