[GH-ISSUE #6559] [Bug] PYZ tables with invalid non-normalized marshal longs are reported decoded #3950

New issue

Closed

opened 2026-10-31 00:45:31 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:31 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6559

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

An executed PyInstallerArtifactReader.entries() production-boundary probe accepts a PYZ table whose namespace type code 3 is serialized as a two-digit marshal long with a zero most-significant digit. It projects PYZ.pyz/pkg, marks the container pyz_status: "decoded" with pyz_entry_count: 1, and reports traversal {"status":"complete","limitations":[]}. CPython rejects those identical table bytes with ValueError: bad marshal data (unnormalized long data). Canonical integer and normalized-long encodings of 3 both load successfully and produce the same REA entry.

Reproduction

Use the main source checkout and its already-installed locked dependencies. The following preserves the fixture bytes and production reader calls actually exercised by an external .mts wrapper; this inline spelling was not separately run. It creates only synthetic files under a fresh scratch directory and reads them without executing target bytecode or invoking a real provider.

WORK="$(mktemp -d)" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS'
import {writeFile} from 'node:fs/promises';
import {join} from 'node:path';
import {cArchiveExecutable,pyzArchive,PYTHON_39_MAGIC} from './src/artifacts/pyinstaller/CArchive.fixture.ts';
import {PyInstallerArtifactReader} from './src/artifacts/PyInstallerArtifactReader.ts';
for (const [name, hex] of [
  ['normal-int', '5b0100000029027a03706b6729036903000000690c0000006900000000'],
  ['normalized-long', '5b0100000029027a03706b6729036c010000000300690c0000006900000000'],
  ['unnormalized-long', '5b0100000029027a03706b6729036c0200000003000000690c0000006900000000'],
]) {
  const toc = Buffer.from(hex, 'hex');
  await writeFile(join(process.env.WORK, name + '.marshal'), toc);
  const pyz = pyzArchive({magic: PYTHON_39_MAGIC, modules: [
    {name: 'pkg', typeCode: 3, body: Buffer.alloc(0)}], toc});
  const carrier = cArchiveExecutable({layout: '2.1', pythonVersion: 309,
    pythonLibrary: 'libpython3.9.so.1.0', members: [
      {name: 'PYZ.pyz', typeCode: 'z', payload: pyz, compress: false}]});
  const file = join(process.env.WORK, name + '.elf');
  await writeFile(file, carrier);
  const reader = new PyInstallerArtifactReader(file);
  try {
    for await (const entry of reader.entries(undefined, {
      onCoverage: value => console.log(JSON.stringify({name, coverage: value}))
    })) console.log(JSON.stringify({name, path: entry.path,
      metadata: entry.formatMetadata, limitations: entry.limitations}));
  } finally { await reader.close(); }
}
JS

For the oracle, the retained wrapper separately ran python3 -I -S -c to call marshal.loads() on each saved .marshal file, catching and printing the exception. It did not load the carrier, import a module, or execute code. Each of the three table cases was observed once. An initial wrapper stopped after the integer control while looking for an unavailable published-reader module; the remaining two cases were then executed without repeating the control. The public inspect_artifact CLI/MCP adapters were not run.

Expected behavior

A marshal long with any digits must have a nonzero highest digit. Do not normalize an invalid serialized number into valid module metadata. This malformed table should follow the existing PYZ format-failure behavior: leave the PYZ opaque, report the parse limitation, and avoid projecting decoded child entries from it. Both canonical encodings must remain readable. CPython's marshal reader explicitly rejects this encoding, and REA already rejects non-normalized longs in its separate pyc parser.

Impact and scope

A corrupted PYZ table can be projected as successfully decoded metadata with complete traversal even though CPython cannot load that table. The demonstrated result invents a decoded namespace child from invalid marshal data. Real-world frequency and other long-field positions were not measured. No bytecode execution, security boundary, resource-exhaustion claim, or severe incident is involved.

REA version

Main source 6c4d880f2dd9ca7d42593be916f90ec19b8a09d7, package metadata 6.7.0. Main-only finding: the PyInstaller reader is absent from the installed npm release, so no published-runtime reproduction is claimed.

Runtime and platform

Node.js v26.12.1 and CPython 3.14.8, Linux x64. The source check used the checkout's locked tsx loader with ambient Node rather than the repository-pinned Node toolchain. The executed Python oracle decoded primitive marshal table values, not 3.9 code objects; the same normalization rule for the declared 3.9 format is also established by CPython source.

Target and artifact context

Synthetic ELF carrier with one uncompressed CArchive member PYZ.pyz, a CPython 3.9 PYZ header, and one zero-payload namespace entry. Its table declares [("pkg", (3, 12, 0))]; only the encoding of 3 differs between cases. The invalid long's bytes are 6c 02 00 00 00 03 00 00 00.

Interface

Other

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

{"case":"normal-int","cpython":"loaded","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}}
{"case":"normalized-long","cpython":"loaded","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}}
{"case":"unnormalized-long","cpython":"ValueError: bad marshal data (unnormalized long data)","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}}

Expected source: CPython's marshal implementation, r_PyLong checks the topmost marshal digit and raises bad marshal data (unnormalized long data); REA's existing PYZ format-failure path; REA pyc reader's normalization check.
Failure source: The executed production reader and CPython outputs above; the PYZ subset reader accumulates the digits and returns the number without checking the top digit.
Evidence provenance: observed, with the declared-format oracle additionally grounded in CPython source.
Local verification: reproduced at the production reader boundary; no public adapter execution.
Reproduction completeness: complete.

Additional context

The existing negative-PYZ-marshal-size report (#6120) concerns signed lengths being misclassified as resource-limit failures and aborting sibling inventory. This report uses positive digit count 2 and valid small list/string lengths; it concerns successful decoding of an invalid normalized-number representation. These behaviors have independent failure signals and restoration checks.

Restoration check (not executed): The invalid table must leave PYZ.pyz opaque with a normalization/format limitation and omit PYZ.pyz/pkg; both canonical table encodings must still project that child successfully. No implementation approach is prescribed.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed reader projects decoded child metadata and complete traversal from a table directly rejected by CPython, while two valid encoding controls agree.
Workaround status: unverified
Workaround basis: A caller could separately validate the primitive table with a compatible marshal reader, but no supported REA workflow that does so was observed; executing or importing the target is unnecessary and was not used.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6559 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior An executed `PyInstallerArtifactReader.entries()` production-boundary probe accepts a PYZ table whose namespace type code 3 is serialized as a two-digit marshal long with a zero most-significant digit. It projects `PYZ.pyz/pkg`, marks the container `pyz_status: "decoded"` with `pyz_entry_count: 1`, and reports traversal `{"status":"complete","limitations":[]}`. CPython rejects those identical table bytes with `ValueError: bad marshal data (unnormalized long data)`. Canonical integer and normalized-long encodings of 3 both load successfully and produce the same REA entry. ### Reproduction Use the main source checkout and its already-installed locked dependencies. The following preserves the fixture bytes and production reader calls actually exercised by an external `.mts` wrapper; this inline spelling was not separately run. It creates only synthetic files under a fresh scratch directory and reads them without executing target bytecode or invoking a real provider. ```sh WORK="$(mktemp -d)" node --import ./node_modules/tsx/dist/loader.mjs --input-type=module <<'JS' import {writeFile} from 'node:fs/promises'; import {join} from 'node:path'; import {cArchiveExecutable,pyzArchive,PYTHON_39_MAGIC} from './src/artifacts/pyinstaller/CArchive.fixture.ts'; import {PyInstallerArtifactReader} from './src/artifacts/PyInstallerArtifactReader.ts'; for (const [name, hex] of [ ['normal-int', '5b0100000029027a03706b6729036903000000690c0000006900000000'], ['normalized-long', '5b0100000029027a03706b6729036c010000000300690c0000006900000000'], ['unnormalized-long', '5b0100000029027a03706b6729036c0200000003000000690c0000006900000000'], ]) { const toc = Buffer.from(hex, 'hex'); await writeFile(join(process.env.WORK, name + '.marshal'), toc); const pyz = pyzArchive({magic: PYTHON_39_MAGIC, modules: [ {name: 'pkg', typeCode: 3, body: Buffer.alloc(0)}], toc}); const carrier = cArchiveExecutable({layout: '2.1', pythonVersion: 309, pythonLibrary: 'libpython3.9.so.1.0', members: [ {name: 'PYZ.pyz', typeCode: 'z', payload: pyz, compress: false}]}); const file = join(process.env.WORK, name + '.elf'); await writeFile(file, carrier); const reader = new PyInstallerArtifactReader(file); try { for await (const entry of reader.entries(undefined, { onCoverage: value => console.log(JSON.stringify({name, coverage: value})) })) console.log(JSON.stringify({name, path: entry.path, metadata: entry.formatMetadata, limitations: entry.limitations})); } finally { await reader.close(); } } JS ``` For the oracle, the retained wrapper separately ran `python3 -I -S -c` to call `marshal.loads()` on each saved `.marshal` file, catching and printing the exception. It did not load the carrier, import a module, or execute code. Each of the three table cases was observed once. An initial wrapper stopped after the integer control while looking for an unavailable published-reader module; the remaining two cases were then executed without repeating the control. The public `inspect_artifact` CLI/MCP adapters were not run. ### Expected behavior A marshal long with any digits must have a nonzero highest digit. Do not normalize an invalid serialized number into valid module metadata. This malformed table should follow the existing PYZ format-failure behavior: leave the PYZ opaque, report the parse limitation, and avoid projecting decoded child entries from it. Both canonical encodings must remain readable. CPython's marshal reader explicitly rejects this encoding, and REA already rejects non-normalized longs in its separate pyc parser. ### Impact and scope A corrupted PYZ table can be projected as successfully decoded metadata with complete traversal even though CPython cannot load that table. The demonstrated result invents a decoded namespace child from invalid marshal data. Real-world frequency and other long-field positions were not measured. No bytecode execution, security boundary, resource-exhaustion claim, or severe incident is involved. ### REA version Main source `6c4d880f2dd9ca7d42593be916f90ec19b8a09d7`, package metadata 6.7.0. Main-only finding: the PyInstaller reader is absent from the installed npm release, so no published-runtime reproduction is claimed. ### Runtime and platform Node.js v26.12.1 and CPython 3.14.8, Linux x64. The source check used the checkout's locked `tsx` loader with ambient Node rather than the repository-pinned Node toolchain. The executed Python oracle decoded primitive marshal table values, not 3.9 code objects; the same normalization rule for the declared 3.9 format is also established by CPython source. ### Target and artifact context Synthetic ELF carrier with one uncompressed CArchive member `PYZ.pyz`, a CPython 3.9 PYZ header, and one zero-payload namespace entry. Its table declares `[("pkg", (3, 12, 0))]`; only the encoding of 3 differs between cases. The invalid long's bytes are `6c 02 00 00 00 03 00 00 00`. ### Interface Other ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence ```json {"case":"normal-int","cpython":"loaded","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}} {"case":"normalized-long","cpython":"loaded","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}} {"case":"unnormalized-long","cpython":"ValueError: bad marshal data (unnormalized long data)","rea_pyz_status":"decoded","rea_pyz_entry_count":1,"rea_child":"PYZ.pyz/pkg","coverage":{"status":"complete","limitations":[]}} ``` - Expected source: [CPython's marshal implementation](https://github.com/python/cpython/blob/v3.9.0/Python/marshal.c), `r_PyLong` checks the topmost marshal digit and raises `bad marshal data (unnormalized long data)`; [REA's existing PYZ format-failure path](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/artifacts/pyinstaller/PyzArchive.ts#L132-L158); [REA pyc reader's normalization check](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/python/PythonPycParser.ts#L1295). - Failure source: The executed production reader and CPython outputs above; [the PYZ subset reader accumulates the digits and returns the number without checking the top digit](https://github.com/morluto/rea/blob/6c4d880f2dd9ca7d42593be916f90ec19b8a09d7/src/artifacts/pyinstaller/MarshalSubset.ts#L207-L232). - Evidence provenance: observed, with the declared-format oracle additionally grounded in CPython source. - Local verification: reproduced at the production reader boundary; no public adapter execution. - Reproduction completeness: complete. ### Additional context The existing negative-PYZ-marshal-size report (#6120) concerns signed lengths being misclassified as resource-limit failures and aborting sibling inventory. This report uses positive digit count 2 and valid small list/string lengths; it concerns successful decoding of an invalid normalized-number representation. These behaviors have independent failure signals and restoration checks. Restoration check (not executed): The invalid table must leave `PYZ.pyz` opaque with a normalization/format limitation and omit `PYZ.pyz/pkg`; both canonical table encodings must still project that child successfully. No implementation approach is prescribed. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed reader projects decoded child metadata and complete traversal from a table directly rejected by CPython, while two valid encoding controls agree. - Workaround status: unverified - Workaround basis: A caller could separately validate the primitive table with a compatible marshal reader, but no supported REA workflow that does so was observed; executing or importing the target is unnecessary and was not used.

bsakaguchi closed this issue

2026-10-31 00:45:32 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:52 +00:00

[PR #4162] [MERGED] fix(apple): count Interface Builder hierarchy records beyond the object limit #6101