morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6674] [Bug] PyInstaller data named PYZ.pyz aborts inventory of a valid onefile executable #4051

New issue

Closed

opened 2026-10-31 00:46:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:23 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6674

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production inventory behind inspect-artifact rejects a real PyInstaller 6.22.3 onefile executable built from pass plus a data file named PYZ.pyz. The producer stores that data as type x and its embedded Python archive as type z, both with the source name PYZ.pyz. REA returns ArtifactReaderFailure, reason integrity, message Duplicate CArchive member name at table index 58, and returns no inventory.

Reproduction

Executed with the runtime below; paths are parameterized. From the source checkout with locked dependencies installed, build the harmless fixture without running the resulting executable:

work=$(mktemp -d /tmp/rea-pyz.XXXXXX)
mkdir -p "$work/input"
printf 'pass\n' > "$work/input/main.py"
printf 'independent data\n' > "$work/input/PYZ.pyz"
UV_CACHE_DIR="$work/uv-cache" uv pip install --python python3 --target "$work/site" 'pyinstaller==6.22.3'
PYTHONPATH="$work/site" PYINSTALLER_CONFIG_DIR="$work/cache" python3 -m PyInstaller \
  --onefile --noconfirm --log-level ERROR --name demo \
  --distpath "$work/dist" --workpath "$work/build" --specpath "$work/input" \
  --add-data "$work/input/PYZ.pyz:." "$work/input/main.py"

Save this executed production-boundary probe as /tmp/rea-inventory.mts:

import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
const [repo, path] = process.argv.slice(2);
const { inventoryArtifact } = await import(
  pathToFileURL(join(repo, 'src/artifacts/inventory/ArtifactInventory.ts')).href);
const { ArtifactResourceScope } = await import(
  pathToFileURL(join(repo, 'src/artifacts/ArtifactResourceScope.ts')).href);
const resourceScope = new ArtifactResourceScope();
try {
  const result = await inventoryArtifact(path, { resourceScope, environment: {} });
  console.log(JSON.stringify({ ok: true, coverage: result.coverage,
    occurrences: result.occurrences.length }));
} catch (error) {
  console.log(JSON.stringify({ ok: false, name: error.name,
    reason: error.reason, message: error.message }));
} finally { await resourceScope.close(); }

node --import "$PWD/node_modules/tsx/dist/loader.mjs" /tmp/rea-inventory.mts "$PWD" "$work/dist/demo"

The producer build succeeded and the probe returned the failure above once. This uses the real inventory, format detection, reader factory and file reader, without injected mocks. CLI/MCP transport was not exercised. A separate 321-byte synthetic CArchive reproduced the same rejection at table index 1; an equally sized control changing only the data member name to resource.pyz returned complete coverage and retained the resource, PYZ namespace and healthy sibling.

Expected behavior

Inventory should preserve both source entries, their types and bytes, and unrelated members. A data entry and embedded PYZ sharing a stored name are not inherently a corrupt CArchive: the released PKG builder retains data names and assigns the embedded PYZ the fixed name PYZ.pyz; the writer's duplicate check is conditional on strict collection mode. The bootloader distinguishes entries by type when deciding what to extract. REA documents static CArchive inventory retaining source names, table indexes and types.

Impact and scope

This bounded compatibility failure blocks the entire artifact inventory, including unrelated members, for the observed valid onefile packaging shape. No executable was launched and no target bytes were changed. Observed frequency is one failure for the real producer output and one for the synthetic duplicate-name fixture; the renamed synthetic control succeeded. Other producer versions, platforms and real-world prevalence were not tested. Renaming the data file before rebuilding changes the artifact and is not an established workaround for inspecting an existing executable.

REA version

Source revision 5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26; package.json identifies 6.7.0. This is newer source than the latest published rea-agents@6.7.0; the published package was not tested.

Runtime and platform

Node.js v24.22.1; CPython 3.14.8; PyInstaller 6.22.3 with hooks-contrib 2026.9; Linux x86_64; locked checkout dependencies and the checkout's tsx loader. PyInstaller 6.22.3 was checked against current PyPI metadata before use and installed only in scratch space.

Target and artifact context

A real 9,935,080-byte Linux onefile executable built with the inline main.py and data fixture. Its CArchive cookie reports Python version 314; its embedded PYZ begins 50595a002b0e0d0a, matching the reader's supported Python 3.14 magic. A read-only raw TOC inspection observed two rows named PYZ.pyz: type x, compressed, decoding to 17 bytes independent data\n; and type z, stored, containing a 1,700,744-byte PYZ. Their payload ranges are separate.

Interface

Other

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Observed output from the production inventory call:

{"ok":false,"name":"ArtifactReaderFailure","reason":"integrity","message":"Duplicate CArchive member name at table index 58"}

Observed synthetic control output, abbreviated only to the relevant fields:

{"label":"duplicate","ok":false,"bytes":321,"reason":"integrity","message":"Duplicate CArchive member name at table index 1"}
{"label":"renamed-control","ok":true,"bytes":321,"coverage":{"scope":"container-entries","status":"complete"}}

Expected source: the released producer and REA inventory contract linked above, plus the successfully generated real executable and its raw TOC rows.
Failure source: the executed production inventory results above; CArchive validation rejects repeated names without considering their distinct source entries or types.
Evidence provenance: observed producer build, raw TOC inspection and production-boundary failure; supplied delegated synthetic failure/control output was inspected before drafting. Bootloader applicability and CLI/MCP reachability are source-derived.
Local verification: reproduced.
Reproduction completeness: complete.

The producer artifact SHA-256 was 02d248ec60fecb0b60483a62ea8f4e06b433c902b555ceafc8c110033224b0f0; reproducibility of the complete executable across environments was not tested. The artifact provider calls this production inventory; its reader factory selects the real PyInstaller reader.

Typed outcome or recovery expected

The unchanged producer artifact should yield inventory preserving both PYZ.pyz source identities, their types, readable data bytes and unrelated members. Keep the renamed control working. Merely deleting the duplicate-name check is insufficient if a later projected file/directory collision still discards either source entry or aborts the inventory. This is a proposed restoration check; no fix was implemented.

Analysis or suspected cause

Source-derived and consistent with the observed error: CArchive validation imposes source-name uniqueness across all nondeclaration types before projection. PyInstaller's public onefile build can produce distinct x and z entries with the same name under its default collection policy. The embedded archive is an analysis container rather than a data-file destination, so conflating those identities rejects this producer output.

Additional context

The open-issue snapshot and fresh all-state searches covered Duplicate CArchive member, PyInstaller/CArchive duplicates, and PYZ.pyz with duplicate or data. Closed #6539 concerns distinct raw source names colliding after PYZ namespace projection. This case fails earlier on identical raw names, and remains reproducible on the current source after that issue's correction. Existing #6441, #6559 and #6120 concern derivation notices or malformed marshal input, not this valid producer output. No bug reproduction was rerun after this draft was written.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The observed valid onefile executable cannot be inventoried because two different source types share a producer-assigned name; the failure hides every member. Scope is bounded to this packaging shape, with no target mutation or data loss observed.
Workaround status: unverified
Workaround basis: The unchanged carrier's data was recovered by a raw TOC probe, but no public REA workflow preserving the complete inventory was demonstrated; renaming and rebuilding alters the artifact.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6674 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The production inventory behind `inspect-artifact` rejects a real PyInstaller 6.22.3 onefile executable built from `pass` plus a data file named `PYZ.pyz`. The producer stores that data as type `x` and its embedded Python archive as type `z`, both with the source name `PYZ.pyz`. REA returns `ArtifactReaderFailure`, reason `integrity`, message `Duplicate CArchive member name at table index 58`, and returns no inventory. ## Reproduction Executed with the runtime below; paths are parameterized. From the source checkout with locked dependencies installed, build the harmless fixture without running the resulting executable: ```sh work=$(mktemp -d /tmp/rea-pyz.XXXXXX) mkdir -p "$work/input" printf 'pass\n' > "$work/input/main.py" printf 'independent data\n' > "$work/input/PYZ.pyz" UV_CACHE_DIR="$work/uv-cache" uv pip install --python python3 --target "$work/site" 'pyinstaller==6.22.3' PYTHONPATH="$work/site" PYINSTALLER_CONFIG_DIR="$work/cache" python3 -m PyInstaller \ --onefile --noconfirm --log-level ERROR --name demo \ --distpath "$work/dist" --workpath "$work/build" --specpath "$work/input" \ --add-data "$work/input/PYZ.pyz:." "$work/input/main.py" ``` Save this executed production-boundary probe as `/tmp/rea-inventory.mts`: ```ts import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; const [repo, path] = process.argv.slice(2); const { inventoryArtifact } = await import( pathToFileURL(join(repo, 'src/artifacts/inventory/ArtifactInventory.ts')).href); const { ArtifactResourceScope } = await import( pathToFileURL(join(repo, 'src/artifacts/ArtifactResourceScope.ts')).href); const resourceScope = new ArtifactResourceScope(); try { const result = await inventoryArtifact(path, { resourceScope, environment: {} }); console.log(JSON.stringify({ ok: true, coverage: result.coverage, occurrences: result.occurrences.length })); } catch (error) { console.log(JSON.stringify({ ok: false, name: error.name, reason: error.reason, message: error.message })); } finally { await resourceScope.close(); } ``` ```sh node --import "$PWD/node_modules/tsx/dist/loader.mjs" /tmp/rea-inventory.mts "$PWD" "$work/dist/demo" ``` The producer build succeeded and the probe returned the failure above once. This uses the real inventory, format detection, reader factory and file reader, without injected mocks. CLI/MCP transport was not exercised. A separate 321-byte synthetic CArchive reproduced the same rejection at table index 1; an equally sized control changing only the data member name to `resource.pyz` returned complete coverage and retained the resource, PYZ namespace and healthy sibling. ## Expected behavior Inventory should preserve both source entries, their types and bytes, and unrelated members. A data entry and embedded PYZ sharing a stored name are not inherently a corrupt CArchive: the released [PKG builder](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/PyInstaller/building/api.py#L324-L350) retains data names and assigns the embedded PYZ the fixed name `PYZ.pyz`; the [writer's duplicate check](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/PyInstaller/archive/writers.py#L198-L214) is conditional on strict collection mode. The [bootloader](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/bootloader/src/pyi_launch.c#L63-L87) distinguishes entries by type when deciding what to extract. REA documents [static CArchive inventory retaining source names, table indexes and types](https://github.com/morluto/rea/blob/5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26/docs/python-packaged-applications.md#L3-L19). ## Impact and scope This bounded compatibility failure blocks the entire artifact inventory, including unrelated members, for the observed valid onefile packaging shape. No executable was launched and no target bytes were changed. Observed frequency is one failure for the real producer output and one for the synthetic duplicate-name fixture; the renamed synthetic control succeeded. Other producer versions, platforms and real-world prevalence were not tested. Renaming the data file before rebuilding changes the artifact and is not an established workaround for inspecting an existing executable. ## REA version Source revision `5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26`; package.json identifies 6.7.0. This is newer source than the latest published `rea-agents@6.7.0`; the published package was not tested. ## Runtime and platform Node.js v24.22.1; CPython 3.14.8; PyInstaller 6.22.3 with hooks-contrib 2026.9; Linux x86_64; locked checkout dependencies and the checkout's tsx loader. PyInstaller 6.22.3 was checked against current PyPI metadata before use and installed only in scratch space. ## Target and artifact context A real 9,935,080-byte Linux onefile executable built with the inline `main.py` and data fixture. Its CArchive cookie reports Python version 314; its embedded PYZ begins `50595a002b0e0d0a`, matching the reader's supported Python 3.14 magic. A read-only raw TOC inspection observed two rows named `PYZ.pyz`: type `x`, compressed, decoding to 17 bytes `independent data\n`; and type `z`, stored, containing a 1,700,744-byte PYZ. Their payload ranges are separate. ## Interface Other ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Observed output from the production inventory call: ```json {"ok":false,"name":"ArtifactReaderFailure","reason":"integrity","message":"Duplicate CArchive member name at table index 58"} ``` Observed synthetic control output, abbreviated only to the relevant fields: ```json {"label":"duplicate","ok":false,"bytes":321,"reason":"integrity","message":"Duplicate CArchive member name at table index 1"} {"label":"renamed-control","ok":true,"bytes":321,"coverage":{"scope":"container-entries","status":"complete"}} ``` - Expected source: the released producer and REA inventory contract linked above, plus the successfully generated real executable and its raw TOC rows. - Failure source: the executed production inventory results above; [CArchive validation](https://github.com/morluto/rea/blob/5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26/src/artifacts/pyinstaller/CArchiveLayout.ts#L337-L345) rejects repeated names without considering their distinct source entries or types. - Evidence provenance: observed producer build, raw TOC inspection and production-boundary failure; supplied delegated synthetic failure/control output was inspected before drafting. Bootloader applicability and CLI/MCP reachability are source-derived. - Local verification: reproduced. - Reproduction completeness: complete. The producer artifact SHA-256 was `02d248ec60fecb0b60483a62ea8f4e06b433c902b555ceafc8c110033224b0f0`; reproducibility of the complete executable across environments was not tested. The [artifact provider](https://github.com/morluto/rea/blob/5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26/src/artifacts/ArtifactProvider.ts#L475-L489) calls this production inventory; its [reader factory](https://github.com/morluto/rea/blob/5a5bcb6b549cfdfba6dd3f02dd681e505ffd2a26/src/artifacts/inventory/reader.ts#L40-L44) selects the real PyInstaller reader. ## Typed outcome or recovery expected The unchanged producer artifact should yield inventory preserving both `PYZ.pyz` source identities, their types, readable data bytes and unrelated members. Keep the renamed control working. Merely deleting the duplicate-name check is insufficient if a later projected file/directory collision still discards either source entry or aborts the inventory. This is a proposed restoration check; no fix was implemented. ## Analysis or suspected cause Source-derived and consistent with the observed error: CArchive validation imposes source-name uniqueness across all nondeclaration types before projection. PyInstaller's public onefile build can produce distinct `x` and `z` entries with the same name under its default collection policy. The embedded archive is an analysis container rather than a data-file destination, so conflating those identities rejects this producer output. ## Additional context The open-issue snapshot and fresh all-state searches covered `Duplicate CArchive member`, PyInstaller/CArchive duplicates, and `PYZ.pyz` with `duplicate` or `data`. Closed [#6539](https://github.com/morluto/rea/issues/6539) concerns distinct raw source names colliding after PYZ namespace projection. This case fails earlier on identical raw names, and remains reproducible on the current source after that issue's correction. Existing #6441, #6559 and #6120 concern derivation notices or malformed marshal input, not this valid producer output. No bug reproduction was rerun after this draft was written. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The observed valid onefile executable cannot be inventoried because two different source types share a producer-assigned name; the failure hides every member. Scope is bounded to this packaging shape, with no target mutation or data loss observed. - Workaround status: unverified - Workaround basis: The unchanged carrier's data was recovered by a raw TOC probe, but no public REA workflow preserving the complete inventory was demonstrated; renaming and rebuilding alters the artifact.

bsakaguchi closed this issue

2026-10-31 00:46:23 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:14 +00:00

[PR #4051] [MERGED] fix(native): report an object file's unnamed segment by its empty name #6021