morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6539] [Bug] PYZ namespace projection makes distinct CArchive members collide #3929

New issue

Closed

opened 2026-10-31 00:45:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:45:19 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6539

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-artifact and extract-artifact fail with Artifact path collision: "PYZ.pyz/ns" for a CArchive containing a PYZ member named PYZ.pyz with namespace package ns, plus an independent data member named PYZ.pyz/ns. The original CArchive names are distinct. REA creates the conflicting directory path while projecting the PYZ namespace.

The real PyInstallerArtifactReader accepts the input, preserves both source identities, and reads the data member and an unrelated sibling correctly. The shared inventory then aborts and neither CLI operation returns Evidence. Changing only the data member's stored name to data/ns allows both operations to succeed.

Reproduction

The initial probe used the checkout's existing fixture constructors, the real reader, the production inventory boundary and fresh source CLI processes. This excerpt preserves the executed fixture inputs and inventory calls, with neutral scratch paths. Start with the stated source checkout, locked dependencies and pinned Node; save this as /tmp/pyz-inventory-probe.mts:

import { mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
const [repo, work] = process.argv.slice(2);
const load = (p: string) => import(pathToFileURL(join(repo, p)).href);
const { cArchiveExecutable, pyzArchive, PYTHON_39_MAGIC } =
  await load('src/artifacts/pyinstaller/CArchive.fixture.ts');
const { inventoryArtifact } =
  await load('src/artifacts/inventory/ArtifactInventory.ts');
const { ArtifactResourceScope } =
  await load('src/artifacts/ArtifactResourceScope.ts');
await mkdir(work, { recursive: true });
const pyz = pyzArchive({
  magic: PYTHON_39_MAGIC,
  modules: [{ name: 'ns', typeCode: 3, body: Buffer.alloc(0) }]
});
for (const [label, name] of [
  ['collision', 'PYZ.pyz/ns'], ['control', 'data/ns']
]) {
  const path = join(work, label);
  await writeFile(path, cArchiveExecutable({
    layout: '2.1', pythonVersion: 309,
    pythonLibrary: 'libpython3.9.so.1.0',
    members: [
      { name: 'PYZ.pyz', typeCode: 'z', payload: pyz, compress: false },
      { name, typeCode: 'x', payload: Buffer.from('independent data\n'),
        compress: false },
      { name: 'sibling.txt', typeCode: 'x',
        payload: Buffer.from('retained sibling\n'), compress: false }
    ]
  }));
  const resourceScope = new ArtifactResourceScope();
  try {
    const result = await inventoryArtifact(path, { resourceScope, environment: {} });
    console.log(label, result.coverage, result.occurrences.length);
  } catch (error) {
    console.log(label, error.reason, error.message);
  } finally {
    await resourceScope.close();
  }
}

Invoke using node --import "$PWD/node_modules/tsx/dist/loader.mjs" /tmp/pyz-inventory-probe.mts "$PWD" /tmp/pyz-fixtures. The collision case raises the path failure; the control returns complete container-entry coverage with five occurrences.

The initial source CLI calls used createCli(isolatedEnv).serve([operation, path, "--format", "json"]) in a fresh Node subprocess for each of inspect-artifact and extract-artifact against each fixture. The CLI runner dynamically imported src/cli.ts through the same locked tsx loader. Those four calls produced the results below. The initial producer-boundary probe additionally enumerated and opened real reader entries; no native provider or packaged application was run.

Expected behavior

Inventory should preserve the independent CArchive member and PYZ namespace without treating the synthetic namespace path as a conflicting source filename. Both byte-bearing CArchive members should remain available for analysis and extraction, with their original source names, container identities and bytes.

REA documents static CArchive/PYZ inventory with retained source names and namespace directories. A PYZ namespace is a valid empty table entry in the published PyInstaller writer. PyInstaller's bootloader extraction switch extracts data entries and skips the embedded PYZ type, so the embedded member PYZ.pyz does not require an output file that would conflict with the data directory. These are source-derived format/applicability facts; a real PyInstaller application was not built or executed.

Impact and scope

The collision aborts the entire inventory, hiding an unrelated readable sibling as well as the conflicting entries from public Evidence. Extraction also fails before materialization. This is a bounded failure for otherwise admitted archives whose outer data names overlap REA's synthetic PYZ paths. The single namespace/data fixture reproduced once through production inventory and once per public CLI operation; the renamed control succeeded. Real-world frequency, other overlapping shapes and other platforms remain untested.

Renaming the source data member works in the synthetic control but changes the artifact and may change application behavior; it is not an established workaround for an existing executable.

REA version

Source revision e073f64be0a5a30c32d8b3cec23f9bd411080903 on main; package version 6.7.0. Production source was read-only and loaded without a build. The published npm artifact was not tested.

Runtime and platform

Node.js v24.18.0; Linux x86_64; locked checkout dependencies and tsx; Python 3.14.8 for the CLI subprocess harness. The source CLI subprocesses used isolated HOME, XDG and temporary directories. No native analysis provider, target execution, encryption or external service was involved.

Target and artifact context

Synthetic 322-byte ELF-header carrier with a layout-2.1 CArchive and Python-3.9 PYZ header. Outer members are PYZ.pyz (z), PYZ.pyz/ns (x), and sibling.txt (x). The PYZ has one zero-payload namespace ns (type_code: 3). Outer member names and payload ranges are distinct. The equally sized control changes only the outer data name to data/ns.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Observed real-reader projection for the collision input:

path          kind       metadata.container  metadata.source_name
PYZ.pyz       directory  carchive            PYZ.pyz
PYZ.pyz/ns    directory  pyz                 ns
PYZ.pyz/ns    file       carchive            PYZ.pyz/ns
sibling.txt   file       carchive            sibling.txt

Reading the two regular entries yielded exactly independent data\n and retained sibling\n. The subsequent production inventory threw ArtifactReaderFailure, reason: "path", message: 'Artifact path collision: "PYZ.pyz/ns"'. The control returned five occurrences and coverage: {scope: "container-entries", status: "complete"}.

Observed fresh-process source CLI results; every stderr was empty:

collision  inspect-artifact  exit=1  artifact_operation_failed / execution_failure
collision  extract-artifact  exit=1  artifact_operation_failed / execution_failure
control    inspect-artifact  exit=0  Evidence returned
control    extract-artifact  exit=0  Evidence returned

Both failures carry details.reason: "path" and details.detail: 'Artifact path collision: "PYZ.pyz/ns"'. Fixture SHA-256 values:

collision: 15ac716bac9629112df3cbe1c65563922ac51f5ce5f7ff63ea0df734f5d83bcd
control:   04eba007a571afb34673481290bfedfb3d409ee4921333beceea461ddf2180a8

Evidence provenance: observed production-boundary and source CLI results, with source-derived format applicability as cited above. Local verification: reproduced. Reproduction completeness: complete for the synthetic failure. No real producer-generated executable, MCP transport run, repair or post-draft rerun is claimed.

Typed outcome or recovery expected

Complete inventory should retain both independent source identities and the readable sibling; extracting the two outer data members should retain their exact bytes. This fixture's empty namespace has no bytecode file to materialize. A restoration check should use the unchanged collision fixture, preserve source names and declaring-container facts, and keep the renamed control working. The representation and handling of genuine output-path collisions remain implementation choices; suppressing one source row would not restore inventory preservation. This check is proposed and was not run against a fix.

Analysis or suspected cause

Source-derived: pyzMemberPath prefixes inner names with the outer member's filename. Namespace projection marks that synthetic path as a directory. The shared scanner registers it alongside ordinary CArchive file paths, and ArtifactPathRegistry rejects the resulting file/directory collision. The reader's distinct adapter keys and format metadata survive until this boundary; the observed failure is not duplicate CArchive-name validation.

Additional context

GitHub searches repo:morluto/rea is:issue "PyInstaller" "path collision" and repo:morluto/rea is:issue "PYZ.pyz" "collision" returned no matches when separately filtered to open and closed issues. Broader source research compared #6312, which concerns diagnostics for malformed compression flags, and closed #6067, which concerns changed-source admission. Their failure conditions and boundaries differ. Existing #6441 (bytecode derivation notices), #6484 (member-type comparison), and #6485 (declaration comparison) do not describe namespace projection collisions.

No bug reproduction was rerun after writing this draft.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A supported empty PYZ namespace and a distinct readable outer data member are projected to the same logical destination, causing complete inventory and extraction failure and hiding an unrelated sibling. The observed scope is one synthetic input shape.
Workaround status: unverified
Workaround basis: Renaming the outer member made the synthetic control pass, but changes the original artifact and can affect application behavior; no safe public workaround for an unchanged executable was demonstrated.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6539 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-artifact` and `extract-artifact` fail with `Artifact path collision: "PYZ.pyz/ns"` for a CArchive containing a PYZ member named `PYZ.pyz` with namespace package `ns`, plus an independent data member named `PYZ.pyz/ns`. The original CArchive names are distinct. REA creates the conflicting directory path while projecting the PYZ namespace. The real `PyInstallerArtifactReader` accepts the input, preserves both source identities, and reads the data member and an unrelated sibling correctly. The shared inventory then aborts and neither CLI operation returns Evidence. Changing only the data member's stored name to `data/ns` allows both operations to succeed. ## Reproduction The initial probe used the checkout's existing fixture constructors, the real reader, the production inventory boundary and fresh source CLI processes. This excerpt preserves the executed fixture inputs and inventory calls, with neutral scratch paths. Start with the stated source checkout, locked dependencies and pinned Node; save this as `/tmp/pyz-inventory-probe.mts`: ```ts import { mkdir, writeFile } from 'node:fs/promises'; import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; const [repo, work] = process.argv.slice(2); const load = (p: string) => import(pathToFileURL(join(repo, p)).href); const { cArchiveExecutable, pyzArchive, PYTHON_39_MAGIC } = await load('src/artifacts/pyinstaller/CArchive.fixture.ts'); const { inventoryArtifact } = await load('src/artifacts/inventory/ArtifactInventory.ts'); const { ArtifactResourceScope } = await load('src/artifacts/ArtifactResourceScope.ts'); await mkdir(work, { recursive: true }); const pyz = pyzArchive({ magic: PYTHON_39_MAGIC, modules: [{ name: 'ns', typeCode: 3, body: Buffer.alloc(0) }] }); for (const [label, name] of [ ['collision', 'PYZ.pyz/ns'], ['control', 'data/ns'] ]) { const path = join(work, label); await writeFile(path, cArchiveExecutable({ layout: '2.1', pythonVersion: 309, pythonLibrary: 'libpython3.9.so.1.0', members: [ { name: 'PYZ.pyz', typeCode: 'z', payload: pyz, compress: false }, { name, typeCode: 'x', payload: Buffer.from('independent data\n'), compress: false }, { name: 'sibling.txt', typeCode: 'x', payload: Buffer.from('retained sibling\n'), compress: false } ] })); const resourceScope = new ArtifactResourceScope(); try { const result = await inventoryArtifact(path, { resourceScope, environment: {} }); console.log(label, result.coverage, result.occurrences.length); } catch (error) { console.log(label, error.reason, error.message); } finally { await resourceScope.close(); } } ``` Invoke using `node --import "$PWD/node_modules/tsx/dist/loader.mjs" /tmp/pyz-inventory-probe.mts "$PWD" /tmp/pyz-fixtures`. The collision case raises the path failure; the control returns complete container-entry coverage with five occurrences. The initial source CLI calls used `createCli(isolatedEnv).serve([operation, path, "--format", "json"])` in a fresh Node subprocess for each of `inspect-artifact` and `extract-artifact` against each fixture. The CLI runner dynamically imported `src/cli.ts` through the same locked `tsx` loader. Those four calls produced the results below. The initial producer-boundary probe additionally enumerated and opened real reader entries; no native provider or packaged application was run. ## Expected behavior Inventory should preserve the independent CArchive member and PYZ namespace without treating the synthetic namespace path as a conflicting source filename. Both byte-bearing CArchive members should remain available for analysis and extraction, with their original source names, container identities and bytes. REA documents [static CArchive/PYZ inventory with retained source names and namespace directories](https://github.com/morluto/rea/blob/e073f64be0a5a30c32d8b3cec23f9bd411080903/docs/python-packaged-applications.md#L3-L25). A PYZ namespace is a valid empty table entry in the published [PyInstaller writer](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/PyInstaller/archive/writers.py#L69-L76). PyInstaller's [bootloader extraction switch](https://github.com/pyinstaller/pyinstaller/blob/v6.22.3/bootloader/src/pyi_launch.c#L63-L87) extracts data entries and skips the embedded PYZ type, so the embedded member `PYZ.pyz` does not require an output file that would conflict with the data directory. These are source-derived format/applicability facts; a real PyInstaller application was not built or executed. ## Impact and scope The collision aborts the entire inventory, hiding an unrelated readable sibling as well as the conflicting entries from public Evidence. Extraction also fails before materialization. This is a bounded failure for otherwise admitted archives whose outer data names overlap REA's synthetic PYZ paths. The single namespace/data fixture reproduced once through production inventory and once per public CLI operation; the renamed control succeeded. Real-world frequency, other overlapping shapes and other platforms remain untested. Renaming the source data member works in the synthetic control but changes the artifact and may change application behavior; it is not an established workaround for an existing executable. ## REA version Source revision `e073f64be0a5a30c32d8b3cec23f9bd411080903` on `main`; package version `6.7.0`. Production source was read-only and loaded without a build. The published npm artifact was not tested. ## Runtime and platform Node.js `v24.18.0`; Linux x86_64; locked checkout dependencies and `tsx`; Python `3.14.8` for the CLI subprocess harness. The source CLI subprocesses used isolated HOME, XDG and temporary directories. No native analysis provider, target execution, encryption or external service was involved. ## Target and artifact context Synthetic 322-byte ELF-header carrier with a layout-2.1 CArchive and Python-3.9 PYZ header. Outer members are `PYZ.pyz` (`z`), `PYZ.pyz/ns` (`x`), and `sibling.txt` (`x`). The PYZ has one zero-payload namespace `ns` (`type_code: 3`). Outer member names and payload ranges are distinct. The equally sized control changes only the outer data name to `data/ns`. ## Interface CLI ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Observed real-reader projection for the collision input: ```text path kind metadata.container metadata.source_name PYZ.pyz directory carchive PYZ.pyz PYZ.pyz/ns directory pyz ns PYZ.pyz/ns file carchive PYZ.pyz/ns sibling.txt file carchive sibling.txt ``` Reading the two regular entries yielded exactly `independent data\n` and `retained sibling\n`. The subsequent production inventory threw `ArtifactReaderFailure`, `reason: "path"`, `message: 'Artifact path collision: "PYZ.pyz/ns"'`. The control returned five occurrences and `coverage: {scope: "container-entries", status: "complete"}`. Observed fresh-process source CLI results; every stderr was empty: ```text collision inspect-artifact exit=1 artifact_operation_failed / execution_failure collision extract-artifact exit=1 artifact_operation_failed / execution_failure control inspect-artifact exit=0 Evidence returned control extract-artifact exit=0 Evidence returned ``` Both failures carry `details.reason: "path"` and `details.detail: 'Artifact path collision: "PYZ.pyz/ns"'`. Fixture SHA-256 values: ```text collision: 15ac716bac9629112df3cbe1c65563922ac51f5ce5f7ff63ea0df734f5d83bcd control: 04eba007a571afb34673481290bfedfb3d409ee4921333beceea461ddf2180a8 ``` Evidence provenance: observed production-boundary and source CLI results, with source-derived format applicability as cited above. Local verification: reproduced. Reproduction completeness: complete for the synthetic failure. No real producer-generated executable, MCP transport run, repair or post-draft rerun is claimed. ## Typed outcome or recovery expected Complete inventory should retain both independent source identities and the readable sibling; extracting the two outer data members should retain their exact bytes. This fixture's empty namespace has no bytecode file to materialize. A restoration check should use the unchanged collision fixture, preserve source names and declaring-container facts, and keep the renamed control working. The representation and handling of genuine output-path collisions remain implementation choices; suppressing one source row would not restore inventory preservation. This check is proposed and was not run against a fix. ## Analysis or suspected cause Source-derived: [`pyzMemberPath`](https://github.com/morluto/rea/blob/e073f64be0a5a30c32d8b3cec23f9bd411080903/src/artifacts/pyinstaller/CArchiveMembers.ts#L299-L305) prefixes inner names with the outer member's filename. [Namespace projection](https://github.com/morluto/rea/blob/e073f64be0a5a30c32d8b3cec23f9bd411080903/src/artifacts/pyinstaller/CArchiveMembers.ts#L208-L238) marks that synthetic path as a directory. The shared scanner registers it alongside ordinary CArchive file paths, and [`ArtifactPathRegistry`](https://github.com/morluto/rea/blob/e073f64be0a5a30c32d8b3cec23f9bd411080903/src/artifacts/ArtifactPaths.ts#L90-L101) rejects the resulting file/directory collision. The reader's distinct adapter keys and format metadata survive until this boundary; the observed failure is not duplicate CArchive-name validation. ## Additional context GitHub searches `repo:morluto/rea is:issue "PyInstaller" "path collision"` and `repo:morluto/rea is:issue "PYZ.pyz" "collision"` returned no matches when separately filtered to open and closed issues. Broader source research compared [#6312](https://github.com/morluto/rea/issues/6312), which concerns diagnostics for malformed compression flags, and closed [#6067](https://github.com/morluto/rea/issues/6067), which concerns changed-source admission. Their failure conditions and boundaries differ. Existing #6441 (bytecode derivation notices), #6484 (member-type comparison), and #6485 (declaration comparison) do not describe namespace projection collisions. No bug reproduction was rerun after writing this draft. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A supported empty PYZ namespace and a distinct readable outer data member are projected to the same logical destination, causing complete inventory and extraction failure and hiding an unrelated sibling. The observed scope is one synthetic input shape. - Workaround status: unverified - Workaround basis: Renaming the outer member made the synthetic control pass, but changes the original artifact and can affect application behavior; no safe public workaround for an unchanged executable was demonstrated.

bsakaguchi closed this issue

2026-10-31 00:45:19 +00:00

bsakaguchi referenced this issue

2026-11-01 21:44:33 +00:00

[PR #5605] [MERGED] fix(browser): honor long passive observation windows #6993