morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6789] [Bug] PyInstaller extraction prepends a second pyc header to legacy modules #4142

New issue

Closed

opened 2026-10-31 00:47:07 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:07 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6789

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Open and closed PyInstaller issues were searched before filing.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production PyInstallerArtifactReader prepends a second 16-byte .pyc header to CArchive m/M members that already contain one, when a sibling PYZ magic agrees with the cookie. The 89-byte legacy-layout fixture becomes 105 bytes, with Python magic 6f0d0d0a at both offsets 0 and 16. REA's production pyc parser accepts the original and rejects the returned bytes with Unsupported marshal type 0x6f.

Observed directly through the production reader's enumeration and member stream, using inert local fixtures. Stored and zlib-compressed m, and compressed M, reproduce the failure. Headerless modern modules and scripts parse successfully; the legacy member without a PYZ sibling is preserved and parses successfully. The larger initial probe also observed complete traversal coverage for the failing archive. CLI/MCP adapters were not run.

Reproduction

From this source checkout with locked dependencies installed, create a scratch directory outside the checkout, save the following as probe.mts there, and run node node_modules/tsx/dist/cli.mjs /absolute/scratch/probe.mts /absolute/scratch. The command below is the final executed probe, with scratch paths parameterized. It writes six small synthetic CArchives, invokes the actual member reader and pyc parser, and asserts each returned byte sequence. It does not build or launch an executable or execute Python bytecode.

The Python 3.10 code body comes from the repository's existing inert format fixture. It is not a compiler-produced application or a claim that a Python interpreter would execute it successfully. The legacy full-header layout is independently grounded in the upstream writer below.

import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { writeFile } from 'node:fs/promises';
import { buffer } from 'node:stream/consumers';
import { deflateSync } from 'node:zlib';
import { pathToFileURL } from 'node:url';
const load = (p: string) => import(pathToFileURL(`${process.cwd()}/${p}`).href);
const { PyInstallerArtifactReader } = await load('src/artifacts/PyInstallerArtifactReader.ts');
const { parsePythonPyc } = await load('src/python/PythonPycParser.ts');
const { inspectPythonPycInputSchema } = await load('src/domain/python/pythonPyc.ts');
const { pythonPycHeader, marshalCode } = await load('src/python/PythonPyc.fixture.ts');
const original = Buffer.concat([pythonPycHeader(3439), marshalCode('py38')]);
const pyz = Buffer.concat([Buffer.from('PYZ\0'), original.subarray(0,4), Buffer.from('0000000c5b00000000','hex')]);
const parse = (bytes: Buffer) => {
  try { return parsePythonPyc(bytes,{path:'/fixtures/module.pyc',bytes:bytes.length,sha256:createHash('sha256').update(bytes).digest('hex')},inspectPythonPycInputSchema.parse({path:'/fixtures/module.pyc'})).coverage.status; }
  catch (e: any) { return e.message; }
};
assert.equal(parse(original),'complete');
for (const [name,type,legacy,compressed,withPyz] of [
  ['legacy-m-stored','m',true,false,true], ['legacy-m-zlib','m',true,true,true],
  ['legacy-M-zlib','M',true,true,true], ['modern-m-zlib','m',false,true,true],
  ['script-zlib','s',false,true,true], ['legacy-no-PYZ','m',true,true,false],
] as const) {
  const payload=legacy?original:original.subarray(16);
  const rows=[{name:'module',type,body:payload,compressed},...(withPyz?[{name:'PYZ.pyz',type:'z',body:pyz,compressed:false}]:[])];
  let data=Buffer.alloc(0),toc=Buffer.alloc(0);
  for(const row of rows){
    const stored=row.compressed?deflateSync(row.body):row.body;
    const record=Buffer.alloc(Math.ceil((19+row.name.length)/16)*16);
    [record.length,data.length,stored.length,row.body.length].forEach((v,i)=>record.writeUInt32BE(v,i*4));
    record[16]=Number(row.compressed);record[17]=row.type.charCodeAt(0);record.write(row.name,18);
    toc=Buffer.concat([toc,record]);data=Buffer.concat([data,stored]);
  }
  const cookie=Buffer.alloc(88);Buffer.from('4d45490c0b0a0b0e','hex').copy(cookie);
  [data.length+toc.length+88,data.length,toc.length,310].forEach((v,i)=>cookie.writeUInt32BE(v,8+i*4));
  const path=`${process.argv[2]}/${name}.pkg`;await writeFile(path,Buffer.concat([data,toc,cookie]));
  const reader=new PyInstallerArtifactReader(path);
  try {
    const entries=[];for await(const e of reader.entries())entries.push(e);
    const out=await buffer(await reader.open(entries.find((e:any)=>e.path==='module')!));
    assert.equal(out.subarray(legacy&&withPyz?16:0).equals(original),true);
    console.log(JSON.stringify({name,source_bytes:payload.length,output_bytes:out.length,offset_16:out.subarray(16,20).toString('hex'),parse:parse(out)}));
  } finally {await reader.close();}
}

The final probe completed once with exit 0; all assertions passed. Earlier broader controls produced the same behavior. No reproduction was rerun after drafting this report.

Expected behavior

Preserve an existing valid .pyc envelope in legacy m/M members rather than inserting a second header. Continue reconstructing a header for genuinely headerless members.

This is a real producer distinction, not an arbitrary filename convention: PyInstaller 4.10's writer retains the existing header when storing a module; its later package classification changes m to M. By contrast, PyInstaller 5.3's writer explicitly removes it. Applicability to producer-generated legacy archives is source-derived; a whole legacy application was not built or run.

Impact and scope

P2: a bounded legacy archive layout yields changed module bytes that cannot be consumed even by REA's own pyc parser, while the reader completes normally. Source files remain unchanged. The synthetic probe establishes the byte transformation for stored and compressed legacy modules and packages. Real-application frequency, first affected REA version, and public CLI/MCP projection are NOT_ENOUGH_INFO: they were not measured.

A verified manual workaround for a confirmed duplicated-header output is to remove the first 16 bytes: the probe asserts that the remainder equals the original member, which parses successfully. This should only be applied after confirming an original header is present.

REA version

rea-agents 6.7.0; source revision a7180370332350bc0f998fbd643888221759570e.

Runtime and platform

Node.js v24.22.1, Linux x86_64; locked dependencies and repository tsx loader. No Python interpreter or external analysis engine was invoked by the reproduction.

Target and artifact context

Synthetic PyInstaller 2.1 CArchives with cookie Python version 310. Matching PYZ uses final Python 3.10 magic 6f0d0d0a and an empty valid marshal list directory. Legacy m/M bodies contain a complete 16-byte-header pyc; modern m and s controls contain its 73-byte marshal body. These are archive bytes only, with no executable bootloader.

Interface

Other

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Exact output from the final executed probe:

{"name":"legacy-m-stored","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."}
{"name":"legacy-m-zlib","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."}
{"name":"legacy-M-zlib","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."}
{"name":"modern-m-zlib","source_bytes":73,"output_bytes":89,"offset_16":"63000000","parse":"complete"}
{"name":"script-zlib","source_bytes":73,"output_bytes":89,"offset_16":"63000000","parse":"complete"}
{"name":"legacy-no-PYZ","source_bytes":89,"output_bytes":89,"offset_16":"63000000","parse":"complete"}

The first three outputs contain the original Python magic where the marshal code object must begin. Every assertion checks the complete returned byte sequence, not just these four bytes.

Typed outcome or recovery expected

Return a single coherent pyc envelope for supported module storage layouts. If the reader cannot establish whether header derivation is appropriate, preserve the original bytes with an explicit limitation rather than silently creating an invalid derived file.

Analysis or suspected cause

Source-derived: projectStoredMember chooses a reconstructed header for all bytecode type codes whenever a sibling magic is available. scriptPythonMagic establishes version agreement, but not whether this particular member already contains a header. pyInstallerMemberStream then prefixes those bytes without inspecting or replacing the existing envelope.

The documented derivation behavior does not distinguish the two producer storage layouts.

Additional context

Open #6441 concerns omitted derivation notices and misleading extraction descriptions; it explicitly observed correct output bytes. This report concerns incorrect returned module bytes. Closed #6736 concerns path normalization, and #3678 is the broader capability proposal. No matching report was found in the supplied open inventory or subsequent open and closed PyInstaller searches. No patch or known-good REA release was tested.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Three inert legacy-layout cases return duplicated headers and fail the production pyc parser despite successful member reading; modern and no-PYZ controls preserve a parseable envelope. Source files are unchanged and the failure is limited to a particular archive layout.
Workaround status: available
Workaround basis: For confirmed double-header output, removing the first 16 bytes restores the exact original member; the executed assertions establish byte equality and the original parser control succeeds.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6789 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. Open and closed PyInstaller issues were searched before filing. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The production `PyInstallerArtifactReader` prepends a second 16-byte `.pyc` header to CArchive `m`/`M` members that already contain one, when a sibling PYZ magic agrees with the cookie. The 89-byte legacy-layout fixture becomes 105 bytes, with Python magic `6f0d0d0a` at both offsets 0 and 16. REA's production pyc parser accepts the original and rejects the returned bytes with `Unsupported marshal type 0x6f.` Observed directly through the production reader's enumeration and member stream, using inert local fixtures. Stored and zlib-compressed `m`, and compressed `M`, reproduce the failure. Headerless modern modules and scripts parse successfully; the legacy member without a PYZ sibling is preserved and parses successfully. The larger initial probe also observed complete traversal coverage for the failing archive. CLI/MCP adapters were not run. ## Reproduction From this source checkout with locked dependencies installed, create a scratch directory outside the checkout, save the following as `probe.mts` there, and run `node node_modules/tsx/dist/cli.mjs /absolute/scratch/probe.mts /absolute/scratch`. The command below is the final executed probe, with scratch paths parameterized. It writes six small synthetic CArchives, invokes the actual member reader and pyc parser, and asserts each returned byte sequence. It does not build or launch an executable or execute Python bytecode. The Python 3.10 code body comes from the repository's existing inert format fixture. It is not a compiler-produced application or a claim that a Python interpreter would execute it successfully. The legacy full-header layout is independently grounded in the upstream writer below. ```ts import assert from 'node:assert/strict'; import { createHash } from 'node:crypto'; import { writeFile } from 'node:fs/promises'; import { buffer } from 'node:stream/consumers'; import { deflateSync } from 'node:zlib'; import { pathToFileURL } from 'node:url'; const load = (p: string) => import(pathToFileURL(`${process.cwd()}/${p}`).href); const { PyInstallerArtifactReader } = await load('src/artifacts/PyInstallerArtifactReader.ts'); const { parsePythonPyc } = await load('src/python/PythonPycParser.ts'); const { inspectPythonPycInputSchema } = await load('src/domain/python/pythonPyc.ts'); const { pythonPycHeader, marshalCode } = await load('src/python/PythonPyc.fixture.ts'); const original = Buffer.concat([pythonPycHeader(3439), marshalCode('py38')]); const pyz = Buffer.concat([Buffer.from('PYZ\0'), original.subarray(0,4), Buffer.from('0000000c5b00000000','hex')]); const parse = (bytes: Buffer) => { try { return parsePythonPyc(bytes,{path:'/fixtures/module.pyc',bytes:bytes.length,sha256:createHash('sha256').update(bytes).digest('hex')},inspectPythonPycInputSchema.parse({path:'/fixtures/module.pyc'})).coverage.status; } catch (e: any) { return e.message; } }; assert.equal(parse(original),'complete'); for (const [name,type,legacy,compressed,withPyz] of [ ['legacy-m-stored','m',true,false,true], ['legacy-m-zlib','m',true,true,true], ['legacy-M-zlib','M',true,true,true], ['modern-m-zlib','m',false,true,true], ['script-zlib','s',false,true,true], ['legacy-no-PYZ','m',true,true,false], ] as const) { const payload=legacy?original:original.subarray(16); const rows=[{name:'module',type,body:payload,compressed},...(withPyz?[{name:'PYZ.pyz',type:'z',body:pyz,compressed:false}]:[])]; let data=Buffer.alloc(0),toc=Buffer.alloc(0); for(const row of rows){ const stored=row.compressed?deflateSync(row.body):row.body; const record=Buffer.alloc(Math.ceil((19+row.name.length)/16)*16); [record.length,data.length,stored.length,row.body.length].forEach((v,i)=>record.writeUInt32BE(v,i*4)); record[16]=Number(row.compressed);record[17]=row.type.charCodeAt(0);record.write(row.name,18); toc=Buffer.concat([toc,record]);data=Buffer.concat([data,stored]); } const cookie=Buffer.alloc(88);Buffer.from('4d45490c0b0a0b0e','hex').copy(cookie); [data.length+toc.length+88,data.length,toc.length,310].forEach((v,i)=>cookie.writeUInt32BE(v,8+i*4)); const path=`${process.argv[2]}/${name}.pkg`;await writeFile(path,Buffer.concat([data,toc,cookie])); const reader=new PyInstallerArtifactReader(path); try { const entries=[];for await(const e of reader.entries())entries.push(e); const out=await buffer(await reader.open(entries.find((e:any)=>e.path==='module')!)); assert.equal(out.subarray(legacy&&withPyz?16:0).equals(original),true); console.log(JSON.stringify({name,source_bytes:payload.length,output_bytes:out.length,offset_16:out.subarray(16,20).toString('hex'),parse:parse(out)})); } finally {await reader.close();} } ``` The final probe completed once with exit 0; all assertions passed. Earlier broader controls produced the same behavior. No reproduction was rerun after drafting this report. ## Expected behavior Preserve an existing valid `.pyc` envelope in legacy `m`/`M` members rather than inserting a second header. Continue reconstructing a header for genuinely headerless members. This is a real producer distinction, not an arbitrary filename convention: [PyInstaller 4.10's writer](https://github.com/pyinstaller/pyinstaller/blob/669313ba4c5c1403ebeb335c35cb68c8c6ba5dd4/PyInstaller/archive/writers.py#L332-L355) retains the existing header when storing a module; its later package classification changes `m` to `M`. By contrast, [PyInstaller 5.3's writer](https://github.com/pyinstaller/pyinstaller/blob/fbf7948be85177dd44b41217e9f039e1d176de6b/PyInstaller/archive/writers.py#L324-L334) explicitly removes it. Applicability to producer-generated legacy archives is source-derived; a whole legacy application was not built or run. ## Impact and scope P2: a bounded legacy archive layout yields changed module bytes that cannot be consumed even by REA's own pyc parser, while the reader completes normally. Source files remain unchanged. The synthetic probe establishes the byte transformation for stored and compressed legacy modules and packages. Real-application frequency, first affected REA version, and public CLI/MCP projection are `NOT_ENOUGH_INFO`: they were not measured. A verified manual workaround for a confirmed duplicated-header output is to remove the first 16 bytes: the probe asserts that the remainder equals the original member, which parses successfully. This should only be applied after confirming an original header is present. ## REA version `rea-agents` 6.7.0; source revision `a7180370332350bc0f998fbd643888221759570e`. ## Runtime and platform Node.js v24.22.1, Linux x86_64; locked dependencies and repository tsx loader. No Python interpreter or external analysis engine was invoked by the reproduction. ## Target and artifact context Synthetic PyInstaller 2.1 CArchives with cookie Python version 310. Matching PYZ uses final Python 3.10 magic `6f0d0d0a` and an empty valid marshal list directory. Legacy `m`/`M` bodies contain a complete 16-byte-header pyc; modern `m` and `s` controls contain its 73-byte marshal body. These are archive bytes only, with no executable bootloader. ## Interface Other ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Exact output from the final executed probe: ```jsonl {"name":"legacy-m-stored","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."} {"name":"legacy-m-zlib","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."} {"name":"legacy-M-zlib","source_bytes":89,"output_bytes":105,"offset_16":"6f0d0d0a","parse":"Unsupported marshal type 0x6f."} {"name":"modern-m-zlib","source_bytes":73,"output_bytes":89,"offset_16":"63000000","parse":"complete"} {"name":"script-zlib","source_bytes":73,"output_bytes":89,"offset_16":"63000000","parse":"complete"} {"name":"legacy-no-PYZ","source_bytes":89,"output_bytes":89,"offset_16":"63000000","parse":"complete"} ``` The first three outputs contain the original Python magic where the marshal code object must begin. Every assertion checks the complete returned byte sequence, not just these four bytes. ## Typed outcome or recovery expected Return a single coherent pyc envelope for supported module storage layouts. If the reader cannot establish whether header derivation is appropriate, preserve the original bytes with an explicit limitation rather than silently creating an invalid derived file. ## Analysis or suspected cause Source-derived: [`projectStoredMember`](https://github.com/morluto/rea/blob/a7180370332350bc0f998fbd643888221759570e/src/artifacts/pyinstaller/CArchiveMembers.ts#L82-L109) chooses a reconstructed header for all bytecode type codes whenever a sibling magic is available. [`scriptPythonMagic`](https://github.com/morluto/rea/blob/a7180370332350bc0f998fbd643888221759570e/src/artifacts/pyinstaller/PyInstallerMetadata.ts#L47-L61) establishes version agreement, but not whether this particular member already contains a header. [`pyInstallerMemberStream`](https://github.com/morluto/rea/blob/a7180370332350bc0f998fbd643888221759570e/src/artifacts/pyinstaller/PyInstallerMemberStream.ts#L26-L29) then prefixes those bytes without inspecting or replacing the existing envelope. The [documented derivation behavior](https://github.com/morluto/rea/blob/a7180370332350bc0f998fbd643888221759570e/docs/python-packaged-applications.md#L132-L137) does not distinguish the two producer storage layouts. ## Additional context Open #6441 concerns omitted derivation notices and misleading extraction descriptions; it explicitly observed correct output bytes. This report concerns incorrect returned module bytes. Closed #6736 concerns path normalization, and #3678 is the broader capability proposal. No matching report was found in the supplied open inventory or subsequent open and closed PyInstaller searches. No patch or known-good REA release was tested. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Three inert legacy-layout cases return duplicated headers and fail the production pyc parser despite successful member reading; modern and no-PYZ controls preserve a parseable envelope. Source files are unchanged and the failure is limited to a particular archive layout. - Workaround status: available - Workaround basis: For confirmed double-header output, removing the first 16 bytes restores the exact original member; the executed assertions establish byte equality and the original parser control succeeds.

bsakaguchi closed this issue

2026-10-31 00:47:07 +00:00

bsakaguchi referenced this issue

2026-10-31 00:50:26 +00:00

[GH-ISSUE #7582] [Bug] One-column process captures silently use a two-column rendered screen #4516

bsakaguchi referenced this issue

2026-10-31 01:11:43 +00:00

[PR #4142] [MERGED] fix(process): preserve terminal cell widths during normalization #6081