morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6441] [Bug] extract-artifact drops PyInstaller derivation notices and misstates decoded PYZ output #3854

New issue

Closed

opened 2026-10-31 00:44:39 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:44:39 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6441

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

extract-artifact successfully writes a decoded PYZ module and a CArchive script with reconstructed Python headers, but its result omits their derivation notices and says nested archive contents remain inside a containing file. The output instead has a PYZ-00.pyz directory containing mod.pyc; there is no regular PYZ carrier file.

inspect-artifact on the same input reports bytecode_header: "derived" and the reconstructed-header limitation for both files. Neither the extraction's normalized limitations nor its outer Evidence limitations preserve that notice, and extraction provenance is empty. Observed on both extraction runs; file bytes themselves were correct.

Reproduction

From a fresh checkout, select the baseline and build with locked dependencies:

git checkout 9eb829aca34be36ff112826337996f6139f82eff
npm ci && npm run build:cached
WORK=$(mktemp -d)
mkdir -p "$WORK/home"
python3 - "$WORK/app" <<'PYTHON'
import marshal, struct, zlib
from pathlib import Path
import sys
module = zlib.compress(b'module-bytecode')
pyz = b'PYZ\0' + bytes.fromhex('610d0d0a') + struct.pack('>I', 12 + len(module)) + module
pyz += marshal.dumps([('mod', (0, 12, len(module)))], 2)
prefix = bytearray(64)
prefix[:7] = b'\x7fELF\x02\x01\x01'
struct.pack_into('<HHI', prefix, 16, 2, 62, 1)
struct.pack_into('<H', prefix, 52, 64)
data, toc = b'', b''
for name, kind, payload in [(b'script', b's', b'script-bytes'), (b'PYZ-00.pyz', b'z', pyz)]:
    stored = zlib.compress(payload)
    size = ((18 + len(name) + 1 + 15) // 16) * 16
    row = struct.pack('>IIIIBc', size, len(data), len(stored), len(payload), 1, kind) + name + b'\0'
    toc += row.ljust(size, b'\0')
    data += stored
cookie = bytes.fromhex('4d45490c0b0a0b0e') + struct.pack('>IIII', len(data) + len(toc) + 88, len(data), len(toc), 309) + bytes(64)
Path(sys.argv[1]).write_bytes(prefix + data + toc + cookie)
PYTHON
HOME="$WORK/home" TMPDIR="$WORK" node scripts/rea.mjs inspect-artifact "$WORK/app" --format json > "$WORK/inspection.json"
HOME="$WORK/home" TMPDIR="$WORK" node scripts/rea.mjs extract-artifact "$WORK/app" --format json > "$WORK/extraction.json"
python3 - "$WORK" <<'PYTHON'
import json, sys
from pathlib import Path
work = Path(sys.argv[1])
e = json.loads((work/'extraction.json').read_text())
r = e['normalized_result']
print('artifacts:', [(a['relative_path'], a['bytes_written']) for a in r['artifacts']])
print('limitations:', r['limitations'])
print('provenance:', r['provenance'])
out = Path(r['output_root'])
print('PYZ carrier is directory:', (out/'PYZ-00.pyz').is_dir())
for path, body in [('script', b'script-bytes'), ('PYZ-00.pyz/mod.pyc', b'module-bytecode')]:
    assert (out/path).read_bytes() == bytes.fromhex('610d0d0a') + bytes(12) + body
inventory = json.loads((work/'inspection.json').read_text())['normalized_result']['substeps'][0]['evidence']['normalized_result']
for row in inventory['occurrences']:
    if row['logical_path'] in ['script', 'PYZ-00.pyz/mod.pyc']:
        print(row['logical_path'], row['format_metadata']['bytecode_header'], row['limitations'])
PYTHON

The fixture construction and CLI calls above were executed. Existing built dependencies were used for verification; the fresh installation/build commands describe checkout preparation. Two extraction calls and one inspection succeeded with exit 0. No recovered code was executed.

Expected behavior

Extraction should retain the relevant reconstructed-header explanation and describe decoded PYZ materialization truthfully. It should distinguish output bytes derived by REA from original member payloads without requiring another command solely to understand provenance.

The shared PyInstaller reader contract applies to both commands. The documented header behavior states that headers are synthesized and digests cover derived files. Tool design requires relevant limitations inline and separates derived relationships from observations.

Impact and scope

P2: callers receive misleading materialization guidance and lose the explanation for why extracted bytes include a header absent from the member payload. This is bounded evidence-provenance loss with an observed workaround: run inspect-artifact and consult its per-occurrence limitations alongside the extraction file list. No source modification, missing extracted module, or incorrect output digest was observed.

Observed scope is the synthetic PyInstaller 2.1/Python-3.9 fixture through CLI on macOS. MCP and other formats/platforms were not executed. The payloads are reduced parser fixtures, not valid executable Python code; successful execution or decompilation is not claimed.

REA version

6.7.0 (source 9eb829aca3)

Runtime and platform

Node.js v26.12.1; macOS 27.0.1; arm64. Python 3 standard library used only to generate the archive fixture. No analysis engine or target process was launched.

Target and artifact context

Synthetic ELF prefix followed by a PyInstaller 2.1 CArchive, cookie Python version 309, compressed type-s script and compressed type-z PYZ. PYZ magic is 610d0d0a and its directory contains one type-0 module.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

Projection of the observed extraction result:

{
  "artifacts": [
    {"relative_path": "PYZ-00.pyz/mod.pyc", "bytes_written": 31},
    {"relative_path": "script", "bytes_written": 28}
  ],
  "provenance": [],
  "limitations": [
    "All regular files in the active artifact were materialized; nested archive contents remain represented by their containing file."
  ]
}

Actual filesystem checks found PYZ-00.pyz to be a directory. Both extracted files equal 610d0d0a, twelve zero bytes, then their respective original payload. Inspection reports both files as bytecode_header: "derived" with this notice:

Python bytecode header was reconstructed from the archive Python magic; this digest covers the derived file.

Typed outcome or recovery expected

Keep successful extraction, its validated file list and hashes, while retaining applicable derivation limitations or equivalent provenance. Use the containing-file caveat only when the nested content actually remains inside a materialized regular carrier.

Analysis or suspected cause

Source-derived: decoded PYZ projection produces a directory, so the regular-ancestor filter correctly leaves its module selected. The reader supplies header limitations for CArchive bytecode and PYZ modules. The extraction result constructor emits empty provenance and a fixed limitation instead of preserving these facts.

Additional context

No known-good version or restoration patch was tested. Related #1318 addresses files disappearing between inventory and materialization; it does not address decoded PYZ directories or lost header notices. #3678 is the broader capability proposal. No matching open or closed issue was found in the searches performed.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6441 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `extract-artifact` successfully writes a decoded PYZ module and a CArchive script with reconstructed Python headers, but its result omits their derivation notices and says nested archive contents remain inside a containing file. The output instead has a `PYZ-00.pyz` directory containing `mod.pyc`; there is no regular PYZ carrier file. `inspect-artifact` on the same input reports `bytecode_header: "derived"` and the reconstructed-header limitation for both files. Neither the extraction's normalized limitations nor its outer Evidence limitations preserve that notice, and extraction `provenance` is empty. Observed on both extraction runs; file bytes themselves were correct. ### Reproduction From a fresh checkout, select the baseline and build with locked dependencies: ```sh git checkout 9eb829aca34be36ff112826337996f6139f82eff npm ci && npm run build:cached WORK=$(mktemp -d) mkdir -p "$WORK/home" python3 - "$WORK/app" <<'PYTHON' import marshal, struct, zlib from pathlib import Path import sys module = zlib.compress(b'module-bytecode') pyz = b'PYZ\0' + bytes.fromhex('610d0d0a') + struct.pack('>I', 12 + len(module)) + module pyz += marshal.dumps([('mod', (0, 12, len(module)))], 2) prefix = bytearray(64) prefix[:7] = b'\x7fELF\x02\x01\x01' struct.pack_into('<HHI', prefix, 16, 2, 62, 1) struct.pack_into('<H', prefix, 52, 64) data, toc = b'', b'' for name, kind, payload in [(b'script', b's', b'script-bytes'), (b'PYZ-00.pyz', b'z', pyz)]: stored = zlib.compress(payload) size = ((18 + len(name) + 1 + 15) // 16) * 16 row = struct.pack('>IIIIBc', size, len(data), len(stored), len(payload), 1, kind) + name + b'\0' toc += row.ljust(size, b'\0') data += stored cookie = bytes.fromhex('4d45490c0b0a0b0e') + struct.pack('>IIII', len(data) + len(toc) + 88, len(data), len(toc), 309) + bytes(64) Path(sys.argv[1]).write_bytes(prefix + data + toc + cookie) PYTHON HOME="$WORK/home" TMPDIR="$WORK" node scripts/rea.mjs inspect-artifact "$WORK/app" --format json > "$WORK/inspection.json" HOME="$WORK/home" TMPDIR="$WORK" node scripts/rea.mjs extract-artifact "$WORK/app" --format json > "$WORK/extraction.json" python3 - "$WORK" <<'PYTHON' import json, sys from pathlib import Path work = Path(sys.argv[1]) e = json.loads((work/'extraction.json').read_text()) r = e['normalized_result'] print('artifacts:', [(a['relative_path'], a['bytes_written']) for a in r['artifacts']]) print('limitations:', r['limitations']) print('provenance:', r['provenance']) out = Path(r['output_root']) print('PYZ carrier is directory:', (out/'PYZ-00.pyz').is_dir()) for path, body in [('script', b'script-bytes'), ('PYZ-00.pyz/mod.pyc', b'module-bytecode')]: assert (out/path).read_bytes() == bytes.fromhex('610d0d0a') + bytes(12) + body inventory = json.loads((work/'inspection.json').read_text())['normalized_result']['substeps'][0]['evidence']['normalized_result'] for row in inventory['occurrences']: if row['logical_path'] in ['script', 'PYZ-00.pyz/mod.pyc']: print(row['logical_path'], row['format_metadata']['bytecode_header'], row['limitations']) PYTHON ``` The fixture construction and CLI calls above were executed. Existing built dependencies were used for verification; the fresh installation/build commands describe checkout preparation. Two extraction calls and one inspection succeeded with exit 0. No recovered code was executed. ### Expected behavior Extraction should retain the relevant reconstructed-header explanation and describe decoded PYZ materialization truthfully. It should distinguish output bytes derived by REA from original member payloads without requiring another command solely to understand provenance. The [shared PyInstaller reader contract](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/python-packaged-applications.md#L3-L6) applies to both commands. [The documented header behavior](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/python-packaged-applications.md#L109-L114) states that headers are synthesized and digests cover derived files. [Tool design](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/docs/tool-design.md#L51-L59) requires relevant limitations inline and separates derived relationships from observations. ### Impact and scope P2: callers receive misleading materialization guidance and lose the explanation for why extracted bytes include a header absent from the member payload. This is bounded evidence-provenance loss with an observed workaround: run `inspect-artifact` and consult its per-occurrence limitations alongside the extraction file list. No source modification, missing extracted module, or incorrect output digest was observed. Observed scope is the synthetic PyInstaller 2.1/Python-3.9 fixture through CLI on macOS. MCP and other formats/platforms were not executed. The payloads are reduced parser fixtures, not valid executable Python code; successful execution or decompilation is not claimed. ### REA version 6.7.0 (source 9eb829aca34be36ff112826337996f6139f82eff) ### Runtime and platform Node.js v26.12.1; macOS 27.0.1; arm64. Python 3 standard library used only to generate the archive fixture. No analysis engine or target process was launched. ### Target and artifact context Synthetic ELF prefix followed by a PyInstaller 2.1 CArchive, cookie Python version 309, compressed type-s script and compressed type-z PYZ. PYZ magic is `610d0d0a` and its directory contains one type-0 module. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence Projection of the observed extraction result: ```json { "artifacts": [ {"relative_path": "PYZ-00.pyz/mod.pyc", "bytes_written": 31}, {"relative_path": "script", "bytes_written": 28} ], "provenance": [], "limitations": [ "All regular files in the active artifact were materialized; nested archive contents remain represented by their containing file." ] } ``` Actual filesystem checks found `PYZ-00.pyz` to be a directory. Both extracted files equal `610d0d0a`, twelve zero bytes, then their respective original payload. Inspection reports both files as `bytecode_header: "derived"` with this notice: ```text Python bytecode header was reconstructed from the archive Python magic; this digest covers the derived file. ``` ### Typed outcome or recovery expected Keep successful extraction, its validated file list and hashes, while retaining applicable derivation limitations or equivalent provenance. Use the containing-file caveat only when the nested content actually remains inside a materialized regular carrier. ### Analysis or suspected cause Source-derived: [decoded PYZ projection](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/artifacts/pyinstaller/CArchiveMembers.ts#L165-L190) produces a directory, so the [regular-ancestor filter](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/artifacts/extraction/ArtifactExtraction.ts#L126-L138) correctly leaves its module selected. The reader supplies header limitations for [CArchive bytecode](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/artifacts/pyinstaller/CArchiveMembers.ts#L82-L102) and [PYZ modules](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/artifacts/pyinstaller/CArchiveMembers.ts#L240-L261). The [extraction result constructor](https://github.com/morluto/rea/blob/9eb829aca34be36ff112826337996f6139f82eff/src/artifacts/extraction/ArtifactExtraction.ts#L382-L416) emits empty provenance and a fixed limitation instead of preserving these facts. ### Additional context No known-good version or restoration patch was tested. Related #1318 addresses files disappearing between inventory and materialization; it does not address decoded PYZ directories or lost header notices. #3678 is the broader capability proposal. No matching open or closed issue was found in the searches performed.

bsakaguchi closed this issue

2026-10-31 00:44:40 +00:00

bsakaguchi referenced this issue

2026-10-31 01:10:40 +00:00

[PR #3854] [MERGED] feat(kuna): serve procedures, pseudocode and assembly from Kuna exports #5950

bsakaguchi referenced this issue

2026-10-31 01:10:41 +00:00

[PR #3855] [MERGED] docs(kuna): document the Kuna provider and list it in the catalog #5952

bsakaguchi referenced this issue

2026-10-31 01:11:05 +00:00

[PR #4029] [MERGED] fix(kuna): preserve ownership through cancellation and failed verification #6001

bsakaguchi referenced this issue