[GH-ISSUE #6852] [Bug] extract-artifact drops unknown traversal coverage and claims an incomplete copy is complete #4192

New issue

Closed

opened 2026-10-31 00:47:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:35 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6852

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

On the stated main revision, extract-artifact exits 0, omits an existing regular file whose POSIX basename contains byte FF, and reports: All regular files in the active artifact were materialized. Neither its normalized result nor outer Evidence retains a warning about the omission.

inspect-artifact on the same unchanged bundle correctly returns inventory coverage unknown and explains the non-UTF-8 filename. Extraction loses that already-established uncertainty rather than rejecting or qualifying the incomplete output.

Reproduction

Executed from the built source checkout using only an inert generated bundle. The setup below is the executed fixture construction with scratch paths generalized:

work_dir="$(mktemp -d)"
export REA_REPRO_ROOT="$work_dir"
python3 - <<'PY'
import os, pathlib, plistlib, struct
r = pathlib.Path(os.environ['REA_REPRO_ROOT'])
c = r / 'Sample.app' / 'Contents'
m = c / 'MacOS'
m.mkdir(parents=True)
(c / 'Info.plist').write_bytes(plistlib.dumps({
    'CFBundlePackageType': 'APPL',
    'CFBundleExecutable': 'sample',
    'CFBundleIdentifier': 'example.sample',
}))
exe = bytearray(32)
struct.pack_into('<8I', exe, 0, 0xfeedfacf, 0x0100000c, 0, 2, 0, 0, 0, 0)
(m / 'sample').write_bytes(exe)
os.chmod(m / 'sample', 0o755)
(c / 'good.txt').write_text('retained\n')
fd = os.open(os.fsencode(c) + b'/opaque-\xff.txt', os.O_WRONLY | os.O_CREAT, 0o600)
os.write(fd, b'required bytes\n')
os.close(fd)
(r / 'temp').mkdir()
PY
TMPDIR="$work_dir/temp" node scripts/rea.mjs inspect-artifact "$work_dir/Sample.app" --json > "$work_dir/inspect.json"
TMPDIR="$work_dir/temp" node scripts/rea.mjs extract-artifact "$work_dir/Sample.app" --json > "$work_dir/extract.json"

Each public CLI command ran once and exited 0. I then inspected the stored JSON, enumerated source/output names as raw bytes, and read the original omitted file: its required bytes\n content remained intact. The regular files copied were Contents/Info.plist, Contents/MacOS/sample and Contents/good.txt; the fourth regular file was absent.

Expected behavior

An extraction result must retain known inventory/traversal uncertainty and avoid claiming that every regular file was materialized when an admitted source entry was omitted. This can be an explicit limitation or a typed refusal; it does not require creating a replacement-decoded filename.

The extraction contract describes extracting regular files exposed by the active reader. The reader already reports the omitted filename as unknown coverage; the inspection path preserves that fact.

Impact and scope

The published extraction directory is incomplete while its Evidence claims all regular files were materialized. Callers can mistake the result for a complete bundle copy. The demonstrated case is a legal byte-valued filename on Linux; source files remain intact. Other platforms and real applications were not tested.

Running inspect-artifact first exposes the omission, as observed here, but does not produce a complete extraction. No workaround preserving the original pathname and complete output was established.

REA version

Unreleased main source 9066444f6b8e9ffe1607a9b02cde3895a37db431, manifest version 6.7.0. The latest published release is also named 6.7.0; its runtime was not tested and this report makes no claim that it has the same omission behavior.

Runtime and platform

Node.js v26.12.1; Linux x86_64; built source runtime and locked dependencies. Python 3.14.8 standard library generated the fixture. No Hopper, Ghidra, Apple tool or target executable was run.

Target and artifact context

A self-authored Sample.app directory with a plist, a 32-byte static Mach-O header fixture, one ordinary text resource, and a text resource with raw basename hex 6f70617175652dff2e747874. The header is sufficient for static artifact admission; no runnable-application claim is made.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The inspection's inventory result contained:

{
  "coverage": {"scope":"container-entries","status":"unknown"},
  "limitations": [
    "Directory contains 1 entry name(s) that are not valid UTF-8; no filesystem operations were performed for them. First raw basename bytes: 6f70617175652dff2e747874."
  ]
}

The extraction result and its outer Evidence instead contained only:

{
  "limitations": [
    "All regular files in the active artifact were materialized; nested archive contents remain represented by their containing file."
  ]
}

The output contained the three named regular files above; raw-byte directory enumeration confirmed the omitted source entry still existed. Evidence provenance: observed public CLI behavior and filesystem readback. Local verification: reproduced. Reproduction completeness: complete for this static bundle fixture.

Typed outcome or recovery expected

Preserve the unknown traversal and its filename explanation in extraction Evidence, or refuse a complete extraction explicitly. Successfully copied files and hashes can remain available with truthful scope. A normal UTF-8-only bundle should still extract completely.

Analysis or suspected cause

Source-derived: inventory traversal captures the reader's coverage callback, but the extraction handoff retains only the manifest and integrity contradictions. The result constructor emits an unconditional completeness sentence, with no inventory coverage or filename limitation.

Additional context

Related #6825 concerns Interface Builder and dylib consumers that never collect the directory callback; here inventory collects it correctly, then extraction drops it. #6743 concerns the separate firmware staging implementation. #6441 concerns correct PyInstaller output losing per-file derivation notices; this case omits source content and loses root traversal coverage. Fixing those format-specific output notices alone does not establish complete directory extraction.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A public CLI extraction omits a real regular file while discarding known uncertainty and claiming complete materialization. The effect is bounded to this pathname representation and leaves source data intact.
Workaround status: unverified
Workaround basis: Inspection exposes the gap but does not materialize the missing file. A practical complete extraction preserving original paths was not verified.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6852 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior On the stated main revision, `extract-artifact` exits 0, omits an existing regular file whose POSIX basename contains byte FF, and reports: `All regular files in the active artifact were materialized`. Neither its normalized result nor outer Evidence retains a warning about the omission. `inspect-artifact` on the same unchanged bundle correctly returns inventory coverage `unknown` and explains the non-UTF-8 filename. Extraction loses that already-established uncertainty rather than rejecting or qualifying the incomplete output. ### Reproduction Executed from the built source checkout using only an inert generated bundle. The setup below is the executed fixture construction with scratch paths generalized: ```sh work_dir="$(mktemp -d)" export REA_REPRO_ROOT="$work_dir" python3 - <<'PY' import os, pathlib, plistlib, struct r = pathlib.Path(os.environ['REA_REPRO_ROOT']) c = r / 'Sample.app' / 'Contents' m = c / 'MacOS' m.mkdir(parents=True) (c / 'Info.plist').write_bytes(plistlib.dumps({ 'CFBundlePackageType': 'APPL', 'CFBundleExecutable': 'sample', 'CFBundleIdentifier': 'example.sample', })) exe = bytearray(32) struct.pack_into('<8I', exe, 0, 0xfeedfacf, 0x0100000c, 0, 2, 0, 0, 0, 0) (m / 'sample').write_bytes(exe) os.chmod(m / 'sample', 0o755) (c / 'good.txt').write_text('retained\n') fd = os.open(os.fsencode(c) + b'/opaque-\xff.txt', os.O_WRONLY | os.O_CREAT, 0o600) os.write(fd, b'required bytes\n') os.close(fd) (r / 'temp').mkdir() PY TMPDIR="$work_dir/temp" node scripts/rea.mjs inspect-artifact "$work_dir/Sample.app" --json > "$work_dir/inspect.json" TMPDIR="$work_dir/temp" node scripts/rea.mjs extract-artifact "$work_dir/Sample.app" --json > "$work_dir/extract.json" ``` Each public CLI command ran once and exited 0. I then inspected the stored JSON, enumerated source/output names as raw bytes, and read the original omitted file: its `required bytes\n` content remained intact. The regular files copied were `Contents/Info.plist`, `Contents/MacOS/sample` and `Contents/good.txt`; the fourth regular file was absent. ### Expected behavior An extraction result must retain known inventory/traversal uncertainty and avoid claiming that every regular file was materialized when an admitted source entry was omitted. This can be an explicit limitation or a typed refusal; it does not require creating a replacement-decoded filename. The [extraction contract](https://github.com/morluto/rea/blob/9066444f6b8e9ffe1607a9b02cde3895a37db431/src/contracts/artifactToolContracts.ts#L86-L90) describes extracting regular files exposed by the active reader. The reader already reports the omitted filename as unknown coverage; the inspection path preserves that fact. ### Impact and scope The published extraction directory is incomplete while its Evidence claims all regular files were materialized. Callers can mistake the result for a complete bundle copy. The demonstrated case is a legal byte-valued filename on Linux; source files remain intact. Other platforms and real applications were not tested. Running `inspect-artifact` first exposes the omission, as observed here, but does not produce a complete extraction. No workaround preserving the original pathname and complete output was established. ### REA version Unreleased main source `9066444f6b8e9ffe1607a9b02cde3895a37db431`, manifest version `6.7.0`. The latest published release is also named `6.7.0`; its runtime was not tested and this report makes no claim that it has the same omission behavior. ### Runtime and platform Node.js v26.12.1; Linux x86_64; built source runtime and locked dependencies. Python 3.14.8 standard library generated the fixture. No Hopper, Ghidra, Apple tool or target executable was run. ### Target and artifact context A self-authored `Sample.app` directory with a plist, a 32-byte static Mach-O header fixture, one ordinary text resource, and a text resource with raw basename hex `6f70617175652dff2e747874`. The header is sufficient for static artifact admission; no runnable-application claim is made. ### Interface CLI ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The inspection's inventory result contained: ```json { "coverage": {"scope":"container-entries","status":"unknown"}, "limitations": [ "Directory contains 1 entry name(s) that are not valid UTF-8; no filesystem operations were performed for them. First raw basename bytes: 6f70617175652dff2e747874." ] } ``` The extraction result and its outer Evidence instead contained only: ```json { "limitations": [ "All regular files in the active artifact were materialized; nested archive contents remain represented by their containing file." ] } ``` The output contained the three named regular files above; raw-byte directory enumeration confirmed the omitted source entry still existed. Evidence provenance: observed public CLI behavior and filesystem readback. Local verification: reproduced. Reproduction completeness: complete for this static bundle fixture. ### Typed outcome or recovery expected Preserve the unknown traversal and its filename explanation in extraction Evidence, or refuse a complete extraction explicitly. Successfully copied files and hashes can remain available with truthful scope. A normal UTF-8-only bundle should still extract completely. ### Analysis or suspected cause Source-derived: [inventory traversal captures the reader's coverage callback](https://github.com/morluto/rea/blob/9066444f6b8e9ffe1607a9b02cde3895a37db431/src/artifacts/inventory/scanReader.ts#L148-L164), but [the extraction handoff retains only the manifest and integrity contradictions](https://github.com/morluto/rea/blob/9066444f6b8e9ffe1607a9b02cde3895a37db431/src/artifacts/extraction/ArtifactExtraction.ts#L153-L160). [The result constructor emits an unconditional completeness sentence](https://github.com/morluto/rea/blob/9066444f6b8e9ffe1607a9b02cde3895a37db431/src/artifacts/extraction/ArtifactExtraction.ts#L401-L420), with no inventory coverage or filename limitation. ### Additional context Related [#6825](https://github.com/morluto/rea/issues/6825) concerns Interface Builder and dylib consumers that never collect the directory callback; here inventory collects it correctly, then extraction drops it. [#6743](https://github.com/morluto/rea/issues/6743) concerns the separate firmware staging implementation. [#6441](https://github.com/morluto/rea/issues/6441) concerns correct PyInstaller output losing per-file derivation notices; this case omits source content and loses root traversal coverage. Fixing those format-specific output notices alone does not establish complete directory extraction. ### Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A public CLI extraction omits a real regular file while discarding known uncertainty and claiming complete materialization. The effect is bounded to this pathname representation and leaves source data intact. - Workaround status: unverified - Workaround basis: Inspection exposes the gap but does not materialize the missing file. A practical complete extraction preserving original paths was not verified.

bsakaguchi closed this issue

2026-10-31 00:47:35 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:20 +00:00

[PR #4290] [MERGED] fix(native): report PE input byte budgets as file-size limits #6161