morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3953] [Bug] ZIP member headers beyond EOF are misreported as file-access failures #1559

New issue

Closed

opened 2026-10-16 21:07:08 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:07:08 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3953

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Observed through the public CLI in the standalone probe and exact draft replay: damaged.zip is 197 bytes, retains note.txt's original local-header offset of 4137, and exits 1 with the following error fields. Stderr is empty.

{
  "code": "artifact_operation_failed",
  "category": "execution_failure",
  "message": "Artifact could not be read or written. Check file access and try again.",
  "retryable": false,
  "remediation": {
    "action": "Artifact could not be read or written. Check file access and try again."
  },
  "details": {
    "operation": "inspect_artifact",
    "reason": "io"
  }
}

Observed controls: valid.zip is 4313 bytes and exits 0 with inspection Evidence. Prefix-only truncation exits 1 with reason format and detail "ZIP archive is malformed: End of central directory not found". The in-file-header control exits 1 with reason format and detail "ZIP archive is malformed: Local file header not found". With the original directory order retained, the same local-region truncation reports format for the earlier padding.txt data failure before reaching note.txt. These observations bound the defect to the missing-header read reached first in this case; they do not establish that all truncated archives are misclassified.

Reproduction

Run from the baseline repository root with the installed dependencies and compiled runtime available. This procedure creates a valid ZIP, truncates the tail of its local-record region while retaining the directory, and runs the supported CLI in an isolated home directory. The directory enumerates note.txt before padding.txt, while the valid local records store padding.txt first. Reversing directory order is valid and makes the missing header the first member read. Only the directory's own end-record location changes after truncation; the original member offsets remain intact. This is a damaged archive with a retained directory, not a simulation of an ordinary prefix-only interrupted download.

python3 - <<'PYTHON'
import io
import json
import os
from pathlib import Path
import struct
import subprocess
import tempfile
import zipfile

with tempfile.TemporaryDirectory(prefix="rea-zip-review-") as work:
    root = Path(work)
    home = root / "home"
    home.mkdir()
    buffer = io.BytesIO()
    with zipfile.ZipFile(buffer, "w", compression=zipfile.ZIP_STORED) as archive:
        archive.writestr("padding.txt", b"A" * 4096)
        archive.writestr("note.txt", b"hello")
        archive.filelist.reverse()
    valid = buffer.getvalue()
    with zipfile.ZipFile(io.BytesIO(valid)) as archive:
        assert archive.testzip() is None
        assert archive.getinfo("note.txt").header_offset == 4137
    directory = valid.index(b"PK\x01\x02")
    damaged = bytearray(valid[:64] + valid[directory:])
    end_record = damaged.rfind(b"PK\x05\x06")
    struct.pack_into("<I", damaged, end_record + 16, 64)
    bad_header = bytearray(damaged)
    struct.pack_into("<I", bad_header, 64 + 42, 1)
    fixtures = {
        "valid.zip": valid,
        "damaged.zip": damaged,
        "prefix-only.zip": valid[:64],
        "in-file-header.zip": bad_header,
    }
    for name, data in fixtures.items():
        path = root / name
        path.write_bytes(data)
        result = subprocess.run(
            ["node", "scripts/rea.mjs", "inspect-artifact", str(path), "--json"],
            env=dict(os.environ, HOME=str(home)),
            text=True,
            capture_output=True,
        )
        output = json.loads(result.stdout)
        print(json.dumps({
            "fixture": name,
            "bytes": len(data),
            "exit": result.returncode,
            "code": output.get("code"),
            "message": output.get("message"),
            "details": output.get("details"),
            "stderr": result.stderr,
        }))
PYTHON

Expected behavior

A missing member header is malformed ZIP structure and should return code: "artifact_operation_failed", details.reason: "format", and a non-empty detail identifying archive corruption. The message and remediation should direct the caller to obtain a fresh copy. Inspection need not recover absent member data. Other damaged ZIP structures already produce this classification.

Impact and scope

A bounded damaged-ZIP case returns the wrong machine-readable reason, omits the archive-corruption detail and advises file-access repairs for missing bytes. The valid control and other corruption controls behave correctly. No valid-input workflow failure or data loss was observed. An intact version of the synthetic ZIP succeeds, but availability of a replacement for affected callers is unknown, and no supported option correcting this fixture's classification was demonstrated.

REA version

rea 6.5.0 at source commit 4692bb3330

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Locally generated stored ZIP with two members. Damaged fixture retains the original central directory but truncates the local-record region; it is not an ordinary prefix-only interrupted download. No extraction or target code execution.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The damaged fixture is 197 bytes; its first enumerated member note.txt retains original local-header offset 4137. The public CLI exited 1 with artifact_operation_failed, category execution_failure, details { "operation": "inspect_artifact", "reason": "io" }, and file-access guidance without a malformed-archive detail. Stderr was empty.

Controls: valid 4313-byte ZIP exited 0; prefix-only truncation returned format with End of central directory not found; an in-file invalid header returned format with Local file header not found. Original directory order instead encounters Entry data out of bounds first and correctly returns format.

A separate production-reader probe enumerated note.txt then raised RangeError ERR_OUT_OF_RANGE from Buffer.alloc, with Received -3940, equal to 197 minus 4137. This corroborates the mechanism; the public CLI establishes reachability.

Expected source: src/artifacts/ZipArtifactReader.ts:369-407, tests/boundary/filesystem/artifactZipIntegrity.test.ts:447-459, and reason-specific guidance at src/domain/analysisErrorPresentation.ts:427-429. Failure path: src/cli/artifactCommands.ts:224-247, src/artifacts/ArtifactProvider.ts:331-349,405-430, src/artifacts/inventory/scanReader.ts:274-293, and src/artifacts/ZipArtifactReader.ts:123-140,396-407.

The failing files are unchanged at current upstream main 22ec8bdb2f7d4626587f23b1eed4a207df600386; the intervening cited launcher change only normalizes output flags. Existing ZIP integrity tests passed all 20 cases via npx --no-install vitest run tests/boundary/filesystem/artifactZipIntegrity.test.ts --maxWorkers=1. They do not cover this missing-header offset.

Typed outcome or recovery expected

Run the same damaged fixture through inspect-artifact. It must still reject the missing bytes, but return reason format with a useful malformed-archive detail and fresh-copy guidance. The exact detail text may vary. The valid control must continue to exit 0, and the prefix-only and in-file-header controls must remain format failures. Genuine filesystem-access failures must retain their appropriate classification rather than being reclassified merely to make the damaged fixture pass. Rerun the existing focused ZIP integrity tests.

Analysis or suspected cause

Source-derived: ZipArtifactReader.ts:123-140 computes a negative read size for a header beyond EOF. Its format-error mapping at lines 396-407 leaves the resulting RangeError untyped, and ArtifactProvider.ts:405-430 maps that untyped error to io.

Additional context

Historical commit cd2ad1f24 already classified structural/data corruption as format failures; this path remains outside that mapping. Related closed #1491 concerned byte-based container-family admission rather than this member-read classification. MCP, APK, IPA and other artifact operations were not executed.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3953 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior Observed through the public CLI in the standalone probe and exact draft replay: damaged.zip is 197 bytes, retains note.txt's original local-header offset of 4137, and exits 1 with the following error fields. Stderr is empty. ```json { "code": "artifact_operation_failed", "category": "execution_failure", "message": "Artifact could not be read or written. Check file access and try again.", "retryable": false, "remediation": { "action": "Artifact could not be read or written. Check file access and try again." }, "details": { "operation": "inspect_artifact", "reason": "io" } } ``` Observed controls: valid.zip is 4313 bytes and exits 0 with inspection Evidence. Prefix-only truncation exits 1 with reason format and detail "ZIP archive is malformed: End of central directory not found". The in-file-header control exits 1 with reason format and detail "ZIP archive is malformed: Local file header not found". With the original directory order retained, the same local-region truncation reports format for the earlier padding.txt data failure before reaching note.txt. These observations bound the defect to the missing-header read reached first in this case; they do not establish that all truncated archives are misclassified. ### Reproduction Run from the baseline repository root with the installed dependencies and compiled runtime available. This procedure creates a valid ZIP, truncates the tail of its local-record region while retaining the directory, and runs the supported CLI in an isolated home directory. The directory enumerates note.txt before padding.txt, while the valid local records store padding.txt first. Reversing directory order is valid and makes the missing header the first member read. Only the directory's own end-record location changes after truncation; the original member offsets remain intact. This is a damaged archive with a retained directory, not a simulation of an ordinary prefix-only interrupted download. ```sh python3 - <<'PYTHON' import io import json import os from pathlib import Path import struct import subprocess import tempfile import zipfile with tempfile.TemporaryDirectory(prefix="rea-zip-review-") as work: root = Path(work) home = root / "home" home.mkdir() buffer = io.BytesIO() with zipfile.ZipFile(buffer, "w", compression=zipfile.ZIP_STORED) as archive: archive.writestr("padding.txt", b"A" * 4096) archive.writestr("note.txt", b"hello") archive.filelist.reverse() valid = buffer.getvalue() with zipfile.ZipFile(io.BytesIO(valid)) as archive: assert archive.testzip() is None assert archive.getinfo("note.txt").header_offset == 4137 directory = valid.index(b"PK\x01\x02") damaged = bytearray(valid[:64] + valid[directory:]) end_record = damaged.rfind(b"PK\x05\x06") struct.pack_into("<I", damaged, end_record + 16, 64) bad_header = bytearray(damaged) struct.pack_into("<I", bad_header, 64 + 42, 1) fixtures = { "valid.zip": valid, "damaged.zip": damaged, "prefix-only.zip": valid[:64], "in-file-header.zip": bad_header, } for name, data in fixtures.items(): path = root / name path.write_bytes(data) result = subprocess.run( ["node", "scripts/rea.mjs", "inspect-artifact", str(path), "--json"], env=dict(os.environ, HOME=str(home)), text=True, capture_output=True, ) output = json.loads(result.stdout) print(json.dumps({ "fixture": name, "bytes": len(data), "exit": result.returncode, "code": output.get("code"), "message": output.get("message"), "details": output.get("details"), "stderr": result.stderr, })) PYTHON ``` ### Expected behavior A missing member header is malformed ZIP structure and should return `code: "artifact_operation_failed"`, `details.reason: "format"`, and a non-empty detail identifying archive corruption. The message and remediation should direct the caller to obtain a fresh copy. Inspection need not recover absent member data. Other damaged ZIP structures already produce this classification. ### Impact and scope A bounded damaged-ZIP case returns the wrong machine-readable reason, omits the archive-corruption detail and advises file-access repairs for missing bytes. The valid control and other corruption controls behave correctly. No valid-input workflow failure or data loss was observed. An intact version of the synthetic ZIP succeeds, but availability of a replacement for affected callers is unknown, and no supported option correcting this fixture's classification was demonstrated. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Locally generated stored ZIP with two members. Damaged fixture retains the original central directory but truncates the local-record region; it is not an ordinary prefix-only interrupted download. No extraction or target code execution. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The damaged fixture is 197 bytes; its first enumerated member `note.txt` retains original local-header offset 4137. The public CLI exited 1 with `artifact_operation_failed`, category `execution_failure`, details `{ "operation": "inspect_artifact", "reason": "io" }`, and file-access guidance without a malformed-archive detail. Stderr was empty. Controls: valid 4313-byte ZIP exited 0; prefix-only truncation returned `format` with `End of central directory not found`; an in-file invalid header returned `format` with `Local file header not found`. Original directory order instead encounters `Entry data out of bounds` first and correctly returns `format`. A separate production-reader probe enumerated `note.txt` then raised `RangeError ERR_OUT_OF_RANGE` from `Buffer.alloc`, with `Received -3940`, equal to 197 minus 4137. This corroborates the mechanism; the public CLI establishes reachability. Expected source: `src/artifacts/ZipArtifactReader.ts:369-407`, `tests/boundary/filesystem/artifactZipIntegrity.test.ts:447-459`, and reason-specific guidance at `src/domain/analysisErrorPresentation.ts:427-429`. Failure path: `src/cli/artifactCommands.ts:224-247`, `src/artifacts/ArtifactProvider.ts:331-349,405-430`, `src/artifacts/inventory/scanReader.ts:274-293`, and `src/artifacts/ZipArtifactReader.ts:123-140,396-407`. The failing files are unchanged at current upstream main `22ec8bdb2f7d4626587f23b1eed4a207df600386`; the intervening cited launcher change only normalizes output flags. Existing ZIP integrity tests passed all 20 cases via `npx --no-install vitest run tests/boundary/filesystem/artifactZipIntegrity.test.ts --maxWorkers=1`. They do not cover this missing-header offset. ### Typed outcome or recovery expected Run the same damaged fixture through `inspect-artifact`. It must still reject the missing bytes, but return reason format with a useful malformed-archive detail and fresh-copy guidance. The exact detail text may vary. The valid control must continue to exit 0, and the prefix-only and in-file-header controls must remain format failures. Genuine filesystem-access failures must retain their appropriate classification rather than being reclassified merely to make the damaged fixture pass. Rerun the existing focused ZIP integrity tests. ### Analysis or suspected cause Source-derived: `ZipArtifactReader.ts:123-140` computes a negative read size for a header beyond EOF. Its format-error mapping at lines 396-407 leaves the resulting RangeError untyped, and `ArtifactProvider.ts:405-430` maps that untyped error to `io`. ### Additional context Historical commit `cd2ad1f24` already classified structural/data corruption as format failures; this path remains outside that mapping. Related closed #1491 concerned byte-based container-family admission rather than this member-read classification. MCP, APK, IPA and other artifact operations were not executed.

bsakaguchi closed this issue

2026-10-16 21:07:09 +00:00

bsakaguchi commented

2026-10-16 21:07:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 15, 2026):

Fixed on main in c209dcb20. The ZIP file reader now classifies offsets beyond EOF as malformed before allocation, preserving the existing behavior for short reads and exact EOF.

The corrupted central-directory offset public CLI regression failed on the baseline with reason=io, then passed after the fix with reason=format and fresh-copy guidance. Existing valid ZIP inspection/extraction and relative-path checks pass. Closing the reported classification defect.

Implementation qualification passed: CI at d9aa06d93. This verifies the implementation revision; the later PCAP-only batch has its own checks.

@morluto commented on GitHub (Oct 15, 2026): Fixed on main in c209dcb20. The ZIP file reader now classifies offsets beyond EOF as malformed before allocation, preserving the existing behavior for short reads and exact EOF. The corrupted central-directory offset public CLI regression failed on the baseline with reason=io, then passed after the fix with reason=format and fresh-copy guidance. Existing valid ZIP inspection/extraction and relative-path checks pass. Closing the reported classification defect. Implementation qualification passed: [CI at d9aa06d93](https://github.com/morluto/rea/actions/runs/38797211865). This verifies the implementation revision; the later PCAP-only batch has its own checks.

bsakaguchi referenced this issue

2026-10-17 21:12:24 +00:00

[PR #1559] [MERGED] docs: celebrate 60,000 GitHub stars 🎉 #2971

bsakaguchi referenced this issue

2026-10-31 00:55:07 +00:00

[PR #1752] [MERGED] docs: celebrate 80,000 GitHub stars 🎉 #4793