morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #7626] [Bug] JavaScript analysis diagnoses unchanged NFC/NFD ASAR siblings as changed bytes #4552

New issue

Closed

opened 2026-10-31 00:50:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:50:43 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7626

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

For a hand-built ASAR containing NFC café.js and NFD café.js with different JavaScript contents and correct declared SHA-256 hashes, this command exits 1:

node scripts/rea.mjs analyze-javascript-application "$archive" --json

It returns a single JSON error with category: "integrity_mismatch", details.reason: "integrity", and details.detail: "Artifact entry changed after inventory: café.js". Its message and remediation say, “Artifact is invalid or has changed. Get a fresh copy and try again.”

The archive does not change. inspect-artifact on those same bytes exits 0, retains both occurrences with verified hashes and distinct occurrence IDs, and reports complete traversal with no integrity contradictions. extract-artifact correctly refuses the same archive with a normalized-path collision diagnosis instead of an integrity diagnosis.

Reproduction

Use the stated baseline's built source checkout with its pinned Node.js v24.22.1 selected. Run this from the repository root. Python constructs the ASAR directly, checks its member offsets and hashes independently, isolates the CLI's state directories, and deletes the disposable files afterward.

python3 -I - <<'PY'
import hashlib
import json
import os
from pathlib import Path
import struct
import subprocess
import tempfile
import unicodedata

with tempfile.TemporaryDirectory(prefix="rea-asar-check-") as directory:
    root = Path(directory)
    env = os.environ.copy()
    for variable in ("HOME", "XDG_CONFIG_HOME", "XDG_CACHE_HOME",
                     "XDG_DATA_HOME", "XDG_STATE_HOME", "TMPDIR"):
        path = root / variable.lower()
        path.mkdir()
        env[variable] = str(path)
    members = [("café.js", b"console.log(1)\n"),
               (unicodedata.normalize("NFD", "café.js"), b"console.log(2)\n")]
    files, data = {}, b""
    for name, content in members:
        files[name] = {"size": len(content), "offset": str(len(data)),
                       "integrity": {"algorithm": "SHA256",
                                     "hash": hashlib.sha256(content).hexdigest()}}
        data += content
    header = json.dumps({"files": files}, ensure_ascii=False,
                        separators=(",", ":")).encode()
    aligned = (len(header) + 3) // 4 * 4
    header_bytes = 8 + aligned
    packed = (struct.pack("<IIII", 4, header_bytes, header_bytes - 4, len(header))
              + header + bytes(aligned - len(header)) + data)
    archive = root / "twins.asar"
    archive.write_bytes(packed)
    parsed = json.loads(packed[16:16 + struct.unpack_from("<I", packed, 12)[0]])
    for name, content in members:
        entry = parsed["files"][name]
        start = 8 + struct.unpack_from("<I", packed, 4)[0] + int(entry["offset"])
        actual = packed[start:start + entry["size"]]
        assert actual == content
        assert hashlib.sha256(actual).hexdigest() == entry["integrity"]["hash"]

    def run(operation):
        result = subprocess.run(["node", "scripts/rea.mjs", operation,
                                 str(archive), "--json"], env=env,
                                capture_output=True, text=True, timeout=60)
        return result.returncode, json.loads(result.stdout)

    status, inspected = run("inspect-artifact")
    assert status == 0
    inventory = next(step["evidence"]["normalized_result"]
                     for step in inspected["normalized_result"]["substeps"]
                     if step["operation"] == "inventory_artifact")
    occurrences = [o for o in inventory["occurrences"] if o["logical_path"] != "."]
    assert len(occurrences) == 2
    assert len({o["occurrence_id"] for o in occurrences}) == 2
    assert all(o["logical_path"] == "café.js" and o["hash_status"] == "verified"
               for o in occurrences)
    assert inventory["integrity_contradictions"] == []
    print("inventory:", inventory["coverage"], "two verified occurrences")
    for attempt in range(3):
        status, reply = run("analyze-javascript-application")
        print("analysis:", status, json.dumps(reply, ensure_ascii=False))
        assert status == 1 and reply["category"] == "integrity_mismatch"
        assert reply["details"]["detail"] == "Artifact entry changed after inventory: café.js"
    assert archive.read_bytes() == packed
PY

Observe the three false integrity failures. The initial independent verification also reproduced the primary failure in three consecutive runs. Single-NFC, single-NFD and noncolliding two-file controls each returned a normal analysis result with exit 0.

Expected behavior

Preserve the distinction between an ambiguous normalized path and changed artifact bytes. If JavaScript analysis cannot represent both occurrences, return a truthful collision or unsupported-layout refusal identifying the shared logical path. Accepting both members is not required to resolve this diagnostic report.

The documented contract says that ASAR entry bytes are rechecked against the inventory digest before parsing. The inventory contract preserves original source spelling alongside NFC logical paths. The repository's identity and failure-reason invariants require retaining source evidence and meaningful causes. A collision between two unchanged, individually verified occurrences does not establish an integrity change. Exact contract locators appear below.

Impact and scope

P3, supported: an ambiguous archive receives an incorrect machine-readable diagnosis and advice to obtain a fresh copy. A truthful collision refusal could still reject this layout, so this report does not establish a requirement for successful analysis or a broader loss of analysis capability. No source modification, incorrect successful result or data loss was observed.

The verified scope is the source CLI on macOS with this tiny self-authored ASAR. No practical workaround correcting the diagnosis for the unchanged archive was demonstrated. The successful reduced fixtures are controls, not substitutes for analyzing both original occurrences. Real-package prevalence, other operating systems, MCP transport and published-package behavior were not tested.

REA version

Exact output of node scripts/rea.mjs --version:

6.8.0

Source baseline: 65f1c447344a423fa2785847dbcea1e9a9e2454a.

Runtime and platform

Node.js v24.22.1; macOS, Darwin arm64. Python 3 standard library generated and independently checked the fixture. No native analysis provider or GUI was used.

Target and artifact context

A packed ASAR with two regular JavaScript members. Their names differ by Unicode composition; each contains 15 bytes including a newline. Both SHA256 declarations match their own member bytes. The outer size Pickle is followed by the header Pickle and aligned UTF-8 JSON; packed member data starts at 8 + header_bytes. No host filesystem sibling files are needed.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The failing command returns this one stdout document, with exit 1:

{"error":"JavaScript application analysis failed","code":"artifact_operation_failed","category":"integrity_mismatch","message":"Artifact is invalid or has changed. Get a fresh copy and try again.","retryable":false,"remediation":{"action":"Artifact is invalid or has changed. Get a fresh copy and try again."},"details":{"operation":"analyze_javascript_application","reason":"integrity","detail":"Artifact entry changed after inventory: café.js"}}

Inspection returns coverage: {"scope":"container-entries","status":"complete"}, an empty integrity_contradictions array, two verified occurrences at logical path café.js, distinct occurrence/content IDs, and the decomposed member's original source_path. Direct fixture-byte comparison after the operations confirms the ASAR remained unchanged.

The extraction control returns details.reason: "path" and explains that multiple selected occurrences share the normalized logical path, cannot occupy the same destination, and remain retained in inventory. This control establishes that the earlier extraction repair still works on the tested input.

At the stated baseline:

Expected sources: docs/javascript-artifact-reconstruction.md:672-675 defines the pre-parse inventory digest check; docs/mcp-contracts.md:1210-1213 defines normalized paths and retained source_path; AGENTS.md, Architecture and Cleanup, requires preserved identity/source evidence and meaningful failure reasons.
Failure sources: src/artifacts/javascript/JavaScriptArtifactFiles.ts:465-471 builds one expected-file value per normalized path; :286-291 retrieves it for each traversed entry; :369-378 compares that entry against the surviving digest.
Existing extraction control: src/artifacts/extraction/ArtifactExtraction.ts:306-313 refuses repeated selected logical paths before materialization.

Evidence provenance: observed CLI behavior and independent fixture-byte validation; source-derived explanation. No repaired build was tested.

Typed outcome or recovery expected

A collision/limitation identifying the ambiguous normalized path, without classifying unchanged bytes as an integrity mismatch or advising that another copy resolves the ambiguity. Preserve real inventory-to-byte verification failures and inventory multiplicity.

Analysis or suspected cause

Source-derived explanation: the acquisition lookup retains one expected digest for each NFC logical path. Both physical ASAR entries are then read, but each is checked against that same surviving expected-file value. Different member contents therefore make one entry appear changed even though it still matches its own inventoried content identity. This report does not prescribe an occurrence-ID schema change or a particular repair.

Additional context

Closed #4271 reports the analogous false diagnosis in extraction. Its closing comment, PR #4273, and fix commit specify extraction's materialization preflight. That commit changes only extraction production code plus extraction/inventory tests. It does not close a JavaScript-analysis reproduction across all surfaces; this is a separate remaining consumer defect, not an observed regression of the extraction repair.

Closed #3152 establishes duplicate-member inventory and asks for explicit downstream ambiguity handling, but reports inventory rejection rather than this post-inventory JavaScript diagnostic. #5108 concerns source-spelling retention, which already works in this verification. #1488 concerns joins that omitted NFC; this input already has correct normalized lookup keys. Searches also distinguished comparison-count, ZIP reader identity and JavaScript recovery-input collision fixes. No matching analysis report or fix was found in the searched open/closed issues, comments, or open/closed/merged PRs.

Triage assessment

Impact level: P3
Assessment status: supported
Impact basis: The demonstrated defect is the false integrity category and fresh-copy guidance for a bounded ambiguous-layout refusal. A truthful refusal remains an acceptable outcome; no data loss or incorrect successful output was observed.
Workaround status: unverified
Workaround basis: No supported way to obtain a truthful analysis diagnosis for the unchanged archive was demonstrated. Removing or renaming a member changes the input and does not establish a workaround preserving both original occurrences.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7626 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior For a hand-built ASAR containing NFC `café.js` and NFD `café.js` with different JavaScript contents and correct declared SHA-256 hashes, this command exits 1: ```sh node scripts/rea.mjs analyze-javascript-application "$archive" --json ``` It returns a single JSON error with `category: "integrity_mismatch"`, `details.reason: "integrity"`, and `details.detail: "Artifact entry changed after inventory: café.js"`. Its message and remediation say, “Artifact is invalid or has changed. Get a fresh copy and try again.” The archive does not change. `inspect-artifact` on those same bytes exits 0, retains both occurrences with verified hashes and distinct occurrence IDs, and reports complete traversal with no integrity contradictions. `extract-artifact` correctly refuses the same archive with a normalized-path collision diagnosis instead of an integrity diagnosis. ### Reproduction 1. Use the stated baseline's built source checkout with its pinned Node.js v24.22.1 selected. Run this from the repository root. Python constructs the ASAR directly, checks its member offsets and hashes independently, isolates the CLI's state directories, and deletes the disposable files afterward. ```sh python3 -I - <<'PY' import hashlib import json import os from pathlib import Path import struct import subprocess import tempfile import unicodedata with tempfile.TemporaryDirectory(prefix="rea-asar-check-") as directory: root = Path(directory) env = os.environ.copy() for variable in ("HOME", "XDG_CONFIG_HOME", "XDG_CACHE_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", "TMPDIR"): path = root / variable.lower() path.mkdir() env[variable] = str(path) members = [("café.js", b"console.log(1)\n"), (unicodedata.normalize("NFD", "café.js"), b"console.log(2)\n")] files, data = {}, b"" for name, content in members: files[name] = {"size": len(content), "offset": str(len(data)), "integrity": {"algorithm": "SHA256", "hash": hashlib.sha256(content).hexdigest()}} data += content header = json.dumps({"files": files}, ensure_ascii=False, separators=(",", ":")).encode() aligned = (len(header) + 3) // 4 * 4 header_bytes = 8 + aligned packed = (struct.pack("<IIII", 4, header_bytes, header_bytes - 4, len(header)) + header + bytes(aligned - len(header)) + data) archive = root / "twins.asar" archive.write_bytes(packed) parsed = json.loads(packed[16:16 + struct.unpack_from("<I", packed, 12)[0]]) for name, content in members: entry = parsed["files"][name] start = 8 + struct.unpack_from("<I", packed, 4)[0] + int(entry["offset"]) actual = packed[start:start + entry["size"]] assert actual == content assert hashlib.sha256(actual).hexdigest() == entry["integrity"]["hash"] def run(operation): result = subprocess.run(["node", "scripts/rea.mjs", operation, str(archive), "--json"], env=env, capture_output=True, text=True, timeout=60) return result.returncode, json.loads(result.stdout) status, inspected = run("inspect-artifact") assert status == 0 inventory = next(step["evidence"]["normalized_result"] for step in inspected["normalized_result"]["substeps"] if step["operation"] == "inventory_artifact") occurrences = [o for o in inventory["occurrences"] if o["logical_path"] != "."] assert len(occurrences) == 2 assert len({o["occurrence_id"] for o in occurrences}) == 2 assert all(o["logical_path"] == "café.js" and o["hash_status"] == "verified" for o in occurrences) assert inventory["integrity_contradictions"] == [] print("inventory:", inventory["coverage"], "two verified occurrences") for attempt in range(3): status, reply = run("analyze-javascript-application") print("analysis:", status, json.dumps(reply, ensure_ascii=False)) assert status == 1 and reply["category"] == "integrity_mismatch" assert reply["details"]["detail"] == "Artifact entry changed after inventory: café.js" assert archive.read_bytes() == packed PY ``` 2. Observe the three false integrity failures. The initial independent verification also reproduced the primary failure in three consecutive runs. Single-NFC, single-NFD and noncolliding two-file controls each returned a normal analysis result with exit 0. ### Expected behavior Preserve the distinction between an ambiguous normalized path and changed artifact bytes. If JavaScript analysis cannot represent both occurrences, return a truthful collision or unsupported-layout refusal identifying the shared logical path. Accepting both members is not required to resolve this diagnostic report. The documented contract says that ASAR entry bytes are rechecked against the inventory digest before parsing. The inventory contract preserves original source spelling alongside NFC logical paths. The repository's identity and failure-reason invariants require retaining source evidence and meaningful causes. A collision between two unchanged, individually verified occurrences does not establish an integrity change. Exact contract locators appear below. ### Impact and scope P3, supported: an ambiguous archive receives an incorrect machine-readable diagnosis and advice to obtain a fresh copy. A truthful collision refusal could still reject this layout, so this report does not establish a requirement for successful analysis or a broader loss of analysis capability. No source modification, incorrect successful result or data loss was observed. The verified scope is the source CLI on macOS with this tiny self-authored ASAR. No practical workaround correcting the diagnosis for the unchanged archive was demonstrated. The successful reduced fixtures are controls, not substitutes for analyzing both original occurrences. Real-package prevalence, other operating systems, MCP transport and published-package behavior were not tested. ### REA version Exact output of `node scripts/rea.mjs --version`: ```text 6.8.0 ``` Source baseline: `65f1c447344a423fa2785847dbcea1e9a9e2454a`. ### Runtime and platform Node.js v24.22.1; macOS, Darwin arm64. Python 3 standard library generated and independently checked the fixture. No native analysis provider or GUI was used. ### Target and artifact context A packed ASAR with two regular JavaScript members. Their names differ by Unicode composition; each contains 15 bytes including a newline. Both SHA256 declarations match their own member bytes. The outer size Pickle is followed by the header Pickle and aligned UTF-8 JSON; packed member data starts at `8 + header_bytes`. No host filesystem sibling files are needed. ### Interface CLI ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The failing command returns this one stdout document, with exit 1: ```json {"error":"JavaScript application analysis failed","code":"artifact_operation_failed","category":"integrity_mismatch","message":"Artifact is invalid or has changed. Get a fresh copy and try again.","retryable":false,"remediation":{"action":"Artifact is invalid or has changed. Get a fresh copy and try again."},"details":{"operation":"analyze_javascript_application","reason":"integrity","detail":"Artifact entry changed after inventory: café.js"}} ``` Inspection returns `coverage: {"scope":"container-entries","status":"complete"}`, an empty `integrity_contradictions` array, two verified occurrences at logical path `café.js`, distinct occurrence/content IDs, and the decomposed member's original `source_path`. Direct fixture-byte comparison after the operations confirms the ASAR remained unchanged. The extraction control returns `details.reason: "path"` and explains that multiple selected occurrences share the normalized logical path, cannot occupy the same destination, and remain retained in inventory. This control establishes that the earlier extraction repair still works on the tested input. At the stated baseline: - Expected sources: `docs/javascript-artifact-reconstruction.md:672-675` defines the pre-parse inventory digest check; `docs/mcp-contracts.md:1210-1213` defines normalized paths and retained `source_path`; `AGENTS.md`, Architecture and Cleanup, requires preserved identity/source evidence and meaningful failure reasons. - Failure sources: `src/artifacts/javascript/JavaScriptArtifactFiles.ts:465-471` builds one expected-file value per normalized path; `:286-291` retrieves it for each traversed entry; `:369-378` compares that entry against the surviving digest. - Existing extraction control: `src/artifacts/extraction/ArtifactExtraction.ts:306-313` refuses repeated selected logical paths before materialization. Evidence provenance: observed CLI behavior and independent fixture-byte validation; source-derived explanation. No repaired build was tested. ### Typed outcome or recovery expected A collision/limitation identifying the ambiguous normalized path, without classifying unchanged bytes as an integrity mismatch or advising that another copy resolves the ambiguity. Preserve real inventory-to-byte verification failures and inventory multiplicity. ### Analysis or suspected cause Source-derived explanation: the acquisition lookup retains one expected digest for each NFC logical path. Both physical ASAR entries are then read, but each is checked against that same surviving expected-file value. Different member contents therefore make one entry appear changed even though it still matches its own inventoried content identity. This report does not prescribe an occurrence-ID schema change or a particular repair. ### Additional context Closed [#4271](https://github.com/morluto/rea/issues/4271) reports the analogous false diagnosis in **extraction**. Its [closing comment](https://github.com/morluto/rea/issues/4271#issuecomment-6206032530), [PR #4273](https://github.com/morluto/rea/pull/4273), and [fix commit](https://github.com/morluto/rea/commit/862a0b2631785492f8dc20c8acd396646c545fd1) specify extraction's materialization preflight. That commit changes only extraction production code plus extraction/inventory tests. It does not close a JavaScript-analysis reproduction across all surfaces; this is a separate remaining consumer defect, not an observed regression of the extraction repair. Closed [#3152](https://github.com/morluto/rea/issues/3152) establishes duplicate-member inventory and asks for explicit downstream ambiguity handling, but reports inventory rejection rather than this post-inventory JavaScript diagnostic. [#5108](https://github.com/morluto/rea/issues/5108) concerns source-spelling retention, which already works in this verification. [#1488](https://github.com/morluto/rea/issues/1488) concerns joins that omitted NFC; this input already has correct normalized lookup keys. Searches also distinguished comparison-count, ZIP reader identity and JavaScript recovery-input collision fixes. No matching analysis report or fix was found in the searched open/closed issues, comments, or open/closed/merged PRs. ## Triage assessment - Impact level: P3 - Assessment status: supported - Impact basis: The demonstrated defect is the false integrity category and fresh-copy guidance for a bounded ambiguous-layout refusal. A truthful refusal remains an acceptable outcome; no data loss or incorrect successful output was observed. - Workaround status: unverified - Workaround basis: No supported way to obtain a truthful analysis diagnosis for the unchanged archive was demonstrated. Removing or renaming a member changes the input and does not establish a workaround preserving both original occurrences.

bsakaguchi closed this issue

2026-10-31 00:50:44 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:25 +00:00

[PR #4552] [CLOSED] fix(ghidra): keep launcher failures primary when cancellation arrives later #6304