morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5620] [Bug] Reference-source manifests change after relocation when a file is unreadable #3408

New issue

Closed

opened 2026-10-31 00:40:02 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-31 00:40:02 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5620

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

node scripts/rea.mjs import-reference-source ROOT --detail summary --format json succeeds for two identical source trees containing an unreadable regular file, but their graph_sha256 and manifest_id differ solely because the retained EACCES diagnostic contains each selected absolute root. Their root_sha256 values match. Explicit --detail full output also contains the root; replacing each root with the same neutral alias makes those full graphs equal.

An independent verifier reproduced this with both detail modes and a readable control. All eight CLI calls exited 0. A second execution of the exact script below reproduced the same results.

Reproduction

From the baseline source checkout with its locked dependencies and compiled runtime available, run the following executed reproduction as a non-root user on a POSIX filesystem that enforces file permissions. It creates disposable fixtures and isolated HOME/XDG/TMPDIR directories. No provider, configuration file, or application database is required. Paths are generated; no original local path is included here.

python3 - <<'PY'
import json
import os
from pathlib import Path
import subprocess
import tempfile

if os.geteuid() == 0:
    raise SystemExit("Run as a non-root user for the permission fixture")
work = Path(tempfile.mkdtemp(prefix="reference-relocation-")).resolve()
env = {"PATH": os.environ["PATH"]}
for key, name in {
    "HOME": "home", "XDG_CONFIG_HOME": "config",
    "XDG_DATA_HOME": "data", "XDG_CACHE_HOME": "cache",
    "XDG_STATE_HOME": "state", "TMPDIR": "temp",
}.items():
    directory = work / name
    directory.mkdir()
    env[key] = str(directory)
roots = [work / "left", work / "right"]
for root in roots:
    root.mkdir()
    (root / "keep.txt").write_bytes(b"ok\n")
    blocked = root / "unreadable.txt"
    blocked.write_bytes(b"blocked\n")
    blocked.chmod(0)

for phase in ("unreadable", "readable"):
    if phase == "readable":
        for root in roots:
            (root / "unreadable.txt").chmod(0o600)
    records = {}
    for detail in ("summary", "full"):
        outputs = []
        for root in roots:
            result = subprocess.run(
                ["node", "scripts/rea.mjs", "import-reference-source",
                 str(root), "--detail", detail, "--format", "json"],
                env=env, capture_output=True, text=True, check=True,
            )
            stem = f"{phase}-{detail}-{root.name}"
            (work / f"{stem}.json").write_text(result.stdout)
            (work / f"{stem}.stderr").write_text(result.stderr)
            outputs.append(json.loads(result.stdout))
        records[detail] = outputs
    left, right = records["summary"]
    graph_left, graph_right = records["full"]
    entry = next(e for e in graph_left["entries"]
                 if e["path"] == "unreadable.txt")
    print(phase, json.dumps({
        "root_digest_equal": left["manifest"]["root_sha256"] == right["manifest"]["root_sha256"],
        "graph_digest_equal": left["manifest"]["graph_sha256"] == right["manifest"]["graph_sha256"],
        "manifest_id_equal": left["manifest"]["manifest_id"] == right["manifest"]["manifest_id"],
        "root_in_full": str(roots[0]) in json.dumps(graph_left),
        "root_in_summary": str(roots[0]) in json.dumps(left),
        "content_state": entry["content_state"],
        "equal_after_root_replacement": json.dumps(graph_left, sort_keys=True).replace(str(roots[0]), "$REFERENCE_ROOT") == json.dumps(graph_right, sort_keys=True).replace(str(roots[1]), "$REFERENCE_ROOT"),
    }))
PY

Expected behavior

Equivalent imports under different selected roots should have equal graph commitments and manifest IDs without embedding the selected absolute root in the committed inventory. Keep the file explicitly unreadable, with useful permission-failure details and recovery guidance. Summary should describe the same committed inventory as full output.

This expectation is source-derived: src/domain/referenceSourceGraph.ts:559-566 specifies no absolute root in the graph commitment and a relocation-independent manifest. tests/boundary/filesystem/referenceSourceImport.test.ts:403-434 checks relocated manifest equality and absence of the selected root. docs/cli.md:277-300 describes summary as a view of the same committed inventory.

Impact and scope

P2. Relocating an otherwise equivalent inventory changes its retained evidence identity and includes a local absolute path in full and summary results. The observed scope is a regular-file EACCES failure on macOS. Other filesystem failure kinds and platforms were not reproduced; no credential exposure or security boundary crossing was established.

Making both files readable restored equality in the control. Its availability as a workaround for affected callers is unverified because they may lack permission; excluding the file changes the inventory.

REA version

6.7.0; source commit 074768901f3cdc8205b61773fb1faf6f6c7b3ca8, using the prebuilt checkout runtime. Latest published release checked: rea-agents-6.7.0.

Runtime and platform

Node.js v24.22.0; macOS 27.0.1 arm64; ordinary non-root user. No Hopper/Ghidra provider involved.

Target and artifact context

Two local source directories with identical keep.txt and unreadable.txt bytes. The latter has mode 000 for the failure and 0600 for the readable control. Both root directories remain readable and imports succeed.

Interface

CLI

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The script prints these comparisons for each phase:

unreadable {"root_digest_equal": true, "graph_digest_equal": false, "manifest_id_equal": false, "root_in_full": true, "root_in_summary": true, "content_state": "unreadable", "equal_after_root_replacement": true}
readable {"root_digest_equal": true, "graph_digest_equal": true, "manifest_id_equal": true, "root_in_full": false, "root_in_summary": false, "content_state": "hashed", "equal_after_root_replacement": true}

The verifier retained stdout/stderr for each import and compared the complete saved graphs after root-only replacement. The unreadable entry's limitation contains an EACCES open diagnostic naming its absolute location, followed by permission-recovery guidance. Particular differing digest values depend on fixture locations.

The verifier also ran npm run test:local -- src/domain/referenceSourceGraph.test.ts: 27/27 passed. The filesystem relocation test cited above was inspected, not executed; the CLI script exercised the production behavior.

Typed outcome or recovery expected

Keep successful partial inventories and content_state: "unreadable" with actionable permission details. Across relocated equivalent trees, both detail modes should have equal manifests and omit their selected absolute root. Keep the readable control equal and root-free. These are proposed restoration checks; no repair is claimed.

Analysis or suspected cause

Source-derived: src/reference/ReferenceSourceReaderFile.ts:298-304 obtains the read-failure detail; src/reference/ReferenceSourceReaderErrors.ts:44-54 retains the OS message; src/application/ReferenceSourceImportEntries.ts:33-35,55-95 incorporates it into entry limitations. The full graph commitment includes these limitations.

Preserve substantive diagnostic distinctions. src/domain/referenceSourceGraph.test.ts:262-284 deliberately checks that diagnostic changes affect the full graph digest while leaving root identity unchanged. Ignoring all limitations would lose that guarantee. This report concerns importer-induced location dependence where only the selected root changes.

Additional context

Prepared with AI assistance and independently reproduced. NOT_ENOUGH_INFO: no last-known-good version was tested.

Upstream origin/main was fetched at 5b676ba9f302228b44c28ab076525abed3e647c1. Its reader, reference-source importer, graph implementation/tests, and CLI utility command file have no changes since the tested baseline. Changes in docs/cli.md concern other commands and output filtering. This freshness check used source comparison, without a fresh runtime replay on that later revision.

Searches included open and closed issues and PRs. #3401 concerns an unreadable selected root that fails import; #5399 concerns missing fields in root-admission errors; #4013 concerns symlinks aborting import. Here both roots are readable and both imports succeed. PR #770 preserves observed entry failure details, and merged PR #2768 adds summaries; neither reports location-dependent identities for equivalent successful imports. Useful diagnostics should remain preserved.

Originally created by @coygeek on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5620 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `node scripts/rea.mjs import-reference-source ROOT --detail summary --format json` succeeds for two identical source trees containing an unreadable regular file, but their `graph_sha256` and `manifest_id` differ solely because the retained EACCES diagnostic contains each selected absolute root. Their `root_sha256` values match. Explicit `--detail full` output also contains the root; replacing each root with the same neutral alias makes those full graphs equal. An independent verifier reproduced this with both detail modes and a readable control. All eight CLI calls exited 0. A second execution of the exact script below reproduced the same results. ### Reproduction From the baseline source checkout with its locked dependencies and compiled runtime available, run the following executed reproduction as a non-root user on a POSIX filesystem that enforces file permissions. It creates disposable fixtures and isolated HOME/XDG/TMPDIR directories. No provider, configuration file, or application database is required. Paths are generated; no original local path is included here. ```sh python3 - <<'PY' import json import os from pathlib import Path import subprocess import tempfile if os.geteuid() == 0: raise SystemExit("Run as a non-root user for the permission fixture") work = Path(tempfile.mkdtemp(prefix="reference-relocation-")).resolve() env = {"PATH": os.environ["PATH"]} for key, name in { "HOME": "home", "XDG_CONFIG_HOME": "config", "XDG_DATA_HOME": "data", "XDG_CACHE_HOME": "cache", "XDG_STATE_HOME": "state", "TMPDIR": "temp", }.items(): directory = work / name directory.mkdir() env[key] = str(directory) roots = [work / "left", work / "right"] for root in roots: root.mkdir() (root / "keep.txt").write_bytes(b"ok\n") blocked = root / "unreadable.txt" blocked.write_bytes(b"blocked\n") blocked.chmod(0) for phase in ("unreadable", "readable"): if phase == "readable": for root in roots: (root / "unreadable.txt").chmod(0o600) records = {} for detail in ("summary", "full"): outputs = [] for root in roots: result = subprocess.run( ["node", "scripts/rea.mjs", "import-reference-source", str(root), "--detail", detail, "--format", "json"], env=env, capture_output=True, text=True, check=True, ) stem = f"{phase}-{detail}-{root.name}" (work / f"{stem}.json").write_text(result.stdout) (work / f"{stem}.stderr").write_text(result.stderr) outputs.append(json.loads(result.stdout)) records[detail] = outputs left, right = records["summary"] graph_left, graph_right = records["full"] entry = next(e for e in graph_left["entries"] if e["path"] == "unreadable.txt") print(phase, json.dumps({ "root_digest_equal": left["manifest"]["root_sha256"] == right["manifest"]["root_sha256"], "graph_digest_equal": left["manifest"]["graph_sha256"] == right["manifest"]["graph_sha256"], "manifest_id_equal": left["manifest"]["manifest_id"] == right["manifest"]["manifest_id"], "root_in_full": str(roots[0]) in json.dumps(graph_left), "root_in_summary": str(roots[0]) in json.dumps(left), "content_state": entry["content_state"], "equal_after_root_replacement": json.dumps(graph_left, sort_keys=True).replace(str(roots[0]), "$REFERENCE_ROOT") == json.dumps(graph_right, sort_keys=True).replace(str(roots[1]), "$REFERENCE_ROOT"), })) PY ``` ### Expected behavior Equivalent imports under different selected roots should have equal graph commitments and manifest IDs without embedding the selected absolute root in the committed inventory. Keep the file explicitly unreadable, with useful permission-failure details and recovery guidance. Summary should describe the same committed inventory as full output. This expectation is source-derived: `src/domain/referenceSourceGraph.ts:559-566` specifies no absolute root in the graph commitment and a relocation-independent manifest. `tests/boundary/filesystem/referenceSourceImport.test.ts:403-434` checks relocated manifest equality and absence of the selected root. `docs/cli.md:277-300` describes summary as a view of the same committed inventory. ### Impact and scope P2. Relocating an otherwise equivalent inventory changes its retained evidence identity and includes a local absolute path in full and summary results. The observed scope is a regular-file EACCES failure on macOS. Other filesystem failure kinds and platforms were not reproduced; no credential exposure or security boundary crossing was established. Making both files readable restored equality in the control. Its availability as a workaround for affected callers is unverified because they may lack permission; excluding the file changes the inventory. ### REA version 6.7.0; source commit `074768901f3cdc8205b61773fb1faf6f6c7b3ca8`, using the prebuilt checkout runtime. Latest published release checked: `rea-agents-6.7.0`. ### Runtime and platform Node.js v24.22.0; macOS 27.0.1 arm64; ordinary non-root user. No Hopper/Ghidra provider involved. ### Target and artifact context Two local source directories with identical `keep.txt` and `unreadable.txt` bytes. The latter has mode `000` for the failure and `0600` for the readable control. Both root directories remain readable and imports succeed. ### Interface CLI ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The script prints these comparisons for each phase: ```text unreadable {"root_digest_equal": true, "graph_digest_equal": false, "manifest_id_equal": false, "root_in_full": true, "root_in_summary": true, "content_state": "unreadable", "equal_after_root_replacement": true} readable {"root_digest_equal": true, "graph_digest_equal": true, "manifest_id_equal": true, "root_in_full": false, "root_in_summary": false, "content_state": "hashed", "equal_after_root_replacement": true} ``` The verifier retained stdout/stderr for each import and compared the complete saved graphs after root-only replacement. The unreadable entry's limitation contains an EACCES open diagnostic naming its absolute location, followed by permission-recovery guidance. Particular differing digest values depend on fixture locations. The verifier also ran `npm run test:local -- src/domain/referenceSourceGraph.test.ts`: 27/27 passed. The filesystem relocation test cited above was inspected, not executed; the CLI script exercised the production behavior. ### Typed outcome or recovery expected Keep successful partial inventories and `content_state: "unreadable"` with actionable permission details. Across relocated equivalent trees, both detail modes should have equal manifests and omit their selected absolute root. Keep the readable control equal and root-free. These are proposed restoration checks; no repair is claimed. ### Analysis or suspected cause Source-derived: `src/reference/ReferenceSourceReaderFile.ts:298-304` obtains the read-failure detail; `src/reference/ReferenceSourceReaderErrors.ts:44-54` retains the OS message; `src/application/ReferenceSourceImportEntries.ts:33-35,55-95` incorporates it into entry limitations. The full graph commitment includes these limitations. Preserve substantive diagnostic distinctions. `src/domain/referenceSourceGraph.test.ts:262-284` deliberately checks that diagnostic changes affect the full graph digest while leaving root identity unchanged. Ignoring all limitations would lose that guarantee. This report concerns importer-induced location dependence where only the selected root changes. ### Additional context Prepared with AI assistance and independently reproduced. NOT_ENOUGH_INFO: no last-known-good version was tested. Upstream `origin/main` was fetched at `5b676ba9f302228b44c28ab076525abed3e647c1`. Its reader, reference-source importer, graph implementation/tests, and CLI utility command file have no changes since the tested baseline. Changes in `docs/cli.md` concern other commands and output filtering. This freshness check used source comparison, without a fresh runtime replay on that later revision. Searches included open and closed issues and PRs. #3401 concerns an unreadable selected root that fails import; #5399 concerns missing fields in root-admission errors; #4013 concerns symlinks aborting import. Here both roots are readable and both imports succeed. PR #770 preserves observed entry failure details, and merged PR #2768 adds summaries; neither reports location-dependent identities for equivalent successful imports. Useful diagnostics should remain preserved.

bsakaguchi closed this issue

2026-10-31 00:40:02 +00:00

bsakaguchi commented

2026-10-31 00:40:03 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with canonical issue #5399 () and sibling #5619. All relate to reference-source manifest generation, root path relativization, file exclusion logic (), and deterministic error representation in . Cataloged under Cluster BJ in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with canonical issue #5399 () and sibling #5619. All relate to reference-source manifest generation, root path relativization, file exclusion logic (), and deterministic error representation in . Cataloged under Cluster BJ in consolidation tracker #4330.

bsakaguchi commented

2026-10-31 00:40:03 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 20, 2026):

TL;DR: This still happens on main on Linux. Take two identical trees, each with one unreadable file. They get different graph digests and manifest IDs. The unreadable file's error text holds each tree's absolute root.

Checked on main dceefa22c with Node 22.23.1, 24.18.0 and 26.11.1. I ran the report's script as a normal user, with an isolated HOME, XDG and TMPDIR. All eight CLI calls per Node exited 0, and the results matched on all three Node versions.

Phase	`graph_sha256` equal	`manifest_id` equal	Root in full output	Root in summary output	`content_state`
`unreadable.txt` at mode 000	false	false	true	true	`unreadable`
Same file at mode 0600 (control)	true	true	false	false	`hashed`

root_sha256 was equal in both phases.
After swapping each root for the same alias, the two full graphs were equal in both phases.
The limitation for the unreadable file says File could not be read safely: EACCES: permission denied, open '<root>/unreadable.txt' This file could not be read. Check its permissions, then try again.

Not run

macOS, other filesystem failure kinds, a fix, and the repo test files.

Thanks, coygeek, for the clear report and the script.

Commands

The report's Python script, adapted to call the built CLI by absolute path:

python3 probe.py <node> 
# per phase and detail: <node> scripts/rea.mjs import-reference-source <root> --detail <summary|full> --format json

Files were created under /tmp, then unreadable.txt was set to mode 000 and later to 0600. Output per Node, trimmed:

unreadable exit codes [0, 0, 0, 0] {"root_digest_equal": true, "graph_digest_equal": false, "manifest_id_equal": false, "root_in_full": true, "root_in_summary": true, "content_state": "unreadable", "equal_after_root_replacement": true}
readable exit codes [0, 0, 0, 0] {"root_digest_equal": true, "graph_digest_equal": true, "manifest_id_equal": true, "root_in_full": false, "root_in_summary": false, "content_state": "hashed", "equal_after_root_replacement": true}

Source check: src/reference/ReferenceSourceReaderErrors.ts still builds the detail as ${operation}: ${cause.message}, so the OS message with the path is kept.

Test setup

Box: Nobara 44 (Fedora 44 base), Linux 7.2.11
CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
RAM: 32 GB DDR4-3600
Node: 22.23.1, 24.18.0, 26.11.1
Head: dceefa22c
Other: another job shared the box during the run

🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

@CassieFox23 commented on GitHub (Oct 20, 2026): **TL;DR:** This still happens on main on Linux. Take two identical trees, each with one unreadable file. They get different graph digests and manifest IDs. The unreadable file's error text holds each tree's absolute root. Checked on main dceefa22c with Node 22.23.1, 24.18.0 and 26.11.1. I ran the report's script as a normal user, with an isolated HOME, XDG and TMPDIR. All eight CLI calls per Node exited 0, and the results matched on all three Node versions. | Phase | `graph_sha256` equal | `manifest_id` equal | Root in full output | Root in summary output | `content_state` | | --- | --- | --- | --- | --- | --- | | `unreadable.txt` at mode 000 | false | false | true | true | `unreadable` | | Same file at mode 0600 (control) | true | true | false | false | `hashed` | - `root_sha256` was equal in both phases. - After swapping each root for the same alias, the two full graphs were equal in both phases. - The limitation for the unreadable file says `File could not be read safely: EACCES: permission denied, open '<root>/unreadable.txt' This file could not be read. Check its permissions, then try again.` ### Not run - macOS, other filesystem failure kinds, a fix, and the repo test files. Thanks, coygeek, for the clear report and the script. <details> <summary>Commands</summary> The report's Python script, adapted to call the built CLI by absolute path: ```sh python3 probe.py <node> # per phase and detail: <node> scripts/rea.mjs import-reference-source <root> --detail <summary|full> --format json ``` Files were created under /tmp, then `unreadable.txt` was set to mode 000 and later to 0600. Output per Node, trimmed: ```text unreadable exit codes [0, 0, 0, 0] {"root_digest_equal": true, "graph_digest_equal": false, "manifest_id_equal": false, "root_in_full": true, "root_in_summary": true, "content_state": "unreadable", "equal_after_root_replacement": true} readable exit codes [0, 0, 0, 0] {"root_digest_equal": true, "graph_digest_equal": true, "manifest_id_equal": true, "root_in_full": false, "root_in_summary": false, "content_state": "hashed", "equal_after_root_replacement": true} ``` Source check: `src/reference/ReferenceSourceReaderErrors.ts` still builds the detail as `${operation}: ${cause.message}`, so the OS message with the path is kept. </details> <details> <summary>Test setup</summary> - Box: Nobara 44 (Fedora 44 base), Linux 7.2.11 - CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads - RAM: 32 GB DDR4-3600 - Node: 22.23.1, 24.18.0, 26.11.1 - Head: dceefa22c - Other: another job shared the box during the run </details> 🤖 Written and tested with Claude Code (Sonnet 5.5, verified by Opus 5.5), posted with my OK 🦊

bsakaguchi commented

2026-10-31 00:40:04 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

Known filesystem failures now separate relocation-independent errno/operation text from exact localDiagnostic text. Entry limitations retain semantic failure/recovery; local_diagnostics retains OS paths. computeHistoricalSourceGraphSha256 excludes only local_diagnostics rather than all limitations, preserving substantive diagnostic commitments.

Implementation: ReferenceSourceReaderErrors.ts, referenceSourceGraph.ts.

The relevant regression checks in referenceSourceImport.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. Known filesystem failures now separate relocation-independent errno/operation text from exact localDiagnostic text. Entry limitations retain semantic failure/recovery; local_diagnostics retains OS paths. computeHistoricalSourceGraphSha256 excludes only local_diagnostics rather than all limitations, preserving substantive diagnostic commitments. Implementation: [ReferenceSourceReaderErrors.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/reference/ReferenceSourceReaderErrors.ts#L63), [referenceSourceGraph.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/referenceSourceGraph.ts#L575). The relevant regression checks in [referenceSourceImport.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/filesystem/referenceSourceImport.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:09:19 +00:00

[PR #3486] fix(setup): default Claude Desktop to compact MCP schemas #5774

bsakaguchi referenced this issue

2026-10-31 01:09:33 +00:00

[PR #3525] fix(mcp): recover structured tool arguments clients serialize as JSON text #5802

bsakaguchi referenced this issue

2026-11-03 13:39:25 +00:00

[PR #7119] [MERGED] fix(mcp): fit every advertised input schema within the 64 KiB client limit #7653