[GH-ISSUE #7341] [Bug] Artifact comparison drops known entry-kind and executable changes when content is unhashed #4361

New issue

Closed

opened 2026-10-31 00:49:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:49:12 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7341

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Production compare_artifacts returns only an availability unknown for a ZIP member whose observed kind changes from symlink to regular file and whose executable marking changes from false to true. Both complete inventories preserve these facts. The member is present in changes, but its row has classification: "unknown" and dimensions: ["availability"]; neither known difference is represented. Reversing the inputs produces the same omission. The aggregate remains conservatively unknown, so this is not false equality.

This reproduced through public CLI inspection followed by production stdio MCP comparison, first with a sibling text entry and again after removing that sibling. Each case set included forward/reverse comparisons and three controls. No target program or analyzer was executed.

Reproduction

From the repository root at the stated baseline, use the pinned Node.js toolchain, locked dependencies and already prepared compiled runtime. Preparation was supplied; no build or installation was run during verification. Run this block. It creates one-member ZIP fixtures, isolates child home/config/cache/data/state/temp directories, performs three serial CLI inspections and five MCP comparisons, then removes its fixtures. The symlink target is inert archive text; it is neither followed nor materialized.

node --input-type=module <<'JS'
import assert from 'node:assert/strict';
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir, release } from 'node:os';
import { resolve, join } from 'node:path';
import { spawnSync } from 'node:child_process';
const require = createRequire(resolve('package.json'));
const { Client } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client')));
const { StdioClientTransport } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client/stdio')));
const { ZipWriter, TextReader, Uint8ArrayWriter } = await import(pathToFileURL(require.resolve('@zip.js/zip.js')));
const work = mkdtempSync(join(tmpdir(), 'artifact-metadata-'));
const env = { PATH: process.env.PATH, REA_LOG_LEVEL: 'silent', NODE_DISABLE_COMPILE_CACHE: '1' };
for (const key of ['HOME', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME', 'XDG_DATA_HOME', 'XDG_STATE_HOME', 'TMPDIR']) {
  env[key] = join(work, key.toLowerCase());
  mkdirSync(env[key], { recursive: true });
}
const dispatcher = resolve('scripts/rea.mjs');
const inventories = {};
const client = new Client({ name: 'artifact-metadata-check', version: '1' });
const transport = new StdioClientTransport({ command: process.execPath, args: [dispatcher, 'mcp'], env, stderr: 'pipe' });
transport.stderr?.on('data', () => {});
try {
  console.log(JSON.stringify({ runtime: process.version, platform: process.platform, arch: process.arch, release: release() }));
  for (const [name, mode, payload] of [
    ['symlink', 0o120777, 'target.txt'],
    ['file', 0o100755, 'target\n'],
    ['plain', 0o100644, 'target\n'],
  ]) {
    const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false });
    await writer.add('entry', new TextReader(payload), { unixMode: mode });
    const path = join(work, `${name}.zip`);
    writeFileSync(path, Buffer.from(await writer.close()));
    const run = spawnSync(process.execPath, [dispatcher, 'inspect-artifact', path, '--detail', 'complete', '--json'], {
      env, encoding: 'utf8', maxBuffer: 10 * 1024 * 1024, timeout: 30000,
    });
    assert.equal(run.status, 0, run.stderr + run.stdout);
    const inspection = JSON.parse(run.stdout);
    const evidence = inspection.normalized_result.substeps.find(s => s.evidence.operation === 'inventory_artifact').evidence;
    const inventory = evidence.normalized_result;
    assert.deepEqual(inventory.coverage, { scope: 'container-entries', status: 'complete' });
    assert.equal(inventory.occurrences.length, 2);
    assert.equal(inventory.nodes.length, inventory.manifest.node_count);
    assert.equal(inventory.occurrences.length, inventory.manifest.occurrence_count);
    assert.equal(inventory.edges.length, inventory.manifest.edge_count);
    inventories[name] = evidence;
    const row = inventory.occurrences.find(o => o.logical_path === 'entry');
    console.log(JSON.stringify({ name, cli_exit: run.status, entry_kind: row.entry_kind, executable: row.executable, hash_status: row.hash_status }));
  }
  await client.connect(transport);
  for (const [name, left, right] of [
    ['kind-change', 'symlink', 'file'], ['reverse', 'file', 'symlink'],
    ['same-symlink', 'symlink', 'symlink'], ['executable-control', 'file', 'plain'],
    ['same-file', 'file', 'file'],
  ]) {
    const reply = await client.callTool({ name: 'compare_artifacts', arguments: { left: inventories[left], right: inventories[right] } });
    assert(!reply.isError && reply.structuredContent, JSON.stringify(reply));
    const result = reply.structuredContent.normalized_result;
    console.log(JSON.stringify({ name, status: result.status, summary: result.summary,
      changes: result.changes.map(({ logical_path, classification, dimensions }) => ({ logical_path, classification, dimensions })) }));
  }
} finally {
  await client.close();
  await transport.close();
  rmSync(work, { recursive: true, force: true });
}
JS

The entry observations are: symlink / not-hashed / executable false; regular file / verified / executable true; control regular file / verified / executable false. Every inventory has complete container-entries coverage, two occurrences including the root, and returned counts matching its manifest. Evidence records are passed unchanged to MCP.

Expected behavior

Retain the definite entry-kind and executable differences in the per-path comparison while keeping the unobserved content identity explicitly unknown. Under the existing dimension vocabulary these facts are metadata for entry_kind and executable, with availability preserving content uncertainty. This does not require following symlinks, asserting equal content, or imposing an aggregate status that conceals remaining uncertainty.

The public contract in src/contracts/sessionToolContracts.ts:107–110 promises comparison by occurrence path, content identity, metadata and graph relations, returning every change inline. docs/tool-design.md:62–69 requires requested facts inline and preservation of partial provider facts. src/domain/artifactComparison.ts:568–577,598 already defines executable and entry-kind comparison semantics. The expectation therefore does not rest solely on the narrower ASAR storage-metadata exception.

Impact and scope

P2: a caller relying on the delta misses observed entry-kind and executable changes. The result is consequential for artifact change review even though the aggregate correctly avoids equality. Manually comparing the two retained occurrence rows is an available workaround: the reproduction reads those facts before comparison and displays the differing values. No data loss, target mutation or downstream incident was observed.

Verified scope is one-member ZIPs, unique matching paths, complete inventory coverage, no integrity mismatch and one intentionally unhashed symlink. Both directions fail in two independently constructed case sets. Other formats, platforms, installed-package behavior and real-application prevalence were not tested. Broader occurrence fields are not claimed by this report.

REA version

Source commit dbe2c97af3, package version 6.8.0, using its prepared compiled runtime.

Runtime and platform

Node.js v24.22.1; macOS/darwin arm64; Darwin release 27.0.0, observed in-process. No external analysis provider was used.

Target and artifact context

Three small static ZIP fixtures, each containing one member named entry: a Unix symlink declaration with mode 0120777, a regular text file with mode 0100755, and a regular text control with mode 0100644. The symlink carries the inert string target.txt; regular entries carry target\n. These archives are parsed only.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

All three reduced-case CLI inspections exited 0. All five MCP calls succeeded. Relevant observed results:

{
  "kind-change": {
    "status": "unknown",
    "summary": {"unchanged":0,"added":0,"removed":0,"changed":1,"unknown":1,"contradiction":0},
    "changes": [
      {"logical_path":".","classification":"changed","dimensions":["content","size"]},
      {"logical_path":"entry","classification":"unknown","dimensions":["availability"]}
    ]
  },
  "reverse": {"status":"unknown","entry_classification":"unknown","entry_dimensions":["availability"]},
  "same-symlink": {"status":"unknown","entry_classification":"unknown","entry_dimensions":["availability"]},
  "executable-control": {"status":"changed","entry_classification":"changed","entry_dimensions":["executable"]},
  "same-file": {"status":"unchanged","summary_unchanged":2,"changes":[]}
}

The last four entries above are labeled projections of the returned schema fields, not literal complete MCP payloads. ZIP limitations concern unmodeled non-entry bytes; they do not limit comparison of observed entry metadata. The root content change is preserved. The same-symlink control confirms that unknown target bytes must remain unknown; the verified-file mode control shows executable changes are compared when both hashes are verified.

Typed outcome or recovery expected

Expose the known occurrence changes together with explicit content uncertainty. Preserve occurrence/Evidence citations, integrity precedence and the valid controls. An availability-only unknown loses facts already established by the producer.

Analysis or suspected cause

Source-derived explanation: src/domain/artifactComparison.ts:392–424 returns early for an unverified hash or absent content node, except for specific storage declarations. That bypasses changedDimensions and metadataChanged, which otherwise compare executable marking and entry kind. A symlink intentionally has artifact_id: null and hash_status: "not-hashed", so this valid reader observation always takes the availability branch. src/artifacts/ZipArtifactReader.ts:338–365 preserves the entry facts; the producer is not losing them.

Additional context

Expected source: src/contracts/sessionToolContracts.ts:107–110, docs/tool-design.md:62–69, and src/domain/artifactComparison.ts:568–577,598, as scoped above. docs/mcp-contracts.md:1210–1223 separately documents storage facts independent of byte availability; that is corroborating precedent rather than the entire contract.
Failure source: the executed CLI-to-stdio-MCP reproduction and observed entry projections above; source explanation at src/domain/artifactComparison.ts:392–425.
Evidence provenance: observed. Public inspections produced unmodified authenticated inventory Evidence; source diagnosis is separately identified as derived reasoning.
Local verification: reproduced, including forward/reverse failures, symlink self-control, verified-file executable control and regular-file self-control. The exact inline block was executed successfully after removing the unrelated sibling member. No builds, installations or full suites ran.
Reproduction completeness: complete for the supported ZIP workflow. A prior bare-directory CLI attempt was rejected as target_unavailable / directory_requires_file; domain-only directory behavior is not used as proof of public reachability here.
Restoration check: repeat both directions and require the entry result to expose the known entry-kind and executable differences while preserving unavailable-content uncertainty and both occurrence citations. Keep symlink self-comparison unknown for content, keep the verified executable-only change visible, and keep file self-comparison unchanged. No fix or post-fix run has been performed.

The complete saved history and final live open/closed issue and pull-request searches found no exact duplicate. Related closed #6485 concerns equality of complete PyInstaller declarations, and closed #6484 concerns omitted type-code metadata with verified content. Those have separate producer/semantic branches and restoration checks; neither restores known symlink/file changes beside genuine content uncertainty. Closed #6706 concerns source-map relation identity, closed #2173 concerns complete root-only coverage, and closed #3956 concerns duplicate-path summary accounting. None matches this unique-path availability gate. Open #7089 concerns rejected summary input attribution; open #7304 concerns inconsistent imported comparison aggregates in reconstruction. All inputs and native comparison aggregates in this reproduction are admitted and internally consistent.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The supported CLI/MCP workflow omits known kind and executable differences from its analytical delta in both tested directions. The aggregate remains conservatively unknown, and no severe downstream incident was observed.
Workaround status: available
Workaround basis: Both retained inventory occurrence rows preserve the exact differing fields; the executed reproduction reads them directly before comparison, allowing safe manual comparison without asserting content equality.

Originally created by @coygeek on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7341 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Production `compare_artifacts` returns only an availability unknown for a ZIP member whose observed kind changes from symlink to regular file and whose executable marking changes from false to true. Both complete inventories preserve these facts. The member is present in `changes`, but its row has `classification: "unknown"` and `dimensions: ["availability"]`; neither known difference is represented. Reversing the inputs produces the same omission. The aggregate remains conservatively unknown, so this is not false equality. This reproduced through public CLI inspection followed by production stdio MCP comparison, first with a sibling text entry and again after removing that sibling. Each case set included forward/reverse comparisons and three controls. No target program or analyzer was executed. ## Reproduction From the repository root at the stated baseline, use the pinned Node.js toolchain, locked dependencies and already prepared compiled runtime. Preparation was supplied; no build or installation was run during verification. Run this block. It creates one-member ZIP fixtures, isolates child home/config/cache/data/state/temp directories, performs three serial CLI inspections and five MCP comparisons, then removes its fixtures. The symlink target is inert archive text; it is neither followed nor materialized. ```sh node --input-type=module <<'JS' import assert from 'node:assert/strict'; import { createRequire } from 'node:module'; import { pathToFileURL } from 'node:url'; import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; import { tmpdir, release } from 'node:os'; import { resolve, join } from 'node:path'; import { spawnSync } from 'node:child_process'; const require = createRequire(resolve('package.json')); const { Client } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client'))); const { StdioClientTransport } = await import(pathToFileURL(require.resolve('@modelcontextprotocol/client/stdio'))); const { ZipWriter, TextReader, Uint8ArrayWriter } = await import(pathToFileURL(require.resolve('@zip.js/zip.js'))); const work = mkdtempSync(join(tmpdir(), 'artifact-metadata-')); const env = { PATH: process.env.PATH, REA_LOG_LEVEL: 'silent', NODE_DISABLE_COMPILE_CACHE: '1' }; for (const key of ['HOME', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME', 'XDG_DATA_HOME', 'XDG_STATE_HOME', 'TMPDIR']) { env[key] = join(work, key.toLowerCase()); mkdirSync(env[key], { recursive: true }); } const dispatcher = resolve('scripts/rea.mjs'); const inventories = {}; const client = new Client({ name: 'artifact-metadata-check', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: [dispatcher, 'mcp'], env, stderr: 'pipe' }); transport.stderr?.on('data', () => {}); try { console.log(JSON.stringify({ runtime: process.version, platform: process.platform, arch: process.arch, release: release() })); for (const [name, mode, payload] of [ ['symlink', 0o120777, 'target.txt'], ['file', 0o100755, 'target\n'], ['plain', 0o100644, 'target\n'], ]) { const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false }); await writer.add('entry', new TextReader(payload), { unixMode: mode }); const path = join(work, `${name}.zip`); writeFileSync(path, Buffer.from(await writer.close())); const run = spawnSync(process.execPath, [dispatcher, 'inspect-artifact', path, '--detail', 'complete', '--json'], { env, encoding: 'utf8', maxBuffer: 10 * 1024 * 1024, timeout: 30000, }); assert.equal(run.status, 0, run.stderr + run.stdout); const inspection = JSON.parse(run.stdout); const evidence = inspection.normalized_result.substeps.find(s => s.evidence.operation === 'inventory_artifact').evidence; const inventory = evidence.normalized_result; assert.deepEqual(inventory.coverage, { scope: 'container-entries', status: 'complete' }); assert.equal(inventory.occurrences.length, 2); assert.equal(inventory.nodes.length, inventory.manifest.node_count); assert.equal(inventory.occurrences.length, inventory.manifest.occurrence_count); assert.equal(inventory.edges.length, inventory.manifest.edge_count); inventories[name] = evidence; const row = inventory.occurrences.find(o => o.logical_path === 'entry'); console.log(JSON.stringify({ name, cli_exit: run.status, entry_kind: row.entry_kind, executable: row.executable, hash_status: row.hash_status })); } await client.connect(transport); for (const [name, left, right] of [ ['kind-change', 'symlink', 'file'], ['reverse', 'file', 'symlink'], ['same-symlink', 'symlink', 'symlink'], ['executable-control', 'file', 'plain'], ['same-file', 'file', 'file'], ]) { const reply = await client.callTool({ name: 'compare_artifacts', arguments: { left: inventories[left], right: inventories[right] } }); assert(!reply.isError && reply.structuredContent, JSON.stringify(reply)); const result = reply.structuredContent.normalized_result; console.log(JSON.stringify({ name, status: result.status, summary: result.summary, changes: result.changes.map(({ logical_path, classification, dimensions }) => ({ logical_path, classification, dimensions })) })); } } finally { await client.close(); await transport.close(); rmSync(work, { recursive: true, force: true }); } JS ``` The `entry` observations are: symlink / not-hashed / executable false; regular file / verified / executable true; control regular file / verified / executable false. Every inventory has complete container-entries coverage, two occurrences including the root, and returned counts matching its manifest. Evidence records are passed unchanged to MCP. ## Expected behavior Retain the definite entry-kind and executable differences in the per-path comparison while keeping the unobserved content identity explicitly unknown. Under the existing dimension vocabulary these facts are `metadata` for `entry_kind` and `executable`, with `availability` preserving content uncertainty. This does not require following symlinks, asserting equal content, or imposing an aggregate status that conceals remaining uncertainty. The public contract in `src/contracts/sessionToolContracts.ts:107–110` promises comparison by occurrence path, content identity, metadata and graph relations, returning every change inline. `docs/tool-design.md:62–69` requires requested facts inline and preservation of partial provider facts. `src/domain/artifactComparison.ts:568–577,598` already defines executable and entry-kind comparison semantics. The expectation therefore does not rest solely on the narrower ASAR storage-metadata exception. ## Impact and scope P2: a caller relying on the delta misses observed entry-kind and executable changes. The result is consequential for artifact change review even though the aggregate correctly avoids equality. Manually comparing the two retained occurrence rows is an available workaround: the reproduction reads those facts before comparison and displays the differing values. No data loss, target mutation or downstream incident was observed. Verified scope is one-member ZIPs, unique matching paths, complete inventory coverage, no integrity mismatch and one intentionally unhashed symlink. Both directions fail in two independently constructed case sets. Other formats, platforms, installed-package behavior and real-application prevalence were not tested. Broader occurrence fields are not claimed by this report. ## REA version Source commit dbe2c97af34361b11b9504e37611dd47715b52cb, package version 6.8.0, using its prepared compiled runtime. ## Runtime and platform Node.js v24.22.1; macOS/darwin arm64; Darwin release 27.0.0, observed in-process. No external analysis provider was used. ## Target and artifact context Three small static ZIP fixtures, each containing one member named `entry`: a Unix symlink declaration with mode 0120777, a regular text file with mode 0100755, and a regular text control with mode 0100644. The symlink carries the inert string `target.txt`; regular entries carry `target\n`. These archives are parsed only. ## Interface MCP ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence All three reduced-case CLI inspections exited 0. All five MCP calls succeeded. Relevant observed results: ```json { "kind-change": { "status": "unknown", "summary": {"unchanged":0,"added":0,"removed":0,"changed":1,"unknown":1,"contradiction":0}, "changes": [ {"logical_path":".","classification":"changed","dimensions":["content","size"]}, {"logical_path":"entry","classification":"unknown","dimensions":["availability"]} ] }, "reverse": {"status":"unknown","entry_classification":"unknown","entry_dimensions":["availability"]}, "same-symlink": {"status":"unknown","entry_classification":"unknown","entry_dimensions":["availability"]}, "executable-control": {"status":"changed","entry_classification":"changed","entry_dimensions":["executable"]}, "same-file": {"status":"unchanged","summary_unchanged":2,"changes":[]} } ``` The last four entries above are labeled projections of the returned schema fields, not literal complete MCP payloads. ZIP limitations concern unmodeled non-entry bytes; they do not limit comparison of observed entry metadata. The root content change is preserved. The same-symlink control confirms that unknown target bytes must remain unknown; the verified-file mode control shows executable changes are compared when both hashes are verified. ## Typed outcome or recovery expected Expose the known occurrence changes together with explicit content uncertainty. Preserve occurrence/Evidence citations, integrity precedence and the valid controls. An availability-only unknown loses facts already established by the producer. ## Analysis or suspected cause Source-derived explanation: `src/domain/artifactComparison.ts:392–424` returns early for an unverified hash or absent content node, except for specific storage declarations. That bypasses `changedDimensions` and `metadataChanged`, which otherwise compare executable marking and entry kind. A symlink intentionally has `artifact_id: null` and `hash_status: "not-hashed"`, so this valid reader observation always takes the availability branch. `src/artifacts/ZipArtifactReader.ts:338–365` preserves the entry facts; the producer is not losing them. ## Additional context - Expected source: `src/contracts/sessionToolContracts.ts:107–110`, `docs/tool-design.md:62–69`, and `src/domain/artifactComparison.ts:568–577,598`, as scoped above. `docs/mcp-contracts.md:1210–1223` separately documents storage facts independent of byte availability; that is corroborating precedent rather than the entire contract. - Failure source: the executed CLI-to-stdio-MCP reproduction and observed entry projections above; source explanation at `src/domain/artifactComparison.ts:392–425`. - Evidence provenance: observed. Public inspections produced unmodified authenticated inventory Evidence; source diagnosis is separately identified as derived reasoning. - Local verification: reproduced, including forward/reverse failures, symlink self-control, verified-file executable control and regular-file self-control. The exact inline block was executed successfully after removing the unrelated sibling member. No builds, installations or full suites ran. - Reproduction completeness: complete for the supported ZIP workflow. A prior bare-directory CLI attempt was rejected as `target_unavailable` / `directory_requires_file`; domain-only directory behavior is not used as proof of public reachability here. - Restoration check: repeat both directions and require the `entry` result to expose the known entry-kind and executable differences while preserving unavailable-content uncertainty and both occurrence citations. Keep symlink self-comparison unknown for content, keep the verified executable-only change visible, and keep file self-comparison unchanged. No fix or post-fix run has been performed. The complete saved history and final live open/closed issue and pull-request searches found no exact duplicate. Related closed [#6485](https://github.com/morluto/rea/issues/6485) concerns equality of complete PyInstaller declarations, and closed [#6484](https://github.com/morluto/rea/issues/6484) concerns omitted type-code metadata with verified content. Those have separate producer/semantic branches and restoration checks; neither restores known symlink/file changes beside genuine content uncertainty. Closed [#6706](https://github.com/morluto/rea/issues/6706) concerns source-map relation identity, closed [#2173](https://github.com/morluto/rea/pull/2173) concerns complete root-only coverage, and closed [#3956](https://github.com/morluto/rea/issues/3956) concerns duplicate-path summary accounting. None matches this unique-path availability gate. Open [#7089](https://github.com/morluto/rea/issues/7089) concerns rejected summary input attribution; open [#7304](https://github.com/morluto/rea/issues/7304) concerns inconsistent imported comparison aggregates in reconstruction. All inputs and native comparison aggregates in this reproduction are admitted and internally consistent. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The supported CLI/MCP workflow omits known kind and executable differences from its analytical delta in both tested directions. The aggregate remains conservatively unknown, and no severe downstream incident was observed. - Workaround status: available - Workaround basis: Both retained inventory occurrence rows preserve the exact differing fields; the executed reproduction reads them directly before comparison, allowing safe manual comparison without asserting content equality.

bsakaguchi closed this issue

2026-10-31 00:49:13 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:39 +00:00

[PR #4361] [CLOSED] feat(clients): add support for Froe client #6203