morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3984] [Bug] Reconstruction verification rejects artifact comparisons sourced from inspections #1579

New issue

Closed

opened 2026-10-16 21:07:24 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:07:24 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3984

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

verify_reconstruction rejects a structural artifact claim when its retained comparison was produced from complete inspect_artifact Evidence. Passing retained summaries of those same inspections to compare_artifacts reaches the same rejection. Both supported comparison routes succeed before verification reports invalid input. Comparing the nested inventory Evidence provides a working evaluation path, with the existing uncertainty limitations preserved.

Observed in two completed local sequences, both the complete-inspection route and the retained-summary route returned comparison status unchanged. Each subsequent verification returned isError: true, omitted structured content, and included this error:

{
  "error": {
    "code": "invalid_request",
    "category": "invalid_input",
    "details": {
      "operation": "verify_reconstruction",
      "issues": [{
        "path": [],
        "reason": "invalid_value",
        "message": "Comparison source Evidence has an unexpected observation type"
      }]
    }
  }
}

The nested-inventory control returned comparison status unchanged and successful verification Evidence with status unknown. Its result passed the reconstruction verification result schema. Its limitations included the same source observation appearing on both comparison sides, unavailable artifact identity, and the ZIP inventory's occurrence scope. The two byte-identical fixtures yielded identical nested inventory IDs. The direct and summary routes retained the same inspection parents and comparison identity.

Reproduction

From the baseline checkout stated under Runtime or environment, use its installed dependencies and prepared compiled runtime with the pinned Node.js on PATH. Run this script from the repository root. It creates disposable local ZIP fixtures and starts one production stdio MCP connection with isolated home and configuration directories. No external analysis provider is needed.

export REA_VERIFY_DIR="$(mktemp -d)"
node --input-type=module <<'JS'
import assert from 'node:assert/strict';
import { mkdir, writeFile, copyFile, readFile } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';
import { TextReader, Uint8ArrayWriter, ZipWriter } from '@zip.js/zip.js';

const directory = process.env.REA_VERIFY_DIR;
await mkdir(join(directory, 'home'));
await mkdir(join(directory, 'config'));
const writer = new ZipWriter(new Uint8ArrayWriter());
await writer.add('entry.txt', new TextReader('fixture\n'));
await writeFile(join(directory, 'left.zip'), await writer.close());
await copyFile(join(directory, 'left.zip'), join(directory, 'right.zip'));
assert.deepEqual(await readFile(join(directory, 'left.zip')),
                 await readFile(join(directory, 'right.zip')));
const client = new Client({ name: 'artifact-check', version: '1' });
const transport = new StdioClientTransport({
  command: process.execPath,
  args: [resolve('scripts/rea.mjs'), 'mcp'],
  cwd: process.cwd(),
  env: {
    PATH: process.env.PATH,
    HOME: join(directory, 'home'),
    XDG_CONFIG_HOME: join(directory, 'config'),
  },
  stderr: 'pipe',
});
const call = (name, args) => client.callTool({ name, arguments: args });
const evidence = (reply) => {
  assert.notEqual(reply.isError, true, JSON.stringify(reply.content));
  assert.ok(reply.structuredContent?.evidence_id);
  return reply.structuredContent;
};
const verify = (comparison) => call('verify_reconstruction', {
  specification: {
    name: 'Artifact structure',
    claims: [{
      kind: 'structural-artifact',
      claim_id: 'artifact',
      title: 'Artifact structure agrees',
      comparison_evidence_id: comparison.evidence_id,
      dimension: 'overall',
    }],
  },
});
async function compareAndVerify(label, left, right) {
  const comparison = evidence(await call('compare_artifacts', { left, right }));
  const verification = await verify(comparison);
  console.log(JSON.stringify({
    route: label,
    comparison: comparison.normalized_result.status,
    isError: verification.isError === true,
    verification: verification.structuredContent?.normalized_result ??
      JSON.parse(verification.content.find(item => item.type === 'text').text),
  }, null, 2));
}
try {
  await client.connect(transport);
  const inspections = [];
  for (const name of ['left.zip', 'right.zip']) {
    const opened = await call('open_binary', { path: join(directory, name) });
    assert.notEqual(opened.isError, true);
    inspections.push(evidence(await call('inspect_artifact', {})));
  }
  await compareAndVerify('complete inspections', ...inspections);
  const summaries = [];
  for (const inspection of inspections) {
    summaries.push(evidence(await call('inspect_analysis_view', {
      source: { kind: 'retained-evidence', evidence_id: inspection.evidence_id },
      view: { kind: 'summary' },
    })));
  }
  await compareAndVerify('retained inspection summaries', ...summaries);
  const inventories = inspections.map(item => item.normalized_result.substeps[0].evidence);
  await compareAndVerify('nested inventories control', ...inventories);
} finally {
  await client.close();
  await transport.close();
}
JS
rm -rf "$REA_VERIFY_DIR"
unset REA_VERIFY_DIR

Expected behavior

Complete inspection Evidence and its retained summaries are supported artifact comparison inputs. A finite structural-artifact claim referencing a successful retained comparison should receive verification Evidence with a pass, fail, or unknown result according to the existing authority, coverage, source independence, and residual unknown rules. Equal ZIP bytes do not justify an unconditional pass.

Impact and scope

A supported artifact comparison workflow cannot evaluate structural verification claims when the retained comparison sources are inspection parents. Both complete and summary carrier routes reproduced this bounded invalid-input failure.

Workaround status: available. Passing the nested atomic inventory Evidence to comparison produced successful, schema-valid verification in the executed control. It preserved the justified unknown result and its source and authority limitations.

REA version

rea 6.5.0 at source commit 4692bb3330

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Two byte-identical local ZIP archives, each containing entry.txt with fixture followed by a newline.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: docs/mcp-contracts.md:18-21 permits complete inspection Evidence in artifact comparison, and :193-204 and :231-242 describe retained inspection parents and summaries. src/contracts/sessionToolContracts.ts:107 permits summary inputs with retained complete parents; :137 defines finite verification and preserves unresolved claims as unknown. src/domain/reconstructionVerificationSchemas.ts:40-44 accepts the reproduced structural claim.
Failure source: The executed production stdio MCP sequence above returned the same invalid-input error for both supported carrier routes while the nested inventory control remained evaluable. Source-derived explanation: src/domain/reconstructionVerification.ts:198-213 requires artifact comparison sources to have operation inventory_artifact, rejecting the retained inspection parents before claim classification; :229 also expects an atomic inventory payload.

Source-derived reachability agrees with the observed comparison parameters. src/application/artifacts/ArtifactComparisonEvidence.ts:35-60 resolves a summary to its authenticated retained parent, and src/server/registerArtifactComparisonTool.ts:61-101 records that parent as the comparison source. src/server/registerInvestigationTools.ts:244-246 evaluates verification against the session bundle. These observations describe one consumer mismatch across two supported input routes.

Typed outcome or recovery expected

Return schema-valid verification Evidence with the justified pass, fail or unknown result rather than invalid_request. The nested inventory comparison is an observed workaround.

Analysis or suspected cause

Source-derived: src/domain/reconstructionVerification.ts:198-213 requires inventory_artifact sources although compare_artifacts accepts retained inspect_artifact parents; the payload branch at line 229 also assumes an atomic inventory.

Additional context

Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision.

Restoration check: Repeat both carrier routes and receive successful verification Evidence with a schema-valid claim result instead of an operation-type invalid-input error. Preserve evidence authentication, complete comparison closure, coverage, authority, independent-source checks, and residual unknown handling. The nested-inventory control must remain evaluable and retain its justified unknown result. A successful operation alone must not force equal bytes to pass verification.

Related closed #3171 concerned compare_artifacts rejecting summary carriers. Here comparison succeeds and its downstream verification rejects the retained inspection parent. Open #3459 concerns later inspection substeps being dropped, rather than verification rejecting the operation type.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3984 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `verify_reconstruction` rejects a structural artifact claim when its retained comparison was produced from complete `inspect_artifact` Evidence. Passing retained summaries of those same inspections to `compare_artifacts` reaches the same rejection. Both supported comparison routes succeed before verification reports invalid input. Comparing the nested inventory Evidence provides a working evaluation path, with the existing uncertainty limitations preserved. Observed in two completed local sequences, both the complete-inspection route and the retained-summary route returned comparison status `unchanged`. Each subsequent verification returned `isError: true`, omitted structured content, and included this error: ```json { "error": { "code": "invalid_request", "category": "invalid_input", "details": { "operation": "verify_reconstruction", "issues": [{ "path": [], "reason": "invalid_value", "message": "Comparison source Evidence has an unexpected observation type" }] } } } ``` The nested-inventory control returned comparison status `unchanged` and successful verification Evidence with status `unknown`. Its result passed the reconstruction verification result schema. Its limitations included the same source observation appearing on both comparison sides, unavailable artifact identity, and the ZIP inventory's occurrence scope. The two byte-identical fixtures yielded identical nested inventory IDs. The direct and summary routes retained the same inspection parents and comparison identity. ### Reproduction From the baseline checkout stated under Runtime or environment, use its installed dependencies and prepared compiled runtime with the pinned Node.js on PATH. Run this script from the repository root. It creates disposable local ZIP fixtures and starts one production stdio MCP connection with isolated home and configuration directories. No external analysis provider is needed. ```sh export REA_VERIFY_DIR="$(mktemp -d)" node --input-type=module <<'JS' import assert from 'node:assert/strict'; import { mkdir, writeFile, copyFile, readFile } from 'node:fs/promises'; import { join, resolve } from 'node:path'; import { Client } from '@modelcontextprotocol/client'; import { StdioClientTransport } from '@modelcontextprotocol/client/stdio'; import { TextReader, Uint8ArrayWriter, ZipWriter } from '@zip.js/zip.js'; const directory = process.env.REA_VERIFY_DIR; await mkdir(join(directory, 'home')); await mkdir(join(directory, 'config')); const writer = new ZipWriter(new Uint8ArrayWriter()); await writer.add('entry.txt', new TextReader('fixture\n')); await writeFile(join(directory, 'left.zip'), await writer.close()); await copyFile(join(directory, 'left.zip'), join(directory, 'right.zip')); assert.deepEqual(await readFile(join(directory, 'left.zip')), await readFile(join(directory, 'right.zip'))); const client = new Client({ name: 'artifact-check', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: [resolve('scripts/rea.mjs'), 'mcp'], cwd: process.cwd(), env: { PATH: process.env.PATH, HOME: join(directory, 'home'), XDG_CONFIG_HOME: join(directory, 'config'), }, stderr: 'pipe', }); const call = (name, args) => client.callTool({ name, arguments: args }); const evidence = (reply) => { assert.notEqual(reply.isError, true, JSON.stringify(reply.content)); assert.ok(reply.structuredContent?.evidence_id); return reply.structuredContent; }; const verify = (comparison) => call('verify_reconstruction', { specification: { name: 'Artifact structure', claims: [{ kind: 'structural-artifact', claim_id: 'artifact', title: 'Artifact structure agrees', comparison_evidence_id: comparison.evidence_id, dimension: 'overall', }], }, }); async function compareAndVerify(label, left, right) { const comparison = evidence(await call('compare_artifacts', { left, right })); const verification = await verify(comparison); console.log(JSON.stringify({ route: label, comparison: comparison.normalized_result.status, isError: verification.isError === true, verification: verification.structuredContent?.normalized_result ?? JSON.parse(verification.content.find(item => item.type === 'text').text), }, null, 2)); } try { await client.connect(transport); const inspections = []; for (const name of ['left.zip', 'right.zip']) { const opened = await call('open_binary', { path: join(directory, name) }); assert.notEqual(opened.isError, true); inspections.push(evidence(await call('inspect_artifact', {}))); } await compareAndVerify('complete inspections', ...inspections); const summaries = []; for (const inspection of inspections) { summaries.push(evidence(await call('inspect_analysis_view', { source: { kind: 'retained-evidence', evidence_id: inspection.evidence_id }, view: { kind: 'summary' }, }))); } await compareAndVerify('retained inspection summaries', ...summaries); const inventories = inspections.map(item => item.normalized_result.substeps[0].evidence); await compareAndVerify('nested inventories control', ...inventories); } finally { await client.close(); await transport.close(); } JS rm -rf "$REA_VERIFY_DIR" unset REA_VERIFY_DIR ``` ### Expected behavior Complete inspection Evidence and its retained summaries are supported artifact comparison inputs. A finite `structural-artifact` claim referencing a successful retained comparison should receive verification Evidence with a `pass`, `fail`, or `unknown` result according to the existing authority, coverage, source independence, and residual unknown rules. Equal ZIP bytes do not justify an unconditional pass. ### Impact and scope A supported artifact comparison workflow cannot evaluate structural verification claims when the retained comparison sources are inspection parents. Both complete and summary carrier routes reproduced this bounded invalid-input failure. Workaround status: available. Passing the nested atomic inventory Evidence to comparison produced successful, schema-valid verification in the executed control. It preserved the justified unknown result and its source and authority limitations. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Two byte-identical local ZIP archives, each containing entry.txt with fixture followed by a newline. ### Interface MCP ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence - Expected source: `docs/mcp-contracts.md:18-21` permits complete inspection Evidence in artifact comparison, and `:193-204` and `:231-242` describe retained inspection parents and summaries. `src/contracts/sessionToolContracts.ts:107` permits summary inputs with retained complete parents; `:137` defines finite verification and preserves unresolved claims as unknown. `src/domain/reconstructionVerificationSchemas.ts:40-44` accepts the reproduced structural claim. - Failure source: The executed production stdio MCP sequence above returned the same invalid-input error for both supported carrier routes while the nested inventory control remained evaluable. Source-derived explanation: `src/domain/reconstructionVerification.ts:198-213` requires artifact comparison sources to have operation `inventory_artifact`, rejecting the retained inspection parents before claim classification; `:229` also expects an atomic inventory payload. Source-derived reachability agrees with the observed comparison parameters. `src/application/artifacts/ArtifactComparisonEvidence.ts:35-60` resolves a summary to its authenticated retained parent, and `src/server/registerArtifactComparisonTool.ts:61-101` records that parent as the comparison source. `src/server/registerInvestigationTools.ts:244-246` evaluates verification against the session bundle. These observations describe one consumer mismatch across two supported input routes. ### Typed outcome or recovery expected Return schema-valid verification Evidence with the justified pass, fail or unknown result rather than invalid_request. The nested inventory comparison is an observed workaround. ### Analysis or suspected cause Source-derived: src/domain/reconstructionVerification.ts:198-213 requires inventory_artifact sources although compare_artifacts accepts retained inspect_artifact parents; the payload branch at line 229 also assumes an atomic inventory. ### Additional context Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750cbe16755c4f78c1f90fdf09fbaca31. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision. Restoration check: Repeat both carrier routes and receive successful verification Evidence with a schema-valid claim result instead of an operation-type invalid-input error. Preserve evidence authentication, complete comparison closure, coverage, authority, independent-source checks, and residual unknown handling. The nested-inventory control must remain evaluable and retain its justified `unknown` result. A successful operation alone must not force equal bytes to pass verification. Related closed #3171 concerned compare_artifacts rejecting summary carriers. Here comparison succeeds and its downstream verification rejects the retained inspection parent. Open #3459 concerns later inspection substeps being dropped, rather than verification rejecting the operation type.

bsakaguchi referenced this issue

2026-10-17 21:12:35 +00:00

[PR #1579] [CLOSED] fix(browser): seed storage once per scenario storage area #2986

bsakaguchi closed this issue

2026-10-31 00:31:33 +00:00

bsakaguchi commented

2026-10-31 00:31:34 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 18, 2026):

Addressed on main in github.com/morluto/rea@ee250816e1. Reconstruction verification now accepts structural-artifact comparisons sourced from inspect_artifact and uses the shared inventory validator to retain nested substep, subject and page checks. The committed regression covers CLI/MCP inspection through comparison and reconstruction verification. Closing as fixed based on source and regression-test review; I have not rerun the original runtime reproduction.

@coygeek commented on GitHub (Oct 18, 2026): Addressed on main in https://github.com/morluto/rea/commit/ee250816e18225782c52a5f3799d91b50881c92f. Reconstruction verification now accepts structural-artifact comparisons sourced from `inspect_artifact` and uses the shared inventory validator to retain nested substep, subject and page checks. The committed regression covers CLI/MCP inspection through comparison and reconstruction verification. Closing as fixed based on source and regression-test review; I have not rerun the original runtime reproduction.

bsakaguchi referenced this issue

2026-10-31 00:44:13 +00:00

[GH-ISSUE #6289] [Bug] Distinct localStorage code units receive the same digest and scenarios compare unchanged #3803

bsakaguchi referenced this issue

2026-10-31 01:05:19 +00:00

[PR #2491] [MERGED] fix(browser): preserve unknown storage inventory and fingerprint coverage #5269