morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5866] [Bug] Artifact integrity comparisons mark the verified right input as contradicting evidence #3555

New issue

Closed

opened 2026-10-31 00:41:48 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:41:48 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5866

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

compare_artifacts records the verified right input as contradicting Evidence when only the left artifact has a declared-integrity mismatch. The comparison correctly returns status: "contradiction", but its automatically created residual unknown has status: "contradicted", the left inspection in supporting_evidence_ids, and the verified right inspection in contradicting_evidence_ids.

This reproduced through the registered MCP tools with two real ASAR fixtures. Their main.js member bytes are identical. The left archive's header still declares the earlier content, so its occurrence is mismatched; the independently packaged right occurrence is verified. Reversing comparison arguments reverses the supporting/contradicting roles, although the integrity mismatch stays with the same archive. list_unknowns and the exported session bundle retain those roles.

Reproduction

I executed a focused source MCP boundary test on the revision below using the real built-in ArtifactProvider, the repository's session fixture, and the MCP SDK's in-memory transport. No inventory records were fabricated and no external decompiler was involved.

The test packages left and right ASARs, replaces the left member bytes without updating its declared hash, opens and inspects each through MCP with integrity_policy: "record-and-continue", compares both argument orders, and reads the unknown ledger. One captured run passed all assertions describing the affected behavior.

Save the test body below as tests/boundary/mcp/artifactIntegrityAttribution.test.ts in the prepared checkout and run it with the pinned Node.js toolchain:

node node_modules/vitest/vitest.mjs run tests/boundary/mcp/artifactIntegrityAttribution.test.ts --maxWorkers=1

import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { createPackageWithOptions } from "@electron/asar";
import { expect, it } from "vitest";
import { ArtifactProvider } from "../../../src/artifacts/ArtifactProvider.js";
import { artifactInventoryResultSchema } from "../../../src/domain/artifactGraph.js";
import { parseEvidence, type Evidence } from "../../../src/domain/evidence.js";
import { createServer } from "../../../src/server/createServer.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";

it("records a verified right artifact as contradicting a left-only integrity mismatch", async () => {
  const root = await mkdtemp(join(process.env.REA_VERIFY_ROOT ?? tmpdir(), "artifact-integrity-"));
  const session = createTestBinarySession(new ArtifactProvider(process.env));
  const server = createServer({ kind: "session", session });
  const client = new Client({ name: "artifact-integrity-repro", version: "1" });
  try {
    const original = "console.log('ok');\n";
    const changed = "console.log('no');\n";
    const archives: string[] = [];
    for (const [side, text] of [["left", original], ["right", changed]] as const) {
      const source = join(root, side);
      await mkdir(source);
      await writeFile(join(source, "main.js"), text);
      const archive = join(root, `${side}.asar`);
      await createPackageWithOptions(source, archive, {});
      archives.push(archive);
    }
    const leftBytes = await readFile(archives[0]!);
    const offset = leftBytes.indexOf(original);
    expect(offset).toBeGreaterThanOrEqual(0);
    leftBytes.write(changed, offset, "utf8");
    await writeFile(archives[0]!, leftBytes);
    const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
    await server.connect(serverTransport);
    await client.connect(clientTransport);
    const observations: Evidence[] = [];
    const inventories = [];
    for (const archive of archives) {
      const opened = await client.callTool({ name: "open_binary", arguments: { path: archive } });
      expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
      const inspected = await client.callTool({ name: "inspect_artifact", arguments: { integrity_policy: "record-and-continue" } });
      expect(inspected.isError, JSON.stringify(inspected)).not.toBe(true);
      const observation = parseEvidence(inspected.structuredContent);
      observations.push(observation);
      const payload = observation.normalized_result as { substeps: { evidence: unknown }[] };
      inventories.push(artifactInventoryResultSchema.parse(parseEvidence(payload.substeps[0]!.evidence).normalized_result));
    }
    const leftMember = inventories[0]!.occurrences.find(row => row.logical_path === "main.js")!;
    const rightMember = inventories[1]!.occurrences.find(row => row.logical_path === "main.js")!;
    expect(leftMember.hash_status).toBe("mismatched");
    expect(rightMember.hash_status).toBe("verified");
    expect(leftMember.artifact_id).toBe(rightMember.artifact_id);
    expect(observations[0]!.evidence_id).not.toBe(observations[1]!.evidence_id);
    const compare = async (left: Evidence, right: Evidence) => {
      const response = await client.callTool({ name: "compare_artifacts", arguments: { left, right } });
      expect(response.isError, JSON.stringify(response)).not.toBe(true);
      expect(parseEvidence(response.structuredContent).normalized_result).toMatchObject({ status: "contradiction" });
      return parseEvidence(response.structuredContent);
    };
    const [left, right] = observations as [Evidence, Evidence];
    const forward = await compare(left, right);
    const reverse = await compare(right, left);
    const listed = await client.callTool({ name: "list_unknowns", arguments: { domain: "artifact-comparison" } });
    expect(listed.isError, JSON.stringify(listed)).not.toBe(true);
    const items = (listed.structuredContent as { result: { items: { question: string; status: string; supporting_evidence_ids: string[]; contradicting_evidence_ids: string[] }[] } }).result.items;
    const forwardUnknown = items.find(item => item.question.includes(forward.evidence_id))!;
    const reverseUnknown = items.find(item => item.question.includes(reverse.evidence_id))!;
    expect(forwardUnknown).toMatchObject({ status: "contradicted", supporting_evidence_ids: [left.evidence_id], contradicting_evidence_ids: [right.evidence_id] });
    expect(reverseUnknown).toMatchObject({ status: "contradicted", supporting_evidence_ids: [right.evidence_id], contradicting_evidence_ids: [left.evidence_id] });
    console.log(JSON.stringify({ memberBytesIdentical: leftMember.artifact_id === rightMember.artifact_id, leftHashStatus: leftMember.hash_status, rightHashStatus: rightMember.hash_status, forward: { status: forwardUnknown.status, contradictsVerifiedRight: forwardUnknown.contradicting_evidence_ids.includes(right.evidence_id) }, reverse: { status: reverseUnknown.status, contradictsMismatchedLeft: reverseUnknown.contradicting_evidence_ids.includes(left.evidence_id) } }));
    const stored = session.exportEvidenceBundle().unknowns;
    expect(stored).toEqual(expect.arrayContaining([forwardUnknown, reverseUnknown]));
  } finally {
    await Promise.allSettled([client.close(), server.close(), session.close()]);
    await rm(root, { recursive: true, force: true });
  }
});

Expected behavior

The unknown's evidence roles should reflect the source of the integrity conflict. A verified right input should not be labeled contradicting simply because the other input has a stale declared hash and the two Evidence IDs differ. Swapping argument order should not change which observation contains that mismatch.

Keep the correct contradiction comparison and both source citations. An open integrity question citing both observations would avoid this false cross-observation attribution; another representation is acceptable if it preserves the actual provenance.

The integrity continuation contract describes a conflict between declared and observed integrity. The comparison classifier classifies a mismatch on either input as an integrity contradiction. Repository guidance requires preserving Evidence provenance.

Impact and scope

P2. A valid artifact comparison stores incorrect supporting/contradicting roles and a derived contradicted status in its residual-unknown ledger. The forward/reverse pair demonstrated that the attribution follows argument order. This affects the retained ledger as well as the MCP list response.

The accurate occurrence hash statuses, content identities, and comparison contradiction remain available, so an analyst can inspect the original inventories to identify the actual mismatch. Those reads succeeded in the test. No automatic closure, downstream audit decision, data loss, or severe incident was exercised. Other artifact formats and transports were not tested.

REA version

Source commit 52415975942a4b9c53a817915b313ccc404f280e; package metadata version 6.7.0.

Runtime and platform

macOS 27.0.1 arm64; Node.js v24.18.0; Vitest 5.0.3. Real built-in artifact provider; no Hopper, Ghidra, or IDA required.

Target and artifact context

Two generated ASAR archives containing main.js. The left was packaged with console.log('ok'); followed by a newline, then its embedded bytes were replaced with the equal-length console.log('no'); text. The right was packaged directly from the latter content. Their observed member artifact IDs match; whole archive digests differ because the headers differ.

Interface

MCP

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The executed test printed this projection of the observed occurrence statuses and public unknown-ledger results:

{"memberBytesIdentical":true,"leftHashStatus":"mismatched","rightHashStatus":"verified","forward":{"status":"contradicted","contradictsVerifiedRight":true},"reverse":{"status":"contradicted","contradictsMismatchedLeft":true}}

Both comparisons returned status: "contradiction". Assertions checked the exact supporting and contradicting Evidence arrays in list_unknowns and confirmed that both unknown records were present unchanged in session.exportEvidenceBundle(). The focused run completed with 1 test file and 1 test passed; assertions describe the incorrect baseline behavior.

Typed outcome or recovery expected

Preserve the integrity contradiction and source records while withholding the unsupported right-side contradicting attribution. An unresolved integrity question should remain available with truthful evidence roles. Repeating the same two observations in reverse order should preserve which observation contains the declared-integrity failure.

Analysis or suspected cause

Source-derived: artifactUnknownInput treats both changed and contradiction as a cross-input disagreement whenever the source IDs differ. It always assigns the right ID to contradicting_evidence_ids. The helper receives only the source IDs and overall status, so it cannot identify which occurrence actually mismatched.

createResidualUnknown then derives contradicted from that nonempty array. The tool's recording path persists this input with the comparison Evidence. This explains the executed ledger behavior; the integrity classifier itself correctly retains the mismatch.

Additional context

Fetched upstream main at f801b2fbfd28f4e3713adefddc7bd73db44c284c; the comparison handler, artifact comparator, and residual-unknown implementation have no changes from the tested baseline. Runtime reproduction was performed on the stated baseline, and persistence on that later revision is source-derived.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5866 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `compare_artifacts` records the verified right input as contradicting Evidence when only the left artifact has a declared-integrity mismatch. The comparison correctly returns `status: "contradiction"`, but its automatically created residual unknown has `status: "contradicted"`, the left inspection in `supporting_evidence_ids`, and the verified right inspection in `contradicting_evidence_ids`. This reproduced through the registered MCP tools with two real ASAR fixtures. Their `main.js` member bytes are identical. The left archive's header still declares the earlier content, so its occurrence is `mismatched`; the independently packaged right occurrence is `verified`. Reversing comparison arguments reverses the supporting/contradicting roles, although the integrity mismatch stays with the same archive. `list_unknowns` and the exported session bundle retain those roles. ### Reproduction I executed a focused source MCP boundary test on the revision below using the real built-in `ArtifactProvider`, the repository's session fixture, and the MCP SDK's in-memory transport. No inventory records were fabricated and no external decompiler was involved. The test packages left and right ASARs, replaces the left member bytes without updating its declared hash, opens and inspects each through MCP with `integrity_policy: "record-and-continue"`, compares both argument orders, and reads the unknown ledger. One captured run passed all assertions describing the affected behavior. Save the test body below as `tests/boundary/mcp/artifactIntegrityAttribution.test.ts` in the prepared checkout and run it with the pinned Node.js toolchain: ```sh node node_modules/vitest/vitest.mjs run tests/boundary/mcp/artifactIntegrityAttribution.test.ts --maxWorkers=1 ``` ```ts import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { createPackageWithOptions } from "@electron/asar"; import { expect, it } from "vitest"; import { ArtifactProvider } from "../../../src/artifacts/ArtifactProvider.js"; import { artifactInventoryResultSchema } from "../../../src/domain/artifactGraph.js"; import { parseEvidence, type Evidence } from "../../../src/domain/evidence.js"; import { createServer } from "../../../src/server/createServer.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; it("records a verified right artifact as contradicting a left-only integrity mismatch", async () => { const root = await mkdtemp(join(process.env.REA_VERIFY_ROOT ?? tmpdir(), "artifact-integrity-")); const session = createTestBinarySession(new ArtifactProvider(process.env)); const server = createServer({ kind: "session", session }); const client = new Client({ name: "artifact-integrity-repro", version: "1" }); try { const original = "console.log('ok');\n"; const changed = "console.log('no');\n"; const archives: string[] = []; for (const [side, text] of [["left", original], ["right", changed]] as const) { const source = join(root, side); await mkdir(source); await writeFile(join(source, "main.js"), text); const archive = join(root, `${side}.asar`); await createPackageWithOptions(source, archive, {}); archives.push(archive); } const leftBytes = await readFile(archives[0]!); const offset = leftBytes.indexOf(original); expect(offset).toBeGreaterThanOrEqual(0); leftBytes.write(changed, offset, "utf8"); await writeFile(archives[0]!, leftBytes); const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); await server.connect(serverTransport); await client.connect(clientTransport); const observations: Evidence[] = []; const inventories = []; for (const archive of archives) { const opened = await client.callTool({ name: "open_binary", arguments: { path: archive } }); expect(opened.isError, JSON.stringify(opened)).not.toBe(true); const inspected = await client.callTool({ name: "inspect_artifact", arguments: { integrity_policy: "record-and-continue" } }); expect(inspected.isError, JSON.stringify(inspected)).not.toBe(true); const observation = parseEvidence(inspected.structuredContent); observations.push(observation); const payload = observation.normalized_result as { substeps: { evidence: unknown }[] }; inventories.push(artifactInventoryResultSchema.parse(parseEvidence(payload.substeps[0]!.evidence).normalized_result)); } const leftMember = inventories[0]!.occurrences.find(row => row.logical_path === "main.js")!; const rightMember = inventories[1]!.occurrences.find(row => row.logical_path === "main.js")!; expect(leftMember.hash_status).toBe("mismatched"); expect(rightMember.hash_status).toBe("verified"); expect(leftMember.artifact_id).toBe(rightMember.artifact_id); expect(observations[0]!.evidence_id).not.toBe(observations[1]!.evidence_id); const compare = async (left: Evidence, right: Evidence) => { const response = await client.callTool({ name: "compare_artifacts", arguments: { left, right } }); expect(response.isError, JSON.stringify(response)).not.toBe(true); expect(parseEvidence(response.structuredContent).normalized_result).toMatchObject({ status: "contradiction" }); return parseEvidence(response.structuredContent); }; const [left, right] = observations as [Evidence, Evidence]; const forward = await compare(left, right); const reverse = await compare(right, left); const listed = await client.callTool({ name: "list_unknowns", arguments: { domain: "artifact-comparison" } }); expect(listed.isError, JSON.stringify(listed)).not.toBe(true); const items = (listed.structuredContent as { result: { items: { question: string; status: string; supporting_evidence_ids: string[]; contradicting_evidence_ids: string[] }[] } }).result.items; const forwardUnknown = items.find(item => item.question.includes(forward.evidence_id))!; const reverseUnknown = items.find(item => item.question.includes(reverse.evidence_id))!; expect(forwardUnknown).toMatchObject({ status: "contradicted", supporting_evidence_ids: [left.evidence_id], contradicting_evidence_ids: [right.evidence_id] }); expect(reverseUnknown).toMatchObject({ status: "contradicted", supporting_evidence_ids: [right.evidence_id], contradicting_evidence_ids: [left.evidence_id] }); console.log(JSON.stringify({ memberBytesIdentical: leftMember.artifact_id === rightMember.artifact_id, leftHashStatus: leftMember.hash_status, rightHashStatus: rightMember.hash_status, forward: { status: forwardUnknown.status, contradictsVerifiedRight: forwardUnknown.contradicting_evidence_ids.includes(right.evidence_id) }, reverse: { status: reverseUnknown.status, contradictsMismatchedLeft: reverseUnknown.contradicting_evidence_ids.includes(left.evidence_id) } })); const stored = session.exportEvidenceBundle().unknowns; expect(stored).toEqual(expect.arrayContaining([forwardUnknown, reverseUnknown])); } finally { await Promise.allSettled([client.close(), server.close(), session.close()]); await rm(root, { recursive: true, force: true }); } }); ``` ### Expected behavior The unknown's evidence roles should reflect the source of the integrity conflict. A verified right input should not be labeled contradicting simply because the other input has a stale declared hash and the two Evidence IDs differ. Swapping argument order should not change which observation contains that mismatch. Keep the correct `contradiction` comparison and both source citations. An open integrity question citing both observations would avoid this false cross-observation attribution; another representation is acceptable if it preserves the actual provenance. The [integrity continuation contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/mcp-contracts.md#L936-L945) describes a conflict between declared and observed integrity. The [comparison classifier](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/artifactComparison.ts#L315-L327) classifies a mismatch on either input as an integrity contradiction. [Repository guidance](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/AGENTS.md#L3-L6) requires preserving Evidence provenance. ### Impact and scope P2. A valid artifact comparison stores incorrect supporting/contradicting roles and a derived contradicted status in its residual-unknown ledger. The forward/reverse pair demonstrated that the attribution follows argument order. This affects the retained ledger as well as the MCP list response. The accurate occurrence hash statuses, content identities, and comparison contradiction remain available, so an analyst can inspect the original inventories to identify the actual mismatch. Those reads succeeded in the test. No automatic closure, downstream audit decision, data loss, or severe incident was exercised. Other artifact formats and transports were not tested. ### REA version Source commit `52415975942a4b9c53a817915b313ccc404f280e`; package metadata version `6.7.0`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v24.18.0; Vitest 5.0.3. Real built-in artifact provider; no Hopper, Ghidra, or IDA required. ### Target and artifact context Two generated ASAR archives containing `main.js`. The left was packaged with `console.log('ok');` followed by a newline, then its embedded bytes were replaced with the equal-length `console.log('no');` text. The right was packaged directly from the latter content. Their observed member artifact IDs match; whole archive digests differ because the headers differ. ### Interface MCP ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The executed test printed this projection of the observed occurrence statuses and public unknown-ledger results: ```json {"memberBytesIdentical":true,"leftHashStatus":"mismatched","rightHashStatus":"verified","forward":{"status":"contradicted","contradictsVerifiedRight":true},"reverse":{"status":"contradicted","contradictsMismatchedLeft":true}} ``` Both comparisons returned `status: "contradiction"`. Assertions checked the exact supporting and contradicting Evidence arrays in `list_unknowns` and confirmed that both unknown records were present unchanged in `session.exportEvidenceBundle()`. The focused run completed with 1 test file and 1 test passed; assertions describe the incorrect baseline behavior. ### Typed outcome or recovery expected Preserve the integrity contradiction and source records while withholding the unsupported right-side contradicting attribution. An unresolved integrity question should remain available with truthful evidence roles. Repeating the same two observations in reverse order should preserve which observation contains the declared-integrity failure. ### Analysis or suspected cause Source-derived: [artifactUnknownInput](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/server/registerArtifactComparisonTool.ts#L105-L123) treats both `changed` and `contradiction` as a cross-input disagreement whenever the source IDs differ. It always assigns the right ID to `contradicting_evidence_ids`. The helper receives only the source IDs and overall status, so it cannot identify which occurrence actually mismatched. [createResidualUnknown](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/residualUnknown.ts#L302-L319) then derives `contradicted` from that nonempty array. The [tool's recording path](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/server/registerArtifactComparisonTool.ts#L86-L98) persists this input with the comparison Evidence. This explains the executed ledger behavior; the integrity classifier itself correctly retains the mismatch. ### Additional context Fetched upstream main at `f801b2fbfd28f4e3713adefddc7bd73db44c284c`; the comparison handler, artifact comparator, and residual-unknown implementation have no changes from the tested baseline. Runtime reproduction was performed on the stated baseline, and persistence on that later revision is source-derived.

bsakaguchi closed this issue

2026-10-31 00:41:48 +00:00

bsakaguchi commented

2026-10-31 00:41:49 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

compareArtifacts now carries the evidence IDs that actually contain integrity mismatches, and artifactUnknownInput uses that list instead of automatically contradicting the right input. Source IDs not in that list remain supporting citations. The MCP regression creates real ASAR integrity cases and checks forward/reverse argument orders, unknown ledger roles and session bundle persistence; verified peers are no longer contradicted merely by position.

Implementation: artifactComparison.ts, registerArtifactComparisonTool.ts.

The relevant regression checks in artifactMcp.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. compareArtifacts now carries the evidence IDs that actually contain integrity mismatches, and artifactUnknownInput uses that list instead of automatically contradicting the right input. Source IDs not in that list remain supporting citations. The MCP regression creates real ASAR integrity cases and checks forward/reverse argument orders, unknown ledger roles and session bundle persistence; verified peers are no longer contradicted merely by position. Implementation: [artifactComparison.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/artifactComparison.ts#L219), [registerArtifactComparisonTool.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/server/registerArtifactComparisonTool.ts#L117). The relevant regression checks in [artifactMcp.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/mcp/artifactMcp.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 00:43:13 +00:00

[GH-ISSUE #6060] [Bug] observe_web_session waits out its window after fatal CDP protocol failure #3705

bsakaguchi referenced this issue

2026-10-31 01:09:40 +00:00

[PR #3555] [MERGED] fix(browser): bound CDP handshakes and command responses #5820

bsakaguchi referenced this issue

2026-10-31 01:09:42 +00:00

[PR #3559] [CLOSED] fix(electron): block and record shell.trashItem #5823