morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6218] [Bug] Overall replay verification passes an imported contradictory process dimension #3787

New issue

Closed

opened 2026-10-31 00:44:03 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:03 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6218

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

After import_evidence_bundle accepts a caller-authored canonical process-comparison record with status: "unchanged", terminal: "contradiction", all other dimensions unchanged, and first_divergence: {"status":"none"}, verify_reconstruction passes its overall behavioral claim. Changing only the claim dimension to terminal returns fail. This reproduced through the registered MCP tools in two of two runs.

The contradictory record was generated with valid content IDs and canonical manifests for this import test. Native compare_process_captures does not emit contradiction. A control that calls that public producer on the same two source captures returns unchanged, and its overall claim passes. This report concerns semantic validation of imported comparison Evidence, not natural production of a contradictory native comparison or authentication of who made an Evidence record.

Reproduction

Save the following temporary workflow test as tests/boundary/mcp/overallReplayContradiction.repro.test.ts. The verification used an equivalent temporary filename, replaced here with a neutral name. It connects the MCP SDK to the production server registration over in-memory transport, imports a caller-authored canonical bundle, and makes the public tool requests. Its assertions intentionally capture the incorrect baseline result. It launches no target or analysis provider.

import { it, expect } from "vitest";
import { writeFile, rm } from "node:fs/promises";
import { join } from "node:path";
import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js";
import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "../../../src/domain/process/processCaptureExample.js";
import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js";
import { processCaptureComparisonSchema } from "../../../src/domain/process/processComparison.js";
import { reconstructionVerificationResultSchema } from "../../../src/domain/reconstructionVerificationSchemas.js";

it("checks imported contradictory process comparison through registered MCP tools", async () => {
  const source = (side: string) => createEvidence(undefined, PROCESS_PROVIDER, {
    predicateType: "rea.process-capture", operation: "capture_process_scenario",
    parameters: { side }, result: EMPTY_PROCESS_CAPTURE_EXAMPLE,
    confidence: "observed", authority: "controlled-replay",
    environment: { id: "fixture", platform: "fixture", architecture: "fixture", isolation: "process" },
  });
  const left = source("left");
  const right = source("right");
  const result = {
    status: "unchanged", terminal: "contradiction", interaction: "unchanged",
    exit: "unchanged", filesystem: "unchanged", process: "unchanged",
    first_divergence: { status: "none" }, limitations: [],
  } as const;
  expect(processCaptureComparisonSchema.safeParse(result).success).toBe(true);
  const comparison = createEvidence(undefined, PROCESS_PROVIDER, {
    predicateType: "rea.process-comparison", operation: "compare_process_captures",
    parameters: { left_evidence_id: left.evidence_id, right_evidence_id: right.evidence_id },
    result, confidence: "derived", authority: "analyst-inference",
    evidenceLinks: [left.evidence_id, right.evidence_id],
  });
  const path = join(process.env.XDG_CACHE_HOME!, "comparison-bundle.json");
  await writeFile(path, JSON.stringify(createEvidenceBundle([left, right, comparison])));
  let launches = 0;
  const session = createTestBinarySession(() => { launches++; throw new Error("No provider should launch"); });
  const harness = await createApplicationMcpHarness({ session });
  const verify = async (evidenceId: string, dimension: "overall" | "terminal") => {
    const reply = await harness.client.callTool({ name: "verify_reconstruction", arguments: {
      specification: { name: "Replay agreement", claims: [{ kind: "behavioral", claim_id: dimension,
        title: "Replay agrees", comparison_evidence_id: evidenceId, dimension }] },
    } });
    expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
    return reconstructionVerificationResultSchema.parse(parseEvidence(reply.structuredContent).normalized_result);
  };
  try {
    const imported = await harness.client.callTool({ name: "import_evidence_bundle", arguments: { path } });
    expect(imported.isError, JSON.stringify(imported)).not.toBe(true);
    expect(session.evidenceById(comparison.evidence_id)).toEqual(comparison);
    const overall = await verify(comparison.evidence_id, "overall");
    const terminal = await verify(comparison.evidence_id, "terminal");
    const nativeReply = await harness.client.callTool({ name: "compare_process_captures", arguments: { left, right } });
    expect(nativeReply.isError, JSON.stringify(nativeReply)).not.toBe(true);
    const native = parseEvidence(nativeReply.structuredContent);
    const nativeResult = processCaptureComparisonSchema.parse(native.normalized_result);
    const control = await verify(native.evidence_id, "overall");
    console.log(JSON.stringify({ schema_accepts_contradiction: true,
      imported_record_retained: true,
      overall: { status: overall.status, observed: overall.claims.items[0].observed_status },
      terminal: { status: terminal.status, observed: terminal.claims.items[0].observed_status },
      native_comparison: { status: nativeResult.status, terminal: nativeResult.terminal },
      native_overall: control.status, provider_launches: launches }));
    expect(overall.status).toBe("pass");
    expect(terminal.status).toBe("fail");
    expect(nativeResult.status).toBe("unchanged");
    expect(control.status).toBe("pass");
    expect(launches).toBe(0);
    await harness.client.ping();
  } finally { await harness.close(); await rm(path, { force: true }); }
});

From the checkout root, run the focused test with isolated configuration directories:

probe_root="$(mktemp -d)"
mkdir -p "$probe_root/home" "$probe_root/dsh" "$probe_root/config" "$probe_root/cache"
HOME="$probe_root/home" DSH_HOME="$probe_root/dsh" \
XDG_CONFIG_HOME="$probe_root/config" XDG_CACHE_HOME="$probe_root/cache" \
node node_modules/vitest/vitest.mjs run tests/boundary/mcp/overallReplayContradiction.repro.test.ts --maxWorkers=1

The runner invocation and test content match the executed reproduction, with only temporary paths and the test filename neutralized. The defect reproduced in two of two runs: one with Node v26.12.0 and one with Node v24.22.0. Delete the temporary test after running it. The fixture removes its bundle and closes its MCP client, server, and session.

Expected behavior

An overall behavioral claim must not pass when a dimension of the comparison retains contradiction. The public verification contract limits pass to complete comparable authority and keeps unresolved evidence unknown. The canonical process-comparison format includes contradiction and enforces aggregate/dimension consistency. Rejecting the inconsistent comparison, failing the claim, or keeping it unknown with an explicit reason would avoid the false pass. Genuinely unchanged comparisons should continue to pass.

Impact and scope

This is a bounded false pass for an imported-format case. A consumer relying on the overall status can accept a reconstruction claim while the same Evidence says a process dimension is contradictory. No target execution, severe downstream incident, or security boundary crossing was observed. Frequency outside the two synthetic runs is NOT_ENOUGH_INFO; ordinary native comparison production does not generate this trigger.

A safe workaround was exercised: recompute the comparison from the retained source captures with compare_process_captures, then verify that consistent result. The control returns unchanged and passes without retaining the caller-injected contradiction in the new comparison.

REA version

Source commit c465ff5a81 (main)

Runtime and platform

Node v26.12.0 and v24.22.0; macOS 27.0.1; arm64; Vitest v5.0.3 with existing locked dependencies. The repository pins Node 24.18.0; the exact pinned runtime was not used. No Hopper or Ghidra was launched.

Target and artifact context

Caller-authored canonical Evidence bundle containing two distinct observed controlled-replay capture records with the same environment and one derived analyst-inference process-comparison record. The captures use the repository's minimal valid process-capture example. No real target is involved.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Both executed focused tests exited successfully because they assert the baseline behavior. Each printed:

{"schema_accepts_contradiction":true,"imported_record_retained":true,"overall":{"status":"pass","observed":"unchanged"},"terminal":{"status":"fail","observed":"contradiction"},"native_comparison":{"status":"unchanged","terminal":"unchanged"},"native_overall":"pass","provider_launches":0}

The test also checks retained import content and a successful MCP ping after verification. This is the SDK's in-memory transport connected to production tool registration; no stdio transport or native process capture is claimed.

Expected contract: verify_reconstruction contract requires complete comparable authority. The canonical process comparison schema explicitly admits contradiction and checks aggregate consistency, while claim classification treats contradiction as failure.
Supported input boundary: import_evidence_bundle contract validates Evidence IDs and canonical manifests and imports the supplied data. The CLI Evidence documentation describes existing/supplied authenticated bundles, including MCP exports, rather than requiring records to have been emitted in the current session.

Typed outcome or recovery expected

For the imported contradictory comparison, a typed validation rejection, failed overall claim, or unknown result with an explicit inconsistency reason; never pass. Preserve the passing genuinely unchanged control and the terminal contradiction's failure semantics.

Analysis or suspected cause

Source-derived explanation: deriveProcessComparisonStatus handles truncated, changed/added/removed, and unknown dimensions but falls through to unchanged for contradiction. Its schema refinement therefore accepts the stored unchanged aggregate. comparisonObservation returns that aggregate for overall when first divergence is none. No implementation change was made or tested.

Additional context

Searched open and closed upstream issues and PRs using nine keyword sets covering compare_process_captures, contradiction, overall, processComparison, deriveProcessComparisonStatus, and imported/aggregate variants. Related #2623 and its fix #2798 concern function-comparison consistency and a different schema. #3980 concerns unknown first divergence; this baseline already has its guard, and this fixture deliberately has first divergence none. #724 concerns explanation of temporal/content differences, not an imported contradictory dimension.

Fetched upstream main at ce5bc8cb01 and inspected the relevant history. The only intervening change to the cited implementation consolidates specification digest computation; the process-comparison implementation is unchanged and the behavioral verification branch is unchanged. No duplicate or fix was found. Current main was inspected rather than runtime-tested.

Originally created by @coygeek on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6218 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior After `import_evidence_bundle` accepts a caller-authored canonical process-comparison record with `status: "unchanged"`, `terminal: "contradiction"`, all other dimensions unchanged, and `first_divergence: {"status":"none"}`, `verify_reconstruction` passes its overall behavioral claim. Changing only the claim dimension to terminal returns fail. This reproduced through the registered MCP tools in two of two runs. The contradictory record was generated with valid content IDs and canonical manifests for this import test. Native `compare_process_captures` does not emit contradiction. A control that calls that public producer on the same two source captures returns unchanged, and its overall claim passes. This report concerns semantic validation of imported comparison Evidence, not natural production of a contradictory native comparison or authentication of who made an Evidence record. ### Reproduction Save the following temporary workflow test as `tests/boundary/mcp/overallReplayContradiction.repro.test.ts`. The verification used an equivalent temporary filename, replaced here with a neutral name. It connects the MCP SDK to the production server registration over in-memory transport, imports a caller-authored canonical bundle, and makes the public tool requests. Its assertions intentionally capture the incorrect baseline result. It launches no target or analysis provider. ```ts import { it, expect } from "vitest"; import { writeFile, rm } from "node:fs/promises"; import { join } from "node:path"; import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "../../../src/domain/process/processCaptureExample.js"; import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js"; import { processCaptureComparisonSchema } from "../../../src/domain/process/processComparison.js"; import { reconstructionVerificationResultSchema } from "../../../src/domain/reconstructionVerificationSchemas.js"; it("checks imported contradictory process comparison through registered MCP tools", async () => { const source = (side: string) => createEvidence(undefined, PROCESS_PROVIDER, { predicateType: "rea.process-capture", operation: "capture_process_scenario", parameters: { side }, result: EMPTY_PROCESS_CAPTURE_EXAMPLE, confidence: "observed", authority: "controlled-replay", environment: { id: "fixture", platform: "fixture", architecture: "fixture", isolation: "process" }, }); const left = source("left"); const right = source("right"); const result = { status: "unchanged", terminal: "contradiction", interaction: "unchanged", exit: "unchanged", filesystem: "unchanged", process: "unchanged", first_divergence: { status: "none" }, limitations: [], } as const; expect(processCaptureComparisonSchema.safeParse(result).success).toBe(true); const comparison = createEvidence(undefined, PROCESS_PROVIDER, { predicateType: "rea.process-comparison", operation: "compare_process_captures", parameters: { left_evidence_id: left.evidence_id, right_evidence_id: right.evidence_id }, result, confidence: "derived", authority: "analyst-inference", evidenceLinks: [left.evidence_id, right.evidence_id], }); const path = join(process.env.XDG_CACHE_HOME!, "comparison-bundle.json"); await writeFile(path, JSON.stringify(createEvidenceBundle([left, right, comparison]))); let launches = 0; const session = createTestBinarySession(() => { launches++; throw new Error("No provider should launch"); }); const harness = await createApplicationMcpHarness({ session }); const verify = async (evidenceId: string, dimension: "overall" | "terminal") => { const reply = await harness.client.callTool({ name: "verify_reconstruction", arguments: { specification: { name: "Replay agreement", claims: [{ kind: "behavioral", claim_id: dimension, title: "Replay agrees", comparison_evidence_id: evidenceId, dimension }] }, } }); expect(reply.isError, JSON.stringify(reply)).not.toBe(true); return reconstructionVerificationResultSchema.parse(parseEvidence(reply.structuredContent).normalized_result); }; try { const imported = await harness.client.callTool({ name: "import_evidence_bundle", arguments: { path } }); expect(imported.isError, JSON.stringify(imported)).not.toBe(true); expect(session.evidenceById(comparison.evidence_id)).toEqual(comparison); const overall = await verify(comparison.evidence_id, "overall"); const terminal = await verify(comparison.evidence_id, "terminal"); const nativeReply = await harness.client.callTool({ name: "compare_process_captures", arguments: { left, right } }); expect(nativeReply.isError, JSON.stringify(nativeReply)).not.toBe(true); const native = parseEvidence(nativeReply.structuredContent); const nativeResult = processCaptureComparisonSchema.parse(native.normalized_result); const control = await verify(native.evidence_id, "overall"); console.log(JSON.stringify({ schema_accepts_contradiction: true, imported_record_retained: true, overall: { status: overall.status, observed: overall.claims.items[0].observed_status }, terminal: { status: terminal.status, observed: terminal.claims.items[0].observed_status }, native_comparison: { status: nativeResult.status, terminal: nativeResult.terminal }, native_overall: control.status, provider_launches: launches })); expect(overall.status).toBe("pass"); expect(terminal.status).toBe("fail"); expect(nativeResult.status).toBe("unchanged"); expect(control.status).toBe("pass"); expect(launches).toBe(0); await harness.client.ping(); } finally { await harness.close(); await rm(path, { force: true }); } }); ``` From the checkout root, run the focused test with isolated configuration directories: ```sh probe_root="$(mktemp -d)" mkdir -p "$probe_root/home" "$probe_root/dsh" "$probe_root/config" "$probe_root/cache" HOME="$probe_root/home" DSH_HOME="$probe_root/dsh" \ XDG_CONFIG_HOME="$probe_root/config" XDG_CACHE_HOME="$probe_root/cache" \ node node_modules/vitest/vitest.mjs run tests/boundary/mcp/overallReplayContradiction.repro.test.ts --maxWorkers=1 ``` The runner invocation and test content match the executed reproduction, with only temporary paths and the test filename neutralized. The defect reproduced in two of two runs: one with Node v26.12.0 and one with Node v24.22.0. Delete the temporary test after running it. The fixture removes its bundle and closes its MCP client, server, and session. ### Expected behavior An overall behavioral claim must not pass when a dimension of the comparison retains contradiction. The public verification contract limits pass to complete comparable authority and keeps unresolved evidence unknown. The canonical process-comparison format includes contradiction and enforces aggregate/dimension consistency. Rejecting the inconsistent comparison, failing the claim, or keeping it unknown with an explicit reason would avoid the false pass. Genuinely unchanged comparisons should continue to pass. ### Impact and scope This is a bounded false pass for an imported-format case. A consumer relying on the overall status can accept a reconstruction claim while the same Evidence says a process dimension is contradictory. No target execution, severe downstream incident, or security boundary crossing was observed. Frequency outside the two synthetic runs is NOT_ENOUGH_INFO; ordinary native comparison production does not generate this trigger. A safe workaround was exercised: recompute the comparison from the retained source captures with `compare_process_captures`, then verify that consistent result. The control returns unchanged and passes without retaining the caller-injected contradiction in the new comparison. ### REA version Source commit c465ff5a814b8e1a386fc547cbcd94df5c35f086 (main) ### Runtime and platform Node v26.12.0 and v24.22.0; macOS 27.0.1; arm64; Vitest v5.0.3 with existing locked dependencies. The repository pins Node 24.18.0; the exact pinned runtime was not used. No Hopper or Ghidra was launched. ### Target and artifact context Caller-authored canonical Evidence bundle containing two distinct observed controlled-replay capture records with the same environment and one derived analyst-inference process-comparison record. The captures use the repository's minimal valid process-capture example. No real target is involved. ### Interface MCP ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Both executed focused tests exited successfully because they assert the baseline behavior. Each printed: ```json {"schema_accepts_contradiction":true,"imported_record_retained":true,"overall":{"status":"pass","observed":"unchanged"},"terminal":{"status":"fail","observed":"contradiction"},"native_comparison":{"status":"unchanged","terminal":"unchanged"},"native_overall":"pass","provider_launches":0} ``` The test also checks retained import content and a successful MCP ping after verification. This is the SDK's in-memory transport connected to production tool registration; no stdio transport or native process capture is claimed. - Expected contract: [verify_reconstruction contract](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/contracts/sessionToolContracts.ts#L141) requires complete comparable authority. The [canonical process comparison schema](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/domain/process/processComparison.ts#L26) explicitly admits contradiction and checks aggregate consistency, while [claim classification](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/domain/reconstructionVerification.ts#L367) treats contradiction as failure. - Supported input boundary: [import_evidence_bundle contract](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/contracts/sessionToolContracts.ts#L92) validates Evidence IDs and canonical manifests and imports the supplied data. The [CLI Evidence documentation](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/docs/cli.md#L276) describes existing/supplied authenticated bundles, including MCP exports, rather than requiring records to have been emitted in the current session. ### Typed outcome or recovery expected For the imported contradictory comparison, a typed validation rejection, failed overall claim, or unknown result with an explicit inconsistency reason; never pass. Preserve the passing genuinely unchanged control and the terminal contradiction's failure semantics. ### Analysis or suspected cause Source-derived explanation: [deriveProcessComparisonStatus](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/domain/process/processComparison.ts#L47) handles truncated, changed/added/removed, and unknown dimensions but falls through to unchanged for contradiction. Its schema refinement therefore accepts the stored unchanged aggregate. [comparisonObservation](https://github.com/morluto/rea/blob/c465ff5a814b8e1a386fc547cbcd94df5c35f086/src/domain/reconstructionVerification.ts#L231) returns that aggregate for overall when first divergence is none. No implementation change was made or tested. ### Additional context Searched open and closed upstream issues and PRs using nine keyword sets covering `compare_process_captures`, `contradiction`, `overall`, `processComparison`, `deriveProcessComparisonStatus`, and imported/aggregate variants. Related #2623 and its fix #2798 concern function-comparison consistency and a different schema. #3980 concerns unknown first divergence; this baseline already has its guard, and this fixture deliberately has first divergence none. #724 concerns explanation of temporal/content differences, not an imported contradictory dimension. Fetched upstream main at ce5bc8cb0149c230e571126e2e7d9f43c86a823d and inspected the relevant history. The only intervening change to the cited implementation consolidates specification digest computation; the process-comparison implementation is unchanged and the behavioral verification branch is unchanged. No duplicate or fix was found. Current main was inspected rather than runtime-tested.

bsakaguchi closed this issue

2026-10-31 00:44:03 +00:00

bsakaguchi commented

2026-10-31 00:44:04 +00:00

Author

Owner

@gongfpp commented on GitHub (Oct 22, 2026):

I would like to work on a focused fix at the shared process-comparison aggregation/schema boundary, following #2798. The plan is to propagate contradiction rather than treating it as unchanged, preserve imported source Evidence and the unchanged native control, and add mixed-status plus MCP import/verification regressions. I propose retaining the current truncation precedence. Is anyone already working on this, or is a different precedence preferred? I noticed #724 proposes additive explanations in the same file and will keep this separate. This is AI-assisted work; the PR will remain draft pending the human owner’s review and readiness declaration.

@gongfpp commented on GitHub (Oct 22, 2026): I would like to work on a focused fix at the shared process-comparison aggregation/schema boundary, following #2798. The plan is to propagate contradiction rather than treating it as unchanged, preserve imported source Evidence and the unchanged native control, and add mixed-status plus MCP import/verification regressions. I propose retaining the current truncation precedence. Is anyone already working on this, or is a different precedence preferred? I noticed #724 proposes additive explanations in the same file and will keep this separate. This is AI-assisted work; the PR will remain draft pending the human owner’s review and readiness declaration.

bsakaguchi referenced this issue

2026-10-31 00:48:54 +00:00

[GH-ISSUE #7282] [Feature] managed-members: filter rows by declaring type or name and project columns #4321

bsakaguchi referenced this issue

2026-10-31 01:10:16 +00:00

[PR #3787] [MERGED] feat(artifacts): inspect and extract .NET single-file bundles #5899

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425