morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #3980] [Bug] Overall replay verification passes despite unresolved capture evidence #1574

New issue

Closed

opened 2026-10-16 21:07:19 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:07:19 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3980

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The public MCP verify_reconstruction tool passes an overall behavioral claim even when its process comparison explicitly says residual unknowns prevent proving that no divergence occurred. This was reproduced by passing two compatible synthetic capture Evidence records with an environment unknown through compare_process_captures, then verifying the returned comparison on the same production MCP connection. The warning remains visible in the claim limitations, but the overall claim still passes with no unknown IDs or recommended probes.

The production stdio probe completed successfully. These are the observed results, with long Evidence identifiers and the repeated diagnostic prose omitted:

complete captures: comparison unchanged; first divergence none; verification pass; claim pass; claim unknown IDs 0
environment unknown: comparison unchanged; first divergence unknown; verification pass; claim pass; claim unknown IDs 0; recommended probes 0
environment unknown after linked record_unknown: verification unknown; claim unknown; claim unknown IDs 1
terminal unknown: comparison unknown; first divergence unknown; verification unknown; claim unknown; claim unknown IDs 1; recommended probes 1

The environment comparison reports first_divergence.reason as Residual unknowns prevent proving that no divergence occurred. Its verification claim limitations contain Inherited host environment variables are not recorded. The complete control and the environment case both have identical replay-environment metadata and all five comparison dimensions unchanged. Their relevant difference is the capture-level environment unknown. The inputs have no subject identity, so all cases also retain the ordinary artifact-identity diagnostic; the complete control demonstrates that this unchanged diagnostic is independent of the reported difference.

The separate focused command npm run test:local -- src/domain/reconstructionVerification.test.ts passed all 10 existing tests. Those tests do not establish restoration of the reproduced failure.

Reproduction

Use the stated baseline with its dependencies and compiled runtime available. Run the following from the repository root. It uses a fresh MCP connection for each case. The assertions document the observed baseline behavior, including the erroneous environment-unknown pass.

probe_dir="$(mktemp -d)"
mkdir "$probe_dir/home"
HOME="$probe_dir/home" node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
import { Client } from '@modelcontextprotocol/client';
import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';
import { createEvidence, parseEvidence } from './dist/domain/evidence.js';
import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from './dist/domain/process/processCaptureExample.js';
import { PROCESS_PROVIDER } from './dist/domain/process/processEvidenceProvider.js';

const run = async scope => {
const client = new Client({ name: 'replay-verification-probe', version: '1' });
const transport = new StdioClientTransport({
  command: process.execPath,
  args: ['scripts/rea.mjs', 'mcp'],
  env: { PATH: process.env.PATH, HOME: process.env.HOME, REA_LOG_LEVEL: 'silent' },
  stderr: 'pipe',
});
let stderr = '';
transport.stderr?.on('data', data => { stderr += data; });
const call = async (name, args) => {
  const reply = await client.callTool({ name, arguments: args });
  assert.notEqual(reply.isError, true, JSON.stringify(reply));
  return reply.structuredContent;
};
const result = async (name, args) => parseEvidence(await call(name, args));
try {
  await client.connect(transport);
    const reason = scope === 'terminal'
      ? 'Terminal observation is incomplete.'
      : 'Inherited host environment variables are not recorded.';
    const capture = {
      ...EMPTY_PROCESS_CAPTURE_EXAMPLE,
      residual_unknowns: scope === null ? [] : [{ scope, reason }],
    };
    const source = side => createEvidence(undefined, PROCESS_PROVIDER, {
      predicateType: 'rea.process-capture',
      operation: 'capture_process_scenario',
      parameters: { side, case: scope },
      result: capture,
      confidence: 'observed', authority: 'controlled-replay',
      environment: {
        id: 'fixture', platform: 'fixture', architecture: 'fixture', isolation: 'process',
      },
    });
    const comparison = await result('compare_process_captures', {
      left: source('left'), right: source('right'),
    });
    const specification = { name: 'Overall replay compatibility', claims: [{
      kind: 'behavioral', claim_id: 'overall', title: 'Overall observed behavior agrees',
      comparison_evidence_id: comparison.evidence_id, dimension: 'overall',
    }] };
    const verification = (await result('verify_reconstruction', { specification })).normalized_result;
    const compared = comparison.normalized_result;
    assert.equal(compared.status, scope === 'terminal' ? 'unknown' : 'unchanged');
    assert.equal(compared.first_divergence.status, scope === null ? 'none' : 'unknown');
    assert.equal(verification.status, scope === 'terminal' ? 'unknown' : 'pass');
    if (scope !== null) assert.ok(verification.claims.items[0].limitations.includes(reason));
    console.log(JSON.stringify({ scope, comparison_status: compared.status,
      first_divergence: compared.first_divergence.status,
      verification_status: verification.status,
      claim_status: verification.claims.items[0].status,
      unknown_ids: verification.claims.items[0].unknown_ids.length,
      limitations: verification.claims.items[0].limitations,
      recommended_probes: verification.recommended_probes.length,
    }));
    if (scope === 'environment') {
      await call('record_unknown', {
        question: 'Does the unrecorded inherited environment affect this replay?',
        severity: 'high', domain: 'reconstruction-verification',
        supporting_evidence_ids: [comparison.evidence_id], contradicting_evidence_ids: [],
        required_authority: 'controlled-replay', required_confidence: 'observed',
        required_environment: null, recommended_probes: [], relationships: [],
      });
      const guarded = (await result('verify_reconstruction', { specification })).normalized_result;
      assert.equal(guarded.status, 'unknown');
      console.log(JSON.stringify({ scope: 'environment-with-recorded-unknown',
        verification_status: guarded.status, claim_status: guarded.claims.items[0].status,
        unknown_ids: guarded.claims.items[0].unknown_ids.length,
      }));
    }
} finally {
  await client.close();
  if (stderr) process.stderr.write(stderr);
}
};
for (const scope of [null, 'environment', 'terminal']) await run(scope);
NODE
rm -rf "$probe_dir"

Compare the complete, environment-unknown, and terminal-unknown results. The environment case retains unchanged dimensions but has an unknown first divergence and still passes verification. Its warning appears in the returned claim limitations.
The same script also exercises a workaround. After the environment case passes, it records an active unknown citing that comparison and repeats the unchanged verification request. That result becomes unknown.

Expected behavior

An overall behavioral claim must remain unknown when the comparison itself says residual capture unknowns prevent establishing no divergence. The public verification contract requires complete comparable authority for a pass and keeps limited or unresolved evidence unknown. The capture contract treats residual unknowns as semantic evidence rather than permission to infer equivalence from matching observations. This expectation concerns the overall claim in the supplied case; it does not assert that every diagnostic must block every independently complete dimension.

Impact and scope

An observed, bounded reconstruction-verification workflow produces a false pass for an overall claim whose comparison explicitly retains unresolved capture evidence. A consumer trusting the status can accept incomplete verification. No downstream deployment failure or severe incident was observed, and real capture execution was outside this reproduction.

Workaround status: available. The warning remains visible. The executed public record_unknown call cites the returned comparison without altering either capture or the claim; repeating the same verification then yields unknown. This preserves the unresolved result until qualifying evidence is available.

REA version

rea 6.5.0 at source commit 4692bb3330

Runtime and platform

Node 24.18.0; macOS arm64

Target and artifact context

Synthetic canonical process capture Evidence with an environment-scoped residual unknown; no target execution.

Interface

MCP

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: src/contracts/sessionToolContracts.ts:137, the verify_reconstruction contract requires complete comparable authority and keeps limited or unresolved evidence unknown; src/domain/process/processCapture.ts:198, residual capture unknowns have semantic significance when assessing equivalence.
Failure source: The observed production MCP sequence and output above; source-derived explanation at src/domain/reconstructionVerification.ts:158, :182, :260, and :398, where behavioral dimension statuses determine classification while comparison limitations are displayed without blocking the pass.

The primary evidence is the executed production stdio MCP comparison and verification sequence with canonical synthetic captures.

Supporting source-derived evidence: src/domain/process/processComparison.ts:490 derives the unknown first divergence for residual capture unknowns; src/server/registerProcessComparisonTool.ts:162 preserves comparison limitations in Evidence, while :204 adds no registry unknown when the comparison status is unchanged; src/server/registerInvestigationTools.ts:244 verifies the actual session-owned bundle.

Typed outcome or recovery expected

An overall behavioral claim should remain unknown until the unresolved capture evidence is addressed. The executed record_unknown workaround preserves that result.

Analysis or suspected cause

Source-derived: src/domain/reconstructionVerification.ts:158,182,260,398 classifies behavioral dimensions without making capture-level comparison limitations blockers. src/server/registerProcessComparisonTool.ts:204 does not register an unknown when the comparison status is unchanged.

Additional context

Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision.

Restoration check: Repeat the known input through the public MCP tools. Without manually recording a second unknown, the environment-unknown case must withhold an overall pass and explain the unresolved capture evidence. Keep the complete control passing and the terminal-unknown control unknown. Preserve the visible warning and the existing ability to record an active unknown linked to a comparison. The implementation may decide how to propagate the semantic uncertainty; merely deleting the warning or relabeling the caller's captures does not restore the contract. No claim is made about untested truncation variants or whether an unrelated unknown should block an otherwise complete dimension.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3980 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The public MCP `verify_reconstruction` tool passes an overall behavioral claim even when its process comparison explicitly says residual unknowns prevent proving that no divergence occurred. This was reproduced by passing two compatible synthetic capture Evidence records with an environment unknown through `compare_process_captures`, then verifying the returned comparison on the same production MCP connection. The warning remains visible in the claim limitations, but the overall claim still passes with no unknown IDs or recommended probes. The production stdio probe completed successfully. These are the observed results, with long Evidence identifiers and the repeated diagnostic prose omitted: ```text complete captures: comparison unchanged; first divergence none; verification pass; claim pass; claim unknown IDs 0 environment unknown: comparison unchanged; first divergence unknown; verification pass; claim pass; claim unknown IDs 0; recommended probes 0 environment unknown after linked record_unknown: verification unknown; claim unknown; claim unknown IDs 1 terminal unknown: comparison unknown; first divergence unknown; verification unknown; claim unknown; claim unknown IDs 1; recommended probes 1 ``` The environment comparison reports `first_divergence.reason` as `Residual unknowns prevent proving that no divergence occurred.` Its verification claim limitations contain `Inherited host environment variables are not recorded.` The complete control and the environment case both have identical replay-environment metadata and all five comparison dimensions unchanged. Their relevant difference is the capture-level environment unknown. The inputs have no subject identity, so all cases also retain the ordinary artifact-identity diagnostic; the complete control demonstrates that this unchanged diagnostic is independent of the reported difference. The separate focused command `npm run test:local -- src/domain/reconstructionVerification.test.ts` passed all 10 existing tests. Those tests do not establish restoration of the reproduced failure. ### Reproduction 1. Use the stated baseline with its dependencies and compiled runtime available. Run the following from the repository root. It uses a fresh MCP connection for each case. The assertions document the observed baseline behavior, including the erroneous environment-unknown pass. ```sh probe_dir="$(mktemp -d)" mkdir "$probe_dir/home" HOME="$probe_dir/home" node --input-type=module <<'NODE' import assert from 'node:assert/strict'; import { Client } from '@modelcontextprotocol/client'; import { StdioClientTransport } from '@modelcontextprotocol/client/stdio'; import { createEvidence, parseEvidence } from './dist/domain/evidence.js'; import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from './dist/domain/process/processCaptureExample.js'; import { PROCESS_PROVIDER } from './dist/domain/process/processEvidenceProvider.js'; const run = async scope => { const client = new Client({ name: 'replay-verification-probe', version: '1' }); const transport = new StdioClientTransport({ command: process.execPath, args: ['scripts/rea.mjs', 'mcp'], env: { PATH: process.env.PATH, HOME: process.env.HOME, REA_LOG_LEVEL: 'silent' }, stderr: 'pipe', }); let stderr = ''; transport.stderr?.on('data', data => { stderr += data; }); const call = async (name, args) => { const reply = await client.callTool({ name, arguments: args }); assert.notEqual(reply.isError, true, JSON.stringify(reply)); return reply.structuredContent; }; const result = async (name, args) => parseEvidence(await call(name, args)); try { await client.connect(transport); const reason = scope === 'terminal' ? 'Terminal observation is incomplete.' : 'Inherited host environment variables are not recorded.'; const capture = { ...EMPTY_PROCESS_CAPTURE_EXAMPLE, residual_unknowns: scope === null ? [] : [{ scope, reason }], }; const source = side => createEvidence(undefined, PROCESS_PROVIDER, { predicateType: 'rea.process-capture', operation: 'capture_process_scenario', parameters: { side, case: scope }, result: capture, confidence: 'observed', authority: 'controlled-replay', environment: { id: 'fixture', platform: 'fixture', architecture: 'fixture', isolation: 'process', }, }); const comparison = await result('compare_process_captures', { left: source('left'), right: source('right'), }); const specification = { name: 'Overall replay compatibility', claims: [{ kind: 'behavioral', claim_id: 'overall', title: 'Overall observed behavior agrees', comparison_evidence_id: comparison.evidence_id, dimension: 'overall', }] }; const verification = (await result('verify_reconstruction', { specification })).normalized_result; const compared = comparison.normalized_result; assert.equal(compared.status, scope === 'terminal' ? 'unknown' : 'unchanged'); assert.equal(compared.first_divergence.status, scope === null ? 'none' : 'unknown'); assert.equal(verification.status, scope === 'terminal' ? 'unknown' : 'pass'); if (scope !== null) assert.ok(verification.claims.items[0].limitations.includes(reason)); console.log(JSON.stringify({ scope, comparison_status: compared.status, first_divergence: compared.first_divergence.status, verification_status: verification.status, claim_status: verification.claims.items[0].status, unknown_ids: verification.claims.items[0].unknown_ids.length, limitations: verification.claims.items[0].limitations, recommended_probes: verification.recommended_probes.length, })); if (scope === 'environment') { await call('record_unknown', { question: 'Does the unrecorded inherited environment affect this replay?', severity: 'high', domain: 'reconstruction-verification', supporting_evidence_ids: [comparison.evidence_id], contradicting_evidence_ids: [], required_authority: 'controlled-replay', required_confidence: 'observed', required_environment: null, recommended_probes: [], relationships: [], }); const guarded = (await result('verify_reconstruction', { specification })).normalized_result; assert.equal(guarded.status, 'unknown'); console.log(JSON.stringify({ scope: 'environment-with-recorded-unknown', verification_status: guarded.status, claim_status: guarded.claims.items[0].status, unknown_ids: guarded.claims.items[0].unknown_ids.length, })); } } finally { await client.close(); if (stderr) process.stderr.write(stderr); } }; for (const scope of [null, 'environment', 'terminal']) await run(scope); NODE rm -rf "$probe_dir" ``` 2. Compare the complete, environment-unknown, and terminal-unknown results. The environment case retains unchanged dimensions but has an unknown first divergence and still passes verification. Its warning appears in the returned claim limitations. 3. The same script also exercises a workaround. After the environment case passes, it records an active unknown citing that comparison and repeats the unchanged verification request. That result becomes unknown. ### Expected behavior An overall behavioral claim must remain unknown when the comparison itself says residual capture unknowns prevent establishing no divergence. The public verification contract requires complete comparable authority for a pass and keeps limited or unresolved evidence unknown. The capture contract treats residual unknowns as semantic evidence rather than permission to infer equivalence from matching observations. This expectation concerns the overall claim in the supplied case; it does not assert that every diagnostic must block every independently complete dimension. ### Impact and scope An observed, bounded reconstruction-verification workflow produces a false pass for an overall claim whose comparison explicitly retains unresolved capture evidence. A consumer trusting the status can accept incomplete verification. No downstream deployment failure or severe incident was observed, and real capture execution was outside this reproduction. Workaround status: available. The warning remains visible. The executed public `record_unknown` call cites the returned comparison without altering either capture or the claim; repeating the same verification then yields unknown. This preserves the unresolved result until qualifying evidence is available. ### REA version rea 6.5.0 at source commit 4692bb33300c7cb77b095b6cc0c33da861c1b860 ### Runtime and platform Node 24.18.0; macOS arm64 ### Target and artifact context Synthetic canonical process capture Evidence with an environment-scoped residual unknown; no target execution. ### Interface MCP ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence - Expected source: `src/contracts/sessionToolContracts.ts:137`, the `verify_reconstruction` contract requires complete comparable authority and keeps limited or unresolved evidence unknown; `src/domain/process/processCapture.ts:198`, residual capture unknowns have semantic significance when assessing equivalence. - Failure source: The observed production MCP sequence and output above; source-derived explanation at `src/domain/reconstructionVerification.ts:158`, `:182`, `:260`, and `:398`, where behavioral dimension statuses determine classification while comparison limitations are displayed without blocking the pass. The primary evidence is the executed production stdio MCP comparison and verification sequence with canonical synthetic captures. - Supporting source-derived evidence: `src/domain/process/processComparison.ts:490` derives the unknown first divergence for residual capture unknowns; `src/server/registerProcessComparisonTool.ts:162` preserves comparison limitations in Evidence, while `:204` adds no registry unknown when the comparison status is unchanged; `src/server/registerInvestigationTools.ts:244` verifies the actual session-owned bundle. ### Typed outcome or recovery expected An overall behavioral claim should remain unknown until the unresolved capture evidence is addressed. The executed record_unknown workaround preserves that result. ### Analysis or suspected cause Source-derived: src/domain/reconstructionVerification.ts:158,182,260,398 classifies behavioral dimensions without making capture-level comparison limitations blockers. src/server/registerProcessComparisonTool.ts:204 does not register an unknown when the comparison status is unchanged. ### Additional context Freshness check: the cited failing implementation is unchanged on origin/main at fc712b3750cbe16755c4f78c1f90fdf09fbaca31. The only change to a cited reproduction entry point normalizes global CLI flag syntax. The baseline reproduction was not rerun on that revision. Restoration check: Repeat the known input through the public MCP tools. Without manually recording a second unknown, the environment-unknown case must withhold an overall pass and explain the unresolved capture evidence. Keep the complete control passing and the terminal-unknown control unknown. Preserve the visible warning and the existing ability to record an active unknown linked to a comparison. The implementation may decide how to propagate the semantic uncertainty; merely deleting the warning or relabeling the caller's captures does not restore the contract. No claim is made about untested truncation variants or whether an unrelated unknown should block an otherwise complete dimension.

bsakaguchi closed this issue

2026-10-16 21:07:20 +00:00

bsakaguchi commented

2026-10-16 21:07:21 +00:00

Author

Owner

@morluto commented on GitHub (Oct 16, 2026):

Fixed on main in 153017d75. Replay verification preserves an unknown overall verdict when residual capture uncertainty prevents proving the absence of divergence.

Domain and production stdio-MCP comparison/verification controls cover the reported false pass using synthetic captures.

Hosted main CI is still pending; this closure is based on the implemented fix and the scoped regression evidence.

@morluto commented on GitHub (Oct 16, 2026): Fixed on `main` in [153017d75](https://github.com/morluto/rea/commit/153017d750322188c1a112b2284c1b8065c67320). Replay verification preserves an unknown overall verdict when residual capture uncertainty prevents proving the absence of divergence. Domain and production stdio-MCP comparison/verification controls cover the reported false pass using synthetic captures. Hosted main CI is still pending; this closure is based on the implemented fix and the scoped regression evidence.

bsakaguchi referenced this issue

2026-10-17 21:12:32 +00:00

[PR #1575] [CLOSED] fix(artifacts): reject Windows alternate stream output paths #2983

bsakaguchi referenced this issue

2026-10-31 00:31:03 +00:00

[GH-ISSUE #3874] [Bug] Literal POSIX names beginning with a letter and colon abort artifact inventory and extraction #1493