[GH-ISSUE #6894] [Bug] Readiness parity ignores conflicting repeated operation outcomes #4228

New issue

Closed

opened 2026-10-31 00:47:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:53 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6894

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The internal reconstruction-readiness evaluator ignores a later conflicting CLI outcome when an earlier CLI/MCP pair agrees. In the shipped example, both permission-failure outcomes report permission-required. Append another CLI outcome for the same operation and call kind with error_code: "different-error": the report still returns status: "pass", nine passed required stages, and zero findings.

A control that changes the first CLI outcome to that same error code correctly returns fail with surface-semantic-drift. Thus the mismatch is detected only when it appears in the first CLI record.

Observed directly in the source evaluator; this is an internal verifier report, not a public CLI/MCP invocation.

Reproduction

With locked dependencies installed at the revision below, place this fixture at ../evidence-state/repeated-parity-probe.mts beside a checkout named rea:

import assert from 'node:assert/strict';
import { RECONSTRUCTION_READINESS_EXAMPLE } from '../rea/src/contracts/reconstructionReadinessExample.ts';
import { createReconstructionReadinessReport } from '../rea/src/domain/reconstructionReadiness.ts';

const control = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE);
control.operation_outcomes[2]!.error_code = 'different-error';
const controlReport = createReconstructionReadinessReport(control);
assert.equal(controlReport.status, 'fail');
assert(controlReport.findings.some(x => x.code === 'surface-semantic-drift'));

const input = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE);
input.operation_outcomes.push({
  ...input.operation_outcomes[2]!,
  sequence: 4,
  error_code: 'different-error'
});
const report = createReconstructionReadinessReport(input);
console.log(JSON.stringify({
  control: { status: controlReport.status, findings: controlReport.findings },
  observed: { status: report.status, findings: report.findings }
}, null, 2));

From the checkout, run:

node --import ./node_modules/tsx/dist/loader.mjs ../evidence-state/repeated-parity-probe.mts

One original probe run completed with exit 0 and established the control failure and false pass above. It also checked the unchanged baseline and retained the exact outcomes and report summary. No post-draft reproduction is claimed.

Expected behavior

Every retained outcome must participate in the parity assessment. Either report the later semantic disagreement/unpaired observation or reject ambiguous repeated records, rather than declaring all stages passed. The documented conformance contract requires CLI/MCP parity; the existing semantic drift finding establishes that differing error codes are material.

Impact and scope

The internal report can certify parity without accounting for a conflicting recorded outcome. Demonstrated with synthetic typed outcomes only; this does not establish that an actual CLI and MCP currently return different errors, or that an existing CI run contains repeated outcomes. Frequency: one original targeted invocation, including its control.

There is no demonstrated workaround that preserves and checks every repeated outcome. Reducing input to only the first pair would lose the observation that exposes the problem.

REA version

6.7.0; tested source SHA ef475fcf8b0601c61052b7091568ba574546417d.

Runtime and platform

Node.js v24.22.1, Linux 7.2.8-9-omarchy, x86_64. Locked tsx loader; no installed REA, native provider, full build, or complete test suite used.

Target and artifact context

The source-owned readiness example and valid embedded Evidence. The added outcome copies the existing CLI permission failure, changes sequence from 2 to 4, and changes only its error code. No process capture was performed.

Interface

Other — direct source call to the internal reconstruction-readiness evaluator.

Earliest failure stage

Transport or result projection

Relevant output or evidence

Original output subset:

{
  "control": {
    "status": "fail",
    "findings": [{
      "code": "surface-semantic-drift",
      "stage_id": "export-replay",
      "status": "fail",
      "detail": "CLI/MCP structured outcomes disagree: capture-process:permission."
    }]
  },
  "observed": {
    "status": "pass",
    "summary": {
      "stages": 9,
      "required_stages": 9,
      "passed_required_stages": 9,
      "findings": 0,
      "failed_findings": 0,
      "unknown_findings": 0
    },
    "findings": []
  }
}

The full original output and fixture are retained by the reporting parent. Verifier consumer. The complete verifier/CI gate was not run with this input.

Typed outcome or recovery expected

A non-passing export-replay finding that cites the conflicting later outcome, or an input error explaining how repeated outcomes must be paired.

Analysis or suspected cause

Source-derived: groupedOutcomes groups every repeat by operation and call kind, but outcomeFindings uses .find to select only the first CLI and first MCP records. Later records contribute surface presence but are never compared for error-code drift.

Additional context

Separate open and closed searches included "readiness" "outcomes"; additional searches included "readiness" "parity" and "surface-semantic-drift". No matching report found. #2178 concerns duplicate missing-Evidence findings, and #3889 concerns a different coverage evaluator's package proofs. Last-known-good version: NOT_ENOUGH_INFO; no historical revision was tested.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The executed internal report passes despite a recorded later semantic disagreement that the control shows should fail parity assessment; observed impact is bounded to report correctness.
Workaround status: unverified
Workaround basis: A caller could independently pair and compare every repeated observation, but that was not executed; dropping repeated observations would lose relevant evidence.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6894 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The internal reconstruction-readiness evaluator ignores a later conflicting CLI outcome when an earlier CLI/MCP pair agrees. In the shipped example, both permission-failure outcomes report `permission-required`. Append another CLI outcome for the same operation and call kind with `error_code: "different-error"`: the report still returns `status: "pass"`, nine passed required stages, and zero findings. A control that changes the first CLI outcome to that same error code correctly returns `fail` with `surface-semantic-drift`. Thus the mismatch is detected only when it appears in the first CLI record. Observed directly in the source evaluator; this is an internal verifier report, not a public CLI/MCP invocation. ### Reproduction With locked dependencies installed at the revision below, place this fixture at `../evidence-state/repeated-parity-probe.mts` beside a checkout named `rea`: ```ts import assert from 'node:assert/strict'; import { RECONSTRUCTION_READINESS_EXAMPLE } from '../rea/src/contracts/reconstructionReadinessExample.ts'; import { createReconstructionReadinessReport } from '../rea/src/domain/reconstructionReadiness.ts'; const control = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE); control.operation_outcomes[2]!.error_code = 'different-error'; const controlReport = createReconstructionReadinessReport(control); assert.equal(controlReport.status, 'fail'); assert(controlReport.findings.some(x => x.code === 'surface-semantic-drift')); const input = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE); input.operation_outcomes.push({ ...input.operation_outcomes[2]!, sequence: 4, error_code: 'different-error' }); const report = createReconstructionReadinessReport(input); console.log(JSON.stringify({ control: { status: controlReport.status, findings: controlReport.findings }, observed: { status: report.status, findings: report.findings } }, null, 2)); ``` From the checkout, run: ```sh node --import ./node_modules/tsx/dist/loader.mjs ../evidence-state/repeated-parity-probe.mts ``` One original probe run completed with exit 0 and established the control failure and false pass above. It also checked the unchanged baseline and retained the exact outcomes and report summary. No post-draft reproduction is claimed. ### Expected behavior Every retained outcome must participate in the parity assessment. Either report the later semantic disagreement/unpaired observation or reject ambiguous repeated records, rather than declaring all stages passed. The [documented conformance contract](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/docs/reconstruction-readiness.md#L27-L49) requires CLI/MCP parity; the existing [semantic drift finding](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/domain/reconstructionReadinessJourneyFindings.ts#L365-L382) establishes that differing error codes are material. ### Impact and scope The internal report can certify parity without accounting for a conflicting recorded outcome. Demonstrated with synthetic typed outcomes only; this does not establish that an actual CLI and MCP currently return different errors, or that an existing CI run contains repeated outcomes. Frequency: one original targeted invocation, including its control. There is no demonstrated workaround that preserves and checks every repeated outcome. Reducing input to only the first pair would lose the observation that exposes the problem. ### REA version `6.7.0`; tested source SHA `ef475fcf8b0601c61052b7091568ba574546417d`. ### Runtime and platform Node.js `v24.22.1`, Linux `7.2.8-9-omarchy`, x86_64. Locked `tsx` loader; no installed REA, native provider, full build, or complete test suite used. ### Target and artifact context The source-owned readiness example and valid embedded Evidence. The added outcome copies the existing CLI permission failure, changes `sequence` from 2 to 4, and changes only its error code. No process capture was performed. ### Interface Other — direct source call to the internal reconstruction-readiness evaluator. ### Earliest failure stage Transport or result projection ### Relevant output or evidence Original output subset: ```json { "control": { "status": "fail", "findings": [{ "code": "surface-semantic-drift", "stage_id": "export-replay", "status": "fail", "detail": "CLI/MCP structured outcomes disagree: capture-process:permission." }] }, "observed": { "status": "pass", "summary": { "stages": 9, "required_stages": 9, "passed_required_stages": 9, "findings": 0, "failed_findings": 0, "unknown_findings": 0 }, "findings": [] } } ``` The full original output and fixture are retained by the reporting parent. [Verifier consumer](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/scripts/verify-reconstruction-readiness.mjs#L100-L127). The complete verifier/CI gate was not run with this input. ### Typed outcome or recovery expected A non-passing `export-replay` finding that cites the conflicting later outcome, or an input error explaining how repeated outcomes must be paired. ### Analysis or suspected cause Source-derived: [groupedOutcomes](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/domain/reconstructionReadinessJourneyFindings.ts#L387-L401) groups every repeat by operation and call kind, but [outcomeFindings](https://github.com/morluto/rea/blob/ef475fcf8b0601c61052b7091568ba574546417d/src/domain/reconstructionReadinessJourneyFindings.ts#L365-L382) uses `.find` to select only the first CLI and first MCP records. Later records contribute surface presence but are never compared for error-code drift. ### Additional context Separate open and closed searches included `"readiness" "outcomes"`; additional searches included `"readiness" "parity"` and `"surface-semantic-drift"`. No matching report found. [#2178](https://github.com/morluto/rea/issues/2178) concerns duplicate missing-Evidence findings, and [#3889](https://github.com/morluto/rea/issues/3889) concerns a different coverage evaluator's package proofs. Last-known-good version: NOT_ENOUGH_INFO; no historical revision was tested. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The executed internal report passes despite a recorded later semantic disagreement that the control shows should fail parity assessment; observed impact is bounded to report correctness. - Workaround status: unverified - Workaround basis: A caller could independently pair and compare every repeated observation, but that was not executed; dropping repeated observations would lose relevant evidence.

bsakaguchi closed this issue

2026-10-31 00:47:53 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:11 +00:00

[PR #4239] [MERGED] fix(javascript): accept empty semantic property seeds #6143