[GH-ISSUE #2623] fix(reconstruction): inconsistent function comparison status produces an overall pass #939

New issue

Closed

opened 2026-10-15 18:29:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:29:44 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2623

Summary

An imported function comparison can report an unchanged overall status while one of its own dimensions reports a change. An overall structural reconstruction claim accepts that record and passes. The same source observations produce a failed claim when the comparison retains its correctly derived changed status.

Steps to reproduce

With the repository's locked dependencies installed, save the following fixture as src/domain/functionStatus.regression.test.ts and run npm run test:focused -- src/domain/functionStatus.regression.test.ts. It creates two artifact-bound function observations that differ in pseudocode, computes a valid changed comparison, then changes only its stored overall status before authenticating the comparison record. The test invokes the reconstruction domain directly. These authenticated records are also accepted by the evidence-bundle import path used to retain records for the reconstruction tool.

import { expect, it } from "vitest";
import { createEvidence } from "./evidence.js";
import { createEvidenceBundle } from "./evidenceBundle.js";
import { compareFunctions } from "./functionComparison.js";
import { functionDossierSchema } from "./hopperValues.js";
import { jsonValueSchema } from "./jsonValue.js";
import { verifyReconstruction } from "./reconstructionVerification.js";

const observation = (digit: string, pseudocode: string) => createEvidence(
  { path: `fixtures/function-${digit}`, sha256: digit.repeat(64), format: "file" },
  { id: "fixture", name: "Function fixture", version: "1" },
  {
    operation: "analyze_function", parameters: { procedure: "main" },
    result: jsonValueSchema.parse(functionDossierSchema.parse({
      procedure: {
        address: "0x1000", name: "main", classification: null,
        body: { available: false, reason: "Fixture body unavailable." },
        signature: "int main(void)", locals: [],
      },
      pseudocode, assembly: [], comments: [], callers: [], callees: [],
      incoming_references: [], outgoing_references: [], referenced_strings: [],
      referenced_names: [], native_api: null, native_value_flow: null,
      basic_blocks: [{ start: "0x1000", end: "0x1001", successors: [] }],
      limitations: [],
    })),
    confidence: "derived", authority: "shipped-artifact",
    relationshipCoverage: { status: "complete" },
  },
);

it("does not pass a function comparison with contradictory overall status", () => {
  const left = observation("1", "return 0;");
  const right = observation("2", "return 1;");
  const comparisonResult = compareFunctions(left, right);
  expect(comparisonResult.status).toBe("changed");
  const comparison = createEvidence(undefined,
    { id: "rea-function-comparison", name: "REA function comparison", version: "1" },
    {
      operation: "compare_functions", predicateType: "rea.function-comparison",
      parameters: { left_evidence_id: left.evidence_id, right_evidence_id: right.evidence_id },
      result: jsonValueSchema.parse({ ...comparisonResult, status: "unchanged" }),
      confidence: "derived", authority: "analyst-inference",
      evidenceLinks: [left.evidence_id, right.evidence_id],
    },
  );
  const verified = verifyReconstruction({ name: "Function reconstruction", claims: [{
    kind: "structural-function", claim_id: "overall", title: "Function matches",
    comparison_evidence_id: comparison.evidence_id, dimension: "overall",
  }] }, createEvidenceBundle([left, right, comparison]));
  console.log(verified.status, verified.claims.items[0]?.observed_status);
  expect(verified.status).not.toBe("pass");
});

Expected behavior

An overall claim must not pass using a function comparison whose unchanged status contradicts its changed pseudocode dimension. Verification should reject the inconsistent comparison, preserve the changed result as a failed claim, or mark the claim unknown with an explicit inconsistency limitation. Genuine unchanged comparisons must still pass.

Actual behavior

The independently executed fixture prints pass unchanged and fails the assertion that the verification status must differ from pass. Both the overall verification and its only claim pass. The pseudocode dimension remains changed in the comparison record. Separately executed controls returned fail for a consistent changed comparison and pass for a consistent unchanged comparison.

Evidence

Expected source: src/domain/functionComparisonResults.ts:68 derives overall status from function match and dimensions; .agents/skills/reverse-engineer-anything/references/evidence-workflows.md:47 limits reconstruction pass to declared comparable claims.
Failure source: src/domain/reconstructionVerification.ts:250 parses the function comparison but immediately returns the stored status for an overall claim. src/domain/functionComparisonSchemas.ts:82 validates shape without checking that aggregate status agrees with the dimensions. The complete fixture above observed a passing claim after changing only the aggregate status.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Bundle import authenticates and retains records through src/application/investigation/EvidenceLedger.ts:75; reconstruction evaluates the retained session bundle at src/server/registerInvestigationTools.ts:242. The native comparison producer derives consistent statuses, so this report concerns inconsistent imported or supplied records. The test does not exercise a native analysis provider or the MCP transport.

Restoration check

The same inconsistent comparison must yield a typed rejection, a failed overall claim, or unknown with an explicit inconsistency limitation, never pass. Preserve fail for the consistent changed comparison and pass for the consistent unchanged comparison. Also check an inconsistent unchanged aggregate with an unknown dimension so uncertainty cannot become equivalence.

Runtime or environment

REA 6.4.0 source at 8b181716ac. Node v26.10.0, npm 11.19.1, installed locked dependencies, Vitest 5.0.3. All observations were in-memory fixtures; no native provider was launched.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: An inconsistent imported function comparison produces a false overall reconstruction pass despite a recorded changed dimension. The demonstrated failure requires a supplied inconsistent record; comparisons produced normally are consistent.
Workaround status: available
Workaround basis: Callers can inspect all comparison dimensions and its function match before relying on overall verification, or regenerate the comparison from the retained source observations.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2623 ## Summary An imported function comparison can report an unchanged overall status while one of its own dimensions reports a change. An overall structural reconstruction claim accepts that record and passes. The same source observations produce a failed claim when the comparison retains its correctly derived changed status. ## Steps to reproduce With the repository's locked dependencies installed, save the following fixture as `src/domain/functionStatus.regression.test.ts` and run `npm run test:focused -- src/domain/functionStatus.regression.test.ts`. It creates two artifact-bound function observations that differ in pseudocode, computes a valid changed comparison, then changes only its stored overall status before authenticating the comparison record. The test invokes the reconstruction domain directly. These authenticated records are also accepted by the evidence-bundle import path used to retain records for the reconstruction tool. ```ts import { expect, it } from "vitest"; import { createEvidence } from "./evidence.js"; import { createEvidenceBundle } from "./evidenceBundle.js"; import { compareFunctions } from "./functionComparison.js"; import { functionDossierSchema } from "./hopperValues.js"; import { jsonValueSchema } from "./jsonValue.js"; import { verifyReconstruction } from "./reconstructionVerification.js"; const observation = (digit: string, pseudocode: string) => createEvidence( { path: `fixtures/function-${digit}`, sha256: digit.repeat(64), format: "file" }, { id: "fixture", name: "Function fixture", version: "1" }, { operation: "analyze_function", parameters: { procedure: "main" }, result: jsonValueSchema.parse(functionDossierSchema.parse({ procedure: { address: "0x1000", name: "main", classification: null, body: { available: false, reason: "Fixture body unavailable." }, signature: "int main(void)", locals: [], }, pseudocode, assembly: [], comments: [], callers: [], callees: [], incoming_references: [], outgoing_references: [], referenced_strings: [], referenced_names: [], native_api: null, native_value_flow: null, basic_blocks: [{ start: "0x1000", end: "0x1001", successors: [] }], limitations: [], })), confidence: "derived", authority: "shipped-artifact", relationshipCoverage: { status: "complete" }, }, ); it("does not pass a function comparison with contradictory overall status", () => { const left = observation("1", "return 0;"); const right = observation("2", "return 1;"); const comparisonResult = compareFunctions(left, right); expect(comparisonResult.status).toBe("changed"); const comparison = createEvidence(undefined, { id: "rea-function-comparison", name: "REA function comparison", version: "1" }, { operation: "compare_functions", predicateType: "rea.function-comparison", parameters: { left_evidence_id: left.evidence_id, right_evidence_id: right.evidence_id }, result: jsonValueSchema.parse({ ...comparisonResult, status: "unchanged" }), confidence: "derived", authority: "analyst-inference", evidenceLinks: [left.evidence_id, right.evidence_id], }, ); const verified = verifyReconstruction({ name: "Function reconstruction", claims: [{ kind: "structural-function", claim_id: "overall", title: "Function matches", comparison_evidence_id: comparison.evidence_id, dimension: "overall", }] }, createEvidenceBundle([left, right, comparison])); console.log(verified.status, verified.claims.items[0]?.observed_status); expect(verified.status).not.toBe("pass"); }); ``` ## Expected behavior An overall claim must not pass using a function comparison whose unchanged status contradicts its changed pseudocode dimension. Verification should reject the inconsistent comparison, preserve the changed result as a failed claim, or mark the claim unknown with an explicit inconsistency limitation. Genuine unchanged comparisons must still pass. ## Actual behavior The independently executed fixture prints `pass unchanged` and fails the assertion that the verification status must differ from pass. Both the overall verification and its only claim pass. The pseudocode dimension remains changed in the comparison record. Separately executed controls returned fail for a consistent changed comparison and pass for a consistent unchanged comparison. ## Evidence - Expected source: `src/domain/functionComparisonResults.ts:68` derives overall status from function match and dimensions; `.agents/skills/reverse-engineer-anything/references/evidence-workflows.md:47` limits reconstruction pass to declared comparable claims. - Failure source: `src/domain/reconstructionVerification.ts:250` parses the function comparison but immediately returns the stored status for an overall claim. `src/domain/functionComparisonSchemas.ts:82` validates shape without checking that aggregate status agrees with the dimensions. The complete fixture above observed a passing claim after changing only the aggregate status. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Bundle import authenticates and retains records through `src/application/investigation/EvidenceLedger.ts:75`; reconstruction evaluates the retained session bundle at `src/server/registerInvestigationTools.ts:242`. The native comparison producer derives consistent statuses, so this report concerns inconsistent imported or supplied records. The test does not exercise a native analysis provider or the MCP transport. ## Restoration check The same inconsistent comparison must yield a typed rejection, a failed overall claim, or unknown with an explicit inconsistency limitation, never pass. Preserve fail for the consistent changed comparison and pass for the consistent unchanged comparison. Also check an inconsistent unchanged aggregate with an unknown dimension so uncertainty cannot become equivalence. ## Runtime or environment REA 6.4.0 source at 8b181716aca2cb467be499cba226abfb5ae19141. Node v26.10.0, npm 11.19.1, installed locked dependencies, Vitest 5.0.3. All observations were in-memory fixtures; no native provider was launched. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: An inconsistent imported function comparison produces a false overall reconstruction pass despite a recorded changed dimension. The demonstrated failure requires a supplied inconsistent record; comparisons produced normally are consistent. - Workaround status: available - Workaround basis: Callers can inspect all comparison dimensions and its function match before relying on overall verification, or regenerate the comparison from the retained source observations.

bsakaguchi

2026-10-15 18:29:44 +00:00