[GH-ISSUE #2622] fix(reconstruction): later replay environments do not limit a multi-source claim #938

New issue

Closed

opened 2026-10-15 18:29:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:29:44 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2622

Summary

A behavioral reconstruction claim can pass when an imported comparison selects several process observations and a later observation used a different environment. Verification checks the first observation on each side, so a mismatch in a later source does not produce the environment limitation. This can make an incomparable declared claim look verified.

Steps to reproduce

From a checkout with locked dependencies installed, save the following test as src/domain/replayEnvironment.regression.test.ts and run npm run test:focused -- src/domain/replayEnvironment.regression.test.ts. The test constructs four authenticated process-capture records and a comparison that explicitly selects two observations per side. It exercises the verification domain used by the reconstruction tool; the same records can be retained through the evidence-bundle import operation.

import { expect, it } from "vitest";

import { createEvidence, type Evidence } from "./evidence.js";
import { createEvidenceBundle } from "./evidenceBundle.js";
import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./process/processCaptureExample.js";
import { verifyReconstruction } from "./reconstructionVerification.js";

const environment = (id: string) => ({
  id,
  platform: "linux",
  architecture: "x86_64",
  isolation: "container" as const,
});

const capture = (digit: string, environmentId: string): Evidence =>
  createEvidence(
    { path: `fixtures/capture-${digit}`, sha256: digit.repeat(64), format: "file" },
    { id: "fixture", name: "Fixture", version: "1" },
    {
      predicateType: "rea.process-capture",
      operation: "capture_process_scenario",
      parameters: {},
      result: EMPTY_PROCESS_CAPTURE_EXAMPLE,
      confidence: "observed",
      authority: "controlled-replay",
      environment: environment(environmentId),
    },
  );

it("checks every replay environment", () => {
  const envs = ["env-a", "env-b", "env-a", "env-a"];
  const leftA = capture("1", envs[0]);
  const leftB = capture("2", envs[1]);
  const rightA = capture("3", envs[2]);
  const rightB = capture("4", envs[3]);
  const sources = [leftA, leftB, rightA, rightB];
  const comparison = createEvidence(
    undefined,
    { id: "rea-process", name: "REA deterministic process harness", version: "3" },
    {
      predicateType: "rea.process-comparison",
      operation: "compare_process_captures",
      parameters: {
        left_evidence_ids: [leftA.evidence_id, leftB.evidence_id],
        right_evidence_ids: [rightA.evidence_id, rightB.evidence_id],
      },
      result: {
        status: "unchanged",
        terminal: "unchanged",
        interaction: "unchanged",
        exit: "unchanged",
        filesystem: "unchanged",
        process: "unchanged",
        first_divergence: { status: "none" },
        limitations: [],
      },
      confidence: "derived",
      authority: "analyst-inference",
      evidenceLinks: sources.map(({ evidence_id: id }) => id),
    },
  );
  const result = verifyReconstruction(
    {
      name: "Multi-source replay",
      claims: [
        {
          kind: "behavioral",
          claim_id: "overall",
          title: "Replay unchanged",
          comparison_evidence_id: comparison.evidence_id,
          dimension: "overall",
        },
      ],
    },
    createEvidenceBundle([...sources, comparison]),
  );
  console.log(result.status, result.claims.items[0]?.limitations);
  expect(result.status).toBe("unknown");
  expect(result.claims.items[0]?.limitations).toContain(
    "Controlled replay environments are unavailable or incompatible.",
  );
});

Expected behavior

Verification should mark this claim unknown and include Controlled replay environments are unavailable or incompatible. because one selected observation has a different environment. A pass should cover only comparable observations for every declared claim.

Actual behavior

The focused test returns pass for both verification and its only claim. The claim reports observed_status: "unchanged" and only the unrelated Artifact identity is unavailable for this observation. limitation. The assertion expecting unknown fails. In separately executed controls, moving the mismatch to the first left source returns unknown with the environment limitation, while four sources with the same environment return pass.

Evidence

Expected source: .agents/skills/reverse-engineer-anything/references/evidence-workflows.md:47 limits pass to declared comparable claims; src/domain/comparisonSourceEvidence.ts:8 permits nonempty source arrays; src/domain/reconstructionVerification.ts:324 defines the environment limitation.
Failure source: src/domain/reconstructionVerification.ts:325 selects only the first left and right source for the environment check. The focused fixture above returned pass with the later mismatch and unknown with the first-source mismatch.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The public process-comparison producer currently selects one observation per side. The multi-source case is accepted by imported bundles: src/server/registerSessionRecordTools.ts:95 imports the bundle into session records, and src/server/registerInvestigationTools.ts:242 exports those records for verification. Bundle authentication and source closure checks accept this fixture; no process is launched by the test.

Restoration check

The fixture must return unknown with the environment limitation for the later mismatch. Preserve pass for all-compatible multi-source evidence and preserve unknown for a mismatch in the first source. Repeat with a missing environment in a later source and a mismatch on the right side so passing cannot depend on source order.

Runtime or environment

REA 6.4.0 source at 8b181716ac. Node v26.10.0, npm 11.19.1, installed locked dependencies, Vitest 5.0.3. Verification used in-memory evidence and did not launch a captured target.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Imported multi-source process comparisons can produce a false pass for an incomparable behavioral claim. The normal process-comparison producer uses one observation per side, which bounds the demonstrated scope.
Workaround status: available
Workaround basis: Callers can check every selected source environment before accepting a pass, or declare separate single-pair claims to retain each comparison and its environment check.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2622 ## Summary A behavioral reconstruction claim can pass when an imported comparison selects several process observations and a later observation used a different environment. Verification checks the first observation on each side, so a mismatch in a later source does not produce the environment limitation. This can make an incomparable declared claim look verified. ## Steps to reproduce From a checkout with locked dependencies installed, save the following test as `src/domain/replayEnvironment.regression.test.ts` and run `npm run test:focused -- src/domain/replayEnvironment.regression.test.ts`. The test constructs four authenticated process-capture records and a comparison that explicitly selects two observations per side. It exercises the verification domain used by the reconstruction tool; the same records can be retained through the evidence-bundle import operation. ```ts import { expect, it } from "vitest"; import { createEvidence, type Evidence } from "./evidence.js"; import { createEvidenceBundle } from "./evidenceBundle.js"; import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./process/processCaptureExample.js"; import { verifyReconstruction } from "./reconstructionVerification.js"; const environment = (id: string) => ({ id, platform: "linux", architecture: "x86_64", isolation: "container" as const, }); const capture = (digit: string, environmentId: string): Evidence => createEvidence( { path: `fixtures/capture-${digit}`, sha256: digit.repeat(64), format: "file" }, { id: "fixture", name: "Fixture", version: "1" }, { predicateType: "rea.process-capture", operation: "capture_process_scenario", parameters: {}, result: EMPTY_PROCESS_CAPTURE_EXAMPLE, confidence: "observed", authority: "controlled-replay", environment: environment(environmentId), }, ); it("checks every replay environment", () => { const envs = ["env-a", "env-b", "env-a", "env-a"]; const leftA = capture("1", envs[0]); const leftB = capture("2", envs[1]); const rightA = capture("3", envs[2]); const rightB = capture("4", envs[3]); const sources = [leftA, leftB, rightA, rightB]; const comparison = createEvidence( undefined, { id: "rea-process", name: "REA deterministic process harness", version: "3" }, { predicateType: "rea.process-comparison", operation: "compare_process_captures", parameters: { left_evidence_ids: [leftA.evidence_id, leftB.evidence_id], right_evidence_ids: [rightA.evidence_id, rightB.evidence_id], }, result: { status: "unchanged", terminal: "unchanged", interaction: "unchanged", exit: "unchanged", filesystem: "unchanged", process: "unchanged", first_divergence: { status: "none" }, limitations: [], }, confidence: "derived", authority: "analyst-inference", evidenceLinks: sources.map(({ evidence_id: id }) => id), }, ); const result = verifyReconstruction( { name: "Multi-source replay", claims: [ { kind: "behavioral", claim_id: "overall", title: "Replay unchanged", comparison_evidence_id: comparison.evidence_id, dimension: "overall", }, ], }, createEvidenceBundle([...sources, comparison]), ); console.log(result.status, result.claims.items[0]?.limitations); expect(result.status).toBe("unknown"); expect(result.claims.items[0]?.limitations).toContain( "Controlled replay environments are unavailable or incompatible.", ); }); ``` ## Expected behavior Verification should mark this claim unknown and include `Controlled replay environments are unavailable or incompatible.` because one selected observation has a different environment. A pass should cover only comparable observations for every declared claim. ## Actual behavior The focused test returns `pass` for both verification and its only claim. The claim reports `observed_status: "unchanged"` and only the unrelated `Artifact identity is unavailable for this observation.` limitation. The assertion expecting unknown fails. In separately executed controls, moving the mismatch to the first left source returns unknown with the environment limitation, while four sources with the same environment return pass. ## Evidence - Expected source: `.agents/skills/reverse-engineer-anything/references/evidence-workflows.md:47` limits pass to declared comparable claims; `src/domain/comparisonSourceEvidence.ts:8` permits nonempty source arrays; `src/domain/reconstructionVerification.ts:324` defines the environment limitation. - Failure source: `src/domain/reconstructionVerification.ts:325` selects only the first left and right source for the environment check. The focused fixture above returned pass with the later mismatch and unknown with the first-source mismatch. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The public process-comparison producer currently selects one observation per side. The multi-source case is accepted by imported bundles: `src/server/registerSessionRecordTools.ts:95` imports the bundle into session records, and `src/server/registerInvestigationTools.ts:242` exports those records for verification. Bundle authentication and source closure checks accept this fixture; no process is launched by the test. ## Restoration check The fixture must return unknown with the environment limitation for the later mismatch. Preserve pass for all-compatible multi-source evidence and preserve unknown for a mismatch in the first source. Repeat with a missing environment in a later source and a mismatch on the right side so passing cannot depend on source order. ## Runtime or environment REA 6.4.0 source at 8b181716aca2cb467be499cba226abfb5ae19141. Node v26.10.0, npm 11.19.1, installed locked dependencies, Vitest 5.0.3. Verification used in-memory evidence and did not launch a captured target. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Imported multi-source process comparisons can produce a false pass for an incomparable behavioral claim. The normal process-comparison producer uses one observation per side, which bounds the demonstrated scope. - Workaround status: available - Workaround basis: Callers can check every selected source environment before accepting a pass, or declare separate single-pair claims to retain each comparison and its environment check.

bsakaguchi

2026-10-15 18:29:44 +00:00

closed this issue
added the
area: domain

bug
labels

bsakaguchi commented

2026-10-15 18:29:45 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Reconstruction verification now checks replay-environment compatibility across every selected behavioral source, not only the first source on each side. The later-source mismatch and compatible-source controls pass on current main.

Root cause: Behavioral verification compared replay environments using only the first left and right source records.

Validation: src/domain/reconstructionVerification.test.ts: 'requires every selected behavioral source to share a replay environment' checks later-left mismatch, later-right mismatch, missing later environment, and compatible multi-source pass. These assertions passed in CI at 500e32399; the reviewed implementation is present on current main. Fix: 55dd21a5e2172187dfe7167df684f85ecb536f5e.

Verification scope: In-memory imported Evidence only; no process is launched.

@morluto commented on GitHub (Oct 14, 2026): Reconstruction verification now checks replay-environment compatibility across every selected behavioral source, not only the first source on each side. The later-source mismatch and compatible-source controls pass on current main. Root cause: Behavioral verification compared replay environments using only the first left and right source records. Validation: src/domain/reconstructionVerification.test.ts: 'requires every selected behavioral source to share a replay environment' checks later-left mismatch, later-right mismatch, missing later environment, and compatible multi-source pass. These assertions passed in [CI at 500e32399](https://github.com/morluto/rea/actions/runs/38514789942); the reviewed implementation is present on current main. Fix: [55dd21a5e2172187dfe7167df684f85ecb536f5e](https://github.com/morluto/rea/commit/55dd21a5e2172187dfe7167df684f85ecb536f5e). Verification scope: In-memory imported Evidence only; no process is launched.

bsakaguchi referenced this issue

2026-10-15 18:31:43 +00:00

[GH-ISSUE #3279] [Bug] Managed static provider marks every VB/MC++-compiled method with exception handling as malformed (small EH DataSize excludes the header) #1171

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:07:03 +00:00

[PR #938] [MERGED] fix(dotnet): validate managed metadata and CIL boundaries #2515