[GH-ISSUE #2178] fix(readiness): stage-level missing-Evidence references produce identical findings #724

New issue

Closed

opened 2026-10-15 18:27:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:27:43 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2178

Summary

The internal reconstruction-readiness evaluator repeats the same missing-Evidence diagnostic for each required check in a stage. Adding one absent Evidence ID to the checked-in example's static-analysis stage produces three identical finding objects. The report's summary.findings and summary.unknown_findings both count all three copies. Maintainers inspecting the report receive redundant diagnostics whose count depends on the stage's required checks. Readiness still correctly remains unknown for the missing reference.

Affected area

Evidence, artifacts, or reconstruction. Interface: Other, the repository's internal maintainer conformance evaluator. The supported maintainer command is npm run verify:readiness; the report evaluator is not a public CLI command or MCP tool. The earliest observed failure is report generation. The target is the source-owned synthetic conformance input with an authenticated embedded Evidence bundle.

Runtime or environment

Source checkout at c22111eca2, using its existing cached build, and the installed published rea-agents 6.4.0 package were tested independently. Both reproduced the same result on Node.js 24.21.0, darwin arm64. This is a supported package runtime; the checkout's contributor pin is Node.js 24.18.0, which was not used for this verification. No native analysis provider is required for the focused reproduction.

Steps to reproduce

Use a repository checkout with dependencies and dist already built through npm ci and npm run build:cached. From its root, run the following small probe. It calls the same evaluator and checked-in example that the documented maintainer verifier uses, without running the native/CLI/MCP fixture journey. The probe changes only an in-memory clone of the input and writes nothing.

node --input-type=module <<'JS'
import assert from "node:assert/strict";
import { RECONSTRUCTION_READINESS_EXAMPLE as example } from "./dist/contracts/reconstructionReadinessExample.js";
import { createReconstructionReadinessReport as evaluate } from "./dist/domain/reconstructionReadiness.js";

const absentA = `ev_${"a".repeat(64)}`;
const absentB = `ev_${"b".repeat(64)}`;
assert.ok(!example.evidence_bundle.records.some(record =>
  [absentA, absentB].includes(record.evidence_id)));
const stage = (input, id = "static-analysis") =>
  input.stages.find(value => value.stage_id === id);
const cases = [
  ["baseline", () => {}],
  ["stage missing one ID", input => stage(input).evidence_ids.push(absentA)],
  ["one check missing one ID", input => stage(input).checks[0].evidence_ids.push(absentA)],
  ["same ID on stage and check", input => {
    stage(input).evidence_ids.push(absentA);
    stage(input).checks[0].evidence_ids.push(absentA);
  }],
  ["two IDs on one stage", input => stage(input).evidence_ids.push(absentA, absentB)],
  ["same ID on two stages", input => {
    stage(input).evidence_ids.push(absentA);
    stage(input, "discover-classify").evidence_ids.push(absentA);
  }],
  ["missing ID plus failed check", input => {
    stage(input).evidence_ids.push(absentA);
    stage(input).checks[0].status = "fail";
  }],
];
for (const [name, mutate] of cases) {
  const input = structuredClone(example);
  mutate(input);
  const report = evaluate(input);
  const missing = report.findings.filter(value =>
    value.code === "referenced-evidence-missing");
  console.log(JSON.stringify({
    name,
    status: report.status,
    missing_rows: missing.length,
    distinct_missing_rows: new Set(missing.map(value => JSON.stringify(value))).size,
    summary: report.summary,
  }));
  if (name === "stage missing one ID") console.log(JSON.stringify(missing));
}
JS

The same imports are present under the installed release package's dist; running this probe with that package directory as the working directory produced the same results. The source and release probes executed locally. The full npm run verify:readiness journey was not run during this verification. That verifier asserts pass before writing its compact summary or retained report, so this reproduction does not establish duplicated output in an actual CI artifact.

Expected behavior

The documented internal report represents failures by stage, stable reason code, Evidence IDs, and detail. A missing stage-level Evidence reference should appear once as a diagnostic for that stage and absent ID. For the one-ID case above, the report should contain one referenced-evidence-missing object and count it once. Separate IDs and separate stages must retain separate diagnostics, and missing Evidence must continue to withhold readiness.

Actual behavior

The baseline returns pass with zero findings. The one absent stage ID returns unknown with three missing-Evidence rows but only one distinct object. All five fields, code, stage_id, status, detail, and evidence_ids, are identical in those rows. The objects contain no check ID or other per-check context. The summary is {"stages":9,"required_stages":9,"passed_required_stages":8,"findings":3,"failed_findings":0,"unknown_findings":3}.

A single absent ID on one check produces one missing-Evidence row. Putting that same ID on the stage and the check still produces three identical rows. Two distinct absent IDs on one stage produce six rows containing two distinct objects. The same absent ID on two stages produces seven rows containing two distinct objects, three for static-analysis and four for discover-classify. A failed check alongside the absent stage ID still returns fail, with three unknown rows and one separate failed finding. These counts are accurate counts of emitted rows; the defect is the redundant rows and the resulting inflated diagnostic counts, not arithmetic in the summary.

Evidence

Expected source: docs/reconstruction-readiness.md:27 establishes the internal project conformance contract; docs/reconstruction-readiness.md:47 requires every stage and check to link to bundled Evidence and preserves non-pass outcomes; docs/reconstruction-readiness.md:71 defines the finding's reported context. AGENTS.md under Architecture and Cleanup requires preserving identity across boundaries. src/domain/reconstructionReadinessFindingHelpers.ts:12 already collapses repeated IDs within a single missing-reference check.
Failure source: The executed probe above produced three fully identical missing-reference objects and summary counts of three. src/domain/reconstructionReadinessJourneyFindings.ts:181 iterates required checks and src/domain/reconstructionReadinessJourneyFindings.ts:213 includes stage references in each call. src/domain/reconstructionReadiness.ts:28 retains the emitted rows and src/domain/reconstructionReadiness.ts:44 counts them. scripts/verify-reconstruction-readiness.mjs:97 uses this evaluator for the documented verifier.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Independent source-build and installed-release probes supplied the runtime evidence, and source inspection established the producing call path. The exact existing source test file, src/domain/reconstructionReadiness.test.ts, also passed all 18 tests. Additional bounded controls accepted the valid baseline and rejected malformed Evidence IDs, duplicate stage IDs, unauthenticated Evidence record IDs, and duplicate bundle records. Missing or duplicate required checks produced fail. Empty check Evidence produced unknown. Explicit stage/check fail, unknown, unsupported, truncated, and skipped inputs retained their respective aggregate non-pass status.

Restoration check

Repeat the inline probe. One absent ID on one stage must yield one missing-reference diagnostic and summary counts of one. Repeating that ID on a stage and its checks must not create indistinguishable extra diagnostics. Two absent IDs in one stage must retain two distinct diagnostics; the same absent ID in two stages must retain one diagnostic for each stage. Three different absent IDs on three checks must retain three distinct diagnostics. The missing-reference cases must still return unknown, and a separate failed or missing required check must still return fail. Preserve input-schema and Evidence-bundle authentication rejection. The restoration concerns report content and counts; it does not ask for a public adapter or a changed readiness decision.

Triage assessment

Impact level: P3
Assessment status: supported
Impact basis: The observed effect is redundant diagnostic text and counts in a maintainer conformance report. Missing Evidence remains visible and correctly withholds readiness; no lost Evidence, changed analysis result, or failure of the readiness decision was observed. The repeats recur for the tested stage/check reference placements in both builds.
Workaround status: available
Workaround basis: The probes measured the complete emitted objects and showed that deduplicating byte-identical objects preserves the distinct stage/ID diagnostics in the tested cases. A consumer can deduplicate for display and recompute diagnostic counts while retaining the original readiness status and separate failed findings.

Originally created by @coygeek on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2178 ## Summary The internal reconstruction-readiness evaluator repeats the same missing-Evidence diagnostic for each required check in a stage. Adding one absent Evidence ID to the checked-in example's `static-analysis` stage produces three identical finding objects. The report's `summary.findings` and `summary.unknown_findings` both count all three copies. Maintainers inspecting the report receive redundant diagnostics whose count depends on the stage's required checks. Readiness still correctly remains `unknown` for the missing reference. ## Affected area Evidence, artifacts, or reconstruction. Interface: Other, the repository's internal maintainer conformance evaluator. The supported maintainer command is `npm run verify:readiness`; the report evaluator is not a public CLI command or MCP tool. The earliest observed failure is report generation. The target is the source-owned synthetic conformance input with an authenticated embedded Evidence bundle. ## Runtime or environment Source checkout at c22111eca2c4bc84a084f12711f405b3e7422a40, using its existing cached build, and the installed published `rea-agents` 6.4.0 package were tested independently. Both reproduced the same result on Node.js 24.21.0, darwin arm64. This is a supported package runtime; the checkout's contributor pin is Node.js 24.18.0, which was not used for this verification. No native analysis provider is required for the focused reproduction. ## Steps to reproduce Use a repository checkout with dependencies and `dist` already built through `npm ci` and `npm run build:cached`. From its root, run the following small probe. It calls the same evaluator and checked-in example that the documented maintainer verifier uses, without running the native/CLI/MCP fixture journey. The probe changes only an in-memory clone of the input and writes nothing. ```sh node --input-type=module <<'JS' import assert from "node:assert/strict"; import { RECONSTRUCTION_READINESS_EXAMPLE as example } from "./dist/contracts/reconstructionReadinessExample.js"; import { createReconstructionReadinessReport as evaluate } from "./dist/domain/reconstructionReadiness.js"; const absentA = `ev_${"a".repeat(64)}`; const absentB = `ev_${"b".repeat(64)}`; assert.ok(!example.evidence_bundle.records.some(record => [absentA, absentB].includes(record.evidence_id))); const stage = (input, id = "static-analysis") => input.stages.find(value => value.stage_id === id); const cases = [ ["baseline", () => {}], ["stage missing one ID", input => stage(input).evidence_ids.push(absentA)], ["one check missing one ID", input => stage(input).checks[0].evidence_ids.push(absentA)], ["same ID on stage and check", input => { stage(input).evidence_ids.push(absentA); stage(input).checks[0].evidence_ids.push(absentA); }], ["two IDs on one stage", input => stage(input).evidence_ids.push(absentA, absentB)], ["same ID on two stages", input => { stage(input).evidence_ids.push(absentA); stage(input, "discover-classify").evidence_ids.push(absentA); }], ["missing ID plus failed check", input => { stage(input).evidence_ids.push(absentA); stage(input).checks[0].status = "fail"; }], ]; for (const [name, mutate] of cases) { const input = structuredClone(example); mutate(input); const report = evaluate(input); const missing = report.findings.filter(value => value.code === "referenced-evidence-missing"); console.log(JSON.stringify({ name, status: report.status, missing_rows: missing.length, distinct_missing_rows: new Set(missing.map(value => JSON.stringify(value))).size, summary: report.summary, })); if (name === "stage missing one ID") console.log(JSON.stringify(missing)); } JS ``` The same imports are present under the installed release package's `dist`; running this probe with that package directory as the working directory produced the same results. The source and release probes executed locally. The full `npm run verify:readiness` journey was not run during this verification. That verifier asserts `pass` before writing its compact summary or retained report, so this reproduction does not establish duplicated output in an actual CI artifact. ## Expected behavior The documented internal report represents failures by stage, stable reason code, Evidence IDs, and detail. A missing stage-level Evidence reference should appear once as a diagnostic for that stage and absent ID. For the one-ID case above, the report should contain one `referenced-evidence-missing` object and count it once. Separate IDs and separate stages must retain separate diagnostics, and missing Evidence must continue to withhold readiness. ## Actual behavior The baseline returns `pass` with zero findings. The one absent stage ID returns `unknown` with three missing-Evidence rows but only one distinct object. All five fields, `code`, `stage_id`, `status`, `detail`, and `evidence_ids`, are identical in those rows. The objects contain no check ID or other per-check context. The summary is `{"stages":9,"required_stages":9,"passed_required_stages":8,"findings":3,"failed_findings":0,"unknown_findings":3}`. A single absent ID on one check produces one missing-Evidence row. Putting that same ID on the stage and the check still produces three identical rows. Two distinct absent IDs on one stage produce six rows containing two distinct objects. The same absent ID on two stages produces seven rows containing two distinct objects, three for `static-analysis` and four for `discover-classify`. A failed check alongside the absent stage ID still returns `fail`, with three unknown rows and one separate failed finding. These counts are accurate counts of emitted rows; the defect is the redundant rows and the resulting inflated diagnostic counts, not arithmetic in the summary. ## Evidence - Expected source: `docs/reconstruction-readiness.md:27` establishes the internal project conformance contract; `docs/reconstruction-readiness.md:47` requires every stage and check to link to bundled Evidence and preserves non-pass outcomes; `docs/reconstruction-readiness.md:71` defines the finding's reported context. `AGENTS.md` under Architecture and Cleanup requires preserving identity across boundaries. `src/domain/reconstructionReadinessFindingHelpers.ts:12` already collapses repeated IDs within a single missing-reference check. - Failure source: The executed probe above produced three fully identical missing-reference objects and summary counts of three. `src/domain/reconstructionReadinessJourneyFindings.ts:181` iterates required checks and `src/domain/reconstructionReadinessJourneyFindings.ts:213` includes stage references in each call. `src/domain/reconstructionReadiness.ts:28` retains the emitted rows and `src/domain/reconstructionReadiness.ts:44` counts them. `scripts/verify-reconstruction-readiness.mjs:97` uses this evaluator for the documented verifier. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Independent source-build and installed-release probes supplied the runtime evidence, and source inspection established the producing call path. The exact existing source test file, `src/domain/reconstructionReadiness.test.ts`, also passed all 18 tests. Additional bounded controls accepted the valid baseline and rejected malformed Evidence IDs, duplicate stage IDs, unauthenticated Evidence record IDs, and duplicate bundle records. Missing or duplicate required checks produced `fail`. Empty check Evidence produced `unknown`. Explicit stage/check `fail`, `unknown`, `unsupported`, `truncated`, and `skipped` inputs retained their respective aggregate non-pass status. ## Restoration check Repeat the inline probe. One absent ID on one stage must yield one missing-reference diagnostic and summary counts of one. Repeating that ID on a stage and its checks must not create indistinguishable extra diagnostics. Two absent IDs in one stage must retain two distinct diagnostics; the same absent ID in two stages must retain one diagnostic for each stage. Three different absent IDs on three checks must retain three distinct diagnostics. The missing-reference cases must still return `unknown`, and a separate failed or missing required check must still return `fail`. Preserve input-schema and Evidence-bundle authentication rejection. The restoration concerns report content and counts; it does not ask for a public adapter or a changed readiness decision. ## Triage assessment - Impact level: P3 - Assessment status: supported - Impact basis: The observed effect is redundant diagnostic text and counts in a maintainer conformance report. Missing Evidence remains visible and correctly withholds readiness; no lost Evidence, changed analysis result, or failure of the readiness decision was observed. The repeats recur for the tested stage/check reference placements in both builds. - Workaround status: available - Workaround basis: The probes measured the complete emitted objects and showed that deduplicating byte-identical objects preserves the distinct stage/ID diagnostics in the tested cases. A consumer can deduplicate for display and recompute diagnostic counts while retaining the original readiness status and separate failed findings.

bsakaguchi

2026-10-15 18:27:43 +00:00