morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3889] [Bug] Reconstruction coverage reports ready when a passing proof conflicts with a failed one #1509

New issue

Closed

opened 2026-10-16 21:06:20 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-16 21:06:20 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 15, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3889

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

rea evaluate-reconstruction-coverage reports a boundary ready when a required package-proof kind has both a passing and a failing proof for the same artifact. The failing proof alone yields failed with package-proof-failed, but adding a passing proof for the identical artifact digest hides the failure: no reason, no probe, and the failure's Evidence ID is absent from the result. The tool contract says failed proof fails closed.

The failed proof alone returns failed with package-proof-failed. Adding a passing proof for the identical artifact digest changes the result to ready with empty reasons and recommended_probes, in either array order. The failing proof's Evidence ID does not appear in the result's evidence_ids. All commands exit 0.

Reproduction

From the root of a built source checkout (the generator imports dist/ modules), write three request files into a scratch directory. The verifier observation time is the current instant so the freshness check does not interfere.

OUT=$(mktemp -d)
node --input-type=module - "$PWD" "$OUT" <<'JS'
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
const [repo, out] = process.argv.slice(2);
const load = p => import(pathToFileURL(join(repo, 'dist', p)));
const {createEvidence} = await load('domain/evidence.js');
const {createEvidenceBundle} = await load('domain/evidenceBundle.js');
const {createReconstructionVerifierContract} = await load('domain/reconstructionCoverage.js');
const d = n => n.repeat(64);
const evidence = (operation, authority, result) => createEvidence(undefined,{id:'fixture',name:'Fixture',version:'1'},{operation,parameters:{},result,authority,confidence:'observed'});
const observation=evidence('inspect_fixture','shipped-artifact',{found:true});
const proof=evidence('verify_fixture','controlled-replay',{passed:true});
const failure=evidence('verify_failed_build','controlled-replay',{passed:false});
const contract=createReconstructionVerifierContract({verifier_id:'verify.help',claim_ids:['claim.help'],dimensions:['output','exit'],authority:'controlled-replay',max_age_ms:86400000,minimum_repeats:1,normalization_sha256:d('4'),normalization_removes_dimensions:false});
const owner={disposition:'implemented',owner_path:'src/help.js',owner_export:'help',owner_sha256:d('3'),path_state:'present',package_state:'distributed',authority_route:'none'};
const surface={surface_id:'cli.help',family:'cli-command',artifact_id:'app',occurrence_id:null,location:'app --help',authority:'observed',dependency_surface_ids:[],evidence_ids:[observation.evidence_id]};
const base={evidence_bundle:createEvidenceBundle([observation,proof,failure]),artifacts:[{artifact_id:'app',artifact_sha256:d('1'),version:'1',environment_sha256:d('2'),evidence_ids:[observation.evidence_id]}],surfaces:[surface],owners:[{surface_id:'cli.help',ownership:owner}],claims:[{claim_id:'claim.help',title:'Help matches',kind:'behavioral',surface_ids:['cli.help'],required_dimensions:['output','exit'],required_authority:'controlled-replay'}],verifier_contracts:[contract],verifier_results:[{verifier_id:contract.verifier_id,contract_sha256:contract.contract_sha256,observed_at:new Date().toISOString(),status:'pass',covered_claim_ids:['claim.help'],covered_dimensions:['output','exit'],artifact_sha256s:[d('1')],owner_sha256s:[d('3')],normalization_sha256:d('4'),repeats:1,evidence_ids:[proof.evidence_id]}],residual_unknown_ids:[],contradictions:[],package_proofs:[{proof_id:'build.pass',kind:'build',status:'pass',artifact_sha256s:[d('1')],evidence_ids:[proof.evidence_id]}],boundaries:[{boundary_id:'replacement',title:'Replacement',required_surface_ids:['cli.help'],required_claim_ids:['claim.help'],required_package_proof_kinds:['build'],allowed_dispositions:[],allowed_unknown_ids:[]}]};
const save=(name,c)=>writeFileSync(join(out,name+'.json'),JSON.stringify({coverage:c,boundary_id:'replacement'}));
const failProof={proof_id:'build.fail',kind:'build',status:'fail',artifact_sha256s:[d('1')],evidence_ids:[failure.evidence_id]};
save('failed-only',{...base,package_proofs:[failProof]});
save('conflicting-proofs',{...base,package_proofs:[...base.package_proofs,failProof]});
save('conflicting-proofs-reversed',{...base,package_proofs:[failProof,...base.package_proofs]});
console.log('failure_evidence_id', failure.evidence_id);
JS

Evaluate each request through the CLI (node scripts/rea.mjs is the rea bin entry in the checkout).

for f in failed-only conflicting-proofs conflicting-proofs-reversed; do node scripts/rea.mjs evaluate-reconstruction-coverage "$OUT/$f.json" --json; done

Compare status, reasons, and evidence_ids across the three results.

Expected behavior

Two retained build proofs for the same required artifact, one passing and one failing with observed failure Evidence, must not silently establish readiness. The result should preserve the failure (failed with package-proof-failed) or explicitly represent the unresolved conflicting proof, and it should cite the failing proof's Evidence. The input schema has no supersession or ordering field that could declare the failing proof obsolete.

Impact and scope

A caller retaining conflicting build proof for the same artifact receives a false ready closure verdict with the failing Evidence dropped, contradicting the fail-closed contract; the effect is limited to callers who supply conflicting proofs.

Workaround: available. Callers can inspect every required-kind proof for the same artifact digests before accepting ready, or retain only the failing proof, which the failed-only control shows is reported as failed.

Reporter's impact estimate: P2 (supported).

REA version

6.5.0, built from source at main commit 034342ba28 (latest npm release is 6.5.0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64; No analysis provider or target execution is involved; the input is authenticated synthetic Evidence.

Target and artifact context

Synthetic reconstruction coverage request with authenticated Evidence bundle

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: src/contracts/applicationToolContracts.ts:221 (evaluate_reconstruction_coverage description: "contradictions, failed proof, missing owners, and authority routing fail closed"); src/domain/reconstructionCoverage.ts:393 classifies package-proof-failed as failed closure
Failure source: public CLI results for the three requests above; static cause at src/domain/reconstructionCoverageEvaluation.ts:439-448
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

Relevant result fields, in command order:

failed-only: status="failed", reasons=[{"code":"package-proof-failed","subject_id":"build","detail":"Required package or integration proof failed."}]
conflicting-proofs: status="ready", reasons=[], recommended_probes=[]
conflicting-proofs-reversed: status="ready", reasons=[], recommended_probes=[]

The generator prints the failing proof's Evidence ID, ev_431d406806a6e1977f6b3326b4d59898483f40e587d7f051a09d6a219874d949, which is absent from the evidence_ids of both conflicting results. Both proofs bind the same artifact_sha256s (64 1 characters).

Source-derived cause: src/domain/reconstructionCoverageEvaluation.ts:439 finds any passing proof covering the required artifacts; lines 444-446 add only that proof's Evidence and continue before the failure check at line 448 runs. The CLI reaches this evaluator through src/cli/applicationCommands.ts:168 and src/application/ReconstructionCoverageService.ts:14. Only the build kind was executed; the other required proof kinds (install, package, routing, independence) take the same branch but were not run individually.

Typed outcome or recovery expected

A non-ready coverage result (failed with package-proof-failed, or an explicit conflicting-proof reason) that cites the failing proof's Evidence ID.

Restoration check:

On the conflicting-proofs and conflicting-proofs-reversed requests, the tested build returns status: "ready" with no reasons; a fixed build should return a non-ready status that names the failing build proof (for example package-proof-failed or an explicit conflicting-proof reason) and includes ev_431d…d949 in evidence_ids. The failed-only control must keep returning failed with package-proof-failed, and a request with only the passing proof must keep returning ready.

Analysis or suspected cause

Source-derived: src/domain/reconstructionCoverageEvaluation.ts:439-446 accepts the first passing proof covering the required artifacts and continues before the failure check at line 448, so a same-artifact failed proof is never examined.

Additional context

Related but distinct open reports on the same evaluator: #3357 (inferred Evidence satisfying controlled replay) and #3354 (missing transitive dependency). Closed #268 introduced the workflow. No existing report covers contradictory package-proof statuses.

This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

Originally created by @coygeek on GitHub (Oct 15, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3889 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `rea evaluate-reconstruction-coverage` reports a boundary `ready` when a required package-proof kind has both a passing and a failing proof for the same artifact. The failing proof alone yields `failed` with `package-proof-failed`, but adding a passing proof for the identical artifact digest hides the failure: no reason, no probe, and the failure's Evidence ID is absent from the result. The tool contract says failed proof fails closed. The failed proof alone returns `failed` with `package-proof-failed`. Adding a passing proof for the identical artifact digest changes the result to `ready` with empty `reasons` and `recommended_probes`, in either array order. The failing proof's Evidence ID does not appear in the result's `evidence_ids`. All commands exit 0. ### Reproduction 1. From the root of a built source checkout (the generator imports `dist/` modules), write three request files into a scratch directory. The verifier observation time is the current instant so the freshness check does not interfere. ```sh OUT=$(mktemp -d) node --input-type=module - "$PWD" "$OUT" <<'JS' import { writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { pathToFileURL } from 'node:url'; const [repo, out] = process.argv.slice(2); const load = p => import(pathToFileURL(join(repo, 'dist', p))); const {createEvidence} = await load('domain/evidence.js'); const {createEvidenceBundle} = await load('domain/evidenceBundle.js'); const {createReconstructionVerifierContract} = await load('domain/reconstructionCoverage.js'); const d = n => n.repeat(64); const evidence = (operation, authority, result) => createEvidence(undefined,{id:'fixture',name:'Fixture',version:'1'},{operation,parameters:{},result,authority,confidence:'observed'}); const observation=evidence('inspect_fixture','shipped-artifact',{found:true}); const proof=evidence('verify_fixture','controlled-replay',{passed:true}); const failure=evidence('verify_failed_build','controlled-replay',{passed:false}); const contract=createReconstructionVerifierContract({verifier_id:'verify.help',claim_ids:['claim.help'],dimensions:['output','exit'],authority:'controlled-replay',max_age_ms:86400000,minimum_repeats:1,normalization_sha256:d('4'),normalization_removes_dimensions:false}); const owner={disposition:'implemented',owner_path:'src/help.js',owner_export:'help',owner_sha256:d('3'),path_state:'present',package_state:'distributed',authority_route:'none'}; const surface={surface_id:'cli.help',family:'cli-command',artifact_id:'app',occurrence_id:null,location:'app --help',authority:'observed',dependency_surface_ids:[],evidence_ids:[observation.evidence_id]}; const base={evidence_bundle:createEvidenceBundle([observation,proof,failure]),artifacts:[{artifact_id:'app',artifact_sha256:d('1'),version:'1',environment_sha256:d('2'),evidence_ids:[observation.evidence_id]}],surfaces:[surface],owners:[{surface_id:'cli.help',ownership:owner}],claims:[{claim_id:'claim.help',title:'Help matches',kind:'behavioral',surface_ids:['cli.help'],required_dimensions:['output','exit'],required_authority:'controlled-replay'}],verifier_contracts:[contract],verifier_results:[{verifier_id:contract.verifier_id,contract_sha256:contract.contract_sha256,observed_at:new Date().toISOString(),status:'pass',covered_claim_ids:['claim.help'],covered_dimensions:['output','exit'],artifact_sha256s:[d('1')],owner_sha256s:[d('3')],normalization_sha256:d('4'),repeats:1,evidence_ids:[proof.evidence_id]}],residual_unknown_ids:[],contradictions:[],package_proofs:[{proof_id:'build.pass',kind:'build',status:'pass',artifact_sha256s:[d('1')],evidence_ids:[proof.evidence_id]}],boundaries:[{boundary_id:'replacement',title:'Replacement',required_surface_ids:['cli.help'],required_claim_ids:['claim.help'],required_package_proof_kinds:['build'],allowed_dispositions:[],allowed_unknown_ids:[]}]}; const save=(name,c)=>writeFileSync(join(out,name+'.json'),JSON.stringify({coverage:c,boundary_id:'replacement'})); const failProof={proof_id:'build.fail',kind:'build',status:'fail',artifact_sha256s:[d('1')],evidence_ids:[failure.evidence_id]}; save('failed-only',{...base,package_proofs:[failProof]}); save('conflicting-proofs',{...base,package_proofs:[...base.package_proofs,failProof]}); save('conflicting-proofs-reversed',{...base,package_proofs:[failProof,...base.package_proofs]}); console.log('failure_evidence_id', failure.evidence_id); JS ``` 2. Evaluate each request through the CLI (`node scripts/rea.mjs` is the `rea` bin entry in the checkout). ```sh for f in failed-only conflicting-proofs conflicting-proofs-reversed; do node scripts/rea.mjs evaluate-reconstruction-coverage "$OUT/$f.json" --json; done ``` 3. Compare `status`, `reasons`, and `evidence_ids` across the three results. ### Expected behavior Two retained build proofs for the same required artifact, one passing and one failing with observed failure Evidence, must not silently establish readiness. The result should preserve the failure (`failed` with `package-proof-failed`) or explicitly represent the unresolved conflicting proof, and it should cite the failing proof's Evidence. The input schema has no supersession or ordering field that could declare the failing proof obsolete. ### Impact and scope A caller retaining conflicting build proof for the same artifact receives a false `ready` closure verdict with the failing Evidence dropped, contradicting the fail-closed contract; the effect is limited to callers who supply conflicting proofs. Workaround: available. Callers can inspect every required-kind proof for the same artifact digests before accepting `ready`, or retain only the failing proof, which the failed-only control shows is reported as `failed`. Reporter's impact estimate: P2 (supported). ### REA version 6.5.0, built from source at main commit 034342ba28c8e56c3746be92b78aa7299afa02bd (latest npm release is 6.5.0) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64; No analysis provider or target execution is involved; the input is authenticated synthetic Evidence. ### Target and artifact context Synthetic reconstruction coverage request with authenticated Evidence bundle ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence - Expected source: src/contracts/applicationToolContracts.ts:221 (`evaluate_reconstruction_coverage` description: "contradictions, failed proof, missing owners, and authority routing fail closed"); src/domain/reconstructionCoverage.ts:393 classifies `package-proof-failed` as failed closure - Failure source: public CLI results for the three requests above; static cause at src/domain/reconstructionCoverageEvaluation.ts:439-448 - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete Relevant result fields, in command order: ```text failed-only: status="failed", reasons=[{"code":"package-proof-failed","subject_id":"build","detail":"Required package or integration proof failed."}] conflicting-proofs: status="ready", reasons=[], recommended_probes=[] conflicting-proofs-reversed: status="ready", reasons=[], recommended_probes=[] ``` The generator prints the failing proof's Evidence ID, `ev_431d406806a6e1977f6b3326b4d59898483f40e587d7f051a09d6a219874d949`, which is absent from the `evidence_ids` of both conflicting results. Both proofs bind the same `artifact_sha256s` (64 `1` characters). Source-derived cause: src/domain/reconstructionCoverageEvaluation.ts:439 finds any passing proof covering the required artifacts; lines 444-446 add only that proof's Evidence and `continue` before the failure check at line 448 runs. The CLI reaches this evaluator through src/cli/applicationCommands.ts:168 and src/application/ReconstructionCoverageService.ts:14. Only the `build` kind was executed; the other required proof kinds (install, package, routing, independence) take the same branch but were not run individually. ### Typed outcome or recovery expected A non-ready coverage result (failed with package-proof-failed, or an explicit conflicting-proof reason) that cites the failing proof's Evidence ID. Restoration check: On the `conflicting-proofs` and `conflicting-proofs-reversed` requests, the tested build returns `status: "ready"` with no reasons; a fixed build should return a non-ready status that names the failing build proof (for example `package-proof-failed` or an explicit conflicting-proof reason) and includes `ev_431d…d949` in `evidence_ids`. The `failed-only` control must keep returning `failed` with `package-proof-failed`, and a request with only the passing proof must keep returning `ready`. ### Analysis or suspected cause Source-derived: src/domain/reconstructionCoverageEvaluation.ts:439-446 accepts the first passing proof covering the required artifacts and continues before the failure check at line 448, so a same-artifact failed proof is never examined. ### Additional context Related but distinct open reports on the same evaluator: #3357 (inferred Evidence satisfying controlled replay) and #3354 (missing transitive dependency). Closed #268 introduced the workflow. No existing report covers contradictory package-proof statuses. This report came from AI-assisted source review with local reproduction where stated above; the evidence labels distinguish observed runs from source-only reasoning.

bsakaguchi commented

2026-10-16 21:06:21 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 15, 2026):

This one violates the tool's own fail-closed contract in a dangerous way: a failing proof for an artifact silently stops counting once any passing proof for the same digest is present, and the failure's Evidence ID even vanishes from evidence_ids. For a verification-coverage tool, conflicting outcomes must fail closed — the whole point is to surface the conflict so a human can investigate, not to let a passing result hide a known failure. The aggregation needs to treat the failing proof as dominant regardless of array order (or explicitly report a conflicting-proof status with both IDs), and the regression fixture should cover both orderings plus the evidence_ids assertion.

@XinwangTao commented on GitHub (Oct 15, 2026): This one violates the tool's own fail-closed contract in a dangerous way: a failing proof for an artifact silently stops counting once any passing proof for the same digest is present, and the failure's Evidence ID even vanishes from evidence_ids. For a verification-coverage tool, conflicting outcomes must fail closed — the whole point is to surface the conflict so a human can investigate, not to let a passing result hide a known failure. The aggregation needs to treat the failing proof as dominant regardless of array order (or explicitly report a conflicting-proof status with both IDs), and the regression fixture should cover both orderings plus the evidence_ids assertion.

bsakaguchi referenced this issue

2026-10-17 21:11:55 +00:00

[PR #1509] [MERGED] fix(browser): accept HAR files that begin with a byte-order mark #2928

bsakaguchi closed this issue

2026-10-31 00:31:09 +00:00

bsakaguchi commented

2026-10-31 00:31:10 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 20, 2026):

Cross-referencing with canonical issue #3354 ([Bug] Reconstruction coverage reports ready with a missing transitive dependency) and siblings #3890, #3937. All relate to reconstruction coverage false-ready reports when proofs conflict or required claims encounter contradictions. Cataloged under Cluster BR in consolidation tracker #4330.

@matbrgz commented on GitHub (Oct 20, 2026): Cross-referencing with canonical issue #3354 (`[Bug] Reconstruction coverage reports ready with a missing transitive dependency`) and siblings #3890, #3937. All relate to reconstruction coverage false-ready reports when proofs conflict or required claims encounter contradictions. Cataloged under Cluster BR in consolidation tracker #4330.

bsakaguchi commented

2026-10-31 00:31:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

The fix direction in 1aifanatic/rea#3 (fail closed whenever any failing proof exists for the artifact) matches the contract. One cross-check before merge: #4337 changes how mixed reference targets are preserved, and this bug is also about conflicting outcomes on the same artifact — if both land, the combined behavior should be pinned by a test where one package-proof kind has a failing proof while a different kind on the same artifact passes, asserting the failure still dominates and both evidence IDs stay in evidence_ids.

@XinwangTao commented on GitHub (Oct 21, 2026): The fix direction in 1aifanatic/rea#3 (fail closed whenever any failing proof exists for the artifact) matches the contract. One cross-check before merge: #4337 changes how mixed reference targets are preserved, and this bug is also about conflicting outcomes on the same artifact — if both land, the combined behavior should be pinned by a test where one package-proof kind has a failing proof while a different kind on the same artifact passes, asserting the failure still dominates and both evidence IDs stay in evidence_ids.

bsakaguchi commented

2026-10-31 00:31:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

@XinwangTao commented on GitHub (Oct 21, 2026): The fix direction in 1aifanatic/rea#3 (fail closed whenever any failing proof exists for the artifact) matches the contract. One cross-check before merge: #4337 changes how mixed reference targets are preserved, and this bug is also about conflicting outcomes on the same artifact — if both land, the combined behavior should be pinned by a test where one package-proof kind has a failing proof while a different kind on the same artifact passes, asserting the failure still dominates and both evidence IDs stay in evidence_ids.

bsakaguchi commented

2026-10-31 00:31:11 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

@XinwangTao commented on GitHub (Oct 21, 2026): The fix direction in 1aifanatic/rea#3 (fail closed whenever any failing proof exists for the artifact) matches the contract. One cross-check before merge: #4337 changes how mixed reference targets are preserved, and this bug is also about conflicting outcomes on the same artifact — if both land, the combined behavior should be pinned by a test where one package-proof kind has a failing proof while a different kind on the same artifact passes, asserting the failure still dominates and both evidence IDs stay in evidence_ids.

bsakaguchi commented

2026-10-31 00:31:11 +00:00

Author

Owner

@morluto commented on GitHub (Oct 23, 2026):

Resolved on main in a35b9335a. Package evaluation retains all applicable outcomes and their Evidence; any applicable failure blocks readiness even when a passing proof exists. Both array orders and unknown/complete controls execute. Validation: src/domain/reconstructionCoverage.test.ts — conflicting package proofs forward / conflicting package proofs reversed; recorded execution at b03a15e6d. Thirty shared evaluator tests passed with synthetic authenticated proof records. No public CLI/MCP rerun or actual build verification is claimed.

@morluto commented on GitHub (Oct 23, 2026): Resolved on main in [a35b9335a](https://github.com/morluto/rea/commit/a35b9335a0124674ee5e3abe8c075f1fff6b4fa6). Package evaluation retains all applicable outcomes and their Evidence; any applicable failure blocks readiness even when a passing proof exists. Both array orders and unknown/complete controls execute. Validation: `src/domain/reconstructionCoverage.test.ts` — conflicting package proofs forward / conflicting package proofs reversed; recorded execution at `b03a15e6d`. Thirty shared evaluator tests passed with synthetic authenticated proof records. No public CLI/MCP rerun or actual build verification is claimed.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1509

No description provided.

Rows
Columns