morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3357] [Bug] Reconstruction coverage reports ready with inferred Evidence backing controlled replay #1224

New issue

Closed

opened 2026-10-16 21:02:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:02:59 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3357

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

evaluate-reconstruction-coverage accepts an inferred, analyst-inference Evidence record as the sole Evidence supporting a passing controlled-replay result and reports ready. The cited record explicitly says that no replay was performed. The public evaluator detects a mismatch in declared contract authority and rejects a missing Evidence ID, but it does not enforce comparable authority or confidence on the Evidence record that the result actually cites.

The base and weak-proof requests both exited zero and returned status: "ready", zero reasons, and an empty recommended_probes array. The weak-proof result included the inferred record's Evidence ID among its returned citations. That record has authority: "analyst-inference", confidence: "inferred", and normalized_result: {"status":"unknown","note":"No replay was performed"}.

The weak record is present in both base and weak input bundles. Only verifier_results[0].evidence_ids changes between those requests; the package proof still cites the observed controlled-replay record. All Evidence IDs and bundle manifests remain authenticated, and the contract digest is valid.

Reproduction

Start from a clean checkout of morluto/rea at 8dde6370d0809ad04cda3342d4e6f290f48aa063. Install the locked dependencies and build the runtime from the repository root:

npm ci
npm run build:cached

Run the following block from the repository root. It creates every input file inline, builds a complete synthetic coverage request, then changes only the result's Evidence reference for the weak-proof case:

probe_dir="$(mktemp -d)"
export PROBE_DIR="$probe_dir"
mkdir "$PROBE_DIR/home"
node --input-type=module <<'NODE'
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { createEvidence } from './dist/domain/evidence.js';
import { createEvidenceBundle } from './dist/domain/evidenceBundle.js';
import { createReconstructionVerifierContract } from './dist/domain/reconstructionCoverage.js';

const digest = character => character.repeat(64);
const record = (operation, authority, confidence, result) => createEvidence(
  undefined, { id: 'sample', name: 'Sample', version: '1' },
  { operation, parameters: {}, result, authority, confidence }
);
const observation = record('observe', 'shipped-artifact', 'observed', { surface: 'cli.help' });
const proof = record('verify', 'controlled-replay', 'observed', { status: 'pass' });
const inference = record('infer', 'analyst-inference', 'inferred', { status: 'unknown', note: 'No replay was performed' });
const contract = createReconstructionVerifierContract({
  verifier_id: 'verify.help', claim_ids: ['claim.help'], dimensions: ['output'],
  authority: 'controlled-replay', max_age_ms: 86400000, minimum_repeats: 1,
  normalization_sha256: digest('4'), normalization_removes_dimensions: false
});
const coverage = {
  evidence_bundle: createEvidenceBundle([observation, proof, inference]),
  artifacts: [{ artifact_id: 'app', artifact_sha256: digest('1'), version: '1',
    environment_sha256: digest('2'), evidence_ids: [observation.evidence_id] }],
  surfaces: [{ surface_id: 'cli.help', family: 'cli-command', artifact_id: 'app',
    occurrence_id: null, location: 'bin/app --help', authority: 'observed',
    dependency_surface_ids: [], evidence_ids: [observation.evidence_id] }],
  owners: [{ surface_id: 'cli.help', ownership: { disposition: 'implemented',
    owner_path: 'src/help.ts', owner_export: null, owner_sha256: digest('3'),
    path_state: 'present', package_state: 'distributed', authority_route: 'none' } }],
  claims: [{ claim_id: 'claim.help', title: 'Help agrees', kind: 'behavioral',
    surface_ids: ['cli.help'], required_dimensions: ['output'],
    required_authority: 'controlled-replay' }],
  verifier_contracts: [contract],
  verifier_results: [{ verifier_id: contract.verifier_id,
    contract_sha256: contract.contract_sha256, observed_at: new Date().toISOString(),
    status: 'pass', covered_claim_ids: ['claim.help'], covered_dimensions: ['output'],
    artifact_sha256s: [digest('1')], owner_sha256s: [digest('3')],
    normalization_sha256: digest('4'), repeats: 1, evidence_ids: [proof.evidence_id] }],
  residual_unknown_ids: [], contradictions: [],
  package_proofs: [{ proof_id: 'build', kind: 'build', status: 'pass',
    artifact_sha256s: [digest('1')], evidence_ids: [proof.evidence_id] }],
  boundaries: [{ boundary_id: 'replacement', title: 'Replacement',
    required_surface_ids: ['cli.help'], required_claim_ids: ['claim.help'],
    required_package_proof_kinds: ['build'], allowed_dispositions: [], allowed_unknown_ids: [] }]
};
const save = (name, value) => writeFileSync(join(process.env.PROBE_DIR, name),
  JSON.stringify({ coverage: value, boundary_id: 'replacement' }));
save('base.json', coverage);

const weak = structuredClone(coverage);
weak.verifier_results[0].evidence_ids = [inference.evidence_id];
save('weak.json', weak);
const declared = structuredClone(coverage);
const { contract_sha256, ...contractInput } = contract;
const otherContract = createReconstructionVerifierContract({ ...contractInput, authority: 'external' });
declared.verifier_contracts = [otherContract];
declared.verifier_results[0].contract_sha256 = otherContract.contract_sha256;
save('declared.json', declared);
const missing = structuredClone(coverage);
missing.verifier_results[0].evidence_ids = ['ev_' + digest('f')];
save('missing.json', missing);
NODE
HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/base.json" --json
HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/weak.json" --json
HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/declared.json" --json
HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/missing.json" --json
rm -rf "$PROBE_DIR"

Compare the four CLI results. Both base.json and weak.json return ready; declared.json returns unknown for the authority mismatch, and missing.json fails as invalid_request.

Expected behavior

A controlled-replay claim supported only by inferred analyst Evidence must remain unknown with an actionable proof/authority limitation. A passing status in the caller's result summary cannot give the cited record stronger authority or confidence than it actually has. Qualifying observed controlled-replay proof should continue to allow closure, while absent Evidence should remain a typed input failure.

src/contracts/applicationToolContracts.ts:218 defines the public evidence-backed evaluation contract and promises an unknown result for weak proof. AGENTS.md:3, Product and Authority, and AGENTS.md:5 require observations, derivations, inferences, and unknowns to remain distinct and require Evidence provenance to survive both adapters.

Impact and scope

Impact level: P2, supported. The public closure evaluator promotes inferred Evidence to a successful replay claim and gives a false ready result for this bounded input.

Workaround status: available. Inspect the cited Evidence authority and confidence before accepting closure and supply qualifying observed replay Evidence. The executed base control, which cites such a record, remains ready without weakening the claim.

REA version

6.4.0 (main at 8dde6370d0)

Runtime and platform

Node.js v26.10.0; macOS 27.0.1 arm64. No external provider or engine was used.

Target and artifact context

A complete synthetic reconstruction-coverage request with an authenticated Evidence bundle, one artifact, one CLI command, one implemented owner, one behavioral claim requiring controlled replay, a matching digest-bound proof contract and result, a passing build package proof, and a replacement boundary. The bundle contains shipped-artifact observed Evidence, controlled-replay observed Evidence, and analyst-inference inferred Evidence whose normalized result says no replay was performed.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

base.json:     exit 0, ready,   reasons []
weak.json:     exit 0, ready,   reasons []
declared.json: exit 0, unknown, reason verifier-result-incompatible
missing.json:  exit 1, invalid_request, Coverage workspace Evidence is missing

Typed outcome or recovery expected

Run the four requests above. The weak-proof case must withhold ready and return an actionable explanation that the cited inference is insufficient for controlled replay. The base case must remain ready, the declared-authority mismatch must remain unknown, and the missing Evidence reference must remain an invalid request. Do not make the weak case pass by relabeling or replacing the caller's Evidence record.

Analysis or suspected cause

Source-derived reasoning: src/domain/reconstructionCoverage.ts:309 authenticates the bundle but retains only a set of record IDs for reference checking. src/domain/reconstructionCoverageEvaluation.ts:336 compares declared contract authority and summary fields without checking the authority/confidence of the cited Evidence records. The observed weak-proof CLI request returns ready even though its cited record disclaims a replay; the controls distinguish metadata admission from proof qualification.

Additional context

#268, completed, defines the reconstruction-coverage workflow and explicitly requires weak Evidence not to satisfy stronger authority; this report identifies a remaining failure of that contract in the public evaluator.
#3005 concerns obtaining qualifying fixture authority through MCP for obligation-ledger verification and an unknown-revision digest failure; this report concerns the coverage evaluator accepting insufficient cited Evidence.
#2622, completed, concerns replay-environment compatibility across imported multi-source observations in reconstruction verification; this report concerns cited-record authority and confidence in coverage evaluation.
#2623, completed, concerns inconsistent overall function-comparison status in reconstruction verification; this report concerns weak backing Evidence for a coverage replay claim.

Originally created by @coygeek on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3357 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `evaluate-reconstruction-coverage` accepts an inferred, analyst-inference Evidence record as the sole Evidence supporting a passing controlled-replay result and reports `ready`. The cited record explicitly says that no replay was performed. The public evaluator detects a mismatch in declared contract authority and rejects a missing Evidence ID, but it does not enforce comparable authority or confidence on the Evidence record that the result actually cites. The base and weak-proof requests both exited zero and returned `status: "ready"`, zero reasons, and an empty `recommended_probes` array. The weak-proof result included the inferred record's Evidence ID among its returned citations. That record has `authority: "analyst-inference"`, `confidence: "inferred"`, and `normalized_result: {"status":"unknown","note":"No replay was performed"}`. The weak record is present in both base and weak input bundles. Only `verifier_results[0].evidence_ids` changes between those requests; the package proof still cites the observed controlled-replay record. All Evidence IDs and bundle manifests remain authenticated, and the contract digest is valid. ### Reproduction 1. Start from a clean checkout of `morluto/rea` at `8dde6370d0809ad04cda3342d4e6f290f48aa063`. Install the locked dependencies and build the runtime from the repository root: ```sh npm ci npm run build:cached ``` 2. Run the following block from the repository root. It creates every input file inline, builds a complete synthetic coverage request, then changes only the result's Evidence reference for the weak-proof case: ```sh probe_dir="$(mktemp -d)" export PROBE_DIR="$probe_dir" mkdir "$PROBE_DIR/home" node --input-type=module <<'NODE' import { writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { createEvidence } from './dist/domain/evidence.js'; import { createEvidenceBundle } from './dist/domain/evidenceBundle.js'; import { createReconstructionVerifierContract } from './dist/domain/reconstructionCoverage.js'; const digest = character => character.repeat(64); const record = (operation, authority, confidence, result) => createEvidence( undefined, { id: 'sample', name: 'Sample', version: '1' }, { operation, parameters: {}, result, authority, confidence } ); const observation = record('observe', 'shipped-artifact', 'observed', { surface: 'cli.help' }); const proof = record('verify', 'controlled-replay', 'observed', { status: 'pass' }); const inference = record('infer', 'analyst-inference', 'inferred', { status: 'unknown', note: 'No replay was performed' }); const contract = createReconstructionVerifierContract({ verifier_id: 'verify.help', claim_ids: ['claim.help'], dimensions: ['output'], authority: 'controlled-replay', max_age_ms: 86400000, minimum_repeats: 1, normalization_sha256: digest('4'), normalization_removes_dimensions: false }); const coverage = { evidence_bundle: createEvidenceBundle([observation, proof, inference]), artifacts: [{ artifact_id: 'app', artifact_sha256: digest('1'), version: '1', environment_sha256: digest('2'), evidence_ids: [observation.evidence_id] }], surfaces: [{ surface_id: 'cli.help', family: 'cli-command', artifact_id: 'app', occurrence_id: null, location: 'bin/app --help', authority: 'observed', dependency_surface_ids: [], evidence_ids: [observation.evidence_id] }], owners: [{ surface_id: 'cli.help', ownership: { disposition: 'implemented', owner_path: 'src/help.ts', owner_export: null, owner_sha256: digest('3'), path_state: 'present', package_state: 'distributed', authority_route: 'none' } }], claims: [{ claim_id: 'claim.help', title: 'Help agrees', kind: 'behavioral', surface_ids: ['cli.help'], required_dimensions: ['output'], required_authority: 'controlled-replay' }], verifier_contracts: [contract], verifier_results: [{ verifier_id: contract.verifier_id, contract_sha256: contract.contract_sha256, observed_at: new Date().toISOString(), status: 'pass', covered_claim_ids: ['claim.help'], covered_dimensions: ['output'], artifact_sha256s: [digest('1')], owner_sha256s: [digest('3')], normalization_sha256: digest('4'), repeats: 1, evidence_ids: [proof.evidence_id] }], residual_unknown_ids: [], contradictions: [], package_proofs: [{ proof_id: 'build', kind: 'build', status: 'pass', artifact_sha256s: [digest('1')], evidence_ids: [proof.evidence_id] }], boundaries: [{ boundary_id: 'replacement', title: 'Replacement', required_surface_ids: ['cli.help'], required_claim_ids: ['claim.help'], required_package_proof_kinds: ['build'], allowed_dispositions: [], allowed_unknown_ids: [] }] }; const save = (name, value) => writeFileSync(join(process.env.PROBE_DIR, name), JSON.stringify({ coverage: value, boundary_id: 'replacement' })); save('base.json', coverage); const weak = structuredClone(coverage); weak.verifier_results[0].evidence_ids = [inference.evidence_id]; save('weak.json', weak); const declared = structuredClone(coverage); const { contract_sha256, ...contractInput } = contract; const otherContract = createReconstructionVerifierContract({ ...contractInput, authority: 'external' }); declared.verifier_contracts = [otherContract]; declared.verifier_results[0].contract_sha256 = otherContract.contract_sha256; save('declared.json', declared); const missing = structuredClone(coverage); missing.verifier_results[0].evidence_ids = ['ev_' + digest('f')]; save('missing.json', missing); NODE HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/base.json" --json HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/weak.json" --json HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/declared.json" --json HOME="$PROBE_DIR/home" node scripts/rea.mjs evaluate-reconstruction-coverage "$PROBE_DIR/missing.json" --json rm -rf "$PROBE_DIR" ``` 3. Compare the four CLI results. Both `base.json` and `weak.json` return `ready`; `declared.json` returns `unknown` for the authority mismatch, and `missing.json` fails as `invalid_request`. ### Expected behavior A controlled-replay claim supported only by inferred analyst Evidence must remain `unknown` with an actionable proof/authority limitation. A passing status in the caller's result summary cannot give the cited record stronger authority or confidence than it actually has. Qualifying observed controlled-replay proof should continue to allow closure, while absent Evidence should remain a typed input failure. `src/contracts/applicationToolContracts.ts:218` defines the public evidence-backed evaluation contract and promises an unknown result for weak proof. `AGENTS.md:3`, Product and Authority, and `AGENTS.md:5` require observations, derivations, inferences, and unknowns to remain distinct and require Evidence provenance to survive both adapters. ### Impact and scope Impact level: P2, supported. The public closure evaluator promotes inferred Evidence to a successful replay claim and gives a false ready result for this bounded input. Workaround status: available. Inspect the cited Evidence authority and confidence before accepting closure and supply qualifying observed replay Evidence. The executed base control, which cites such a record, remains ready without weakening the claim. ### REA version 6.4.0 (main at 8dde6370d0809ad04cda3342d4e6f290f48aa063) ### Runtime and platform Node.js v26.10.0; macOS 27.0.1 arm64. No external provider or engine was used. ### Target and artifact context A complete synthetic reconstruction-coverage request with an authenticated Evidence bundle, one artifact, one CLI command, one implemented owner, one behavioral claim requiring controlled replay, a matching digest-bound proof contract and result, a passing build package proof, and a replacement boundary. The bundle contains shipped-artifact observed Evidence, controlled-replay observed Evidence, and analyst-inference inferred Evidence whose normalized result says no replay was performed. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence ```text base.json: exit 0, ready, reasons [] weak.json: exit 0, ready, reasons [] declared.json: exit 0, unknown, reason verifier-result-incompatible missing.json: exit 1, invalid_request, Coverage workspace Evidence is missing ``` ### Typed outcome or recovery expected Run the four requests above. The weak-proof case must withhold ready and return an actionable explanation that the cited inference is insufficient for controlled replay. The base case must remain ready, the declared-authority mismatch must remain unknown, and the missing Evidence reference must remain an invalid request. Do not make the weak case pass by relabeling or replacing the caller's Evidence record. ### Analysis or suspected cause Source-derived reasoning: `src/domain/reconstructionCoverage.ts:309` authenticates the bundle but retains only a set of record IDs for reference checking. `src/domain/reconstructionCoverageEvaluation.ts:336` compares declared contract authority and summary fields without checking the authority/confidence of the cited Evidence records. The observed weak-proof CLI request returns ready even though its cited record disclaims a replay; the controls distinguish metadata admission from proof qualification. ### Additional context - [#268](https://github.com/morluto/rea/issues/268), completed, defines the reconstruction-coverage workflow and explicitly requires weak Evidence not to satisfy stronger authority; this report identifies a remaining failure of that contract in the public evaluator. - [#3005](https://github.com/morluto/rea/issues/3005) concerns obtaining qualifying fixture authority through MCP for obligation-ledger verification and an unknown-revision digest failure; this report concerns the coverage evaluator accepting insufficient cited Evidence. - [#2622](https://github.com/morluto/rea/issues/2622), completed, concerns replay-environment compatibility across imported multi-source observations in reconstruction verification; this report concerns cited-record authority and confidence in coverage evaluation. - [#2623](https://github.com/morluto/rea/issues/2623), completed, concerns inconsistent overall function-comparison status in reconstruction verification; this report concerns weak backing Evidence for a coverage replay claim.

bsakaguchi closed this issue

2026-10-16 21:03:00 +00:00

bsakaguchi referenced this issue

2026-10-17 21:09:40 +00:00

[PR #1224] [MERGED] fix: name host and tool requirements in optional adapter refusals #2737

bsakaguchi referenced this issue

2026-10-17 21:09:46 +00:00

[PR #1258] [MERGED] fix(native): name the plist selection that works when Info.plist is missing #2745

bsakaguchi referenced this issue

2026-10-31 00:56:58 +00:00

[PR #1946] [MERGED] fix(apktool): name host requirements in configuration refusals #4926

bsakaguchi referenced this issue