[GH-ISSUE #4396] [Feature] Require a failing negative control before a reconstruction verifier closes an obligation #1748

Open
opened 2026-10-16 21:09:48 +00:00 by bsakaguchi · 2 comments
Owner

Originally created by @coygeek on GitHub (Oct 16, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4396

Before submitting

  • I searched the open and closed issues and pull requests for a related proposal.
  • I understand the contribution scope must be reviewed before integration; open questions are welcome here.
  • This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

build_reconstruction_obligation_ledger closes an obligation when a manifest binding supplies a unique owner, required fixtures, and "a passing verifier whose authority is comparable to the original observation. The verifier must enumerate the obligation ID and its result must be present in the input Evidence bundle." (docs/reconstruction-obligation-ledgers.md). The verifier binding carries command, authority, status: "pass", result_evidence_id, and enumerated_obligation_ids.

Nothing in that binding shows the verifier can distinguish a correct reconstruction from a wrong one. A verifier that always passes, for example one that enumerates the obligation ID but asserts nothing, or one whose comparison silently skips the relevant dimension, meets the stated closure rule. The negative value of obligationCaseKindSchema (src/domain/reconstructionObligationLedgerSchemas.ts) describes negative inputs to the target's behavior ("reject an invalid configuration option"), not a negative control on the verifier itself. verify_reconstruction takes only a specification and checks claim authority; evaluate_reconstruction_coverage checks evidence provenance.

REA already applies this discipline internally: npm run verify:readiness fails closed if "a deliberate primary divergence is not localized" and keeps "every deliberate fault" represented (docs/reconstruction-readiness.md). That protects REA's own comparators, not the verifiers a user binds into a ledger.

Use cases

  • An agent reconstructs a protocol handler and writes a conformance test as the verifier. Before the ledger closes the obligation, it should show the same test failing against a build with one deliberate defect (wrong field order, dropped error path).
  • Reviewing someone else's reconstruction claim: the ledger shows which closures rest on verifiers that were demonstrated to fail on a known-wrong input and which were never challenged.

Open false-green and false-red reports in the reconstruction area (#3937, #3984; closed #3980) show that closure logic is already a source of incorrect results; this request addresses the verifier side, which those issues do not cover.

Desired outcome and success criteria

Let a reconstruction obligation ledger record a negative control for each verifier binding (a deliberately wrong candidate and the verifier's observed failing result), and keep closure open or unknown when no control shows that the verifier can fail.

  • A verifier binding can include one or more controls, each naming the deliberately wrong candidate (by artifact identity or fixture), the defect it introduces, and the Evidence ID of the verifier's observed failing result.
  • The ledger reports, per obligation, whether its verifier has a passing result on the candidate and a failing result on at least one control; a control that also passes is reported as a vacuous verifier, not ignored.
  • Closure policy is explicit: an obligation without a demonstrated failing control does not report the same closed status as one with it (either it stays open or it closes with a named limitation; maintainers choose which).
  • Existing ledgers without controls still parse; their closures are labeled as unchallenged rather than silently upgraded or rejected.
  • Documentation explains how to construct a one-defect control and why a control that passes is evidence against the verifier.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

The ledger reads Evidence and manifests only; running the verifier and the control stays with the caller, as it does today for the passing result. Non-goals: REA generating mutants automatically, or running caller verifier commands.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Possible fixture: a ledger with two obligations bound to the same trivially passing verifier, one with a control whose result also passes and one with a control that fails. Expected: the first is reported as a vacuous verifier and not closed; the second closes.

Alternatives considered

  • Relying on reviewers to notice weak verifiers: does not scale and is not recorded in Evidence.
  • Expressing controls as additional negative cases: conflates target behavior with verifier sensitivity and would let a verifier that ignores its input satisfy both.

Possible direction

Model a control as an optional array on the verifier binding with its own result_evidence_id and expected status: "fail", and surface the per-obligation result in the ledger output next to existing closure reasons.

References

  • Checked against main at commit 8ab5024 (rea-agents 6.5.0).
Originally created by @coygeek on GitHub (Oct 16, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4396 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation `build_reconstruction_obligation_ledger` closes an obligation when a manifest binding supplies a unique owner, required fixtures, and "a passing verifier whose authority is comparable to the original observation. The verifier must enumerate the obligation ID and its result must be present in the input Evidence bundle." (`docs/reconstruction-obligation-ledgers.md`). The verifier binding carries `command`, `authority`, `status: "pass"`, `result_evidence_id`, and `enumerated_obligation_ids`. Nothing in that binding shows the verifier can distinguish a correct reconstruction from a wrong one. A verifier that always passes, for example one that enumerates the obligation ID but asserts nothing, or one whose comparison silently skips the relevant dimension, meets the stated closure rule. The `negative` value of `obligationCaseKindSchema` (`src/domain/reconstructionObligationLedgerSchemas.ts`) describes negative inputs to the target's behavior ("reject an invalid configuration option"), not a negative control on the verifier itself. `verify_reconstruction` takes only a specification and checks claim authority; `evaluate_reconstruction_coverage` checks evidence provenance. REA already applies this discipline internally: `npm run verify:readiness` fails closed if "a deliberate primary divergence is not localized" and keeps "every deliberate fault" represented (`docs/reconstruction-readiness.md`). That protects REA's own comparators, not the verifiers a user binds into a ledger. ### Use cases - An agent reconstructs a protocol handler and writes a conformance test as the verifier. Before the ledger closes the obligation, it should show the same test failing against a build with one deliberate defect (wrong field order, dropped error path). - Reviewing someone else's reconstruction claim: the ledger shows which closures rest on verifiers that were demonstrated to fail on a known-wrong input and which were never challenged. Open false-green and false-red reports in the reconstruction area (#3937, #3984; closed #3980) show that closure logic is already a source of incorrect results; this request addresses the verifier side, which those issues do not cover. ### Desired outcome and success criteria Let a reconstruction obligation ledger record a negative control for each verifier binding (a deliberately wrong candidate and the verifier's observed failing result), and keep closure open or unknown when no control shows that the verifier can fail. - A verifier binding can include one or more controls, each naming the deliberately wrong candidate (by artifact identity or fixture), the defect it introduces, and the Evidence ID of the verifier's observed failing result. - The ledger reports, per obligation, whether its verifier has a passing result on the candidate and a failing result on at least one control; a control that also passes is reported as a vacuous verifier, not ignored. - Closure policy is explicit: an obligation without a demonstrated failing control does not report the same closed status as one with it (either it stays open or it closes with a named limitation; maintainers choose which). - Existing ledgers without controls still parse; their closures are labeled as unchallenged rather than silently upgraded or rejected. - Documentation explains how to construct a one-defect control and why a control that passes is evidence against the verifier. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) The ledger reads Evidence and manifests only; running the verifier and the control stays with the caller, as it does today for the passing result. Non-goals: REA generating mutants automatically, or running caller verifier commands. ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) Possible fixture: a ledger with two obligations bound to the same trivially passing verifier, one with a control whose result also passes and one with a control that fails. Expected: the first is reported as a vacuous verifier and not closed; the second closes. ### Alternatives considered - Relying on reviewers to notice weak verifiers: does not scale and is not recorded in Evidence. - Expressing controls as additional `negative` cases: conflates target behavior with verifier sensitivity and would let a verifier that ignores its input satisfy both. ### Possible direction Model a control as an optional array on the verifier binding with its own `result_evidence_id` and expected `status: "fail"`, and surface the per-obligation result in the ledger output next to existing closure reasons. ### References - Checked against `main` at commit 8ab5024 (`rea-agents` 6.5.0).
Author
Owner

@mhmdmhmdtahapnahy-star commented on GitHub (Oct 25, 2026):

Before submitting

  • I searched the open and closed issues and pull requests for a related proposal.
  • I understand the contribution scope must be reviewed before integration; open questions are welcome here.
  • This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

build_reconstruction_obligation_ledger closes an obligation when a manifest binding supplies a unique owner, required fixtures, and "a passing verifier whose authority is comparable to the original observation. The verifier must enumerate the obligation ID and its result must be present in the input Evidence bundle." (docs/reconstruction-obligation-ledgers.md). The verifier binding carries command, authority, status: "pass", result_evidence_id, and enumerated_obligation_ids.

Nothing in that binding shows the verifier can distinguish a correct reconstruction from a wrong one. A verifier that always passes, for example one that enumerates the obligation ID but asserts nothing, or one whose comparison silently skips the relevant dimension, meets the stated closure rule. The negative value of obligationCaseKindSchema (src/domain/reconstructionObligationLedgerSchemas.ts) describes negative inputs to the target's behavior ("reject an invalid configuration option"), not a negative control on the verifier itself. verify_reconstruction takes only a specification and checks claim authority; evaluate_reconstruction_coverage checks evidence provenance.

REA already applies this discipline internally: npm run verify:readiness fails closed if "a deliberate primary divergence is not localized" and keeps "every deliberate fault" represented (docs/reconstruction-readiness.md). That protects REA's own comparators, not the verifiers a user binds into a ledger.

Use cases

  • An agent reconstructs a protocol handler and writes a conformance test as the verifier. Before the ledger closes the obligation, it should show the same test failing against a build with one deliberate defect (wrong field order, dropped error path).
  • Reviewing someone else's reconstruction claim: the ledger shows which closures rest on verifiers that were demonstrated to fail on a known-wrong input and which were never challenged.

Open false-green and false-red reports in the reconstruction area (#3937, #3984; closed #3980) show that closure logic is already a source of incorrect results; this request addresses the verifier side, which those issues do not cover.

Desired outcome and success criteria

Let a reconstruction obligation ledger record a negative control for each verifier binding (a deliberately wrong candidate and the verifier's observed failing result), and keep closure open or unknown when no control shows that the verifier can fail.

  • A verifier binding can include one or more controls, each naming the deliberately wrong candidate (by artifact identity or fixture), the defect it introduces, and the Evidence ID of the verifier's observed failing result.
  • The ledger reports, per obligation, whether its verifier has a passing result on the candidate and a failing result on at least one control; a control that also passes is reported as a vacuous verifier, not ignored.
  • Closure policy is explicit: an obligation without a demonstrated failing control does not report the same closed status as one with it (either it stays open or it closes with a named limitation; maintainers choose which).
  • Existing ledgers without controls still parse; their closures are labeled as unchallenged rather than silently upgraded or rejected.
  • Documentation explains how to construct a one-defect control and why a control that passes is evidence against the verifier.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

The ledger reads Evidence and manifests only; running the verifier and the control stays with the caller, as it does today for the passing result. Non-goals: REA generating mutants automatically, or running caller verifier commands.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Possible fixture: a ledger with two obligations bound to the same trivially passing verifier, one with a control whose result also passes and one with a control that fails. Expected: the first is reported as a vacuous verifier and not closed; the second closes.

Alternatives considered

  • Relying on reviewers to notice weak verifiers: does not scale and is not recorded in Evidence.
  • Expressing controls as additional negative cases: conflates target behavior with verifier sensitivity and would let a verifier that ignores its input satisfy both.

Possible direction

Model a control as an optional array on the verifier binding with its own result_evidence_id and expected status: "fail", and surface the per-obligation result in the ledger output next to existing closure reasons.

References

  • Checked against main at commit 8ab5024 (rea-agents 6.5.0).
<!-- gh-comment-id:6404783433 --> @mhmdmhmdtahapnahy-star commented on GitHub (Oct 25, 2026): > ### Before submitting > > - [x] I searched the open and closed issues and pull requests for a related proposal. > - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. > - [x] This request does not include credentials, private samples, or sensitive target data. > > ### Area > > Evidence, artifacts, or reconstruction > > ### Current limitation > > `build_reconstruction_obligation_ledger` closes an obligation when a manifest binding supplies a unique owner, required fixtures, and "a passing verifier whose authority is comparable to the original observation. The verifier must enumerate the obligation ID and its result must be present in the input Evidence bundle." (`docs/reconstruction-obligation-ledgers.md`). The verifier binding carries `command`, `authority`, `status: "pass"`, `result_evidence_id`, and `enumerated_obligation_ids`. > > Nothing in that binding shows the verifier can distinguish a correct reconstruction from a wrong one. A verifier that always passes, for example one that enumerates the obligation ID but asserts nothing, or one whose comparison silently skips the relevant dimension, meets the stated closure rule. The `negative` value of `obligationCaseKindSchema` (`src/domain/reconstructionObligationLedgerSchemas.ts`) describes negative inputs to the target's behavior ("reject an invalid configuration option"), not a negative control on the verifier itself. `verify_reconstruction` takes only a specification and checks claim authority; `evaluate_reconstruction_coverage` checks evidence provenance. > > REA already applies this discipline internally: `npm run verify:readiness` fails closed if "a deliberate primary divergence is not localized" and keeps "every deliberate fault" represented (`docs/reconstruction-readiness.md`). That protects REA's own comparators, not the verifiers a user binds into a ledger. > > ### Use cases > > - An agent reconstructs a protocol handler and writes a conformance test as the verifier. Before the ledger closes the obligation, it should show the same test failing against a build with one deliberate defect (wrong field order, dropped error path). > - Reviewing someone else's reconstruction claim: the ledger shows which closures rest on verifiers that were demonstrated to fail on a known-wrong input and which were never challenged. > > Open false-green and false-red reports in the reconstruction area (#3937, #3984; closed #3980) show that closure logic is already a source of incorrect results; this request addresses the verifier side, which those issues do not cover. > > ### Desired outcome and success criteria > > Let a reconstruction obligation ledger record a negative control for each verifier binding (a deliberately wrong candidate and the verifier's observed failing result), and keep closure open or unknown when no control shows that the verifier can fail. > > - A verifier binding can include one or more controls, each naming the deliberately wrong candidate (by artifact identity or fixture), the defect it introduces, and the Evidence ID of the verifier's observed failing result. > - The ledger reports, per obligation, whether its verifier has a passing result on the candidate and a failing result on at least one control; a control that also passes is reported as a vacuous verifier, not ignored. > - Closure policy is explicit: an obligation without a demonstrated failing control does not report the same closed status as one with it (either it stays open or it closes with a named limitation; maintainers choose which). > - Existing ledgers without controls still parse; their closures are labeled as unchallenged rather than silently upgraded or rejected. > - Documentation explains how to construct a one-defect control and why a control that passes is evidence against the verifier. > > ### Expected interaction with targets > > Read existing files or captured data > > ### Target and scope details (optional) > > The ledger reads Evidence and manifests only; running the verifier and the control stays with the caller, as it does today for the passing result. Non-goals: REA generating mutants automatically, or running caller verifier commands. > > ### Verification available so far > > I can suggest a sample or manual check > > ### Verification details (optional) > > Possible fixture: a ledger with two obligations bound to the same trivially passing verifier, one with a control whose result also passes and one with a control that fails. Expected: the first is reported as a vacuous verifier and not closed; the second closes. > > ### Alternatives considered > > - Relying on reviewers to notice weak verifiers: does not scale and is not recorded in Evidence. > - Expressing controls as additional `negative` cases: conflates target behavior with verifier sensitivity and would let a verifier that ignores its input satisfy both. > > ### Possible direction > > Model a control as an optional array on the verifier binding with its own `result_evidence_id` and expected `status: "fail"`, and surface the per-obligation result in the ledger output next to existing closure reasons. > > ### References > > - Checked against `main` at commit 8ab5024 (`rea-agents` 6.5.0). >
Author
Owner

@morluto commented on GitHub (Oct 26, 2026):

Thank you for the concrete false-green example. #4991 and #7178 both propose negative-control closure rules; #6367 separately addresses control-content identity and depends on #4991. These proposals remain open while the closure policy and one compatible implementation are decided, including the current proof owner/revision requirements.

<!-- gh-comment-id:6428621436 --> @morluto commented on GitHub (Oct 26, 2026): Thank you for the concrete false-green example. #4991 and #7178 both propose negative-control closure rules; #6367 separately addresses control-content identity and depends on #4991. These proposals remain open while the closure policy and one compatible implementation are decided, including the current proof owner/revision requirements.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#1748
No description provided.