[PR #4160] [CLOSED] fix(reconstruction): keep overall behavioral claims unknown when capture unknowns remain #6099

Closed
opened 2026-10-31 01:11:51 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4160
Author: @rudycelekli
Created: 10/15/2026
Status: ❌ Closed

Base: main ← Head: fix/reconstruction-capture-unknowns


📝 Commits (1)

  • 2d01247 fix(reconstruction): keep overall behavioral claims unknown when capture unknowns remain

📊 Changes

2 files changed (+45 additions, -5 deletions)

View changed files

📝 src/domain/reconstructionVerification.test.ts (+37 -0)
📝 src/domain/reconstructionVerification.ts (+8 -5)

📄 Description

Summary

verify_reconstruction passed an overall behavioral claim when the cited process comparison had status unchanged but first_divergence.status: "unknown" ("Residual unknowns prevent proving that no divergence occurred."). The overall claim now stays unknown; the comparison's reason is added to the claim limitations.

Fixes #3980

AI-assisted: written with Claude Sonnet 5.5 via Claude Code; a human has not yet reviewed it.

Problem and expected behavior

Trigger: two compatible rea.process-capture Evidence records whose residual_unknowns has an environment entry, compared with compare_process_captures and then verified with a behavioral claim with dimension: "overall". The comparison is unchanged (all five dimensions unchanged) with an unknown first divergence. verification.status was pass with no unknown IDs and no recommended probes. Invariant: limited or unresolved evidence must not produce a pass.

Change and scope

comparisonObservation (src/domain/reconstructionVerification.ts), the owner of how a comparison maps to an observed status, now reports unknown for an overall behavioral claim when status === "unchanged" and first_divergence.status === "unknown". Observed differences still outrank (changed stays fail), and per-dimension claims are untouched, since the issue does not ask for them to block. The same rule applies when the unknown divergence comes from differing capture normalization, which compareProcessCaptures handles identically. The warning is kept visible, not removed. Not included: any change to the comparison tool, which already carries the unknown correctly.

Follows the contribution scope: ordinary fix within existing boundaries; maintainer scope decision not applicable.

Contract and boundary impact

  • Semantic owner and earliest changed stage: verifyReconstruction (domain), interpretation of a process comparison for an overall claim
  • Contribution scope fit and linked maintainer decision: not applicable
  • CLI and MCP/tool-catalog contract: no schema change; verify_reconstruction returns unknown instead of pass in the case above, which the tool contract already requires
  • Provider, bridge, target-format, or platform compatibility: not applicable
  • Evidence, artifact, provenance, or reconstruction contract: overall behavioral verdict now respects comparison-level unknowns. Verification of an unknown result registers its own residual unknown and recommended probe through the existing path
  • Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable
  • Generated metadata, package, or installation impact: none

Expected effects:

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Executed evidence.

  • Tests added or updated: src/domain/reconstructionVerification.test.ts, new test "keeps an overall claim unknown when residual capture unknowns prevent proving no divergence". It also asserts that a terminal-dimension claim on the same comparison still passes.
  • Base reproduction or other evidence: the issue's production stdio MCP script run unmodified on upstream/main 9d6651e5b. Output, abbreviated:
    • complete: verification_status: pass
    • environment unknown: comparison_status: unchanged, first_divergence: unknown, verification_status: pass, claim_status: pass, unknown_ids: 0, recommended_probes: 0
    • terminal unknown: verification_status: unknown
  • User-visible CLI/MCP output after the fix (same script, same fresh-HOME setup, assertion on the environment verdict flipped to expect unknown):
    • complete: verification_status: pass, claim_status: pass (unchanged)
    • environment unknown: comparison_status: unchanged, first_divergence: unknown, verification_status: unknown, claim_status: unknown, recommended_probes: 1; limitations now include both "Inherited host environment variables are not recorded." and "Residual unknowns prevent proving that no divergence occurred."
    • terminal unknown: unchanged (unknown, 1 unknown ID)
  • Regression failure on base: the new test fails with the src/domain/reconstructionVerification.ts change stashed (1 failed, 10 passed), and passes with it (11 passed).
  • Remaining proof gaps:
    • Real process capture execution was not exercised, only synthetic canonical Evidence as in the issue.
    • The original script's last step (record_unknown citing the comparison, then repeating the verification) now fails with a retryable revision_conflict at the repeat verification, because the unknown verification now records its own unknown. I did not investigate whether this is a pre-existing registry interaction for any unknown verify result. The workaround is no longer needed for this case.
    • tests/acceptance/tools/localToolContracts.test.ts and other boundary tests mentioning verification were not run.

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run build:cached — passed
  • npm run test:local -- src/domain/reconstructionVerification.test.ts — 11 passed
  • npm run test:local -- src/domain/reconstructionVerification.functionSources.test.ts — 69 passed
  • npm run test:focused -- tests/boundary/mcp/processEvidenceIdentityMcp.test.ts tests/boundary/mcp/investigationAnalysisMcp.test.ts — 38 passed
  • npm run check:fast — typecheck and lint passed
  • Issue reproduction script (before and after, see above)
  • npm run verify:test-discovery — not run (no new test files)
  • npm run check:pr — not run

Compatibility, safety, and release

  • Breaking changes or migration steps: none. A verdict that was incorrectly pass is now unknown.
  • Real Hopper/Ghidra, browser, or OS coverage: none; macOS arm64 with Node 24.5.0, synthetic captures only
  • Package or release metadata impact: none
  • Security, privacy, process, or containment review: not applicable; no target execution

Author readiness

Human owner:

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement.
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked, or the reason for not linking one is stated above.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🤖 Generated with Claude Code


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4160 **Author:** [@rudycelekli](https://github.com/rudycelekli) **Created:** 10/15/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/reconstruction-capture-unknowns` --- ### 📝 Commits (1) - [`2d01247`](https://github.com/morluto/rea/commit/2d01247b5fcc0e32ef0d03b96b9c6bd7b4567fe2) fix(reconstruction): keep overall behavioral claims unknown when capture unknowns remain ### 📊 Changes **2 files changed** (+45 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `src/domain/reconstructionVerification.test.ts` (+37 -0) 📝 `src/domain/reconstructionVerification.ts` (+8 -5) </details> ### 📄 Description <!-- PR title: fix(reconstruction): keep overall behavioral claims unknown when capture unknowns remain --> ## Summary `verify_reconstruction` passed an overall behavioral claim when the cited process comparison had status `unchanged` but `first_divergence.status: "unknown"` ("Residual unknowns prevent proving that no divergence occurred."). The overall claim now stays `unknown`; the comparison's reason is added to the claim limitations. Fixes #3980 AI-assisted: written with Claude Sonnet 5.5 via Claude Code; a human has not yet reviewed it. ## Problem and expected behavior Trigger: two compatible `rea.process-capture` Evidence records whose `residual_unknowns` has an `environment` entry, compared with `compare_process_captures` and then verified with a `behavioral` claim with `dimension: "overall"`. The comparison is `unchanged` (all five dimensions unchanged) with an unknown first divergence. `verification.status` was `pass` with no unknown IDs and no recommended probes. Invariant: limited or unresolved evidence must not produce a pass. ## Change and scope `comparisonObservation` (src/domain/reconstructionVerification.ts), the owner of how a comparison maps to an observed status, now reports `unknown` for an `overall` behavioral claim when `status === "unchanged"` and `first_divergence.status === "unknown"`. Observed differences still outrank (`changed` stays `fail`), and per-dimension claims are untouched, since the issue does not ask for them to block. The same rule applies when the unknown divergence comes from differing capture normalization, which `compareProcessCaptures` handles identically. The warning is kept visible, not removed. Not included: any change to the comparison tool, which already carries the unknown correctly. Follows the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope): ordinary fix within existing boundaries; maintainer scope decision not applicable. ## Contract and boundary impact - Semantic owner and earliest changed stage: `verifyReconstruction` (domain), interpretation of a process comparison for an overall claim - Contribution scope fit and linked maintainer decision: not applicable - CLI and MCP/tool-catalog contract: no schema change; `verify_reconstruction` returns `unknown` instead of `pass` in the case above, which the tool contract already requires - Provider, bridge, target-format, or platform compatibility: not applicable - Evidence, artifact, provenance, or reconstruction contract: overall behavioral verdict now respects comparison-level unknowns. Verification of an unknown result registers its own residual unknown and recommended probe through the existing path - Target selection, authority, upstream behavior, and cleanup details (if applicable): not applicable - Generated metadata, package, or installation impact: none Expected effects: - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage Executed evidence. - Tests added or updated: `src/domain/reconstructionVerification.test.ts`, new test "keeps an overall claim unknown when residual capture unknowns prevent proving no divergence". It also asserts that a terminal-dimension claim on the same comparison still passes. - Base reproduction or other evidence: the issue's production stdio MCP script run unmodified on upstream/main 9d6651e5b. Output, abbreviated: - complete: `verification_status: pass` - environment unknown: `comparison_status: unchanged, first_divergence: unknown, verification_status: pass, claim_status: pass, unknown_ids: 0, recommended_probes: 0` - terminal unknown: `verification_status: unknown` - User-visible CLI/MCP output after the fix (same script, same fresh-HOME setup, assertion on the environment verdict flipped to expect `unknown`): - complete: `verification_status: pass, claim_status: pass` (unchanged) - environment unknown: `comparison_status: unchanged, first_divergence: unknown, verification_status: unknown, claim_status: unknown, recommended_probes: 1`; limitations now include both "Inherited host environment variables are not recorded." and "Residual unknowns prevent proving that no divergence occurred." - terminal unknown: unchanged (`unknown`, 1 unknown ID) - Regression failure on base: the new test fails with the `src/domain/reconstructionVerification.ts` change stashed (1 failed, 10 passed), and passes with it (11 passed). - Remaining proof gaps: - Real process capture execution was not exercised, only synthetic canonical Evidence as in the issue. - The original script's last step (record_unknown citing the comparison, then repeating the verification) now fails with a retryable `revision_conflict` at the repeat verification, because the unknown verification now records its own unknown. I did not investigate whether this is a pre-existing registry interaction for any unknown verify result. The workaround is no longer needed for this case. - `tests/acceptance/tools/localToolContracts.test.ts` and other boundary tests mentioning verification were not run. For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run build:cached` — passed - `npm run test:local -- src/domain/reconstructionVerification.test.ts` — 11 passed - `npm run test:local -- src/domain/reconstructionVerification.functionSources.test.ts` — 69 passed - `npm run test:focused -- tests/boundary/mcp/processEvidenceIdentityMcp.test.ts tests/boundary/mcp/investigationAnalysisMcp.test.ts` — 38 passed - `npm run check:fast` — typecheck and lint passed - Issue reproduction script (before and after, see above) - `npm run verify:test-discovery` — not run (no new test files) - `npm run check:pr` — not run ## Compatibility, safety, and release - Breaking changes or migration steps: none. A verdict that was incorrectly `pass` is now `unknown`. - Real Hopper/Ghidra, browser, or OS coverage: none; macOS arm64 with Node 24.5.0, synthetic captures only - Package or release metadata impact: none - Security, privacy, process, or containment review: not applicable; no target execution ## Author readiness Human owner: <!-- @GitHub-handle --> - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [ ] This PR addresses a concrete problem or an agreed enhancement. - [ ] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [ ] The PR has one focused outcome and the title follows `type(scope): outcome`. - [ ] Related issue is linked, or the reason for not linking one is stated above. - [ ] Tests cover changed observable behavior and meaningful failure paths. - [ ] Owning docs, contracts, and generated metadata are updated where needed. - [ ] User-visible CLI/MCP changes include representative output. - [ ] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:11:51 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#6099
No description provided.