[PR #1211] [MERGED] fix!: restore caching, preserve failures, and verify fixture claims #2724

New issue

Closed

opened 2026-10-17 21:09:30 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:09:30 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1211
Author: @Y0UW1SH
Created: 10/9/2026
Status: ✅ Merged
Merged: 10/9/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/analysis-cache-cleanup

📝 Commits (5)

91732cb fix!: preserve analysis cleanup errors and label evaluation heuristics
040701e fix(ghidra): cache immutable direct and composed analysis results
5f36e26 chore: integrate main for snapshot cache verification
d538c97 fix(evaluation)!: grade fixture claims against returned evidence
4488557 fix(ci): register the agent evaluation fixture entry

📊 Changes

20 files changed (+2711 additions, -207 deletions)

View changed files

📝 docs/testing.md (+42 -2)
📝 knip.json (+1 -0)
📝 scripts/verify-agent-experience.mjs (+39 -74)
📝 src/application/AnalysisProvider.ts (+6 -1)
📝 src/application/DirectAnalysis.ts (+117 -108)
➕ src/application/DirectAnalysisCleanup.ts (+97 -0)
📝 src/application/binary/AnalysisSnapshotCache.test.ts (+68 -0)
📝 src/application/binary/AnalysisSnapshotCache.ts (+2 -1)
➕ src/evaluation/AgentEvaluationFixtures.ts (+315 -0)
➕ src/evaluation/AgentEvaluationReport.ts (+56 -0)
📝 src/evaluation/CodexAgentEval.ts (+32 -12)
➕ src/evaluation/KnownAnswerEvaluation.ts (+421 -0)
📝 src/ghidra/GhidraProviderCapabilities.ts (+6 -0)
➕ src/ghidra/GhidraSnapshotCache.test.ts (+236 -0)
➕ tests/boundary/mcp/agentKnownAnswerMcp.test.ts (+302 -0)
➕ tests/composition/directAnalysisCleanup.test.ts (+242 -0)
📝 tests/composition/directAnalysisSnapshotReplay.test.ts (+59 -0)
➕ tests/evaluation/model-evals/agentEvaluationReport.test.ts (+163 -0)
📝 tests/evaluation/model-evals/codexAgentEval.test.ts (+58 -9)
➕ tests/evaluation/model-evals/knownAnswerEvaluation.test.ts (+449 -0)

📄 Description

Summary

Repeated Ghidra analysis bypassed reusable snapshots, one-shot cleanup discarded failures, and answer keyword/citation matching could accept fabricated claims. This PR fixes those paths and adds evidence-backed correctness checks for configured benchmark fixtures.

Based on main at 5b769d3a, including its xrefs resolution and cleanup updates.

Changes

Enable snapshots for explicitly deterministic Ghidra reads while retaining live-state, mutation, permission, document and cursor exclusions. Preserve annotation invalidation and composed workflow replay.
Preserve completed native Evidence, managed execution results, and primary errors when session cleanup also fails.
Grade packaged desktop and parser-comparison answers against known fixture values and complete Evidence returned by successful REA calls.
Require a closed JSON claim manifest with exact claim IDs, values, citations, confidence and authority. Reject missing, duplicate, extra, contradictory or unsupported claims, altered Evidence, conflicting result aliases, and malformed manifests.
Bind desktop claims to the artifact path and SHA-256. Bind parser claims to the requested module/export selectors and the two delivered source analyses.
Keep golden values outside the prompts. Keep routing checks and text heuristics separate from factual assessment.
Require managed member inspection for the existing public-type question.

Assessment scope and output

The desktop rubric covers the bridge API, members, world, IPC operations and resolved preload. The parser rubric covers the exact heading depth addition, discriminator, comparison coverage, and limits on runtime conclusions.

A factual pass establishes only these configured fixture claims. Native, managed, browser, navigation-context and address-context scenarios remain not_assessed. Their routing and text checks remain active.

Scenario results contain factualCorrectness and per-claim failure diagnostics. Factual failures fail the benchmark gate even when all keyword and citation heuristics pass. Correct structured claims do not need superficial prose cues to pass.

CLI/MCP tool names and schemas are unchanged. Cleanup failures use the existing cleanup_incomplete envelope and retain observations or primary diagnostics. No bridge, installation, dependency, or generated-output changes.

Tested

209 tests passed across 19 affected files, covering evaluator regressions, report/exit gating, actual MCP desktop/parser producers, CLI/MCP boundaries, managed cleanup, xrefs, snapshots and client cleanup.
Packaged desktop and parser fixtures exercised through actual MCP calls. Correct claims pass; wrong values, extra claims, unrelated citations, changed artifact identity, missing source links and runtime overclaims fail.
npm run build:cached: passed.
npm run check:fast: build, strict typecheck and lint passed.
Oxfmt checks, whitespace checks and runner syntax check: passed.
Independent grader and fixture/report reviews: approved.

Additional Ghidra client socket fixture suite: 5 passed, 19 failed, 1 skipped. The failed cases report listen EPERM. The client, socket fixture and test code are unchanged by the evaluation fix.

Not tested

Live-agent benchmark execution.
Real Ghidra or Hopper engine execution.

Compatibility and migration

BREAKING CHANGE: evaluation reports use schema version 3. Top-level factualCorrectness changes from a constant string to a summary containing status, scope and scenario counts. Per-scenario correctness is passed, failed or not_assessed, with configured_fixture_claims scope.

Report consumers must also migrate the earlier renamed fields: completionQuality → answerHeuristicsMet, authorityHonesty → epistemicCuePresent, contentCriteriaMet → answerTermCoverageMet, and qualityCriteria → requiredAnswerTermGroups. These remain diagnostic heuristics, not factual assessments. Migration and benchmark scope are documented in docs/testing.md.

Review

Meaningful failure paths and producer representations covered.
Artifact identity, source provenance, observations and primary failures preserved.
Unsupported and unassessed conclusions remain explicit.
Final diff reviewed for unrelated changes and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1211 **Author:** [@Y0UW1SH](https://github.com/Y0UW1SH) **Created:** 10/9/2026 **Status:** ✅ Merged **Merged:** 10/9/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/analysis-cache-cleanup` --- ### 📝 Commits (5) - [`91732cb`](https://github.com/morluto/rea/commit/91732cba6de590a4fc9db224fb32f8ee388cf7de) fix!: preserve analysis cleanup errors and label evaluation heuristics - [`040701e`](https://github.com/morluto/rea/commit/040701ec85818df6283b5d9a4c3e19acedd80e32) fix(ghidra): cache immutable direct and composed analysis results - [`5f36e26`](https://github.com/morluto/rea/commit/5f36e263b21270b47d4146931aaf1bf8538243cd) chore: integrate main for snapshot cache verification - [`d538c97`](https://github.com/morluto/rea/commit/d538c97dafc1471195aff397dcfca0017640824a) fix(evaluation)!: grade fixture claims against returned evidence - [`4488557`](https://github.com/morluto/rea/commit/4488557973c125b4bb86eab8e6c2cafb9a7a9a59) fix(ci): register the agent evaluation fixture entry ### 📊 Changes **20 files changed** (+2711 additions, -207 deletions) <details> <summary>View changed files</summary> 📝 `docs/testing.md` (+42 -2) 📝 `knip.json` (+1 -0) 📝 `scripts/verify-agent-experience.mjs` (+39 -74) 📝 `src/application/AnalysisProvider.ts` (+6 -1) 📝 `src/application/DirectAnalysis.ts` (+117 -108) ➕ `src/application/DirectAnalysisCleanup.ts` (+97 -0) 📝 `src/application/binary/AnalysisSnapshotCache.test.ts` (+68 -0) 📝 `src/application/binary/AnalysisSnapshotCache.ts` (+2 -1) ➕ `src/evaluation/AgentEvaluationFixtures.ts` (+315 -0) ➕ `src/evaluation/AgentEvaluationReport.ts` (+56 -0) 📝 `src/evaluation/CodexAgentEval.ts` (+32 -12) ➕ `src/evaluation/KnownAnswerEvaluation.ts` (+421 -0) 📝 `src/ghidra/GhidraProviderCapabilities.ts` (+6 -0) ➕ `src/ghidra/GhidraSnapshotCache.test.ts` (+236 -0) ➕ `tests/boundary/mcp/agentKnownAnswerMcp.test.ts` (+302 -0) ➕ `tests/composition/directAnalysisCleanup.test.ts` (+242 -0) 📝 `tests/composition/directAnalysisSnapshotReplay.test.ts` (+59 -0) ➕ `tests/evaluation/model-evals/agentEvaluationReport.test.ts` (+163 -0) 📝 `tests/evaluation/model-evals/codexAgentEval.test.ts` (+58 -9) ➕ `tests/evaluation/model-evals/knownAnswerEvaluation.test.ts` (+449 -0) </details> ### 📄 Description ## Summary Repeated Ghidra analysis bypassed reusable snapshots, one-shot cleanup discarded failures, and answer keyword/citation matching could accept fabricated claims. This PR fixes those paths and adds evidence-backed correctness checks for configured benchmark fixtures. Based on `main` at `5b769d3a`, including its xrefs resolution and cleanup updates. ## Changes - Enable snapshots for explicitly deterministic Ghidra reads while retaining live-state, mutation, permission, document and cursor exclusions. Preserve annotation invalidation and composed workflow replay. - Preserve completed native Evidence, managed execution results, and primary errors when session cleanup also fails. - Grade packaged desktop and parser-comparison answers against known fixture values and complete Evidence returned by successful REA calls. - Require a closed JSON claim manifest with exact claim IDs, values, citations, confidence and authority. Reject missing, duplicate, extra, contradictory or unsupported claims, altered Evidence, conflicting result aliases, and malformed manifests. - Bind desktop claims to the artifact path and SHA-256. Bind parser claims to the requested module/export selectors and the two delivered source analyses. - Keep golden values outside the prompts. Keep routing checks and text heuristics separate from factual assessment. - Require managed member inspection for the existing public-type question. ## Assessment scope and output The desktop rubric covers the bridge API, members, world, IPC operations and resolved preload. The parser rubric covers the exact heading depth addition, discriminator, comparison coverage, and limits on runtime conclusions. A factual pass establishes only these configured fixture claims. Native, managed, browser, navigation-context and address-context scenarios remain `not_assessed`. Their routing and text checks remain active. Scenario results contain `factualCorrectness` and per-claim failure diagnostics. Factual failures fail the benchmark gate even when all keyword and citation heuristics pass. Correct structured claims do not need superficial prose cues to pass. CLI/MCP tool names and schemas are unchanged. Cleanup failures use the existing `cleanup_incomplete` envelope and retain observations or primary diagnostics. No bridge, installation, dependency, or generated-output changes. ## Tested - **209 tests passed across 19 affected files**, covering evaluator regressions, report/exit gating, actual MCP desktop/parser producers, CLI/MCP boundaries, managed cleanup, xrefs, snapshots and client cleanup. - Packaged desktop and parser fixtures exercised through actual MCP calls. Correct claims pass; wrong values, extra claims, unrelated citations, changed artifact identity, missing source links and runtime overclaims fail. - `npm run build:cached`: passed. - `npm run check:fast`: build, strict typecheck and lint passed. - Oxfmt checks, whitespace checks and runner syntax check: passed. - Independent grader and fixture/report reviews: approved. Additional Ghidra client socket fixture suite: **5 passed, 19 failed, 1 skipped**. The failed cases report `listen EPERM`. The client, socket fixture and test code are unchanged by the evaluation fix. ## Not tested - Live-agent benchmark execution. - Real Ghidra or Hopper engine execution. ## Compatibility and migration **BREAKING CHANGE:** evaluation reports use schema version 3. Top-level `factualCorrectness` changes from a constant string to a summary containing `status`, `scope` and scenario counts. Per-scenario correctness is `passed`, `failed` or `not_assessed`, with `configured_fixture_claims` scope. Report consumers must also migrate the earlier renamed fields: `completionQuality` → `answerHeuristicsMet`, `authorityHonesty` → `epistemicCuePresent`, `contentCriteriaMet` → `answerTermCoverageMet`, and `qualityCriteria` → `requiredAnswerTermGroups`. These remain diagnostic heuristics, not factual assessments. Migration and benchmark scope are documented in `docs/testing.md`. ## Review - [x] Meaningful failure paths and producer representations covered. - [x] Artifact identity, source provenance, observations and primary failures preserved. - [x] Unsupported and unassessed conclusions remain explicit. - [x] Final diff reviewed for unrelated changes and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:09:30 +00:00