[PR #2451] [CLOSED] test(agent): grade notes export facts against produced evidence #5239

Closed
opened 2026-10-31 01:05:03 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2451
Author: @optimization2026
Created: 10/13/2026
Status: ❌ Closed

Base: main ← Head: test/notes-export-factual-rubric


📝 Commits (3)

  • fde35be test(agent): add evidence-backed notes export rubric
  • 193f58b Add notes-export analysis task to verify-agent-experience
  • 9240420 Add tests for Notes export known-answer evaluation

📊 Changes

3 files changed (+485 additions, -12 deletions)

View changed files

📝 scripts/verify-agent-experience.mjs (+12 -0)
📝 src/evaluation/AgentEvaluationFixtures.ts (+145 -12)
📝 tests/boundary/mcp/agentKnownAnswerMcp.test.ts (+328 -0)

📄 Description

Summary

The Notes export scenario proposed in #1704 checks tool order, answer terms and a produced Evidence ID, but those checks do not establish that the selected facts are correct or supported. Add a guided notes-export scenario with six closed claims checked against fixture expectations and the actual delivered producer Evidence. Refs #1704 and #778.

Problem and expected behavior

A reply containing notes:export and an existing Evidence ID can still give an incorrect API, cite another operation or target, or erase a semantic unknown. The factual gate now requires the exposed API, renderer/main IPC operations, resolved preload, feature-trace source-module set and semantic-query boundary to match their supporting Evidence.

Change and scope

Reuse KnownAnswerEvaluation and the existing source-owned six-file Notes example. Package it as ASAR and independently hash its bytes in the shared fixture factory. Bind analysis claims to the intended path/digest and derived claims to delivered parent analysis, operation, authority/confidence and query semantics.

The channel query accepts REA's effective default exact match as well as explicit exact; a real agent trial exposed the initial overly strict parameter comparison. Contains matching remains rejected. Shared bridge assertions and answer descriptions serve both existing ASAR and Notes fixtures.

This is a three-file evaluation contribution directly against main. #1704 remains open at 7dfd2b6d; its website/demo portfolio is not included here. Its author will need to reconcile the overlapping scenario entry if both PRs proceed.

Contract and boundary impact

  • Semantic owner: source-owned evaluation fixtures, selected-fact rubric and verifier scenario.
  • Scope decision: not applicable to this change within existing static-analysis/evaluation boundaries.
  • CLI/MCP, tool catalog, provider behavior and Evidence schemas: unchanged.
  • Target/lifecycle: static analysis of owned packaged bytes; existing MCP harness and isolated verifier cleanup. Electron is not launched.
  • Generated metadata, installation and package distribution: unchanged; generated outputs and raw records are excluded from the diff.

Expected effects:

  • Read existing files or captured data.
  • Launch existing REA/agent verifier processes.
  • Create owned fixture/archive files and temporary isolated client configuration.
  • Contact the existing model service during the optional live agent verifier.
  • No target execution, instrumentation, mutation or protection recovery is added.

Evidence and regression coverage

Two public MCP workflow regressions exercise actual Notes analysis, feature trace and semantic trace. They accept supported claims and reject incorrect values, erased ambiguity, unrelated existing citations, missing parent Evidence, unsupported extra runtime claims, keyword-only prose, contains queries, another artifact path and changed bytes at the same path. The default exact query is explicitly accepted.

Source/module expectations are established from the owned fixture. The semantic claim checks the bounded static-query contract. The module list is the trace's returned source-module set; it is not proof that every module executed.

Validation performed

Windows x64, Node.js 24.18.0 and npm 11.16.0; source baseline ca2d15b3.

  • npm ci, npm run build:cached — passed.
  • npm run test:focused -- tests/boundary/mcp/agentKnownAnswerMcp.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts — 4 files / 69 tests passed.
  • npm run check — passed, including typecheck, lint, format, knip, metadata and architecture guards.
  • npm run jscpd — passed, 2.98% duplicated lines.
  • npm run verify:test-discovery — all 1,004 files discovered exactly once.
  • git diff --check — passed.
  • Real stdio MCP probe — all three static operations completed against packaged Notes bytes. Windows native controls came from the verified published 6.4.0 artifact; the source checkout itself does not build that addon on Windows.
  • Live isolated REA_AGENT_EVAL_SCENARIOS=notes-export npm run verify:agent with Codex CLI 0.161.0 completed three successful REA calls, no invalid or repeated calls, and correct six-claim JSON. The original command's factual gate failed only on omitted match (REA's default exact). After fixing that interpretation, the unchanged original transcript was regraded with an independently regenerated fixture digest: six claims and the existing gate passed. This is replayed grading of that original execution, not a fresh post-fix model run. The original failed report and raw transcript remain local.

The packaged fixture SHA-256 is a8f3fe3170ca3332e6bc4708c9cdcb07ce69c01433694d8b26a43636323cc07c; the original transcript SHA-256 is 0939640bfb0bbc40d53ced38333cbf3666873b73451d463eeb581163c24e3781. An earlier attempt with the installed Codex CLI 0.154.0 did not start a model task because it lacks --no-daemon.

Compatibility, safety, and release

This guided fixture does not grade arbitrary free-form factual correctness, autonomous IPC discovery, CSV formatter correctness, actual Downloads resolution or runtime clicks/writes. It adds no AISOP/AISP runtime or dependency. No performance advantage is claimed: the one live trial reported 298,309 input tokens (231,552 cached) and 1,119 output tokens; CPU, peak process-family memory and model identity were not independently established.

Existing public contracts and historical ASAR/parser rubrics are preserved. Linux/macOS model workflows, Electron runtime execution and the full local check:pr suite were not run. CI and maintainer review remain pending.

Before merge

  • One focused evaluation outcome and conventional title.
  • Existing related work and overlap are identified.
  • Changed behavior and meaningful failure paths are covered.
  • Observed, derived, inferred and unknown boundaries remain explicit.
  • Final diff excludes credentials, generated files, raw records and unrelated cleanup.
  • Maintainer review and current-head CI complete.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2451 **Author:** [@optimization2026](https://github.com/optimization2026) **Created:** 10/13/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `test/notes-export-factual-rubric` --- ### 📝 Commits (3) - [`fde35be`](https://github.com/morluto/rea/commit/fde35be10ade0bbcb12cc8409b8e9e9af632cb43) test(agent): add evidence-backed notes export rubric - [`193f58b`](https://github.com/morluto/rea/commit/193f58bb4449fd2223c1e2aea40ce17072e0ef5b) Add notes-export analysis task to verify-agent-experience - [`9240420`](https://github.com/morluto/rea/commit/92404209a458c6e9c15ecd821ab51160be78ee57) Add tests for Notes export known-answer evaluation ### 📊 Changes **3 files changed** (+485 additions, -12 deletions) <details> <summary>View changed files</summary> 📝 `scripts/verify-agent-experience.mjs` (+12 -0) 📝 `src/evaluation/AgentEvaluationFixtures.ts` (+145 -12) 📝 `tests/boundary/mcp/agentKnownAnswerMcp.test.ts` (+328 -0) </details> ### 📄 Description ## Summary The Notes export scenario proposed in #1704 checks tool order, answer terms and a produced Evidence ID, but those checks do not establish that the selected facts are correct or supported. Add a guided `notes-export` scenario with six closed claims checked against fixture expectations and the actual delivered producer Evidence. Refs #1704 and #778. ## Problem and expected behavior A reply containing `notes:export` and an existing Evidence ID can still give an incorrect API, cite another operation or target, or erase a semantic unknown. The factual gate now requires the exposed API, renderer/main IPC operations, resolved preload, feature-trace source-module set and semantic-query boundary to match their supporting Evidence. ## Change and scope Reuse `KnownAnswerEvaluation` and the existing source-owned six-file Notes example. Package it as ASAR and independently hash its bytes in the shared fixture factory. Bind analysis claims to the intended path/digest and derived claims to delivered parent analysis, operation, authority/confidence and query semantics. The channel query accepts REA's effective default exact match as well as explicit exact; a real agent trial exposed the initial overly strict parameter comparison. Contains matching remains rejected. Shared bridge assertions and answer descriptions serve both existing ASAR and Notes fixtures. This is a three-file evaluation contribution directly against main. #1704 remains open at `7dfd2b6d`; its website/demo portfolio is not included here. Its author will need to reconcile the overlapping scenario entry if both PRs proceed. ## Contract and boundary impact - Semantic owner: source-owned evaluation fixtures, selected-fact rubric and verifier scenario. - Scope decision: not applicable to this change within existing static-analysis/evaluation boundaries. - CLI/MCP, tool catalog, provider behavior and Evidence schemas: unchanged. - Target/lifecycle: static analysis of owned packaged bytes; existing MCP harness and isolated verifier cleanup. Electron is not launched. - Generated metadata, installation and package distribution: unchanged; generated outputs and raw records are excluded from the diff. Expected effects: - [x] Read existing files or captured data. - [x] Launch existing REA/agent verifier processes. - [x] Create owned fixture/archive files and temporary isolated client configuration. - [x] Contact the existing model service during the optional live agent verifier. - No target execution, instrumentation, mutation or protection recovery is added. ## Evidence and regression coverage Two public MCP workflow regressions exercise actual Notes analysis, feature trace and semantic trace. They accept supported claims and reject incorrect values, erased ambiguity, unrelated existing citations, missing parent Evidence, unsupported extra runtime claims, keyword-only prose, contains queries, another artifact path and changed bytes at the same path. The default exact query is explicitly accepted. Source/module expectations are established from the owned fixture. The semantic claim checks the bounded static-query contract. The module list is the trace's returned source-module set; it is not proof that every module executed. ## Validation performed Windows x64, Node.js 24.18.0 and npm 11.16.0; source baseline `ca2d15b3`. - `npm ci`, `npm run build:cached` — passed. - `npm run test:focused -- tests/boundary/mcp/agentKnownAnswerMcp.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts` — 4 files / 69 tests passed. - `npm run check` — passed, including typecheck, lint, format, knip, metadata and architecture guards. - `npm run jscpd` — passed, 2.98% duplicated lines. - `npm run verify:test-discovery` — all 1,004 files discovered exactly once. - `git diff --check` — passed. - Real stdio MCP probe — all three static operations completed against packaged Notes bytes. Windows native controls came from the verified published 6.4.0 artifact; the source checkout itself does not build that addon on Windows. - Live isolated `REA_AGENT_EVAL_SCENARIOS=notes-export npm run verify:agent` with Codex CLI 0.161.0 completed three successful REA calls, no invalid or repeated calls, and correct six-claim JSON. The original command's factual gate failed only on omitted `match` (REA's default exact). After fixing that interpretation, the unchanged original transcript was regraded with an independently regenerated fixture digest: six claims and the existing gate passed. This is replayed grading of that original execution, not a fresh post-fix model run. The original failed report and raw transcript remain local. The packaged fixture SHA-256 is `a8f3fe3170ca3332e6bc4708c9cdcb07ce69c01433694d8b26a43636323cc07c`; the original transcript SHA-256 is `0939640bfb0bbc40d53ced38333cbf3666873b73451d463eeb581163c24e3781`. An earlier attempt with the installed Codex CLI 0.154.0 did not start a model task because it lacks `--no-daemon`. ## Compatibility, safety, and release This guided fixture does not grade arbitrary free-form factual correctness, autonomous IPC discovery, CSV formatter correctness, actual Downloads resolution or runtime clicks/writes. It adds no AISOP/AISP runtime or dependency. No performance advantage is claimed: the one live trial reported 298,309 input tokens (231,552 cached) and 1,119 output tokens; CPU, peak process-family memory and model identity were not independently established. Existing public contracts and historical ASAR/parser rubrics are preserved. Linux/macOS model workflows, Electron runtime execution and the full local `check:pr` suite were not run. CI and maintainer review remain pending. ## Before merge - [x] One focused evaluation outcome and conventional title. - [x] Existing related work and overlap are identified. - [x] Changed behavior and meaningful failure paths are covered. - [x] Observed, derived, inferred and unknown boundaries remain explicit. - [x] Final diff excludes credentials, generated files, raw records and unrelated cleanup. - [ ] Maintainer review and current-head CI complete. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-10-31 01:05:03 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#5239
No description provided.