[GH-ISSUE #8475] [RFC] Benchmark oracle adapter: expose rea_verify_reconstruction for AgentRE-Bench / SRE-Bench task packs #6619

New issue

Open

opened 2026-11-01 21:12:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:44 +00:00

Owner

Originally created by @lostsock1 on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8475

Context

Agent-assisted RE benchmarks (AgentRE-Bench, SRE-Bench) appear in the same community resource maps that feature REA, but there is no adapter connecting them. REA already has the hard part: rea_verify_reconstruction scores finite typed claims against authenticated Evidence, and reconstruction-obligation ledgers + rea_verify_unknown_resolution give per-unknown resolution semantics. That is a natural scoring oracle for benchmark tasks, and REA's worked showcases (Notes export, CTF Flag Checker) are ready-made golden tasks.

Suggested direction

A benchmark adapter: run a task pack through the public CLI/MCP surface and score agent claims via rea_verify_reconstruction (pass/fail/unknown per claim), emitting a machine-readable results file with Evidence IDs for audit.
Ship a small offline task pack (bundle the Notes-style fixture target + rules + claims) so agents can be scored without external downloads.
Keeps REA's core promise intact: a passing score means claims had complete comparable authority, not global source equivalence.

Open questions

Should the adapter live in-repo (a rea benchmark command) or as a separate harness package consuming the MCP surface?
Scoring semantics: is claim-level verify_reconstruction output the agreed metric, or do benchmarks require task-level pass/fail rollups?
Is there appetite for publishing REA's own task set to those benchmarks rather than only consuming theirs?

Originally created by @lostsock1 on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8475 ## Context Agent-assisted RE benchmarks (AgentRE-Bench, SRE-Bench) appear in the same community resource maps that feature REA, but there is no adapter connecting them. REA already has the hard part: `rea_verify_reconstruction` scores finite typed claims against authenticated Evidence, and reconstruction-obligation ledgers + `rea_verify_unknown_resolution` give per-unknown resolution semantics. That is a natural scoring oracle for benchmark tasks, and REA's worked showcases (Notes export, CTF Flag Checker) are ready-made golden tasks. ## Suggested direction - A benchmark adapter: run a task pack through the public CLI/MCP surface and score agent claims via `rea_verify_reconstruction` (pass/fail/unknown per claim), emitting a machine-readable results file with Evidence IDs for audit. - Ship a small offline task pack (bundle the Notes-style fixture target + rules + claims) so agents can be scored without external downloads. - Keeps REA's core promise intact: a passing score means claims had complete comparable authority, not global source equivalence. ## Open questions 1. Should the adapter live in-repo (a `rea benchmark` command) or as a separate harness package consuming the MCP surface? 2. Scoring semantics: is claim-level verify_reconstruction output the agreed metric, or do benchmarks require task-level pass/fail rollups? 3. Is there appetite for publishing REA's own task set to those benchmarks rather than only consuming theirs?

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6619

No description provided.

Rows
Columns