[PR #1924] [MERGED] test(evaluation): grade managed agent fixture facts #4913

New issue

Closed

opened 2026-10-31 00:56:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:56:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1924
Author: @Ramcharan49
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @morluto

Base: main ← Head: test/managed-agent-known-answers

📝 Commits (7)

29bdaf3 test(evaluation): grade managed agent fixture facts
2f1a05a test(evaluation): cover distinct managed artifact deliveries
de8bd6c Merge upstream main to resolve agent evaluation documentation conflict
0ad1590 fix(evaluation): require unfiltered managed type inventories
bd25ecf fix(evaluation): reject selected inventories for whole-assembly claims
0248564 chore: preserve concurrent pull request branch updates
e93f25a fix(evaluation): preserve selected entry point grounding

📊 Changes

5 files changed (+506 additions, -10 deletions)

View changed files

📝 docs/client-verification.md (+14 -5)
📝 scripts/verify-agent-experience.mjs (+9 -1)
📝 src/evaluation/AgentEvaluationFixtures.ts (+120 -1)
📝 src/evaluation/KnownAnswerEvaluation.ts (+16 -2)
📝 tests/evaluation/model-evals/knownAnswerEvaluation.test.ts (+347 -1)

📄 Description

Managed agent evaluations currently verify the artifact/member workflow without grading the answers. Add closed known-answer claims for the authored fixture's exact public top-level types and CLI entry point, bound to the freshly built DLL's SHA-256 and authenticated delivered Evidence.

Resolve the actual header token against complete member metadata under matching non-null module MVIDs. Reject fabricated values, wrong identities, decoy or duplicate methods, partial inventories, and missing or conflicting companion deliveries. The rubric conservatively rejects multiple distinct matching artifact inspections. Existing JavaScript selectors remain unchanged; documentation describes the limited static factual scope.

Validation and remaining gaps:

Focused selection: 90/94 tests passed, including 83 evaluator tests, two real managed MCP tests, and five compiled managed CLI byte-fixture tests. Four existing JavaScript known-answer boundary tests failed before grading; a diagnostic reported missing Windows native controls. These failures remain unresolved, and pristine-upstream reproduction was not obtained.
Explicit two-checker typechecking passed. Lint/module boundaries, formatting, metadata, architecture guards, and git diff checks passed.
The aggregate npm check and Knip validation are incomplete because Windows memory allocation failed. A bounded serial follow-up also failed on host resources; the full gate is not green.
The SDK-built AgentEval DLL was not validated because .NET was unavailable. Live agent evaluation was not run because separate model execution approval was not provided. Synthetic transcripts validate evaluator logic; existing real MCP/CLI byte-fixture tests do not establish SDK-built fixture or live-agent correctness.

This draft is for review with these validation gaps explicit; it is not presented as merge-ready.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1924 **Author:** [@Ramcharan49](https://github.com/Ramcharan49) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `test/managed-agent-known-answers` --- ### 📝 Commits (7) - [`29bdaf3`](https://github.com/morluto/rea/commit/29bdaf39163e14805d347a86e0b504432baf2e17) test(evaluation): grade managed agent fixture facts - [`2f1a05a`](https://github.com/morluto/rea/commit/2f1a05a95f515d74a167e6d5cc5e38e49cef3b0b) test(evaluation): cover distinct managed artifact deliveries - [`de8bd6c`](https://github.com/morluto/rea/commit/de8bd6c476ad998b09a19ebfd3e44b5ab2549e73) Merge upstream main to resolve agent evaluation documentation conflict - [`0ad1590`](https://github.com/morluto/rea/commit/0ad1590d6c1824a090b958b29ff10933dc1594a8) fix(evaluation): require unfiltered managed type inventories - [`bd25ecf`](https://github.com/morluto/rea/commit/bd25ecf5e24a3380d01a7a3e12557cddc2050c20) fix(evaluation): reject selected inventories for whole-assembly claims - [`0248564`](https://github.com/morluto/rea/commit/0248564f70178209d30714bfb94b614290c02714) chore: preserve concurrent pull request branch updates - [`e93f25a`](https://github.com/morluto/rea/commit/e93f25a9b40a390098291bf7bf12cdb784f6a097) fix(evaluation): preserve selected entry point grounding ### 📊 Changes **5 files changed** (+506 additions, -10 deletions) <details> <summary>View changed files</summary> 📝 `docs/client-verification.md` (+14 -5) 📝 `scripts/verify-agent-experience.mjs` (+9 -1) 📝 `src/evaluation/AgentEvaluationFixtures.ts` (+120 -1) 📝 `src/evaluation/KnownAnswerEvaluation.ts` (+16 -2) 📝 `tests/evaluation/model-evals/knownAnswerEvaluation.test.ts` (+347 -1) </details> ### 📄 Description Managed agent evaluations currently verify the artifact/member workflow without grading the answers. Add closed known-answer claims for the authored fixture's exact public top-level types and CLI entry point, bound to the freshly built DLL's SHA-256 and authenticated delivered Evidence. Resolve the actual header token against complete member metadata under matching non-null module MVIDs. Reject fabricated values, wrong identities, decoy or duplicate methods, partial inventories, and missing or conflicting companion deliveries. The rubric conservatively rejects multiple distinct matching artifact inspections. Existing JavaScript selectors remain unchanged; documentation describes the limited static factual scope. Validation and remaining gaps: - Focused selection: 90/94 tests passed, including 83 evaluator tests, two real managed MCP tests, and five compiled managed CLI byte-fixture tests. Four existing JavaScript known-answer boundary tests failed before grading; a diagnostic reported missing Windows native controls. These failures remain unresolved, and pristine-upstream reproduction was not obtained. - Explicit two-checker typechecking passed. Lint/module boundaries, formatting, metadata, architecture guards, and git diff checks passed. - The aggregate npm check and Knip validation are incomplete because Windows memory allocation failed. A bounded serial follow-up also failed on host resources; the full gate is not green. - The SDK-built AgentEval DLL was not validated because .NET was unavailable. Live agent evaluation was not run because separate model execution approval was not provided. Synthetic transcripts validate evaluator logic; existing real MCP/CLI byte-fixture tests do not establish SDK-built fixture or live-agent correctness. This draft is for review with these validation gaps explicit; it is not presented as merge-ready. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:56:43 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4913

No description provided.

Rows
Columns