morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #3045] [CLOSED] fix(evaluation): require explicit REA server identity for routing credit #5578

New issue

Closed

opened 2026-10-31 01:07:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:07:43 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/3045
Author: @Arthurandrealmeida
Created: 10/14/2026
Status: ❌ Closed

Base: main ← Head: fix/evaluation-server-identity

📝 Commits (3)

88ebf2c fix(evaluation): require explicit REA server identity for routing credit
276fc41 test(evaluation): cover missing and explicit MCP server identities
3171c07 Merge origin/main into fix/evaluation-server-identity

📊 Changes

1 file changed (+43 additions, -0 deletions)

View changed files

📝 tests/evaluation/model-evals/codexAgentEval.test.ts (+43 -0)

📄 Description

Summary

Fixes #2615.

Require explicit REA server identity before a completed MCP call receives REA routing credit. Cover missing identity, foreign servers, both identity fields, and case-insensitive REA names.

Problem and expected behavior

A completed call naming inspect_managed_artifact but omitting both server and server_name currently counts as REA use, the correct first tool, and a successful required tool sequence. The tool name alone does not identify its server.

For that transcript, the corrected metrics are naturalUse: false, firstTool: null, correctFirstTool: false, and requiredToolSubsequenceMet: false. Explicit rea identity continues to receive credit through either field, case-insensitively.

Change and scope

The existing transcript parser still owns alias resolution, completion validation, arguments, and Evidence authentication. Its parsed server identity now controls the routing filter consistently with the Evidence path. Replace the old test that credited unidentified legacy calls with negative and positive identity cases.

This is an ordinary evaluator bug fix within the existing contribution scope; an advance maintainer scope decision is not applicable. No new provider, target interaction, execution authority, or installation behavior is introduced.

Contract and boundary impact

Semantic owner: evaluateCodexEvents, when selecting parsed REA calls for transcript metrics.
CLI/MCP schemas, tool catalog, provider/platform support, target selection, and cleanup: unchanged.
Evidence authentication, retained artifact identity, provenance, and factual assessment: unchanged. Unknown server identity cannot establish REA routing credit.
Generated metadata, packaging, and installation: no changes.
Expected effect: evaluate caller-supplied transcript data in memory; no target interaction or network requests added.

Evidence and regression coverage

Executed the updated evaluator suite before the production fix: the missing-identity case failed with the incorrect routing metrics; 27 other tests passed.
After the fix, all 101 tests in the evaluator, known-answer assessment, and evaluation-report suites passed.
Regression cases cover absent identity, foreign server and server_name, and explicit rea/REA through both fields. Existing conflicting-field, Evidence, error, and workflow tests remain covered.
A read-only review found no actionable issues.

Validation performed

Using the pinned Node.js 24.18.0 and npm 11.16.0 on Linux x64:

npm run test:local -- tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts — 3 files, 101 tests passed.
npm run check — passed typecheck, compile, format, lint, Knip, package-metadata checks, module boundaries, and architecture guards. Lint reported 0 errors and 1,211 warnings.
git diff --check — passed.

Compatibility, safety, and release

Legacy transcripts without server identity no longer establish REA usage; record server: "rea" or server_name: "rea" to identify those calls. No schema or migration changes. A live model trial, full deterministic suite, and real native providers were not run; the exercised behavior is the production transcript evaluator and its related assessment/report consumers.

Before merge

Concrete issue linked; one focused outcome and conventional title.
Scope and authority questions resolved for this ordinary fix.
Regression demonstrated on the affected base and passed after the fix.
Relevant static checks and related evaluator suites passed.
No documentation, catalog, or package updates required.
Final diff reviewed for secrets, unrelated changes, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/3045 **Author:** [@Arthurandrealmeida](https://github.com/Arthurandrealmeida) **Created:** 10/14/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/evaluation-server-identity` --- ### 📝 Commits (3) - [`88ebf2c`](https://github.com/morluto/rea/commit/88ebf2c32d247de83fa17a8ab5aa3f82b1ab0cde) fix(evaluation): require explicit REA server identity for routing credit - [`276fc41`](https://github.com/morluto/rea/commit/276fc419c6719bd54902eaccc565ef1a9b06429e) test(evaluation): cover missing and explicit MCP server identities - [`3171c07`](https://github.com/morluto/rea/commit/3171c0709718d8dcf027f5f7a8f179f6744070c4) Merge origin/main into fix/evaluation-server-identity ### 📊 Changes **1 file changed** (+43 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `tests/evaluation/model-evals/codexAgentEval.test.ts` (+43 -0) </details> ### 📄 Description ## Summary Fixes #2615. Require explicit REA server identity before a completed MCP call receives REA routing credit. Cover missing identity, foreign servers, both identity fields, and case-insensitive REA names. ## Problem and expected behavior A completed call naming `inspect_managed_artifact` but omitting both `server` and `server_name` currently counts as REA use, the correct first tool, and a successful required tool sequence. The tool name alone does not identify its server. For that transcript, the corrected metrics are `naturalUse: false`, `firstTool: null`, `correctFirstTool: false`, and `requiredToolSubsequenceMet: false`. Explicit `rea` identity continues to receive credit through either field, case-insensitively. ## Change and scope The existing transcript parser still owns alias resolution, completion validation, arguments, and Evidence authentication. Its parsed server identity now controls the routing filter consistently with the Evidence path. Replace the old test that credited unidentified legacy calls with negative and positive identity cases. This is an ordinary evaluator bug fix within the existing contribution scope; an advance maintainer scope decision is not applicable. No new provider, target interaction, execution authority, or installation behavior is introduced. ## Contract and boundary impact - Semantic owner: `evaluateCodexEvents`, when selecting parsed REA calls for transcript metrics. - CLI/MCP schemas, tool catalog, provider/platform support, target selection, and cleanup: unchanged. - Evidence authentication, retained artifact identity, provenance, and factual assessment: unchanged. Unknown server identity cannot establish REA routing credit. - Generated metadata, packaging, and installation: no changes. - Expected effect: evaluate caller-supplied transcript data in memory; no target interaction or network requests added. ## Evidence and regression coverage - Executed the updated evaluator suite before the production fix: the missing-identity case failed with the incorrect routing metrics; 27 other tests passed. - After the fix, all 101 tests in the evaluator, known-answer assessment, and evaluation-report suites passed. - Regression cases cover absent identity, foreign `server` and `server_name`, and explicit `rea`/`REA` through both fields. Existing conflicting-field, Evidence, error, and workflow tests remain covered. - A read-only review found no actionable issues. ## Validation performed Using the pinned Node.js 24.18.0 and npm 11.16.0 on Linux x64: - `npm run test:local -- tests/evaluation/model-evals/codexAgentEval.test.ts tests/evaluation/model-evals/knownAnswerEvaluation.test.ts tests/evaluation/model-evals/agentEvaluationReport.test.ts` — 3 files, 101 tests passed. - `npm run check` — passed typecheck, compile, format, lint, Knip, package-metadata checks, module boundaries, and architecture guards. Lint reported 0 errors and 1,211 warnings. - `git diff --check` — passed. ## Compatibility, safety, and release Legacy transcripts without server identity no longer establish REA usage; record `server: "rea"` or `server_name: "rea"` to identify those calls. No schema or migration changes. A live model trial, full deterministic suite, and real native providers were not run; the exercised behavior is the production transcript evaluator and its related assessment/report consumers. ## Before merge - [x] Concrete issue linked; one focused outcome and conventional title. - [x] Scope and authority questions resolved for this ordinary fix. - [x] Regression demonstrated on the affected base and passed after the fix. - [x] Relevant static checks and related evaluator suites passed. - [x] No documentation, catalog, or package updates required. - [x] Final diff reviewed for secrets, unrelated changes, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 01:07:43 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:44:18 +00:00

[PR #5544] fix(cli): reject invalid token-window bounds #6955

bsakaguchi referenced this issue

2026-11-01 21:44:26 +00:00

[PR #5578] [CLOSED] fix(cli): reject negative and fractional token window values #6976

bsakaguchi changed title from ~~[PR #3045] [CLOSED] fix(evaluation): require explicit REA server identity for routing credit~~ to [PR #3045] fix(evaluation): require explicit REA server identity for routing credit

2026-11-03 13:30:02 +00:00

bsakaguchi changed title from ~~[PR #3045] fix(evaluation): require explicit REA server identity for routing credit~~ to [PR #3045] [CLOSED] fix(evaluation): require explicit REA server identity for routing credit

2026-11-03 14:31:48 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5578

No description provided.

Rows
Columns