morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #5024] feat(experiments): add MCP tool-selection eval harness #6743

New issue

Open

opened 2026-11-01 21:42:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:42:44 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5024
Author: @jonjicjan
Created: 10/18/2026
Status: 🔄 Open

Base: main ← Head: feat/tool-selection-eval

📝 Commits (3)

86bbe92 feat(experiments): add MCP tool-selection eval harness
77f0a71 fix(tool-selection-eval): improve experiment reproducibility
e49a1ec fix(tool-selection-eval): address review comments on validation, paths, and resilience

📊 Changes

6 files changed (+4875 additions, -0 deletions)

View changed files

➕ experiments/tool-selection-eval/README.md (+67 -0)
➕ experiments/tool-selection-eval/fixtures.example.json (+98 -0)
➕ experiments/tool-selection-eval/package-lock.json (+4011 -0)
➕ experiments/tool-selection-eval/package.json (+24 -0)
➕ experiments/tool-selection-eval/run.ts (+662 -0)
➕ experiments/tool-selection-eval/tsconfig.json (+13 -0)

📄 Description

Summary

Adds experiments/tool-selection-eval/, a standalone harness for evaluating which REA MCP tool an agent selects first for a natural-language task. It includes an audit mode that reports catalog size, schema and description statistics, and similar-description candidates.

The harness does not execute the selected tools.

Problem and Expected Behavior

REA exposes a large MCP tool catalog, but there is no dedicated experiment for measuring whether an agent selects the expected tool for a given task or for inspecting the size and similarity of tool definitions.

This contribution provides a repeatable evaluation workflow using starter fixtures, a TypeScript runner, and an audit mode. Expected tool names are validated against the live catalog.

The audit can help identify large schemas and potentially confusing tool descriptions without modifying existing tool contracts or schemas.

Changes

Adds the following files under experiments/tool-selection-eval/:

run.ts — catalog audit and model-based evaluation runner.
fixtures.example.json — 16 starter evaluation fixtures.
README.md — setup, usage, and limitations.
package.json — experiment dependencies and configuration.
package-lock.json — dependency lockfile.
tsconfig.json — TypeScript configuration.

The changes are isolated to the experiment directory. No changes are intended to TOOL_CONTRACTS, tool descriptions, schemas, result envelopes, or application code outside this folder.

Contract and Boundary Impact

Scope: Standalone experiment; no intended changes to existing tool contracts.
MCP interaction: Uses tools/list to retrieve tool definitions. It does not invoke the listed REA tools.
Target interaction: No target interaction or provider-tool execution is intended.
Network access: Model-based evaluation sends tool definitions and task prompts to the Anthropic API.
Configuration: The runner reads ANTHROPIC_API_KEY from the environment and uses REA_MCP_CMD to configure the local MCP server command.
Release impact: No application build outputs or release metadata changes are intended.

Evidence and Regression Coverage

Tests added or updated: None.
Audit executed: npx tsx .\experiments\tool-selection-eval\run.ts --audit completed successfully.
Catalog size: 207 tools, 759,906 definition characters, and approximately 189,977 estimated tokens using the runner's characters-divided-by-four heuristic.
Largest schema: compare_web_captures, with 165,694 schema characters and 885 description characters.
Description similarity: procedure_callees and procedure_callers had a reported Jaccard similarity of approximately 0.553. These are candidates for review, not proof of redundant tools.
Remaining proof gaps: Model-based evaluation has not been run against an Anthropic model. Evaluation scoring and reporting remain unverified.

Validation Performed

npm run check:fast — passed; typecheck and lint succeeded.
npx turbo run format:check — passed.
npx turbo run knip — passed; compilation emitted INEFFECTIVE_DYNAMIC_IMPORT warnings.
npx tsx .\experiments\tool-selection-eval\run.ts --audit — completed successfully and produced the catalog statistics above.
git diff main...HEAD --check — passed with no whitespace errors.
npm run check — exited with code 1 on Windows because packaged native artifacts were missing, including native/windows/build/manifest.json. The same failure was reproduced on main and the feature branch.

The full check is not green. The native-artifact failure was reproduced on both branches.

Compatibility, Safety, and Release

Breaking changes or migrations: None intended.
Provider execution: No REA provider tools are executed by the evaluation harness.
Evaluation behavior: Measures first-tool selection only. Subset-catalog results should not be interpreted as full-catalog accuracy.
Limitations: Model-based evaluation and its scoring/reporting behavior have not yet been verified end to end.

Author Readiness

Human owner: @jonjicjan

I have reviewed the complete diff.
I have verified the experiment's documented commands and expected behavior.
I have reviewed the validation results and remaining proof gaps.
I have confirmed any required maintainer scope approval.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5024 **Author:** [@jonjicjan](https://github.com/jonjicjan) **Created:** 10/18/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/tool-selection-eval` --- ### 📝 Commits (3) - [`86bbe92`](https://github.com/morluto/rea/commit/86bbe92ff0dbdf8417e8b0941633b86f3456d8e8) feat(experiments): add MCP tool-selection eval harness - [`77f0a71`](https://github.com/morluto/rea/commit/77f0a716e4916858080e7c3827e31cad664b70ec) fix(tool-selection-eval): improve experiment reproducibility - [`e49a1ec`](https://github.com/morluto/rea/commit/e49a1ec5d89a647bcff0854ec1312851687a255a) fix(tool-selection-eval): address review comments on validation, paths, and resilience ### 📊 Changes **6 files changed** (+4875 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `experiments/tool-selection-eval/README.md` (+67 -0) ➕ `experiments/tool-selection-eval/fixtures.example.json` (+98 -0) ➕ `experiments/tool-selection-eval/package-lock.json` (+4011 -0) ➕ `experiments/tool-selection-eval/package.json` (+24 -0) ➕ `experiments/tool-selection-eval/run.ts` (+662 -0) ➕ `experiments/tool-selection-eval/tsconfig.json` (+13 -0) </details> ### 📄 Description ## Summary Adds `experiments/tool-selection-eval/`, a standalone harness for evaluating which REA MCP tool an agent selects first for a natural-language task. It includes an audit mode that reports catalog size, schema and description statistics, and similar-description candidates. The harness does not execute the selected tools. ## Problem and Expected Behavior REA exposes a large MCP tool catalog, but there is no dedicated experiment for measuring whether an agent selects the expected tool for a given task or for inspecting the size and similarity of tool definitions. This contribution provides a repeatable evaluation workflow using starter fixtures, a TypeScript runner, and an audit mode. Expected tool names are validated against the live catalog. The audit can help identify large schemas and potentially confusing tool descriptions without modifying existing tool contracts or schemas. ## Changes Adds the following files under `experiments/tool-selection-eval/`: - `run.ts` — catalog audit and model-based evaluation runner. - `fixtures.example.json` — 16 starter evaluation fixtures. - `README.md` — setup, usage, and limitations. - `package.json` — experiment dependencies and configuration. - `package-lock.json` — dependency lockfile. - `tsconfig.json` — TypeScript configuration. The changes are isolated to the experiment directory. No changes are intended to `TOOL_CONTRACTS`, tool descriptions, schemas, result envelopes, or application code outside this folder. ## Contract and Boundary Impact - **Scope:** Standalone experiment; no intended changes to existing tool contracts. - **MCP interaction:** Uses `tools/list` to retrieve tool definitions. It does not invoke the listed REA tools. - **Target interaction:** No target interaction or provider-tool execution is intended. - **Network access:** Model-based evaluation sends tool definitions and task prompts to the Anthropic API. - **Configuration:** The runner reads `ANTHROPIC_API_KEY` from the environment and uses `REA_MCP_CMD` to configure the local MCP server command. - **Release impact:** No application build outputs or release metadata changes are intended. ## Evidence and Regression Coverage - **Tests added or updated:** None. - **Audit executed:** `npx tsx .\experiments\tool-selection-eval\run.ts --audit` completed successfully. - **Catalog size:** 207 tools, 759,906 definition characters, and approximately 189,977 estimated tokens using the runner's characters-divided-by-four heuristic. - **Largest schema:** `compare_web_captures`, with 165,694 schema characters and 885 description characters. - **Description similarity:** `procedure_callees` and `procedure_callers` had a reported Jaccard similarity of approximately 0.553. These are candidates for review, not proof of redundant tools. - **Remaining proof gaps:** Model-based evaluation has not been run against an Anthropic model. Evaluation scoring and reporting remain unverified. ## Validation Performed - `npm run check:fast` — passed; typecheck and lint succeeded. - `npx turbo run format:check` — passed. - `npx turbo run knip` — passed; compilation emitted `INEFFECTIVE_DYNAMIC_IMPORT` warnings. - `npx tsx .\experiments\tool-selection-eval\run.ts --audit` — completed successfully and produced the catalog statistics above. - `git diff main...HEAD --check` — passed with no whitespace errors. - `npm run check` — exited with code 1 on Windows because packaged native artifacts were missing, including `native/windows/build/manifest.json`. The same failure was reproduced on `main` and the feature branch. The full check is not green. The native-artifact failure was reproduced on both branches. ## Compatibility, Safety, and Release - **Breaking changes or migrations:** None intended. - **Provider execution:** No REA provider tools are executed by the evaluation harness. - **Evaluation behavior:** Measures first-tool selection only. Subset-catalog results should not be interpreted as full-catalog accuracy. - **Limitations:** Model-based evaluation and its scoring/reporting behavior have not yet been verified end to end. ## Author Readiness Human owner: @jonjicjan - [ ] I have reviewed the complete diff. - [ ] I have verified the experiment's documented commands and expected behavior. - [ ] I have reviewed the validation results and remaining proof gaps. - [ ] I have confirmed any required maintainer scope approval. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-11-01 21:42:44 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6743

No description provided.

Rows
Columns