morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #3246] [Feature] Scope advertised MCP tools by family through REA_MCP_TOOL_FAMILIES #1164

New issue

Closed

opened 2026-10-15 18:31:39 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-15 18:31:39 +00:00

Owner

Originally created by @0xLLVEL on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3246

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

MCP or agent integration

Current limitation

REA advertises its complete MCP tool catalog (200+ tools) to every client. Agents pay context for each advertised tool schema, and clients that resend catalog payloads per request can spend more on the advertisement than on the investigation itself. For local models with small context windows this is a hard wall: the full catalog does not fit, so REA is unusable with them. There is currently no supported way to narrow the advertised tool set to the families an investigation actually needs.

Use cases

Run a local model with a 32K context window against REA by advertising only the needed tool families (e.g. session,direct).
Persist a scoped selection per registration via rea setup --tool-families <ids> so every session starts lean without client-side filtering.
Keep the default behavior unchanged: with no selection, the complete catalog is advertised exactly as before.

Desired outcome and success criteria

REA_MCP_TOOL_FAMILIES scopes the advertised MCP tool set by family, settable via environment or rea setup --tool-families <ids> (persisted into the registration entry). Out-of-scope tools stay registered but disabled: a direct call fails with the SDK's Tool ... disabled invalid-params error instead of vanishing silently. A selection outside the published family ids fails server startup with the complete id list, and doctor validates the selection against the same ids. The family grouping lives in one source of truth (src/contracts/toolFamilies.ts) shared by runtime advertising and the product catalog, and a workflow-to-family recommendation table documents which families each investigation type needs. A prototype is available in #3026.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Scoping only changes which tool schemas are advertised over MCP and which calls are accepted; target analysis, providers, and evidence behavior are unchanged. No new network, execution, or protection-recovery capability is introduced.

Verification available so far

Someone else verified this with evidence

Verification details (optional)

Independent end-to-end benchmark by @CassieFox23 in discussion #3098: a local Qwen3-27B (IQ3_XXS, 32K context, llama.cpp on a single 12 GB AMD card) drove REA over MCP against a small C target at PR #3026 (head d993ffeb1):

Setup	Tools	Prompt tokens	Result
Full tool list	206	248,762	Refused at once: too big for 32K
`session,direct` + compact	62	28,628	12.7 min to read the prompt, then a GPU reset killed the run
Same, plus a 10-tool client allowlist	10	4,407	Right answer, 3 of 3 runs (~80 s each)

Per-tool token accounting (model's own tokenizer, name + description + input schema only): full catalog 193,578 tokens (79,788 compact), session only 26,972 (15,489 compact), session,native compact 21,313. open_binary lives in session, so any file-opening setup starts near 15K tokens even with compact schemas.

Prototype-side checks in PR #3026: metadata:check passes (family inventory covers every TOOL_CONTRACTS entry), typecheck/lint/oxfmt --check clean, catalog regeneration byte-identical, and unit/conformance/boundary/setup tests (toolFamilies, mcpToolFamilies, toolFamiliesMcp, Setup.configuration) pass.

References (optional)

PR #3026 (prototype implementation)
Discussion #3098 (independent local-model benchmark and token accounting)

Originally created by @0xLLVEL on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3246 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area MCP or agent integration ### Current limitation REA advertises its complete MCP tool catalog (200+ tools) to every client. Agents pay context for each advertised tool schema, and clients that resend catalog payloads per request can spend more on the advertisement than on the investigation itself. For local models with small context windows this is a hard wall: the full catalog does not fit, so REA is unusable with them. There is currently no supported way to narrow the advertised tool set to the families an investigation actually needs. ### Use cases - Run a local model with a 32K context window against REA by advertising only the needed tool families (e.g. `session,direct`). - Persist a scoped selection per registration via `rea setup --tool-families <ids>` so every session starts lean without client-side filtering. - Keep the default behavior unchanged: with no selection, the complete catalog is advertised exactly as before. ### Desired outcome and success criteria `REA_MCP_TOOL_FAMILIES` scopes the advertised MCP tool set by family, settable via environment or `rea setup --tool-families <ids>` (persisted into the registration entry). Out-of-scope tools stay registered but disabled: a direct call fails with the SDK's `Tool ... disabled` invalid-params error instead of vanishing silently. A selection outside the published family ids fails server startup with the complete id list, and doctor validates the selection against the same ids. The family grouping lives in one source of truth (`src/contracts/toolFamilies.ts`) shared by runtime advertising and the product catalog, and a workflow-to-family recommendation table documents which families each investigation type needs. A prototype is available in #3026. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) Scoping only changes which tool schemas are advertised over MCP and which calls are accepted; target analysis, providers, and evidence behavior are unchanged. No new network, execution, or protection-recovery capability is introduced. ### Verification available so far Someone else verified this with evidence ### Verification details (optional) Independent end-to-end benchmark by @CassieFox23 in discussion #3098: a local Qwen3-27B (IQ3_XXS, 32K context, llama.cpp on a single 12 GB AMD card) drove REA over MCP against a small C target at PR #3026 (head `d993ffeb1`): | Setup | Tools | Prompt tokens | Result | |---|--:|--:|---| | Full tool list | 206 | 248,762 | Refused at once: too big for 32K | | `session,direct` + compact | 62 | 28,628 | 12.7 min to read the prompt, then a GPU reset killed the run | | Same, plus a 10-tool client allowlist | 10 | 4,407 | Right answer, 3 of 3 runs (~80 s each) | Per-tool token accounting (model's own tokenizer, name + description + input schema only): full catalog 193,578 tokens (79,788 compact), `session` only 26,972 (15,489 compact), `session,native` compact 21,313. `open_binary` lives in `session`, so any file-opening setup starts near 15K tokens even with compact schemas. Prototype-side checks in PR #3026: `metadata:check` passes (family inventory covers every `TOOL_CONTRACTS` entry), typecheck/lint/`oxfmt --check` clean, catalog regeneration byte-identical, and unit/conformance/boundary/setup tests (`toolFamilies`, `mcpToolFamilies`, `toolFamiliesMcp`, `Setup.configuration`) pass. ### References (optional) - PR #3026 (prototype implementation) - Discussion #3098 (independent local-model benchmark and token accounting)

bsakaguchi added the

enhancement

label

2026-10-15 18:31:39 +00:00

bsakaguchi commented

2026-10-16 21:02:42 +00:00

Author

Owner

@pablodiazjorge commented on GitHub (Oct 16, 2026):

A data point on what the catalog costs with one client that defers schemas. In rea-lab study 3 (rea 6.4.0 advertising 153 tools, Claude Code 2.1.296, 72 runs with REA and 72 without), Claude Code listed the REA tools by name only and loaded schemas on demand through its tool search. Every run with REA used it, a median of once per run.

The first request of a run carried a median of 12,187 input tokens with REA against 8,970 without, so the tool names plus the server's instructions cost about 3,200 tokens per request.
That overhead repeats on every model request. Multiplied by each run's requests, it is a median of about 12% of the run's input tokens (3 to 19%), counting input, cache creation and cache reads. The rest of the gap between the conditions (median 300,695 input tokens with REA against 112,967 without) is what the investigation itself put into the context and carried from request to request; #4249 has the breakdown of the oversized results.

So with a deferring client the catalog is a modest, fixed cost. For clients that send every schema up front, like the 32K local case in this issue, scoping is what makes REA fit at all. I did not measure a client of that kind.

@pablodiazjorge commented on GitHub (Oct 16, 2026): A data point on what the catalog costs with one client that defers schemas. In [rea-lab study 3](https://github.com/pablodiazjorge/rea-lab/tree/main/writeups/2026-10-study-3) (rea 6.4.0 advertising 153 tools, Claude Code 2.1.296, 72 runs with REA and 72 without), Claude Code listed the REA tools by name only and loaded schemas on demand through its tool search. Every run with REA used it, a median of once per run. - The first request of a run carried a median of 12,187 input tokens with REA against 8,970 without, so the tool names plus the server's instructions cost about 3,200 tokens per request. - That overhead repeats on every model request. Multiplied by each run's requests, it is a median of about 12% of the run's input tokens (3 to 19%), counting input, cache creation and cache reads. The rest of the gap between the conditions (median 300,695 input tokens with REA against 112,967 without) is what the investigation itself put into the context and carried from request to request; #4249 has the breakdown of the oversized results. So with a deferring client the catalog is a modest, fixed cost. For clients that send every schema up front, like the 32K local case in this issue, scoping is what makes REA fit at all. I did not measure a client of that kind.

bsakaguchi commented

2026-10-16 21:02:43 +00:00

Author

Owner

@pablodiazjorge commented on GitHub (Oct 16, 2026):

A follow-up with the case I said I had not measured. Pointing the same Claude Code 2.1.296 at a non-Anthropic endpoint (DeepSeek's Anthropic-compatible API, for rea-lab's next study) turns tool search off; the client logs [ToolSearch:optimistic] disabled: ANTHROPIC_BASE_URL=… is not a first-party Anthropic host. It then sends every REA schema with each request:

Trivial run (reply OK, no tools), rea 6.4.0	Tools advertised	Input tokens of the first request
Without REA	5	5,745
With REA, tool search off	158	127,161
With REA, `ENABLE_TOOL_SEARCH=true`	159	7,903

The counts are DeepSeek's tokens. So the 153 REA tools cost about 121,000 tokens per request when they are all sent, roughly 790 per tool, against about 2,200 when deferred. That is the per-request wall a 32K local model hits. Going by that log message, it also applies to anyone running Claude Code through a gateway or router with a non-Anthropic base URL: unless they set ENABLE_TOOL_SEARCH=true and the backend supports it, each request carries the whole catalog. Family scoping would help there, and documenting ENABLE_TOOL_SEARCH for proxied setups may be a cheap first step.

@pablodiazjorge commented on GitHub (Oct 16, 2026): A follow-up with the case I said I had not measured. Pointing the same Claude Code 2.1.296 at a non-Anthropic endpoint (DeepSeek's Anthropic-compatible API, for [rea-lab](https://github.com/pablodiazjorge/rea-lab)'s next study) turns tool search off; the client logs `[ToolSearch:optimistic] disabled: ANTHROPIC_BASE_URL=… is not a first-party Anthropic host`. It then sends every REA schema with each request: | Trivial run (reply OK, no tools), rea 6.4.0 | Tools advertised | Input tokens of the first request | |---|---|---| | Without REA | 5 | 5,745 | | With REA, tool search off | 158 | 127,161 | | With REA, `ENABLE_TOOL_SEARCH=true` | 159 | 7,903 | The counts are DeepSeek's tokens. So the 153 REA tools cost about 121,000 tokens per request when they are all sent, roughly 790 per tool, against about 2,200 when deferred. That is the per-request wall a 32K local model hits. Going by that log message, it also applies to anyone running Claude Code through a gateway or router with a non-Anthropic base URL: unless they set `ENABLE_TOOL_SEARCH=true` and the backend supports it, each request carries the whole catalog. Family scoping would help there, and documenting `ENABLE_TOOL_SEARCH` for proxied setups may be a cheap first step.

bsakaguchi commented

2026-10-16 21:02:43 +00:00

Author

Owner

@hj01857655 commented on GitHub (Oct 16, 2026):

Adding a confirming data point for the proxied / non-Anthropic-backend case, on rea-agents@6.5.0.

Setup: Claude Code → a 2api bridge (Anthropic-format → OpenAI-format) → Qwen3.8-Flash. Tool search is off because the backend is not a first-party Anthropic host, so every REA schema is sent on each request.

Single-request measurement:

prompt_tokens: 314267 (context full)
top-level tools array: ~430KB+ (the request dump was truncated at 512KB mid-schema, so the real array is larger)
34 mcp__rea__* tools observed in the truncated portion (consistent with the 153–158 advertised in 6.4.0 once the rest of the array is counted)

Knock-on effect in a long RE session: the host agent hit overflow_compacted up to (3/3) — the request would not fit even after three compaction passes, because the tools array is re-sent every turn and cannot be compacted away. The only workaround was to detach the REA MCP server entirely, which defeats having it attached.

On ENABLE_TOOL_SEARCH=true for this case: it does not help here. The backend is an OpenAI-format model (Qwen) behind a format-translating proxy; Anthropic tool search is not something the backend can execute, so the client sending tool-search-style requests has nothing to talk to. For proxied / non-Anthropic setups, family scoping (REA_MCP_TOOL_FAMILIES) is the only structural fix.

Implementation status check: grepping the repo for REA_MCP_TOOL_FAMILIES / TOOL_FAMILIES returns 0 hits as of 6.5.0, so the scoping mechanism proposed here is not yet in the codebase. Would be great to see it prioritized — the proxy/gateway case has no other escape hatch.

@hj01857655 commented on GitHub (Oct 16, 2026): Adding a confirming data point for the proxied / non-Anthropic-backend case, on `rea-agents@6.5.0`. **Setup:** Claude Code → a 2api bridge (Anthropic-format → OpenAI-format) → `Qwen3.8-Flash`. Tool search is off because the backend is not a first-party Anthropic host, so every REA schema is sent on each request. **Single-request measurement:** - `prompt_tokens: 314267` (context full) - top-level `tools` array: **~430KB+** (the request dump was truncated at 512KB mid-schema, so the real array is larger) - 34 `mcp__rea__*` tools observed in the truncated portion (consistent with the 153–158 advertised in 6.4.0 once the rest of the array is counted) **Knock-on effect in a long RE session:** the host agent hit `overflow_compacted` up to **(3/3)** — the request would not fit even after three compaction passes, because the `tools` array is re-sent every turn and **cannot be compacted away**. The only workaround was to detach the REA MCP server entirely, which defeats having it attached. **On `ENABLE_TOOL_SEARCH=true` for this case:** it does not help here. The backend is an OpenAI-format model (Qwen) behind a format-translating proxy; Anthropic tool search is not something the backend can execute, so the client sending tool-search-style requests has nothing to talk to. For proxied / non-Anthropic setups, family scoping (`REA_MCP_TOOL_FAMILIES`) is the only structural fix. **Implementation status check:** grepping the repo for `REA_MCP_TOOL_FAMILIES` / `TOOL_FAMILIES` returns 0 hits as of 6.5.0, so the scoping mechanism proposed here is not yet in the codebase. Would be great to see it prioritized — the proxy/gateway case has no other escape hatch.

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-17 21:01:37 +00:00

bsakaguchi referenced this issue

2026-10-17 21:09:14 +00:00

[PR #1177] [MERGED] fix(setup): stop uninstall before removals when a client config is unsafe #2700

bsakaguchi closed this issue

2026-10-31 00:29:58 +00:00