morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #2146] [Feature] Help agents recover from common argument-name mistakes in procedure and search tools #703

New issue

Open

opened 2026-10-15 18:27:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:27:33 +00:00

Owner

Originally created by @pablodiazjorge on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2146

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Current limitation

Agents guess argument names from neighbouring tools, and two guesses fail often:

resolve_containing_procedure, xrefs and read_bytes take address, while procedure_pseudo_code, procedure_assembly and read_function_instructions take procedure ("The procedure name or address"). An agent that just got an address from one tool sends address to the next.
search_strings and search_procedures take pattern; agents send query.

The call is rejected before it reaches REA, for example:

Input validation error: Invalid arguments for tool search_strings: pattern: Invalid input: expected string, received undefined, Unrecognized key: "query"

Measured in a preregistered study: Claude Code 2.1.296 with claude-sonnet-5-5, REA 6.4.0 over MCP, Ghidra 12.1.4, 40 runs on small Windows PE crackmes, 321 REA calls. There were 18 input validation failures in 10 of the 40 runs:

Tool	Sent	Expected	Failures	Next call to the same tool succeeded
`procedure_pseudo_code`	`address`	`procedure`	13	9
`read_function_instructions`	`address`	`procedure`	1	0
`search_strings`	`query`	`pattern`	4	0

Each failure costs a turn. For search_strings, the agent never called the tool again in any of the four runs, so the string search it wanted was simply dropped. The contract is the same on main at fe4b94a66.

Use cases

The common static workflow moves from an address to its procedure: search_strings or xrefs gives an address, and the agent decompiles or disassembles the code there. That hand-off is where the address guess happens, in any MCP client.

Desired outcome and success criteria

A call that uses one of these known wrong keys either succeeds with the intended meaning or fails with an error that names the right key, for example: "procedure_pseudo_code takes procedure (a procedure name or address); address is not one of its inputs."
Evidence does not change: recorded parameters keep the canonical key, so Evidence identity does not depend on how the agent spelled the argument.
Genuinely unknown keys are still rejected.
Success: the cases above either succeed or come back with the hint. With aliases, 17 of the 18 calls become valid; the remaining read_function_instructions call also sent an unknown count, which should still be rejected.

Authority boundaries and non-goals

No change to what the tools do or return. No fuzzy matching of arbitrary keys. No renaming of existing parameters, which would break callers.

Alternatives considered

Description wording only: the procedure field already says "name or address", and agents still sent address.
Renaming procedure to address: breaking, and procedure also accepts symbol names.

Possible direction

I'd like to implement this. Either direction is additive:

Declared aliases for a short, fixed list (address for procedure on procedure-scoped tools that accept an address, query for pattern on the search tools), normalized before validation and recorded under the canonical key.
Keep strict input and add a remediation line to the validation error for those keys.

My preference is 1: the current error already names the expected key (pattern: Invalid input: expected string, received undefined), and the agent still recovered in only 9 of the 18 cases, none of the four search_strings ones, so a clearer message alone may not be enough. I'm opening a small PR for 1 right after this issue so there is something concrete to look at; if you prefer strict input, I'll switch it to 2.

References

Study writeup: https://github.com/pablodiazjorge/rea-lab/blob/main/writeups/2026-10-study-1/README.md
Per-run results (input validation failure counts in the rea field): https://github.com/pablodiazjorge/rea-lab/blob/main/bench/results/study-1/results.jsonl
Related: #1065, #723

Originally created by @pablodiazjorge on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2146 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Current limitation Agents guess argument names from neighbouring tools, and two guesses fail often: - `resolve_containing_procedure`, `xrefs` and `read_bytes` take `address`, while `procedure_pseudo_code`, `procedure_assembly` and `read_function_instructions` take `procedure` ("The procedure name or address"). An agent that just got an address from one tool sends `address` to the next. - `search_strings` and `search_procedures` take `pattern`; agents send `query`. The call is rejected before it reaches REA, for example: ```text Input validation error: Invalid arguments for tool search_strings: pattern: Invalid input: expected string, received undefined, Unrecognized key: "query" ``` Measured in a preregistered study: Claude Code 2.1.296 with `claude-sonnet-5-5`, REA 6.4.0 over MCP, Ghidra 12.1.4, 40 runs on small Windows PE crackmes, 321 REA calls. There were 18 input validation failures in 10 of the 40 runs: | Tool | Sent | Expected | Failures | Next call to the same tool succeeded | |---|---|---|---|---| | `procedure_pseudo_code` | `address` | `procedure` | 13 | 9 | | `read_function_instructions` | `address` | `procedure` | 1 | 0 | | `search_strings` | `query` | `pattern` | 4 | 0 | Each failure costs a turn. For `search_strings`, the agent never called the tool again in any of the four runs, so the string search it wanted was simply dropped. The contract is the same on `main` at `fe4b94a66`. ### Use cases The common static workflow moves from an address to its procedure: `search_strings` or `xrefs` gives an address, and the agent decompiles or disassembles the code there. That hand-off is where the `address` guess happens, in any MCP client. ### Desired outcome and success criteria - A call that uses one of these known wrong keys either succeeds with the intended meaning or fails with an error that names the right key, for example: "`procedure_pseudo_code` takes `procedure` (a procedure name or address); `address` is not one of its inputs." - Evidence does not change: recorded parameters keep the canonical key, so Evidence identity does not depend on how the agent spelled the argument. - Genuinely unknown keys are still rejected. - Success: the cases above either succeed or come back with the hint. With aliases, 17 of the 18 calls become valid; the remaining `read_function_instructions` call also sent an unknown `count`, which should still be rejected. ### Authority boundaries and non-goals No change to what the tools do or return. No fuzzy matching of arbitrary keys. No renaming of existing parameters, which would break callers. ### Alternatives considered - Description wording only: the `procedure` field already says "name or address", and agents still sent `address`. - Renaming `procedure` to `address`: breaking, and `procedure` also accepts symbol names. ### Possible direction I'd like to implement this. Either direction is additive: 1. Declared aliases for a short, fixed list (`address` for `procedure` on procedure-scoped tools that accept an address, `query` for `pattern` on the search tools), normalized before validation and recorded under the canonical key. 2. Keep strict input and add a remediation line to the validation error for those keys. My preference is 1: the current error already names the expected key (`pattern: Invalid input: expected string, received undefined`), and the agent still recovered in only 9 of the 18 cases, none of the four `search_strings` ones, so a clearer message alone may not be enough. I'm opening a small PR for 1 right after this issue so there is something concrete to look at; if you prefer strict input, I'll switch it to 2. ### References - Study writeup: https://github.com/pablodiazjorge/rea-lab/blob/main/writeups/2026-10-study-1/README.md - Per-run results (input validation failure counts in the `rea` field): https://github.com/pablodiazjorge/rea-lab/blob/main/bench/results/study-1/results.jsonl - Related: #1065, #723

bsakaguchi added the

area: domain

enhancement

P2: medium

labels

2026-10-15 18:27:33 +00:00

bsakaguchi commented

2026-10-15 18:27:34 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 12, 2026):

Agent Ergonomics & Schema Guidance Review

LLM agents frequently hallucinate synonym argument names (query instead of pattern, address instead of procedure):

Zod Schema Preprocessing vs Guidance: Rather than silently accepting arbitrary aliases (which can create schema ambiguity and mask client bugs), REA should provide high-signal validation error messages that guide recovery:
- If query is passed to search_strings, the error diagnostic should explicitly state: Unrecognized argument 'query'; did you mean 'pattern'?.
- Alternatively, declare optional deprecated aliases in the input schema that normalize to the canonical property inside the contract handler with an emitted warning.

@matbrgz commented on GitHub (Oct 12, 2026): ### Agent Ergonomics & Schema Guidance Review LLM agents frequently hallucinate synonym argument names (`query` instead of `pattern`, `address` instead of `procedure`): - **Zod Schema Preprocessing vs Guidance:** Rather than silently accepting arbitrary aliases (which can create schema ambiguity and mask client bugs), REA should provide high-signal validation error messages that guide recovery: - If `query` is passed to `search_strings`, the error diagnostic should explicitly state: `Unrecognized argument 'query'; did you mean 'pattern'?`. - Alternatively, declare optional deprecated aliases in the input schema that normalize to the canonical property inside the contract handler with an emitted warning.

bsakaguchi referenced this issue

2026-10-16 21:15:22 +00:00

[PR #703] [MERGED] fix(process): honor disabled PID normalization in samples #2184

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:28 +00:00