[GH-ISSUE #5431] [Feature] Make MCP tool failures self-recovering for agents #3294

New issue

Closed

opened 2026-10-31 00:38:52 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:38:52 +00:00

Owner

Originally created by @gensokyo-dev on GitHub (Oct 20, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5431

Problem

REA already has a shared, typed error projection for CLI and MCP tool-handler failures, but several failures that an agent encounters before or around the handler do not tell the agent how to recover. This makes a wrong call look like an invitation to retry blindly or to brute-force the catalog.

The gap is especially visible for timeouts and MCP schema/tool-selection mistakes:

On origin/main at aa314ab682e4bcd1e050a1e073bc90930660807f, a real stdio MCP client listed 210 tools. Calling inspect_wasm_artifact with {"path": 42} produced the SDK text-only error Input validation error: Invalid arguments for tool inspect_wasm_artifact: path: Invalid input: expected string, received number. It did not point to the advertised input schema, tools/list, or a non-repeating recovery step.
Calling inspect_analysis_view with malformed arguments did produce REA's canonical error object, but its remediation was only Correct the listed arguments and retry.; it did not tell the agent to re-read the tool schema or refresh the catalog if the schema was stale.
Calling an unknown tool produced the MCP JSON-RPC -32602 error Tool ... not found, with no direction to list the advertised tools or choose an available operation.
Timeout projections expose the elapsed REA deadline, but the generic recovery text does not distinguish the client deadline from the server/provider deadline or explain when an unchanged retry is unsafe.

The first case is source-derived: ErrorEvidenceTransport currently preserves SDK-generated validation errors as text-only results unless they exceed the response budget, so they bypass the canonical ToolResultDelivery projection. The other cases are observed through the same production stdio path.

Prior art and boundaries

RepoPrompt CE uses bounded structured fields such as error_code, retryable, retry_after_ms, and explicit next_step/next_call guidance in its agent-facing workflows. Its recovery guidance also distinguishes state that may have committed from safe-to-retry failures. Relevant references:

This is separate from #3221 (oversized error retention), #1720 (a Ghidra/client timeout incident), and #1587 (Codex skill onboarding). Those should remain intact; this issue is about a small shared recovery contract and clear messages at the REA MCP boundary.

Proposed outcome

Design and implement an additive, bounded recovery projection for agent-facing tool failures, reusing the existing canonical error contract and shared CLI/MCP delivery path:

Give schema/input failures a machine-readable recovery kind and concise message that names the failed operation, tells the agent to inspect that operation's advertised inputSchema (using tools/list when the catalog may be stale), and says not to repeat the unchanged call.
Give unknown-tool protocol failures a concise next step to refresh/list advertised tools while preserving MCP's protocol error shape.
Make timeout recovery state which REA/provider deadline elapsed, tell the caller how to distinguish a client-side deadline from a server/provider timeout, and advise a bounded smaller request or a health/session inspection before retrying when state may still be active.
Keep the response bounded: do not attach the complete catalog, raw caller arguments, credentials, or an unbounded provider diagnostic. Preserve Evidence references, partial observations, unknown coverage, target markers, and existing retryability semantics.
Add contract, transport-boundary, malformed-input, timeout, and response-budget regressions. Validate advertised JSON Schema after SDK conversion and exercise the real local stdio MCP client, including a raw call that bypasses client-side prevalidation.

Do not add automatic retries, mutate targets/providers, or make a new provider-specific tool. If the final design shows that an unknown-tool error cannot be safely enriched without changing the MCP protocol contract, document that limitation and keep the improvement to the canonical errors and SDK validation message.

Acceptance evidence

The canonical error schema remains valid for every CLI and MCP error consumer and is additive for existing clients.
A real stdio call makes the next recovery action discoverable without returning a full tool list or large context blob.
Before/after Codex CLI probes record selected tool, arguments, error, recovery action, retries, outcome, elapsed time, and peak process-family memory; report unmeasured client/model costs explicitly.
Focused tests and the relevant MCP/CLI acceptance lanes pass; no provider-specific behavior is claimed without the matching provider lane.

Originally created by @gensokyo-dev on GitHub (Oct 20, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5431 ## Problem REA already has a shared, typed error projection for CLI and MCP tool-handler failures, but several failures that an agent encounters before or around the handler do not tell the agent how to recover. This makes a wrong call look like an invitation to retry blindly or to brute-force the catalog. The gap is especially visible for timeouts and MCP schema/tool-selection mistakes: - On `origin/main` at `aa314ab682e4bcd1e050a1e073bc90930660807f`, a real stdio MCP client listed 210 tools. Calling `inspect_wasm_artifact` with `{"path": 42}` produced the SDK text-only error `Input validation error: Invalid arguments for tool inspect_wasm_artifact: path: Invalid input: expected string, received number`. It did not point to the advertised input schema, `tools/list`, or a non-repeating recovery step. - Calling `inspect_analysis_view` with malformed arguments did produce REA's canonical error object, but its remediation was only `Correct the listed arguments and retry.`; it did not tell the agent to re-read the tool schema or refresh the catalog if the schema was stale. - Calling an unknown tool produced the MCP JSON-RPC `-32602` error `Tool ... not found`, with no direction to list the advertised tools or choose an available operation. - Timeout projections expose the elapsed REA deadline, but the generic recovery text does not distinguish the client deadline from the server/provider deadline or explain when an unchanged retry is unsafe. The first case is source-derived: `ErrorEvidenceTransport` currently preserves SDK-generated validation errors as text-only results unless they exceed the response budget, so they bypass the canonical `ToolResultDelivery` projection. The other cases are observed through the same production stdio path. ## Prior art and boundaries RepoPrompt CE uses bounded structured fields such as `error_code`, `retryable`, `retry_after_ms`, and explicit `next_step`/`next_call` guidance in its agent-facing workflows. Its recovery guidance also distinguishes state that may have committed from safe-to-retry failures. Relevant references: - https://github.com/repoprompt/repoprompt-ce/blob/773a0f2e/Sources/RepoPromptShared/MCP/MCPHostRoutingValueContracts.swift - https://github.com/repoprompt/repoprompt-ce/blob/773a0f2e/Sources/RepoPrompt/Infrastructure/AI/Prompts/AgentModePrompts.swift - https://github.com/repoprompt/repoprompt-ce/blob/773a0f2e/Sources/RepoPromptShared/MCP/MCPTimeoutPolicy.swift This is separate from #3221 (oversized error retention), #1720 (a Ghidra/client timeout incident), and #1587 (Codex skill onboarding). Those should remain intact; this issue is about a small shared recovery contract and clear messages at the REA MCP boundary. ## Proposed outcome Design and implement an additive, bounded recovery projection for agent-facing tool failures, reusing the existing canonical error contract and shared CLI/MCP delivery path: 1. Give schema/input failures a machine-readable recovery kind and concise message that names the failed operation, tells the agent to inspect that operation's advertised `inputSchema` (using `tools/list` when the catalog may be stale), and says not to repeat the unchanged call. 2. Give unknown-tool protocol failures a concise next step to refresh/list advertised tools while preserving MCP's protocol error shape. 3. Make timeout recovery state which REA/provider deadline elapsed, tell the caller how to distinguish a client-side deadline from a server/provider timeout, and advise a bounded smaller request or a health/session inspection before retrying when state may still be active. 4. Keep the response bounded: do not attach the complete catalog, raw caller arguments, credentials, or an unbounded provider diagnostic. Preserve Evidence references, partial observations, unknown coverage, target markers, and existing retryability semantics. 5. Add contract, transport-boundary, malformed-input, timeout, and response-budget regressions. Validate advertised JSON Schema after SDK conversion and exercise the real local stdio MCP client, including a raw call that bypasses client-side prevalidation. Do not add automatic retries, mutate targets/providers, or make a new provider-specific tool. If the final design shows that an unknown-tool error cannot be safely enriched without changing the MCP protocol contract, document that limitation and keep the improvement to the canonical errors and SDK validation message. ## Acceptance evidence - The canonical error schema remains valid for every CLI and MCP error consumer and is additive for existing clients. - A real stdio call makes the next recovery action discoverable without returning a full tool list or large context blob. - Before/after Codex CLI probes record selected tool, arguments, error, recovery action, retries, outcome, elapsed time, and peak process-family memory; report unmeasured client/model costs explicitly. - Focused tests and the relevant MCP/CLI acceptance lanes pass; no provider-specific behavior is claimed without the matching provider lane.

bsakaguchi

2026-10-31 00:38:52 +00:00

closed this issue
added the
area: server

area: mcp

enhancement

P2: medium
labels

bsakaguchi commented

2026-10-31 00:38:53 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 20, 2026):

Post-merge validation update: the final compiled CI exposed four legacy remediation-text assertions in a CLI test introduced by the main update. Follow-up #5536 updates the consumer regression to preserve typed recovery, operation identity, discovery guidance and original diagnostics across JSON/TOON/YAML/Markdown. All five CLI tests and eleven MCP recovery tests pass locally; npm run check passes; independent Codex CLI review found no actionable defects. The five other observed compiled failures also reproduce in an isolated pre-change c374c2fe7 worktree and are intentionally not changed. #5470 now records the final CI limitations rather than claiming a green compiled suite.

@gensokyo-dev commented on GitHub (Oct 20, 2026): Post-merge validation update: the final compiled CI exposed four legacy remediation-text assertions in a CLI test introduced by the main update. Follow-up #5536 updates the consumer regression to preserve typed recovery, operation identity, discovery guidance and original diagnostics across JSON/TOON/YAML/Markdown. All five CLI tests and eleven MCP recovery tests pass locally; npm run check passes; independent Codex CLI review found no actionable defects. The five other observed compiled failures also reproduce in an isolated pre-change c374c2fe7 worktree and are intentionally not changed. #5470 now records the final CI limitations rather than claiming a green compiled suite.

bsakaguchi commented

2026-10-31 00:38:53 +00:00

Author

Owner

@gensokyo-dev commented on GitHub (Oct 20, 2026):

Completed: #5470 is merged as e50a271c7 and its CLI regression follow-up #5536 is merged as 55cce6fc1, using the owner-authorized override for reproduced unrelated baseline failures. No co-author trailers. Real Codex CLI probes recovered from wrong input and forced timeouts, including the installed Commander 14.0.3 package; the real-package task preserved 128 MiB analysis heap, completed in three MCP calls, and retained explicit partial/unknown coverage and Evidence references. Follow-up CI passed build, static checks and 12,301 source tests. Its three compiled failures reproduced exactly on clean cf5194f68 (39 passed, three failed, two skipped); cancelled peer coverage is explicitly not claimed as passing. Complete executed validation, agent token/time/memory costs and limitations are in the PR descriptions.

@gensokyo-dev commented on GitHub (Oct 20, 2026): Completed: #5470 is merged as e50a271c7 and its CLI regression follow-up #5536 is merged as 55cce6fc1, using the owner-authorized override for reproduced unrelated baseline failures. No co-author trailers. Real Codex CLI probes recovered from wrong input and forced timeouts, including the installed Commander 14.0.3 package; the real-package task preserved 128 MiB analysis heap, completed in three MCP calls, and retained explicit partial/unknown coverage and Evidence references. Follow-up CI passed build, static checks and 12,301 source tests. Its three compiled failures reproduced exactly on clean cf5194f68 (39 passed, three failed, two skipped); cancelled peer coverage is explicitly not claimed as passing. Complete executed validation, agent token/time/memory costs and limitations are in the PR descriptions.

bsakaguchi referenced this issue

2026-10-31 01:08:49 +00:00

[PR #3294] [MERGED] fix(apple): order projection paths by Unicode code point #5707

bsakaguchi referenced this issue

2026-10-31 01:08:51 +00:00

[PR #3302] [MERGED] ci: validate initial PR readiness declarations #5712