morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #261] Make trace_feature operation-budget limits discoverable and errors actionable #152

New issue

Closed

opened 2026-10-15 18:21:42 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-15 18:21:42 +00:00

Owner

Originally created by @morluto on GitHub (Jul 16, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/261

Originally assigned to: @N0zoM1z0 on GitHub.

Problem

trace_feature rejects max_operations values above 100, but the effective
ceiling is not visible on at least one real MCP host tool surface, and the
validation failure does not provide a structured repair path.

Observed with REA 1.7.0 through Codex MCP:

rea.trace_feature({
  "query": "Finding useful first work",
  "case_sensitive": true,
  "limit": 50,
  "max_operations": 300
})
Input validation error: Invalid arguments for tool trace_feature:
max_operations: Too big: expected number to be <=100

The same failure then repeated for three independent queries:

signup-draft.
codex-home
config.toml

This was not a query-specific failure. The invalid budget was part of the
analysis strategy, so every call in the batch failed before producing evidence.

Discovery-surface evidence

Codex rendered the REA tool declaration as:

declare const tools: { mcp__rea__trace_feature(args: {
  case_sensitive?: boolean;
  limit?: number;
  max_operations?: number;
  query: string;
  unknown_registry_approved?: true;
}): Promise<...>; };

The human-facing tool description explained that tracing is bounded, but did
not state the accepted range or default. The rendered TypeScript declaration
showed only number; it did not expose a maximum: 100 constraint.

REA may already publish maximum: 100 in its raw JSON Schema. If so, part of
the loss is likely in the MCP SDK/host schema-to-tool rendering boundary. REA
still needs a host-robust description and actionable runtime error because it
cannot assume every client will display numeric JSON Schema constraints.

Analysis context

The target was the shipped a shipped minified JavaScript CLI JavaScript bundle. REA could
open it as a JavaScript artifact, but binary_overview correctly reported that
Hopper and Ghidra do not analyze JavaScript artifact targets. The caller then
tried trace_feature for bounded literal recovery of signup behavior. The
budget error prevented all four traces before target/provider capability could
be evaluated.

This context matters because callers naturally raise operation budgets after a
deep-provider route is unavailable. The valid continuation route needs to be
discoverable without trial-and-error.

Why this matters

A bounded analysis request fails before producing any Evidence v2 record.
Batched or repeated calls reproduce the same avoidable failure.
Agents cannot infer the ceiling from max_operations?: number.
Generic prose requires error-string parsing rather than typed recovery.
Callers do not know whether to retry at 100, paginate, narrow the query, or
switch tools.
A correct server-side constraint is insufficient when clients omit that
constraint from their rendered tool signature.

Requested behavior

1. Make the range explicit at discovery time

Keep the JSON Schema constraint and add the range to the human-facing argument
or tool description, for example:

max_operations: maximum trace operations; integer 1-100; default 50

Use the real default in place of the example if it differs. This redundant
description is intentional because host renderers may discard numeric schema
facets.

2. Return a typed, actionable validation error

Return structured error content such as:

{
  "code": "argument_out_of_range",
  "category": "invalid_input",
  "retryable": true,
  "details": {
    "argument": "max_operations",
    "received": 300,
    "minimum": 1,
    "maximum": 100
  },
  "remediation": {
    "action": "Retry with max_operations <= 100. If the trace is truncated, narrow the query or follow the returned evidence with bounded tools."
  }
}

The MCP call should remain an error and must not emit analysis evidence for an
operation that never ran.

3. Preserve batch-level repairability

When multiple calls fail for the same argument constraint, each response must
carry the same machine-readable bounds. Hosts can then repair the shared call
template once rather than interpreting four unrelated strings.

If MCP Tasks or another batch abstraction is later used, argument validation
should happen before task creation and return the same typed contract.

4. Document the supported continuation workflow

Explain what to do when 100 operations are insufficient:

Retry at the maximum supported value.
Inspect truncated, matches, references, and residual unknowns.
Narrow the literal query or use a more specific follow-up operation.
Continue from returned evidence rather than pretending independent traces
form one exhaustive trace.
For JavaScript artifacts, identify which source/artifact analysis tools are
supported instead of directing callers toward native-only providers.

5. Optional explicit clamping

An explicit mode such as budget_policy: "reject" | "clamp" could permit
ergonomic clamping. The default should remain reject. If clamping is added,
Evidence v2 must preserve both requested and effective values in parameters or
limitations. Silent clamping is not requested.

Acceptance criteria

The raw MCP input schema declares the integer minimum and maximum for
max_operations.
The tool or argument description states its accepted range and default,
even when a host omits numeric JSON Schema facets.
Values above 100 return a typed error containing argument name, received
value, minimum, maximum, retryability, and exact remediation.
No Evidence v2 record or task is created for validation failures.
Repeated/batched failures expose identical structured constraint data.
Documentation explains the supported continuation path at the ceiling.
If explicit clamping is implemented, requested/effective budgets are
preserved and silent clamping remains impossible.
Tests cover omitted/default behavior, the minimum, 100, 101, zero,
negative values, non-integers, and repeated calls sharing an invalid
template.
The same schema-description-error contract is audited across other
bounded REA arguments (limit, operation budgets, capture limits, and
comparison page sizes).
A compatibility test inspects the actual MCP tools/list schema, not
only an internal validator schema.

Environment

REA CLI/package: 1.7.0
REA catalog: 78 MCP tools
MCP SDK packages reported by rea doctor: 2.0.0-beta.4
Host: Codex on macOS arm64
Target: a shipped minified JavaScript CLI JavaScript artifact

Originally created by @morluto on GitHub (Jul 16, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/261 Originally assigned to: @N0zoM1z0 on GitHub. ## Problem `trace_feature` rejects `max_operations` values above 100, but the effective ceiling is not visible on at least one real MCP host tool surface, and the validation failure does not provide a structured repair path. Observed with REA 1.7.0 through Codex MCP: ```text rea.trace_feature({ "query": "Finding useful first work", "case_sensitive": true, "limit": 50, "max_operations": 300 }) Input validation error: Invalid arguments for tool trace_feature: max_operations: Too big: expected number to be <=100 ``` The same failure then repeated for three independent queries: - `signup-draft.` - `codex-home` - `config.toml` This was not a query-specific failure. The invalid budget was part of the analysis strategy, so every call in the batch failed before producing evidence. ## Discovery-surface evidence Codex rendered the REA tool declaration as: ```ts declare const tools: { mcp__rea__trace_feature(args: { case_sensitive?: boolean; limit?: number; max_operations?: number; query: string; unknown_registry_approved?: true; }): Promise<...>; }; ``` The human-facing tool description explained that tracing is bounded, but did not state the accepted range or default. The rendered TypeScript declaration showed only `number`; it did not expose a `maximum: 100` constraint. REA may already publish `maximum: 100` in its raw JSON Schema. If so, part of the loss is likely in the MCP SDK/host schema-to-tool rendering boundary. REA still needs a host-robust description and actionable runtime error because it cannot assume every client will display numeric JSON Schema constraints. ## Analysis context The target was the shipped `a shipped minified JavaScript CLI` JavaScript bundle. REA could open it as a JavaScript artifact, but `binary_overview` correctly reported that Hopper and Ghidra do not analyze JavaScript artifact targets. The caller then tried `trace_feature` for bounded literal recovery of signup behavior. The budget error prevented all four traces before target/provider capability could be evaluated. This context matters because callers naturally raise operation budgets after a deep-provider route is unavailable. The valid continuation route needs to be discoverable without trial-and-error. ## Why this matters - A bounded analysis request fails before producing any Evidence v2 record. - Batched or repeated calls reproduce the same avoidable failure. - Agents cannot infer the ceiling from `max_operations?: number`. - Generic prose requires error-string parsing rather than typed recovery. - Callers do not know whether to retry at 100, paginate, narrow the query, or switch tools. - A correct server-side constraint is insufficient when clients omit that constraint from their rendered tool signature. ## Requested behavior ### 1. Make the range explicit at discovery time Keep the JSON Schema constraint and add the range to the human-facing argument or tool description, for example: ```text max_operations: maximum trace operations; integer 1-100; default 50 ``` Use the real default in place of the example if it differs. This redundant description is intentional because host renderers may discard numeric schema facets. ### 2. Return a typed, actionable validation error Return structured error content such as: ```json { "code": "argument_out_of_range", "category": "invalid_input", "retryable": true, "details": { "argument": "max_operations", "received": 300, "minimum": 1, "maximum": 100 }, "remediation": { "action": "Retry with max_operations <= 100. If the trace is truncated, narrow the query or follow the returned evidence with bounded tools." } } ``` The MCP call should remain an error and must not emit analysis evidence for an operation that never ran. ### 3. Preserve batch-level repairability When multiple calls fail for the same argument constraint, each response must carry the same machine-readable bounds. Hosts can then repair the shared call template once rather than interpreting four unrelated strings. If MCP Tasks or another batch abstraction is later used, argument validation should happen before task creation and return the same typed contract. ### 4. Document the supported continuation workflow Explain what to do when 100 operations are insufficient: 1. Retry at the maximum supported value. 2. Inspect `truncated`, matches, references, and residual unknowns. 3. Narrow the literal query or use a more specific follow-up operation. 4. Continue from returned evidence rather than pretending independent traces form one exhaustive trace. 5. For JavaScript artifacts, identify which source/artifact analysis tools are supported instead of directing callers toward native-only providers. ### 5. Optional explicit clamping An explicit mode such as `budget_policy: "reject" | "clamp"` could permit ergonomic clamping. The default should remain `reject`. If clamping is added, Evidence v2 must preserve both requested and effective values in parameters or limitations. Silent clamping is not requested. ## Acceptance criteria - [ ] The raw MCP input schema declares the integer minimum and maximum for `max_operations`. - [ ] The tool or argument description states its accepted range and default, even when a host omits numeric JSON Schema facets. - [ ] Values above 100 return a typed error containing argument name, received value, minimum, maximum, retryability, and exact remediation. - [ ] No Evidence v2 record or task is created for validation failures. - [ ] Repeated/batched failures expose identical structured constraint data. - [ ] Documentation explains the supported continuation path at the ceiling. - [ ] If explicit clamping is implemented, requested/effective budgets are preserved and silent clamping remains impossible. - [ ] Tests cover omitted/default behavior, the minimum, 100, 101, zero, negative values, non-integers, and repeated calls sharing an invalid template. - [ ] The same schema-description-error contract is audited across other bounded REA arguments (`limit`, operation budgets, capture limits, and comparison page sizes). - [ ] A compatibility test inspects the actual MCP `tools/list` schema, not only an internal validator schema. ## Environment - REA CLI/package: 1.7.0 - REA catalog: 78 MCP tools - MCP SDK packages reported by `rea doctor`: 2.0.0-beta.4 - Host: Codex on macOS arm64 - Target: `a shipped minified JavaScript CLI` JavaScript artifact

bsakaguchi

2026-10-15 18:21:42 +00:00

closed this issue
added the
area: mcp

enhancement
labels

bsakaguchi commented

2026-10-15 18:21:44 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

Confirmed root cause and scope

I reproduced this against the published rea-agents@1.7.0 MCP server and traced the registration path.

Discovery boundary

The live tools/list response is correct. trace_feature.inputSchema.properties.max_operations contains:

{
  "type": "integer",
  "minimum": 1,
  "maximum": 100,
  "default": 20
}

Codex's generated TypeScript declaration drops those JSON Schema facets and renders only max_operations?: number. REA therefore needs the redundant field/tool prose requested above even though its raw schema is valid.

This is systemic rather than trace-specific: an audit of the live 1.7.0 tools/list result found 208 bounded numeric input fields and no field-level descriptions on those numeric schemas. The audit should distinguish meaningful product ceilings from generated safe-integer bounds, but limit, depth, operation-budget, capture-limit, and page-size fields all share the host-loss problem.

Validation and error ownership

The MCP SDK validates the registered Zod schema before invoking REA's callback. In @modelcontextprotocol/server@2.0.0-beta.4, failed validation is caught by the high-level tool handler and converted directly to text-only { content, isError: true }. Consequently:

registerEnhancedTools never reaches its callback;
REA's toCallToolResult structured error projection never runs;
no provider operation, task, unknown record, or Evidence v2 record is created;
repeated calls receive repeatable prose but no machine-readable constraint data.

REA then parses enhanced input twice more when the callback does run: once in registerEnhancedTools and once in EnhancedTools.execute. The three validation layers have ambiguous ownership, and only the outer SDK layer controls this failure's representation.

The implementation should establish one application-owned validation/error contract while preserving the strict discovery schema. Options include an upstream SDK validation-error projector/hook or a narrow REA adapter/low-level handler that separates advertised JSON Schema from application parsing. Simply weakening the advertised schema would not meet the acceptance criteria.

Internal error algebra gap

Even if validation reaches REA, AnalysisInputError currently retains only the operation. It projects as generic invalid_request, has no argument/received/minimum/maximum details, is non-retryable, and uses generic remediation. A typed constraint violation (or richer AnalysisInputError) is required in addition to changing the MCP boundary.

CLI parity

The same gap exists through the CLI:

rea trace --help shows the default but not the 1-100 range;
rea trace --schema contains the correct minimum and maximum;
rea trace /tmp/nonexistent x --max-operations 300 --format json returns only a generic VALIDATION_ERROR message.

The fix should therefore share constraint projection and remediation across CLI and MCP rather than special-case only trace_feature registration.

Suggested implementation sequence

Add field descriptions with range/default and continuation guidance.
Add typed constraint-violation data to the domain error algebra.
Route MCP and CLI validation failures through that projection without executing the operation.
Test the actual tools/list result and actual invalid calls at 0, 1, 100, 101, negative, fractional, omitted/default, and repeated-template cases.
Audit other caller-meaningful bounded inputs.

Explicit clamping is not needed for the initial fix; rejection with deterministic repair preserves the current evidence semantics.

@morluto commented on GitHub (Jul 16, 2026): ## Confirmed root cause and scope I reproduced this against the published `rea-agents@1.7.0` MCP server and traced the registration path. ### Discovery boundary The live `tools/list` response is correct. `trace_feature.inputSchema.properties.max_operations` contains: ```json { "type": "integer", "minimum": 1, "maximum": 100, "default": 20 } ``` Codex's generated TypeScript declaration drops those JSON Schema facets and renders only `max_operations?: number`. REA therefore needs the redundant field/tool prose requested above even though its raw schema is valid. This is systemic rather than trace-specific: an audit of the live 1.7.0 `tools/list` result found 208 bounded numeric input fields and no field-level descriptions on those numeric schemas. The audit should distinguish meaningful product ceilings from generated safe-integer bounds, but `limit`, depth, operation-budget, capture-limit, and page-size fields all share the host-loss problem. ### Validation and error ownership The MCP SDK validates the registered Zod schema before invoking REA's callback. In `@modelcontextprotocol/server@2.0.0-beta.4`, failed validation is caught by the high-level tool handler and converted directly to text-only `{ content, isError: true }`. Consequently: - `registerEnhancedTools` never reaches its callback; - REA's `toCallToolResult` structured error projection never runs; - no provider operation, task, unknown record, or Evidence v2 record is created; - repeated calls receive repeatable prose but no machine-readable constraint data. REA then parses enhanced input twice more when the callback does run: once in `registerEnhancedTools` and once in `EnhancedTools.execute`. The three validation layers have ambiguous ownership, and only the outer SDK layer controls this failure's representation. The implementation should establish one application-owned validation/error contract while preserving the strict discovery schema. Options include an upstream SDK validation-error projector/hook or a narrow REA adapter/low-level handler that separates advertised JSON Schema from application parsing. Simply weakening the advertised schema would not meet the acceptance criteria. ### Internal error algebra gap Even if validation reaches REA, `AnalysisInputError` currently retains only the operation. It projects as generic `invalid_request`, has no argument/received/minimum/maximum details, is non-retryable, and uses generic remediation. A typed constraint violation (or richer `AnalysisInputError`) is required in addition to changing the MCP boundary. ### CLI parity The same gap exists through the CLI: - `rea trace --help` shows the default but not the 1-100 range; - `rea trace --schema` contains the correct minimum and maximum; - `rea trace /tmp/nonexistent x --max-operations 300 --format json` returns only a generic `VALIDATION_ERROR` message. The fix should therefore share constraint projection and remediation across CLI and MCP rather than special-case only `trace_feature` registration. ### Suggested implementation sequence 1. Add field descriptions with range/default and continuation guidance. 2. Add typed constraint-violation data to the domain error algebra. 3. Route MCP and CLI validation failures through that projection without executing the operation. 4. Test the actual `tools/list` result and actual invalid calls at 0, 1, 100, 101, negative, fractional, omitted/default, and repeated-template cases. 5. Audit other caller-meaningful bounded inputs. Explicit clamping is not needed for the initial fix; rejection with deterministic repair preserves the current evidence semantics.

bsakaguchi commented

2026-10-15 18:21:44 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

Closing this incident report. The reusable JavaScript-analysis and capability-routing work is tracked in #117 and #101.

@morluto commented on GitHub (Jul 16, 2026): Closing this incident report. The reusable JavaScript-analysis and capability-routing work is tracked in #117 and #101.

bsakaguchi commented

2026-10-15 18:21:45 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

Reopening now that the report is generalized; the discovery-time bounds and typed-recovery gap remains actionable.

@morluto commented on GitHub (Jul 16, 2026): Reopening now that the report is generalized; the discovery-time bounds and typed-recovery gap remains actionable.

bsakaguchi commented

2026-10-15 18:21:45 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

REA 2.0 validation slice: the adapter advertises Standard Schema JSON Schema while bypassing SDK text-only prevalidation, then safe-parses once at the MCP boundary. Errors use invalid_request with secret-safe path/reason/bounds metadata and correction-only remediation; wire tests cover misspelled arguments and max_operations=101. CLI JSON workflows use the same issue projection.

@morluto commented on GitHub (Jul 16, 2026): REA 2.0 validation slice: the adapter advertises Standard Schema JSON Schema while bypassing SDK text-only prevalidation, then safe-parses once at the MCP boundary. Errors use invalid_request with secret-safe path/reason/bounds metadata and correction-only remediation; wire tests cover misspelled arguments and max_operations=101. CLI JSON workflows use the same issue projection.

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:28:46 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#152

No description provided.

Rows
Columns