morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #101] Make MCP tool discovery capability- and policy-aware #67

New issue

Closed

opened 2026-10-15 18:20:46 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-15 18:20:46 +00:00

Owner

Originally created by @morluto on GitHub (Jul 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/101

Summary

Make REA's MCP discovery and workflow guidance reflect negotiated client features, installed providers, active target state, operator/session policy, and operation-specific limits.

Accurate static annotations have shipped. The remaining work is dynamic availability, truthful routing, and schema-visible constraints.

Current problem

Agents still often discover only after invocation that:

a provider or target does not support the operation;
no target is open;
process/evidence/investigation/mount/replay policy is disabled;
a session grant is missing;
the client lacks elicitation, Tasks, resources, progress, or cancellation support;
an input budget has a smaller maximum than the generated tool declaration exposed;
a stale MCP server is connected after the CLI/skill upgraded;
a different workflow is the correct route for the target and goal.

This produces malformed calls, native-tool attempts on JavaScript, unnecessary fallbacks, and abandoned workflows.

Capability lattice

Expose per-operation state across:

classification/inventory/extraction;
native analysis;
JavaScript static semantics and source maps;
managed-code analysis;
browser/Electron observation;
controlled process capture/replay;
comparison/correlation;
evidence/workspace operations;
reconstruction verification and conformance export.

Each operation reports one of: available, degraded, unavailable, approval-required, policy-disabled, setup-required, target-required, provider-missing, unsupported-host, or client-capability-missing.

Discovery contract

For every visible tool expose or make queryable:

accepted target kinds/formats/architectures;
required provider and currently selected provider;
active-target applicability;
approval and policy/grant requirements;
filesystem/network/process side effects;
exact numeric/string/collection limits and defaults;
pagination behavior;
required client MCP capabilities;
expected Evidence authority and known limitations;
closest supported alternative workflow;
installed CLI, connected MCP server, skill/catalog, and provider versions/digests.

Limits must exist in the actual MCP input schema, not only prose or runtime validation. Errors repeat the accepted range and closest recovery action.

Dynamic behavior

Recompute discovery when target, provider health, setup, policy/grant, client capability, or registration version changes. Advertise tools.listChanged and emit notifications/tools/list_changed when the visible or materially described catalog changes.

When omission would make recovery confusing, retain the tool with typed availability metadata/preflight rather than silently hiding it. Apply one documented policy consistently by category.

Goal-aware routing

Given target plus goal, recommend the next supported workflow with:

prerequisites;
authority limits;
side effects and approvals;
budgets;
expected Evidence;
fallback if unavailable.

For example, a JavaScript bundle with Hopper installed should route to JavaScript application analysis, historical/source-map mapping, or bounded runtime observation—not native procedure analysis merely because the artifact can be opened or inventoried.

Acceptance criteria

Tool discovery incorporates client, host, provider, target, policy, and session-grant state.
Every operation exposes exact accepted limits/defaults in its MCP schema.
Runtime validation errors repeat the precise accepted range and recovery action.
Tool discovery distinguishes target acceptance from operation-level support.
Goal-aware routing never recommends an incompatible provider family.
Material state changes emit tools/list_changed.
Disabled/degraded tools have stable machine-readable reasons.
#96 grants update availability without restart where the client supports it.
Tasks/resources/progress/cancellation recommendations respect negotiated client capabilities.
Installed CLI, connected server, skill/catalog, and provider version drift is diagnosable.
Annotation and availability contracts have catalog-level matrix tests.
Controlled agent evaluations measure first-tool success, malformed calls, fallback rate, and recovery quality.
Documentation explains omission versus retained-unavailable behavior.

Non-goals

Treating MCP annotations as security enforcement.
Treating roots/list as authorization.
Renaming tools dynamically based on model preference.
Claiming a provider supports operations it cannot implement.

Relationships

#96 supplies elicited grants.
#97 supplies Tasks negotiation.
#100 supplies typed results/errors.
#404 supplies JavaScript runtime semantic analysis.
Provider descriptors/SDKs feed the lattice.

Machine-readable capability record

Use a stable, versioned record shared by MCP discovery, preflight, CLI doctor, and workflow prompts. For example:

{
  "operation": "process.capture",
  "state": "approval-required",
  "target": {"accepted": true, "kind": "executable", "digest": "..."},
  "provider": {"id": "process-v4", "available": true},
  "requirements": {
    "client": ["elicitation"],
    "grant": ["process_capture", "filesystem_write"],
    "approval": "per-call"
  },
  "limits": {"max_operations": 100, "max_duration_ms": 30000},
  "side_effects": ["process_execution", "filesystem_write"],
  "alternatives": ["process.preflight"],
  "reason_codes": ["session_grant_missing"]
}

The operation identifier and reason codes must remain stable even if human-facing tool names or descriptions evolve.

Availability evaluation

Compute capability state from a documented decision table:

Is the operation implemented by this server/catalog version?
Does the host satisfy OS/runtime dependencies?
Is a compatible provider installed and healthy?
Is the target classified, open where required, and accepted by that operation?
Does the administrator policy permit the operation in principle?
Is a session/project grant present, or can one be elicited?
Does the client support the required MCP features?
Are current budgets and resource state sufficient?

Return every applicable reason internally, but nominate one primary actionable reason deterministically. Avoid generic unsupported when a narrower reason is known.

Visibility rules

available: advertise normally.
approval-required: advertise with accurate side effects and approval metadata.
setup-required, provider-missing, or recoverable target-required: retain when invocation/preflight gives an actionable path.
permanently incompatible target/host tools: omit when the client handles list changes reliably; otherwise retain with typed unavailability to prevent guesswork.
policy-disabled operations that cannot be elicited: retain only through a safe capability/doctor surface, not as apparently callable analysis tools.

Document this table so agents do not have to infer why two hosts expose different catalogs.

Schema fidelity

Generate MCP input schemas and runtime validators from the same source of truth.
Encode inclusive minima/maxima, enums, patterns, defaults, collection lengths, and mutually exclusive fields structurally.
Avoid documenting a numeric maximum only in prose.
When a limit depends on provider/host/policy, expose the effective value through the capability record and reject above-limit calls with the same value.
Include a catalog/schema digest so clients can identify stale cached declarations.

Change notification and caching

Increment a capability revision when target, provider health, policy/grant, dependency, client feature, or effective limit changes. Debounce bursts but do not lose the final state. A tools/list_changed notification should be followed by a list whose revision reflects the triggering state.

Because not all clients refresh dynamically, rea doctor and a lightweight MCP status/resource should report:

installed CLI version;
connected server version and process start time;
tool/prompt/catalog version and digest;
provider versions and health;
capability revision;
explicit restart action when the connected server is stale.

Goal-routing contract

Routing should accept a target plus a goal vocabulary such as inventory, extract, find_feature, trace_behavior, compare_versions, capture_runtime, or verify_reconstruction. Return ranked workflows, but explain hard compatibility and authority constraints before softer cost/quality preferences.

A recommendation must never turn provider installed into operation supported. It should cite the capability records used, expected Evidence authority, required approvals, and fallback route.

Conformance and evaluation matrix

Add table-driven tests across:

client feature combinations (elicitation, resources, Tasks, progress, cancellation, list changes);
supported/unsupported hosts;
provider absent/unhealthy/healthy/version-mismatched;
no target, compatible target, incompatible format, wrong architecture, stale target;
ceiling denied, grant missing, grant elicitable, granted, expired, revoked;
static versus dynamic limits;
connected-server/CLI/catalog version skew.

Agent evaluations should measure first valid tool selection, malformed-call rate, calls to incompatible provider families, recovery after a typed denial, unnecessary fallback rate, and successful completion—not merely whether a tool name appeared in the prompt.

Delivery slices

Shared capability record, stable reason codes, exact schema limits, and CLI/MCP preflight.
Target/provider/host/policy decision table plus catalog matrix tests.
#96 grant integration and dynamic revision/list-change behavior.
Version/digest drift diagnostics and stale-server guidance.
Goal-aware routing and controlled agent discoverability evaluations.

Originally created by @morluto on GitHub (Jul 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/101 ## Summary Make REA's MCP discovery and workflow guidance reflect negotiated client features, installed providers, active target state, operator/session policy, and operation-specific limits. Accurate static annotations have shipped. The remaining work is dynamic availability, truthful routing, and schema-visible constraints. ## Current problem Agents still often discover only after invocation that: - a provider or target does not support the operation; - no target is open; - process/evidence/investigation/mount/replay policy is disabled; - a session grant is missing; - the client lacks elicitation, Tasks, resources, progress, or cancellation support; - an input budget has a smaller maximum than the generated tool declaration exposed; - a stale MCP server is connected after the CLI/skill upgraded; - a different workflow is the correct route for the target and goal. This produces malformed calls, native-tool attempts on JavaScript, unnecessary fallbacks, and abandoned workflows. ## Capability lattice Expose per-operation state across: - classification/inventory/extraction; - native analysis; - JavaScript static semantics and source maps; - managed-code analysis; - browser/Electron observation; - controlled process capture/replay; - comparison/correlation; - evidence/workspace operations; - reconstruction verification and conformance export. Each operation reports one of: available, degraded, unavailable, approval-required, policy-disabled, setup-required, target-required, provider-missing, unsupported-host, or client-capability-missing. ## Discovery contract For every visible tool expose or make queryable: - accepted target kinds/formats/architectures; - required provider and currently selected provider; - active-target applicability; - approval and policy/grant requirements; - filesystem/network/process side effects; - exact numeric/string/collection limits and defaults; - pagination behavior; - required client MCP capabilities; - expected Evidence authority and known limitations; - closest supported alternative workflow; - installed CLI, connected MCP server, skill/catalog, and provider versions/digests. Limits must exist in the actual MCP input schema, not only prose or runtime validation. Errors repeat the accepted range and closest recovery action. ## Dynamic behavior Recompute discovery when target, provider health, setup, policy/grant, client capability, or registration version changes. Advertise `tools.listChanged` and emit `notifications/tools/list_changed` when the visible or materially described catalog changes. When omission would make recovery confusing, retain the tool with typed availability metadata/preflight rather than silently hiding it. Apply one documented policy consistently by category. ## Goal-aware routing Given target plus goal, recommend the next supported workflow with: - prerequisites; - authority limits; - side effects and approvals; - budgets; - expected Evidence; - fallback if unavailable. For example, a JavaScript bundle with Hopper installed should route to JavaScript application analysis, historical/source-map mapping, or bounded runtime observation—not native procedure analysis merely because the artifact can be opened or inventoried. ## Acceptance criteria - [ ] Tool discovery incorporates client, host, provider, target, policy, and session-grant state. - [ ] Every operation exposes exact accepted limits/defaults in its MCP schema. - [ ] Runtime validation errors repeat the precise accepted range and recovery action. - [ ] Tool discovery distinguishes target acceptance from operation-level support. - [ ] Goal-aware routing never recommends an incompatible provider family. - [ ] Material state changes emit tools/list_changed. - [ ] Disabled/degraded tools have stable machine-readable reasons. - [ ] #96 grants update availability without restart where the client supports it. - [ ] Tasks/resources/progress/cancellation recommendations respect negotiated client capabilities. - [ ] Installed CLI, connected server, skill/catalog, and provider version drift is diagnosable. - [ ] Annotation and availability contracts have catalog-level matrix tests. - [ ] Controlled agent evaluations measure first-tool success, malformed calls, fallback rate, and recovery quality. - [ ] Documentation explains omission versus retained-unavailable behavior. ## Non-goals - Treating MCP annotations as security enforcement. - Treating `roots/list` as authorization. - Renaming tools dynamically based on model preference. - Claiming a provider supports operations it cannot implement. ## Relationships - #96 supplies elicited grants. - #97 supplies Tasks negotiation. - #100 supplies typed results/errors. - #404 supplies JavaScript runtime semantic analysis. - Provider descriptors/SDKs feed the lattice. ## Machine-readable capability record Use a stable, versioned record shared by MCP discovery, preflight, CLI doctor, and workflow prompts. For example: ```json { "operation": "process.capture", "state": "approval-required", "target": {"accepted": true, "kind": "executable", "digest": "..."}, "provider": {"id": "process-v4", "available": true}, "requirements": { "client": ["elicitation"], "grant": ["process_capture", "filesystem_write"], "approval": "per-call" }, "limits": {"max_operations": 100, "max_duration_ms": 30000}, "side_effects": ["process_execution", "filesystem_write"], "alternatives": ["process.preflight"], "reason_codes": ["session_grant_missing"] } ``` The operation identifier and reason codes must remain stable even if human-facing tool names or descriptions evolve. ## Availability evaluation Compute capability state from a documented decision table: 1. Is the operation implemented by this server/catalog version? 2. Does the host satisfy OS/runtime dependencies? 3. Is a compatible provider installed and healthy? 4. Is the target classified, open where required, and accepted by that operation? 5. Does the administrator policy permit the operation in principle? 6. Is a session/project grant present, or can one be elicited? 7. Does the client support the required MCP features? 8. Are current budgets and resource state sufficient? Return every applicable reason internally, but nominate one primary actionable reason deterministically. Avoid generic `unsupported` when a narrower reason is known. ## Visibility rules - `available`: advertise normally. - `approval-required`: advertise with accurate side effects and approval metadata. - `setup-required`, `provider-missing`, or recoverable `target-required`: retain when invocation/preflight gives an actionable path. - permanently incompatible target/host tools: omit when the client handles list changes reliably; otherwise retain with typed unavailability to prevent guesswork. - policy-disabled operations that cannot be elicited: retain only through a safe capability/doctor surface, not as apparently callable analysis tools. Document this table so agents do not have to infer why two hosts expose different catalogs. ## Schema fidelity - Generate MCP input schemas and runtime validators from the same source of truth. - Encode inclusive minima/maxima, enums, patterns, defaults, collection lengths, and mutually exclusive fields structurally. - Avoid documenting a numeric maximum only in prose. - When a limit depends on provider/host/policy, expose the effective value through the capability record and reject above-limit calls with the same value. - Include a catalog/schema digest so clients can identify stale cached declarations. ## Change notification and caching Increment a capability revision when target, provider health, policy/grant, dependency, client feature, or effective limit changes. Debounce bursts but do not lose the final state. A `tools/list_changed` notification should be followed by a list whose revision reflects the triggering state. Because not all clients refresh dynamically, `rea doctor` and a lightweight MCP status/resource should report: - installed CLI version; - connected server version and process start time; - tool/prompt/catalog version and digest; - provider versions and health; - capability revision; - explicit restart action when the connected server is stale. ## Goal-routing contract Routing should accept a target plus a goal vocabulary such as `inventory`, `extract`, `find_feature`, `trace_behavior`, `compare_versions`, `capture_runtime`, or `verify_reconstruction`. Return ranked workflows, but explain hard compatibility and authority constraints before softer cost/quality preferences. A recommendation must never turn `provider installed` into `operation supported`. It should cite the capability records used, expected Evidence authority, required approvals, and fallback route. ## Conformance and evaluation matrix Add table-driven tests across: - client feature combinations (elicitation, resources, Tasks, progress, cancellation, list changes); - supported/unsupported hosts; - provider absent/unhealthy/healthy/version-mismatched; - no target, compatible target, incompatible format, wrong architecture, stale target; - ceiling denied, grant missing, grant elicitable, granted, expired, revoked; - static versus dynamic limits; - connected-server/CLI/catalog version skew. Agent evaluations should measure first valid tool selection, malformed-call rate, calls to incompatible provider families, recovery after a typed denial, unnecessary fallback rate, and successful completion—not merely whether a tool name appeared in the prompt. ## Delivery slices 1. Shared capability record, stable reason codes, exact schema limits, and CLI/MCP preflight. 2. Target/provider/host/policy decision table plus catalog matrix tests. 3. #96 grant integration and dynamic revision/list-change behavior. 4. Version/digest drift diagnostics and stale-server guidance. 5. Goal-aware routing and controlled agent discoverability evaluations.

bsakaguchi

2026-10-15 18:20:46 +00:00

closed this issue
added the
area: mcp

enhancement
labels

bsakaguchi commented

2026-10-15 18:20:47 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

Concrete annotation audit from the current 84-tool wire surface

All tools publish the four MCP annotation booleans, but the values are not consistently derived from actual operation effects.

The central annotations(name, kind) helper uses tool family plus a small name allow/deny list. This causes obvious read operations such as list_names, list_strings, procedure_info, procedure_references, search_strings, and xrefs to publish readOnlyHint: false because they are official-proxy tools. Native/browser/application families use separate hand-written policies, so equivalent behavior is classified differently by constructor.

Two semantic decisions also need to be made explicit:

Whether recording session Evidence or residual unknowns counts as modifying the MCP tool's environment. Today some evidence-producing tools publish read-only and others do not.
Whether destructiveHint means destructive environmental updates, as MCP defines it, rather than general risk. For example, run_controlled_replay publishes destructiveHint: true; controlled execution may be high-risk and approval-gated without necessarily performing a destructive update. Optional additive file export should not be conflated with destructive overwrite.

Recommendation: make one canonical effect descriptor the source for capability inventory, MCP annotations, setup/approval prose, and tests. Project readOnlyHint, destructiveHint, idempotentHint, and openWorldHint from those effects with explicit policy for internal session/evidence mutation. Name-based exceptions should fail a contract test when a new tool is added.

Additional acceptance checks suggested for this issue:

representative official read operations publish readOnlyHint: true;
GUI cursor/document navigation is distinguished from artifact mutation;
additive writes publish destructiveHint: false unless overwrite/removal is possible;
tools that may overwrite, remove, or discard caller state publish destructiveHint: true;
controlled execution risk remains represented in descriptions/approval/policy even when not destructive;
browser/external network interactions and local closed-world process/provider operations classify openWorldHint consistently;
annotation snapshots are generated from canonical effects, not duplicated name lists.

@morluto commented on GitHub (Jul 16, 2026): ## Concrete annotation audit from the current 84-tool wire surface All tools publish the four MCP annotation booleans, but the values are not consistently derived from actual operation effects. The central `annotations(name, kind)` helper uses tool family plus a small name allow/deny list. This causes obvious read operations such as `list_names`, `list_strings`, `procedure_info`, `procedure_references`, `search_strings`, and `xrefs` to publish `readOnlyHint: false` because they are `official-proxy` tools. Native/browser/application families use separate hand-written policies, so equivalent behavior is classified differently by constructor. Two semantic decisions also need to be made explicit: 1. Whether recording session Evidence or residual unknowns counts as modifying the MCP tool's environment. Today some evidence-producing tools publish read-only and others do not. 2. Whether `destructiveHint` means destructive environmental updates, as MCP defines it, rather than general risk. For example, `run_controlled_replay` publishes `destructiveHint: true`; controlled execution may be high-risk and approval-gated without necessarily performing a destructive update. Optional additive file export should not be conflated with destructive overwrite. Recommendation: make one canonical effect descriptor the source for capability inventory, MCP annotations, setup/approval prose, and tests. Project `readOnlyHint`, `destructiveHint`, `idempotentHint`, and `openWorldHint` from those effects with explicit policy for internal session/evidence mutation. Name-based exceptions should fail a contract test when a new tool is added. Additional acceptance checks suggested for this issue: - representative official read operations publish `readOnlyHint: true`; - GUI cursor/document navigation is distinguished from artifact mutation; - additive writes publish `destructiveHint: false` unless overwrite/removal is possible; - tools that may overwrite, remove, or discard caller state publish `destructiveHint: true`; - controlled execution risk remains represented in descriptions/approval/policy even when not destructive; - browser/external network interactions and local closed-world process/provider operations classify `openWorldHint` consistently; - annotation snapshots are generated from canonical effects, not duplicated name lists.

bsakaguchi commented

2026-10-15 18:20:47 +00:00

Author

Owner

@morluto commented on GitHub (Jul 16, 2026):

REA 2.0 annotation slice: all 84 tools now carry explicit ToolEffects (target/session/filesystem/process/network/UI/discard/idempotence). MCP annotations and capability inventory derive from that model; catalog tests reject missing or heuristic fallback entries. Dynamic availability, Tasks, and elicitation remain separate work in this issue.

@morluto commented on GitHub (Jul 16, 2026): REA 2.0 annotation slice: all 84 tools now carry explicit ToolEffects (target/session/filesystem/process/network/UI/discard/idempotence). MCP annotations and capability inventory derive from that model; catalog tests reject missing or heuristic fallback entries. Dynamic availability, Tasks, and elicitation remain separate work in this issue.

bsakaguchi commented

2026-10-15 18:20:47 +00:00

Author

Owner

@morluto commented on GitHub (Jul 20, 2026):

Split into focused child issues:

#392
#393

These replace the broad remaining scope in this issue. Closing this parent can happen once all children are closed.

@morluto commented on GitHub (Jul 20, 2026): Split into focused child issues: - [ ] #392 - [ ] #393 These replace the broad remaining scope in this issue. Closing this parent can happen once all children are closed.

bsakaguchi commented

2026-10-15 18:20:48 +00:00

Author

Owner

@morluto commented on GitHub (Jul 25, 2026):

The focused children #392 and #393 are closed: merged PR #433 delivers dynamic availability integration and merged PR #424 delivers target/policy-aware filtering and catalog tests.

@morluto commented on GitHub (Jul 25, 2026): The focused children #392 and #393 are closed: merged PR #433 delivers dynamic availability integration and merged PR #424 delivers target/policy-aware filtering and catalog tests.

bsakaguchi referenced this issue

2026-10-15 18:21:47 +00:00

[GH-ISSUE #268] Build an evidence-backed reconstruction workspace and coverage closure workflow #157

bsakaguchi referenced this issue

2026-10-15 18:22:27 +00:00

[GH-ISSUE #361] Close browser and source-to-bundle comparison gaps #208

bsakaguchi referenced this issue

2026-10-15 18:22:40 +00:00

[GH-ISSUE #379] Define compare_source_to_bundle contract and scoring model #226

bsakaguchi referenced this issue

2026-10-15 18:22:40 +00:00

[GH-ISSUE #380] Implement source-to-bundle signal pipeline and classification #227