morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #7307] [CLOSED] test(skill): guard bundled SKILL.md references against the advertised MCP catalog #7813

New issue

Closed

opened 2026-11-05 01:47:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-05 01:47:52 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/7307
Author: @akushonkamen
Created: 10/26/2026
Status: ❌ Closed

Base: main ← Head: fix/5244-skill-mcp-catalog-consistency

📝 Commits (1)

5d41501 test(skill): guard bundled SKILL.md references against the advertised MCP catalog

📊 Changes

1 file changed (+146 additions, -0 deletions)

View changed files

➕ tests/boundary/skillCatalogConsistency.test.ts (+146 -0)

📄 Description

Summary

Adds a boundary test that guards the consistency between the bundled agent skills and the advertised MCP tool catalog (#5244). A bundled SKILL.md that documents a tool the server does not advertise makes the documented workflow fail with Unknown tool for every agent that follows it, even when doctor reports the catalog digest as aligned, because the digest covers only the catalog itself and not the identifiers referenced in the skill prose.

Fixes #5244

Problem and expected behavior

On rea-agents 6.7.0 the bundled skill v42 documents tools such as inspect_pcap_capture, inspect_wasm_artifact, and inspect_sqlite_database, while a fresh server session advertises 170 tools that do not include them; calling them fails with Unknown tool 'rea.inspect_pcap_capture'. Nothing in the repository currently detects this divergence: the catalog digest validates the catalog against itself, so a skill that drifts from the advertised surface ships silently.

Expected behavior: any snake_case identifier referenced in a bundled SKILL.md must belong to the advertised surface — an MCP tool name, a documented tool input/output property, a CLI command, or a supported client name.

Change and scope

The branch adds one CPU-only, name-based guard, tests/boundary/skillCatalogConsistency.test.ts (146 lines, no production changes). It walks every bundled SKILL.md, extracts backtick-quoted snake_case identifiers, and requires each to resolve against the advertised surface built from TOOL_CONTRACTS (including advertised input and output schema properties), CLI_COMMAND_NAMES, and SUPPORTED_CLIENT_DEFINITIONS.

Scope is intentionally limited to detection: the test fails on current drift and names the offending identifiers, but does not decide per-tool whether a missing tool should be re-advertised or removed from the prose — those follow-up choices belong to the maintainers, and the discussion in #5244 (environment gates such as REA_TSHARK_COMMAND suppressing otherwise-registered tools) shows the resolution direction is not settled yet.

Contract and boundary impact

Semantic owner and earliest changed stage: test-only; owner unchanged. Consumers: anyone editing bundled skills or the tool catalog gets an earlier, actionable failure signal in CI.
Contribution scope fit and linked maintainer decision: ordinary test coverage within existing boundaries; not applicable.
CLI and MCP/tool-catalog contract: none; the test reads the existing contracts and asserts consistency, it does not change them.
Provider, bridge, target-format, or platform compatibility: none.
Evidence, artifact, provenance, or reconstruction contract: none.
Target selection, authority, upstream behavior, and cleanup details: not applicable.
Generated metadata, package, or installation impact: none; no build outputs committed.

Expected effects:

Documentation or usability only; no target interaction.

Evidence and regression coverage

Tests added or updated: tests/boundary/skillCatalogConsistency.test.ts (new, 2 cases).
Base reproduction or other evidence: source-derived — the divergence reported in #5244 (207 documented vs 170 advertised in a fresh session) exists at the rea-agents-6.7.0 tag; the guard is the regression that would fail on the affected base revision.
Remaining proof gaps: the guard is name-based and cannot catch semantic drift (a documented tool that exists but behaves differently than the prose implies).
Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npx vitest run tests/boundary/skillCatalogConsistency.test.ts — 2 passed (2) on this branch at 5d41501f2.

Compatibility, safety, and release

Breaking changes or migration steps: none.
Real Hopper/Ghidra, browser, or OS coverage: not applicable; the test is filesystem- and CPU-only.
Package or release metadata impact: none.
Security, privacy, process, or containment review: none; no target interaction.

Author readiness

Human owner:

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/7307 **Author:** [@akushonkamen](https://github.com/akushonkamen) **Created:** 10/26/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/5244-skill-mcp-catalog-consistency` --- ### 📝 Commits (1) - [`5d41501`](https://github.com/morluto/rea/commit/5d41501f2a84b1258ffc8dc8cb363c12123b1d94) test(skill): guard bundled SKILL.md references against the advertised MCP catalog ### 📊 Changes **1 file changed** (+146 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `tests/boundary/skillCatalogConsistency.test.ts` (+146 -0) </details> ### 📄 Description ## Summary Adds a boundary test that guards the consistency between the bundled agent skills and the advertised MCP tool catalog (#5244). A bundled `SKILL.md` that documents a tool the server does not advertise makes the documented workflow fail with `Unknown tool` for every agent that follows it, even when `doctor` reports the catalog digest as aligned, because the digest covers only the catalog itself and not the identifiers referenced in the skill prose. Fixes #5244 ## Problem and expected behavior On rea-agents 6.7.0 the bundled skill v42 documents tools such as `inspect_pcap_capture`, `inspect_wasm_artifact`, and `inspect_sqlite_database`, while a fresh server session advertises 170 tools that do not include them; calling them fails with `Unknown tool 'rea.inspect_pcap_capture'`. Nothing in the repository currently detects this divergence: the catalog digest validates the catalog against itself, so a skill that drifts from the advertised surface ships silently. Expected behavior: any snake_case identifier referenced in a bundled `SKILL.md` must belong to the advertised surface — an MCP tool name, a documented tool input/output property, a CLI command, or a supported client name. ## Change and scope The branch adds one CPU-only, name-based guard, `tests/boundary/skillCatalogConsistency.test.ts` (146 lines, no production changes). It walks every bundled `SKILL.md`, extracts backtick-quoted snake_case identifiers, and requires each to resolve against the advertised surface built from `TOOL_CONTRACTS` (including advertised input and output schema properties), `CLI_COMMAND_NAMES`, and `SUPPORTED_CLIENT_DEFINITIONS`. Scope is intentionally limited to detection: the test fails on current drift and names the offending identifiers, but does not decide per-tool whether a missing tool should be re-advertised or removed from the prose — those follow-up choices belong to the maintainers, and the discussion in #5244 (environment gates such as `REA_TSHARK_COMMAND` suppressing otherwise-registered tools) shows the resolution direction is not settled yet. ## Contract and boundary impact - Semantic owner and earliest changed stage: test-only; owner unchanged. Consumers: anyone editing bundled skills or the tool catalog gets an earlier, actionable failure signal in CI. - Contribution scope fit and linked maintainer decision: ordinary test coverage within existing boundaries; not applicable. - CLI and MCP/tool-catalog contract: none; the test reads the existing contracts and asserts consistency, it does not change them. - Provider, bridge, target-format, or platform compatibility: none. - Evidence, artifact, provenance, or reconstruction contract: none. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata, package, or installation impact: none; no build outputs committed. Expected effects: - [x] Documentation or usability only; no target interaction. ## Evidence and regression coverage - Tests added or updated: `tests/boundary/skillCatalogConsistency.test.ts` (new, 2 cases). - Base reproduction or other evidence: source-derived — the divergence reported in #5244 (207 documented vs 170 advertised in a fresh session) exists at the rea-agents-6.7.0 tag; the guard is the regression that would fail on the affected base revision. - Remaining proof gaps: the guard is name-based and cannot catch semantic drift (a documented tool that exists but behaves differently than the prose implies). - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npx vitest run tests/boundary/skillCatalogConsistency.test.ts` — 2 passed (2) on this branch at `5d41501f2`. ## Compatibility, safety, and release - Breaking changes or migration steps: none. - Real Hopper/Ghidra, browser, or OS coverage: not applicable; the test is filesystem- and CPU-only. - Package or release metadata impact: none. - Security, privacy, process, or containment review: none; no target interaction. ## Author readiness Human owner:  - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-05 01:47:52 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7813

No description provided.

Rows
Columns