[PR #6632] [CLOSED] fix(mcp): advertise attached-capture incompleteness in compare_web_captures example #7384

Closed
opened 2026-11-01 21:47:27 +00:00 by bsakaguchi · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/6632
Author: @ashyyhere
Created: 10/23/2026
Status: ❌ Closed

Base: main ← Head: fix/compare-web-captures-example


📝 Commits (1)

  • 15483fe fix(mcp): advertise attached-capture incompleteness in compare_web_captures example

📊 Changes

2 files changed (+45 additions, -4 deletions)

View changed files

📝 src/contracts/browserToolContracts.ts (+23 -2)
📝 tests/boundary/mcp/browserComparisonInput.test.ts (+22 -2)

📄 Description

Summary

The default MCP tools/list input example for compare_web_captures advertised two attached-browser captures (browser.mode: "connect") with completeness.status: "complete", equality_eligible: true, no missing sections, and limitations: []. A connected capture cannot recover pre-attach events, so the producer can never emit that state. The example now mirrors the producer's attached state: capture-wide incomplete, equality_eligible: false, missing_sections: ["events"], and the attachment limitations.

Fixes #5897.

Problem and expected behavior

Smallest trigger: call MCP tools/list, take the advertised compare_web_captures example unchanged, and submit it as before_scenario/after_scenario. It compares as overall_status: "unchanged".

Violated invariant: connected captures always declare missing event coverage. globalCompleteness (src/browser/PlaywrightBrowserScenarioProvider.ts) unconditionally records "events" as missing for mode: "connect", which classifies to {status: "incomplete", equality_eligible: false}, and browserScenarioLimitations (src/browser/BrowserScenarioPartialObservation.ts) always returns a non-empty limitation list, including three attachment-specific strings. The advertised example (complete, equality_eligible: true, limitations: []) is a capture the normal producer cannot emit, so clients reading the catalog get the wrong model of attached capture coverage.

Expected: the advertised example describes a capture the producer can emit, so the example compares as unknown — missing or truncated evidence never proves equality.

Change and scope

  • src/contracts/browserToolContracts.ts: the shared exampleScenarioCapture() now uses a new attachedScenarioCompleteness (incomplete, equality_eligible: false, missing_sections: ["events"], truncated_sections: []) and attachedScenarioLimitations (the six unconditional limitation strings plus the three connect-mode strings, with the en-US locale default from browserScenarioEnvironmentSchema) at capture level. This mirrors globalCompleteness and browserScenarioLimitations for this example's shape (no captured response body, no storage seeds, no session event gaps). Per-step completeness stays complete, matching stepCompleteness, which derives only from step artifacts.
  • tests/boundary/mcp/browserComparisonInput.test.ts: the boundary test that pinned the scenario example as unchanged now pins unknown (the diff requires both captures to be equality-eligible before claiming equality), with the test name updated; a new focused test pins the advertised capture state (completeness record and the attachment limitation) so the example cannot regress to advertising complete coverage.
  • Intentionally not included: no change to the capture schema, the comparison contract, docs/browser-scenario-contract.md (already the correct authority), or any provider behavior. The passive before/after example group is untouched; its example is not a scenario capture.

Follow the contribution scope. Ordinary fix within existing boundaries; no maintainer scope decision required (no new provider or capability; no target execution, injection, mutation, or protection recovery).

Contract and boundary impact

  • Semantic owner and earliest changed stage: MCP tool-catalog example metadata owned by src/contracts/browserToolContracts.ts; changed at advertisement time only.
  • Contribution scope fit and linked maintainer decision: ordinary fix within existing boundaries; not applicable.
  • CLI and MCP/tool-catalog contract: the advertised compare_web_captures input example values change (completeness, limitations); input/output schemas unchanged; ignored catalogs are CI-regenerated.
  • Provider, bridge, target-format, or platform compatibility: none; no provider code changes.
  • Evidence, artifact, provenance, or reconstruction contract: the example now preserves the incomplete-coverage semantics the contract requires.
  • Target selection, authority, upstream behavior, and cleanup details: not applicable.
  • Generated metadata (docs/public/product-catalog.json), package, or installation impact: none authored; CI regenerates ignored catalogs.

Expected effects (check known effects; use documentation-only on its own).

  • Read existing files or captured data.
  • Launch or attach to a process.
  • Inject instrumentation or run scripts/code.
  • Change process memory, registers, files, or configuration.
  • Contact network services.
  • Recover protected code or work around protections.
  • Documentation or usability only; no target interaction.
  • Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

  • Tests added or updated: new focused test compare_web_captures advertised example > advertises the attached scenario capture in the producer's state parses the advertised example and asserts the producer-attached state (fails on the base revision, where the example advertises complete); the boundary test's pinned scenario outcome moved unchanged -> unknown (fails on the base revision).
  • Base reproduction or other evidence: executed. The boundary test drives the advertised example through a live in-memory MCP server (client.callTool with examples[0].input) and asserts normalized_result matches the domain comparison; on this branch that comparison returns unknown, observed passing. The producer-side expectations (incomplete, equality_eligible: false, missing_sections: ["events"], attachment limitations present) are asserted by the existing PlaywrightBrowserScenarioProvider.test.ts connect-capture test, which is unchanged and still passes.
  • User-visible CLI/MCP output (if applicable): the advertised example published by tools/list for compare_web_captures now contains "completeness": {"status": "incomplete", "equality_eligible": false, "missing_sections": ["events"], "truncated_sections": []} and the nine applicable limitation strings for the attached example capture.
  • Remaining proof gaps: limitation strings for the example are source-derived from browserScenarioLimitations for this scenario shape; a live Playwright connect session with popup families, storage seeds, or event-budget drops would additionally carry conditional strings not represented in the synthetic example. No real-browser session was run for this change (the boundary test uses the fake CDP browser; provider connect behavior is covered by its existing simulated-session tests).

For evidence-bearing changes:

  • Observed, derived, and inferred claims remain distinguishable.
  • Artifact identity, source provenance, and failed attempts remain preserved.
  • Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

  • npm run test:focused -- tests/boundary/mcp/browserComparisonInput.test.ts — result: 2/2 passed.
  • npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/toolInputAdmission.test.ts tests/boundary/mcp/toolResult.test.ts tests/boundary/mcp/optionalProviderIsolation.test.ts tests/boundary/mcp/observationPartialEvidenceDelivery.test.ts tests/boundary/mcp/targetContentFramingMcp.test.ts tests/boundary/mcp/applicationExportShapeMcp.test.ts tests/boundary/mcp/scriptInventoryRace.test.ts tests/boundary/mcp/browserMcp.test.ts tests/boundary/mcp/browserScenarioMcp.test.ts src/contracts/toolInputReferences.test.ts — result: 103/107 passed; 4 failures were timeouts under 11-file parallel load plus one applicationExportShapeMcp evidence_id assertion that reproduces identically with the change stashed on the clean base (pre-existing, unrelated); the three timeout-prone files re-ran with this change and passed 31/31.
  • npm run check:fast — result: typecheck and lint pass (0 errors).

Compatibility, safety, and release

  • Breaking changes or migration steps: none; input schemas unchanged, only advertised example values.
  • Real Hopper/Ghidra, browser, or OS coverage: not applicable; no target interaction. Browser boundary behavior exercised through the existing fake-CDP tests.
  • Package or release metadata impact: none.
  • Security, privacy, process, or containment review: none; static example metadata.

Author readiness

Human owner: @ashyyhere

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

Before merge

  • This PR addresses a concrete problem or an agreed enhancement (open issue #5897 with reproduction).
  • Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff (not applicable; ordinary fix within existing boundaries).
  • The PR has one focused outcome and the title follows type(scope): outcome.
  • Related issue is linked.
  • Tests cover changed observable behavior and meaningful failure paths.
  • Owning docs, contracts, and generated metadata are updated where needed.
  • User-visible CLI/MCP changes include representative output.
  • I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/6632 **Author:** [@ashyyhere](https://github.com/ashyyhere) **Created:** 10/23/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/compare-web-captures-example` --- ### 📝 Commits (1) - [`15483fe`](https://github.com/morluto/rea/commit/15483fea8035985b7a6672bb1a9bf8099f0f6774) fix(mcp): advertise attached-capture incompleteness in compare_web_captures example ### 📊 Changes **2 files changed** (+45 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `src/contracts/browserToolContracts.ts` (+23 -2) 📝 `tests/boundary/mcp/browserComparisonInput.test.ts` (+22 -2) </details> ### 📄 Description ## Summary The default MCP `tools/list` input example for `compare_web_captures` advertised two attached-browser captures (`browser.mode: "connect"`) with `completeness.status: "complete"`, `equality_eligible: true`, no missing sections, and `limitations: []`. A connected capture cannot recover pre-attach events, so the producer can never emit that state. The example now mirrors the producer's attached state: capture-wide `incomplete`, `equality_eligible: false`, `missing_sections: ["events"]`, and the attachment limitations. Fixes #5897. ## Problem and expected behavior Smallest trigger: call MCP `tools/list`, take the advertised `compare_web_captures` example unchanged, and submit it as `before_scenario`/`after_scenario`. It compares as `overall_status: "unchanged"`. Violated invariant: connected captures always declare missing event coverage. `globalCompleteness` (`src/browser/PlaywrightBrowserScenarioProvider.ts`) unconditionally records `"events"` as missing for `mode: "connect"`, which classifies to `{status: "incomplete", equality_eligible: false}`, and `browserScenarioLimitations` (`src/browser/BrowserScenarioPartialObservation.ts`) always returns a non-empty limitation list, including three attachment-specific strings. The advertised example (`complete`, `equality_eligible: true`, `limitations: []`) is a capture the normal producer cannot emit, so clients reading the catalog get the wrong model of attached capture coverage. Expected: the advertised example describes a capture the producer can emit, so the example compares as `unknown` — missing or truncated evidence never proves equality. ## Change and scope - `src/contracts/browserToolContracts.ts`: the shared `exampleScenarioCapture()` now uses a new `attachedScenarioCompleteness` (`incomplete`, `equality_eligible: false`, `missing_sections: ["events"]`, `truncated_sections: []`) and `attachedScenarioLimitations` (the six unconditional limitation strings plus the three connect-mode strings, with the `en-US` locale default from `browserScenarioEnvironmentSchema`) at capture level. This mirrors `globalCompleteness` and `browserScenarioLimitations` for this example's shape (no captured response body, no storage seeds, no session event gaps). Per-step completeness stays `complete`, matching `stepCompleteness`, which derives only from step artifacts. - `tests/boundary/mcp/browserComparisonInput.test.ts`: the boundary test that pinned the scenario example as `unchanged` now pins `unknown` (the diff requires both captures to be equality-eligible before claiming equality), with the test name updated; a new focused test pins the advertised capture state (completeness record and the attachment limitation) so the example cannot regress to advertising complete coverage. - Intentionally not included: no change to the capture schema, the comparison contract, `docs/browser-scenario-contract.md` (already the correct authority), or any provider behavior. The passive `before`/`after` example group is untouched; its example is not a scenario capture. Follow the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). Ordinary fix within existing boundaries; no maintainer scope decision required (no new provider or capability; no target execution, injection, mutation, or protection recovery). ## Contract and boundary impact - Semantic owner and earliest changed stage: MCP tool-catalog example metadata owned by `src/contracts/browserToolContracts.ts`; changed at advertisement time only. - Contribution scope fit and linked maintainer decision: ordinary fix within existing boundaries; not applicable. - CLI and MCP/tool-catalog contract: the advertised `compare_web_captures` input example values change (`completeness`, `limitations`); input/output schemas unchanged; ignored catalogs are CI-regenerated. - Provider, bridge, target-format, or platform compatibility: none; no provider code changes. - Evidence, artifact, provenance, or reconstruction contract: the example now preserves the incomplete-coverage semantics the contract requires. - Target selection, authority, upstream behavior, and cleanup details: not applicable. - Generated metadata (`docs/public/product-catalog.json`), package, or installation impact: none authored; CI regenerates ignored catalogs. Expected effects (check known effects; use documentation-only on its own). - [x] Read existing files or captured data. - [ ] Launch or attach to a process. - [ ] Inject instrumentation or run scripts/code. - [ ] Change process memory, registers, files, or configuration. - [ ] Contact network services. - [ ] Recover protected code or work around protections. - [x] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Tests added or updated: new focused test `compare_web_captures advertised example > advertises the attached scenario capture in the producer's state` parses the advertised example and asserts the producer-attached state (fails on the base revision, where the example advertises `complete`); the boundary test's pinned scenario outcome moved `unchanged` -> `unknown` (fails on the base revision). - Base reproduction or other evidence: executed. The boundary test drives the advertised example through a live in-memory MCP server (`client.callTool` with `examples[0].input`) and asserts `normalized_result` matches the domain comparison; on this branch that comparison returns `unknown`, observed passing. The producer-side expectations (`incomplete`, `equality_eligible: false`, `missing_sections: ["events"]`, attachment limitations present) are asserted by the existing `PlaywrightBrowserScenarioProvider.test.ts` connect-capture test, which is unchanged and still passes. - User-visible CLI/MCP output (if applicable): the advertised example published by `tools/list` for `compare_web_captures` now contains `"completeness": {"status": "incomplete", "equality_eligible": false, "missing_sections": ["events"], "truncated_sections": []}` and the nine applicable limitation strings for the attached example capture. - Remaining proof gaps: limitation strings for the example are source-derived from `browserScenarioLimitations` for this scenario shape; a live Playwright connect session with popup families, storage seeds, or event-budget drops would additionally carry conditional strings not represented in the synthetic example. No real-browser session was run for this change (the boundary test uses the fake CDP browser; provider connect behavior is covered by its existing simulated-session tests). For evidence-bearing changes: - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run test:focused -- tests/boundary/mcp/browserComparisonInput.test.ts` — result: 2/2 passed. - `npm run test:focused -- tests/boundary/mcp/toolSchemaValidity.test.ts tests/boundary/mcp/toolInputAdmission.test.ts tests/boundary/mcp/toolResult.test.ts tests/boundary/mcp/optionalProviderIsolation.test.ts tests/boundary/mcp/observationPartialEvidenceDelivery.test.ts tests/boundary/mcp/targetContentFramingMcp.test.ts tests/boundary/mcp/applicationExportShapeMcp.test.ts tests/boundary/mcp/scriptInventoryRace.test.ts tests/boundary/mcp/browserMcp.test.ts tests/boundary/mcp/browserScenarioMcp.test.ts src/contracts/toolInputReferences.test.ts` — result: 103/107 passed; 4 failures were timeouts under 11-file parallel load plus one `applicationExportShapeMcp` `evidence_id` assertion that reproduces identically with the change stashed on the clean base (pre-existing, unrelated); the three timeout-prone files re-ran with this change and passed 31/31. - `npm run check:fast` — result: typecheck and lint pass (0 errors). ## Compatibility, safety, and release - Breaking changes or migration steps: none; input schemas unchanged, only advertised example values. - Real Hopper/Ghidra, browser, or OS coverage: not applicable; no target interaction. Browser boundary behavior exercised through the existing fake-CDP tests. - Package or release metadata impact: none. - Security, privacy, process, or containment review: none; static example metadata. ## Author readiness Human owner: @ashyyhere - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement (open issue #5897 with reproduction). - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff (not applicable; ordinary fix within existing boundaries). - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
bsakaguchi 2026-11-01 21:47:27 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
morluto/rea#7384
No description provided.