[GH-ISSUE #2578] fix(identity): example-only catalog changes retain the same digest #897

New issue

Closed

opened 2026-10-15 18:29:23 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:29:23 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2578

Summary

Changing the input examples advertised for an existing MCP tool leaves the catalog identity unchanged. Removing the valid example for annotate_native_function changes the registration's advertised input schema, but both the tool digest and combined catalog digest stay equal. A caller expecting the edited catalog receives aligned from the identity comparison against the original catalog. The documented catalog identity guarantee covers fields that change what clients receive.

Steps to reproduce

Use the source baseline under Runtime or environment with its locked dependencies installed. Save this fixture as src/catalogExamples.regression.test.ts and run the focused command below. The fixture copies one contract, leaves input validation and all other catalog fields unchanged, and checks the production registration projection.

import { expect, it } from "vitest";
import { CATALOG_IDENTITY, createCatalogDigests } from "./catalogIdentity.js";
import { TOOL_CONTRACTS, toolContract } from "./contracts/toolContracts.js";
import { toolRegistrationOptions } from "./server/toolRegistrationOptions.js";
import { createServerIdentity } from "./serverIdentity.js";

it("detects removal of an advertised input example", () => {
  const original = toolContract("annotate_native_function");
  const edited = { ...original, examples: [] };
  const contracts = TOOL_CONTRACTS.map(c => c === original ? edited : c);
  const project = (c: typeof original) => toolRegistrationOptions(c)
    .inputSchema["~standard"].jsonSchema.input({ target: "draft-2020-12" });
  expect(original.examples.every(e => original.inputSchema.safeParse(e.input).success)).toBe(true);
  expect(project(original).examples).toBeDefined();
  expect(project(edited).examples).toBeUndefined();
  expect(project(edited)).not.toEqual(project(original));
  const baseline = CATALOG_IDENTITY.digests;
  const changed = createCatalogDigests(contracts);
  const alignment = createServerIdentity({
    startedAt: "2026-01-01T00:00:00Z",
    expected: { catalog_digest: changed.combined_sha256 },
  }).alignment.state;
  console.log(JSON.stringify({
    toolsDigestEqual: changed.tools_sha256 === baseline.tools_sha256,
    combinedDigestEqual: changed.combined_sha256 === baseline.combined_sha256,
    alignment,
  }));
  expect(changed.combined_sha256).not.toBe(baseline.combined_sha256);
});

npm run test:focused -- src/catalogExamples.regression.test.ts

The verifier executed the same inputs and operations in a focused probe, plus a control that toggles returnsTargetContent. No live MCP process with an edited catalog or full doctor command was run.

Expected behavior

Removing a valid example from an advertised tool input schema should change the tool and combined catalog digests. Comparing the original catalog against the edited catalog's expected digest should detect a mismatch rather than report aligned, because the catalog identity promises to cover contract fields that change what the client receives.

Actual behavior

The original advertised schema includes the example {"procedure":"0x10100","name":"entry","comment":"Analyst observation"}; the edited projection has no examples property. Despite that difference, both digests stay equal. The identity helper reports aligned. The expected digest-change assertion fails; the observation and existing-field control both pass. The focused process exits with code 1.

Evidence

Expected source: src/catalogIdentity.ts:21-25 states that client-visible contract fields belong in the digest; docs/mcp-contracts.md:293-303 documents expected catalog comparison and the canonical complete tools/list schemas.
Failure source: src/catalogIdentity.ts:27-48 hashes the canonical schemas without examples, while src/contracts/toolSchemaMetadata.ts:56-73 adds them to the advertised input projection. The focused fixture above observes unequal advertised schemas with equal digests.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The observed comparison output was {"toolsDigestEqual":true,"combinedDigestEqual":true,"alignment":"aligned"}. The independent control changed the combined digest when returnsTargetContent changed, showing that the observed equality is specific to the omitted examples field.

The production official-tool registration uses the same projection in src/server/registerOfficialTools.ts:35-57. Source inspection shows that the session adapter forwards expected_catalog_digest to the identity helper in src/server/registerSessionStatusTool.ts:76-84, and the doctor's server-identity check compares package version and catalog digest in src/mcpDoctor.ts:178-184. Those public diagnostic consequences are established from source for examples-only differences with otherwise matching identity.

The historical catalog drift feature in https://github.com/morluto/rea/issues/115 is related context, not an existing report of this example-only failure.

Restoration check

Repeat the example-removal fixture. The advertised schemas must still differ, and the tool and combined digests must now differ as well. Comparing the original identity with the edited catalog's expected digest must report catalog_digest_mismatch. Keep the existing control that changes returnsTargetContent and confirm it still changes the combined digest. These checks must preserve the original contract's input validation and the advertised example content when examples are present.

Runtime or environment

Failure was verified at a2d8b94dd5 and is still present at 8b181716ac by source recheck, package rea-agents 6.4.0, Node v26.10.0, npm 11.19.1, Vitest 5.0.3, with locked dependencies already installed. The focused probe completed in 19.33 seconds.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Example-only catalog drift produces a false alignment result. This is a bounded diagnostic correctness failure; input validation and tool execution were not shown to fail, and no data loss was observed.
Workaround status: available
Workaround basis: Compare the canonical tools/list input schemas directly, including their examples, instead of treating equal identity digests as proof of catalog equality. The focused projection comparison distinguishes the catalogs without changing source behavior.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2578 ## Summary Changing the input examples advertised for an existing MCP tool leaves the catalog identity unchanged. Removing the valid example for annotate_native_function changes the registration's advertised input schema, but both the tool digest and combined catalog digest stay equal. A caller expecting the edited catalog receives aligned from the identity comparison against the original catalog. The documented catalog identity guarantee covers fields that change what clients receive. ## Steps to reproduce Use the source baseline under Runtime or environment with its locked dependencies installed. Save this fixture as src/catalogExamples.regression.test.ts and run the focused command below. The fixture copies one contract, leaves input validation and all other catalog fields unchanged, and checks the production registration projection. ```ts import { expect, it } from "vitest"; import { CATALOG_IDENTITY, createCatalogDigests } from "./catalogIdentity.js"; import { TOOL_CONTRACTS, toolContract } from "./contracts/toolContracts.js"; import { toolRegistrationOptions } from "./server/toolRegistrationOptions.js"; import { createServerIdentity } from "./serverIdentity.js"; it("detects removal of an advertised input example", () => { const original = toolContract("annotate_native_function"); const edited = { ...original, examples: [] }; const contracts = TOOL_CONTRACTS.map(c => c === original ? edited : c); const project = (c: typeof original) => toolRegistrationOptions(c) .inputSchema["~standard"].jsonSchema.input({ target: "draft-2020-12" }); expect(original.examples.every(e => original.inputSchema.safeParse(e.input).success)).toBe(true); expect(project(original).examples).toBeDefined(); expect(project(edited).examples).toBeUndefined(); expect(project(edited)).not.toEqual(project(original)); const baseline = CATALOG_IDENTITY.digests; const changed = createCatalogDigests(contracts); const alignment = createServerIdentity({ startedAt: "2026-01-01T00:00:00Z", expected: { catalog_digest: changed.combined_sha256 }, }).alignment.state; console.log(JSON.stringify({ toolsDigestEqual: changed.tools_sha256 === baseline.tools_sha256, combinedDigestEqual: changed.combined_sha256 === baseline.combined_sha256, alignment, })); expect(changed.combined_sha256).not.toBe(baseline.combined_sha256); }); ``` ```sh npm run test:focused -- src/catalogExamples.regression.test.ts ``` The verifier executed the same inputs and operations in a focused probe, plus a control that toggles returnsTargetContent. No live MCP process with an edited catalog or full doctor command was run. ## Expected behavior Removing a valid example from an advertised tool input schema should change the tool and combined catalog digests. Comparing the original catalog against the edited catalog's expected digest should detect a mismatch rather than report aligned, because the catalog identity promises to cover contract fields that change what the client receives. ## Actual behavior The original advertised schema includes the example {"procedure":"0x10100","name":"entry","comment":"Analyst observation"}; the edited projection has no examples property. Despite that difference, both digests stay equal. The identity helper reports aligned. The expected digest-change assertion fails; the observation and existing-field control both pass. The focused process exits with code 1. ## Evidence - Expected source: src/catalogIdentity.ts:21-25 states that client-visible contract fields belong in the digest; docs/mcp-contracts.md:293-303 documents expected catalog comparison and the canonical complete tools/list schemas. - Failure source: src/catalogIdentity.ts:27-48 hashes the canonical schemas without examples, while src/contracts/toolSchemaMetadata.ts:56-73 adds them to the advertised input projection. The focused fixture above observes unequal advertised schemas with equal digests. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The observed comparison output was {"toolsDigestEqual":true,"combinedDigestEqual":true,"alignment":"aligned"}. The independent control changed the combined digest when returnsTargetContent changed, showing that the observed equality is specific to the omitted examples field. The production official-tool registration uses the same projection in src/server/registerOfficialTools.ts:35-57. Source inspection shows that the session adapter forwards expected_catalog_digest to the identity helper in src/server/registerSessionStatusTool.ts:76-84, and the doctor's server-identity check compares package version and catalog digest in src/mcpDoctor.ts:178-184. Those public diagnostic consequences are established from source for examples-only differences with otherwise matching identity. The historical catalog drift feature in https://github.com/morluto/rea/issues/115 is related context, not an existing report of this example-only failure. ## Restoration check Repeat the example-removal fixture. The advertised schemas must still differ, and the tool and combined digests must now differ as well. Comparing the original identity with the edited catalog's expected digest must report catalog_digest_mismatch. Keep the existing control that changes returnsTargetContent and confirm it still changes the combined digest. These checks must preserve the original contract's input validation and the advertised example content when examples are present. ## Runtime or environment Failure was verified at a2d8b94dd50dab8d966239dd63ab84718aa8b4b4 and is still present at 8b181716aca2cb467be499cba226abfb5ae19141 by source recheck, package rea-agents 6.4.0, Node v26.10.0, npm 11.19.1, Vitest 5.0.3, with locked dependencies already installed. The focused probe completed in 19.33 seconds. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Example-only catalog drift produces a false alignment result. This is a bounded diagnostic correctness failure; input validation and tool execution were not shown to fail, and no data loss was observed. - Workaround status: available - Workaround basis: Compare the canonical tools/list input schemas directly, including their examples, instead of treating equal identity digests as proof of catalog equality. The focused projection comparison distinguishes the catalogs without changing source behavior.

bsakaguchi

2026-10-15 18:29:23 +00:00

closed this issue
added the
area: domain

bug
labels

bsakaguchi commented

2026-10-15 18:29:24 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Fixed on main in 80de3145e4: catalog identity uses the owner-generated advertised input schema, including input examples, instead of independently projecting a narrower schema.

Verified with tests/boundary/mcp/serverIdentity.test.ts in the current focused run: all 8 tests passed. The example-only mutation changes tools and combined digests while leaving prompt identity stable. No duplicate examples representation or extra public field was introduced. Whole-main CI qualification remains in progress.

@morluto commented on GitHub (Oct 14, 2026): Fixed on main in 80de3145e4: catalog identity uses the owner-generated advertised input schema, including input examples, instead of independently projecting a narrower schema. Verified with tests/boundary/mcp/serverIdentity.test.ts in the current focused run: all 8 tests passed. The example-only mutation changes tools and combined digests while leaving prompt identity stable. No duplicate examples representation or extra public field was introduced. Whole-main CI qualification remains in progress.

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-16 21:17:21 +00:00

[PR #897] [MERGED] feat(browser): trace captured website source-map locations #2329