morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5961] [Bug] match_native_functions misclassifies authentic wrong-role Evidence as an integrity failure #3633

New issue

Closed

opened 2026-10-31 00:42:33 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:33 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5961

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

match_native_functions reports authentic Evidence supplied in the wrong request role as evidence_integrity_mismatch, with advice to recreate or re-import it. The records pass parseEvidence; correcting their selection succeeds without recreating them.

I reproduced six cases through the registered MCP server: wrong-operation inventory input on either side, an observation from the opposite target on either side, and a same-context list_procedures record in either observation list. Each returned the integrity error family with only details.reason, no rejected field/index, and no input-contract recovery.

Reproduction

Save the following test as tests/boundary/mcp/nativeMatchingRoleDiagnostics.test.ts in the source checkout with dependencies installed, then run:

./node_modules/.bin/vitest run tests/boundary/mcp/nativeMatchingRoleDiagnostics.test.ts --maxWorkers 1

This test was executed using the real SDK client, InMemoryTransport, and the registered production server. Evidence comes from the repository's synthetic fixtures. The session throws if a provider starts; no native analysis engine or live binary is used.

import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { expect, it } from "vitest";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
import { nativeMatchingInventory, nativeMatchingDetails } from "../../fixtures/nativeFunctionMatching.js";
import { createServer } from "../../../src/server/createServer.js";
import { parseEvidence } from "../../../src/domain/evidence.js";

it("misclassifies authentic wrong-role and wrong-side Evidence", async () => {
  let providerLaunches = 0;
  const session = createTestBinarySession(() => {
    providerLaunches++;
    throw new Error("No provider should launch");
  });
  const server = createServer({ kind: "session", session });
  const client = new Client({ name: "matching-role-test", version: "1" });
  const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
  const left = nativeMatchingInventory("1", [{ address: "0x10", value: "main" }]);
  const right = nativeMatchingInventory("2", [{ address: "0x20", value: "main" }]);
  const leftDetails = nativeMatchingDetails("1", "0x10", "c3", true);
  const rightDetails = nativeMatchingDetails("2", "0x20", "c3");
  for (const record of [left, right, ...leftDetails, ...rightDetails])
    expect(parseEvidence(record)).toEqual(record);
  const cases = [
    { name: "left wrong operation", input: { left: leftDetails[0] }, reason: "Expected artifact-bound list_procedures Evidence" },
    { name: "right wrong operation", input: { right: rightDetails[0] }, reason: "Expected artifact-bound list_procedures Evidence" },
    { name: "left wrong context", input: { left_observations: [rightDetails[0]] }, reason: "Function observation does not belong to its selected inventory context" },
    { name: "right wrong context", input: { right_observations: [leftDetails[0]] }, reason: "Function observation does not belong to its selected inventory context" },
    { name: "left unsupported observation", input: { left_observations: [left] }, reason: "Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence" },
    { name: "right unsupported observation", input: { right_observations: [right] }, reason: "Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence" },
  ];
  try {
    await server.connect(serverTransport);
    await client.connect(clientTransport);
    for (const entry of cases) {
      const result = await client.callTool({ name: "match_native_functions", arguments: { left, right, detail: "complete", ...entry.input } });
      const diagnostic = parseMcpToolError(result);
      console.log(JSON.stringify({ case: entry.name, diagnostic }));
      expect(diagnostic.error).toMatchObject({
        code: "evidence_integrity_mismatch", category: "integrity_mismatch",
        message: "Evidence is invalid or has changed. Recreate or re-import it, then try again.",
        details: { reason: entry.reason },
      });
      expect(diagnostic.error).not.toHaveProperty("recovery");
      expect(diagnostic.error.details).not.toHaveProperty("issues");
    }
    const corrected = await client.callTool({ name: "match_native_functions", arguments: {
      left, right, left_observations: leftDetails, right_observations: rightDetails, detail: "complete",
    } });
    expect(corrected.isError).not.toBe(true);
    expect(parseEvidence(corrected.structuredContent).normalized_result).toMatchObject({ summary: { matched: 1 } });
    expect(providerLaunches).toBe(0);
    console.log(JSON.stringify({ corrected: "matched: 1", providerLaunches }));
    await client.ping();
  } finally {
    await client.close();
    await server.close();
    await session.close();
  }
});

One selected test passed. It asserts the currently incorrect diagnostic for all six cases, followed by a successful corrected-input control. This does not assert that the current classification is desirable.

Expected behavior

Classify authentic Evidence selected for the wrong inventory or observation role as invalid_request / invalid_input. Identify left, right, or the observation field and index, explain the expected operation or side context, and direct the caller to correct the selection. Preserve integrity errors for actual contradictory producer facts, authentication failures, and missing bundle dependencies.

The matching contract specifies the inventory operations and same-context observation kinds. The matching guide requires the selected side's exact context. The analogous process comparison test requires an input error naming the rejected side for authentic wrong-workflow Evidence.

Impact and scope

P2: callers that use the typed error family receive integrity-recovery advice for a correctable selection error. Selecting the right inventories and observations is a demonstrated workaround: the corrected call returned one match with zero provider launches. No corruption, accepted invalid matching result, actual automated retry loop, or repeated native analysis was observed.

Executed coverage includes wrong operations and opposite-target context on both sides using inline authenticated Evidence. Provider/profile/document-only mismatches, retained references, and CLI behavior were not separately executed. The CLI calls the same service by source inspection.

REA version

rea-agents 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3. The repository pins Node 24.18.0; that exact runtime was not separately tested.

Target and artifact context

Synthetic authenticated native list_procedures, analyze_function, procedure_info, and read_bytes Evidence for two fixture targets. No live target binary was opened or executed.

Interface

MCP

Earliest failure stage

Request parsing or validation

Relevant output or evidence

The wrong-operation inventory cases returned this canonical diagnostic as MCP text content with isError: true:

{
  "error": {
    "code": "evidence_integrity_mismatch",
    "category": "integrity_mismatch",
    "message": "Evidence is invalid or has changed. Recreate or re-import it, then try again.",
    "retryable": false,
    "remediation": {
      "action": "Evidence is invalid or has changed. Recreate or re-import it, then try again."
    },
    "details": {
      "reason": "Expected artifact-bound list_procedures Evidence"
    }
  }
}

Wrong-side observations returned the same diagnostic shape with details.reason: "Function observation does not belong to its selected inventory context". Unsupported observation operations returned "Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence". None included recovery or details.issues.

The corrected-input call returned successful matching Evidence with normalized_result.summary.matched: 1; the client remained responsive to ping and the provider launch count remained zero.

Typed outcome or recovery expected

invalid_request / invalid_input, details.issues naming the rejected side or observation index with the role/context constraint, and recovery.kind: inspect_input_contract for match_native_functions.

Analysis or suspected cause

Source-derived: the shared service authenticates the selected records, then calls the domain inventory reader. That reader throws EvidenceIntegrityError for wrong inventory roles, wrong observation context, and unsupported observation operations. The application's existing AnalysisError passthrough preserves that class and its generic integrity recovery.

Role-aware input validation needs to distinguish these caller selections from actual producer contradictions; converting every integrity error to an input error would also change unrelated genuine integrity failures.

Additional context

Related reports checked: #5932 covers a separate operation/predicate gate in compare_functions. Changing that handler leaves the native matching reader unchanged. #5950 covers structurally malformed inline values that fail deferred Evidence parsing, already return invalid_request, and lose their field paths. Here the records authenticate and later role checks select the wrong error family, so preserving parse-error paths alone does not fix this behavior.

The matching service, reader, contract, registration, CLI adapter, and relevant integrity classification/recovery branches remain unchanged at fetched origin/main revision 25e1a24a3ed3671a0148757555f18d16c29d936a. That later source was inspected, not executed.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5961 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Observed behavior `match_native_functions` reports authentic Evidence supplied in the wrong request role as `evidence_integrity_mismatch`, with advice to recreate or re-import it. The records pass `parseEvidence`; correcting their selection succeeds without recreating them. I reproduced six cases through the registered MCP server: wrong-operation inventory input on either side, an observation from the opposite target on either side, and a same-context `list_procedures` record in either observation list. Each returned the integrity error family with only `details.reason`, no rejected field/index, and no input-contract recovery. ### Reproduction Save the following test as `tests/boundary/mcp/nativeMatchingRoleDiagnostics.test.ts` in the source checkout with dependencies installed, then run: ```sh ./node_modules/.bin/vitest run tests/boundary/mcp/nativeMatchingRoleDiagnostics.test.ts --maxWorkers 1 ``` This test was executed using the real SDK client, `InMemoryTransport`, and the registered production server. Evidence comes from the repository's synthetic fixtures. The session throws if a provider starts; no native analysis engine or live binary is used. ```ts import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { expect, it } from "vitest"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import { nativeMatchingInventory, nativeMatchingDetails } from "../../fixtures/nativeFunctionMatching.js"; import { createServer } from "../../../src/server/createServer.js"; import { parseEvidence } from "../../../src/domain/evidence.js"; it("misclassifies authentic wrong-role and wrong-side Evidence", async () => { let providerLaunches = 0; const session = createTestBinarySession(() => { providerLaunches++; throw new Error("No provider should launch"); }); const server = createServer({ kind: "session", session }); const client = new Client({ name: "matching-role-test", version: "1" }); const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); const left = nativeMatchingInventory("1", [{ address: "0x10", value: "main" }]); const right = nativeMatchingInventory("2", [{ address: "0x20", value: "main" }]); const leftDetails = nativeMatchingDetails("1", "0x10", "c3", true); const rightDetails = nativeMatchingDetails("2", "0x20", "c3"); for (const record of [left, right, ...leftDetails, ...rightDetails]) expect(parseEvidence(record)).toEqual(record); const cases = [ { name: "left wrong operation", input: { left: leftDetails[0] }, reason: "Expected artifact-bound list_procedures Evidence" }, { name: "right wrong operation", input: { right: rightDetails[0] }, reason: "Expected artifact-bound list_procedures Evidence" }, { name: "left wrong context", input: { left_observations: [rightDetails[0]] }, reason: "Function observation does not belong to its selected inventory context" }, { name: "right wrong context", input: { right_observations: [leftDetails[0]] }, reason: "Function observation does not belong to its selected inventory context" }, { name: "left unsupported observation", input: { left_observations: [left] }, reason: "Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence" }, { name: "right unsupported observation", input: { right_observations: [right] }, reason: "Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence" }, ]; try { await server.connect(serverTransport); await client.connect(clientTransport); for (const entry of cases) { const result = await client.callTool({ name: "match_native_functions", arguments: { left, right, detail: "complete", ...entry.input } }); const diagnostic = parseMcpToolError(result); console.log(JSON.stringify({ case: entry.name, diagnostic })); expect(diagnostic.error).toMatchObject({ code: "evidence_integrity_mismatch", category: "integrity_mismatch", message: "Evidence is invalid or has changed. Recreate or re-import it, then try again.", details: { reason: entry.reason }, }); expect(diagnostic.error).not.toHaveProperty("recovery"); expect(diagnostic.error.details).not.toHaveProperty("issues"); } const corrected = await client.callTool({ name: "match_native_functions", arguments: { left, right, left_observations: leftDetails, right_observations: rightDetails, detail: "complete", } }); expect(corrected.isError).not.toBe(true); expect(parseEvidence(corrected.structuredContent).normalized_result).toMatchObject({ summary: { matched: 1 } }); expect(providerLaunches).toBe(0); console.log(JSON.stringify({ corrected: "matched: 1", providerLaunches })); await client.ping(); } finally { await client.close(); await server.close(); await session.close(); } }); ``` One selected test passed. It asserts the currently incorrect diagnostic for all six cases, followed by a successful corrected-input control. This does not assert that the current classification is desirable. ### Expected behavior Classify authentic Evidence selected for the wrong inventory or observation role as `invalid_request` / `invalid_input`. Identify `left`, `right`, or the observation field and index, explain the expected operation or side context, and direct the caller to correct the selection. Preserve integrity errors for actual contradictory producer facts, authentication failures, and missing bundle dependencies. The [matching contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/native/functionMatchingContracts.ts#L9-L28) specifies the inventory operations and same-context observation kinds. The [matching guide](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/native-function-matching.md#L28-L31) requires the selected side's exact context. The [analogous process comparison test](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts#L448-L475) requires an input error naming the rejected side for authentic wrong-workflow Evidence. ### Impact and scope P2: callers that use the typed error family receive integrity-recovery advice for a correctable selection error. Selecting the right inventories and observations is a demonstrated workaround: the corrected call returned one match with zero provider launches. No corruption, accepted invalid matching result, actual automated retry loop, or repeated native analysis was observed. Executed coverage includes wrong operations and opposite-target context on both sides using inline authenticated Evidence. Provider/profile/document-only mismatches, retained references, and CLI behavior were not separately executed. The CLI calls the same service by source inspection. ### REA version `rea-agents` 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3. The repository pins Node 24.18.0; that exact runtime was not separately tested. ### Target and artifact context Synthetic authenticated native `list_procedures`, `analyze_function`, `procedure_info`, and `read_bytes` Evidence for two fixture targets. No live target binary was opened or executed. ### Interface MCP ### Earliest failure stage Request parsing or validation ### Relevant output or evidence The wrong-operation inventory cases returned this canonical diagnostic as MCP text content with `isError: true`: ```json { "error": { "code": "evidence_integrity_mismatch", "category": "integrity_mismatch", "message": "Evidence is invalid or has changed. Recreate or re-import it, then try again.", "retryable": false, "remediation": { "action": "Evidence is invalid or has changed. Recreate or re-import it, then try again." }, "details": { "reason": "Expected artifact-bound list_procedures Evidence" } } } ``` Wrong-side observations returned the same diagnostic shape with `details.reason: "Function observation does not belong to its selected inventory context"`. Unsupported observation operations returned `"Function matching observations must be list_names, procedure_info, read_bytes or analyze_function Evidence"`. None included `recovery` or `details.issues`. The corrected-input call returned successful matching Evidence with `normalized_result.summary.matched: 1`; the client remained responsive to `ping` and the provider launch count remained zero. ### Typed outcome or recovery expected `invalid_request` / `invalid_input`, `details.issues` naming the rejected side or observation index with the role/context constraint, and `recovery.kind: inspect_input_contract` for `match_native_functions`. ### Analysis or suspected cause Source-derived: the shared service [authenticates the selected records](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/native/NativeFunctionMatchingService.ts#L65-L97), then calls the domain inventory reader. That reader throws `EvidenceIntegrityError` for [wrong inventory roles](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/functionInventory.ts#L245-L252), [wrong observation context](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/functionInventory.ts#L50-L65), and [unsupported observation operations](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/functionInventory.ts#L261-L279). The application's [existing AnalysisError passthrough](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/native/NativeFunctionMatchingService.ts#L192-L209) preserves that class and its generic integrity recovery. Role-aware input validation needs to distinguish these caller selections from actual producer contradictions; converting every integrity error to an input error would also change unrelated genuine integrity failures. ### Additional context Related reports checked: #5932 covers a separate operation/predicate gate in `compare_functions`. Changing that handler leaves the native matching reader unchanged. [#5950](https://github.com/morluto/rea/issues/5950) covers structurally malformed inline values that fail deferred Evidence parsing, already return `invalid_request`, and lose their field paths. Here the records authenticate and later role checks select the wrong error family, so preserving parse-error paths alone does not fix this behavior. The matching service, reader, contract, registration, CLI adapter, and relevant integrity classification/recovery branches remain unchanged at fetched `origin/main` revision `25e1a24a3ed3671a0148757555f18d16c29d936a`. That later source was inspected, not executed.

bsakaguchi closed this issue

2026-10-31 00:42:34 +00:00

bsakaguchi commented

2026-10-31 00:42:34 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

Source authentication precedes explicit role validation in NativeMatchingSources. Wrong inventory/observation operations and exact side-context mismatches now produce AnalysisInputError with the rejected field/index; integrity errors remain for authentication, missing dependencies and contradictory producer payloads. Tests inspect wrong-role authentic Evidence, opposite-target observations, unsupported observation operations, field/index diagnostics and corrected selection.

Implementation: NativeMatchingSources.ts, NativeMatchingSources.ts.

The relevant regression checks in NativeFunctionMatchingService.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. Source authentication precedes explicit role validation in NativeMatchingSources. Wrong inventory/observation operations and exact side-context mismatches now produce AnalysisInputError with the rejected field/index; integrity errors remain for authentication, missing dependencies and contradictory producer payloads. Tests inspect wrong-role authentic Evidence, opposite-target observations, unsupported observation operations, field/index diagnostics and corrected selection. Implementation: [NativeMatchingSources.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/native/NativeMatchingSources.ts#L65), [NativeMatchingSources.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/native/NativeMatchingSources.ts#L111). The relevant regression checks in [NativeFunctionMatchingService.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/application/native/NativeFunctionMatchingService.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:09:58 +00:00

[PR #3634] [MERGED] feat(artifacts): inspect and extract Godot PCK with retained facts #5859

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3633

No description provided.

Rows
Columns