morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5932] [Bug] compare_functions misclassifies authentic wrong-kind Evidence as an integrity failure #3608

New issue

Closed

opened 2026-10-31 00:42:20 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:20 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5932

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

MCP or agent integration

Observed behavior

Calling compare_functions with authentic Evidence from another operation returns evidence_integrity_mismatch, category: integrity_mismatch, and instructions to recreate or re-import the record. The record passes parseEvidence; it is simply the wrong selection for function comparison.

I reproduced this through the registered MCP server with the wrong record on each side. Both diagnostics contain only details.reason: "Expected analyze_function Evidence": neither identifies the rejected side or actual operation/predicate, and neither contains input issues or input-contract recovery. Replacing the wrong record with function Evidence succeeds.

Reproduction

With dependencies installed at the source revision below, save this focused test as tests/boundary/mcp/functionInputKindMcp.test.ts and run:

./node_modules/.bin/vitest run tests/boundary/mcp/functionInputKindMcp.test.ts --maxWorkers 1

The executed test used the repository's binary-session fixture and a real SDK client connected to the registered server by InMemoryTransport. The fixture throws if an analysis provider starts, and the test checks that none starts. The non-function record is synthetic authenticated Evidence; no real inventory provider output or native analysis engine was exercised.

import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { expect, it } from "vitest";
import { createTestBinarySession } from "../../fixtures/binarySession.js";
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComparisonExample.js";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
import { createServer } from "../../../src/server/createServer.js";

it("reports valid wrong-kind Evidence as an integrity mismatch on either side", async () => {
  let providerLaunches = 0;
  const session = createTestBinarySession(() => {
    providerLaunches++;
    throw new Error("No provider should launch");
  });
  const server = createServer({ kind: "session", session });
  const client = new Client({ name: "comparison-input-test", version: "1" });
  const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
  const good = FUNCTION_COMPARISON_EXAMPLE.left;
  const wrong = createEvidence(undefined, good.provider, {
    operation: "inventory_artifact", predicateType: "rea.artifact-inventory",
    parameters: {}, result: {},
  });
  expect(parseEvidence(wrong)).toEqual(wrong);
  try {
    await server.connect(serverTransport);
    await client.connect(clientTransport);
    for (const side of ["left", "right"] as const) {
      const result = await client.callTool({
        name: "compare_functions",
        arguments: { left: side === "left" ? wrong : good,
          right: side === "right" ? wrong : good },
      });
      const diagnostic = parseMcpToolError(result);
      console.log(JSON.stringify({ side, diagnostic }));
      expect(diagnostic.error).toMatchObject({
        code: "evidence_integrity_mismatch", category: "integrity_mismatch",
        details: { reason: "Expected analyze_function Evidence" },
        message: "Evidence is invalid or has changed. Recreate or re-import it, then try again.",
      });
      expect(diagnostic.error.details).not.toHaveProperty("issues");
    }
    const corrected = await client.callTool({
      name: "compare_functions", arguments: { left: good, right: good },
    });
    expect(corrected.isError).not.toBe(true);
    expect(parseEvidence(corrected.structuredContent).normalized_result).toMatchObject({ status: "unchanged" });
    expect(providerLaunches).toBe(0);
    await client.ping();
  } finally {
    await client.close();
    await server.close();
    await session.close();
  }
});

One selected test passed. It checks the currently incorrect diagnostic on both sides and a successful corrected-input control; it is a reproduction test rather than an assertion that this classification is desirable.

Expected behavior

Reject authentic Evidence of the wrong operation/predicate as an input-selection error, identify left or right, and explain the expected analyze_function operation and rea.analysis predicate alongside the supplied values. Preserve integrity errors for records that fail authentication.

The function-comparison contract requires function Evidence. The process-comparison boundary handles the analogous wrong-workflow case as AnalysisInputError, and its MCP regression requires invalid_request with the failing side.

Impact and scope

P2: callers that branch on the diagnostic family receive integrity-recovery advice for a correctable selection error. The observed failure is bounded to wrong-kind input; selecting function Evidence succeeds. Both left and right inline-input cases were exercised. No corruption, accepted invalid comparison, target execution, or actual automated retry loop was observed. Retained-reference and predicate-only cases were not separately executed.

REA version

rea-agents 6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3. This run differs from the repository's Node 24.18.0 pin; behavior on the pinned runtime was not separately tested.

Target and artifact context

Synthetic authenticated inventory_artifact Evidence with predicate rea.artifact-inventory, paired with the repository's canonical function Evidence example. No live binary target is required.

Interface

MCP

Earliest failure stage

Request parsing or validation

Relevant output or evidence

Both wrong-side calls returned this canonical diagnostic in MCP text content with isError: true:

{
  "error": {
    "code": "evidence_integrity_mismatch",
    "category": "integrity_mismatch",
    "message": "Evidence is invalid or has changed. Recreate or re-import it, then try again.",
    "retryable": false,
    "remediation": {
      "action": "Evidence is invalid or has changed. Recreate or re-import it, then try again."
    },
    "details": {
      "reason": "Expected analyze_function Evidence"
    }
  }
}

The corrected-input control returned successful comparison Evidence with normalized_result.status: "unchanged"; provider launches remained zero. The client remained responsive to ping.

Source corroboration: the operation/predicate gate runs after Evidence authentication and creates EvidenceIntegrityError. Its default message supplies the recreation/re-import advice.

Typed outcome or recovery expected

invalid_request / invalid_input, details.issues naming the rejected side with reason: invalid_value, and recovery.kind: inspect_input_contract for compare_functions. The recovery should direct callers to select function Evidence. The existing diagnostic's details.reason gives a useful expected-operation hint, but its error family and recovery action still describe an integrity failure.

Analysis or suspected cause

Source-derived: the combined operation/predicate check uses the same error class as the preceding authentication-failure branch. That class represents Evidence integrity checks; the wrong-workflow selection passes authentication. Reporting this check through input-error handling would preserve the distinction already made by process comparison.

Additional context

The relevant gate and error-classification branches are unchanged at fetched origin/main revision 33f912919d5884d1019eb7f17b202db1b77296a9. That later revision was inspected, not separately executed. No dedicated function-comparison CLI command was tested.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5932 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area MCP or agent integration ### Observed behavior Calling `compare_functions` with authentic Evidence from another operation returns `evidence_integrity_mismatch`, `category: integrity_mismatch`, and instructions to recreate or re-import the record. The record passes `parseEvidence`; it is simply the wrong selection for function comparison. I reproduced this through the registered MCP server with the wrong record on each side. Both diagnostics contain only `details.reason: "Expected analyze_function Evidence"`: neither identifies the rejected side or actual operation/predicate, and neither contains input issues or input-contract recovery. Replacing the wrong record with function Evidence succeeds. ### Reproduction With dependencies installed at the source revision below, save this focused test as `tests/boundary/mcp/functionInputKindMcp.test.ts` and run: ```sh ./node_modules/.bin/vitest run tests/boundary/mcp/functionInputKindMcp.test.ts --maxWorkers 1 ``` The executed test used the repository's binary-session fixture and a real SDK client connected to the registered server by `InMemoryTransport`. The fixture throws if an analysis provider starts, and the test checks that none starts. The non-function record is synthetic authenticated Evidence; no real inventory provider output or native analysis engine was exercised. ```ts import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { expect, it } from "vitest"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComparisonExample.js"; import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { createServer } from "../../../src/server/createServer.js"; it("reports valid wrong-kind Evidence as an integrity mismatch on either side", async () => { let providerLaunches = 0; const session = createTestBinarySession(() => { providerLaunches++; throw new Error("No provider should launch"); }); const server = createServer({ kind: "session", session }); const client = new Client({ name: "comparison-input-test", version: "1" }); const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); const good = FUNCTION_COMPARISON_EXAMPLE.left; const wrong = createEvidence(undefined, good.provider, { operation: "inventory_artifact", predicateType: "rea.artifact-inventory", parameters: {}, result: {}, }); expect(parseEvidence(wrong)).toEqual(wrong); try { await server.connect(serverTransport); await client.connect(clientTransport); for (const side of ["left", "right"] as const) { const result = await client.callTool({ name: "compare_functions", arguments: { left: side === "left" ? wrong : good, right: side === "right" ? wrong : good }, }); const diagnostic = parseMcpToolError(result); console.log(JSON.stringify({ side, diagnostic })); expect(diagnostic.error).toMatchObject({ code: "evidence_integrity_mismatch", category: "integrity_mismatch", details: { reason: "Expected analyze_function Evidence" }, message: "Evidence is invalid or has changed. Recreate or re-import it, then try again.", }); expect(diagnostic.error.details).not.toHaveProperty("issues"); } const corrected = await client.callTool({ name: "compare_functions", arguments: { left: good, right: good }, }); expect(corrected.isError).not.toBe(true); expect(parseEvidence(corrected.structuredContent).normalized_result).toMatchObject({ status: "unchanged" }); expect(providerLaunches).toBe(0); await client.ping(); } finally { await client.close(); await server.close(); await session.close(); } }); ``` One selected test passed. It checks the currently incorrect diagnostic on both sides and a successful corrected-input control; it is a reproduction test rather than an assertion that this classification is desirable. ### Expected behavior Reject authentic Evidence of the wrong operation/predicate as an input-selection error, identify `left` or `right`, and explain the expected `analyze_function` operation and `rea.analysis` predicate alongside the supplied values. Preserve integrity errors for records that fail authentication. The [function-comparison contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/contracts/sessionToolContracts.ts#L116-L120) requires function Evidence. The [process-comparison boundary](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/server/registerProcessComparisonTool.ts#L67-L78) handles the analogous wrong-workflow case as `AnalysisInputError`, and its [MCP regression](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts#L448-L475) requires `invalid_request` with the failing side. ### Impact and scope P2: callers that branch on the diagnostic family receive integrity-recovery advice for a correctable selection error. The observed failure is bounded to wrong-kind input; selecting function Evidence succeeds. Both left and right inline-input cases were exercised. No corruption, accepted invalid comparison, target execution, or actual automated retry loop was observed. Retained-reference and predicate-only cases were not separately executed. ### REA version `rea-agents` 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v26.12.0; Vitest 5.0.3. This run differs from the repository's Node 24.18.0 pin; behavior on the pinned runtime was not separately tested. ### Target and artifact context Synthetic authenticated `inventory_artifact` Evidence with predicate `rea.artifact-inventory`, paired with the repository's canonical function Evidence example. No live binary target is required. ### Interface MCP ### Earliest failure stage Request parsing or validation ### Relevant output or evidence Both wrong-side calls returned this canonical diagnostic in MCP text content with `isError: true`: ```json { "error": { "code": "evidence_integrity_mismatch", "category": "integrity_mismatch", "message": "Evidence is invalid or has changed. Recreate or re-import it, then try again.", "retryable": false, "remediation": { "action": "Evidence is invalid or has changed. Recreate or re-import it, then try again." }, "details": { "reason": "Expected analyze_function Evidence" } } } ``` The corrected-input control returned successful comparison Evidence with `normalized_result.status: "unchanged"`; provider launches remained zero. The client remained responsive to `ping`. Source corroboration: [the operation/predicate gate](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/server/registerFunctionComparisonTool.ts#L42-L77) runs after Evidence authentication and creates `EvidenceIntegrityError`. Its [default message](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisErrorPresentation.ts#L269-L274) supplies the recreation/re-import advice. ### Typed outcome or recovery expected `invalid_request` / `invalid_input`, `details.issues` naming the rejected side with `reason: invalid_value`, and `recovery.kind: inspect_input_contract` for `compare_functions`. The recovery should direct callers to select function Evidence. The existing diagnostic's `details.reason` gives a useful expected-operation hint, but its error family and recovery action still describe an integrity failure. ### Analysis or suspected cause Source-derived: the combined operation/predicate check uses the same error class as the preceding authentication-failure branch. That class [represents Evidence integrity checks](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/evidenceErrors.ts#L13-L25); the wrong-workflow selection passes authentication. Reporting this check through input-error handling would preserve the distinction already made by process comparison. ### Additional context The relevant gate and error-classification branches are unchanged at fetched `origin/main` revision `33f912919d5884d1019eb7f17b202db1b77296a9`. That later revision was inspected, not separately executed. No dedicated function-comparison CLI command was tested.

bsakaguchi closed this issue

2026-10-31 00:42:20 +00:00

bsakaguchi commented

2026-10-31 00:42:21 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

The original misclassification is addressed in source at d87e73f. compare_functions now passes the rejected left/right path into the shared parser. That parser authenticates first, then returns AnalysisInputError for the wrong operation/predicate, so authentic wrong-kind input follows input-error handling rather than the old integrity branch. The error projection also supplies inspect_input_contract recovery.

The narrower remaining requirement is to report the supplied operation and predicate alongside the expected analyze_function / rea.analysis pair. The producer currently constructs only a fixed expected-values message and an issue path; it does not attach either actual value. The existing wrong-operation MCP test checks only isError and retained-record count, so it does not establish the requested diagnostic details. These are source/test-assertion observations, not a current runtime reproduction.

For closure, a focused registered-MCP regression should check both sides, independent operation and predicate mismatches, and inline/retained inputs: invalid_request / invalid_input, the failing side, expected and supplied values, and input-contract recovery. Keep a valid-input success control and a tampered-record integrity control, with no provider launch. If expected-only guidance is intentional, explicitly resolve that narrower diagnostic contract rather than treating the original supplied-values requirement as implemented.

@coygeek commented on GitHub (Oct 24, 2026): The original misclassification is addressed in source at `d87e73f`. `compare_functions` now [passes the rejected left/right path into the shared parser](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/server/registerFunctionComparisonTool.ts#L38-L57). That parser [authenticates first, then returns AnalysisInputError for the wrong operation/predicate](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/functionDossierEvidence.ts#L29-L44), so authentic wrong-kind input follows input-error handling rather than the old integrity branch. The error projection also supplies [inspect_input_contract recovery](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/analysisErrorProjection.ts#L155-L162). The narrower remaining requirement is to report the supplied operation and predicate alongside the expected `analyze_function` / `rea.analysis` pair. The producer currently constructs only a fixed expected-values message and an issue path; it does not attach either actual value. The [existing wrong-operation MCP test](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/mcp/functionComparisonMcp.test.ts#L357-L392) checks only `isError` and retained-record count, so it does not establish the requested diagnostic details. These are source/test-assertion observations, not a current runtime reproduction. For closure, a focused registered-MCP regression should check both sides, independent operation and predicate mismatches, and inline/retained inputs: `invalid_request` / `invalid_input`, the failing side, expected and supplied values, and input-contract recovery. Keep a valid-input success control and a tampered-record integrity control, with no provider launch. If expected-only guidance is intentional, explicitly resolve that narrower diagnostic contract rather than treating the original supplied-values requirement as implemented.

bsakaguchi referenced this issue

2026-10-31 00:48:11 +00:00

[GH-ISSUE #7038] [Feature] Read Hermes bytecode bundles: header, integrity, function and string tables #4258

bsakaguchi referenced this issue

2026-10-31 00:48:12 +00:00

[GH-ISSUE #7038] [Feature] Read Hermes bytecode bundles: header, integrity, function and string tables #4258