[GH-ISSUE #2615] fix(evaluation): missing server identity counts as REA tool use #932

New issue

Closed

opened 2026-10-15 18:29:41 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:29:41 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2615

Summary

When the Codex transcript evaluator receives a completed MCP call without either server-name field, it counts the call as REA use. If the call names the expected first tool, the routing score also passes despite unresolved server identity. This failure is established from source; runtime reproduction and a live model trial were not run. It can affect the evaluator's routing and workflow measurements independently of its stricter delivered-Evidence checks.

Runtime or environment

Failure verified at a2d8b94dd5 and still present at 8b181716ac. Node v26.10.0 and npm 11.19.1 are installed, and locked dependencies are available. The target is the source evaluator, not an installed REA runtime.

Steps to reproduce

In the stated source checkout, add a focused evaluator test at tests/evaluation/model-evals/unidentifiedServer.test.ts with the fixture below. These execution steps have not been run by this verifier.
Run npm run test:focused -- tests/evaluation/model-evals/unidentifiedServer.test.ts.
The first assertion should fail because the unidentified call receives the same REA routing credit as the explicit REA control. The foreign-server control should remain excluded.

import { expect, it } from "vitest";
import { evaluateCodexEvents } from "../../../src/evaluation/CodexAgentEval.js";

it("preserves unknown server identity in routing scores", () => {
  const tool = "analyze_javascript_application";
  const evaluate = (identity: Record<string, string>) =>
    evaluateCodexEvents(
      [{
        type: "item.completed",
        item: {
          type: "mcp_tool_call",
          status: "completed",
          tool,
          arguments: {},
          ...identity,
        },
      }],
      tool,
      { requiredToolSubsequence: [tool] },
    );

  expect(evaluate({})).toMatchObject({
    naturalUse: false,
    firstTool: null,
    correctFirstTool: false,
    requiredToolSubsequenceMet: false,
  });
  expect(evaluate({ server: "rea" })).toMatchObject({
    naturalUse: true,
    firstTool: tool,
    correctFirstTool: true,
    requiredToolSubsequenceMet: true,
  });
  expect(evaluate({ server: "other" }).naturalUse).toBe(false);
});

Expected behavior

A completed call that omits both server and server_name has unresolved server identity. It should not establish REA use, an expected first REA tool, or a required REA tool sequence. The supplied tool name alone does not identify which MCP server handled the call. Explicit REA identity should continue to count.

Actual behavior

Source inspection establishes that the unidentified call is included in the REA call list. For the fixture, the derived metrics are naturalUse: true, firstTool: "analyze_javascript_application", correctFirstTool: true, and requiredToolSubsequenceMet: true. Its repeatedCallCount is zero and its authenticated Evidence list is empty. These are source-derived results, not captured test output.

The one-call fixture does not establish a complete release-gate pass because it has no final answer or token usage. However, scenarios without required Evidence or configured fixture facts can accept this routing credit when their separate execution, token and answer checks also pass. The existing browser scenario is one such case: a server-less call naming its expected tool, an answer of at least 80 trimmed characters containing unavailable, which satisfies both its term requirement and the separate epistemic-cue check, positive input usage and a successful exit satisfy the current gate by source inspection. Scenarios requiring authenticated Evidence or configured facts retain those independent requirements.

Evidence

Expected source: AGENTS.md:21, under Architecture and Cleanup, requires unresolved identity or applicability to remain unknown rather than be treated as absent, foreign, or aligned.
Failure source: src/evaluation/CodexAgentEval.ts:116 accepts the completed item and nonempty tool, assigns null when both server aliases are absent, and inserts the call into the initially empty deduplication set; src/evaluation/CodexAgentEval.ts:221 includes that null-server call in REA routing before returning the affected metrics.
Evidence provenance: static
Local verification: static-proof
Reproduction completeness: complete

The production evaluator parses JSON stdout lines without a server-identity guard, forwards those events unchanged to the exported evaluator, and passes its metrics into the gate. This establishes reachability for this transcript shape; it does not establish how often a live Codex version emits it. The Evidence parser rejects missing server identity before authenticating a delivered record, but that rejection does not remove the call from routing. No earlier guard authenticates REA identity using the tool name.

The release-gate consequence follows from scripts/verify-agent-experience.mjs:216 for the browser scenario and src/evaluation/AgentEvaluationReport.ts:23 for the independent gate conditions. Runtime execution was not run. Explicit REA identity remains included by the same filter; an explicit foreign server is excluded, so these controls distinguish identity preservation from disabling routing scoring.

Related report: https://github.com/morluto/rea/issues/1728 covers diagnostic strings counting as produced Evidence. Its merged Evidence-authentication fix leaves the unknown-server routing filter intact, so this report has a separate failure and restoration check.

Restoration check

With the same unidentified event and non-null expected tool, REA use and first-tool credit must be false, the first REA tool must be null, and the configured one-tool subsequence must be unmet. An otherwise identical event with server: "rea", case-insensitive REA identity, or server_name: "rea" must retain the appropriate REA credit. An explicit foreign-server call must remain excluded. Preserve the independent Evidence and factual checks while restoring routing identity.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: The evaluator misattributes completed calls with omitted identity and can incorrectly satisfy routing and sequence requirements. The failure is bounded to this transcript shape. No live incorrect release decision or production analysis failure was observed.
Workaround status: unverified
Workaround basis: The evaluator has no caller option to exclude unidentified calls. Manual transcript review might identify affected runs, but no practical complete workaround was tested.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2615 ## Summary When the Codex transcript evaluator receives a completed MCP call without either server-name field, it counts the call as REA use. If the call names the expected first tool, the routing score also passes despite unresolved server identity. This failure is established from source; runtime reproduction and a live model trial were not run. It can affect the evaluator's routing and workflow measurements independently of its stricter delivered-Evidence checks. ## Runtime or environment Failure verified at a2d8b94dd50dab8d966239dd63ab84718aa8b4b4 and still present at 8b181716aca2cb467be499cba226abfb5ae19141. Node v26.10.0 and npm 11.19.1 are installed, and locked dependencies are available. The target is the source evaluator, not an installed REA runtime. ## Steps to reproduce 1. In the stated source checkout, add a focused evaluator test at `tests/evaluation/model-evals/unidentifiedServer.test.ts` with the fixture below. These execution steps have not been run by this verifier. 2. Run `npm run test:focused -- tests/evaluation/model-evals/unidentifiedServer.test.ts`. 3. The first assertion should fail because the unidentified call receives the same REA routing credit as the explicit REA control. The foreign-server control should remain excluded. ```ts import { expect, it } from "vitest"; import { evaluateCodexEvents } from "../../../src/evaluation/CodexAgentEval.js"; it("preserves unknown server identity in routing scores", () => { const tool = "analyze_javascript_application"; const evaluate = (identity: Record<string, string>) => evaluateCodexEvents( [{ type: "item.completed", item: { type: "mcp_tool_call", status: "completed", tool, arguments: {}, ...identity, }, }], tool, { requiredToolSubsequence: [tool] }, ); expect(evaluate({})).toMatchObject({ naturalUse: false, firstTool: null, correctFirstTool: false, requiredToolSubsequenceMet: false, }); expect(evaluate({ server: "rea" })).toMatchObject({ naturalUse: true, firstTool: tool, correctFirstTool: true, requiredToolSubsequenceMet: true, }); expect(evaluate({ server: "other" }).naturalUse).toBe(false); }); ``` ## Expected behavior A completed call that omits both `server` and `server_name` has unresolved server identity. It should not establish REA use, an expected first REA tool, or a required REA tool sequence. The supplied tool name alone does not identify which MCP server handled the call. Explicit REA identity should continue to count. ## Actual behavior Source inspection establishes that the unidentified call is included in the REA call list. For the fixture, the derived metrics are `naturalUse: true`, `firstTool: "analyze_javascript_application"`, `correctFirstTool: true`, and `requiredToolSubsequenceMet: true`. Its `repeatedCallCount` is zero and its authenticated Evidence list is empty. These are source-derived results, not captured test output. The one-call fixture does not establish a complete release-gate pass because it has no final answer or token usage. However, scenarios without required Evidence or configured fixture facts can accept this routing credit when their separate execution, token and answer checks also pass. The existing browser scenario is one such case: a server-less call naming its expected tool, an answer of at least 80 trimmed characters containing `unavailable`, which satisfies both its term requirement and the separate epistemic-cue check, positive input usage and a successful exit satisfy the current gate by source inspection. Scenarios requiring authenticated Evidence or configured facts retain those independent requirements. ## Evidence - Expected source: `AGENTS.md:21`, under Architecture and Cleanup, requires unresolved identity or applicability to remain unknown rather than be treated as absent, foreign, or aligned. - Failure source: `src/evaluation/CodexAgentEval.ts:116` accepts the completed item and nonempty tool, assigns null when both server aliases are absent, and inserts the call into the initially empty deduplication set; `src/evaluation/CodexAgentEval.ts:221` includes that null-server call in REA routing before returning the affected metrics. - Evidence provenance: static - Local verification: static-proof - Reproduction completeness: complete The production evaluator parses JSON stdout lines without a server-identity guard, forwards those events unchanged to the exported evaluator, and passes its metrics into the gate. This establishes reachability for this transcript shape; it does not establish how often a live Codex version emits it. The Evidence parser rejects missing server identity before authenticating a delivered record, but that rejection does not remove the call from routing. No earlier guard authenticates REA identity using the tool name. The release-gate consequence follows from `scripts/verify-agent-experience.mjs:216` for the browser scenario and `src/evaluation/AgentEvaluationReport.ts:23` for the independent gate conditions. Runtime execution was not run. Explicit REA identity remains included by the same filter; an explicit foreign server is excluded, so these controls distinguish identity preservation from disabling routing scoring. Related report: https://github.com/morluto/rea/issues/1728 covers diagnostic strings counting as produced Evidence. Its merged Evidence-authentication fix leaves the unknown-server routing filter intact, so this report has a separate failure and restoration check. ## Restoration check With the same unidentified event and non-null expected tool, REA use and first-tool credit must be false, the first REA tool must be null, and the configured one-tool subsequence must be unmet. An otherwise identical event with `server: "rea"`, case-insensitive REA identity, or `server_name: "rea"` must retain the appropriate REA credit. An explicit foreign-server call must remain excluded. Preserve the independent Evidence and factual checks while restoring routing identity. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: The evaluator misattributes completed calls with omitted identity and can incorrectly satisfy routing and sequence requirements. The failure is bounded to this transcript shape. No live incorrect release decision or production analysis failure was observed. - Workaround status: unverified - Workaround basis: The evaluator has no caller option to exclude unidentified calls. Manual transcript review might identify affected runs, but no practical complete workaround was tested.

bsakaguchi

2026-10-15 18:29:41 +00:00

closed this issue
added the
area: server

bug
labels

bsakaguchi commented

2026-10-15 18:29:42 +00:00

Author

Owner

@morluto commented on GitHub (Oct 14, 2026):

Addressed on main in 443c7aff77c862d19fca296149fce28091ab0592.

Evaluation attribution requires an explicit REA server identity. Missing or unrelated identity earns no REA tool-use credit; the explicit alias remains accepted.

Validation: Affected suites passed in the merged 65-file run (1169 passed, 2 explicit skips), with full static/architecture checks and documentation checks passing. Final installed-package verification is recorded separately in the audit.

Verification scope: Controlled source and production-boundary verification on macOS with pinned Node 24.18.0; real external engines and Windows hosts were not independently rerun. Closure covers the described source defect, not broader provider conformance or performance.

@morluto commented on GitHub (Oct 14, 2026): Addressed on main in [443c7aff77c862d19fca296149fce28091ab0592](https://github.com/morluto/rea/commit/443c7aff77c862d19fca296149fce28091ab0592). Evaluation attribution requires an explicit REA server identity. Missing or unrelated identity earns no REA tool-use credit; the explicit alias remains accepted. Validation: Affected suites passed in the merged 65-file run (1169 passed, 2 explicit skips), with full static/architecture checks and documentation checks passing. Final installed-package verification is recorded separately in the audit. Verification scope: Controlled source and production-boundary verification on macOS with pinned Node 24.18.0; real external engines and Windows hosts were not independently rerun. Closure covers the described source defect, not broader provider conformance or performance.

bsakaguchi referenced this issue

2026-10-16 21:02:34 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:06:58 +00:00

[PR #932] [MERGED] perf(javascript): analyze member chains without recursive copying #2509

bsakaguchi referenced this issue

2026-10-17 21:07:04 +00:00

[PR #939] [MERGED] fix(javascript): handle deep static member paths #2516