morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5829] [Bug] Attached IDA analysis omits UI effects from tool discovery #3526

New issue

Open

opened 2026-10-31 00:41:31 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-31 00:41:31 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5829

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Provider or capability discovery

Observed behavior

An executed in-memory MCP reproduction returned contradictory effect metadata for four successful attached IDA operations: procedure_pseudo_code, procedure_assembly, read_function_instructions, and analyze_function.

For each operation, binary_session included may_show_ui and may_access_network in the IDA provider capability group, while its tool availability group omitted changesUiState and accessesNetwork. The corresponding tools/list entry also returned openWorldHint: false.

The test used the production REA server and IDA adapter with the repository's recording MCP producer. It observed dispatch and returned metadata, without a real IDA engine, GUI, or external network request. Actual window movement was not tested.

Reproduction

From a checkout of the source revision below with locked dependencies installed, save this test as tests/composition/repro.test.ts, then run npm run test:local -- tests/composition/repro.test.ts. It uses existing repository fixtures, creates a synthetic ELF target and attached registration, and removes its target files afterward. The test asserts the current mismatch so the reproduction passes when it is present.

import { rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { parseConfig } from "../../src/config/parseConfig.js";
import { IdaProvider } from "../../src/ida/IdaProvider.js";
import { createServer } from "../../src/server/createServer.js";
import { createTestBinarySession } from "../fixtures/binarySession.js";
import { createIdaTarget, RecordingIdaMcp } from "../fixtures/idaMcp.js";
import { createTestMcp } from "../support/mcp/mcpFixture.js";

it("reproduces omitted attached IDA tool effects over MCP", async () => {
  const { root, target } = await createIdaTarget();
  const mcp = createTestMcp();
  let session: ReturnType<typeof createTestBinarySession> | undefined;
  try {
    const registration = join(root, "registration.json");
    await writeFile(registration, JSON.stringify({ command: "fixture", mode: "attached" }));
    const config = parseConfig({ REA_IDA_MCP_CONFIG: registration });
    if (!config.ok) throw config.error;
    const producer = new RecordingIdaMcp(target);
    session = createTestBinarySession(new IdaProvider(config.value, {}, () => producer));
    expect((await session.open(target.path)).ok).toBe(true);
    const server = createServer({ kind: "session", session }, { environment: { REA_LOG_LEVEL: "silent" } });
    const client = await mcp.connect(server);
    const tools = (await client.listTools()).tools;
    const names = ["procedure_pseudo_code", "procedure_assembly", "read_function_instructions", "analyze_function"];
    for (const name of names) {
      const response = await client.callTool({ name, arguments: { procedure: "main" } });
      expect(response.isError, name).not.toBe(true);
    }
    const response = await client.callTool({ name: "binary_session", arguments: {} });
    expect(response.isError).not.toBe(true);
    const status = (response.structuredContent as any).result;
    const rows = names.map(name => {
      const capabilitySet = status.capabilities.find((set: any) => set.provider.id === "ida");
      const capability = capabilitySet.groups.find((group: any) => group.operations.includes(name));
      const tool = status.tool_availability.groups.find((group: any) => group.names.includes(name));
      return {
        name,
        providerMayShowUi: capability.effects.includes("may_show_ui"),
        toolChangesUiState: tool.effects.includes("changesUiState"),
        providerMayAccessNetwork: capability.effects.includes("may_access_network"),
        toolAccessesNetwork: tool.effects.includes("accessesNetwork"),
        openWorldHint: tools.find(tool => tool.name === name)?.annotations?.openWorldHint,
      };
    });
    console.log(JSON.stringify(rows));
    console.log(JSON.stringify({ upstreamCalls: [...new Set(producer.calls.map(call => call.name))] }));
    expect(rows).toHaveLength(4);
    for (const row of rows) expect(row).toMatchObject({ providerMayShowUi: true, toolChangesUiState: false, providerMayAccessNetwork: true, toolAccessesNetwork: false, openWorldHint: false });
  } finally {
    await mcp.close();
    await session?.close();
    await rm(root, { recursive: true, force: true });
  }
});

I ran this test once; all four MCP analysis calls succeeded and all four metadata comparisons showed the omission. No real IDA installation is required for this metadata reproduction.

Expected behavior

The canonical tool metadata should include possible UI and network effects of its supported provider paths. A global tool audit may conservatively include effects from other profiles; it need not equal each selected profile's exact effects.

The contract requires truthful effect declarations in the tool design guide, and the MCP contract explicitly says effects cover possible behavior across supported inputs. The IDA guide documents the supported attached operations' GUI side effects. Their tool metadata must therefore include changesUiState, alongside the correctly advertised provider effect. The same tested records must not omit their supported network effect or derive a false openWorldHint from that omission.

Impact and scope

P2: clients relying on tool discovery receive an incomplete description of possible effects for supported analysis calls. The four tested operations returned successful results while understating their effect scope. This is consequential discovery behavior, rather than a demonstrated analytical-result failure.

The current global audit reproduces the omission without a particular real IDA build or operating system. The executed coverage is attached legacy-profile dispatch through a recording producer; native and headless profiles and real-client confirmation behavior were not exercised. Provider capability metadata is a verified alternative source of the omitted effect information. No data loss, target execution, or security boundary crossing was established; annotations are documented hints, not authorization controls.

REA version

rea-agents 6.7.0; source revision a5f982de8d

Runtime and platform

Node.js v26.12.0; macOS 27.0.1 arm64; locked MCP SDK over in-memory transports; no IDA engine or GUI used.

Target and artifact context

Synthetic ELF x86_64 executable from createIdaTarget, function main; RecordingIdaMcp attached producer fixture; no private target data.

Interface

MCP

Earliest failure stage

Transport or result projection

Relevant output or evidence

The executed test printed the following rows (formatting expanded for readability):

[
  {"name":"procedure_pseudo_code","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false},
  {"name":"procedure_assembly","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false},
  {"name":"read_function_instructions","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false},
  {"name":"analyze_function","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false}
]

The recording producer also reported:

{"upstreamCalls":["get_metadata","get_function_by_name","decompile_function","disassemble_function","get_callers","get_callees"]}

Trimmed runner summary:

Test Files  1 passed (1)
     Tests  1 passed (1)

The provider effect declaration is in IdaProviderCapabilities; BinarySessionStatus projects its enabled flags into the response. These are REA metadata observations, not proof that a real GUI moved or that network traffic occurred.

Typed outcome or recovery expected

For the same successful calls, binary_session tool availability should include changesUiState and the relevant network effect, and tools/list should carry the resulting truthful openWorldHint. The provider capability facts should remain available. Restoring these declarations should preserve successful analysis and cleanup.

Analysis or suspected cause

Source-derived: the Evidence-only default leaves UI and network flags false. Three affected primitive entries and the dossier entry use it. CapabilityInventory copies those contract effects rather than the provider descriptors, while annotation derivation computes openWorldHint from process/network flags. This explains the two different descriptions observed for the same calls.

Additional context

Related closed issue 1712 and merged PR 1764 corrected IDA provider descriptors. That PR changed the provider declaration, its conformance test, and documentation; it did not change the canonical tool audit or tool availability projection implicated here. The metadata omission remains present at the tested revision. Read-only issue and PR searches found no matching report for this tool-audit boundary.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5829 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Provider or capability discovery ### Observed behavior An executed in-memory MCP reproduction returned contradictory effect metadata for four successful attached IDA operations: `procedure_pseudo_code`, `procedure_assembly`, `read_function_instructions`, and `analyze_function`. For each operation, `binary_session` included `may_show_ui` and `may_access_network` in the IDA provider capability group, while its tool availability group omitted `changesUiState` and `accessesNetwork`. The corresponding `tools/list` entry also returned `openWorldHint: false`. The test used the production REA server and IDA adapter with the repository's recording MCP producer. It observed dispatch and returned metadata, without a real IDA engine, GUI, or external network request. Actual window movement was not tested. ### Reproduction From a checkout of the source revision below with locked dependencies installed, save this test as `tests/composition/repro.test.ts`, then run `npm run test:local -- tests/composition/repro.test.ts`. It uses existing repository fixtures, creates a synthetic ELF target and attached registration, and removes its target files afterward. The test asserts the current mismatch so the reproduction passes when it is present. ```typescript import { rm, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { expect, it } from "vitest"; import { parseConfig } from "../../src/config/parseConfig.js"; import { IdaProvider } from "../../src/ida/IdaProvider.js"; import { createServer } from "../../src/server/createServer.js"; import { createTestBinarySession } from "../fixtures/binarySession.js"; import { createIdaTarget, RecordingIdaMcp } from "../fixtures/idaMcp.js"; import { createTestMcp } from "../support/mcp/mcpFixture.js"; it("reproduces omitted attached IDA tool effects over MCP", async () => { const { root, target } = await createIdaTarget(); const mcp = createTestMcp(); let session: ReturnType<typeof createTestBinarySession> | undefined; try { const registration = join(root, "registration.json"); await writeFile(registration, JSON.stringify({ command: "fixture", mode: "attached" })); const config = parseConfig({ REA_IDA_MCP_CONFIG: registration }); if (!config.ok) throw config.error; const producer = new RecordingIdaMcp(target); session = createTestBinarySession(new IdaProvider(config.value, {}, () => producer)); expect((await session.open(target.path)).ok).toBe(true); const server = createServer({ kind: "session", session }, { environment: { REA_LOG_LEVEL: "silent" } }); const client = await mcp.connect(server); const tools = (await client.listTools()).tools; const names = ["procedure_pseudo_code", "procedure_assembly", "read_function_instructions", "analyze_function"]; for (const name of names) { const response = await client.callTool({ name, arguments: { procedure: "main" } }); expect(response.isError, name).not.toBe(true); } const response = await client.callTool({ name: "binary_session", arguments: {} }); expect(response.isError).not.toBe(true); const status = (response.structuredContent as any).result; const rows = names.map(name => { const capabilitySet = status.capabilities.find((set: any) => set.provider.id === "ida"); const capability = capabilitySet.groups.find((group: any) => group.operations.includes(name)); const tool = status.tool_availability.groups.find((group: any) => group.names.includes(name)); return { name, providerMayShowUi: capability.effects.includes("may_show_ui"), toolChangesUiState: tool.effects.includes("changesUiState"), providerMayAccessNetwork: capability.effects.includes("may_access_network"), toolAccessesNetwork: tool.effects.includes("accessesNetwork"), openWorldHint: tools.find(tool => tool.name === name)?.annotations?.openWorldHint, }; }); console.log(JSON.stringify(rows)); console.log(JSON.stringify({ upstreamCalls: [...new Set(producer.calls.map(call => call.name))] })); expect(rows).toHaveLength(4); for (const row of rows) expect(row).toMatchObject({ providerMayShowUi: true, toolChangesUiState: false, providerMayAccessNetwork: true, toolAccessesNetwork: false, openWorldHint: false }); } finally { await mcp.close(); await session?.close(); await rm(root, { recursive: true, force: true }); } }); ``` I ran this test once; all four MCP analysis calls succeeded and all four metadata comparisons showed the omission. No real IDA installation is required for this metadata reproduction. ### Expected behavior The canonical tool metadata should include possible UI and network effects of its supported provider paths. A global tool audit may conservatively include effects from other profiles; it need not equal each selected profile's exact effects. The contract requires truthful effect declarations in [the tool design guide](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/docs/tool-design.md#L65-L66), and [the MCP contract](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/docs/mcp-contracts.md#L741-L755) explicitly says effects cover possible behavior across supported inputs. [The IDA guide](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/docs/ida-provider.md#L83) documents the supported attached operations' GUI side effects. Their tool metadata must therefore include `changesUiState`, alongside the correctly advertised provider effect. The same tested records must not omit their supported network effect or derive a false `openWorldHint` from that omission. ### Impact and scope P2: clients relying on tool discovery receive an incomplete description of possible effects for supported analysis calls. The four tested operations returned successful results while understating their effect scope. This is consequential discovery behavior, rather than a demonstrated analytical-result failure. The current global audit reproduces the omission without a particular real IDA build or operating system. The executed coverage is attached legacy-profile dispatch through a recording producer; native and headless profiles and real-client confirmation behavior were not exercised. Provider capability metadata is a verified alternative source of the omitted effect information. No data loss, target execution, or security boundary crossing was established; annotations are documented hints, not authorization controls. ### REA version rea-agents 6.7.0; source revision a5f982de8dbe4b092cc20ab89d492b7e7028b758 ### Runtime and platform Node.js v26.12.0; macOS 27.0.1 arm64; locked MCP SDK over in-memory transports; no IDA engine or GUI used. ### Target and artifact context Synthetic ELF x86_64 executable from createIdaTarget, function main; RecordingIdaMcp attached producer fixture; no private target data. ### Interface MCP ### Earliest failure stage Transport or result projection ### Relevant output or evidence The executed test printed the following rows (formatting expanded for readability): ```json [ {"name":"procedure_pseudo_code","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false}, {"name":"procedure_assembly","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false}, {"name":"read_function_instructions","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false}, {"name":"analyze_function","providerMayShowUi":true,"toolChangesUiState":false,"providerMayAccessNetwork":true,"toolAccessesNetwork":false,"openWorldHint":false} ] ``` The recording producer also reported: ```json {"upstreamCalls":["get_metadata","get_function_by_name","decompile_function","disassemble_function","get_callers","get_callees"]} ``` Trimmed runner summary: ```text Test Files 1 passed (1) Tests 1 passed (1) ``` The provider effect declaration is in [IdaProviderCapabilities](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/ida/IdaProviderCapabilities.ts#L87-L99); [BinarySessionStatus](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/application/binary/BinarySessionStatus.ts#L341-L353) projects its enabled flags into the response. These are REA metadata observations, not proof that a real GUI moved or that network traffic occurred. ### Typed outcome or recovery expected For the same successful calls, `binary_session` tool availability should include `changesUiState` and the relevant network effect, and `tools/list` should carry the resulting truthful `openWorldHint`. The provider capability facts should remain available. Restoring these declarations should preserve successful analysis and cleanup. ### Analysis or suspected cause Source-derived: the [Evidence-only default](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/contracts/toolEffects.ts#L32-L46) leaves UI and network flags false. [Three affected primitive entries](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/contracts/toolEffects.ts#L508-L520) and [the dossier entry](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/contracts/toolEffects.ts#L555-L562) use it. [CapabilityInventory](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/application/CapabilityInventory.ts#L255-L270) copies those contract effects rather than the provider descriptors, while [annotation derivation](https://github.com/morluto/rea/blob/a5f982de8dbe4b092cc20ab89d492b7e7028b758/src/contracts/toolEffects.ts#L725-L743) computes `openWorldHint` from process/network flags. This explains the two different descriptions observed for the same calls. ### Additional context Related closed [issue 1712](https://github.com/morluto/rea/issues/1712) and merged [PR 1764](https://github.com/morluto/rea/pull/1764) corrected IDA provider descriptors. That PR changed the provider declaration, its conformance test, and documentation; it did not change the canonical tool audit or tool availability projection implicated here. The metadata omission remains present at the tested revision. Read-only issue and PR searches found no matching report for this tool-audit boundary.

bsakaguchi commented

2026-10-31 00:41:31 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 21, 2026):

Confirmed — for the four tested attached IDA operations, the provider capability group correctly declares may_show_ui / may_access_network, yet the tool-level effect array omits changesUiState and accessesNetwork and derives a false openWorldHint: false. Since discovery metadata is the caller's only pre-invocation signal for GUI/network side effects, propagating the provider-level effects into the tool records (or at least not contradicting them) would keep the tool design guide and MCP contract's "possible behavior across supported inputs" honest.

@XinwangTao commented on GitHub (Oct 21, 2026): Confirmed — for the four tested attached IDA operations, the provider capability group correctly declares may_show_ui / may_access_network, yet the tool-level effect array omits changesUiState and accessesNetwork and derives a false openWorldHint: false. Since discovery metadata is the caller's only pre-invocation signal for GUI/network side effects, propagating the provider-level effects into the tool records (or at least not contradicting them) would keep the tool design guide and MCP contract's "possible behavior across supported inputs" honest.

bsakaguchi commented

2026-10-31 00:41:31 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

There is progress on one discovery boundary at d87e73f: session inventory now merges matching provider effects and recomputes its annotations. The new regression asserts attached/headless UI, process and network effects plus open_world: true for selected direct/composed operations.

The remaining source-derived gap is SDK discovery. The four reported operations still use the static evidence audit entries, whose base enables session mutation only. Annotation derivation therefore still computes openWorldHint: false, and SDK registration forwards the contract annotations unchanged. Updating session inventory does not update that registration. I have not rerun the current server, so this is a source-derived mismatch, not a new wire reproduction or evidence of actual GUI/network activity.

For closure, compare tools/list and binary_session from the same recording-producer attached IDA fixture for all four operations: procedure_pseudo_code, procedure_assembly, read_function_instructions, and analyze_function. Inventory should retain the applicable UI/network effects, and SDK openWorldHint must include the supported network path. The current inventory test neither reads tools/list nor independently asserts all four operations. Add that boundary coverage while preserving provider-specific attached/headless/native distinctions; a global tool audit can conservatively include possible effects across supported profiles. A real IDA GUI is unnecessary for this metadata check.

@coygeek commented on GitHub (Oct 24, 2026): There is progress on one discovery boundary at `d87e73f`: session inventory now merges matching provider effects and recomputes its annotations. The [new regression](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/tests/boundary/providers/ida/idaCapabilityInventory.test.ts#L14-L37) asserts attached/headless UI, process and network effects plus `open_world: true` for selected direct/composed operations. The remaining source-derived gap is SDK discovery. The four reported operations still use the [static `evidence` audit entries](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/contracts/toolEffects.ts#L518-L577), whose base enables session mutation only. [Annotation derivation](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/contracts/toolEffects.ts#L742-L762) therefore still computes `openWorldHint: false`, and [SDK registration](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/server/toolRegistrationOptions.ts#L9-L23) forwards the contract annotations unchanged. Updating session inventory does not update that registration. I have not rerun the current server, so this is a source-derived mismatch, not a new wire reproduction or evidence of actual GUI/network activity. For closure, compare `tools/list` and `binary_session` from the same recording-producer attached IDA fixture for all four operations: `procedure_pseudo_code`, `procedure_assembly`, `read_function_instructions`, and `analyze_function`. Inventory should retain the applicable UI/network effects, and SDK `openWorldHint` must include the supported network path. The current inventory test neither reads `tools/list` nor independently asserts all four operations. Add that boundary coverage while preserving provider-specific attached/headless/native distinctions; a global tool audit can conservatively include possible effects across supported profiles. A real IDA GUI is unnecessary for this metadata check.

bsakaguchi referenced this issue

2026-10-31 01:09:33 +00:00

[PR #3526] [MERGED] fix(reconstruction): withhold readiness for manifest references to absent Evidence #5803

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3526

No description provided.

Rows
Columns