[GH-ISSUE #5943] [Bug] Electron scenarios omit unexecuted actions after a failed action #3613

New issue

Closed

opened 2026-10-31 00:42:23 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:23 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5943

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Process capture or runtime observation

Observed behavior

capture_electron_scenario drops the remaining action outcomes after a click fails. In an executed MCP test requesting fail followed by settle, the returned Evidence retains both requested actions in parameters.actions, but normalized_result.actions contains only the failed fail action. The later wait is never called and has no cancelled/not-executed record or omission explanation. The MCP response is successful; the first action's failure is correctly visible.

This reproduction uses a simulated Electron application and process supervision. It executes REA's registered MCP handler, capture service, actual provider/action runner, snapshot parsing, projection and Evidence delivery. No real Electron runtime was launched or signaled.

Reproduction

At the source revision below with its locked dependencies installed, save the following as src/browser/electronSkippedActionsRepro.test.ts:

import { afterEach, expect, it, vi } from "vitest";
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { PlaywrightElectronActiveProvider } from "./PlaywrightElectronActiveProvider.js";
import { createServer } from "../server/createServer.js";
import { createTestBinarySession } from "../../tests/fixtures/binarySession.js";
import { observed } from "../../tests/fixtures/analysisExecution.js";
import { ok } from "../domain/result.js";

vi.mock("../process/ProcessOwnershipObservation.js", () => ({
  prepareProcessOwnershipInspection: async () => undefined,
  systemProcessOwnershipHost: {},
  observeOwnedProcessLineage: async (owner: any) => ({
    status: "verified", observedAt: new Date().toISOString(),
    lineage: { runId: owner.runId, launcherPid: owner.leaderPid,
      launcherParentPid: process.pid, processGroupId: owner.leaderPid, descendants: [] },
  }),
}));
vi.mock("./ElectronProcessCleanup.js", () => ({
  cleanupElectronProcesses: async () => ({ cleaned: true, signaled: false }),
}));
afterEach(() => vi.restoreAllMocks());

it("returns successful MCP Evidence with an omitted post-failure action", async () => {
  const click = vi.fn(async () => { throw new Error("simulated strict mode violation"); });
  const wait = vi.fn(async () => undefined);
  const page = { locator: () => ({ click }), waitForTimeout: wait };
  const snapshot = {
    events: [], retention_budget_bytes: 0, estimated_retained_bytes: 0,
    event_serialized_byte_upper_bound: 0, retained: 0, dropped: 0,
    dropped_ipc: 0, dropped_runtime: 0, dropped_event_families: [],
    dropped_event_roles: [], observed: 0, observed_ipc: 0,
    observed_runtime: 0, hook_error: false,
  };
  let read = 0;
  const close = vi.fn(async () => undefined);
  const app = {
    process: () => ({ pid: 424242 }), close,
    windows: () => [page], firstWindow: async () => page,
    evaluate: async () => [[], [], "simulated-electron", snapshot][read++ % 4],
  };
  const provider = new PlaywrightElectronActiveProvider(
    {}, async () => app as any, async () => ({ slices: [] }) as any,
  );
  const session = createTestBinarySession(() => ({
    execute: async () => observed(null), close: async () => ok(null),
  }));
  const server = createServer({ kind: "session", session }, { electronActiveObservation: provider });
  const client = new Client({ name: "electron-action-reproduction", version: "1" });
  const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
  try {
    await server.connect(serverTransport);
    await client.connect(clientTransport);
    const response = await client.callTool({ name: "capture_electron_scenario", arguments: {
      executable_path: process.execPath, application_path: process.execPath,
      actions: [
        { step_id: "fail", kind: "click", selector: "#duplicate" },
        { step_id: "settle", kind: "wait", duration_ms: 0 },
      ],
    } });
    expect(response.isError, JSON.stringify(response)).not.toBe(true);
    const evidence = response.structuredContent as any;
    const result = evidence.normalized_result;
    expect(click).toHaveBeenCalledOnce();
    expect(wait).not.toHaveBeenCalled();
    expect(result.actions.map((action: any) => [action.step_id, action.status])).toEqual([["fail", "failed"]]);
    expect(result.coverage.status).toBe("partial_attach");
    expect(result.limitations.some((text: string) => /action|step|skipp/i.test(text))).toBe(false);
    expect(evidence.parameters.actions).toHaveLength(2);
    expect(close).toHaveBeenCalledOnce();
    console.log(JSON.stringify({
      isError: response.isError ?? false,
      declaredActions: evidence.parameters.actions.map((action: any) => action.step_id),
      returnedActions: result.actions.map((action: any) => ({ step_id: action.step_id, status: action.status, error: action.error })),
      waitCalls: wait.mock.calls.length,
      coverage: result.coverage.status,
      actionLimitations: result.limitations.filter((text: string) => /action|step|skipp/i.test(text)),
      cleanup: result.application.cleanup,
    }));
  } finally {
    await client.close(); await server.close(); await session.close(); await provider.close();
  }
});

Run only that test:

node node_modules/vitest/vitest.mjs run src/browser/electronSkippedActionsRepro.test.ts --maxWorkers 1

The focused run passed its assertions of the current incorrect result. Delete the temporary test afterward. The injected click rejection is deterministic; this does not establish a real Playwright selector failure.

Expected behavior

Represent every requested action's disposition, including a clear not-executed/cancelled outcome for later actions after an earlier failure, or explicitly report the omitted action IDs/count and reason. Continuing the remaining actions is not required, and a capture may still retain useful observations after an action fails.

Electron documentation says the result records action status and targets. Tool design guidance calls for preserving per-item outcomes and meaningful action choices, with coverage and limitations. The separate browser scenario implementation already records later actions as cancelled; that is a possible existing representation, rather than an Electron-specific contract guarantee.

Impact and scope

P2: action-result consumers cannot account for the entire declared scenario from normalized_result.actions alone. The suffix disappears on the first ordinary action rejection, even though the request itself returns Evidence. A practical workaround is to compare parameters.actions with the returned action records and treat missing steps as unexecuted. The original failure remains visible, so this is not a claim that the failed action is reported as completed.

One deterministic simulated application was tested. Real Electron behavior and natural failure frequency were not measured. CLI reachability follows the shared service wiring; the reproduction exercised MCP only.

REA version

6.7.0 source checkout at 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3. The exact repository-pinned Node 24.18.0 runtime was not used. Electron application and process supervision were simulated.

Target and artifact context

Two-action owned Electron scenario: a simulated rejected click followed by a zero-duration wait. The target boundary is simulated; paths are supplied dynamically from the running Node executable solely to satisfy the real service's path admission.

Interface

MCP

Earliest failure stage

Capture or observation

Relevant output or evidence

The focused MCP test printed:

{"isError":false,"declaredActions":["fail","settle"],"returnedActions":[{"step_id":"fail","status":"failed","error":"simulated strict mode violation"}],"waitCalls":0,"coverage":"partial_attach","actionLimitations":[],"cleanup":"terminated-owned-process"}

One test passed. Cleanup in this output is the simulated cleanup result, not proof of a real process termination. partial_attach describes event-hook coverage; it does not explain why the requested wait is missing.

Typed outcome or recovery expected

For this input, retain fail as failed and represent settle as unexecuted after the earlier failure, or supply equivalent explicit action-omission information. Preserve the original failure diagnostic and collected observations.

Analysis or suspected cause

Source-derived: the action loop appends the failed action and immediately breaks. The provider reads final state and returns a successful capture unless cancelled. Projection copies that prefix without reconciling it with the requested actions. Evidence construction retains the complete request separately, so the discrepancy survives MCP delivery.

Additional context

The relevant implementation and Electron documentation are unchanged between the tested revision and fetched origin/main at ce1f4b1bbaced57529d609f3007a76de583e11c4.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5943 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Process capture or runtime observation ### Observed behavior `capture_electron_scenario` drops the remaining action outcomes after a click fails. In an executed MCP test requesting `fail` followed by `settle`, the returned Evidence retains both requested actions in `parameters.actions`, but `normalized_result.actions` contains only the failed `fail` action. The later wait is never called and has no cancelled/not-executed record or omission explanation. The MCP response is successful; the first action's failure is correctly visible. This reproduction uses a simulated Electron application and process supervision. It executes REA's registered MCP handler, capture service, actual provider/action runner, snapshot parsing, projection and Evidence delivery. No real Electron runtime was launched or signaled. ### Reproduction At the source revision below with its locked dependencies installed, save the following as `src/browser/electronSkippedActionsRepro.test.ts`: ```typescript import { afterEach, expect, it, vi } from "vitest"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { PlaywrightElectronActiveProvider } from "./PlaywrightElectronActiveProvider.js"; import { createServer } from "../server/createServer.js"; import { createTestBinarySession } from "../../tests/fixtures/binarySession.js"; import { observed } from "../../tests/fixtures/analysisExecution.js"; import { ok } from "../domain/result.js"; vi.mock("../process/ProcessOwnershipObservation.js", () => ({ prepareProcessOwnershipInspection: async () => undefined, systemProcessOwnershipHost: {}, observeOwnedProcessLineage: async (owner: any) => ({ status: "verified", observedAt: new Date().toISOString(), lineage: { runId: owner.runId, launcherPid: owner.leaderPid, launcherParentPid: process.pid, processGroupId: owner.leaderPid, descendants: [] }, }), })); vi.mock("./ElectronProcessCleanup.js", () => ({ cleanupElectronProcesses: async () => ({ cleaned: true, signaled: false }), })); afterEach(() => vi.restoreAllMocks()); it("returns successful MCP Evidence with an omitted post-failure action", async () => { const click = vi.fn(async () => { throw new Error("simulated strict mode violation"); }); const wait = vi.fn(async () => undefined); const page = { locator: () => ({ click }), waitForTimeout: wait }; const snapshot = { events: [], retention_budget_bytes: 0, estimated_retained_bytes: 0, event_serialized_byte_upper_bound: 0, retained: 0, dropped: 0, dropped_ipc: 0, dropped_runtime: 0, dropped_event_families: [], dropped_event_roles: [], observed: 0, observed_ipc: 0, observed_runtime: 0, hook_error: false, }; let read = 0; const close = vi.fn(async () => undefined); const app = { process: () => ({ pid: 424242 }), close, windows: () => [page], firstWindow: async () => page, evaluate: async () => [[], [], "simulated-electron", snapshot][read++ % 4], }; const provider = new PlaywrightElectronActiveProvider( {}, async () => app as any, async () => ({ slices: [] }) as any, ); const session = createTestBinarySession(() => ({ execute: async () => observed(null), close: async () => ok(null), })); const server = createServer({ kind: "session", session }, { electronActiveObservation: provider }); const client = new Client({ name: "electron-action-reproduction", version: "1" }); const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); try { await server.connect(serverTransport); await client.connect(clientTransport); const response = await client.callTool({ name: "capture_electron_scenario", arguments: { executable_path: process.execPath, application_path: process.execPath, actions: [ { step_id: "fail", kind: "click", selector: "#duplicate" }, { step_id: "settle", kind: "wait", duration_ms: 0 }, ], } }); expect(response.isError, JSON.stringify(response)).not.toBe(true); const evidence = response.structuredContent as any; const result = evidence.normalized_result; expect(click).toHaveBeenCalledOnce(); expect(wait).not.toHaveBeenCalled(); expect(result.actions.map((action: any) => [action.step_id, action.status])).toEqual([["fail", "failed"]]); expect(result.coverage.status).toBe("partial_attach"); expect(result.limitations.some((text: string) => /action|step|skipp/i.test(text))).toBe(false); expect(evidence.parameters.actions).toHaveLength(2); expect(close).toHaveBeenCalledOnce(); console.log(JSON.stringify({ isError: response.isError ?? false, declaredActions: evidence.parameters.actions.map((action: any) => action.step_id), returnedActions: result.actions.map((action: any) => ({ step_id: action.step_id, status: action.status, error: action.error })), waitCalls: wait.mock.calls.length, coverage: result.coverage.status, actionLimitations: result.limitations.filter((text: string) => /action|step|skipp/i.test(text)), cleanup: result.application.cleanup, })); } finally { await client.close(); await server.close(); await session.close(); await provider.close(); } }); ``` Run only that test: ```sh node node_modules/vitest/vitest.mjs run src/browser/electronSkippedActionsRepro.test.ts --maxWorkers 1 ``` The focused run passed its assertions of the current incorrect result. Delete the temporary test afterward. The injected click rejection is deterministic; this does not establish a real Playwright selector failure. ### Expected behavior Represent every requested action's disposition, including a clear not-executed/cancelled outcome for later actions after an earlier failure, or explicitly report the omitted action IDs/count and reason. Continuing the remaining actions is not required, and a capture may still retain useful observations after an action fails. [Electron documentation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/electron-observation.md#L119-L134) says the result records action status and targets. [Tool design guidance](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/tool-design.md#L35-L55) calls for preserving per-item outcomes and meaningful action choices, with coverage and limitations. The separate browser scenario implementation already [records later actions as cancelled](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightBrowserScenarioProvider.ts#L171-L187); that is a possible existing representation, rather than an Electron-specific contract guarantee. ### Impact and scope P2: action-result consumers cannot account for the entire declared scenario from `normalized_result.actions` alone. The suffix disappears on the first ordinary action rejection, even though the request itself returns Evidence. A practical workaround is to compare `parameters.actions` with the returned action records and treat missing steps as unexecuted. The original failure remains visible, so this is not a claim that the failed action is reported as completed. One deterministic simulated application was tested. Real Electron behavior and natural failure frequency were not measured. CLI reachability follows the shared service wiring; the reproduction exercised MCP only. ### REA version 6.7.0 source checkout at `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3. The exact repository-pinned Node 24.18.0 runtime was not used. Electron application and process supervision were simulated. ### Target and artifact context Two-action owned Electron scenario: a simulated rejected click followed by a zero-duration wait. The target boundary is simulated; paths are supplied dynamically from the running Node executable solely to satisfy the real service's path admission. ### Interface MCP ### Earliest failure stage Capture or observation ### Relevant output or evidence The focused MCP test printed: ```json {"isError":false,"declaredActions":["fail","settle"],"returnedActions":[{"step_id":"fail","status":"failed","error":"simulated strict mode violation"}],"waitCalls":0,"coverage":"partial_attach","actionLimitations":[],"cleanup":"terminated-owned-process"} ``` One test passed. Cleanup in this output is the simulated cleanup result, not proof of a real process termination. `partial_attach` describes event-hook coverage; it does not explain why the requested wait is missing. ### Typed outcome or recovery expected For this input, retain `fail` as failed and represent `settle` as unexecuted after the earlier failure, or supply equivalent explicit action-omission information. Preserve the original failure diagnostic and collected observations. ### Analysis or suspected cause Source-derived: [the action loop](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightElectronActiveActions.ts#L121-L138) appends the failed action and immediately breaks. [The provider](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightElectronActiveProvider.ts#L195-L205) reads final state and returns a successful capture unless cancelled. [Projection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/browser/PlaywrightElectronActiveProjection.ts#L137-L216) copies that prefix without reconciling it with the requested actions. [Evidence construction](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/ElectronActiveEvidence.ts#L9-L41) retains the complete request separately, so the discrepancy survives MCP delivery. ### Additional context The relevant implementation and Electron documentation are unchanged between the tested revision and fetched `origin/main` at `ce1f4b1bbaced57529d609f3007a76de583e11c4`.

bsakaguchi closed this issue

2026-10-31 00:42:24 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:56 +00:00

[PR #3613] [MERGED] fix(analysis-view): unify inert inline Evidence admission #5855