[GH-ISSUE #2674] fix(cli): passive native UI observation silently accepts scenario steps #974

New issue

Closed

opened 2026-10-15 18:30:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:30:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2674

Runtime or environment

Verified against source commit a2d8b94dd5 using Node.js 24.18.0, npm 11.16.0 and the locked incur 0.6.1 dependency. Verification used source command registration and an injected analysis recorder. No published package or built executable was tested, and no native application was opened.

A source-only recheck at 3ae826ceba confirms that the observation option schema, dispatch branch, passive/scenario input contracts and downstream observation step selection remain unchanged. No runtime reproduction was rerun for this recheck; the six executed source checks below belong to the original tested baseline.

Summary

A caller who follows observe-native-ui --help can supply a JSON step array that the command accepts but never forwards to analysis. Even a valid zero-duration wait disappears from the request. The command should preserve its passive capture contract while explicitly refusing an unsupported scenario input. Silent acceptance prevents callers from noticing that they selected the wrong command.

Steps to reproduce

From a checkout with its locked dependencies installed, save this probe as src/native-ui-cli-input.test.ts. The source test below was executed with the repository focused-test runner. It uses the production command registration and Incur parser, with a recording dependency that never opens the fixture path or accesses a PID/window. The numeric IDs are parser inputs only.

import { Cli } from "incur";
import { afterEach, expect, it, vi } from "vitest";
import { registerUtilityCommands } from "./cli/utilityCommands.js";
import { silentLogger } from "./logger.js";

const savedExitCode = process.exitCode;
afterEach(() => { process.exitCode = savedExitCode; });

async function invoke(command: string, flags: string[]) {
  const cli = Cli.create("rea", { sync: false });
  const calls: unknown[][] = [];
  const runProviderAnalysis = vi.fn(async (...args: unknown[]) => {
    calls.push(args.slice(0, 3));
    return { recorded: true };
  });
  registerUtilityCommands(cli, silentLogger, {}, {
    runProviderAnalysis,
    runSessionStatus: vi.fn(),
  });
  let stdout = "";
  await cli.serve([command, "fixture.app", "--pid", "123", "--window-id", "456", ...flags, "--json"], {
    env: {}, stdout: text => { stdout += text; }, exit: () => {},
  });
  return { calls, output: JSON.parse(stdout) };
}

it.each([undefined, '[{"kind":"wait","milliseconds":0}]', "not-json"])(
  "observation drops steps %s at the real parser/handler boundary",
  async steps => {
    const result = await invoke("observe-native-ui", steps === undefined ? [] : ["--steps", steps]);
    expect(result.calls).toEqual([["fixture.app", "observe_native_ui", {
      pid: 123, window_id: 456, screenshot: true, accessibility: true, max_nodes: 500,
    }]]);
    expect(result.output).toEqual({ recorded: true });
    console.log(JSON.stringify({ command: "observe-native-ui", steps: steps ?? "omitted", ...result }));
  },
);

it("scenario preserves a valid wait step", async () => {
  const result = await invoke("capture-native-ui-scenario", ["--steps", '[{"kind":"wait","milliseconds":0}]']);
  expect(result.calls).toEqual([["fixture.app", "capture_native_ui_scenario", {
    pid: 123, window_id: 456, screenshot: true, accessibility: true, max_nodes: 500,
    steps: [{ kind: "wait", milliseconds: 0 }],
  }]]);
  console.log(JSON.stringify({ command: "capture-native-ui-scenario", ...result }));
});

it("scenario rejects malformed JSON before dispatch", async () => {
  const result = await invoke("capture-native-ui-scenario", ["--steps", "not-json"]);
  expect(result.calls).toEqual([]);
  expect(result.output).toMatchObject({ code: "invalid_request", details: { operation: "capture_native_ui_scenario" } });
  expect(JSON.stringify(result.output)).toContain("steps");
  console.log(JSON.stringify({ command: "capture-native-ui-scenario", ...result }));
});

it("observation help advertises steps", async () => {
  const cli = Cli.create("rea", { sync: false });
  registerUtilityCommands(cli, silentLogger, {}, { runProviderAnalysis: vi.fn(), runSessionStatus: vi.fn() });
  let stdout = "";
  await cli.serve(["observe-native-ui", "--help"], { env: {}, stdout: text => { stdout += text; }, exit: () => {} });
  expect(stdout).toContain("--steps");
  expect(stdout).toContain("JSON array of declarative AX actions or waits");
  console.log(stdout);
});

Run the focused source test:

npm run test:focused -- src/native-ui-cli-input.test.ts

The assertions characterize the failing behavior. The valid wait and malformed JSON cases both reach the observation recorder without a steps property; the scenario control forwards the wait and rejects malformed JSON.

Expected behavior

Passive native UI observation should advertise its supported capture inputs and explicitly reject an unsupported --steps input before dispatch. The separate scenario command should continue to accept and validate step arrays. The documented separation between passive observation and scenario actions does not require adding actions to observation.

Actual behavior

All six source checks passed. Observation help included this option:

--steps <string>      JSON array of declarative AX actions or waits

Observation with omitted steps, the valid wait array, and not-json made identical analysis calls. The recorded parameters were:

{"pid":123,"window_id":456,"screenshot":true,"accessibility":true,"max_nodes":500}

The scenario control preserved steps: [{kind:"wait",milliseconds:0}]. Its malformed-JSON case made no analysis call and returned:

{"error":"Application workflow failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"capture_native_ui_scenario","issues":[{"path":["steps"],"reason":"invalid_format","expected":"JSON"}]}}

The injected {recorded:true} output is a test observation of dispatch, not a successful native capture result.

Evidence

Expected source: src/cli/utilityCommands.ts:35-55 advertises the option for both commands. src/domain/native/nativeUiObservation.ts:24-29, src/contracts/native/nativeToolContracts.ts:81-88 and docs/native-investigation.md:384-401 establish distinct passive and scenario inputs.
Failure source: The executed probe above observes identical observation dispatch parameters with omitted, valid and malformed steps, plus contrasting scenario validation. src/cli/utilityCommands.ts:74-111 builds parameters without steps and only decodes them on the scenario branch.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The original native UI capability request, morluto/rea#497, describes passive capture and separate declarative scenarios. Its feature closure does not address this CLI input mismatch.

Related: #2183 tracks Windows window capture; #778 tracks reproducible demos. Neither covers the ignored observation steps.

Restoration check

The valid wait array and malformed JSON must no longer reach observation analysis as if --steps had been omitted. Observation help must describe supported passive inputs, and an unsupported scenario flag must fail explicitly before dispatch. Observation without steps must still dispatch its capture scope. The scenario controls must continue forwarding a valid wait and rejecting malformed JSON with a steps-specific input error.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: An advertised caller-selected step sequence is silently discarded at the CLI boundary. Callers can inspect the initial state when they expected a wait or action sequence; this is a bounded behavioral error, and no data loss was observed.
Workaround status: available
Workaround basis: Use the separate capture-native-ui-scenario command for step input. The source probe directly verifies valid-step forwarding and malformed-input rejection on that command. It does not verify native helper or real UI execution.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2674 ## Runtime or environment Verified against source commit a2d8b94dd50dab8d966239dd63ab84718aa8b4b4 using Node.js 24.18.0, npm 11.16.0 and the locked incur 0.6.1 dependency. Verification used source command registration and an injected analysis recorder. No published package or built executable was tested, and no native application was opened. A source-only recheck at 3ae826ceba6f25b08bce50bcb51001209bc16b57 confirms that the observation option schema, dispatch branch, passive/scenario input contracts and downstream observation step selection remain unchanged. No runtime reproduction was rerun for this recheck; the six executed source checks below belong to the original tested baseline. ## Summary A caller who follows `observe-native-ui --help` can supply a JSON step array that the command accepts but never forwards to analysis. Even a valid zero-duration wait disappears from the request. The command should preserve its passive capture contract while explicitly refusing an unsupported scenario input. Silent acceptance prevents callers from noticing that they selected the wrong command. ## Steps to reproduce From a checkout with its locked dependencies installed, save this probe as `src/native-ui-cli-input.test.ts`. The source test below was executed with the repository focused-test runner. It uses the production command registration and Incur parser, with a recording dependency that never opens the fixture path or accesses a PID/window. The numeric IDs are parser inputs only. ```ts import { Cli } from "incur"; import { afterEach, expect, it, vi } from "vitest"; import { registerUtilityCommands } from "./cli/utilityCommands.js"; import { silentLogger } from "./logger.js"; const savedExitCode = process.exitCode; afterEach(() => { process.exitCode = savedExitCode; }); async function invoke(command: string, flags: string[]) { const cli = Cli.create("rea", { sync: false }); const calls: unknown[][] = []; const runProviderAnalysis = vi.fn(async (...args: unknown[]) => { calls.push(args.slice(0, 3)); return { recorded: true }; }); registerUtilityCommands(cli, silentLogger, {}, { runProviderAnalysis, runSessionStatus: vi.fn(), }); let stdout = ""; await cli.serve([command, "fixture.app", "--pid", "123", "--window-id", "456", ...flags, "--json"], { env: {}, stdout: text => { stdout += text; }, exit: () => {}, }); return { calls, output: JSON.parse(stdout) }; } it.each([undefined, '[{"kind":"wait","milliseconds":0}]', "not-json"])( "observation drops steps %s at the real parser/handler boundary", async steps => { const result = await invoke("observe-native-ui", steps === undefined ? [] : ["--steps", steps]); expect(result.calls).toEqual([["fixture.app", "observe_native_ui", { pid: 123, window_id: 456, screenshot: true, accessibility: true, max_nodes: 500, }]]); expect(result.output).toEqual({ recorded: true }); console.log(JSON.stringify({ command: "observe-native-ui", steps: steps ?? "omitted", ...result })); }, ); it("scenario preserves a valid wait step", async () => { const result = await invoke("capture-native-ui-scenario", ["--steps", '[{"kind":"wait","milliseconds":0}]']); expect(result.calls).toEqual([["fixture.app", "capture_native_ui_scenario", { pid: 123, window_id: 456, screenshot: true, accessibility: true, max_nodes: 500, steps: [{ kind: "wait", milliseconds: 0 }], }]]); console.log(JSON.stringify({ command: "capture-native-ui-scenario", ...result })); }); it("scenario rejects malformed JSON before dispatch", async () => { const result = await invoke("capture-native-ui-scenario", ["--steps", "not-json"]); expect(result.calls).toEqual([]); expect(result.output).toMatchObject({ code: "invalid_request", details: { operation: "capture_native_ui_scenario" } }); expect(JSON.stringify(result.output)).toContain("steps"); console.log(JSON.stringify({ command: "capture-native-ui-scenario", ...result })); }); it("observation help advertises steps", async () => { const cli = Cli.create("rea", { sync: false }); registerUtilityCommands(cli, silentLogger, {}, { runProviderAnalysis: vi.fn(), runSessionStatus: vi.fn() }); let stdout = ""; await cli.serve(["observe-native-ui", "--help"], { env: {}, stdout: text => { stdout += text; }, exit: () => {} }); expect(stdout).toContain("--steps"); expect(stdout).toContain("JSON array of declarative AX actions or waits"); console.log(stdout); }); ``` Run the focused source test: ```sh npm run test:focused -- src/native-ui-cli-input.test.ts ``` The assertions characterize the failing behavior. The valid wait and malformed JSON cases both reach the observation recorder without a `steps` property; the scenario control forwards the wait and rejects malformed JSON. ## Expected behavior Passive native UI observation should advertise its supported capture inputs and explicitly reject an unsupported `--steps` input before dispatch. The separate scenario command should continue to accept and validate step arrays. The documented separation between passive observation and scenario actions does not require adding actions to observation. ## Actual behavior All six source checks passed. Observation help included this option: ```text --steps <string> JSON array of declarative AX actions or waits ``` Observation with omitted steps, the valid wait array, and `not-json` made identical analysis calls. The recorded parameters were: ```json {"pid":123,"window_id":456,"screenshot":true,"accessibility":true,"max_nodes":500} ``` The scenario control preserved `steps: [{kind:"wait",milliseconds:0}]`. Its malformed-JSON case made no analysis call and returned: ```json {"error":"Application workflow failed","code":"invalid_request","category":"invalid_input","message":"Analysis input is invalid. Check the arguments and try again.","retryable":true,"remediation":{"action":"Correct the listed arguments and retry."},"details":{"operation":"capture_native_ui_scenario","issues":[{"path":["steps"],"reason":"invalid_format","expected":"JSON"}]}} ``` The injected `{recorded:true}` output is a test observation of dispatch, not a successful native capture result. ## Evidence - Expected source: `src/cli/utilityCommands.ts:35-55` advertises the option for both commands. `src/domain/native/nativeUiObservation.ts:24-29`, `src/contracts/native/nativeToolContracts.ts:81-88` and `docs/native-investigation.md:384-401` establish distinct passive and scenario inputs. - Failure source: The executed probe above observes identical observation dispatch parameters with omitted, valid and malformed steps, plus contrasting scenario validation. `src/cli/utilityCommands.ts:74-111` builds parameters without steps and only decodes them on the scenario branch. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The original native UI capability request, [morluto/rea#497](https://github.com/morluto/rea/issues/497), describes passive capture and separate declarative scenarios. Its feature closure does not address this CLI input mismatch. Related: #2183 tracks Windows window capture; #778 tracks reproducible demos. Neither covers the ignored observation steps. ## Restoration check The valid wait array and malformed JSON must no longer reach observation analysis as if `--steps` had been omitted. Observation help must describe supported passive inputs, and an unsupported scenario flag must fail explicitly before dispatch. Observation without steps must still dispatch its capture scope. The scenario controls must continue forwarding a valid wait and rejecting malformed JSON with a steps-specific input error. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: An advertised caller-selected step sequence is silently discarded at the CLI boundary. Callers can inspect the initial state when they expected a wait or action sequence; this is a bounded behavioral error, and no data loss was observed. - Workaround status: available - Workaround basis: Use the separate capture-native-ui-scenario command for step input. The source probe directly verifies valid-step forwarding and malformed-input rejection on that command. It does not verify native helper or real UI execution.

bsakaguchi

2026-10-15 18:30:01 +00:00