[GH-ISSUE #2518] fix(package): platform verification accepts missing overview results #835

New issue

Closed

opened 2026-10-15 18:28:52 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:28:52 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 13, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/2518

Summary

On a non-Linux host, the packaged CLI platform check can finish successfully when analyze or inspect returns a binary_overview response without its required result. A controlled CLI fixture reproduced this behavior in the production verifier. This gives maintainers a false-success verification signal for these malformed responses; it does not establish that an installed REA release currently produces them.

Steps to reproduce

Use the source baseline and installed dependencies listed under Runtime or environment. On macOS, run the following from the repository root. The fixture emits exit-zero CLI responses and varies one overview command at a time; the other commands return the fields checked by the platform verifier. This invokes the exported production verifier without running installation, a real analysis provider, or the full package verification pipeline.

node --input-type=module <<'JS'
import assert from "node:assert/strict";
import { mkdtemp, writeFile, chmod, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { verifyPackagePlatform } from "./scripts/verify-package-platform.mjs";
assert.notEqual(process.platform, "linux");
const directory = await mkdtemp(join(tmpdir(), "platform-check-"));
const cli = join(directory, "cli.mjs");
try {
  await writeFile(cli, `#!${process.execPath}
const command = process.argv[2];
const overview = {
  operation: "binary_overview",
  normalized_result: { procedure_count: 1 },
};
const responses = {
  analyze: overview,
  inspect: overview,
  function: {
    operation: "analyze_function",
    provider: { id: "hopper" },
    normalized_result: { procedure: { address: "0x1000" } },
  },
  xrefs: { operation: "xrefs", normalized_result: ["0x1000"] },
  trace: { operation: "trace_feature" },
};
if (!(command in responses)) process.exit(3);
if (command === process.env.CHECK_COMMAND)
  responses[command] = JSON.parse(process.env.CHECK_RESPONSE);
console.log(JSON.stringify(responses[command]));
`);
  await chmod(cli, 0o755);
  const check = (command, response) => verifyPackagePlatform({
    cli,
    environment: {
      CHECK_COMMAND: command,
      CHECK_RESPONSE: JSON.stringify(response),
    },
  });
  await check("unused", {});
  console.log("positive control: accepted");
  for (const command of ["analyze", "inspect"]) {
    for (const [name, response] of [
      ["missing", { operation: "binary_overview" }],
      ["null", { operation: "binary_overview", normalized_result: null }],
    ]) {
      await check(command, response);
      console.log(`${command}/${name}: malformed response accepted`);
    }
    await assert.rejects(() => check(command, {
      operation: "binary_overview",
      normalized_result: { procedure_count: 0 },
    }));
    await assert.rejects(() => check(command, {
      operation: "wrong",
      normalized_result: { procedure_count: 1 },
    }));
    console.log(`${command}: zero-count and wrong-operation rejected`);
  }
} finally {
  await rm(directory, { recursive: true, force: true });
}
JS

Expected behavior

The platform smoke check must reject an analyze or inspect response that lacks the required overview result or cannot demonstrate at least one procedure. A response with the expected overview operation and positive procedure count should pass this check. Successful process exit and parseable JSON alone do not meet the overview contract.

Actual behavior

The executed verifier accepted both missing and null results independently for analyze and inspect, then completed its remaining function, cross-reference, and trace checks. It also accepted an empty result object with no procedure count in an additional local case. Positive-count controls passed, while zero-count and wrong-operation controls failed. The observation concerns controlled CLI stdout consumed by the verifier; the installed packaged CLI and real providers were not exercised.

Evidence

Expected source: scripts/verify-package-platform.mjs:13-30 requires a binary overview and at least one procedure; src/contracts/toolOutputSchemaGroups.ts:356-371 and src/contracts/toolOutputSchemaPrimitives.ts:32 require the overview result object and numeric count; docs/testing.md:160-168 preserves malformed-input verification and producer reply shapes.
Failure source: Executed verifyPackagePlatform with the inline CLI fixture. Both guards at scripts/verify-package-platform.mjs:18 and :28 accept missing or null results because their count expression is undefined and its comparison with 1 is false. scripts/lib/verify-package-core.mjs:20-22 and :41 only require successful process exit and parseable JSON, so no earlier result validation blocks the fixture.
Evidence provenance: observed
Local verification: reproduced
Reproduction completeness: complete

The package entry point calls this function in its platform CLI stage at scripts/verify-package.mjs:137-142. Later checks use separate command responses and do not revisit the two overview results. The Linux branch checks a different unsupported-provider outcome, so the malformed-overview reproduction applies to the non-Linux branch.

Restoration check

Run each missing-result and null-result case independently for both overview commands. Each must reject before the platform stage reports completion. Apply the same check to a result object without a procedure count. Keep the positive-count control accepted and the zero-count and wrong-operation controls rejected. Retain the later function, cross-reference, and trace checks. These signals restore the existing platform verification contract without prescribing a particular implementation.

Runtime or environment

Failure verified at a2d8b94dd5 and still present at 8b181716ac, package manifest version 6.4.0, Node v26.10.0, npm 11.19.1, macOS, and existing installed dependencies. The local replay invoked the source verifier with a disposable CLI fixture.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Missing overview results produce a false-success result in the non-Linux packaged platform verifier. This can conceal a CLI-output regression in that verification stage. No malformed installed release, production analysis failure, or security impact was observed.
Workaround status: available
Workaround basis: Maintainers can inspect the analyze and inspect JSON directly and require an object result with a positive numeric procedure count before accepting the platform verification result. This preserves the required observation without a source modification.

Originally created by @coygeek on GitHub (Oct 13, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/2518 ## Summary On a non-Linux host, the packaged CLI platform check can finish successfully when `analyze` or `inspect` returns a `binary_overview` response without its required result. A controlled CLI fixture reproduced this behavior in the production verifier. This gives maintainers a false-success verification signal for these malformed responses; it does not establish that an installed REA release currently produces them. ## Steps to reproduce Use the source baseline and installed dependencies listed under Runtime or environment. On macOS, run the following from the repository root. The fixture emits exit-zero CLI responses and varies one overview command at a time; the other commands return the fields checked by the platform verifier. This invokes the exported production verifier without running installation, a real analysis provider, or the full package verification pipeline. ```sh node --input-type=module <<'JS' import assert from "node:assert/strict"; import { mkdtemp, writeFile, chmod, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { verifyPackagePlatform } from "./scripts/verify-package-platform.mjs"; assert.notEqual(process.platform, "linux"); const directory = await mkdtemp(join(tmpdir(), "platform-check-")); const cli = join(directory, "cli.mjs"); try { await writeFile(cli, `#!${process.execPath} const command = process.argv[2]; const overview = { operation: "binary_overview", normalized_result: { procedure_count: 1 }, }; const responses = { analyze: overview, inspect: overview, function: { operation: "analyze_function", provider: { id: "hopper" }, normalized_result: { procedure: { address: "0x1000" } }, }, xrefs: { operation: "xrefs", normalized_result: ["0x1000"] }, trace: { operation: "trace_feature" }, }; if (!(command in responses)) process.exit(3); if (command === process.env.CHECK_COMMAND) responses[command] = JSON.parse(process.env.CHECK_RESPONSE); console.log(JSON.stringify(responses[command])); `); await chmod(cli, 0o755); const check = (command, response) => verifyPackagePlatform({ cli, environment: { CHECK_COMMAND: command, CHECK_RESPONSE: JSON.stringify(response), }, }); await check("unused", {}); console.log("positive control: accepted"); for (const command of ["analyze", "inspect"]) { for (const [name, response] of [ ["missing", { operation: "binary_overview" }], ["null", { operation: "binary_overview", normalized_result: null }], ]) { await check(command, response); console.log(`${command}/${name}: malformed response accepted`); } await assert.rejects(() => check(command, { operation: "binary_overview", normalized_result: { procedure_count: 0 }, })); await assert.rejects(() => check(command, { operation: "wrong", normalized_result: { procedure_count: 1 }, })); console.log(`${command}: zero-count and wrong-operation rejected`); } } finally { await rm(directory, { recursive: true, force: true }); } JS ``` ## Expected behavior The platform smoke check must reject an `analyze` or `inspect` response that lacks the required overview result or cannot demonstrate at least one procedure. A response with the expected overview operation and positive procedure count should pass this check. Successful process exit and parseable JSON alone do not meet the overview contract. ## Actual behavior The executed verifier accepted both missing and null results independently for `analyze` and `inspect`, then completed its remaining function, cross-reference, and trace checks. It also accepted an empty result object with no procedure count in an additional local case. Positive-count controls passed, while zero-count and wrong-operation controls failed. The observation concerns controlled CLI stdout consumed by the verifier; the installed packaged CLI and real providers were not exercised. ## Evidence - Expected source: `scripts/verify-package-platform.mjs:13-30` requires a binary overview and at least one procedure; `src/contracts/toolOutputSchemaGroups.ts:356-371` and `src/contracts/toolOutputSchemaPrimitives.ts:32` require the overview result object and numeric count; `docs/testing.md:160-168` preserves malformed-input verification and producer reply shapes. - Failure source: Executed `verifyPackagePlatform` with the inline CLI fixture. Both guards at `scripts/verify-package-platform.mjs:18` and `:28` accept missing or null results because their count expression is undefined and its comparison with 1 is false. `scripts/lib/verify-package-core.mjs:20-22` and `:41` only require successful process exit and parseable JSON, so no earlier result validation blocks the fixture. - Evidence provenance: observed - Local verification: reproduced - Reproduction completeness: complete The package entry point calls this function in its platform CLI stage at `scripts/verify-package.mjs:137-142`. Later checks use separate command responses and do not revisit the two overview results. The Linux branch checks a different unsupported-provider outcome, so the malformed-overview reproduction applies to the non-Linux branch. ## Restoration check Run each missing-result and null-result case independently for both overview commands. Each must reject before the platform stage reports completion. Apply the same check to a result object without a procedure count. Keep the positive-count control accepted and the zero-count and wrong-operation controls rejected. Retain the later function, cross-reference, and trace checks. These signals restore the existing platform verification contract without prescribing a particular implementation. ## Runtime or environment Failure verified at a2d8b94dd50dab8d966239dd63ab84718aa8b4b4 and still present at 8b181716aca2cb467be499cba226abfb5ae19141, package manifest version 6.4.0, Node v26.10.0, npm 11.19.1, macOS, and existing installed dependencies. The local replay invoked the source verifier with a disposable CLI fixture. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Missing overview results produce a false-success result in the non-Linux packaged platform verifier. This can conceal a CLI-output regression in that verification stage. No malformed installed release, production analysis failure, or security impact was observed. - Workaround status: available - Workaround basis: Maintainers can inspect the analyze and inspect JSON directly and require an object result with a positive numeric procedure count before accepting the platform verification result. This preserves the required observation without a source modification.

bsakaguchi

2026-10-15 18:28:52 +00:00