[GH-ISSUE #5983] [Bug] JavaScript summary limitation count excludes semantic graph caveats #3651

New issue

Closed

opened 2026-10-31 00:42:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:43 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5983

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A JavaScript application summary from inspect-analysis-view reports summary.limitation_count: 5 while its own limitations array contains 16 distinct messages. The count excludes all 11 semantic graph limitations included in that response.

This occurs with a directory containing only main.js with export const answer = 42; and a trailing newline. Both analysis and summary projection succeed. No message-budget omission notice is present.

Reproduction

From the built repository root, set REPRO_DIR to a disposable absolute directory and run this script with Node. I executed this script successfully after an initial pair of equivalent CLI calls. It creates the inert fixture, runs both public CLI commands, and checks the current incorrect counts; it does not execute the analyzed source or use an external provider.

import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { join, resolve } from 'node:path';

assert(process.env.REPRO_DIR, 'Set REPRO_DIR to a disposable absolute directory.');
const root = resolve(process.env.REPRO_DIR);
const app = join(root, 'application');
mkdirSync(app, { recursive: true });
writeFileSync(join(app, 'main.js'), 'export const answer = 42;\n');
const run = args => JSON.parse(execFileSync(process.execPath,
  ['scripts/rea.mjs', ...args, '--json'],
  { encoding: 'utf8', maxBuffer: 16 * 1024 * 1024,
    stdio: ['ignore', 'pipe', 'pipe'] }));
const parent = run(['analyze-javascript-application', app]);
const evidenceFile = join(root, 'application-evidence.json');
writeFileSync(evidenceFile, JSON.stringify(parent));
const request = join(root, 'view-request.json');
writeFileSync(request, JSON.stringify({
  source: { kind: 'evidence-file', path: evidenceFile },
  view: { kind: 'summary' }
}));
const result = run(['inspect-analysis-view', request]);
writeFileSync(join(root, 'summary-evidence.json'), JSON.stringify(result));
const analysis = parent.normalized_result;
const view = result.normalized_result;
const merged = [...new Set([...parent.limitations, ...analysis.limitations,
  ...analysis.graph.limitations, ...analysis.semantic_graph.limitations])].sort();
assert.equal(view.kind, 'summary');
assert.deepEqual(view.limitations, merged);
assert.equal(view.summary.limitation_count, 5);
assert.equal(view.limitations.length, 16);
const missing = analysis.semantic_graph.limitations.filter(
  text => !analysis.limitations.includes(text));
assert.equal(missing.length, 11);
assert(missing.every(text => view.limitations.includes(text)));
assert(!view.limitations.some(text => text.includes('omitted from this view')));
console.log(JSON.stringify({
  summary_limitation_count: view.summary.limitation_count,
  returned_limitation_count: view.limitations.length,
  analysis_limitation_count: analysis.limitations.length,
  graph_limitation_count: analysis.graph.limitations.length,
  semantic_limitation_count: analysis.semantic_graph.limitations.length,
  distinct_semantic_limitations_not_counted: missing.length,
  representative_uncounted_limitation: missing.find(text =>
    text.startsWith('Local data flow'))
}, null, 2));

Expected behavior

The unqualified summary limitation count should account for the same merged, deduplicated limitation sources represented by the summary's limitations field. In this untruncated case, the expected count is 16.

This expectation is source-derived: the JavaScript view merges parent, analysis, application graph and semantic graph limitations, and the sibling Mach-O summary counts its merged caveats. The documented CLI workflow recommends saving complete analysis Evidence and reading this summary.

For oversized message sets, preserve the existing omission notice and define the count consistently with the full source set; this reproduction does not exercise that boundary.

Impact and scope

P2: a machine-readable summary counter understates reported caveats, including the limitation on control-flow-sensitive reaching definitions and arbitrary dynamic property resolution. A consumer using this field to count caveats receives an incomplete result.

The observed scope is the ordinary single-module fixture, reproduced through two successful analysis/summary pairs. All 16 messages remain readable in the same small response, which provides a verified workaround. No evidence loss, failed downstream consumer, false zero count, target mutation or security effect was observed. Frequency in real applications and other platforms was not measured.

REA version

rea-agents 6.7.0 source build; revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node.js v26.12.0. Existing locked dependencies and built runtime. The repository's Node.js 24.18.0 development pin was not separately exercised.

Target and artifact context

An inert application directory containing only main.js, with the bytes export const answer = 42; followed by a newline. No Electron process, ASAR archive, browser or native analysis provider is required.

Interface

CLI

Earliest failure stage

Transport or result projection

Relevant output or evidence

The executed reproduction printed:

{
  "summary_limitation_count": 5,
  "returned_limitation_count": 16,
  "analysis_limitation_count": 5,
  "graph_limitation_count": 5,
  "semantic_limitation_count": 11,
  "distinct_semantic_limitations_not_counted": 11,
  "representative_uncounted_limitation": "Local data flow does not claim control-flow-sensitive reaching definitions or arbitrary dynamic property resolution."
}

The script also verified that the returned list equals the complete distinct union of parent, analysis, structural graph and semantic graph limitations, and that no omission notice is present. These are real producer-generated and authenticated CLI results, not manually constructed Evidence.

Typed outcome or recovery expected

Retain the successful summary and all existing caveats, but make summary.limitation_count include semantic graph caveats. Preserve message deduplication and the separate byte-budget behavior.

Analysis or suspected cause

Source-derived: the summary branch counts only analysis.limitations.length, while its shared fields merge several sources. The message helper deduplicates the merged values.

Structural graph limitations are already assigned to the final analysis result by the application service. In this fixture, those five caveats are counted; the additional eleven semantic graph caveats are omitted from the count. No repair was applied or tested.

Additional context

This differs from #5973, which loses an encountered provenance expression-depth limit before public coverage projection, and #5955, which concerns hard-coded example statistics. Here the semantic caveats already exist and appear in the live result, but the summary count excludes them. It also differs from #5464 / PR #5651, which address module-versus-graph-node units in summary coverage.

Fetched upstream main at 25e1a24a3ed3671a0148757555f18d16c29d936a; the relevant projection, message helper and final limitation assignment are unchanged. This is a source freshness comparison, not a separate runtime execution at the newer revision. MCP transport and a last-known-good version were not established.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5983 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A JavaScript application summary from `inspect-analysis-view` reports `summary.limitation_count: 5` while its own `limitations` array contains 16 distinct messages. The count excludes all 11 semantic graph limitations included in that response. This occurs with a directory containing only `main.js` with `export const answer = 42;` and a trailing newline. Both analysis and summary projection succeed. No message-budget omission notice is present. ### Reproduction From the built repository root, set `REPRO_DIR` to a disposable absolute directory and run this script with Node. I executed this script successfully after an initial pair of equivalent CLI calls. It creates the inert fixture, runs both public CLI commands, and checks the current incorrect counts; it does not execute the analyzed source or use an external provider. ```js import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { join, resolve } from 'node:path'; assert(process.env.REPRO_DIR, 'Set REPRO_DIR to a disposable absolute directory.'); const root = resolve(process.env.REPRO_DIR); const app = join(root, 'application'); mkdirSync(app, { recursive: true }); writeFileSync(join(app, 'main.js'), 'export const answer = 42;\n'); const run = args => JSON.parse(execFileSync(process.execPath, ['scripts/rea.mjs', ...args, '--json'], { encoding: 'utf8', maxBuffer: 16 * 1024 * 1024, stdio: ['ignore', 'pipe', 'pipe'] })); const parent = run(['analyze-javascript-application', app]); const evidenceFile = join(root, 'application-evidence.json'); writeFileSync(evidenceFile, JSON.stringify(parent)); const request = join(root, 'view-request.json'); writeFileSync(request, JSON.stringify({ source: { kind: 'evidence-file', path: evidenceFile }, view: { kind: 'summary' } })); const result = run(['inspect-analysis-view', request]); writeFileSync(join(root, 'summary-evidence.json'), JSON.stringify(result)); const analysis = parent.normalized_result; const view = result.normalized_result; const merged = [...new Set([...parent.limitations, ...analysis.limitations, ...analysis.graph.limitations, ...analysis.semantic_graph.limitations])].sort(); assert.equal(view.kind, 'summary'); assert.deepEqual(view.limitations, merged); assert.equal(view.summary.limitation_count, 5); assert.equal(view.limitations.length, 16); const missing = analysis.semantic_graph.limitations.filter( text => !analysis.limitations.includes(text)); assert.equal(missing.length, 11); assert(missing.every(text => view.limitations.includes(text))); assert(!view.limitations.some(text => text.includes('omitted from this view'))); console.log(JSON.stringify({ summary_limitation_count: view.summary.limitation_count, returned_limitation_count: view.limitations.length, analysis_limitation_count: analysis.limitations.length, graph_limitation_count: analysis.graph.limitations.length, semantic_limitation_count: analysis.semantic_graph.limitations.length, distinct_semantic_limitations_not_counted: missing.length, representative_uncounted_limitation: missing.find(text => text.startsWith('Local data flow')) }, null, 2)); ``` ### Expected behavior The unqualified summary limitation count should account for the same merged, deduplicated limitation sources represented by the summary's `limitations` field. In this untruncated case, the expected count is 16. This expectation is source-derived: [the JavaScript view merges parent, analysis, application graph and semantic graph limitations](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisView/javascriptApplicationView.ts#L220-L242), and the [sibling Mach-O summary counts its merged caveats](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisView/machoView.ts#L112-L148). The [documented CLI workflow](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/cli.md#L24-L39) recommends saving complete analysis Evidence and reading this summary. For oversized message sets, preserve the existing omission notice and define the count consistently with the full source set; this reproduction does not exercise that boundary. ### Impact and scope P2: a machine-readable summary counter understates reported caveats, including the limitation on control-flow-sensitive reaching definitions and arbitrary dynamic property resolution. A consumer using this field to count caveats receives an incomplete result. The observed scope is the ordinary single-module fixture, reproduced through two successful analysis/summary pairs. All 16 messages remain readable in the same small response, which provides a verified workaround. No evidence loss, failed downstream consumer, false zero count, target mutation or security effect was observed. Frequency in real applications and other platforms was not measured. ### REA version rea-agents 6.7.0 source build; revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v26.12.0. Existing locked dependencies and built runtime. The repository's Node.js 24.18.0 development pin was not separately exercised. ### Target and artifact context An inert application directory containing only `main.js`, with the bytes `export const answer = 42;` followed by a newline. No Electron process, ASAR archive, browser or native analysis provider is required. ### Interface CLI ### Earliest failure stage Transport or result projection ### Relevant output or evidence The executed reproduction printed: ```json { "summary_limitation_count": 5, "returned_limitation_count": 16, "analysis_limitation_count": 5, "graph_limitation_count": 5, "semantic_limitation_count": 11, "distinct_semantic_limitations_not_counted": 11, "representative_uncounted_limitation": "Local data flow does not claim control-flow-sensitive reaching definitions or arbitrary dynamic property resolution." } ``` The script also verified that the returned list equals the complete distinct union of parent, analysis, structural graph and semantic graph limitations, and that no omission notice is present. These are real producer-generated and authenticated CLI results, not manually constructed Evidence. ### Typed outcome or recovery expected Retain the successful summary and all existing caveats, but make `summary.limitation_count` include semantic graph caveats. Preserve message deduplication and the separate byte-budget behavior. ### Analysis or suspected cause Source-derived: [the summary branch counts only `analysis.limitations.length`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisView/javascriptApplicationView.ts#L352-L391), while its shared fields merge several sources. [The message helper deduplicates the merged values](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/analysisView/analysisViewCoverage.ts#L40-L71). Structural graph limitations are already [assigned to the final analysis result by the application service](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptApplicationService.ts#L78-L83). In this fixture, those five caveats are counted; the additional eleven semantic graph caveats are omitted from the count. No repair was applied or tested. ### Additional context This differs from #5973, which loses an encountered provenance expression-depth limit before public coverage projection, and #5955, which concerns hard-coded example statistics. Here the semantic caveats already exist and appear in the live result, but the summary count excludes them. It also differs from #5464 / PR #5651, which address module-versus-graph-node units in summary coverage. Fetched upstream main at `25e1a24a3ed3671a0148757555f18d16c29d936a`; the relevant projection, message helper and final limitation assignment are unchanged. This is a source freshness comparison, not a separate runtime execution at the newer revision. MCP transport and a last-known-good version were not established.

bsakaguchi closed this issue

2026-10-31 00:42:43 +00:00

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471