morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5995] [Bug] Invalid-UTF-8 JavaScript is omitted from semantic unavailable-source coverage #3661

New issue

Closed

opened 2026-10-31 00:42:49 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:49 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5995

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production analyze-javascript-application CLI omits an in-scope JavaScript file from semantic unavailable-source coverage when its bytes cannot be decoded as UTF-8.

With ok.js containing valid source and bad.js containing bytes ff fe ff, the command succeeds. The application graph correctly retains an unavailable parse-javascript scope for bad.js, and statistics.invalid_utf8_files is 1. However, the semantic graph contains only ok.js, no unknowns, and zero omitted nodes and relations.

As a control, selecting only ok.js makes the same bad.js file appear in the semantic graph with 12 incomplete-module unknowns and unknown omission extents. Decoder exclusion therefore loses source-gap attribution that deliberate source exclusion preserves.

Reproduction

From a built checkout at the revision below, set REPRO_DIR to a disposable absolute directory and run this script with Node from the repository root. It creates the two inert source files and invokes the real CLI twice. Both cases were first run individually, and this recipe then reproduced them. Its assertions describe the current incorrect decoder result and the existing source-selection control.

import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { mkdirSync, writeFileSync } from 'node:fs';
import { join, isAbsolute } from 'node:path';

assert(process.env.REPRO_DIR && isAbsolute(process.env.REPRO_DIR),
  'Set REPRO_DIR to a disposable absolute directory.');
const app = join(process.env.REPRO_DIR, 'app');
mkdirSync(app, { recursive: true });
writeFileSync(join(app, 'ok.js'), 'export const answer = 42;\n');
writeFileSync(join(app, 'bad.js'), Buffer.from([0xff, 0xfe, 0xff]));
for (const selectedOnly of [false, true]) {
  const args = ['scripts/rea.mjs', 'analyze-javascript-application', app];
  if (selectedOnly) args.push('--source-path', 'ok.js');
  args.push('--format', 'json');
  const stdout = execFileSync(process.execPath, args, {
    encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'],
    timeout: 60000, maxBuffer: 16 * 1024 * 1024,
  });
  const r = JSON.parse(stdout).normalized_result;
  const s = r.semantic_graph;
  const modulePaths = [...new Set(s.nodes.map(n => n.identity.module_path))];
  const unknowns = r.graph.nodes.filter(n => n.kind === 'unknown')
    .flatMap(n => n.observations.map(o => ({
      path: o.properties.path, operation: o.properties.operation,
    })));
  assert.equal(r.graph.coverage.status, 'partial');
  assert.equal(s.coverage.truncated, false);
  assert.equal(s.nodes.some(n => n.identity.module_path === 'bad.js'), selectedOnly);
  assert.equal(r.statistics.invalid_utf8_files, selectedOnly ? 0 : 1);
  assert.equal(s.coverage.status, selectedOnly ? 'partial' : 'unknown');
  assert.equal(s.coverage.omitted_nodes, selectedOnly ? null : 0);
  assert.equal(s.coverage.omitted_relations, selectedOnly ? null : 0);
  assert.equal(s.unknowns.length, selectedOnly ? 12 : 0);
  console.log(JSON.stringify({
    case: selectedOnly ? 'unselected-control' : 'invalid-utf8',
    invalid_utf8_files: r.statistics.invalid_utf8_files,
    application_unknowns: unknowns,
    application_status: r.graph.coverage.status,
    semantic_status: s.coverage.status,
    semantic_truncated: s.coverage.truncated,
    semantic_omitted_nodes: s.coverage.omitted_nodes,
    semantic_omitted_relations: s.coverage.omitted_relations,
    semantic_module_paths: modulePaths,
    semantic_unknown_count: s.unknowns.length,
  }));
}

Expected behavior

Retain a semantic unavailable-source record identifying bad.js and its decoding limitation, with incomplete semantic coverage and unknown omitted extents. No resource truncation or named limit is needed for an ordinary decoding failure.

The existing semantic projection contract explicitly preserves JavaScript-source unknowns. Its unavailable-file implementation retains a module where capacity permits, emits per-family unknowns, and makes omitted node/relation extents unknown. The general contract requires missing coverage to remain explicit.

Impact and scope

P2: semantic-only consumers lose the identity and reason of an unanalyzed source and receive unsupported zero omission extents. The semantic graph says unknown, so this is not a claim that it reports complete coverage.

The application graph still contains the file, its invalid-UTF-8 status and an explicit unavailable parse scope. Inspecting those records alongside the semantic graph provides a demonstrated cross-check. No original-byte loss, target execution, security effect, or incorrect downstream trace/comparison was demonstrated. Runtime verification covers this small directory artifact and its source-selection control; ASAR and MCP transport were not exercised.

REA version

6.7.0 source build; revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node.js v26.12.0. Used the prebuilt CLI and production static JavaScript worker, with no external analysis provider. The repository's Node.js 24.18.0 development pin was not exercised.

Target and artifact context

A local directory containing exactly ok.js with export const answer = 42; followed by a newline, and bad.js with exactly three bytes ff fe ff. Both files are inventoried. The first analysis includes all sources; the control selects only ok.js.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Reduced output from the executed recipe:

{"case":"invalid-utf8","invalid_utf8_files":1,"application_unknowns":[{"path":"bad.js","operation":"parse-javascript"}],"application_status":"partial","semantic_status":"unknown","semantic_truncated":false,"semantic_omitted_nodes":0,"semantic_omitted_relations":0,"semantic_module_paths":["ok.js"],"semantic_unknown_count":0}
{"case":"unselected-control","invalid_utf8_files":0,"application_unknowns":[{"path":"bad.js","operation":"select-javascript-sources"}],"application_status":"partial","semantic_status":"partial","semantic_truncated":false,"semantic_omitted_nodes":null,"semantic_omitted_relations":null,"semantic_module_paths":["ok.js","bad.js"],"semantic_unknown_count":12}

Both CLI invocations exit 0. The first also reports relevant_files: 2, parsed_javascript_files: 1, and parse_failures: 0. The application unknown explicitly states JavaScript text was unavailable: invalid-utf8.

Typed outcome or recovery expected

A source-specific semantic incomplete-module unknown carrying the decoding limitation, with partial coverage and unknown omitted node/relation extents. Preserve the existing application inventory and unavailable parse scope. Ordinary decoder unavailability can remain truncated: false.

Analysis or suspected cause

Source-derived: the artifact reader preserves fatal UTF-8 decoding failure as excluded text. The incremental analysis branch routes excluded text through analyzeArtifactFileSteps and continues before worker analysis. That helper pushes javascript: null, semantic: null without notifying the semantic projection.

The outside-scope classifier records unavailable semantics only for outside-analysis-scope. Decoder exclusion therefore never increments the semantic builder's unavailable-file count. Its final coverage calculation consequently retains zero omission extents.

Additional context

The full diff and relevant head source of PR #2092 were inspected at b6a097b35d61fecc8c689f6bd4ef47bde49577ab. Its projection-null fix handles decoded syntax failures after worker analysis. Its excluded-text branch and null helper result still bypass semantic unavailable-source recording. No PR-head execution is claimed.

Issue #5951 concerns missing HTML-reference gaps in the application graph; this reproduction's application graph already records its JavaScript gap. Issue #5904 concerns validating contradictory imported coverage; this result is generated directly from original source bytes. Issue #5881 concerns JSON omission-count projection. These require different producing-boundary fixes.

Fetched upstream main at 25e1a24a3ed3671a0148757555f18d16c29d936a leaves the reader, excluded-text analysis branch and semantic builder unchanged. Its worker-reuse changes do not change this branch. This was a source freshness check, not execution of the newer revision. The introducing revision and a last known good version have not been established.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5995 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior The production `analyze-javascript-application` CLI omits an in-scope JavaScript file from semantic unavailable-source coverage when its bytes cannot be decoded as UTF-8. With `ok.js` containing valid source and `bad.js` containing bytes `ff fe ff`, the command succeeds. The application graph correctly retains an unavailable `parse-javascript` scope for `bad.js`, and `statistics.invalid_utf8_files` is 1. However, the semantic graph contains only `ok.js`, no unknowns, and zero omitted nodes and relations. As a control, selecting only `ok.js` makes the same `bad.js` file appear in the semantic graph with 12 `incomplete-module` unknowns and unknown omission extents. Decoder exclusion therefore loses source-gap attribution that deliberate source exclusion preserves. ### Reproduction From a built checkout at the revision below, set `REPRO_DIR` to a disposable absolute directory and run this script with Node from the repository root. It creates the two inert source files and invokes the real CLI twice. Both cases were first run individually, and this recipe then reproduced them. Its assertions describe the current incorrect decoder result and the existing source-selection control. ```js import assert from 'node:assert/strict'; import { execFileSync } from 'node:child_process'; import { mkdirSync, writeFileSync } from 'node:fs'; import { join, isAbsolute } from 'node:path'; assert(process.env.REPRO_DIR && isAbsolute(process.env.REPRO_DIR), 'Set REPRO_DIR to a disposable absolute directory.'); const app = join(process.env.REPRO_DIR, 'app'); mkdirSync(app, { recursive: true }); writeFileSync(join(app, 'ok.js'), 'export const answer = 42;\n'); writeFileSync(join(app, 'bad.js'), Buffer.from([0xff, 0xfe, 0xff])); for (const selectedOnly of [false, true]) { const args = ['scripts/rea.mjs', 'analyze-javascript-application', app]; if (selectedOnly) args.push('--source-path', 'ok.js'); args.push('--format', 'json'); const stdout = execFileSync(process.execPath, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: 60000, maxBuffer: 16 * 1024 * 1024, }); const r = JSON.parse(stdout).normalized_result; const s = r.semantic_graph; const modulePaths = [...new Set(s.nodes.map(n => n.identity.module_path))]; const unknowns = r.graph.nodes.filter(n => n.kind === 'unknown') .flatMap(n => n.observations.map(o => ({ path: o.properties.path, operation: o.properties.operation, }))); assert.equal(r.graph.coverage.status, 'partial'); assert.equal(s.coverage.truncated, false); assert.equal(s.nodes.some(n => n.identity.module_path === 'bad.js'), selectedOnly); assert.equal(r.statistics.invalid_utf8_files, selectedOnly ? 0 : 1); assert.equal(s.coverage.status, selectedOnly ? 'partial' : 'unknown'); assert.equal(s.coverage.omitted_nodes, selectedOnly ? null : 0); assert.equal(s.coverage.omitted_relations, selectedOnly ? null : 0); assert.equal(s.unknowns.length, selectedOnly ? 12 : 0); console.log(JSON.stringify({ case: selectedOnly ? 'unselected-control' : 'invalid-utf8', invalid_utf8_files: r.statistics.invalid_utf8_files, application_unknowns: unknowns, application_status: r.graph.coverage.status, semantic_status: s.coverage.status, semantic_truncated: s.coverage.truncated, semantic_omitted_nodes: s.coverage.omitted_nodes, semantic_omitted_relations: s.coverage.omitted_relations, semantic_module_paths: modulePaths, semantic_unknown_count: s.unknowns.length, })); } ``` ### Expected behavior Retain a semantic unavailable-source record identifying `bad.js` and its decoding limitation, with incomplete semantic coverage and unknown omitted extents. No resource truncation or named limit is needed for an ordinary decoding failure. The [existing semantic projection contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptSemanticGraphBuilder.ts#L180-L186) explicitly preserves JavaScript-source unknowns. Its [unavailable-file implementation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptSemanticGraphBuilder.ts#L431-L491) retains a module where capacity permits, emits per-family unknowns, and makes omitted node/relation extents unknown. The [general contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/tool-design.md#L50-L57) requires missing coverage to remain explicit. ### Impact and scope P2: semantic-only consumers lose the identity and reason of an unanalyzed source and receive unsupported zero omission extents. The semantic graph says `unknown`, so this is not a claim that it reports complete coverage. The application graph still contains the file, its invalid-UTF-8 status and an explicit unavailable parse scope. Inspecting those records alongside the semantic graph provides a demonstrated cross-check. No original-byte loss, target execution, security effect, or incorrect downstream trace/comparison was demonstrated. Runtime verification covers this small directory artifact and its source-selection control; ASAR and MCP transport were not exercised. ### REA version 6.7.0 source build; revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v26.12.0. Used the prebuilt CLI and production static JavaScript worker, with no external analysis provider. The repository's Node.js 24.18.0 development pin was not exercised. ### Target and artifact context A local directory containing exactly `ok.js` with `export const answer = 42;` followed by a newline, and `bad.js` with exactly three bytes `ff fe ff`. Both files are inventoried. The first analysis includes all sources; the control selects only `ok.js`. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Reduced output from the executed recipe: ```jsonl {"case":"invalid-utf8","invalid_utf8_files":1,"application_unknowns":[{"path":"bad.js","operation":"parse-javascript"}],"application_status":"partial","semantic_status":"unknown","semantic_truncated":false,"semantic_omitted_nodes":0,"semantic_omitted_relations":0,"semantic_module_paths":["ok.js"],"semantic_unknown_count":0} {"case":"unselected-control","invalid_utf8_files":0,"application_unknowns":[{"path":"bad.js","operation":"select-javascript-sources"}],"application_status":"partial","semantic_status":"partial","semantic_truncated":false,"semantic_omitted_nodes":null,"semantic_omitted_relations":null,"semantic_module_paths":["ok.js","bad.js"],"semantic_unknown_count":12} ``` Both CLI invocations exit 0. The first also reports `relevant_files: 2`, `parsed_javascript_files: 1`, and `parse_failures: 0`. The application unknown explicitly states `JavaScript text was unavailable: invalid-utf8.` ### Typed outcome or recovery expected A source-specific semantic `incomplete-module` unknown carrying the decoding limitation, with partial coverage and unknown omitted node/relation extents. Preserve the existing application inventory and unavailable parse scope. Ordinary decoder unavailability can remain `truncated: false`. ### Analysis or suspected cause Source-derived: the [artifact reader](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/javascript/JavaScriptArtifactFiles.ts#L358-L372) preserves fatal UTF-8 decoding failure as excluded text. The [incremental analysis branch](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysis.ts#L165-L175) routes excluded text through `analyzeArtifactFileSteps` and continues before worker analysis. That [helper](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysis.ts#L397-L399) pushes `javascript: null, semantic: null` without notifying the semantic projection. The [outside-scope classifier](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysis.ts#L449-L460) records unavailable semantics only for `outside-analysis-scope`. Decoder exclusion therefore never increments the semantic builder's unavailable-file count. Its [final coverage calculation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptSemanticGraphBuilder.ts#L305-L317) consequently retains zero omission extents. ### Additional context The full diff and relevant head source of PR #2092 were inspected at `b6a097b35d61fecc8c689f6bd4ef47bde49577ab`. Its [projection-null fix](https://github.com/morluto/rea/blob/b6a097b35d61fecc8c689f6bd4ef47bde49577ab/src/application/javascript/JavaScriptArtifactAnalysis.ts#L223-L233) handles decoded syntax failures after worker analysis. Its [excluded-text branch](https://github.com/morluto/rea/blob/b6a097b35d61fecc8c689f6bd4ef47bde49577ab/src/application/javascript/JavaScriptArtifactAnalysis.ts#L173-L182) and [null helper result](https://github.com/morluto/rea/blob/b6a097b35d61fecc8c689f6bd4ef47bde49577ab/src/application/javascript/JavaScriptArtifactAnalysis.ts#L374-L376) still bypass semantic unavailable-source recording. No PR-head execution is claimed. Issue #5951 concerns missing HTML-reference gaps in the application graph; this reproduction's application graph already records its JavaScript gap. Issue #5904 concerns validating contradictory imported coverage; this result is generated directly from original source bytes. Issue #5881 concerns JSON omission-count projection. These require different producing-boundary fixes. Fetched upstream main at `25e1a24a3ed3671a0148757555f18d16c29d936a` leaves the reader, excluded-text analysis branch and semantic builder unchanged. Its worker-reuse changes do not change this branch. This was a source freshness check, not execution of the newer revision. The introducing revision and a last known good version have not been established.

bsakaguchi closed this issue

2026-10-31 00:42:49 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:42 +00:00

[PR #3559] [CLOSED] fix(electron): block and record shell.trashItem #5823

bsakaguchi referenced this issue

2026-10-31 01:10:04 +00:00

[PR #3661] [MERGED] fix(mcp): keep a snapshot open ahead of later calls on the same connection #5869