[GH-ISSUE #5951] [Bug] Invalid-UTF-8 HTML omits its unavailable reference-parsing scope #3621

New issue

Closed

opened 2026-10-31 00:42:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:42:27 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5951

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

analyze-javascript-application returns successful partial Evidence for a selected index.html containing invalid UTF-8, but the graph has no unknown processing scope for that file. Its inventory observation preserves text_status: "invalid-utf8" and parse_status: null; the applicable parse-html-script-references operation is absent.

A four-case public CLI reproduction confirms this difference. Valid HTML with an external reference produces a load edge. Invalid-UTF-8 HTML produces neither a load nor an unavailable parsing scope. Invalid-UTF-8 JSON, in contrast, produces an explicit parse-json-module unknown scope.

Reproduction

Run this script as node reproduce.mjs from a built checkout. It creates disposable directory artifacts, invokes the compiled public CLI through its normal dispatcher, prints reduced observations, and removes the fixtures afterward. All four CLI invocations exited successfully in the executed run.

import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { execFileSync } from 'node:child_process';

const workspace = await mkdtemp(join(tmpdir(), 'rea-html-scope-'));
try {
  const cases = [
    ['invalid-html', Buffer.concat([Buffer.from('<script src="main.js"></script>'), Buffer.from([0xff])]), '{}'],
    ['valid-empty-html', '<html></html>', '{}'],
    ['valid-reference-html', '<script src="main.js"></script>', '{}'],
    ['invalid-json-control', '<html></html>', Buffer.from([0xff])],
  ];
  for (const [name, html, json] of cases) {
    const root = join(workspace, name);
    await mkdir(root);
    await writeFile(join(root, 'package.json'), '{"name":"example","main":"main.js"}');
    await writeFile(join(root, 'main.js'), 'console.log("example");');
    await writeFile(join(root, 'index.html'), html);
    await writeFile(join(root, 'data.json'), json);
    const args = ['analyze-javascript-application', root];
    for (const path of ['package.json', 'main.js', 'index.html', 'data.json'])
      args.push('--source-path', path);
    args.push('--format', 'json');
    const output = execFileSync(process.execPath, [resolve('scripts/rea.mjs'), ...args], {
      env: { ...process.env, REA_LOG_LEVEL: 'silent' }, encoding: 'utf8',
      timeout: 60000, maxBuffer: 16 * 1024 * 1024, stdio: ['ignore', 'pipe', 'pipe'],
    });
    const { graph } = JSON.parse(output).normalized_result;
    const htmlObservation = graph.nodes.flatMap(n => n.observations)
      .find(o => o.properties.path === 'index.html');
    const unknown = graph.nodes.filter(n => n.kind === 'unknown')
      .flatMap(n => n.observations.map(o => ({ path: o.properties.path, operation: o.properties.operation })));
    console.log(JSON.stringify({
      case: name, coverage: graph.coverage.status,
      text_status: htmlObservation.properties.text_status,
      parse_status: htmlObservation.properties.parse_status,
      unknown,
      html_loads: graph.edges.filter(e => e.relation === 'loads' && e.properties.script_path !== undefined).length,
    }));
  }
} finally {
  await rm(workspace, { recursive: true, force: true });
}

Expected behavior

Keep the HTML inventory observation and partial graph coverage, and also record the unavailable HTML-reference processing phase with the HTML file's identity, operation: "parse-html-script-references", artifact-byte authority, and a limitation explaining that the text could not be decoded. This makes the per-file gap discoverable in the same structured form as unreadable JSON and JavaScript inputs.

The applicable processing-scope definition explicitly assigns that operation to HTML. The per-file gap documentation identifies HTML-reference processing as an artifact-byte phase, and the graph evidence contract requires uncertainty and unavailable values to remain explicit. The gap-documentation paragraph discusses interruption; the reproduction here completes normally and exercises decoding failure.

Impact and scope

P2. A consumer that enumerates unknown processing scopes cannot identify skipped HTML reference discovery for this input. Overall coverage remains correctly partial, and the HTML file's text_status survives, so consumers can work around the missing scope by treating invalid-UTF-8 HTML inventory entries as unanalyzed. No data loss or target execution was observed. Runtime reproduction covers invalid UTF-8 in a selected directory-artifact HTML file; other text-acquisition failures and ASAR inputs were not tested.

REA version

6.7.0 source checkout at 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node.js v24.22.0. The prebuilt CLI was executed; no external analysis provider was needed.

Target and artifact context

Local directory artifact containing package.json, main.js, index.html, and data.json, with all four files selected explicitly. The invalid HTML is an ordinary external script tag followed by byte 0xff. Controls use valid empty HTML, valid HTML with the same script reference, and invalid-UTF-8 JSON.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed output from the script above:

{"case":"invalid-html","coverage":"partial","text_status":"invalid-utf8","parse_status":null,"unknown":[],"html_loads":0}
{"case":"valid-empty-html","coverage":"complete","text_status":"included","parse_status":null,"unknown":[],"html_loads":0}
{"case":"valid-reference-html","coverage":"complete","text_status":"included","parse_status":null,"unknown":[],"html_loads":1}
{"case":"invalid-json-control","coverage":"partial","text_status":"included","parse_status":null,"unknown":[{"path":"data.json","operation":"parse-json-module"}],"html_loads":0}

The invalid HTML is the only unavailable text in its case, and its graph contains zero unknown nodes. The JSON control demonstrates that an unavailable-text processing gap is represented for another artifact-byte parser.

Typed outcome or recovery expected

An unknown scope attached to index.html, with unavailable Evidence identifying parse-html-script-references and the decoding limitation. A known inline-script count or inline body range cannot be inferred from unreadable text and is not required to reproduce this defect.

Analysis or suspected cause

Source-derived: the artifact reader preserves fatal UTF-8 decode failure as unavailable text. HTML extraction is guarded by file.text.included, so that file yields no HTML observations. The file-gap projection has branches for selection, interruption, JavaScript and structured-data parsing, and recognized inline scripts, but no branch for selected HTML whose text is unavailable. Overall graph coverage independently counts invalid UTF-8 and therefore becomes partial.

Additional context

The relevant reader and analysis/graph source files are unchanged on fetched origin/main at 705058f3e2df82503f1c601fc464aea7cbb626b8.

Existing reports concerning readable inline script bodies or interruption authority address different paths. This report concerns the missing HTML-reference scope when decoding fails before reference extraction can begin.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5951 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `analyze-javascript-application` returns successful partial Evidence for a selected `index.html` containing invalid UTF-8, but the graph has no unknown processing scope for that file. Its inventory observation preserves `text_status: "invalid-utf8"` and `parse_status: null`; the applicable `parse-html-script-references` operation is absent. A four-case public CLI reproduction confirms this difference. Valid HTML with an external reference produces a load edge. Invalid-UTF-8 HTML produces neither a load nor an unavailable parsing scope. Invalid-UTF-8 JSON, in contrast, produces an explicit `parse-json-module` unknown scope. ### Reproduction Run this script as `node reproduce.mjs` from a built checkout. It creates disposable directory artifacts, invokes the compiled public CLI through its normal dispatcher, prints reduced observations, and removes the fixtures afterward. All four CLI invocations exited successfully in the executed run. ```js import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { execFileSync } from 'node:child_process'; const workspace = await mkdtemp(join(tmpdir(), 'rea-html-scope-')); try { const cases = [ ['invalid-html', Buffer.concat([Buffer.from('<script src="main.js"></script>'), Buffer.from([0xff])]), '{}'], ['valid-empty-html', '<html></html>', '{}'], ['valid-reference-html', '<script src="main.js"></script>', '{}'], ['invalid-json-control', '<html></html>', Buffer.from([0xff])], ]; for (const [name, html, json] of cases) { const root = join(workspace, name); await mkdir(root); await writeFile(join(root, 'package.json'), '{"name":"example","main":"main.js"}'); await writeFile(join(root, 'main.js'), 'console.log("example");'); await writeFile(join(root, 'index.html'), html); await writeFile(join(root, 'data.json'), json); const args = ['analyze-javascript-application', root]; for (const path of ['package.json', 'main.js', 'index.html', 'data.json']) args.push('--source-path', path); args.push('--format', 'json'); const output = execFileSync(process.execPath, [resolve('scripts/rea.mjs'), ...args], { env: { ...process.env, REA_LOG_LEVEL: 'silent' }, encoding: 'utf8', timeout: 60000, maxBuffer: 16 * 1024 * 1024, stdio: ['ignore', 'pipe', 'pipe'], }); const { graph } = JSON.parse(output).normalized_result; const htmlObservation = graph.nodes.flatMap(n => n.observations) .find(o => o.properties.path === 'index.html'); const unknown = graph.nodes.filter(n => n.kind === 'unknown') .flatMap(n => n.observations.map(o => ({ path: o.properties.path, operation: o.properties.operation }))); console.log(JSON.stringify({ case: name, coverage: graph.coverage.status, text_status: htmlObservation.properties.text_status, parse_status: htmlObservation.properties.parse_status, unknown, html_loads: graph.edges.filter(e => e.relation === 'loads' && e.properties.script_path !== undefined).length, })); } } finally { await rm(workspace, { recursive: true, force: true }); } ``` ### Expected behavior Keep the HTML inventory observation and partial graph coverage, and also record the unavailable HTML-reference processing phase with the HTML file's identity, `operation: "parse-html-script-references"`, artifact-byte authority, and a limitation explaining that the text could not be decoded. This makes the per-file gap discoverable in the same structured form as unreadable JSON and JavaScript inputs. The [applicable processing-scope definition](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysisTypes.ts#L126-L140) explicitly assigns that operation to HTML. The [per-file gap documentation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/javascript-artifact-reconstruction.md#L191-L197) identifies HTML-reference processing as an artifact-byte phase, and the [graph evidence contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/javascript-application-graph.md#L148-L159) requires uncertainty and unavailable values to remain explicit. The gap-documentation paragraph discusses interruption; the reproduction here completes normally and exercises decoding failure. ### Impact and scope P2. A consumer that enumerates unknown processing scopes cannot identify skipped HTML reference discovery for this input. Overall coverage remains correctly partial, and the HTML file's `text_status` survives, so consumers can work around the missing scope by treating invalid-UTF-8 HTML inventory entries as unanalyzed. No data loss or target execution was observed. Runtime reproduction covers invalid UTF-8 in a selected directory-artifact HTML file; other text-acquisition failures and ASAR inputs were not tested. ### REA version 6.7.0 source checkout at `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node.js v24.22.0. The prebuilt CLI was executed; no external analysis provider was needed. ### Target and artifact context Local directory artifact containing `package.json`, `main.js`, `index.html`, and `data.json`, with all four files selected explicitly. The invalid HTML is an ordinary external script tag followed by byte `0xff`. Controls use valid empty HTML, valid HTML with the same script reference, and invalid-UTF-8 JSON. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Observed output from the script above: ```jsonl {"case":"invalid-html","coverage":"partial","text_status":"invalid-utf8","parse_status":null,"unknown":[],"html_loads":0} {"case":"valid-empty-html","coverage":"complete","text_status":"included","parse_status":null,"unknown":[],"html_loads":0} {"case":"valid-reference-html","coverage":"complete","text_status":"included","parse_status":null,"unknown":[],"html_loads":1} {"case":"invalid-json-control","coverage":"partial","text_status":"included","parse_status":null,"unknown":[{"path":"data.json","operation":"parse-json-module"}],"html_loads":0} ``` The invalid HTML is the only unavailable text in its case, and its graph contains zero unknown nodes. The JSON control demonstrates that an unavailable-text processing gap is represented for another artifact-byte parser. ### Typed outcome or recovery expected An unknown scope attached to `index.html`, with unavailable Evidence identifying `parse-html-script-references` and the decoding limitation. A known inline-script count or inline body range cannot be inferred from unreadable text and is not required to reproduce this defect. ### Analysis or suspected cause Source-derived: the [artifact reader](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/artifacts/javascript/JavaScriptArtifactFiles.ts#L358-L372) preserves fatal UTF-8 decode failure as unavailable text. [HTML extraction](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactAnalysis.ts#L389-L399) is guarded by `file.text.included`, so that file yields no HTML observations. The [file-gap projection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactGraphStructure.ts#L222-L349) has branches for selection, interruption, JavaScript and structured-data parsing, and recognized inline scripts, but no branch for selected HTML whose text is unavailable. [Overall graph coverage](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/application/javascript/JavaScriptArtifactGraphBuilder.ts#L242-L255) independently counts invalid UTF-8 and therefore becomes partial. ### Additional context The relevant reader and analysis/graph source files are unchanged on fetched `origin/main` at `705058f3e2df82503f1c601fc464aea7cbb626b8`. Existing reports concerning readable inline script bodies or interruption authority address different paths. This report concerns the missing HTML-reference scope when decoding fails before reference extraction can begin.

bsakaguchi closed this issue

2026-10-31 00:42:27 +00:00

bsakaguchi referenced this issue

2026-11-01 21:10:54 +00:00

[GH-ISSUE #3641] [Feature] USB protocol reverse engineering: descriptors, HID report schemas, and recorded transaction reconstruction #1363

bsakaguchi referenced this issue

2026-11-01 21:10:57 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471