morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #623] analyze_javascript_application fails on large targets: RangeError: Invalid string length masked as reason 'io' #300

New issue

Closed

opened 2026-10-15 18:23:37 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-15 18:23:37 +00:00

Owner

Originally created by @Ali-Jemo on GitHub (Oct 5, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/623

Originally assigned to: @N0zoM1z0 on GitHub.

Summary

analyze_javascript_application fails on large targets with a bare {"reason":"io"} error. The underlying cause is RangeError: Invalid string length, thrown when the canonical JSON form of a whole graph exceeds the engine's maximum string length.

Two independent sites materialize the entire value as one string before hashing it:

dist/domain/javascriptSemanticGraph.js — digestCanonical(semantic) computing graph_id
dist/domain/evidence.js — computeEvidenceId() computing evidence_id

Both call canonicalize(value), which builds one contiguous string via Array.prototype.join, then pass it to createHash().update().

Reproduction target: rea-agents's own dist/ directory (719 JS files, ~5.7 MB of source).

Impact

Analysis cannot be run at all on large JavaScript/Electron applications — precisely the targets REA is designed for. The error is also unhelpful: JavaScriptApplicationService.js catches every non-Zod exception and rethrows a bare ArtifactOperationError(OPERATION, "io"), discarding the real cause.

Reproduction

npm install --global rea-agents
NODE_OPTIONS=--max-old-space-size=16384 \
  rea analyze-javascript-application \
    "$(npm root -g)/rea-agents/dist" \
    --artifact-format directory --format json

Observed (before fix), ~121 s:

{
  "error": "JavaScript application analysis failed",
  "code": "artifact_operation_failed",
  "details": { "operation": "analyze_javascript_application", "reason": "io" }
}

With the real cause restored:

RangeError: Invalid string length
    at canonicalize (canonicalize/lib/canonicalize.js:44:24)
    at canonicalJson (dist/domain/javascriptSemanticGraph.js:7:21)
    at digestCanonical (dist/domain/javascriptSemanticGraph.js:12:64)
    at createJavaScriptSemanticGraph (dist/domain/javascriptSemanticGraph.js:354:27)
    at buildJavaScriptSemanticGraph (dist/application/JavaScriptSemanticGraphBuilder.js:27:12)

Measurements

Per-directory, showing superlinear growth in the semantic projection:

directory	source	semantic nodes	semantic relations	time	peak heap
`cli`	0.04 MB	3,194	3,471	3.3 s	111 MB
`server`	0.14 MB	12,259	13,650	9.1 s	388 MB
`browser`	0.26 MB	24,972	21,891	17.7 s	525 MB
`application`	0.84 MB	71,561	67,398	48 s	2.1 GB
`domain`	1.39 MB	131,007	118,579	87 s	3.1 GB

Total source for the whole tree is 2.91 MB; the serialized semantic graph is ~306 MB, but the process holds >512 MB transiently.

Suggested fix

Stream the canonical form into the hash instead of materializing it, preserving byte-identical output so graph_id, evidence_id, and analysis-snapshot cache keys stay stable:

const streamCanonical = (hash, object, seen) => {
  // emit the same bytes canonicalize() produces, piecewise
};
const digestCanonical = (value) => {
  const hash = createHash("sha256");
  streamCanonical(hash, value, new Set());
  return hash.digest("hex");
};

Verified byte-identical against canonicalize for: nested objects, arrays, undefined/symbol handling, sparse arrays, toJSON, Date, circular-reference rejection, and circular/NaN/Infinity errors. Digests for existing small graphs are unchanged.

Separately, JavaScriptApplicationService.js should not collapse arbitrary exceptions into "io"; a RangeError should surface as an engine-limit/protocol error so this is diagnosable without a debugger.

Even after the analysis completes, printing an unfiltered full result fails at incur (node_modules/incur/dist/internal/json.js:3), which calls JSON.stringify on the entire output envelope:

RangeError: Invalid string length
    at Module.stringify (incur/dist/internal/json.js:3:17)
    at Module.format (incur/dist/Formatter.js:18:21)
    at write (incur/dist/Cli.js:984:37)

--filter-output works because filtering happens before formatting, but a full result should be printable. Incremental/streaming JSON writing in the formatter would fix this.

Environment

rea-agents 3.2.1 (npm global)
Node v26.8.1
Linux x64 (Omarchy/Arch-based)

Originally created by @Ali-Jemo on GitHub (Oct 5, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/623 Originally assigned to: @N0zoM1z0 on GitHub. ## Summary `analyze_javascript_application` fails on large targets with a bare `{"reason":"io"}` error. The underlying cause is `RangeError: Invalid string length`, thrown when the canonical JSON form of a whole graph exceeds the engine's maximum string length. Two independent sites materialize the entire value as one string before hashing it: 1. `dist/domain/javascriptSemanticGraph.js` — `digestCanonical(semantic)` computing `graph_id` 2. `dist/domain/evidence.js` — `computeEvidenceId()` computing `evidence_id` Both call `canonicalize(value)`, which builds one contiguous string via `Array.prototype.join`, then pass it to `createHash().update()`. Reproduction target: `rea-agents`'s own `dist/` directory (719 JS files, ~5.7 MB of source). ## Impact Analysis cannot be run at all on large JavaScript/Electron applications — precisely the targets REA is designed for. The error is also unhelpful: `JavaScriptApplicationService.js` catches every non-Zod exception and rethrows a bare `ArtifactOperationError(OPERATION, "io")`, discarding the real cause. ## Reproduction ```bash npm install --global rea-agents NODE_OPTIONS=--max-old-space-size=16384 \ rea analyze-javascript-application \ "$(npm root -g)/rea-agents/dist" \ --artifact-format directory --format json ``` Observed (before fix), ~121 s: ```json { "error": "JavaScript application analysis failed", "code": "artifact_operation_failed", "details": { "operation": "analyze_javascript_application", "reason": "io" } } ``` With the real cause restored: ``` RangeError: Invalid string length at canonicalize (canonicalize/lib/canonicalize.js:44:24) at canonicalJson (dist/domain/javascriptSemanticGraph.js:7:21) at digestCanonical (dist/domain/javascriptSemanticGraph.js:12:64) at createJavaScriptSemanticGraph (dist/domain/javascriptSemanticGraph.js:354:27) at buildJavaScriptSemanticGraph (dist/application/JavaScriptSemanticGraphBuilder.js:27:12) ``` ### Measurements Per-directory, showing superlinear growth in the semantic projection: | directory | source | semantic nodes | semantic relations | time | peak heap | |---|---|---|---|---|---| | `cli` | 0.04 MB | 3,194 | 3,471 | 3.3 s | 111 MB | | `server` | 0.14 MB | 12,259 | 13,650 | 9.1 s | 388 MB | | `browser` | 0.26 MB | 24,972 | 21,891 | 17.7 s | 525 MB | | `application` | 0.84 MB | 71,561 | 67,398 | 48 s | 2.1 GB | | `domain` | 1.39 MB | 131,007 | 118,579 | 87 s | 3.1 GB | Total source for the whole tree is 2.91 MB; the serialized semantic graph is ~306 MB, but the process holds >512 MB transiently. ## Suggested fix Stream the canonical form into the hash instead of materializing it, preserving byte-identical output so `graph_id`, `evidence_id`, and analysis-snapshot cache keys stay stable: ```js const streamCanonical = (hash, object, seen) => { // emit the same bytes canonicalize() produces, piecewise }; const digestCanonical = (value) => { const hash = createHash("sha256"); streamCanonical(hash, value, new Set()); return hash.digest("hex"); }; ``` Verified byte-identical against `canonicalize` for: nested objects, arrays, `undefined`/symbol handling, sparse arrays, `toJSON`, `Date`, circular-reference rejection, and circular/`NaN`/`Infinity` errors. Digests for existing small graphs are unchanged. Separately, `JavaScriptApplicationService.js` should not collapse arbitrary exceptions into `"io"`; a `RangeError` should surface as an engine-limit/protocol error so this is diagnosable without a debugger. ## Related: CLI output path Even after the analysis completes, printing an unfiltered full result fails at `incur` (`node_modules/incur/dist/internal/json.js:3`), which calls `JSON.stringify` on the entire output envelope: ``` RangeError: Invalid string length at Module.stringify (incur/dist/internal/json.js:3:17) at Module.format (incur/dist/Formatter.js:18:21) at write (incur/dist/Cli.js:984:37) ``` `--filter-output` works because filtering happens before formatting, but a full result should be printable. Incremental/streaming JSON writing in the formatter would fix this. ## Environment - rea-agents 3.2.1 (npm global) - Node v26.8.1 - Linux x64 (Omarchy/Arch-based)

bsakaguchi

2026-10-15 18:23:37 +00:00

closed this issue
added the
area: runtime

bug
labels

bsakaguchi commented

2026-10-15 18:23:38 +00:00

Author

Owner

@morluto commented on GitHub (Oct 5, 2026):

Hey this should be fixed, please update to rea 4.0.1

@morluto commented on GitHub (Oct 5, 2026): Hey this should be fixed, please update to rea 4.0.1

bsakaguchi commented

2026-10-15 18:23:38 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 6, 2026):

Reconfirmed both aggregate hash failures on current main/4.0.1. Follow-up #646 hashes canonical bytes incrementally and preserves unexpected analysis failure diagnostics. Verified a 537,921,040-byte canonical value and actual CLI/stdio MCP results with matching Evidence IDs. Whole-result formatter/transport string limits remain separate, so this does not close the issue.

@N0zoM1z0 commented on GitHub (Oct 6, 2026): Reconfirmed both aggregate hash failures on current main/4.0.1. Follow-up #646 hashes canonical bytes incrementally and preserves unexpected analysis failure diagnostics. Verified a 537,921,040-byte canonical value and actual CLI/stdio MCP results with matching Evidence IDs. Whole-result formatter/transport string limits remain separate, so this does not close the issue.

bsakaguchi commented

2026-10-15 18:23:39 +00:00

Author

Owner

@DariuszCiesielski commented on GitHub (Oct 7, 2026):

Additional data point, much smaller target than the ones above.

Target: a Vite/React production dist/ of 2.9 MB total; after excluding one chunk (see the separate /label issue I filed), the remaining set with a single ~800 KB index-*.js chunk fails after ~140 s with:

RangeError: Invalid string length at JSON.stringify (incur/dist/internal/json.js:3)

rea-agents@4.1.0, npx -y, macOS Apple Silicon, Node 25.6.1 (outside supported range; noting for completeness).
Here the throw site is the final JSON.stringify of the result (via incur), not canonicalize() — so even after the hashing sites are streamed, the serialized output itself can exceed V8's max string length. For reference, 12 small chunks (312 KB) that do succeed produce a 254 MB JSON result (~800× input).
--token-limit is rejected for this command ("Token windows cannot preserve structured output"), so there is no way to shrink the output from the CLI.

So the threshold is far below "large Electron app": one ordinary SPA chunk under 1 MB is enough.

@DariuszCiesielski commented on GitHub (Oct 7, 2026): Additional data point, much smaller target than the ones above. - Target: a Vite/React production `dist/` of **2.9 MB** total; after excluding one chunk (see the separate `/label` issue I filed), the remaining set with a single **~800 KB `index-*.js`** chunk fails after ~140 s with: ``` RangeError: Invalid string length at JSON.stringify (incur/dist/internal/json.js:3) ``` - `rea-agents@4.1.0`, `npx -y`, macOS Apple Silicon, Node 25.6.1 (outside supported range; noting for completeness). - Here the throw site is the final `JSON.stringify` of the result (via `incur`), not `canonicalize()` — so even after the hashing sites are streamed, the serialized output itself can exceed V8's max string length. For reference, 12 small chunks (312 KB) that *do* succeed produce a **254 MB** JSON result (~800× input). - `--token-limit` is rejected for this command ("Token windows cannot preserve structured output"), so there is no way to shrink the output from the CLI. So the threshold is far below "large Electron app": one ordinary SPA chunk under 1 MB is enough.

bsakaguchi commented

2026-10-15 18:23:39 +00:00

Author

Owner

@morluto commented on GitHub (Oct 7, 2026):

Additional reproduction on 2026-10-07: REA's existing built dist/ tree remains silent beyond a 60-second observation window on supported Node v22.22.2, macOS 26.3 arm64.

Checkout: faaee693d5bb4bb957387ebd0f4338513c4c914a, with concurrent source changes; this run used existing built artifacts. Direct command:

/absolute/path/to/node-v22.22.2 scripts/rea.mjs \
  analyze-javascript-application dist \
  --format json --filter-output graph.coverage

Observed: zero stdout/stderr bytes before an external 60-second watchdog terminated the process. The supervised attempt allowed up to 4 GiB process-tree RSS; it ended on the time budget, not a reported memory-limit violation. A CPU-profile attempt produced no native profile before termination.

This is an externally interrupted attempt, not a reproduction of the old terminal RangeError. Current source already streams canonical hashing; no conclusion about an eventual native failure is asserted. Output filtering did not make the full analysis finish within the observation window.

The dist/server control (59 JS files, 143,782 source bytes) completed in 5.45 seconds and produced a native CPU profile. Both runs were concurrent, so their elapsed times do not establish an isolated scaling benchmark. The completed control identifies per-token canonical digest/crypto updates as an optimization candidate, tracked separately in #904. Related silent-analysis behavior: #746.

@morluto commented on GitHub (Oct 7, 2026): Additional reproduction on 2026-10-07: REA's existing built `dist/` tree remains silent beyond a 60-second observation window on supported Node v22.22.2, macOS 26.3 arm64. Checkout: `faaee693d5bb4bb957387ebd0f4338513c4c914a`, with concurrent source changes; this run used existing built artifacts. Direct command: ```console /absolute/path/to/node-v22.22.2 scripts/rea.mjs \ analyze-javascript-application dist \ --format json --filter-output graph.coverage ``` Observed: zero stdout/stderr bytes before an external 60-second watchdog terminated the process. The supervised attempt allowed up to 4 GiB process-tree RSS; it ended on the time budget, not a reported memory-limit violation. A CPU-profile attempt produced no native profile before termination. This is an externally interrupted attempt, not a reproduction of the old terminal RangeError. Current source already streams canonical hashing; no conclusion about an eventual native failure is asserted. Output filtering did not make the full analysis finish within the observation window. The `dist/server` control (59 JS files, 143,782 source bytes) completed in 5.45 seconds and produced a native CPU profile. Both runs were concurrent, so their elapsed times do not establish an isolated scaling benchmark. The completed control identifies per-token canonical digest/crypto updates as an optimization candidate, tracked separately in #904. Related silent-analysis behavior: #746.

bsakaguchi referenced this issue

2026-10-15 18:28:29 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-16 21:12:39 +00:00

[PR #300] [MERGED] docs(managed): align normalized CIL claims with shipped v1 semantics #1936