[PR #1024] [MERGED] fix(cli): stream large JavaScript JSON results #2589

New issue

Closed

opened 2026-10-17 21:07:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:07:55 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1024
Author: @N0zoM1z0
Created: 10/7/2026
Status: ✅ Merged
Merged: 10/7/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/stream-javascript-json

📝 Commits (8)

bbadb3c fix(javascript): report application analysis progress on the CLI
91fdff6 test(cli): preserve progress in filtered output checks
251dd81 fix(cli): stream large JavaScript JSON results
acf20bf Merge remote-tracking branch 'origin/main' into fix/javascript-analysis-progress
956190b Merge remote-tracking branch 'origin/main' into fix/javascript-analysis-progress
74a72ed Merge branch 'fix/javascript-analysis-progress' into fix/stream-javascript-json
d3f6e2e Merge remote-tracking branch 'origin/main' into fix/stream-javascript-json
7a46cfc test(cli): include streaming output in dispatcher fixture

📊 Changes

16 files changed (+781 additions, -34 deletions)

View changed files

📝 docs/javascript-artifact-reconstruction.md (+13 -5)
📝 docs/testing.md (+12 -0)
📝 package.json (+1 -0)
📝 scripts/rea.mjs (+11 -3)
➕ scripts/verify/javascript/output.mjs (+134 -0)
📝 src/cli.ts (+4 -2)
📝 src/cli/coreAnalysisCommands.ts (+3 -1)
📝 src/cli/coreBinaryCommands.ts (+25 -7)
📝 src/cli/electronCommands.ts (+15 -6)
📝 src/cli/javascriptApplicationAnalysis.ts (+15 -2)
➕ src/cli/jsonOutput.ts (+149 -0)
➕ src/cli/streamedJsonOutput.ts (+84 -0)
📝 src/cliOutput.ts (+22 -8)
📝 tests/boundary/cli/dispatcher.test.ts (+5 -0)
➕ tests/boundary/cli/javascriptApplicationJsonOutput.test.ts (+118 -0)
➕ tests/boundary/cli/jsonOutput.test.ts (+170 -0)

📄 Description

Large JavaScript application analysis can produce valid Evidence that exceeds V8's single-string limit during Incur JSON formatting. The CLI now writes JSON and JSONL incrementally for both analyze-javascript-application and directory/ASAR analyze, preserving filtering, full-output envelopes, Unicode, complete Evidence, and existing output ordering. Writes await destination completion; output errors preserve the cause on stderr without appending a second document. Application and MCP contracts remain unchanged.

Closes #623.

Validation:

Reproduced native RangeError: Invalid string length in the pinned Incur formatter after successfully constructing valid Evidence.
An opt-in real-Incur format-boundary verifier writes and independently hashes JSON (537,035,518 bytes) and JSONL (536,961,605 bytes), both larger than V8's 536,870,888-code-unit single-string limit. Linux and Windows runs passed and removed the temporary outputs. The verifier checks available disk space and keeps a 1-GiB reserve.
Windows verification used an existing Node 22.19.0, two CPUs, lowered priority, a 1-GiB heap, and a bounded owned workspace. The final integrated run peaked near 96 MiB working set for large-output verification and 418 MiB for the real 24.2-MB ASAR fixture. No owned processes remained.
The real ASAR fixture contains 61 compiled application modules and an inert resource pad. On the original frozen baseline, streaming preserved output bytes exactly and reduced peak family RSS from approximately 876 to 689 MiB, with elapsed time increasing from approximately 9.4 to 12.7 seconds.
A real Linux stdout-to-/dev/full run exited 1 and preserved ENOSPC on stderr, without writing a second JSON document.
111 focused tests across eight suites pass, including golden formats, actual CLI/MCP, filters/slices, Unicode, backpressure, delayed completion, and failing destinations. Static, formatting, and generated-document checks pass. Installed-package verification passes with 133 tools and six prompts. Both engine-limit output checks and canonical-digest verification were repeated after integration with the merged progress fix #1008. CI identified an incomplete dispatcher fixture; its new output module is now included and both routing/recovery cases pass locally.

JSON and JSONL output avoid a document-sized string; analysis still needs memory for its source and graphs. Other formats, token-count requests, and MCP text serialization retain their existing formatting boundaries. The reporter's original Obsidian/SPA assets were not available. No real Hopper/Ghidra verification was performed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1024 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/7/2026 **Status:** ✅ Merged **Merged:** 10/7/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/stream-javascript-json` --- ### 📝 Commits (8) - [`bbadb3c`](https://github.com/morluto/rea/commit/bbadb3c6a7ee357169598c4bfafbb780e67f9191) fix(javascript): report application analysis progress on the CLI - [`91fdff6`](https://github.com/morluto/rea/commit/91fdff63c8a25ec7ff9a6f24f729a1fd8acbcc91) test(cli): preserve progress in filtered output checks - [`251dd81`](https://github.com/morluto/rea/commit/251dd819a2b78e05d037c896a6ddf217703b5a34) fix(cli): stream large JavaScript JSON results - [`acf20bf`](https://github.com/morluto/rea/commit/acf20bf1fad82e94d58d2d7074bb3ebd549eac01) Merge remote-tracking branch 'origin/main' into fix/javascript-analysis-progress - [`956190b`](https://github.com/morluto/rea/commit/956190b07b793d57a88c8466ddc4282ab2882837) Merge remote-tracking branch 'origin/main' into fix/javascript-analysis-progress - [`74a72ed`](https://github.com/morluto/rea/commit/74a72ed0b2f160d17bd889be8ef9a9d8a4147343) Merge branch 'fix/javascript-analysis-progress' into fix/stream-javascript-json - [`d3f6e2e`](https://github.com/morluto/rea/commit/d3f6e2ed2a89314c4629b80baf9632ab6ecb8ab6) Merge remote-tracking branch 'origin/main' into fix/stream-javascript-json - [`7a46cfc`](https://github.com/morluto/rea/commit/7a46cfc0fa53b5a5b476f02a6cef3ae081178d3b) test(cli): include streaming output in dispatcher fixture ### 📊 Changes **16 files changed** (+781 additions, -34 deletions) <details> <summary>View changed files</summary> 📝 `docs/javascript-artifact-reconstruction.md` (+13 -5) 📝 `docs/testing.md` (+12 -0) 📝 `package.json` (+1 -0) 📝 `scripts/rea.mjs` (+11 -3) ➕ `scripts/verify/javascript/output.mjs` (+134 -0) 📝 `src/cli.ts` (+4 -2) 📝 `src/cli/coreAnalysisCommands.ts` (+3 -1) 📝 `src/cli/coreBinaryCommands.ts` (+25 -7) 📝 `src/cli/electronCommands.ts` (+15 -6) 📝 `src/cli/javascriptApplicationAnalysis.ts` (+15 -2) ➕ `src/cli/jsonOutput.ts` (+149 -0) ➕ `src/cli/streamedJsonOutput.ts` (+84 -0) 📝 `src/cliOutput.ts` (+22 -8) 📝 `tests/boundary/cli/dispatcher.test.ts` (+5 -0) ➕ `tests/boundary/cli/javascriptApplicationJsonOutput.test.ts` (+118 -0) ➕ `tests/boundary/cli/jsonOutput.test.ts` (+170 -0) </details> ### 📄 Description Large JavaScript application analysis can produce valid Evidence that exceeds V8's single-string limit during Incur JSON formatting. The CLI now writes JSON and JSONL incrementally for both `analyze-javascript-application` and directory/ASAR `analyze`, preserving filtering, full-output envelopes, Unicode, complete Evidence, and existing output ordering. Writes await destination completion; output errors preserve the cause on stderr without appending a second document. Application and MCP contracts remain unchanged. Closes #623. Validation: - Reproduced native `RangeError: Invalid string length` in the pinned Incur formatter after successfully constructing valid Evidence. - An opt-in real-Incur format-boundary verifier writes and independently hashes JSON (537,035,518 bytes) and JSONL (536,961,605 bytes), both larger than V8's 536,870,888-code-unit single-string limit. Linux and Windows runs passed and removed the temporary outputs. The verifier checks available disk space and keeps a 1-GiB reserve. - Windows verification used an existing Node 22.19.0, two CPUs, lowered priority, a 1-GiB heap, and a bounded owned workspace. The final integrated run peaked near 96 MiB working set for large-output verification and 418 MiB for the real 24.2-MB ASAR fixture. No owned processes remained. - The real ASAR fixture contains 61 compiled application modules and an inert resource pad. On the original frozen baseline, streaming preserved output bytes exactly and reduced peak family RSS from approximately 876 to 689 MiB, with elapsed time increasing from approximately 9.4 to 12.7 seconds. - A real Linux stdout-to-`/dev/full` run exited 1 and preserved `ENOSPC` on stderr, without writing a second JSON document. - 111 focused tests across eight suites pass, including golden formats, actual CLI/MCP, filters/slices, Unicode, backpressure, delayed completion, and failing destinations. Static, formatting, and generated-document checks pass. Installed-package verification passes with 133 tools and six prompts. Both engine-limit output checks and canonical-digest verification were repeated after integration with the merged progress fix #1008. CI identified an incomplete dispatcher fixture; its new output module is now included and both routing/recovery cases pass locally. JSON and JSONL output avoid a document-sized string; analysis still needs memory for its source and graphs. Other formats, token-count requests, and MCP text serialization retain their existing formatting boundaries. The reporter's original Obsidian/SPA assets were not available. No real Hopper/Ghidra verification was performed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:07:55 +00:00