[PR #1360] [MERGED] fix(cli): stream JSON inputs beyond the whole-document string limit #2826

New issue

Closed

opened 2026-10-17 21:10:44 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-17 21:10:44 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1360
Author: @N0zoM1z0
Created: 10/9/2026
Status: ✅ Merged
Merged: 10/9/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/streamed-cli-json-1331

📝 Commits (10+)

9591815 fix(cli): parse JSON files without a whole-document string
16e58c5 fix(cli): preserve runtime JSON string-limit diagnostics
26a1a38 fix(cli): integrate string-limit follow-up with current main
2cbcf1d perf(cli): retain native JSON parsing for bounded file reads
3868c9b fix(cli): combine streaming input with typed JSON resource diagnostics
fc281c2 chore: integrate current CLI and setup fixes before JSON verification
7a22868 test(json): check actionable oversized Evidence recovery
6e920eb fix(json): decode large UTF-8 inputs by character length
612eae6 fix(json): bound string assembly copies and preserve usable recovery advice
773df0c test(json): reject truncated UTF-8 after large input decode

📊 Changes

10 files changed (+554 additions, -196 deletions)

View changed files

📝 docs/javascript-artifact-reconstruction.md (+15 -5)
📝 package-lock.json (+22 -0)
📝 package.json (+1 -0)
📝 src/application/JsonFiles.ts (+13 -12)
📝 src/application/RegularFileRead.ts (+18 -5)
📝 src/application/Utf8JsonInput.ts (+5 -2)
➕ src/cliJsonFile.ts (+261 -0)
📝 src/cliJsonInput.ts (+20 -59)
📝 tests/boundary/cli/jsonInputSize.test.ts (+104 -14)
📝 tests/boundary/filesystem/cliJsonInput.test.ts (+95 -99)

📄 Description

CLI workflows can now consume a JSON file larger than V8's whole-document string limit. Larger inputs use strict incremental UTF-8 decoding and the standard JSON tokenizer/assembler on one verified regular-file handle; small files retain native parsing. Files growing after admission continue from their existing handle without losing the prefix. The parsed value remains complete.

String fragments are coalesced, with native UTF-16 length checks and a heap headroom check before potentially large flattening allocations. Resource errors preserve the selected path, observed counts and actionable recovery. Evidence bundle/snapshot readers retain #1350's character-based decoder diagnostics and now reuse the verified regular-file reader.

Validation:

108 focused filesystem, CLI/status and Evidence bundle cases passed; after integrating the final #1350 decoder changes, all 40 affected cases passed again. These cover strict JSON/UTF-8, chunked Unicode and surrogate escapes, long keys, duplicates and __proto__, cancellation, regular-file admission and CLI failure status.
Actual CLI analysis and follow-up tracing of a valid 768,736,012-byte request match the smaller request's Evidence identity and normalized result. The previous whole-document decoder fails on those bytes with ERR_STRING_TOO_LONG. A compiled CLI regression also checks full result parity above the running engine's string limit.
An oversized individual string returns a typed memory constraint under a 768 MiB heap, with about 649 MiB sampled family RSS, instead of the prototype's V8 abort.
On native Windows Node 22.19, the final #1350 decoder accepts 536,870,894 UTF-8 bytes representing 268,435,446 UTF-16 units, preserves the complete value's digest, and rejects an incomplete UTF-8 suffix. Two CPUs, 768 MiB heap, 1.36 GiB sampled family RSS, no surviving owned processes. This is a decoder-boundary proof, not Windows CLI/package or provider acceptance.
Small-file benchmarks retain native-parser latency after correcting a streaming-only prototype regression. Static checks and docs:check pass.

The large tracing fixture uses whitespace padding; it does not establish handling of a dense 733 MB Evidence object. The whole value still requires memory, and this does not guarantee recovery from every V8 allocation failure. No real Hopper/Ghidra verification is needed or claimed.

Refs #1331, #1065. Follow-up to #1350.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1360 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/9/2026 **Status:** ✅ Merged **Merged:** 10/9/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/streamed-cli-json-1331` --- ### 📝 Commits (10+) - [`9591815`](https://github.com/morluto/rea/commit/9591815fe284ce95188faa79ee763db0f2f1c306) fix(cli): parse JSON files without a whole-document string - [`16e58c5`](https://github.com/morluto/rea/commit/16e58c5e5a68ac00503f566e06ff517729e8314e) fix(cli): preserve runtime JSON string-limit diagnostics - [`26a1a38`](https://github.com/morluto/rea/commit/26a1a38632e10225b87e628e8afcf23513a30dd6) fix(cli): integrate string-limit follow-up with current main - [`2cbcf1d`](https://github.com/morluto/rea/commit/2cbcf1df2d89c4231ef803b7d2dfa2b45dc44d45) perf(cli): retain native JSON parsing for bounded file reads - [`3868c9b`](https://github.com/morluto/rea/commit/3868c9b59dff2efdba04b1dcc8e30ac046902da2) fix(cli): combine streaming input with typed JSON resource diagnostics - [`fc281c2`](https://github.com/morluto/rea/commit/fc281c20111f28849543fb6c8a628ba5f2bcf021) chore: integrate current CLI and setup fixes before JSON verification - [`7a22868`](https://github.com/morluto/rea/commit/7a228686da31994a556fceb762347b7cdc0b4dd0) test(json): check actionable oversized Evidence recovery - [`6e920eb`](https://github.com/morluto/rea/commit/6e920ebf2160676ef0dd2bfca65a08e07338018e) fix(json): decode large UTF-8 inputs by character length - [`612eae6`](https://github.com/morluto/rea/commit/612eae61781399b3b0d9f9ad9fe060e28ee5760c) fix(json): bound string assembly copies and preserve usable recovery advice - [`773df0c`](https://github.com/morluto/rea/commit/773df0cefe184f3948ce7230f9921c402582ded0) test(json): reject truncated UTF-8 after large input decode ### 📊 Changes **10 files changed** (+554 additions, -196 deletions) <details> <summary>View changed files</summary> 📝 `docs/javascript-artifact-reconstruction.md` (+15 -5) 📝 `package-lock.json` (+22 -0) 📝 `package.json` (+1 -0) 📝 `src/application/JsonFiles.ts` (+13 -12) 📝 `src/application/RegularFileRead.ts` (+18 -5) 📝 `src/application/Utf8JsonInput.ts` (+5 -2) ➕ `src/cliJsonFile.ts` (+261 -0) 📝 `src/cliJsonInput.ts` (+20 -59) 📝 `tests/boundary/cli/jsonInputSize.test.ts` (+104 -14) 📝 `tests/boundary/filesystem/cliJsonInput.test.ts` (+95 -99) </details> ### 📄 Description CLI workflows can now consume a JSON file larger than V8's whole-document string limit. Larger inputs use strict incremental UTF-8 decoding and the standard JSON tokenizer/assembler on one verified regular-file handle; small files retain native parsing. Files growing after admission continue from their existing handle without losing the prefix. The parsed value remains complete. String fragments are coalesced, with native UTF-16 length checks and a heap headroom check before potentially large flattening allocations. Resource errors preserve the selected path, observed counts and actionable recovery. Evidence bundle/snapshot readers retain #1350's character-based decoder diagnostics and now reuse the verified regular-file reader. Validation: - 108 focused filesystem, CLI/status and Evidence bundle cases passed; after integrating the final #1350 decoder changes, all 40 affected cases passed again. These cover strict JSON/UTF-8, chunked Unicode and surrogate escapes, long keys, duplicates and `__proto__`, cancellation, regular-file admission and CLI failure status. - Actual CLI analysis and follow-up tracing of a valid 768,736,012-byte request match the smaller request's Evidence identity and normalized result. The previous whole-document decoder fails on those bytes with `ERR_STRING_TOO_LONG`. A compiled CLI regression also checks full result parity above the running engine's string limit. - An oversized individual string returns a typed memory constraint under a 768 MiB heap, with about 649 MiB sampled family RSS, instead of the prototype's V8 abort. - On native Windows Node 22.19, the final #1350 decoder accepts 536,870,894 UTF-8 bytes representing 268,435,446 UTF-16 units, preserves the complete value's digest, and rejects an incomplete UTF-8 suffix. Two CPUs, 768 MiB heap, 1.36 GiB sampled family RSS, no surviving owned processes. This is a decoder-boundary proof, not Windows CLI/package or provider acceptance. - Small-file benchmarks retain native-parser latency after correcting a streaming-only prototype regression. Static checks and `docs:check` pass. The large tracing fixture uses whitespace padding; it does not establish handling of a dense 733 MB Evidence object. The whole value still requires memory, and this does not guarantee recovery from every V8 allocation failure. No real Hopper/Ghidra verification is needed or claimed. Refs #1331, #1065. Follow-up to #1350. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-17 21:10:44 +00:00