[PR #1694] [MERGED] perf(evidence): fuse streamed JSON token handling #4760

New issue

Closed

opened 2026-10-31 00:54:45 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:54:45 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1694
Author: @N0zoM1z0
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/evidence-json-stream-allocation

📝 Commits (2)

9d57542 perf(evidence): fuse streamed JSON token handling
0efc3a3 fix(build): declare the streamed JSON chain dependency

📊 Changes

4 files changed (+189 additions, -128 deletions)

View changed files

📝 package-lock.json (+1 -0)
📝 package.json (+1 -0)
📝 src/application/JsonInputFile.ts (+130 -128)
📝 tests/boundary/cli/jsonInputSize.test.ts (+57 -0)

📄 Description

Large evidence JSON files currently create async iterator and stream work for every parser token. On an 18.54 MB snapshot exported from a real Ghidra analysis, that work dominates the sampled allocation window.

Fuse the upstream parser, scalar packing, heap admission, and upstream value assembler into synchronous token stages. File reads remain asynchronous and cancellation-aware. Keep the existing capacity estimates, UTF-8 validation, scalar limits, file identity checks, and typed failures. Declare the existing stream-chain 4.2.6 as a direct runtime dependency. Add a public CLI container-expansion regression plus a same-process successful read after rejection.

Validation:

Final head 0efc3a3d passed 30 focused tests, including malformed input, cancellation, long strings, the existing 512 MiB whitespace control, and the new container admission/recovery case. check:fast, Knip, and whole-repository formatting passed on the final head. A real-evidence Linux CLI/MCP replay also passed on the benchmarked candidate; its reader source and compiled file are byte-identical to the final head.
Actual Flameox 0.2.9 ran five paired blocks under two CPUs and a 768 MiB Node heap. All ten workloads passed a separate oracle checking the complete canonical snapshot digest and retained Evidence identity. Median paired wall-time difference: −3.63 s; separate baseline/candidate medians: 7.64 / 3.62 s. The workload includes Node startup, production snapshot reading/validation, full canonical hashing, and exit. Flameox classifies this as a descriptive practical improvement; these are local measurements, not a universal latency guarantee.
Scoped V8 sampling of production snapshot reading, analyzed with Flameox, estimated cumulative allocations of 10.77 / 3.17 GB before/after (single capture each). This includes collected objects and excludes module initialization; it is not peak RSS or a leak measurement.
Integrated #1687 and #1691 without changing the measured reader source or compiled file. Final-head CI run 38110917770 passed 31/31 jobs, including Windows package verification.
Actual Windows Node 22.19 / SDK 2.3.1 verification passed with the full source-built 6.4.0 archive, standard CLI/MCP entrypoints, existing matching locked production dependencies, and same-head CI native controls. A fresh real Ghidra 12.1.4 analysis saved a current-profile snapshot; adding 9 MiB of JSON whitespace forced streaming. Restarted MCP, verified the old reference was unavailable before import, restored exact retained Evidence/view identity, and checked CLI/MCP summary parity. JVM 512 MiB / Node heap 768 MiB / two CPUs; target unchanged, temporary directories empty, no remaining observed owned descendants. This is source-built package acceptance, not a published npm release or Windows large-payload benchmark.

The larger 83 MB native snapshot capacity case from #1680 is not established as fixed by the 18.54 MB measurements. Keep that issue open.

Refs #1065, #1680.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1694 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/evidence-json-stream-allocation` --- ### 📝 Commits (2) - [`9d57542`](https://github.com/morluto/rea/commit/9d575428cd4c77173aaa7c79cd591e7ac1c0de31) perf(evidence): fuse streamed JSON token handling - [`0efc3a3`](https://github.com/morluto/rea/commit/0efc3a3d97697d01a7a34ddf85fc35f4c44c48a2) fix(build): declare the streamed JSON chain dependency ### 📊 Changes **4 files changed** (+189 additions, -128 deletions) <details> <summary>View changed files</summary> 📝 `package-lock.json` (+1 -0) 📝 `package.json` (+1 -0) 📝 `src/application/JsonInputFile.ts` (+130 -128) 📝 `tests/boundary/cli/jsonInputSize.test.ts` (+57 -0) </details> ### 📄 Description Large evidence JSON files currently create async iterator and stream work for every parser token. On an 18.54 MB snapshot exported from a real Ghidra analysis, that work dominates the sampled allocation window. Fuse the upstream parser, scalar packing, heap admission, and upstream value assembler into synchronous token stages. File reads remain asynchronous and cancellation-aware. Keep the existing capacity estimates, UTF-8 validation, scalar limits, file identity checks, and typed failures. Declare the existing stream-chain 4.2.6 as a direct runtime dependency. Add a public CLI container-expansion regression plus a same-process successful read after rejection. Validation: - Final head `0efc3a3d` passed 30 focused tests, including malformed input, cancellation, long strings, the existing 512 MiB whitespace control, and the new container admission/recovery case. `check:fast`, Knip, and whole-repository formatting passed on the final head. A real-evidence Linux CLI/MCP replay also passed on the benchmarked candidate; its reader source and compiled file are byte-identical to the final head. - Actual [Flameox](https://github.com/morluto/flameox) 0.2.9 ran five paired blocks under two CPUs and a 768 MiB Node heap. All ten workloads passed a separate oracle checking the complete canonical snapshot digest and retained Evidence identity. Median paired wall-time difference: **−3.63 s**; separate baseline/candidate medians: **7.64 / 3.62 s**. The workload includes Node startup, production snapshot reading/validation, full canonical hashing, and exit. Flameox classifies this as a descriptive practical improvement; these are local measurements, not a universal latency guarantee. - Scoped V8 sampling of production snapshot reading, analyzed with Flameox, estimated cumulative allocations of **10.77 / 3.17 GB** before/after (single capture each). This includes collected objects and excludes module initialization; it is **not peak RSS** or a leak measurement. - Integrated #1687 and #1691 without changing the measured reader source or compiled file. Final-head CI run [38110917770](https://github.com/morluto/rea/actions/runs/38110917770) passed **31/31 jobs**, including Windows package verification. - Actual Windows Node 22.19 / SDK 2.3.1 verification passed with the full source-built 6.4.0 archive, standard CLI/MCP entrypoints, existing matching locked production dependencies, and same-head CI native controls. A fresh real Ghidra 12.1.4 analysis saved a current-profile snapshot; adding 9 MiB of JSON whitespace forced streaming. Restarted MCP, verified the old reference was unavailable before import, restored exact retained Evidence/view identity, and checked CLI/MCP summary parity. JVM 512 MiB / Node heap 768 MiB / two CPUs; target unchanged, temporary directories empty, no remaining observed owned descendants. This is source-built package acceptance, not a published npm release or Windows large-payload benchmark. The larger 83 MB native snapshot capacity case from #1680 is not established as fixed by the 18.54 MB measurements. Keep that issue open. Refs #1065, #1680. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:54:45 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-10-31 01:14:08 +00:00

[PR #4760] [CLOSED] fix(native): reject NativeAOT reads beyond declared PE header extents (#4750) #6394

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4760

No description provided.

Rows
Columns