[GH-ISSUE #1249] perf(snapshot): reduce eager materialization of large retained investigations #446

New issue

Closed

opened 2026-10-15 18:25:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:25:08 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1249

Current limitation

Snapshot load reads/parses/validates the complete JSON before staging cloned cache entries. Export clones retained Evidence and entries, validates again, then builds one complete canonical string. Workflow entries and Evidence can repeat large result payloads. Current main still uses these eager operations despite successful replay improvements.

Evidence and measurement limits

The current-main real native audit saved a 7,584,032-byte snapshot containing 3 primitive entries, 1 workflow entry and 19 Evidence records. Close/save took 2.29 s; fresh open took 4.39 s; eligible primitive/dossier reads took approximately 53/119 ms and preserved observations without launching Ghidra. Open includes other lifecycle/probe work: these numbers do not isolate JSON cost.

The original working log reports a much larger retained investigation and approximately 1 GiB point-in-time Node RSS under 4.1.0. That large-case timing/RSS has not been reproduced on 6.1.0/current main and must not be advertised as a current regression or peak. Current source establishes eager materialization, rather than a measured current out-of-memory defect.

Required outcome

Measure representative generated retained investigations under bounded CPU, Node heap and sampled family RSS, separating parse/validation, cloning, replay lookup and serialization. Reduce repeated full materialization and unnecessary payload duplication while preserving validation, immutable results, atomic writes, target/profile bindings and complete provenance.

Evaluate incremental/lazy access or lossless shared/compressed payloads only where the measured workload benefits. Preserve compatibility or provide explicit migration; no hidden store, automatic path choice, unsafe mutable hardlinks, silent record eviction or semantic digest changes. A downstream helper repeatedly archiving tool catalogs is helper overhead, not REA automatically making those copies.

Verify corruption/mismatch diagnostics, large-record and many-record cases, cross-session replay, changed target/profile rejection and resource/cleanup behavior. Keep serialized bytes, disk allocation, Node heap and process-family RSS distinct.

Source and relationship

src/application/binary/AnalysisSnapshotFiles.ts:16; src/application/binary/AnalysisSnapshotCache.ts:108; src/domain/analysisSnapshot.ts:679. Performance follow-up to #1211 under #1065; provider database persistence belongs to #1048.

Verification and scope

Small real snapshot workflow measured on main; eager load/clone/export architecture source-confirmed. The historical large-case cost and compressed-storage savings remain motivating observations from the old release, not latest-version benchmarks.

Audit date: 2026-10-09. Latest npm dist-tag verified as rea-agents@6.1.0; current main tested at f4c99ab8a53806dd57dc24b3d5a600905bcb4686. Linux x64, Node 24.18.0, MCP client SDK 2.3.1. Native observations use unmodified Ghidra 12.1.4 and JDK 21, two CPUs and a 512 MiB Java heap. No target execution or executable mutation was used for the static native checks.

Originally created by @N0zoM1z0 on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1249 ### Current limitation Snapshot load reads/parses/validates the complete JSON before staging cloned cache entries. Export clones retained Evidence and entries, validates again, then builds one complete canonical string. Workflow entries and Evidence can repeat large result payloads. Current main still uses these eager operations despite successful replay improvements. ### Evidence and measurement limits The current-main real native audit saved a 7,584,032-byte snapshot containing 3 primitive entries, 1 workflow entry and 19 Evidence records. Close/save took 2.29 s; fresh open took 4.39 s; eligible primitive/dossier reads took approximately 53/119 ms and preserved observations without launching Ghidra. Open includes other lifecycle/probe work: these numbers do not isolate JSON cost. The original working log reports a much larger retained investigation and approximately 1 GiB point-in-time Node RSS under 4.1.0. That large-case timing/RSS has not been reproduced on 6.1.0/current main and must not be advertised as a current regression or peak. Current source establishes eager materialization, rather than a measured current out-of-memory defect. ### Required outcome Measure representative generated retained investigations under bounded CPU, Node heap and sampled family RSS, separating parse/validation, cloning, replay lookup and serialization. Reduce repeated full materialization and unnecessary payload duplication while preserving validation, immutable results, atomic writes, target/profile bindings and complete provenance. Evaluate incremental/lazy access or lossless shared/compressed payloads only where the measured workload benefits. Preserve compatibility or provide explicit migration; no hidden store, automatic path choice, unsafe mutable hardlinks, silent record eviction or semantic digest changes. A downstream helper repeatedly archiving tool catalogs is helper overhead, not REA automatically making those copies. Verify corruption/mismatch diagnostics, large-record and many-record cases, cross-session replay, changed target/profile rejection and resource/cleanup behavior. Keep serialized bytes, disk allocation, Node heap and process-family RSS distinct. ### Source and relationship [src/application/binary/AnalysisSnapshotFiles.ts:16](https://github.com/morluto/rea/blob/f4c99ab8a53806dd57dc24b3d5a600905bcb4686/src/application/binary/AnalysisSnapshotFiles.ts#L16); [src/application/binary/AnalysisSnapshotCache.ts:108](https://github.com/morluto/rea/blob/f4c99ab8a53806dd57dc24b3d5a600905bcb4686/src/application/binary/AnalysisSnapshotCache.ts#L108); [src/domain/analysisSnapshot.ts:679](https://github.com/morluto/rea/blob/f4c99ab8a53806dd57dc24b3d5a600905bcb4686/src/domain/analysisSnapshot.ts#L679). Performance follow-up to #1211 under #1065; provider database persistence belongs to #1048. ### Verification and scope Small real snapshot workflow measured on main; eager load/clone/export architecture source-confirmed. The historical large-case cost and compressed-storage savings remain motivating observations from the old release, not latest-version benchmarks. Audit date: 2026-10-09. Latest npm dist-tag verified as `rea-agents@6.1.0`; current main tested at `f4c99ab8a53806dd57dc24b3d5a600905bcb4686`. Linux x64, Node 24.18.0, MCP client SDK 2.3.1. Native observations use unmodified Ghidra 12.1.4 and JDK 21, two CPUs and a 512 MiB Java heap. No target execution or executable mutation was used for the static native checks.

bsakaguchi

2026-10-15 18:25:08 +00:00