morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1680] feat(evidence): reduce heap expansion when importing complete large snapshots #562

New issue

Closed

opened 2026-10-15 18:26:11 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-15 18:26:11 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1680

A complete, source-owned native analysis can be saved successfully but cannot be reopened within the same selected Node heap because importing the snapshot materializes expanded JSON representations.

Independent Linux x64 verification on main f88c99021b88ce67e513025901f57478d8107d0c, Node 24.18.0, MCP SDK 2.3.1, Ghidra 12.1.4 and Java 21:

A source-owned unrolled native C function produced a 20,749,645-byte normalized dossier and a 41,502,586-byte complete Evidence record. Its complete response received the expected typed transport constraint; the connection remained usable and selected native pages matched the complete exported parent.
close_binary saved a complete 83,369,526-byte snapshot and removed the owned Ghidra runtime. A fresh MCP session with NODE_OPTIONS=--max-old-space-size=768 rejected open_binary({snapshot_path: ...}) with resource_constraint, resource memory, operation read_evidence_file.
Reported input bytes: 83,369,526; projected value bytes: 567,749,520; required assembly headroom: 567,815,056; available heap: 566,591,240; V8 heap limit: 1,006,632,960. These are admission estimates, not independently measured retained-heap sizes.
The guarded run peaked at 2,567,151,616 bytes of process-family RSS with a 512 MiB JVM and two CPUs; no external guard stop or fatal Node OOM occurred. The failure is actionable resource admission, not connection loss or invalid JSON.

A smaller control using the same workflow produced a 4,543,224-byte dossier and an 18,543,311-byte snapshot. Its fresh-session import and exact retained-view replay passed without starting Ghidra. Thus snapshot reuse works within current memory admission; the large saved result remains the gap.

The source-owned fixture is an -O0 -fno-inline -g0 -no-pie C executable containing one noinline uint32_t function with 4,500 unrolled statements. Each statement updates state = (state ^ values[i % 64]) + (i + 17) and stores it into volatile values[(i + 1) % 64]; main supplies a local 64-element array. It is analyzed statically and never executed. This produces 44,869 assembly rows and 22,505 outgoing-reference rows; high-pcode remains explicitly truncated at the provider's existing boundary.

Improve complete snapshot import/discovery so analysts can reuse already saved large results without materializing avoidable copies of their payloads. Keep Evidence authentication, canonical profile/target binding, atomic import, cancellation, and complete retained content. Preserve typed budget failures where genuine headroom is insufficient; raising arbitrary caps or trimming signed records is not a fix. Measure fresh-session import latency and process-family memory through ordinary CLI/MCP consumers.

Related: #1247 and #1065. No Windows or Hopper coverage is claimed for this large-snapshot case.

Originally created by @N0zoM1z0 on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1680 A complete, source-owned native analysis can be saved successfully but cannot be reopened within the same selected Node heap because importing the snapshot materializes expanded JSON representations. Independent Linux x64 verification on main `f88c99021b88ce67e513025901f57478d8107d0c`, Node 24.18.0, MCP SDK 2.3.1, Ghidra 12.1.4 and Java 21: - A source-owned unrolled native C function produced a 20,749,645-byte normalized dossier and a 41,502,586-byte complete Evidence record. Its complete response received the expected typed transport constraint; the connection remained usable and selected native pages matched the complete exported parent. - `close_binary` saved a complete 83,369,526-byte snapshot and removed the owned Ghidra runtime. A fresh MCP session with `NODE_OPTIONS=--max-old-space-size=768` rejected `open_binary({snapshot_path: ...})` with `resource_constraint`, resource `memory`, operation `read_evidence_file`. - Reported input bytes: 83,369,526; projected value bytes: 567,749,520; required assembly headroom: 567,815,056; available heap: 566,591,240; V8 heap limit: 1,006,632,960. These are admission estimates, not independently measured retained-heap sizes. - The guarded run peaked at 2,567,151,616 bytes of process-family RSS with a 512 MiB JVM and two CPUs; no external guard stop or fatal Node OOM occurred. The failure is actionable resource admission, not connection loss or invalid JSON. A smaller control using the same workflow produced a 4,543,224-byte dossier and an 18,543,311-byte snapshot. Its fresh-session import and exact retained-view replay passed without starting Ghidra. Thus snapshot reuse works within current memory admission; the large saved result remains the gap. The source-owned fixture is an `-O0 -fno-inline -g0 -no-pie` C executable containing one `noinline uint32_t` function with 4,500 unrolled statements. Each statement updates `state = (state ^ values[i % 64]) + (i + 17)` and stores it into volatile `values[(i + 1) % 64]`; `main` supplies a local 64-element array. It is analyzed statically and never executed. This produces 44,869 assembly rows and 22,505 outgoing-reference rows; high-pcode remains explicitly truncated at the provider's existing boundary. Improve complete snapshot import/discovery so analysts can reuse already saved large results without materializing avoidable copies of their payloads. Keep Evidence authentication, canonical profile/target binding, atomic import, cancellation, and complete retained content. Preserve typed budget failures where genuine headroom is insufficient; raising arbitrary caps or trimming signed records is not a fix. Measure fresh-session import latency and process-family memory through ordinary CLI/MCP consumers. Related: #1247 and #1065. No Windows or Hopper coverage is claimed for this large-snapshot case.

bsakaguchi

2026-10-15 18:26:11 +00:00

closed this issue
added the
performance

area: evidence

enhancement

P2: medium
labels

bsakaguchi commented

2026-10-15 18:26:12 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 11, 2026):

A first allocation fix is now in #1694, based on actual Flameox 0.2.9 profiling and paired experiments with the 18.54 MB real Ghidra snapshot. Per-token async iteration dominated temporary allocation; synchronous token stages reduced estimated cumulative sampled allocations from 10.77 to 3.17 GB in the scoped reader window (single capture each, not peak RSS).

Five paired whole-process workloads under two CPUs / a 768 MiB heap all passed an independent full canonical snapshot digest and Evidence identity oracle. Median paired wall-time difference was −3.63 s. Parsing, capacity admission, file identity, cancellation and typed failures retain their existing semantics; 30 focused tests passed on the benchmarked candidate.

This does not yet establish that the larger 83 MB snapshot imports under the same heap. Keep #1680 open for that capacity case. Final-head checks and Windows verification are tracked in the PR.

@N0zoM1z0 commented on GitHub (Oct 11, 2026): A first allocation fix is now in #1694, based on actual Flameox 0.2.9 profiling and paired experiments with the 18.54 MB real Ghidra snapshot. Per-token async iteration dominated temporary allocation; synchronous token stages reduced estimated cumulative sampled allocations from 10.77 to 3.17 GB in the scoped reader window (single capture each, not peak RSS). Five paired whole-process workloads under two CPUs / a 768 MiB heap all passed an independent full canonical snapshot digest and Evidence identity oracle. Median paired wall-time difference was −3.63 s. Parsing, capacity admission, file identity, cancellation and typed failures retain their existing semantics; 30 focused tests passed on the benchmarked candidate. This does not yet establish that the larger 83 MB snapshot imports under the same heap. Keep #1680 open for that capacity case. Final-head checks and Windows verification are tracked in the PR.

bsakaguchi commented

2026-10-15 18:26:12 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 11, 2026):

The allocation-churn fix #1694 is now merged (13e03e94). Final-head CI passed all 31 jobs; local 30 focused tests and complete source-built Windows CLI/MCP + real Ghidra snapshot recovery passed. Existing heap admission and typed failures remain in force.

The Flameox measurements use the 18.54 MB real snapshot, with full canonical-content and retained-Evidence oracles. They establish reduced temporary allocation and whole-process wall time for that workload. The larger 83 MB capacity failure has not been rerun on this fix, so this issue remains open.

@N0zoM1z0 commented on GitHub (Oct 11, 2026): The allocation-churn fix #1694 is now merged (`13e03e94`). Final-head CI passed all 31 jobs; local 30 focused tests and complete source-built Windows CLI/MCP + real Ghidra snapshot recovery passed. Existing heap admission and typed failures remain in force. The Flameox measurements use the 18.54 MB real snapshot, with full canonical-content and retained-Evidence oracles. They establish reduced temporary allocation and whole-process wall time for that workload. The larger 83 MB capacity failure has not been rerun on this fix, so this issue remains open.

bsakaguchi commented

2026-10-15 18:26:12 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 11, 2026):

Reproduced on merged main 13e03e94 after #1694 and #1691, with the exact original 4,500-statement executable digest and the same Node 24.18 / 768 MiB heap / Ghidra 12.1.4 / JVM 512 MiB / two-CPU controls.

The complete snapshot is now 83,049,642 bytes (this focused reproduction retains fewer derived view records). Fresh-process import still returns a typed memory constraint during JSON assembly: projected 612,671,096 bytes, required 612,736,632, available 612,556,080, V8 limit 1,006,632,960. The connection remains usable, the target/snapshot bytes are unchanged, and owned runtime cleanup succeeds. The later canonical validation phase is not reached, so optimizing its index alone would not fix this reproduction.

I am working on reading legacy cache executions one at a time, validating bounded fingerprints against the complete authenticated Evidence, and reusing its payload graphs. The persisted format and complete records will remain unchanged. This issue stays open until the same saved snapshot passes fresh CLI/MCP import and complete bundle round-trip checks under the original heap.

@N0zoM1z0 commented on GitHub (Oct 11, 2026): Reproduced on merged main `13e03e94` after #1694 and #1691, with the exact original 4,500-statement executable digest and the same Node 24.18 / 768 MiB heap / Ghidra 12.1.4 / JVM 512 MiB / two-CPU controls. The complete snapshot is now 83,049,642 bytes (this focused reproduction retains fewer derived view records). Fresh-process import still returns a typed memory constraint during JSON assembly: projected 612,671,096 bytes, required 612,736,632, available 612,556,080, V8 limit 1,006,632,960. The connection remains usable, the target/snapshot bytes are unchanged, and owned runtime cleanup succeeds. The later canonical validation phase is not reached, so optimizing its index alone would not fix this reproduction. I am working on reading legacy cache executions one at a time, validating bounded fingerprints against the complete authenticated Evidence, and reusing its payload graphs. The persisted format and complete records will remain unchanged. This issue stays open until the same saved snapshot passes fresh CLI/MCP import and complete bundle round-trip checks under the original heap.

bsakaguchi commented

2026-10-15 18:26:13 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 11, 2026):

Verified the fix in #1716 with the original real Ghidra snapshot: 83,049,642 bytes now import through SDK 2.3.1 stdio MCP on Node 24.18 with the same 768 MiB old-space limit. The complete 41,546,956-byte exported bundle retains its original SHA-256; parent metadata/procedure and the first/last two rows of 44,869 assembly entries and 22,505 outgoing references remain accessible. Cancellation followed by a healthy closed session and successful retry also passes. Target/snapshot hashes are unchanged; replay launches no provider runtime.

Actual Windows Node 22.19 verification used a freshly installed full source-built package. Public CLI/MCP recovery of real Windows PE evidence passes with a streaming-sized input, exact retained view identity, complete bundle hash and summary parity. The same 83 MB Linux-produced file also passes the installed Windows file boundary with its full bundle digest preserved; this is decoder coverage, not a Windows provider claim for the ELF. Hopper/IDA recovery remains unverified.

No heap increase, payload trimming or snapshot format migration was needed.

@N0zoM1z0 commented on GitHub (Oct 11, 2026): Verified the fix in #1716 with the original real Ghidra snapshot: 83,049,642 bytes now import through SDK 2.3.1 stdio MCP on Node 24.18 with the same 768 MiB old-space limit. The complete 41,546,956-byte exported bundle retains its original SHA-256; parent metadata/procedure and the first/last two rows of 44,869 assembly entries and 22,505 outgoing references remain accessible. Cancellation followed by a healthy closed session and successful retry also passes. Target/snapshot hashes are unchanged; replay launches no provider runtime. Actual Windows Node 22.19 verification used a freshly installed full source-built package. Public CLI/MCP recovery of real Windows PE evidence passes with a streaming-sized input, exact retained view identity, complete bundle hash and summary parity. The same 83 MB Linux-produced file also passes the installed Windows file boundary with its full bundle digest preserved; this is decoder coverage, not a Windows provider claim for the ELF. Hopper/IDA recovery remains unverified. No heap increase, payload trimming or snapshot format migration was needed.

bsakaguchi referenced this issue

2026-10-16 21:14:14 +00:00

[PR #597] [MERGED] fix!: repair boundary contracts across CLI and MCP #2097

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:17 +00:00