[PR #1716] [MERGED] perf(snapshot): restore large legacy caches within live memory bounds #4775

New issue

Closed

opened 2026-10-31 00:54:54 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:54:54 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/1716
Author: @N0zoM1z0
Created: 10/11/2026
Status: ✅ Merged
Merged: 10/11/2026
Merged by: @N0zoM1z0

Base: main ← Head: fix/evidence-snapshot-import-capacity

📝 Commits (5)

409ae9a perf(snapshot): stream legacy cache payload validation
c8ec321 Merge remote-tracking branch 'origin/main' into fix/evidence-snapshot-import-capacity
857f3ce refactor(snapshot): reuse the composite key owner
d712ff8 Merge remote-tracking branch 'origin/main' into fix/evidence-snapshot-import-capacity
5e7f62f fix(snapshot): preserve sealed authentication after readiness extraction

📊 Changes

9 files changed (+998 additions, -303 deletions)

View changed files

📝 src/application/DirectAnalysis.ts (+1 -1)
📝 src/application/JsonFiles.ts (+44 -5)
📝 src/application/JsonInputFile.ts (+270 -56)
📝 src/application/binary/AnalysisSnapshotCache.ts (+5 -3)
📝 src/application/binary/AnalysisSnapshotFiles.ts (+89 -6)
📝 src/domain/analysisSnapshot.ts (+341 -200)
📝 src/domain/evidenceBundle.ts (+27 -2)
📝 src/server/registerSessionTools.ts (+4 -1)
📝 tests/boundary/filesystem/analysisSnapshot.test.ts (+217 -29)

📄 Description

An 83,049,642-byte snapshot from a real Ghidra analysis still fails open_binary(snapshot_path) on main with the existing 768 MiB Node old-space limit. Legacy files duplicate complete execution payloads in cache entries and authenticated Evidence. The reader currently assembles those copies together before validation can start.

Read large legacy snapshots in two scans on one admitted descriptor. Parse cache rows independently, retain private metadata/content-digest handles, then authenticate the complete Evidence bundle and restore cache payloads from exact matching records. Check that both scans saw identical bytes. Keep the existing JSON format, all content and caller metadata, strict JSON/UTF-8 semantics, live resource admission, and canonical query/profile/target checks. Small files retain the bounded native parser path.

The cache now owns immutable authenticated snapshots and reuses their payloads. Exported snapshots remain detached and mutable. CLI and MCP snapshot reads receive cancellation; queued cancellation is checked before publishing the restored session.

Validation:

70 affected snapshot/cache/reconstruction-readiness/filesystem/CLI/MCP composition tests pass after integrating current main. The unchanged 24-case strict CLI JSON input lane also passed. Final typecheck, lint, Knip and architecture guards pass.
Actual SDK 2.3.1 stdio MCP, Node 24.18, 768 MiB old-space, two CPUs: cancelling the same large import leaves the session closed and healthy, then a fresh import succeeds. Original parent metadata, procedure, and the first/last two rows of 44,869 assembly entries and 22,505 outgoing references are accessible.
The complete 41,546,956-byte exported bundle has exactly the original SHA-256. Target and snapshot digests remain unchanged; replay launches no provider runtime and leaves its scratch directory empty. Guarded process-family peak is 1,095,970,816 bytes; one successful import took 14.54 seconds. These are single-workflow observations, not a general throughput benchmark or exhaustive page coverage.
Windows x64, Node 22.19, existing npm 10.9.3: freshly installed the complete source-built package from integrity-verified locked offline archives. Real Windows Ghidra PE snapshot recovery through public MCP passes with a 9 MiB whitespace prefix forcing streaming; the complete original bundle hash and procedure-view Evidence ID are exact, CLI/MCP summary records agree, close invalidates retained references, and replay leaves no provider process or scratch files. The same original 83 MB Linux-produced file also passes the installed Windows snapshot file boundary with its complete 41 MB bundle hash preserved; this additional lane does not claim Windows provider analysis of that ELF. Sampled process-family peak across the combined Windows workflow is 428,752,896 bytes.
The packaged Windows native helper and Ghidra bridge are source-equivalent to the already verified 6.4.0 artifact; no new native compilation claim. Real Hopper and IDA recovery remain unverified.

Fixes #1680. Refs #1065.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/1716 **Author:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Created:** 10/11/2026 **Status:** ✅ Merged **Merged:** 10/11/2026 **Merged by:** [@N0zoM1z0](https://github.com/N0zoM1z0) **Base:** `main` ← **Head:** `fix/evidence-snapshot-import-capacity` --- ### 📝 Commits (5) - [`409ae9a`](https://github.com/morluto/rea/commit/409ae9a951313da42309cc7b9631d1f0fe51744b) perf(snapshot): stream legacy cache payload validation - [`c8ec321`](https://github.com/morluto/rea/commit/c8ec321cb343e2157a37272fd3b409f4e9a034b0) Merge remote-tracking branch 'origin/main' into fix/evidence-snapshot-import-capacity - [`857f3ce`](https://github.com/morluto/rea/commit/857f3ce0291dd12aa1facdfddbee38dfe15f3186) refactor(snapshot): reuse the composite key owner - [`d712ff8`](https://github.com/morluto/rea/commit/d712ff878def88b3f624f09831dc3951cf43986b) Merge remote-tracking branch 'origin/main' into fix/evidence-snapshot-import-capacity - [`5e7f62f`](https://github.com/morluto/rea/commit/5e7f62fa232e62a00d553c888faec4cc912d3fd3) fix(snapshot): preserve sealed authentication after readiness extraction ### 📊 Changes **9 files changed** (+998 additions, -303 deletions) <details> <summary>View changed files</summary> 📝 `src/application/DirectAnalysis.ts` (+1 -1) 📝 `src/application/JsonFiles.ts` (+44 -5) 📝 `src/application/JsonInputFile.ts` (+270 -56) 📝 `src/application/binary/AnalysisSnapshotCache.ts` (+5 -3) 📝 `src/application/binary/AnalysisSnapshotFiles.ts` (+89 -6) 📝 `src/domain/analysisSnapshot.ts` (+341 -200) 📝 `src/domain/evidenceBundle.ts` (+27 -2) 📝 `src/server/registerSessionTools.ts` (+4 -1) 📝 `tests/boundary/filesystem/analysisSnapshot.test.ts` (+217 -29) </details> ### 📄 Description An 83,049,642-byte snapshot from a real Ghidra analysis still fails `open_binary(snapshot_path)` on main with the existing 768 MiB Node old-space limit. Legacy files duplicate complete execution payloads in cache entries and authenticated Evidence. The reader currently assembles those copies together before validation can start. Read large legacy snapshots in two scans on one admitted descriptor. Parse cache rows independently, retain private metadata/content-digest handles, then authenticate the complete Evidence bundle and restore cache payloads from exact matching records. Check that both scans saw identical bytes. Keep the existing JSON format, all content and caller metadata, strict JSON/UTF-8 semantics, live resource admission, and canonical query/profile/target checks. Small files retain the bounded native parser path. The cache now owns immutable authenticated snapshots and reuses their payloads. Exported snapshots remain detached and mutable. CLI and MCP snapshot reads receive cancellation; queued cancellation is checked before publishing the restored session. Validation: - 70 affected snapshot/cache/reconstruction-readiness/filesystem/CLI/MCP composition tests pass after integrating current main. The unchanged 24-case strict CLI JSON input lane also passed. Final typecheck, lint, Knip and architecture guards pass. - Actual SDK 2.3.1 stdio MCP, Node 24.18, 768 MiB old-space, two CPUs: cancelling the same large import leaves the session closed and healthy, then a fresh import succeeds. Original parent metadata, procedure, and the first/last two rows of 44,869 assembly entries and 22,505 outgoing references are accessible. - The complete 41,546,956-byte exported bundle has exactly the original SHA-256. Target and snapshot digests remain unchanged; replay launches no provider runtime and leaves its scratch directory empty. Guarded process-family peak is 1,095,970,816 bytes; one successful import took 14.54 seconds. These are single-workflow observations, not a general throughput benchmark or exhaustive page coverage. - Windows x64, Node 22.19, existing npm 10.9.3: freshly installed the complete source-built package from integrity-verified locked offline archives. Real Windows Ghidra PE snapshot recovery through public MCP passes with a 9 MiB whitespace prefix forcing streaming; the complete original bundle hash and procedure-view Evidence ID are exact, CLI/MCP summary records agree, close invalidates retained references, and replay leaves no provider process or scratch files. The same original 83 MB Linux-produced file also passes the installed Windows snapshot file boundary with its complete 41 MB bundle hash preserved; this additional lane does not claim Windows provider analysis of that ELF. Sampled process-family peak across the combined Windows workflow is 428,752,896 bytes. - The packaged Windows native helper and Ghidra bridge are source-equivalent to the already verified 6.4.0 artifact; no new native compilation claim. Real Hopper and IDA recovery remain unverified. Fixes #1680. Refs #1065. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:54:54 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-10-31 01:14:13 +00:00

[PR #4775] [CLOSED] Update display name and description in openai.yaml #6405

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4775

No description provided.

Rows
Columns