morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8869] [Design] Replay source-bearing analysis without losing Evidence closure #7778

New issue

Open

opened 2026-11-04 05:58:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-04 05:58:17 +00:00

Owner

Originally created by @morluto on GitHub (Nov 4, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8869

Before submitting

I searched open and closed issues and pull requests for related discussions.
I understand the contribution scope must be reviewed before integration.
This report contains no credentials or private targets.

Decision needed

Can snapshot replay retain the complete source Evidence closure of a provider result, so source-bearing observations can be reused without dropping provenance?

Context and evidence

On REA d7df9df33 plus the unrelated snapshot-import guidance in draft #8867, real Ghidra 12.1.4/JDK 21 on macOS arm64 was exercised through stdio MCP in both schema profiles against the same owned static Mach-O.

A fresh imported program with an explicit document replayed the second procedure_pseudo_code call. The closed snapshot held one replay entry; reopening it and repeating the exact call replayed again.
A disposable saved project with an intentionally synthetic historical analyzer report attached a health source Evidence link to the same observation. Even with an explicit document, neither the repeated call nor the reopened snapshot replayed; the snapshot held zero provider entries, although it retained the source and parent Evidence.
An omitted document did not replay in either control. This is separately relevant to the one-shot CLI report #8085. These are fixture observations, not latency or prevalence measurements. The target and saved project's .gpr descriptor hashes were unchanged; other project files were not independently hashed.

Source inspection identifies a deliberate boundary: AnalysisSnapshotCache.record refuses any execution carrying sourceEvidence because provider snapshot entries do not preserve the source closure. GhidraProviderClient attaches the historical report as source Evidence. This is a fail-closed provenance rule, not evidence that Ghidra cannot persist projects or that REA discarded the recorded source.

Expected interaction with targets

Read an explicitly selected artifact and existing analysis snapshot; write only a caller-selected snapshot. No target execution or mutation is proposed.

Boundary details

Preserve each source identity, link, uncertainty, target/profile binding and explicit unknowns. Missing, invalid or capacity-refused source records must never become a successful source-free replay. Account for snapshot format/version, imported historical snapshots, lifecycle, retained Evidence and bounded memory before integrating a new representation.

Options and tradeoffs

Compare retaining authenticated source references in snapshot entries with the current reanalysis behavior. Do not infer completeness from a link alone or silently duplicate large provider output. Source-bearing and documentless cases have different causes.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Exercise real Ghidra source-bearing and source-free controls, omitted and explicit document selectors, repeated calls and reopen. Verify source closure and exact identities across capacity failures, missing/corrupt sources and cancellation; measure completed tasks if changing performance. Other providers and hosts remain unverified.

References

#8085, #8590, #8655, #8867; src/application/binary/AnalysisSnapshotCache.ts and src/ghidra/GhidraProviderClient.ts.

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Source-closure replay prototype

Draft #8870 uses the retained parent and existing source records for source-bearing snapshot entries, and requires the complete transitive Evidence closure before replay. It adds no separate copy of provider source rows. On real Ghidra 12.1.4 in both MCP profiles, the saved-project control replayed the repeated document-selected decompile and replayed it again after snapshot reopen with the same direct source ID. Removing the linked historical source from a copied, otherwise accepted snapshot caused a live provider call instead; the source was re-established. The owned target and .gpr descriptor were unchanged. An injected nested-source control refused both in-memory and imported replay when a transitive source was missing. These are controlled cases, not an exhaustive lifecycle or provider claim.

The one-shot pre-provider path needs its own closure check: a filesystem boundary regression without that check returned the cached parent after its source was removed; with the check it launched the provider and restored the source. The existing default CLI source-handoff question remains #8590.

A small local comparison alternated three completed saved-project tasks per variant: open, two decompiles, snapshot close, reopen and decompile. The baseline was d7df9df33 plus unrelated #8867 import guidance; the prototype was d7df9df33 plus the replay experiment. Both used real Ghidra 12.1.4/JDK 21, compact MCP, macOS arm64, the same target and synthetic historical report. Medians: first call 3,328/3,151 ms, second 3/1 ms, reopened call 3,117/1 ms, full task 8,999/5,056 ms (baseline/prototype). Sampled process-family peak RSS was 889,440/881,152 KiB and sampled CPU lower bounds were 13.26/7.45 s; shared RSS may be counted more than once and short-lived processes may be missed. The serialized complete MCP tool catalog, explicit requests and returned SDK call-result objects were 132,431/132,499 o200k_base tokens; this excludes protocol framing and any autonomous agent conversation. Answer hashes and source IDs matched. These are three-run host-specific controls, not general speed or memory guarantees.

Four focused suites passed (96 tests, 2 skipped), including public MCP snapshot replay, one-shot replay and missing-source fallback. The new source-bearing regressions failed on the baseline; the missing-source one-shot regression also failed if the pre-provider closure check was removed. Source metadata replacement and injected final cleanup failure or cancellation with retry are now covered; a real Ghidra zero-retention control returned resource errors and saved no entry. Review, real cleanup failure, actual low memory, other providers and large-closure costs remain open.

A real Ghidra freshness control on #8870, in both MCP profiles, added one file to a disposable saved-project copy after creating a source-bearing snapshot. REA rejected the old snapshot for a profile mismatch and successfully analyzed the changed copy without the snapshot. Target and snapshot hashes were unchanged. This does not test concurrent project mutation.

Additional real Ghidra controls on #8870, both MCP profiles, refused a source-bearing snapshot for a byte-modified target copy and refused a copied snapshot containing tampered source Evidence. Original target, snapshot and project descriptor were unchanged.

On #8870, a real Ghidra control opened a snapshot missing its historical source, imported that source, then repeated the query in both MCP profiles. Before lazy revalidation, the query reran Ghidra; with it, the query replayed and kept the source ID. A nested missing-source control remains blocked until the leaf is imported, and ambiguous parent provenance remains non-replayable. The snapshot and target were unchanged.

Originally created by @morluto on GitHub (Nov 4, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8869 ### Before submitting - [x] I searched open and closed issues and pull requests for related discussions. - [x] I understand the contribution scope must be reviewed before integration. - [x] This report contains no credentials or private targets. ### Decision needed Can snapshot replay retain the complete source Evidence closure of a provider result, so source-bearing observations can be reused without dropping provenance? ### Context and evidence On REA `d7df9df33` plus the unrelated snapshot-import guidance in draft #8867, real Ghidra 12.1.4/JDK 21 on macOS arm64 was exercised through stdio MCP in both schema profiles against the same owned static Mach-O. - A fresh imported program with an explicit document replayed the second `procedure_pseudo_code` call. The closed snapshot held one replay entry; reopening it and repeating the exact call replayed again. - A disposable saved project with an intentionally synthetic historical analyzer report attached a `health` source Evidence link to the same observation. Even with an explicit document, neither the repeated call nor the reopened snapshot replayed; the snapshot held zero provider entries, although it retained the source and parent Evidence. - An omitted document did not replay in either control. This is separately relevant to the one-shot CLI report #8085. These are fixture observations, not latency or prevalence measurements. The target and saved project's `.gpr` descriptor hashes were unchanged; other project files were not independently hashed. Source inspection identifies a deliberate boundary: `AnalysisSnapshotCache.record` refuses any execution carrying `sourceEvidence` because provider snapshot entries do not preserve the source closure. `GhidraProviderClient` attaches the historical report as source Evidence. This is a fail-closed provenance rule, not evidence that Ghidra cannot persist projects or that REA discarded the recorded source. ### Expected interaction with targets Read an explicitly selected artifact and existing analysis snapshot; write only a caller-selected snapshot. No target execution or mutation is proposed. ### Boundary details Preserve each source identity, link, uncertainty, target/profile binding and explicit unknowns. Missing, invalid or capacity-refused source records must never become a successful source-free replay. Account for snapshot format/version, imported historical snapshots, lifecycle, retained Evidence and bounded memory before integrating a new representation. ### Options and tradeoffs Compare retaining authenticated source references in snapshot entries with the current reanalysis behavior. Do not infer completeness from a link alone or silently duplicate large provider output. Source-bearing and documentless cases have different causes. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Exercise real Ghidra source-bearing and source-free controls, omitted and explicit document selectors, repeated calls and reopen. Verify source closure and exact identities across capacity failures, missing/corrupt sources and cancellation; measure completed tasks if changing performance. Other providers and hosts remain unverified. ### References #8085, #8590, #8655, #8867; `src/application/binary/AnalysisSnapshotCache.ts` and `src/ghidra/GhidraProviderClient.ts`. cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Source-closure replay prototype Draft #8870 uses the retained parent and existing source records for source-bearing snapshot entries, and requires the complete transitive Evidence closure before replay. It adds no separate copy of provider source rows. On real Ghidra 12.1.4 in both MCP profiles, the saved-project control replayed the repeated document-selected decompile and replayed it again after snapshot reopen with the same direct source ID. Removing the linked historical source from a copied, otherwise accepted snapshot caused a live provider call instead; the source was re-established. The owned target and `.gpr` descriptor were unchanged. An injected nested-source control refused both in-memory and imported replay when a transitive source was missing. These are controlled cases, not an exhaustive lifecycle or provider claim. The one-shot pre-provider path needs its own closure check: a filesystem boundary regression without that check returned the cached parent after its source was removed; with the check it launched the provider and restored the source. The existing default CLI source-handoff question remains #8590. A small local comparison alternated three completed saved-project tasks per variant: open, two decompiles, snapshot close, reopen and decompile. The baseline was `d7df9df33` plus unrelated #8867 import guidance; the prototype was `d7df9df33` plus the replay experiment. Both used real Ghidra 12.1.4/JDK 21, compact MCP, macOS arm64, the same target and synthetic historical report. Medians: first call 3,328/3,151 ms, second 3/1 ms, reopened call 3,117/1 ms, full task 8,999/5,056 ms (baseline/prototype). Sampled process-family peak RSS was 889,440/881,152 KiB and sampled CPU lower bounds were 13.26/7.45 s; shared RSS may be counted more than once and short-lived processes may be missed. The serialized complete MCP tool catalog, explicit requests and returned SDK call-result objects were 132,431/132,499 `o200k_base` tokens; this excludes protocol framing and any autonomous agent conversation. Answer hashes and source IDs matched. These are three-run host-specific controls, not general speed or memory guarantees. Four focused suites passed (96 tests, 2 skipped), including public MCP snapshot replay, one-shot replay and missing-source fallback. The new source-bearing regressions failed on the baseline; the missing-source one-shot regression also failed if the pre-provider closure check was removed. Source metadata replacement and injected final cleanup failure or cancellation with retry are now covered; a real Ghidra zero-retention control returned resource errors and saved no entry. Review, real cleanup failure, actual low memory, other providers and large-closure costs remain open. A real Ghidra freshness control on #8870, in both MCP profiles, added one file to a disposable saved-project copy after creating a source-bearing snapshot. REA rejected the old snapshot for a profile mismatch and successfully analyzed the changed copy without the snapshot. Target and snapshot hashes were unchanged. This does not test concurrent project mutation. Additional real Ghidra controls on #8870, both MCP profiles, refused a source-bearing snapshot for a byte-modified target copy and refused a copied snapshot containing tampered source Evidence. Original target, snapshot and project descriptor were unchanged. On #8870, a real Ghidra control opened a snapshot missing its historical source, imported that source, then repeated the query in both MCP profiles. Before lazy revalidation, the query reran Ghidra; with it, the query replayed and kept the source ID. A nested missing-source control remains blocked until the leaf is imported, and ambiguous parent provenance remains non-replayable. The snapshot and target were unchanged.

bsakaguchi added the

enhancement

triage

labels

2026-11-04 05:58:17 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7778

No description provided.

Rows
Columns