morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8732] [Bug] Snapshot replay returns unlinked duplicate provider Evidence #7528

New issue

Closed

opened 2026-11-03 07:17:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 07:17:03 +00:00

Owner

Originally created by @morluto on GitHub (Nov 3, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8732

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Replaying a Ghidra pseudocode observation through public MCP returns a new Evidence ID with no link to the original observation. The exported investigation bundle retains both records, but neither record connects them. The replayed record correctly says that the provider was not rerun. This makes it harder to tell that the two records came from the same underlying observation and can make a replay appear to be independent corroboration if that limitation is overlooked.

Reproduction

On REA main 9ebbe67e05af08b567e215340da993c08dae378f, macOS arm64, Node 24.22.1, Ghidra 12.1.4 and JDK 21:

Open a self-built thin Mach-O with provider ghidra, call list_documents, and call procedure_pseudo_code for _audit_value with the returned document selector. Record its Evidence ID.
Call close_binary with an absolute, new snapshot_path.
Start a new stdio MCP process, open the same target and snapshot, then repeat the same pseudocode request.
Read get_evidence_bundle.

In both compact and full output profiles, pseudocode was identical and the new response included Loaded from a local REA analysis snapshot; this call did not re-run the provider. The saved snapshot contained the pseudocode replay entry. The original Evidence ID was ev_22f7818b79b85d2f077ce498f5f4d66fd7e93ea27965599f9008b43b9ee2c829; the replay returned ev_31097f3e64cd593563796854b77b1adaf4bb1e014eea1d1fcd3e2301e1cf532f. A separate compact-profile check found empty evidence_links on both responses and confirmed that both records were retained and unlinked in the bundle. Link fields and bundle retention were not separately checked in the full profile. The target hash remained unchanged and it was not executed.

Existing Ghidra project control

A separate compact-profile control used REA's private-copy workflow with a caller-owned Ghidra 12.1.4 project containing an authored function name. An explicit-document pseudocode request returned that name; its analysis profile committed the project and said original settings and completeness were unknown. After saving and reopening the snapshot, the same request returned the snapshot-replay limitation without starting Ghidra; it again produced a different, unlinked Evidence ID. The original and replay records were both in the exported bundle. File hashes for the selected project and target remained unchanged. This particular project yielded no separate project-history source Evidence, so it does not exercise the source-chain case described below.

Composed-workflow control

In a separate real-Ghidra compact MCP control on the same target, get_call_graph before and after a snapshot handoff returned the same workflow Evidence ID and linked to the same retained procedure_callees Evidence ID. The saved snapshot contained a get_call_graph workflow entry and no primitive replay entries. A follow-up reopened that same snapshot in a fresh process, checked binary_session before and after the repeated graph call, and observed process lineage change from not_observed to a verified Ghidra process family. The call returned the same workflow ID and source link and took approximately 3.4 seconds in this one run. Thus Ghidra started during that call; the check does not independently establish which internal reads ran or benchmark typical performance. Source inspection shows that executeComposedWorkflow collects execution.evidenceId as an upstream link, whereas the direct provider tool delivery described below constructs a new record. Keep those ownership paths consistent while preserving their different semantics.

Expected behavior

A consumer should be able to identify the retained observation underlying a replay while also seeing that the current call did not rerun the provider. Decide whether to preserve the original identity with a separate delivery caveat or represent a linked replay receipt; preserve exact request parameters, caller input provenance, source Evidence, snapshot validity, and retention failures. Avoid treating the replay as a second independent provider observation.

Impact and scope

Observed for one Ghidra pseudocode query on one Mach-O. The text and replay limitation are correct and the original is retained; this is a provenance and identity relationship gap, not loss of the pseudocode or proof that an agent actually drew a wrong conclusion. Other tools, providers, CLI, and snapshots without a matching original record were not tested in this control.

REA version

9ebbe67e05af08b567e215340da993c08dae378f

Runtime and platform

macOS arm64, Node 24.22.1, Ghidra 12.1.4, JDK 21.

Target and artifact context

Self-built, unexecuted thin arm64 Mach-O, SHA-256 0e0ecf82c96e4918ee2ca1287dc25513d8fd13d5ab7fa5b98db00ad342f03930. Caller-owned temporary snapshot.

Interface

MCP

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

The same request and text had distinct IDs before and after replay. The replay had the explicit snapshot limitation; both records had evidence_links: [], and both were present in the exported bundle.

Analysis or suspected cause

Source-derived: AnalysisSnapshotCache restores a binding to the original Evidence ID and adds the replay limitation to the execution. registerEvidenceTools.ts then independently constructs and records Evidence from that modified execution rather than using the restored evidenceId as a link or delivery identity. The extra limitation changes the resulting content identity. Check the producing and delivery owners together; simply forcing an ID onto changed Evidence would violate content identity. Related #8282 concerns missing source records for cached native value traces; #8380 concerns a separate Ghidra verifier assertion.

Existing provenance constraint

tests/boundary/mcp/analysisSnapshotProvenanceMcp.test.ts exercises an imported replay whose original observation already links to a source and that source links to an ancestor. It checks refusal when the immediate source is missing, immutable source ownership, and metadata refresh after a bundle import; current assertions require the delivered replay to link to the original source. A fix must preserve that source closure and metadata replacement behavior while representing the replay-to-original relationship. Also verify the live in-session cache path, which binds an ID when recording the entry, and source-retention failures. Do not add an unowned ID in just the MCP renderer or force a changed record to reuse the original content ID. These are source/test-derived constraints; the existing test uses the managed provider fixture, not a real Ghidra process.

Typed outcome or recovery expected

Make the relationship and replay status available in the public result and exported bundle. Verify real-provider snapshot replay and non-replay controls, Evidence import/export and source-linked records, both MCP profiles, ambiguity and cancellation/retention failure behavior.

Additional context

@gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Originally created by @morluto on GitHub (Nov 3, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8732 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior Replaying a Ghidra pseudocode observation through public MCP returns a new Evidence ID with no link to the original observation. The exported investigation bundle retains both records, but neither record connects them. The replayed record correctly says that the provider was not rerun. This makes it harder to tell that the two records came from the same underlying observation and can make a replay appear to be independent corroboration if that limitation is overlooked. ### Reproduction On REA main `9ebbe67e05af08b567e215340da993c08dae378f`, macOS arm64, Node 24.22.1, Ghidra 12.1.4 and JDK 21: 1. Open a self-built thin Mach-O with provider `ghidra`, call `list_documents`, and call `procedure_pseudo_code` for `_audit_value` with the returned document selector. Record its Evidence ID. 2. Call `close_binary` with an absolute, new `snapshot_path`. 3. Start a new stdio MCP process, open the same target and snapshot, then repeat the same pseudocode request. 4. Read `get_evidence_bundle`. In both compact and full output profiles, pseudocode was identical and the new response included `Loaded from a local REA analysis snapshot; this call did not re-run the provider.` The saved snapshot contained the pseudocode replay entry. The original Evidence ID was `ev_22f7818b79b85d2f077ce498f5f4d66fd7e93ea27965599f9008b43b9ee2c829`; the replay returned `ev_31097f3e64cd593563796854b77b1adaf4bb1e014eea1d1fcd3e2301e1cf532f`. A separate compact-profile check found empty `evidence_links` on both responses and confirmed that both records were retained and unlinked in the bundle. Link fields and bundle retention were not separately checked in the full profile. The target hash remained unchanged and it was not executed. ### Existing Ghidra project control A separate compact-profile control used REA's private-copy workflow with a caller-owned Ghidra 12.1.4 project containing an authored function name. An explicit-document pseudocode request returned that name; its analysis profile committed the project and said original settings and completeness were unknown. After saving and reopening the snapshot, the same request returned the snapshot-replay limitation without starting Ghidra; it again produced a different, unlinked Evidence ID. The original and replay records were both in the exported bundle. File hashes for the selected project and target remained unchanged. This particular project yielded no separate project-history source Evidence, so it does not exercise the source-chain case described below. ### Composed-workflow control In a separate real-Ghidra compact MCP control on the same target, `get_call_graph` before and after a snapshot handoff returned the same workflow Evidence ID and linked to the same retained `procedure_callees` Evidence ID. The saved snapshot contained a `get_call_graph` workflow entry and no primitive replay entries. A follow-up reopened that same snapshot in a fresh process, checked `binary_session` before and after the repeated graph call, and observed process lineage change from `not_observed` to a verified Ghidra process family. The call returned the same workflow ID and source link and took approximately 3.4 seconds in this one run. Thus Ghidra started during that call; the check does not independently establish which internal reads ran or benchmark typical performance. Source inspection shows that `executeComposedWorkflow` collects `execution.evidenceId` as an upstream link, whereas the direct provider tool delivery described below constructs a new record. Keep those ownership paths consistent while preserving their different semantics. ### Expected behavior A consumer should be able to identify the retained observation underlying a replay while also seeing that the current call did not rerun the provider. Decide whether to preserve the original identity with a separate delivery caveat or represent a linked replay receipt; preserve exact request parameters, caller input provenance, source Evidence, snapshot validity, and retention failures. Avoid treating the replay as a second independent provider observation. ### Impact and scope Observed for one Ghidra pseudocode query on one Mach-O. The text and replay limitation are correct and the original is retained; this is a provenance and identity relationship gap, not loss of the pseudocode or proof that an agent actually drew a wrong conclusion. Other tools, providers, CLI, and snapshots without a matching original record were not tested in this control. ### REA version `9ebbe67e05af08b567e215340da993c08dae378f` ### Runtime and platform macOS arm64, Node 24.22.1, Ghidra 12.1.4, JDK 21. ### Target and artifact context Self-built, unexecuted thin arm64 Mach-O, SHA-256 `0e0ecf82c96e4918ee2ca1287dc25513d8fd13d5ab7fa5b98db00ad342f03930`. Caller-owned temporary snapshot. ### Interface MCP ### Earliest failure stage Artifact or evidence registration ### Relevant output or evidence The same request and text had distinct IDs before and after replay. The replay had the explicit snapshot limitation; both records had `evidence_links: []`, and both were present in the exported bundle. ### Analysis or suspected cause Source-derived: `AnalysisSnapshotCache` restores a binding to the original Evidence ID and adds the replay limitation to the execution. `registerEvidenceTools.ts` then independently constructs and records Evidence from that modified execution rather than using the restored `evidenceId` as a link or delivery identity. The extra limitation changes the resulting content identity. Check the producing and delivery owners together; simply forcing an ID onto changed Evidence would violate content identity. Related #8282 concerns missing source records for cached native value traces; #8380 concerns a separate Ghidra verifier assertion. ### Existing provenance constraint `tests/boundary/mcp/analysisSnapshotProvenanceMcp.test.ts` exercises an imported replay whose original observation already links to a source and that source links to an ancestor. It checks refusal when the immediate source is missing, immutable source ownership, and metadata refresh after a bundle import; current assertions require the delivered replay to link to the original source. A fix must preserve that source closure and metadata replacement behavior while representing the replay-to-original relationship. Also verify the live in-session cache path, which binds an ID when recording the entry, and source-retention failures. Do not add an unowned ID in just the MCP renderer or force a changed record to reuse the original content ID. These are source/test-derived constraints; the existing test uses the managed provider fixture, not a real Ghidra process. ### Typed outcome or recovery expected Make the relationship and replay status available in the public result and exported bundle. Verify real-provider snapshot replay and non-replay controls, Evidence import/export and source-linked records, both MCP profiles, ambiguity and cancellation/retention failure behavior. ### Additional context @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

bsakaguchi closed this issue

2026-11-04 18:21:30 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7528

No description provided.

Rows
Columns