morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8781] [Design] Select analysis snapshots in native value trace CLI #7543

New issue

Open

opened 2026-11-03 12:24:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-03 12:24:59 +00:00

Owner

Originally created by @morluto on GitHub (Nov 3, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8781

Before submitting

I searched open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration.
This discussion contains no credentials or private target data.

Decision needed

Should the one-shot trace-native-values CLI expose the existing selected analysis snapshot workflow, so a caller can reuse a prior analysis and export the exact source Evidence cited by the trace?

Context and evidence

On macOS arm64, Node 24.22.1 and Ghidra 12.1.4, a source-built, unexecuted Mach-O was first opened through MCP, _evaluate was analyzed, and the session was closed with snapshot_path. Reopening that target and snapshot through real MCP and calling trace_native_values reused the original function Evidence ID and reported that the function observation came from the snapshot. The exported session bundle contained all source IDs cited by the returned page, in both compact and full schema profiles. The fixture SHA-256 b814f3cc4226f9fe3788dedf8a14f869ca45339c5f49fc2a7709eddd2df0bd6b remained unchanged.

The equivalent one-shot rea trace-native-values <target> _evaluate --provider ghidra --snapshot <snapshot> --format json exits 1 with Unknown flag: --snapshot; its help does not advertise that flag. This was observed against a checkout containing main 8c857ce56 plus unrelated feature-trace-view changes, and against an isolated combination of #8297 095e8f394 and #8384 a20c186eb. By contrast, rea trace --help advertises a snapshot option. The CLI behavior is an unsupported capability, not a claim that it promises snapshot support. The MCP replay workaround is available today.

In the same combined control with Kuna 1.989, MCP returned a new resolvable source ID and no snapshot-replay limitation; the separately recorded Kuna source ID was not reused. This does not establish that Kuna replayed that observation. Provider-specific caching and snapshot applicability require their own verification.

Expected interaction with targets

Read the selected target and snapshot; launch the explicitly selected analysis provider where needed; write the selected snapshot or Evidence bundle only under the existing explicit CLI options and their overwrite rules. No target execution or mutation proposed.

Boundary details

Preserve the caller's target, provider and snapshot selection. Reuse existing profile/identity validation and Evidence ownership; retain original source IDs on replay. Preserve typed errors, cleanup and publication semantics. Old snapshots and incompatible profiles must not be silently treated as fresh analysis.

Options and tradeoffs

Consider reusing the existing CLI snapshot selection and direct-analysis lifecycle, or document the MCP workflow if supporting one-shot replay is undesirable. #8297 addresses exporting source closure but its user-facing CLI currently has no snapshot selection; direct-analysis callback tests alone do not show that callers can select a snapshot through this command.

Verification available so far

I have observed results on one self-built Mach-O and two providers. The CLI refusal and Ghidra MCP source identity were exercised; a CLI snapshot implementation, cancellation/cleanup, snapshot write failures and other hosts remain unverified.

Success criteria and open questions

Check fresh and replayed observations, mixed cached/live source IDs, snapshot profile mismatch, close and publication failures, and offline bundle source resolution. Determine when an explicit snapshot should be updated and how that interacts with the caller-selected output path.

References

Related: #8282 (cached trace source identity), #8285 and #8297 (one-shot Evidence export), #8384 (source retention).

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Isolated option experiment

In the uncommitted combination of #8297 095e8f394 and #8384 a20c186eb, I added snapshot: snapshotSelectionOption to this CLI command, letting its existing directAnalysisOptionsFromCli pass the caller's selection to the shared lifecycle. With real Ghidra 12.1.4 and the same source-built fixture, the CLI summary plus --evidence_output succeeded; its saved trace linked the original snapshot function Evidence ID, its two linked source records were in the bundle, and the trace disclosed snapshot replay. The target SHA-256 remained unchanged.

A separate target with a different digest, selected against the same snapshot, returned evidence_integrity_mismatch; the selected snapshot's SHA-256 remained unchanged. This is one non-shipped prototype based on two unmerged PRs, not a claim that adding the option independently fixes source retention or CLI export. I did not exercise concurrent updates, cleanup failures, cancellation, output-path collisions, or other providers in this experiment.

Additional one-shot prototype controls

In the same isolated combination with the temporary snapshot option, a real Kuna 1.989 CLI run using its selected snapshot exported a trace with one linked function source and the explicit no-def-use unknown; the source was present in the bundle. It did not report a snapshot replay limitation, so this is not evidence of Kuna replay. A repeated Ghidra CLI run also exported both linked function records and disclosed a snapshot-loaded function observation. The unexecuted target digest remained unchanged.

For a separate Ghidra control I copied the selected snapshot to a test-owned file, then used that same path for --snapshot and --evidence_output. The CLI returned invalid_request saying the Evidence file already exists, with the completed trace in details.partial_observation. The test-owned snapshot's SHA-256 stayed unchanged and its embedded bundle contained both linked source IDs. The error did not itself include those source records or explain how to recover them from the snapshot. This is one path-collision control on the unmerged prototype; it does not prove all publication failures preserve the snapshot or source closure, or that an agent can recover sources without knowing the snapshot format.

Follow-up: failure handoff in a newer combined prototype

A newer isolated combination of #8786 (e870ea43a, based on #8785), #8384 (a20c186eb), and #8297 (095e8f394), with the same local one-line --snapshot option, changes the collision result above. On macOS arm64 with real Ghidra 12.1.4 and the same unexecuted Mach-O (SHA-256 b814f3cc4226f9fe3788dedf8a14f869ca45339c5f49fc2a7709eddd2df0bd6b), a fresh shared snapshot/Evidence output path produced invalid_request because the snapshot had just been written and the Evidence publisher will not overwrite it. The error now includes the completed trace and source_closure with publication: not_published, complete: true, and both source records named by the trace. Thus, the earlier missing-closure result describes the older prototype, not this newer combination. The caller still needs to choose a separate bundle destination.

Re-running with the newly created snapshot and a separate output path succeeded and exported the same trace Evidence ID with both referenced source records in the bundle. This is one Ghidra collision and replay control on an unmerged experimental combination; it does not establish behavior for all cleanup, cancellation or publication failures, other providers, or platforms. The target digest stayed unchanged.

Snapshot publication failure still loses captured sources

On the same newer combined prototype, I selected a snapshot path under a nonexistent parent directory and a separate Evidence output path. Real Ghidra completed the trace, then snapshot writing failed with invalid_request. The response retained the trace and explicitly reported source_closure.complete: false, two not_captured source IDs, and the need to rerun; no Evidence bundle was published. The target remained unexecuted. This is a distinct boundary from the same-path collision above.

Source inspection suggests the session's onEvidenceBundle callback runs only after successful snapshot publication, and the CLI's failure branch does not supply captured source records to delivery. The session had produced the trace's source records, but this experiment does not establish which failure and cleanup stages can safely export them or how to preserve resource ownership under cancellation. Verify those paths before selecting a fix; do not treat the honest incomplete-closure response as successful offline handoff.

Experimental recovery before snapshot publication

An uncommitted modification to the newer combined prototype captures the session Evidence bundle after the completed trace is recorded but before snapshot writing, then selects the trace's transitive source closure for failure delivery. On the same real Ghidra missing-parent control, the original invalid_request and publication: not_published remain, while source_closure.complete: true and both cited source records are returned inline. Seven focused CLI-boundary tests passed, including a synthetic session snapshot-write failure and a failure handoff that excludes unrelated session records. The provider fixture and real Ghidra target were not executed.

This is an experiment, not a proposed integration or a complete fix: it depends on unmerged work, has not been verified through cancellation or cleanup failure, and the prototype currently falls back to no captured records if authenticated closure resolution fails. Passing the entire session bundle to the existing failure formatter is incorrect: that formatter also includes supplied unrelated records. Preserve partial authenticated sources without exposing unrelated history before considering the design complete.

Follow-up: partial sources and final-close controls

The local experiment now supplies session records as a lookup inventory to the existing bounded failure-closure owner, distinct from records explicitly selected as sources. A focused synthetic CLI case with one available source, one missing transitive source, and unrelated session history returns the available source, reports the missing ID as not_captured, and excludes the unrelated record. Existing explicitly selected but unlinked source behavior is preserved. The experiment's ten CLI-boundary tests pass, and check:ci:fast passes.

Synthetic production-session final-close controls also retain the source when cancellation or a simulated cleanup error occurs; they do not establish recovery from an actual still-owned resource or every finalization stage. On the same macOS arm64 fixture, real Ghidra 12.1.4 and Kuna 1.989 missing-parent snapshot failures retained respectively two and one cited source records with source_closure.complete: true while keeping the original invalid_request and publication: not_published. The executable digest remained unchanged. The experiment remains uncommitted and depends on unmerged PRs; publication, cancellation, cleanup, and provider coverage are not exhaustive.

Retained-session cleanup retry control

The experimental production-session test now injects a close failure before the synthetic session releases its active target. It verifies the cleanup_incomplete response retains the owned-resource name and the captured source closure, then retries the session close and verifies the target is released. A separate variant also cancels during the failed close; cleanup remains the reported failure and the source closure remains available. Eleven focused CLI-boundary tests passed. This is a controlled synthetic close failure and retry, not a reproduction of a real Ghidra or Kuna process-cleanup failure; late handshakes and other cleanup stages remain unverified.

Originally created by @morluto on GitHub (Nov 3, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8781 ### Before submitting - [x] I searched open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration. - [x] This discussion contains no credentials or private target data. ### Decision needed Should the one-shot `trace-native-values` CLI expose the existing selected analysis snapshot workflow, so a caller can reuse a prior analysis and export the exact source Evidence cited by the trace? ### Context and evidence On macOS arm64, Node 24.22.1 and Ghidra 12.1.4, a source-built, unexecuted Mach-O was first opened through MCP, `_evaluate` was analyzed, and the session was closed with `snapshot_path`. Reopening that target and snapshot through real MCP and calling `trace_native_values` reused the original function Evidence ID and reported that the function observation came from the snapshot. The exported session bundle contained all source IDs cited by the returned page, in both compact and full schema profiles. The fixture SHA-256 `b814f3cc4226f9fe3788dedf8a14f869ca45339c5f49fc2a7709eddd2df0bd6b` remained unchanged. The equivalent one-shot `rea trace-native-values <target> _evaluate --provider ghidra --snapshot <snapshot> --format json` exits 1 with `Unknown flag: --snapshot`; its help does not advertise that flag. This was observed against a checkout containing main `8c857ce56` plus unrelated feature-trace-view changes, and against an isolated combination of #8297 `095e8f394` and #8384 `a20c186eb`. By contrast, `rea trace --help` advertises a snapshot option. The CLI behavior is an unsupported capability, not a claim that it promises snapshot support. The MCP replay workaround is available today. In the same combined control with Kuna 1.989, MCP returned a new resolvable source ID and no snapshot-replay limitation; the separately recorded Kuna source ID was not reused. This does not establish that Kuna replayed that observation. Provider-specific caching and snapshot applicability require their own verification. ### Expected interaction with targets Read the selected target and snapshot; launch the explicitly selected analysis provider where needed; write the selected snapshot or Evidence bundle only under the existing explicit CLI options and their overwrite rules. No target execution or mutation proposed. ### Boundary details Preserve the caller's target, provider and snapshot selection. Reuse existing profile/identity validation and Evidence ownership; retain original source IDs on replay. Preserve typed errors, cleanup and publication semantics. Old snapshots and incompatible profiles must not be silently treated as fresh analysis. ### Options and tradeoffs Consider reusing the existing CLI snapshot selection and direct-analysis lifecycle, or document the MCP workflow if supporting one-shot replay is undesirable. #8297 addresses exporting source closure but its user-facing CLI currently has no snapshot selection; direct-analysis callback tests alone do not show that callers can select a snapshot through this command. ### Verification available so far I have observed results on one self-built Mach-O and two providers. The CLI refusal and Ghidra MCP source identity were exercised; a CLI snapshot implementation, cancellation/cleanup, snapshot write failures and other hosts remain unverified. ### Success criteria and open questions Check fresh and replayed observations, mixed cached/live source IDs, snapshot profile mismatch, close and publication failures, and offline bundle source resolution. Determine when an explicit snapshot should be updated and how that interacts with the caller-selected output path. ### References Related: #8282 (cached trace source identity), #8285 and #8297 (one-shot Evidence export), #8384 (source retention). cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Isolated option experiment In the uncommitted combination of #8297 `095e8f394` and #8384 `a20c186eb`, I added `snapshot: snapshotSelectionOption` to this CLI command, letting its existing `directAnalysisOptionsFromCli` pass the caller's selection to the shared lifecycle. With real Ghidra 12.1.4 and the same source-built fixture, the CLI summary plus `--evidence_output` succeeded; its saved trace linked the original snapshot function Evidence ID, its two linked source records were in the bundle, and the trace disclosed snapshot replay. The target SHA-256 remained unchanged. A separate target with a different digest, selected against the same snapshot, returned `evidence_integrity_mismatch`; the selected snapshot's SHA-256 remained unchanged. This is one non-shipped prototype based on two unmerged PRs, not a claim that adding the option independently fixes source retention or CLI export. I did not exercise concurrent updates, cleanup failures, cancellation, output-path collisions, or other providers in this experiment. ### Additional one-shot prototype controls In the same isolated combination with the temporary snapshot option, a real Kuna 1.989 CLI run using its selected snapshot exported a trace with one linked function source and the explicit no-def-use unknown; the source was present in the bundle. It did not report a snapshot replay limitation, so this is not evidence of Kuna replay. A repeated Ghidra CLI run also exported both linked function records and disclosed a snapshot-loaded function observation. The unexecuted target digest remained unchanged. For a separate Ghidra control I copied the selected snapshot to a test-owned file, then used that same path for `--snapshot` and `--evidence_output`. The CLI returned `invalid_request` saying the Evidence file already exists, with the completed trace in `details.partial_observation`. The test-owned snapshot's SHA-256 stayed unchanged and its embedded bundle contained both linked source IDs. The error did not itself include those source records or explain how to recover them from the snapshot. This is one path-collision control on the unmerged prototype; it does not prove all publication failures preserve the snapshot or source closure, or that an agent can recover sources without knowing the snapshot format. ### Follow-up: failure handoff in a newer combined prototype A newer isolated combination of #8786 (`e870ea43a`, based on #8785), #8384 (`a20c186eb`), and #8297 (`095e8f394`), with the same local one-line `--snapshot` option, changes the collision result above. On macOS arm64 with real Ghidra 12.1.4 and the same unexecuted Mach-O (SHA-256 `b814f3cc4226f9fe3788dedf8a14f869ca45339c5f49fc2a7709eddd2df0bd6b`), a fresh shared snapshot/Evidence output path produced `invalid_request` because the snapshot had just been written and the Evidence publisher will not overwrite it. The error now includes the completed trace and `source_closure` with `publication: not_published`, `complete: true`, and both source records named by the trace. Thus, the earlier missing-closure result describes the older prototype, not this newer combination. The caller still needs to choose a separate bundle destination. Re-running with the newly created snapshot and a separate output path succeeded and exported the same trace Evidence ID with both referenced source records in the bundle. This is one Ghidra collision and replay control on an unmerged experimental combination; it does not establish behavior for all cleanup, cancellation or publication failures, other providers, or platforms. The target digest stayed unchanged. ### Snapshot publication failure still loses captured sources On the same newer combined prototype, I selected a snapshot path under a nonexistent parent directory and a separate Evidence output path. Real Ghidra completed the trace, then snapshot writing failed with `invalid_request`. The response retained the trace and explicitly reported `source_closure.complete: false`, two `not_captured` source IDs, and the need to rerun; no Evidence bundle was published. The target remained unexecuted. This is a distinct boundary from the same-path collision above. Source inspection suggests the session's `onEvidenceBundle` callback runs only after successful snapshot publication, and the CLI's failure branch does not supply captured source records to delivery. The session had produced the trace's source records, but this experiment does not establish which failure and cleanup stages can safely export them or how to preserve resource ownership under cancellation. Verify those paths before selecting a fix; do not treat the honest incomplete-closure response as successful offline handoff. ### Experimental recovery before snapshot publication An uncommitted modification to the newer combined prototype captures the session Evidence bundle after the completed trace is recorded but before snapshot writing, then selects the trace's transitive source closure for failure delivery. On the same real Ghidra missing-parent control, the original `invalid_request` and `publication: not_published` remain, while `source_closure.complete: true` and both cited source records are returned inline. Seven focused CLI-boundary tests passed, including a synthetic session snapshot-write failure and a failure handoff that excludes unrelated session records. The provider fixture and real Ghidra target were not executed. This is an experiment, not a proposed integration or a complete fix: it depends on unmerged work, has not been verified through cancellation or cleanup failure, and the prototype currently falls back to no captured records if authenticated closure resolution fails. Passing the entire session bundle to the existing failure formatter is incorrect: that formatter also includes supplied unrelated records. Preserve partial authenticated sources without exposing unrelated history before considering the design complete. ### Follow-up: partial sources and final-close controls The local experiment now supplies session records as a lookup inventory to the existing bounded failure-closure owner, distinct from records explicitly selected as sources. A focused synthetic CLI case with one available source, one missing transitive source, and unrelated session history returns the available source, reports the missing ID as `not_captured`, and excludes the unrelated record. Existing explicitly selected but unlinked source behavior is preserved. The experiment's ten CLI-boundary tests pass, and `check:ci:fast` passes. Synthetic production-session final-close controls also retain the source when cancellation or a simulated cleanup error occurs; they do not establish recovery from an actual still-owned resource or every finalization stage. On the same macOS arm64 fixture, real Ghidra 12.1.4 and Kuna 1.989 missing-parent snapshot failures retained respectively two and one cited source records with `source_closure.complete: true` while keeping the original `invalid_request` and `publication: not_published`. The executable digest remained unchanged. The experiment remains uncommitted and depends on unmerged PRs; publication, cancellation, cleanup, and provider coverage are not exhaustive. ### Retained-session cleanup retry control The experimental production-session test now injects a close failure before the synthetic session releases its active target. It verifies the `cleanup_incomplete` response retains the owned-resource name and the captured source closure, then retries the session close and verifies the target is released. A separate variant also cancels during the failed close; cleanup remains the reported failure and the source closure remains available. Eleven focused CLI-boundary tests passed. This is a controlled synthetic close failure and retry, not a reproduction of a real Ghidra or Kuna process-cleanup failure; late handshakes and other cleanup stages remain unverified.

bsakaguchi added the

enhancement

triage

labels

2026-11-03 12:24:59 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7543

No description provided.

Rows
Columns