morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8260] [Design] Preserve per-function producer provenance across decompile export resume #6516

New issue

Open

opened 2026-11-01 21:11:49 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-11-01 21:11:49 +00:00

Owner

Originally created by @morluto on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8260

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

How should a resumed decompile_all directory retain the exact provider observation or project-export identity that produced each function, especially when successful files come from different completed exports with the same target and analysis profile?

Context and evidence

On REA 53a4ab78451a0654e9c3beb3d43a5e371a288fcc, Kuna 1.986 and an unexecuted owned arm64 Mach-O, I used two isolated cache directories with the same target, wrapper binary, jobs, export key and profile. A test-only wrapper injected an upstream worker stall for _adjust in the first export; a second clean cache produced a complete export. This deliberately induced condition does not establish natural timeout frequency.

A two-function decompile_all against the partial cache wrote _evaluate and recorded _adjust as failed. Resuming against the same partial cache retried only _adjust, kept the successful file's SHA-256 unchanged, and failed again. Resuming against the separately built clean cache skipped _evaluate, wrote _adjust, and kept the original successful file's digest unchanged. The append-only journal retained both failed attempts and the later success, and the final index recorded both files and their digests.

Separate real procedure_pseudo_code calls show that the two _evaluate strings are identical in this sample, but their raw_result.source values point to different project directories and different full project-export C-file SHA-256 values. Their Evidence IDs differ. The journal and final index carry per-function output digests and limitations, target/provider/profile, but no project-export source digest or per-function Evidence ID. Each separately exported decompile_all Evidence bundle in this test contained one composed decompile_all record, raw_result: null, and no evidence_links; it did not retain per-function source records. Therefore the directory and those bundles alone cannot identify which producer export supplied each successful function. This is an observed provenance limitation, not a claim that the two sampled bodies differ or that the directory's file hashes are wrong.

Expected interaction with targets

Read a caller-selected binary and launch an owned static provider; write only the explicitly selected export directory, Evidence bundles and existing owned cache. The target is not executed or modified.

Boundary details

Keep each file's digest, exact source observation, provider/profile and project-export identity distinct from output paths and cache lookup keys. Preserve per-function failure caveats and append-only recovery history. An export key is not a content digest of each completed project. A resumed directory can legitimately combine attempts; do not assert cross-run pseudocode equivalence or retain every full body inline solely to establish provenance.

Options and tradeoffs

Consider optional retained per-function source Evidence or a compact authenticated source descriptor in journal entries and final index, with explicit unknown for earlier records. Assess bundle closure, storage size, cancellation, resume compatibility and migration without inventing a new result from an unavailable source. Compare this with existing per-function output digests and limitations before expanding the contract.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Exercise mixed success/failure, same-cache retry, independently completed export, missing/tampered source, cancellation and existing-journal compatibility. Verify CLI and MCP plus source Evidence retrieval without overwriting caller files. This report does not propose silently rebuilding or deleting a cache.

References

#8167 tracks retrying a failed cached Kuna function; #8226 tracks loss of per-item failure association in batch_decompile. This question concerns provenance of successfully written directory files across resume, whose existing journal does retain per-function caveats.

Originally created by @morluto on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8260 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed How should a resumed `decompile_all` directory retain the exact provider observation or project-export identity that produced each function, especially when successful files come from different completed exports with the same target and analysis profile? ### Context and evidence On REA `53a4ab78451a0654e9c3beb3d43a5e371a288fcc`, Kuna 1.986 and an unexecuted owned arm64 Mach-O, I used two isolated cache directories with the same target, wrapper binary, jobs, export key and profile. A test-only wrapper injected an upstream worker stall for `_adjust` in the first export; a second clean cache produced a complete export. This deliberately induced condition does not establish natural timeout frequency. A two-function `decompile_all` against the partial cache wrote `_evaluate` and recorded `_adjust` as failed. Resuming against the same partial cache retried only `_adjust`, kept the successful file's SHA-256 unchanged, and failed again. Resuming against the separately built clean cache skipped `_evaluate`, wrote `_adjust`, and kept the original successful file's digest unchanged. The append-only journal retained both failed attempts and the later success, and the final index recorded both files and their digests. Separate real `procedure_pseudo_code` calls show that the two `_evaluate` strings are identical in this sample, but their `raw_result.source` values point to different project directories and different full project-export C-file SHA-256 values. Their Evidence IDs differ. The journal and final index carry per-function output digests and limitations, target/provider/profile, but no project-export source digest or per-function Evidence ID. Each separately exported `decompile_all` Evidence bundle in this test contained one composed `decompile_all` record, `raw_result: null`, and no `evidence_links`; it did not retain per-function source records. Therefore the directory and those bundles alone cannot identify which producer export supplied each successful function. This is an observed provenance limitation, not a claim that the two sampled bodies differ or that the directory's file hashes are wrong. ### Expected interaction with targets Read a caller-selected binary and launch an owned static provider; write only the explicitly selected export directory, Evidence bundles and existing owned cache. The target is not executed or modified. ### Boundary details Keep each file's digest, exact source observation, provider/profile and project-export identity distinct from output paths and cache lookup keys. Preserve per-function failure caveats and append-only recovery history. An export key is not a content digest of each completed project. A resumed directory can legitimately combine attempts; do not assert cross-run pseudocode equivalence or retain every full body inline solely to establish provenance. ### Options and tradeoffs Consider optional retained per-function source Evidence or a compact authenticated source descriptor in journal entries and final index, with explicit unknown for earlier records. Assess bundle closure, storage size, cancellation, resume compatibility and migration without inventing a new result from an unavailable source. Compare this with existing per-function output digests and limitations before expanding the contract. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Exercise mixed success/failure, same-cache retry, independently completed export, missing/tampered source, cancellation and existing-journal compatibility. Verify CLI and MCP plus source Evidence retrieval without overwriting caller files. This report does not propose silently rebuilding or deleting a cache. ### References [#8167](https://github.com/morluto/rea/issues/8167) tracks retrying a failed cached Kuna function; [#8226](https://github.com/morluto/rea/issues/8226) tracks loss of per-item failure association in `batch_decompile`. This question concerns provenance of successfully written directory files across resume, whose existing journal does retain per-function caveats.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:11:49 +00:00

bsakaguchi commented

2026-11-01 21:11:50 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:50 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

The mixed partial-cache / clean-cache resume scenario also works through the public CLI: the control in #8167 ran three rea decompile-all --format json processes, retained the first successful file unchanged, and wrote the failed file after selecting the separately prepared clean cache. The final index and journal kept output hashes and per-function caveats; each CLI response was composed decompile_all Evidence with raw_result: null and evidence_links: []. The directory records therefore still do not identify the exact project-export producer for each successful file in this control. This extends the observed interface coverage; the injected stall and independent caches do not establish natural failure frequency or semantic equivalence of exports.

@morluto commented on GitHub (Oct 31, 2026): The mixed partial-cache / clean-cache resume scenario also works through the public CLI: [the control in #8167](https://github.com/morluto/rea/issues/8167#issuecomment-6544380376) ran three `rea decompile-all --format json` processes, retained the first successful file unchanged, and wrote the failed file after selecting the separately prepared clean cache. The final index and journal kept output hashes and per-function caveats; each CLI response was composed `decompile_all` Evidence with `raw_result: null` and `evidence_links: []`. The directory records therefore still do not identify the exact project-export producer for each successful file in this control. This extends the observed interface coverage; the injected stall and independent caches do not establish natural failure frequency or semantic equivalence of exports.

bsakaguchi commented

2026-11-01 21:11:50 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

A complementary Ghidra control on REA 60149282: an export created from a fresh import could not be resumed in a separate session selecting a saved, byte-edited Ghidra project for the same target. Their target SHA-256 and provider matched, but their committed profile digests differed. REA rejected the resume and kept all directory-file hashes unchanged; resuming under the original fresh-import profile then skipped the verified function. The target and selected source-project files were unchanged. This is a working cross-profile safety boundary, distinct from this issue's same-profile, different producer-export identity.

The mismatch error does not identify the differing component and recommends inspecting the input schema; I filed that separate recovery issue as #8349. One fixture and Ghidra version were tested; no target execution.

@morluto commented on GitHub (Oct 31, 2026): A complementary Ghidra control on REA `60149282`: an export created from a fresh import could not be resumed in a separate session selecting a saved, byte-edited Ghidra project for the same target. Their target SHA-256 and provider matched, but their committed profile digests differed. REA rejected the resume and kept all directory-file hashes unchanged; resuming under the original fresh-import profile then skipped the verified function. The target and selected source-project files were unchanged. This is a working cross-profile safety boundary, distinct from this issue's same-profile, different producer-export identity. The mismatch error does not identify the differing component and recommends inspecting the input schema; I filed that separate recovery issue as #8349. One fixture and Ghidra version were tested; no target execution.

bsakaguchi commented

2026-11-01 21:11:50 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

The same-profile provenance problem also occurs with Ghidra's live session edits, without separate Kuna caches. On REA 60149282 and real Ghidra 12.1.4 MCP, I used an owned unexecuted Mach-O where _evaluate calls _adjust:

Exporting _evaluate to a new directory wrote a file referring to _adjust.
annotate_native_function renamed _adjust to audit_new_adjust in the ephemeral session. A fresh procedure_pseudo_code(_evaluate) then referred to audit_new_adjust.
Resuming the original directory for _evaluate skipped the verified existing file and kept its digest and old reference. Resuming the directory for audit_new_adjust wrote the renamed callee alongside it. The index retained both file hashes but no per-file session-edit revision; its top-level limitations were empty. The manifest's provider profile digest was the same before and after the rename.
Exporting _evaluate to a new directory after the rename produced the updated call name. Individual pre/post pseudocode Evidence IDs were distinct; the decompile_all Evidence records had null raw results and no source Evidence links in this run.

Keeping the verified existing file is consistent with resume's documented no-overwrite behavior; the issue is that the resulting directory alone does not establish whether its files were produced before or after the session edit. This is a concrete mixed-state control for any per-file provenance design, not evidence of a semantic behavior change or a recommendation to overwrite caller files. Compact MCP schemas were fetched before calls; the target digest stayed unchanged and the target was not executed. One fixture and provider version were tested.

@morluto commented on GitHub (Oct 31, 2026): The same-profile provenance problem also occurs with **Ghidra's live session edits**, without separate Kuna caches. On REA `60149282` and real Ghidra 12.1.4 MCP, I used an owned unexecuted Mach-O where `_evaluate` calls `_adjust`: 1. Exporting `_evaluate` to a new directory wrote a file referring to `_adjust`. 2. `annotate_native_function` renamed `_adjust` to `audit_new_adjust` in the ephemeral session. A fresh `procedure_pseudo_code(_evaluate)` then referred to `audit_new_adjust`. 3. Resuming the original directory for `_evaluate` skipped the verified existing file and kept its digest and old reference. Resuming the directory for `audit_new_adjust` wrote the renamed callee alongside it. The index retained both file hashes but no per-file session-edit revision; its top-level limitations were empty. The manifest's provider profile digest was the same before and after the rename. 4. Exporting `_evaluate` to a **new** directory after the rename produced the updated call name. Individual pre/post pseudocode Evidence IDs were distinct; the `decompile_all` Evidence records had null raw results and no source Evidence links in this run. Keeping the verified existing file is consistent with `resume`'s documented no-overwrite behavior; the issue is that the resulting directory alone does not establish whether its files were produced before or after the session edit. This is a concrete mixed-state control for any per-file provenance design, not evidence of a semantic behavior change or a recommendation to overwrite caller files. Compact MCP schemas were fetched before calls; the target digest stayed unchanged and the target was not executed. One fixture and provider version were tested.

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 14:28:59 +00:00

bsakaguchi commented

2026-11-03 14:29:00 +00:00

Author

Owner

@morluto commented on GitHub (Nov 3, 2026):

A source-level follow-up on the current decompile_all boundary narrows the design choice (this is not a new runtime reproduction):

publishOne receives the actual procedure_pseudo_code execution before writing its file, but its journal record keeps the output-file digest, body fact and caveats rather than the execution's source. This is the earliest point where the producer's source can be associated with the committed file.
The lack of per-function Evidence is deliberate for documentless export reads: isSourceEvidenceRetainable excludes those pseudocode bodies because retaining every body in the in-memory ledger would duplicate the file-backed export. Simply adding a document selector to force source Evidence would change parameters and defeat that resource boundary; a fix should not do so implicitly.
Kuna's execution can report a project-export file digest and byte range; Ghidra's ordinary function result need not include a session-edit revision or an equivalent project-export descriptor. A provider-neutral source field must therefore permit an explicit unknown and must not turn the profile digest, output-file digest, or a directory path into a claim about a specific session revision.

A possible direction is a bounded, provider-owned source observation recorded with each journal outcome at commit time, preserving only facts actually available; assess whether any separate retained record can be retrieved and authenticated without copying every function body into memory. A digest or Evidence ID by itself would not make the corresponding source record retrievable. Resume should preserve earlier records and their unknowns, including after a matching existing-file commit; any schema change needs its own old-journal and torn-write tests. This is a design direction, not a completed implementation or a claim that current Ghidra output already supplies revision identity.

@morluto commented on GitHub (Nov 3, 2026): A source-level follow-up on the current `decompile_all` boundary narrows the design choice (this is not a new runtime reproduction): - `publishOne` receives the actual `procedure_pseudo_code` execution before writing its file, but its journal record keeps the output-file digest, body fact and caveats rather than the execution's source. This is the earliest point where the producer's source can be associated with the committed file. - The lack of per-function Evidence is deliberate for documentless export reads: `isSourceEvidenceRetainable` excludes those pseudocode bodies because retaining every body in the in-memory ledger would duplicate the file-backed export. Simply adding a document selector to force source Evidence would change parameters and defeat that resource boundary; a fix should not do so implicitly. - Kuna's execution can report a project-export file digest and byte range; Ghidra's ordinary function result need not include a session-edit revision or an equivalent project-export descriptor. A provider-neutral source field must therefore permit an explicit unknown and must not turn the profile digest, output-file digest, or a directory path into a claim about a specific session revision. A possible direction is a bounded, provider-owned source observation recorded with each journal outcome at commit time, preserving only facts actually available; assess whether any separate retained record can be retrieved and authenticated without copying every function body into memory. A digest or Evidence ID by itself would not make the corresponding source record retrievable. Resume should preserve earlier records and their unknowns, including after a matching existing-file commit; any schema change needs its own old-journal and torn-write tests. This is a design direction, not a completed implementation or a claim that current Ghidra output already supplies revision identity.

bsakaguchi commented

2026-11-03 14:29:00 +00:00

Author

Owner

@morluto commented on GitHub (Nov 3, 2026):

A further public MCP control on REA 00159ef7c, Kuna 1.989 and Ghidra 12.1.4 (macOS arm64) exported one selected function for each provider, then inspected the index and bundle before a separate direct pseudocode request. Advertised output schemas validated. In each case the index's written row had address, name, output path/digest, body and limitations, but no producer observation; the decompile_all Evidence had no links and its bundle had no pseudocode record. The selected target's SHA-256 remained unchanged and it was not executed.

I ran a cold control and a separate control that first waited for procedure_assembly. In the cold control the subsequent Kuna pseudocode observation reported source.kind: single-function. In the warm control it reported source.kind: project-export with the C-file digest and byte range. The corresponding sampled Ghidra pseudocode observations had raw_result: null in both controls. These are separate direct observations, not a claim that the existing directory record identifies its own producer. They demonstrate that even for the same target, a future record must capture what each export call actually observed; a provider profile or inferred source from a later query is insufficient. One function, one host and these provider versions were tested.

@morluto commented on GitHub (Nov 3, 2026): A further public MCP control on REA `00159ef7c`, Kuna 1.989 and Ghidra 12.1.4 (macOS arm64) exported one selected function for each provider, then inspected the index and bundle before a separate direct pseudocode request. Advertised output schemas validated. In each case the index's written row had address, name, output path/digest, body and limitations, but no producer observation; the `decompile_all` Evidence had no links and its bundle had no pseudocode record. The selected target's SHA-256 remained unchanged and it was not executed. I ran a cold control and a separate control that first waited for `procedure_assembly`. In the cold control the subsequent Kuna pseudocode observation reported `source.kind: single-function`. In the warm control it reported `source.kind: project-export` with the C-file digest and byte range. The corresponding sampled Ghidra pseudocode observations had `raw_result: null` in both controls. These are **separate direct observations**, not a claim that the existing directory record identifies its own producer. They demonstrate that even for the same target, a future record must capture what each export call actually observed; a provider profile or inferred source from a later query is insufficient. One function, one host and these provider versions were tested.

bsakaguchi commented

2026-11-03 15:31:08 +00:00

Author

Owner

@morluto commented on GitHub (Nov 3, 2026):

Ghidra's DomainObject.getModificationNumber() documentation describes a change/undo/redo counter intended for uses such as cache invalidation; it does not claim a globally unique or content-addressed revision.

I exercised Ghidra 12.1.4 headless on the owned arm64 fixture using a disposable project, without executing or modifying the target. The script explicitly ended GhidraScript's outer transaction before the probe, as REA's bridge does. In two independent fresh imports, both counters were 438 before an edit and 439 after a committed rename, although one session named the function rea_revision_probe and the other named it rea_revision_other. A subsequent aborted rename restored the respective committed name and moved both counters to 441. The simple read between measurements left 438 unchanged. These are observations for this fixture, not a general determinism or uniqueness guarantee; Ghidra also logged DWARF expression errors during analysis.

An initial probe left the script's outer transaction open; a nested abort had not completed its rollback when measured. That run is not evidence of REA behavior: the bridge explicitly ends the outer transaction first. The counter may help label changes within one identified session, but the measured collision shows that the counter alone cannot identify a program state across sessions or establish content equivalence. A durable record would need a separately owned session identity and semantics for in-flight changes; this check does not establish those or propose persisting a Ghidra database.

@morluto commented on GitHub (Nov 3, 2026): Ghidra's [`DomainObject.getModificationNumber()` documentation](https://ghidra.re/ghidra_docs/api/ghidra/framework/model/DomainObject.html#getModificationNumber()) describes a change/undo/redo counter intended for uses such as cache invalidation; it does not claim a globally unique or content-addressed revision. I exercised Ghidra 12.1.4 headless on the owned arm64 fixture using a disposable project, without executing or modifying the target. The script explicitly ended GhidraScript's outer transaction before the probe, as REA's bridge does. In two independent fresh imports, both counters were 438 before an edit and 439 after a committed rename, although one session named the function `rea_revision_probe` and the other named it `rea_revision_other`. A subsequent aborted rename restored the respective committed name and moved both counters to 441. The simple read between measurements left 438 unchanged. These are observations for this fixture, not a general determinism or uniqueness guarantee; Ghidra also logged DWARF expression errors during analysis. An initial probe left the script's outer transaction open; a nested abort had not completed its rollback when measured. That run is not evidence of REA behavior: the bridge explicitly ends the outer transaction first. The counter may help label changes **within one identified session**, but the measured collision shows that the counter alone cannot identify a program state across sessions or establish content equivalence. A durable record would need a separately owned session identity and semantics for in-flight changes; this check does not establish those or propose persisting a Ghidra database.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6516

No description provided.

Rows
Columns