morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8215] [Design] Surface relevant analysis-profile differences in function comparisons #6491

New issue

Open

opened 2026-11-01 21:11:37 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-11-01 21:11:37 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8215

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Should compare_functions expose a bounded explanation when two source dossiers were produced with different analysis profiles, so an agent can assess whether a textual difference may reflect analysis configuration?

Context and evidence

On macOS arm64, REA 38433cfb4577b133d7598f9439c6448299cb44eb, and real Kuna 1.986, I analyzed the same self-built unexecuted Mach-O in separate MCP sessions with REA_KUNA_JOBS=1 and 2. Both forced a complete project export before reading _audit_score. Direct Kuna runs and REA agreed on the selected outputs. The one-worker pseudocode rendered a floating-point call argument as 2.5; the two-worker output rendered it as 0x4004000000000000 and added casts. Other displayed declarations also differed. The target SHA-256 was unchanged.

Each REA pseudocode record preserved the requested jobs in its analysis profile and used a distinct profile digest and project-export key. After exporting and importing both bundles, compare_functions accurately marked pseudocode text changed, assembly unchanged, and overall identity unknown because Kuna symbol origin was unavailable. The comparison's text limitation explains that textual changes do not establish behavior change, but the comparison result does not identify the profiles' differing job settings; the caller must follow source Evidence to find them. REA's Kuna guide documents the one-worker versus parallel prototype tradeoff. This example shows a concrete reason that text can differ for the same artifact; it does not prove that one representation is universally more correct, that any runtime behavior changed, or that the selected worker count was always used on other targets.

Expected interaction with targets

Read authenticated existing Evidence; the example's two provider runs read local binary files and wrote owned caches. No target execution is proposed.

Boundary details (optional)

Keep exact source profiles and their Evidence references authoritative. Different profile digests alone do not prove which setting caused a change, and equal profiles do not establish identical provider state. Any field-level explanation must preserve absent/unknown values, avoid conflating secret credentials with ordinary selected values, bound output, and not equate addresses or behavior across contexts. A textual changed result remains a valid observation.

Options and tradeoffs

Keep the existing source-Evidence lookup and improve its discovery; add a small profile-identity/difference summary to the existing comparison; or expose a reusable bounded view over source profiles. Compare costs and usefulness before choosing. Avoid new Kuna-specific wrappers or embedding entire repeated profiles in every comparison.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

Exercise same/different profiles on the same and different artifacts; explicit unknown/missing profile facts; allowed provider changes; bounded large profiles; retained/bundled Evidence; and comparison behavior when name or coordinate identity is unknown. Distinguish observed configuration differences from causal explanations. Check the cold single-function versus project-export transition separately; the configured worker count alone does not identify an actual producer mode.

References

Kuna provider guide; function comparison contract. Related: #8173 (unknown Kuna symbol origin), #8191 (cold versus project representation), #8213 (same-entry rename identity), and #1947 (text normalization).

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8215 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Should `compare_functions` expose a bounded explanation when two source dossiers were produced with different analysis profiles, so an agent can assess whether a textual difference may reflect analysis configuration? ### Context and evidence On macOS arm64, REA `38433cfb4577b133d7598f9439c6448299cb44eb`, and real Kuna 1.986, I analyzed the same self-built unexecuted Mach-O in separate MCP sessions with `REA_KUNA_JOBS=1` and `2`. Both forced a complete project export before reading `_audit_score`. Direct Kuna runs and REA agreed on the selected outputs. The one-worker pseudocode rendered a floating-point call argument as `2.5`; the two-worker output rendered it as `0x4004000000000000` and added casts. Other displayed declarations also differed. The target SHA-256 was unchanged. Each REA pseudocode record preserved the requested `jobs` in its analysis profile and used a distinct profile digest and project-export key. After exporting and importing both bundles, `compare_functions` accurately marked pseudocode text `changed`, assembly `unchanged`, and overall identity `unknown` because Kuna symbol origin was unavailable. The comparison's text limitation explains that textual changes do not establish behavior change, but the comparison result does not identify the profiles' differing job settings; the caller must follow source Evidence to find them. REA's Kuna guide documents the one-worker versus parallel prototype tradeoff. This example shows a concrete reason that text can differ for the same artifact; it does not prove that one representation is universally more correct, that any runtime behavior changed, or that the selected worker count was always used on other targets. ### Expected interaction with targets Read authenticated existing Evidence; the example's two provider runs read local binary files and wrote owned caches. No target execution is proposed. ### Boundary details (optional) Keep exact source profiles and their Evidence references authoritative. Different profile digests alone do not prove which setting caused a change, and equal profiles do not establish identical provider state. Any field-level explanation must preserve absent/unknown values, avoid conflating secret credentials with ordinary selected values, bound output, and not equate addresses or behavior across contexts. A textual `changed` result remains a valid observation. ### Options and tradeoffs Keep the existing source-Evidence lookup and improve its discovery; add a small profile-identity/difference summary to the existing comparison; or expose a reusable bounded view over source profiles. Compare costs and usefulness before choosing. Avoid new Kuna-specific wrappers or embedding entire repeated profiles in every comparison. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions Exercise same/different profiles on the same and different artifacts; explicit unknown/missing profile facts; allowed provider changes; bounded large profiles; retained/bundled Evidence; and comparison behavior when name or coordinate identity is unknown. Distinguish observed configuration differences from causal explanations. Check the cold single-function versus project-export transition separately; the configured worker count alone does not identify an actual producer mode. ### References [Kuna provider guide](https://github.com/morluto/rea/blob/38433cfb4577b133d7598f9439c6448299cb44eb/docs/kuna-provider.md); [function comparison contract](https://github.com/morluto/rea/blob/38433cfb4577b133d7598f9439c6448299cb44eb/docs/mcp-contracts.md). Related: #8173 (unknown Kuna symbol origin), #8191 (cold versus project representation), #8213 (same-entry rename identity), and #1947 (text normalization).

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:11:37 +00:00

bsakaguchi commented

2026-11-01 21:11:37 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A Ghidra existing-project control shows a different reason the same artifact can yield changed text. I analyzed the same unexecuted Mach-O in two real MCP sessions: one fresh import, and one caller-selected read-only project whose in-program instruction had been deliberately changed earlier by a separate headless script (#8232). Each dossier retained the same target SHA-256 and procedure name/address, but different profile digests; the existing-project profile carried its selected project fingerprint and warned that prior edits may exist. Both bundles were exported, then imported into a fresh MCP server for comparison.

compare_functions matched entry by symbol, marked pseudocode and assembly text changed, and left references unknown because a common coordinate commitment was not established. It retained links to both source dossiers and the existing-project caveat. find_changed_behavior reported two static candidates, one unresolved dimension and behavior_status: unknown; no runtime change was claimed. An explicitly requested structural verification failed the changed pseudocode, assembly and overall claims, and passed identity, calls and CFG. Those text differences are genuine for the selected analysis programs, but the identical executable digest means they are not evidence of a change to the selected file.

The project profile and limitations allow a caller to inspect the actual inputs; the comparison does not summarize the differing import mode or selected project fingerprint. This is another reason to make observed profile differences discoverable without asserting they alone caused the text change. A generic Artifact identity is unavailable for this observation limitation was also present on the comparison created without an active target, even though its result included both source subject digests; that wording concerns the derived observation's absent single subject and should not be read as absence of the source identities. This one comparison does not validate all snapshot, coordinate or matching behavior. The target digest stayed unchanged; the project edit preceded REA's read-only sessions.

@morluto commented on GitHub (Oct 30, 2026): A Ghidra existing-project control shows a different reason the same artifact can yield changed text. I analyzed the same unexecuted Mach-O in two real MCP sessions: one fresh import, and one caller-selected read-only project whose in-program instruction had been deliberately changed earlier by a separate headless script (#8232). Each dossier retained the same target SHA-256 and procedure name/address, but different profile digests; the existing-project profile carried its selected project fingerprint and warned that prior edits may exist. Both bundles were exported, then imported into a fresh MCP server for comparison. `compare_functions` matched `entry` by symbol, marked pseudocode and assembly text changed, and left references unknown because a common coordinate commitment was not established. It retained links to both source dossiers and the existing-project caveat. `find_changed_behavior` reported two static candidates, one unresolved dimension and `behavior_status: unknown`; no runtime change was claimed. An explicitly requested structural verification failed the changed pseudocode, assembly and overall claims, and passed identity, calls and CFG. Those text differences are genuine for the selected analysis programs, but the identical executable digest means they are not evidence of a change to the selected file. The project profile and limitations allow a caller to inspect the actual inputs; the comparison does not summarize the differing import mode or selected project fingerprint. This is another reason to make **observed** profile differences discoverable without asserting they alone caused the text change. A generic `Artifact identity is unavailable for this observation` limitation was also present on the comparison created without an active target, even though its result included both source subject digests; that wording concerns the derived observation's absent single subject and should not be read as absence of the source identities. This one comparison does not validate all snapshot, coordinate or matching behavior. The target digest stayed unchanged; the project edit preceded REA's read-only sessions.

bsakaguchi commented

2026-11-01 21:11:37 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:37 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

An equal-profile control illustrates the limitation already noted in this issue. On REA 60149282, real Ghidra 12.1.4 MCP analyzed _evaluate, renamed its callee _adjust to audit_changed_callee in the ephemeral database, and analyzed _evaluate again. Both caller dossiers had the same target and profile digest, but the callee name and its pseudocode call changed; the callee address did not. The address-only get_call_graph result and Evidence ID were unchanged.

compare_functions matched the caller by same entry, marked identity, assembly, calls and references unchanged, and marked pseudocode and referenced names changed. find_changed_behavior returned two static candidates, zero observed changes and behavior_status: unknown. Both source dossier Evidence IDs were retained. After exporting the bundle and importing it into a fresh target-free server, the same comparison and behavior results and their Evidence IDs were reproduced exactly; the old and new pseudocode views remained separately accessible. SDK compact output schemas were fetched before the successful calls; the target and bundle hashes were unchanged, and the target was not executed.

This is working conservative behavior for one fixture. It supports keeping “same profile” separate from “same live analysis state”: profile-difference summaries alone cannot explain a text change caused by a session edit. It also does not prove that the rename caused a runtime behavior change or that all metadata changes are captured. Related directory-export mixed-state control: #8260.

@morluto commented on GitHub (Oct 31, 2026): An equal-profile control illustrates the limitation already noted in this issue. On REA `60149282`, real Ghidra 12.1.4 MCP analyzed `_evaluate`, renamed its callee `_adjust` to `audit_changed_callee` in the ephemeral database, and analyzed `_evaluate` again. Both caller dossiers had the same target and profile digest, but the callee name and its pseudocode call changed; the callee address did not. The address-only `get_call_graph` result and Evidence ID were unchanged. `compare_functions` matched the caller by same entry, marked identity, assembly, calls and references unchanged, and marked pseudocode and referenced names changed. `find_changed_behavior` returned two static candidates, zero observed changes and `behavior_status: unknown`. Both source dossier Evidence IDs were retained. After exporting the bundle and importing it into a fresh target-free server, the same comparison and behavior results **and their Evidence IDs** were reproduced exactly; the old and new pseudocode views remained separately accessible. SDK compact output schemas were fetched before the successful calls; the target and bundle hashes were unchanged, and the target was not executed. This is working conservative behavior for one fixture. It supports keeping “same profile” separate from “same live analysis state”: profile-difference summaries alone cannot explain a text change caused by a session edit. It also does not prove that the rename caused a runtime behavior change or that all metadata changes are captured. Related directory-export mixed-state control: #8260.

bsakaguchi commented

2026-11-01 21:11:37 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

A cross-version control on the same unexecuted ARM64 Mach-O adds a distinct case from the worker-count example above. Verified Kuna 1.986 produced pseudocode ending return audit_global, while an unverified build from current Kuna source (8c74a944d74f8d3349755e05441a9628b4753111) produced return &_audit_global. The instruction sequence moves the global's computed address into the return register; Ghidra's decompile also returns the address. This is one static counterexample, not a version-wide correctness assessment or upgrade recommendation.

I obtained complete REA dossiers after each project's export, exported their bundles, imported both into a fresh MCP server without opening a target, and called compare_functions with the retained IDs. Both source records and provider/profile identities survived. The result reports pseudocode and assembly unknown with “Exact text comparison requires one provider identity” because the reported provider versions differ (1.986 versus the source build's unverified 0.1.0), rather than exposing the observed textual difference; function matching is also ambiguous because no common coordinate commitment or stable symbol origin was established. Comparing the 1.986 dossier with itself marks its pseudocode and assembly unchanged. The comparator correctly avoids claiming a binary behavior change.

This illustrates an upgrade-assessment question alongside this issue's profile explanation: can an agent explicitly request or discover a bounded textual comparison across provider versions, without treating different decompiler renderings as semantic evidence or asserting function correspondence beyond the sources? Compare that with reading the two retained dossier facets directly. This is not a request to implicitly treat the source build as a verified release. The target SHA-256 9b1f167a30cd17cb5d4393d720336fc14d7ac3bce20ade06182b941ecf8918fd stayed unchanged. The separate identical-record function identity issue is #8367.

@morluto commented on GitHub (Oct 31, 2026): A cross-version control on the same unexecuted ARM64 Mach-O adds a distinct case from the worker-count example above. Verified Kuna 1.986 produced pseudocode ending `return audit_global`, while an **unverified** build from current Kuna source (`8c74a944d74f8d3349755e05441a9628b4753111`) produced `return &_audit_global`. The instruction sequence moves the global's computed address into the return register; Ghidra's decompile also returns the address. This is one static counterexample, not a version-wide correctness assessment or upgrade recommendation. I obtained complete REA dossiers after each project's export, exported their bundles, imported both into a fresh MCP server without opening a target, and called `compare_functions` with the retained IDs. Both source records and provider/profile identities survived. The result reports pseudocode and assembly `unknown` with “Exact text comparison requires one provider identity” because the reported provider versions differ (`1.986` versus the source build's unverified `0.1.0`), rather than exposing the observed textual difference; function matching is also ambiguous because no common coordinate commitment or stable symbol origin was established. Comparing the 1.986 dossier with itself marks its pseudocode and assembly unchanged. The comparator correctly avoids claiming a binary behavior change. This illustrates an upgrade-assessment question alongside this issue's profile explanation: can an agent explicitly request or discover a bounded **textual** comparison across provider versions, without treating different decompiler renderings as semantic evidence or asserting function correspondence beyond the sources? Compare that with reading the two retained dossier facets directly. This is not a request to implicitly treat the source build as a verified release. The target SHA-256 `9b1f167a30cd17cb5d4393d720336fc14d7ac3bce20ade06182b941ecf8918fd` stayed unchanged. The separate identical-record function identity issue is #8367.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6491

No description provided.

Rows
Columns