morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8173] [Design] Use independent symbol evidence to match functions when a provider omits origins #4699

New issue

Open

opened 2026-10-31 00:52:36 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-31 00:52:36 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8173

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Can function matching use independently observed, target-bound symbol facts to establish symbol origin when a provider such as Kuna cannot supply it, without treating generated names or data exports as functions?

Context and evidence

Two self-built, unexecuted arm64 Mach-O versions differed only in a small helper's addition versus subtraction. On REA source 47fa6ab3795d547207001ffb01643c0aa39dcad8, Ghidra 12.1.4 list_procedures Evidence let match_native_functions pair the three reported functions by observed symbol origin; compare_functions marked pseudocode and assembly changed for the helper, with its normal limitations.

On the same binaries, Kuna 1.833 inventories had six entries per side. Matching those inventories yielded zero matches, six only_left and six only_right, with source_coverage.status: partial and per-row Symbol origin unavailable; exact-name matching is unavailable limitations. Those states are explicitly not proof of added or removed functions. Directly comparing the selected helper dossiers marked text changed but overall status unknown. Kuna does not currently supply REA list_names, procedure_info, or read_bytes observations for independent matching.

Separately, REA inspect_macho on one of these binaries returned an _adjust export at the same address as the Kuna procedure. Supplying that retained Evidence in left_observations was refused: matching accepts only list_names, procedure_info, read_bytes, or analyze_function Evidence from the exact provider/profile context. This is an observed UX gap, not evidence that every export is a function: the same inspection listed _global_count and _message as exports too. #8168 covers those data entries appearing in Kuna's procedure inventory.

Expected interaction with targets

Read caller-selected local binaries and existing Evidence; invoke the existing caller-supplied static provider where selected. No target execution or binary modification is proposed.

Boundary details (optional)

Any candidate must preserve exact binary identity, architecture/slice, source names, address-space and image-base mapping, aliases, symbol type and origin, conflicting or incomplete observations, and independent Evidence links. An export name alone does not establish a function body, behavior, or safe one-to-one correspondence. Avoid assuming Ghidra and Kuna addresses agree: on a separate x86-64 PIE fixture their bases differed.

Options and tradeoffs

Explore reusing format-owned symbol Evidence or enriching Kuna inventory with a separately verified native symbol observation, then joining only compatible exact rows. Compare the effort and coverage with the existing Ghidra workflow and with explicit analyst selection. Do not add a raw Kuna command wrapper just for parity; names that cannot be grounded remain unknown.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

Determine whether a format-aware source can safely distinguish function symbols, generated names, imports, aliases and data on Mach-O and ELF, and how it relates to provider addresses. Exercise symbolized, stripped, duplicate/alias, conflicting and rebased inputs; test CLI/MCP Evidence retention and cross-provider behavior. Preserve unknown coverage and do not infer behavioral equivalence. No implementation or other-architecture verification was performed.

References

In a follow-up on the same sessions' exported bundles, supplying the two retained _adjust dossiers as left_observations and right_observations produced the same zero Kuna matches and no structural candidates. Ghidra retained three symbol matches and supplied a ready-to-use compare_functions input for _adjust. This is one symbolized fixture, not evidence about all binaries or providers.

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8173 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Can function matching use independently observed, target-bound symbol facts to establish symbol origin when a provider such as Kuna cannot supply it, without treating generated names or data exports as functions? ### Context and evidence Two self-built, unexecuted arm64 Mach-O versions differed only in a small helper's addition versus subtraction. On REA source `47fa6ab3795d547207001ffb01643c0aa39dcad8`, Ghidra 12.1.4 `list_procedures` Evidence let `match_native_functions` pair the three reported functions by observed symbol origin; `compare_functions` marked pseudocode and assembly changed for the helper, with its normal limitations. On the same binaries, Kuna 1.833 inventories had six entries per side. Matching those inventories yielded zero matches, six `only_left` and six `only_right`, with `source_coverage.status: partial` and per-row `Symbol origin unavailable; exact-name matching is unavailable` limitations. Those states are explicitly not proof of added or removed functions. Directly comparing the selected helper dossiers marked text changed but overall status unknown. Kuna does not currently supply REA `list_names`, `procedure_info`, or `read_bytes` observations for independent matching. Separately, REA `inspect_macho` on one of these binaries returned an `_adjust` export at the same address as the Kuna procedure. Supplying that retained Evidence in `left_observations` was refused: matching accepts only `list_names`, `procedure_info`, `read_bytes`, or `analyze_function` Evidence from the exact provider/profile context. This is an observed UX gap, not evidence that every export is a function: the same inspection listed `_global_count` and `_message` as exports too. #8168 covers those data entries appearing in Kuna's procedure inventory. ### Expected interaction with targets Read caller-selected local binaries and existing Evidence; invoke the existing caller-supplied static provider where selected. No target execution or binary modification is proposed. ### Boundary details (optional) Any candidate must preserve exact binary identity, architecture/slice, source names, address-space and image-base mapping, aliases, symbol type and origin, conflicting or incomplete observations, and independent Evidence links. An export name alone does not establish a function body, behavior, or safe one-to-one correspondence. Avoid assuming Ghidra and Kuna addresses agree: on a separate x86-64 PIE fixture their bases differed. ### Options and tradeoffs Explore reusing format-owned symbol Evidence or enriching Kuna inventory with a separately verified native symbol observation, then joining only compatible exact rows. Compare the effort and coverage with the existing Ghidra workflow and with explicit analyst selection. Do not add a raw Kuna command wrapper just for parity; names that cannot be grounded remain unknown. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions Determine whether a format-aware source can safely distinguish function symbols, generated names, imports, aliases and data on Mach-O and ELF, and how it relates to provider addresses. Exercise symbolized, stripped, duplicate/alias, conflicting and rebased inputs; test CLI/MCP Evidence retention and cross-provider behavior. Preserve unknown coverage and do not infer behavioral equivalence. No implementation or other-architecture verification was performed. ### References - [Native function matching](https://github.com/morluto/rea/blob/47fa6ab3795d547207001ffb01643c0aa39dcad8/docs/native-function-matching.md) - [Matching source admission](https://github.com/morluto/rea/blob/47fa6ab3795d547207001ffb01643c0aa39dcad8/src/application/native/NativeMatchingSources.ts) - Related: #8168 and #8170. In a follow-up on the same sessions' exported bundles, supplying the two retained `_adjust` dossiers as `left_observations` and `right_observations` produced the same zero Kuna matches and no structural candidates. Ghidra retained three symbol matches and supplied a ready-to-use `compare_functions` input for `_adjust`. This is one symbolized fixture, not evidence about all binaries or providers.

bsakaguchi added the

triage

enhancement

labels

2026-10-31 00:52:36 +00:00

bsakaguchi referenced this issue

2026-10-31 01:13:53 +00:00

[PR #4699] [CLOSED] fix(kuna): resolve retained process failures once their owner releases #6364

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-01 21:11:17 +00:00

bsakaguchi commented

2026-11-01 21:11:17 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A further real Kuna 1.986 control used the same self-built Mach-O in two REA sessions, with project exports selected by REA_KUNA_JOBS=1 and 2. Both _audit_score dossiers reported the same name and entry, but Kuna did not establish symbol origin, so compare_functions kept function_match: ambiguous. The project-export pseudocode differed on this fixture (notably 2.5 versus 0x4004000000000000 and casts); the text dimension was correctly changed, while overall comparison stayed unknown. Profiles and cache keys differed and the target digest was unchanged.

This is consistent with the same unresolved identity boundary even without a binary version change; the currently emitted identity limitation mentions “cross-version identity,” which can be confusing in this same-artifact scenario. It is not proof that same numerical addresses across profiles/providers are always interchangeable. #8215 separately discusses making profile differences discoverable, not relaxing identity or claiming why a particular text changed.

@morluto commented on GitHub (Oct 30, 2026): A further real Kuna 1.986 control used the **same** self-built Mach-O in two REA sessions, with project exports selected by `REA_KUNA_JOBS=1` and `2`. Both `_audit_score` dossiers reported the same name and entry, but Kuna did not establish symbol origin, so `compare_functions` kept `function_match: ambiguous`. The project-export pseudocode differed on this fixture (notably `2.5` versus `0x4004000000000000` and casts); the text dimension was correctly `changed`, while overall comparison stayed `unknown`. Profiles and cache keys differed and the target digest was unchanged. This is consistent with the same unresolved identity boundary even without a binary version change; the currently emitted identity limitation mentions “cross-version identity,” which can be confusing in this same-artifact scenario. It is not proof that same numerical addresses across profiles/providers are always interchangeable. #8215 separately discusses making profile differences discoverable, not relaxing identity or claiming why a particular text changed.

bsakaguchi commented

2026-11-01 21:11:18 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A stripped-build follow-up exercised the existing byte-matching route on the two self-built Mach-O versions. I stripped disposable copies with strip -u -r; nm still reported the Mach-O header symbol, so this is not a claim that every symbol or metadata source was removed. The fixture targets were not executed and both SHA-256 values were unchanged by the REA sessions.

With real REA/Ghidra 12.1.4, each inventory contained two default-origin generated functions and an imported entry. Inventory-only matching paired the entry by exact symbol and left two rows on each side unpaired. I then captured procedure_info body ranges and read_bytes for all three functions on each side, exported/imported both Evidence bundles, and supplied those observations. The matcher paired one additional generated function (0x100000430) by identical-bytes; the modified helper (0x100000410) remained unpaired on each side. Coverage stayed partial with inventory completeness unknown. This validates the documented byte workflow on this fixture; byte identity does not establish source or behavioral equivalence.

With REA/Kuna 1.986 on the same copies, inventories contained four rows per side, including the Mach-O header as a procedure candidate. procedure_info returned typed capability unavailability for all eight rows, so the scripted body-range/byte route could not acquire its prerequisite ranges. Matching remained zero pairs, four unpaired per side, with partial coverage and zero byte fingerprints. Separately, #8170 records the unavailable REA/Kuna read_bytes operation. This is a real-provider control for the stripped case, not proof that all stripped binaries behave this way or that same-address rows should automatically be paired.

@morluto commented on GitHub (Oct 30, 2026): A stripped-build follow-up exercised the existing byte-matching route on the two self-built Mach-O versions. I stripped disposable copies with `strip -u -r`; `nm` still reported the Mach-O header symbol, so this is not a claim that every symbol or metadata source was removed. The fixture targets were not executed and both SHA-256 values were unchanged by the REA sessions. With real REA/Ghidra 12.1.4, each inventory contained two `default`-origin generated functions and an `imported` entry. Inventory-only matching paired the entry by exact symbol and left two rows on each side unpaired. I then captured `procedure_info` body ranges and `read_bytes` for all three functions on each side, exported/imported both Evidence bundles, and supplied those observations. The matcher paired one additional generated function (`0x100000430`) by `identical-bytes`; the modified helper (`0x100000410`) remained unpaired on each side. Coverage stayed partial with inventory completeness unknown. This validates the documented byte workflow on this fixture; byte identity does not establish source or behavioral equivalence. With REA/Kuna 1.986 on the same copies, inventories contained four rows per side, including the Mach-O header as a procedure candidate. `procedure_info` returned typed capability unavailability for all eight rows, so the scripted body-range/byte route could not acquire its prerequisite ranges. Matching remained zero pairs, four unpaired per side, with partial coverage and zero byte fingerprints. Separately, #8170 records the unavailable REA/Kuna `read_bytes` operation. This is a real-provider control for the stripped case, not proof that all stripped binaries behave this way or that same-address rows should automatically be paired.

bsakaguchi commented

2026-11-01 21:11:18 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

A stripped x86-64 ELF duplicate-body control exercises the existing Ghidra byte route. Two functions on the left and three on the right have identical six-byte bodies; a separate unique six-byte function matches across builds. After collecting real procedure_info and read_bytes Evidence, exporting/importing both bundles, and matching, REA returned the repeated bodies as one unresolved two-versus-three group without forcing a pair; it paired the unique body by identical-bytes. Inventory completeness remained unknown. Neither binary was executed or modified. This confirms the intended ambiguity handling for this constructed case, not a universal equivalence claim.

The same control exposed a separate origin problem: Ghidra reports a loader-created stripped ELF entry label as imported, which REA accepts as exact-symbol identity. A different-entry-body control and direct comparison are filed at #8257. This is distinct from Kuna's unavailable origin and reinforces the need to distinguish source-symbol provenance from provider-created role labels.

@morluto commented on GitHub (Oct 31, 2026): A stripped x86-64 ELF duplicate-body control exercises the existing Ghidra byte route. Two functions on the left and three on the right have identical six-byte bodies; a separate unique six-byte function matches across builds. After collecting real `procedure_info` and `read_bytes` Evidence, exporting/importing both bundles, and matching, REA returned the repeated bodies as one unresolved two-versus-three group without forcing a pair; it paired the unique body by `identical-bytes`. Inventory completeness remained unknown. Neither binary was executed or modified. This confirms the intended ambiguity handling for this constructed case, not a universal equivalence claim. The same control exposed a separate origin problem: Ghidra reports a loader-created stripped ELF entry label as imported, which REA accepts as exact-symbol identity. A different-entry-body control and direct comparison are filed at [#8257](https://github.com/morluto/rea/issues/8257). This is distinct from Kuna's unavailable origin and reinforces the need to distinguish source-symbol provenance from provider-created role labels.

bsakaguchi commented

2026-11-01 21:11:18 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:18 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Cross-provider matching on the same stripped artifact

I imported previously captured real Ghidra 12.1.4 and Kuna 1.986 Evidence for a self-owned stripped arm64 Mach-O into a fresh, target-free REA b43c329df779708e43ae297f498a844e45d9c923 MCP server. The two inventory records had the same target SHA-256; Ghidra reported three entries and Kuna four. Inventory-only match_native_functions returned zero matches, three only_left, four only_right and partial source coverage. These mean unpaired within the supplied inventories, not added/deleted functions.

Adding the Ghidra-owned body ranges and byte observations for all three entries on the Ghidra side produced three left byte fingerprints but zero Kuna-side fingerprints and still zero matches. Supplying a Ghidra observation on the Kuna side instead was refused with invalid_request, explicitly requiring the inventory's exact target, provider, profile and document context. This refusal also occurred in the same-target case, so equal artifact digests do not bypass source ownership. Repeating against the separately built other version gave the same counts on this small fixture.

This confirms an existing safe boundary and a task-level gap: an analyst can collect independently useful Ghidra observations, but cannot pass them off as Kuna-produced observations to complete a cross-provider match. It does not justify weakening ownership, equating numeric addresses, or assuming an export is a function; a cross-provider join would need explicit mapping/applicability and provenance. The matching calls used imported Evidence only; no provider or target was reopened. Both source targets and the three input bundles retained their SHA-256 values and neither target was executed.

@morluto commented on GitHub (Oct 31, 2026): ### Cross-provider matching on the same stripped artifact I imported previously captured real Ghidra 12.1.4 and Kuna 1.986 Evidence for a self-owned stripped arm64 Mach-O into a fresh, target-free REA `b43c329df779708e43ae297f498a844e45d9c923` MCP server. The two inventory records had the **same target SHA-256**; Ghidra reported three entries and Kuna four. Inventory-only `match_native_functions` returned zero matches, three `only_left`, four `only_right` and partial source coverage. These mean unpaired within the supplied inventories, not added/deleted functions. Adding the Ghidra-owned body ranges and byte observations for all three entries on the Ghidra side produced three left byte fingerprints but zero Kuna-side fingerprints and still zero matches. Supplying a Ghidra observation on the Kuna side instead was refused with `invalid_request`, explicitly requiring the inventory's exact target, provider, profile and document context. This refusal also occurred in the same-target case, so equal artifact digests do not bypass source ownership. Repeating against the separately built other version gave the same counts on this small fixture. This confirms an existing safe boundary and a task-level gap: an analyst can collect independently useful Ghidra observations, but cannot pass them off as Kuna-produced observations to complete a cross-provider match. It does not justify weakening ownership, equating numeric addresses, or assuming an export is a function; a cross-provider join would need explicit mapping/applicability and provenance. The matching calls used imported Evidence only; no provider or target was reopened. Both source targets and the three input bundles retained their SHA-256 values and neither target was executed.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4699

No description provided.

Rows
Columns