morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1193] [Feature] Analyze a caller-selected matching dSYM with the Ghidra provider #427

New issue

Open

opened 2026-10-15 18:24:58 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-15 18:24:58 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 9, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1193

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Current limitation

REA imports an immutable private executable copy into its ephemeral Ghidra project. A dSYM beside the original Mach-O is not available beside that private copy. On an observed source-built C fixture, REA decompiles pair_sum as two generated scalar parameters and cannot find its source Pair type, while an independent Ghidra session importing the original executable beside the matching dSYM recovers a single by-value Pair parameter, field names, source parameter names, and the type at /DWARF/fixture.c/Pair. This blocks an analyst who already owns the debug companion from obtaining Ghidra's more informative function semantics through REA.

This is a new capability request. The current tool contracts promise recovered database observations and do not promise dSYM companion admission, so the observed difference is not asserted as a violation of that contract.

Use cases

Let a Ghidra analysis session include an explicitly selected matching dSYM companion so native decompilation, recovered API boundaries, and data-type inspection can use the debug information available to the analyst.

Direct Ghidra or GhidraMCP analysis beside the original executable recovers these facts, and an embedded-DWARF object works for source-controlled fixtures, but neither supplies a shipped linked executable plus its dSYM to REA's composed dossiers and comparisons.

Desired outcome and success criteria

Allow the caller to select a local dSYM companion when opening a native Mach-O for Ghidra analysis through CLI or MCP. Establish that the companion matches the selected executable before analysis. Preserve the existing immutable artifact behavior and ephemeral annotation lifecycle. Report whether debug information was admitted and used, retain its identity in the analysis evidence, and keep unsupported or mismatched companions explicit.

Function dossiers, pseudocode, native API inspection, and exact data-type inspection should expose the facts Ghidra recovers from the selected companion. Debug declarations and decompiler inferences must retain distinct attribution; using debug information does not make pseudocode original source.

Success criteria:

Compile a thin arm64 Mach-O and matching dSYM from the inline source below. With the selected matching companion, CLI and MCP recover pair_sum's single by-value Pair parameter, its field names and 16-byte size, and expose /DWARF/fixture.c/Pair through exact type inspection.
With no companion selected, retain the current executable-only analysis behavior and report its narrower inputs accurately.
An explicitly selected dSYM for a different build must produce a mismatch or unsupported result with the relevant identities; it must not silently supply the other build's declarations or fall back while claiming debug coverage.
A missing, unreadable, or malformed selected companion must produce an actionable result that distinguishes those conditions.
Evidence and analysis-profile identity must distinguish analysis with the selected debug companion from executable-only analysis, so a snapshot cannot silently replay the wrong input set.
Original executable and dSYM bytes remain unchanged. Reopening the same live target with the same selected input identities preserves authorized session annotations under the existing lifecycle contract.

Authority boundaries and non-goals

Affected area: Native Ghidra target selection and its function-level CLI/MCP workflows. The existing private immutable import and evidence identity are binding constraints because the additional input must remain attributable without modifying the user's Ghidra installation or source artifacts.

Automatic symbol-server downloads, debugger execution, global filesystem searches for debug data, source-code reconstruction, and broad PDB or ELF debug-sidecar support are outside this request.

Alternatives considered

Direct Ghidra or GhidraMCP analysis beside the original executable already recovers the companion facts, but leaves the analyst unable to use those facts in REA's composed dossiers and comparisons. Compiling and selecting a DWARF-bearing object file works for source-controlled fixtures, but is not a replacement for analyzing a shipped linked executable and its existing dSYM. Explicit companion selection is preferable to automatically importing nearby files because it preserves the caller's chosen inputs and permits clear identity and mismatch checks.

Possible direction

No response

References

The observed comparison used rea-agents@6.1.0 and main commit b34bdcdb2877be394c59c9cbf70269dc1841ff78, Ghidra 12.1.4, OpenJDK 21.0.12.1, Node 24.21.0, and macOS 27 arm64. The independent oracle was bethington/ghidra-mcp stable v6.0.0 at commit 8cd2078e10b9ba28b188cb84ce5b9051a904b995, built against that Ghidra install and run headlessly on loopback with a disposable project.

The larger observed fixture contained this function and type. The following smaller source is an acceptance fixture, not a separately executed reduction.

struct Pair { long a; long b; };
__attribute__((noinline)) long pair_sum(struct Pair p) { return p.a + 3*p.b; }
int main(void) { return (int)pair_sum((struct Pair){1,2}); }

clang -arch arm64 -O2 -g -gdwarf-4 fixture.c -o fixture

For the executed control, the oracle returned long pair_sum(Pair p) and return p.b * 3 + p.a; when the matching dSYM was beside the original file. Importing a byte-identical executable copy under a new basename without a dSYM returned the same generated scalar-parameter pseudocode as REA. That control establishes the companion-input difference rather than a decompiler disagreement. The exact-source DWARF-bearing object also gave REA a recovered Pair parameter, demonstrating that embedded debug facts already reach the function workflows.

src/ghidra/GhidraTargetSnapshot.ts identifies the executable-only immutable snapshot boundary. docs/native-investigation.md, "Native instruction, call and type primitives", defines exact database pathname inspection and database authority. Read-only issue searches for dSYM found no existing issue for this selected-companion workflow.

Ghidra and Kuna-backend input control

A separate source-built arm64 Mach-O with struct Sample { int count; int scale; } and its adjacent dSYM gave Ghidra 12.1.4 headless enough type information for both the stock decompiler and the released Kuna 1.989 Ghidra backend to render the named fields. Ghidra reported some unrecoverable DWARF frame/register expressions. On a byte-identical copy without the companion, both headless backends rendered integer-pointer expressions instead. Real REA Ghidra and independent Kuna function dossiers on the original executable also rendered integer-pointer expressions. The executable SHA-256 stayed c244f7d7d9fa9b7f4cfcb1f69d69cc1f83ad98fe38b0293b223b3ca7d9076363; neither copy was executed.

This independently supports the selected-companion concern for one additional fixture and shows why changing decompiler backend alone does not substitute for binding debug inputs. It does not demonstrate a general dSYM import guarantee, prove all debug facts were applied, or test an REA hybrid backend. See #8177 for the complete experimental scope.

Existing-project workaround and offline handoff

For the typed fixture above, I imported the executable with its adjacent dSYM into a disposable Ghidra 12.1.4 project and saved it. Real REA/Ghidra, configured with that explicitly selected project and program, opened its private copy and analyze_function(0x100000410) returned Sample * and both named fields in full and compact MCP profiles. The session Evidence reported a selected-project fingerprint and unknown original analysis completeness/settings; it did not independently identify the dSYM or certify which prior input produced the fields. The original project's ten regular files and the executable remained byte-identical through both REA sessions.

Each session's exported dossier was identical to its live record. A fresh target-free MCP server imported each bundle and paged the retained pseudocode, preserving both field names in both profiles. This is a usable workaround when the analyst already has a suitable project, not direct dSYM selection or independently verified companion provenance. Ghidra's project-import log reported the same DWARF frame/register errors noted above. The project was created specifically for this control and the target was never executed.

Originally created by @coygeek on GitHub (Oct 9, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1193 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Current limitation REA imports an immutable private executable copy into its ephemeral Ghidra project. A dSYM beside the original Mach-O is not available beside that private copy. On an observed source-built C fixture, REA decompiles `pair_sum` as two generated scalar parameters and cannot find its source `Pair` type, while an independent Ghidra session importing the original executable beside the matching dSYM recovers a single by-value `Pair` parameter, field names, source parameter names, and the type at `/DWARF/fixture.c/Pair`. This blocks an analyst who already owns the debug companion from obtaining Ghidra's more informative function semantics through REA. This is a new capability request. The current tool contracts promise recovered database observations and do not promise dSYM companion admission, so the observed difference is not asserted as a violation of that contract. ### Use cases Let a Ghidra analysis session include an explicitly selected matching dSYM companion so native decompilation, recovered API boundaries, and data-type inspection can use the debug information available to the analyst. Direct Ghidra or GhidraMCP analysis beside the original executable recovers these facts, and an embedded-DWARF object works for source-controlled fixtures, but neither supplies a shipped linked executable plus its dSYM to REA's composed dossiers and comparisons. ### Desired outcome and success criteria Allow the caller to select a local dSYM companion when opening a native Mach-O for Ghidra analysis through CLI or MCP. Establish that the companion matches the selected executable before analysis. Preserve the existing immutable artifact behavior and ephemeral annotation lifecycle. Report whether debug information was admitted and used, retain its identity in the analysis evidence, and keep unsupported or mismatched companions explicit. Function dossiers, pseudocode, native API inspection, and exact data-type inspection should expose the facts Ghidra recovers from the selected companion. Debug declarations and decompiler inferences must retain distinct attribution; using debug information does not make pseudocode original source. Success criteria: - Compile a thin arm64 Mach-O and matching dSYM from the inline source below. With the selected matching companion, CLI and MCP recover `pair_sum`'s single by-value `Pair` parameter, its field names and 16-byte size, and expose `/DWARF/fixture.c/Pair` through exact type inspection. - With no companion selected, retain the current executable-only analysis behavior and report its narrower inputs accurately. - An explicitly selected dSYM for a different build must produce a mismatch or unsupported result with the relevant identities; it must not silently supply the other build's declarations or fall back while claiming debug coverage. - A missing, unreadable, or malformed selected companion must produce an actionable result that distinguishes those conditions. - Evidence and analysis-profile identity must distinguish analysis with the selected debug companion from executable-only analysis, so a snapshot cannot silently replay the wrong input set. - Original executable and dSYM bytes remain unchanged. Reopening the same live target with the same selected input identities preserves authorized session annotations under the existing lifecycle contract. ### Authority boundaries and non-goals Affected area: Native Ghidra target selection and its function-level CLI/MCP workflows. The existing private immutable import and evidence identity are binding constraints because the additional input must remain attributable without modifying the user's Ghidra installation or source artifacts. Automatic symbol-server downloads, debugger execution, global filesystem searches for debug data, source-code reconstruction, and broad PDB or ELF debug-sidecar support are outside this request. ### Alternatives considered Direct Ghidra or GhidraMCP analysis beside the original executable already recovers the companion facts, but leaves the analyst unable to use those facts in REA's composed dossiers and comparisons. Compiling and selecting a DWARF-bearing object file works for source-controlled fixtures, but is not a replacement for analyzing a shipped linked executable and its existing dSYM. Explicit companion selection is preferable to automatically importing nearby files because it preserves the caller's chosen inputs and permits clear identity and mismatch checks. ### Possible direction _No response_ ### References The observed comparison used `rea-agents@6.1.0` and main commit `b34bdcdb2877be394c59c9cbf70269dc1841ff78`, Ghidra 12.1.4, OpenJDK 21.0.12.1, Node 24.21.0, and macOS 27 arm64. The independent oracle was bethington/ghidra-mcp stable v6.0.0 at commit `8cd2078e10b9ba28b188cb84ce5b9051a904b995`, built against that Ghidra install and run headlessly on loopback with a disposable project. The larger observed fixture contained this function and type. The following smaller source is an acceptance fixture, not a separately executed reduction. ```c struct Pair { long a; long b; }; __attribute__((noinline)) long pair_sum(struct Pair p) { return p.a + 3*p.b; } int main(void) { return (int)pair_sum((struct Pair){1,2}); } ``` ```sh clang -arch arm64 -O2 -g -gdwarf-4 fixture.c -o fixture ``` For the executed control, the oracle returned `long pair_sum(Pair p)` and `return p.b * 3 + p.a;` when the matching dSYM was beside the original file. Importing a byte-identical executable copy under a new basename without a dSYM returned the same generated scalar-parameter pseudocode as REA. That control establishes the companion-input difference rather than a decompiler disagreement. The exact-source DWARF-bearing object also gave REA a recovered `Pair` parameter, demonstrating that embedded debug facts already reach the function workflows. `src/ghidra/GhidraTargetSnapshot.ts` identifies the executable-only immutable snapshot boundary. `docs/native-investigation.md`, "Native instruction, call and type primitives", defines exact database pathname inspection and database authority. Read-only issue searches for dSYM found no existing issue for this selected-companion workflow. ### Ghidra and Kuna-backend input control A separate source-built arm64 Mach-O with `struct Sample { int count; int scale; }` and its adjacent dSYM gave Ghidra 12.1.4 headless enough type information for both the stock decompiler and the released Kuna 1.989 Ghidra backend to render the named fields. Ghidra reported some unrecoverable DWARF frame/register expressions. On a byte-identical copy without the companion, both headless backends rendered integer-pointer expressions instead. Real REA Ghidra and independent Kuna function dossiers on the original executable also rendered integer-pointer expressions. The executable SHA-256 stayed `c244f7d7d9fa9b7f4cfcb1f69d69cc1f83ad98fe38b0293b223b3ca7d9076363`; neither copy was executed. This independently supports the selected-companion concern for one additional fixture and shows why changing decompiler backend alone does not substitute for binding debug inputs. It does not demonstrate a general dSYM import guarantee, prove all debug facts were applied, or test an REA hybrid backend. See #8177 for the complete experimental scope. ### Existing-project workaround and offline handoff For the typed fixture above, I imported the executable with its adjacent dSYM into a disposable Ghidra 12.1.4 project and saved it. Real REA/Ghidra, configured with that explicitly selected project and program, opened its private copy and `analyze_function(0x100000410)` returned `Sample *` and both named fields in full and compact MCP profiles. The session Evidence reported a selected-project fingerprint and unknown original analysis completeness/settings; it did not independently identify the dSYM or certify which prior input produced the fields. The original project's ten regular files and the executable remained byte-identical through both REA sessions. Each session's exported dossier was identical to its live record. A fresh target-free MCP server imported each bundle and paged the retained pseudocode, preserving both field names in both profiles. This is a usable workaround when the analyst already has a suitable project, not direct dSYM selection or independently verified companion provenance. Ghidra's project-import log reported the same DWARF frame/register errors noted above. The project was created specifically for this control and the target was never executed.

bsakaguchi added the

area: native

enhancement

P2: medium

labels

2026-10-15 18:24:58 +00:00

bsakaguchi commented

2026-10-15 18:24:59 +00:00

Author

Owner

@jimpablo commented on GitHub (Oct 12, 2026):

I'd like to work on this. Environment is ready: macOS 27 arm64, Ghidra 12.1.4 (Homebrew formula build, native mac_arm_64 decompiler verified by rea doctor), JDK accepted.

Reproduced the baseline with a two-step fixture (clang -O0 -g -c pair.c → link → dsymutil pair -o pair.dSYM, so the DWARF companion is real and sits beside the original executable):

Through REA (rea function <pair> _pair_sum --provider ghidra --json): native_api.signature_source is default, return-type confidence low — the private ephemeral import has no debug companion, exactly as reported.
The independent control (a Ghidra session importing the original executable beside the same dSYM) recovers the by-value Pair parameter per the report; I will re-verify it on this host as part of the change.

Planned direction, following the tool-design guide and the caller-selected-input rule:

Admit an explicitly selected dSYM companion at target admission: validate that its UUID matches the executable's LC_UUID (reject mismatches with an honest, typed reason rather than silently loading a non-matching companion), record both digests as Evidence, and stage the companion beside the private import so Ghidra's DWARF applies to decompilation, recovered types, and parameter names.
Contract surface: a new optional caller input on the binary-opening routes (CLI flag + MCP field), designed so that omitting it keeps today's behavior byte-identical.
Verification: real Ghidra lane on this host with the fixture above — signature source, parameter identity, and type recovery before/after; mismatch rejection; cancellation and cleanup of the staged companion.

One design question before I start: should companion admission also apply to the already-open-session route (open_binary) only, or also to one-shot commands (rea function etc.)? My default is both, since both share the same import path — I'll follow whatever the maintainers prefer.

@jimpablo commented on GitHub (Oct 12, 2026): I'd like to work on this. Environment is ready: macOS 27 arm64, Ghidra 12.1.4 (Homebrew formula build, native `mac_arm_64` decompiler verified by `rea doctor`), JDK accepted. Reproduced the baseline with a two-step fixture (`clang -O0 -g -c pair.c` → link → `dsymutil pair -o pair.dSYM`, so the DWARF companion is real and sits beside the original executable): - Through REA (`rea function <pair> _pair_sum --provider ghidra --json`): `native_api.signature_source` is `default`, return-type confidence `low` — the private ephemeral import has no debug companion, exactly as reported. - The independent control (a Ghidra session importing the original executable beside the same dSYM) recovers the by-value `Pair` parameter per the report; I will re-verify it on this host as part of the change. Planned direction, following the tool-design guide and the caller-selected-input rule: - Admit an explicitly selected dSYM companion at target admission: validate that its UUID matches the executable's `LC_UUID` (reject mismatches with an honest, typed reason rather than silently loading a non-matching companion), record both digests as Evidence, and stage the companion beside the private import so Ghidra's DWARF applies to decompilation, recovered types, and parameter names. - Contract surface: a new optional caller input on the binary-opening routes (CLI flag + MCP field), designed so that omitting it keeps today's behavior byte-identical. - Verification: real Ghidra lane on this host with the fixture above — signature source, parameter identity, and type recovery before/after; mismatch rejection; cancellation and cleanup of the staged companion. One design question before I start: should companion admission also apply to the already-open-session route (open_binary) only, or also to one-shot commands (`rea function` etc.)? My default is both, since both share the same import path — I'll follow whatever the maintainers prefer.

bsakaguchi commented

2026-10-15 18:24:59 +00:00

Author

Owner

@matbrgz commented on GitHub (Oct 12, 2026):

Symbol Binding & dSYM Invariants Review

Staging and binding companion Apple dSYM bundles during headless Ghidra Mach-O analysis bridges a major symbol recovery gap on macOS:

UUID Verification: The dSYM bundle's LC_UUID load command MUST strictly match the target Mach-O's LC_UUID. If the UUIDs differ, REA must fail closed with dsym_uuid_mismatch rather than attributing mismatched DWARF symbol names.
Staging: When copying the immutable private executable copy into the ephemeral Ghidra project root, copy the validated .dSYM bundle alongside it so Ghidra's DWARF analyzer can automatically ingest functions, parameter types, and line numbers.

@matbrgz commented on GitHub (Oct 12, 2026): ### Symbol Binding & dSYM Invariants Review Staging and binding companion Apple `dSYM` bundles during headless Ghidra Mach-O analysis bridges a major symbol recovery gap on macOS: - **UUID Verification:** The dSYM bundle's `LC_UUID` load command MUST strictly match the target Mach-O's `LC_UUID`. If the UUIDs differ, REA must fail closed with `dsym_uuid_mismatch` rather than attributing mismatched DWARF symbol names. - **Staging:** When copying the immutable private executable copy into the ephemeral Ghidra project root, copy the validated `.dSYM` bundle alongside it so Ghidra's DWARF analyzer can automatically ingest functions, parameter types, and line numbers.

bsakaguchi referenced this issue

2026-10-16 21:13:00 +00:00

[PR #427] [MERGED] chore(main): release rea-agents 2.4.0 #1975

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-10-31 00:29:09 +00:00

bsakaguchi added

P2: medium

and removed