morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #2101] [MERGED] fix(ghidra): resolve file offsets in decompile #5027

New issue

Closed

opened 2026-10-31 00:58:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:58:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2101
Author: @Zhong-fan
Created: 10/12/2026
Status: ✅ Merged
Merged: 10/13/2026
Merged by: @morluto

Base: main ← Head: fix/2078-address-context

📝 Commits (10+)

6f11e6e fix(native): expose loaded image bases in overviews
a39562c Merge remote-tracking branch 'origin/main' into fix/2078-address-context
2127611 test(ghidra): retain resources when verifier cleanup fails
c2adf40 docs(ghidra): spell out file-offset address mapping
63fa812 fix(ci): separate and retry NativeAOT dependency restoration
85726c1 fix(test): resolve CI script file URLs on Windows
5f03f4b chore: merge main and preserve Windows CI path handling
4dcc187 fix(native): retain conflict observations and actionable CLI guidance
f17581d fix(fixtures): keep NativeAOT retry logs credential-safe
6789e1e fix(fixtures): redact authenticated URLs at diagnostic creation

📊 Changes

34 files changed (+1588 additions, -86 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+2 -0)
📝 .github/workflows/ci.yml (+2 -0)
📝 .github/workflows/nativeaot-fixtures.yml (+2 -1)
📝 bridge/ghidra/ReaGhidraBridge.java (+112 -15)
📝 docs/ghidra-provider.md (+74 -0)
📝 docs/testing.md (+38 -0)
📝 package.json (+1 -0)
📝 scripts/ci/scopes.mjs (+1 -0)
📝 scripts/generate-nativeaot-fixtures.mjs (+12 -17)
➕ scripts/lib/nativeaot-build.d.mts (+25 -0)
➕ scripts/lib/nativeaot-build.mjs (+60 -0)
➕ scripts/lib/real-ghidra-file-offsets.mjs (+264 -0)
➕ scripts/verify-real-ghidra-address-context.mjs (+313 -0)
📝 scripts/verify-real-ghidra-windows.mjs (+6 -0)
📝 src/application/EnhancedTools.ts (+15 -9)
📝 src/application/InvestigationProviders.ts (+5 -5)
📝 src/cli/coreBinaryCommands.ts (+18 -2)
📝 src/contracts/officialToolContracts.ts (+1 -1)
📝 src/contracts/toolOutputSchemaGroups.ts (+3 -1)
📝 src/domain/hopperValues.test.ts (+0 -14)

...and 14 more files

📄 Description

Summary

Fixes #2078. Support Ghidra file-offset decompilation on top of the image-base contract merged in #2083.

Problem and expected behavior

rea decompile fixture.exe 0x210 --provider ghidra previously failed with "No procedure exists". It now maps that source offset to 0x140001010 and decompiles the controlled return-42 function. File offsets must identify the admitted target, rather than another source in the analysis database.

Change and scope

Add CLI --file-offset=0x... and MCP procedure: "file-offset:0x...". Unresolved numeric selectors try source mappings; existing symbols and analysis addresses retain precedence.
Require complete original FileBytes matching the target SHA-256 in both mapping directions. Reject unverified or ambiguous mappings and cache immutable source identity.
Preserve segment address context, retain #2083's optional overview base, and version profile/workflow semantics to prevent stale replay.
Preserve mapped coordinates when no procedure contains the bytes, or a resolved function fails/cancels decompilation; retain backend failure codes.
Retain bounded NativeAOT restore/retry and credential-safe diagnostics. Generator and subprocess regressions share host applicability; existing native-platform CI runs the diagnostic tests.

Contract and boundary impact

Semantic owner: Ghidra resolves source coordinates; the shared segment parser owns overview observations.
Scope decision: not applicable; ordinary static analysis within existing contribution scope.
CLI/MCP: additive offset selection through the existing decompile operation; no new MCP tool.
Compatibility: Ghidra coordinate conversion; other procedure operations retain address semantics.
Evidence: original selector, mapped address, procedure entry and file-offset locations remain available.
Authority/cleanup: caller-selected immutable target, owned analyzer and temporary project; no target execution or instrumentation injection.
Metadata/package: authored docs/skill updated; generated projections checked. No dependency, release-version or installer change.

Expected effects:

Read existing files or captured data.
Launch or attach to a process (owned Ghidra analyzer).
Inject instrumentation or run scripts/code (REA bridge script only).
Change process memory, registers, files, or configuration (Ghidra analysis, cache and runtime files; target bytes unchanged).
Contact network services (authenticated loopback RPC; CI fixture restore may contact configured NuGet feeds).
Recover protected code or work around protections.
Documentation or usability only; no target interaction.
Not sure yet; investigation or maintainer guidance needed.

Evidence and regression coverage

Reproduced the original offset failure. A real Ghidra multi-source probe also reproduced selecting foreign_only at 0x910 in a 1536-byte target and reporting its auxiliary offset in reverse.
Regressions cover source identity, partial/auxiliary rejection, ambiguity, SDK schemas, Evidence and replay. Real CLI/MCP reproduces mapped bytes without a procedure; injected backend failures verify retained coordinates/codes without claiming natural engine failures.

Representative raw_result.selection from the controlled PE fixture:

{"file_offset":528,"mapped_address":"0x140001010","procedure_address":"0x140001010"}

Remaining proof gaps: real Hopper/IDA execution is unverified. Backend failure/cancellation is injected, not a natural engine-failure reproduction.
Observed, derived, and inferred claims remain distinguishable.
Artifact identity, source provenance, and failed attempts remain preserved.
Unsupported, incomplete, unavailable, or uncertain outcomes remain visible.

Validation performed

npm run check and npm run docs:check — passed.
Focused mapping/Evidence/schema/replay/CLI and CI diagnostics — 118 tests passed.
node scripts/verify-real-ghidra-windows.mjs — real Windows x64 baseline passed; its public file-offset helper also passed after the final bridge fix.
ReaFileOffsetSourceProbe.java — real Ghidra source-boundary regression and injected failure/cancellation checks passed; steps are in docs/testing.md.
node scripts/verify-real-ghidra-address-context.mjs — real Linux ELF CLI/MCP workflow, explicit reverse-mapping round trip, unchanged bytes and empty process lineage passed.
NativeAOT diagnostics/build and CI-scope regressions — 92 passed, 1 unsupported-host case skipped on Windows x64.

Compatibility, safety, and release

Existing symbol/address selection and main's overview contract are preserved; unverified original-file mappings now fail explicitly.
Real-provider coverage: Windows x64, Node 24.18.0, JDK 25, Ghidra 12.1.4. Linux x64 uses GCC 13.3, Node 24.18.0, JDK 21 and Ghidra 12.1.4.
Source bytes remain unchanged; owned runtime cleanup was verified. Verifier-wide Windows lineage remains unavailable; provider-owned native controls passed.
No release or dependency changes. CPU/memory costs were not measured.
Final commit 321a283 fork CI passed all 36 executed jobs, including six full-suite shards and diagnostic regressions on macOS/Linux x64/arm64. Upstream workflows require maintainer approval; maintainer review remains pending.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved; a separate maintainer scope decision is not applicable.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
Final diff checked for secrets, unrelated cleanup, and unsupported claims.
Current-head upstream CI has passed.
Maintainer review is complete.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2101 **Author:** [@Zhong-fan](https://github.com/Zhong-fan) **Created:** 10/12/2026 **Status:** ✅ Merged **Merged:** 10/13/2026 **Merged by:** [@morluto](https://github.com/morluto) **Base:** `main` ← **Head:** `fix/2078-address-context` --- ### 📝 Commits (10+) - [`6f11e6e`](https://github.com/morluto/rea/commit/6f11e6ea55c97d56a10bdb0ca2dbd02318e5a8d5) fix(native): expose loaded image bases in overviews - [`a39562c`](https://github.com/morluto/rea/commit/a39562c05846a4f07f6f95e78a0db692f967c183) Merge remote-tracking branch 'origin/main' into fix/2078-address-context - [`2127611`](https://github.com/morluto/rea/commit/21276119897da7d21ad820d223743b7e243f5417) test(ghidra): retain resources when verifier cleanup fails - [`c2adf40`](https://github.com/morluto/rea/commit/c2adf40401a61451e930c3ccbc6e150a18007acc) docs(ghidra): spell out file-offset address mapping - [`63fa812`](https://github.com/morluto/rea/commit/63fa812135616b4bca2c5779646694d6eab27c45) fix(ci): separate and retry NativeAOT dependency restoration - [`85726c1`](https://github.com/morluto/rea/commit/85726c1e546314aee438aeb6e374d34e07e9ac27) fix(test): resolve CI script file URLs on Windows - [`5f03f4b`](https://github.com/morluto/rea/commit/5f03f4bd627b83fe9f52e2ad8adc470e0e4c469a) chore: merge main and preserve Windows CI path handling - [`4dcc187`](https://github.com/morluto/rea/commit/4dcc1875b0c03804c2d3723fd8ecf2a6657e6e25) fix(native): retain conflict observations and actionable CLI guidance - [`f17581d`](https://github.com/morluto/rea/commit/f17581d4028ee7ffcdf6ffd917fd72af325bdc07) fix(fixtures): keep NativeAOT retry logs credential-safe - [`6789e1e`](https://github.com/morluto/rea/commit/6789e1e55a5b5cd1df8697b485419988c332b2d3) fix(fixtures): redact authenticated URLs at diagnostic creation ### 📊 Changes **34 files changed** (+1588 additions, -86 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+2 -0) 📝 `.github/workflows/ci.yml` (+2 -0) 📝 `.github/workflows/nativeaot-fixtures.yml` (+2 -1) 📝 `bridge/ghidra/ReaGhidraBridge.java` (+112 -15) 📝 `docs/ghidra-provider.md` (+74 -0) 📝 `docs/testing.md` (+38 -0) 📝 `package.json` (+1 -0) 📝 `scripts/ci/scopes.mjs` (+1 -0) 📝 `scripts/generate-nativeaot-fixtures.mjs` (+12 -17) ➕ `scripts/lib/nativeaot-build.d.mts` (+25 -0) ➕ `scripts/lib/nativeaot-build.mjs` (+60 -0) ➕ `scripts/lib/real-ghidra-file-offsets.mjs` (+264 -0) ➕ `scripts/verify-real-ghidra-address-context.mjs` (+313 -0) 📝 `scripts/verify-real-ghidra-windows.mjs` (+6 -0) 📝 `src/application/EnhancedTools.ts` (+15 -9) 📝 `src/application/InvestigationProviders.ts` (+5 -5) 📝 `src/cli/coreBinaryCommands.ts` (+18 -2) 📝 `src/contracts/officialToolContracts.ts` (+1 -1) 📝 `src/contracts/toolOutputSchemaGroups.ts` (+3 -1) 📝 `src/domain/hopperValues.test.ts` (+0 -14) _...and 14 more files_ </details> ### 📄 Description ## Summary Fixes #2078. Support Ghidra file-offset decompilation on top of the image-base contract merged in #2083. ## Problem and expected behavior `rea decompile fixture.exe 0x210 --provider ghidra` previously failed with "No procedure exists". It now maps that source offset to `0x140001010` and decompiles the controlled return-42 function. File offsets must identify the admitted target, rather than another source in the analysis database. ## Change and scope - Add CLI `--file-offset=0x...` and MCP `procedure: "file-offset:0x..."`. Unresolved numeric selectors try source mappings; existing symbols and analysis addresses retain precedence. - Require complete original FileBytes matching the target SHA-256 in both mapping directions. Reject unverified or ambiguous mappings and cache immutable source identity. - Preserve segment address context, retain #2083's optional overview base, and version profile/workflow semantics to prevent stale replay. - Preserve mapped coordinates when no procedure contains the bytes, or a resolved function fails/cancels decompilation; retain backend failure codes. - Retain bounded NativeAOT restore/retry and credential-safe diagnostics. Generator and subprocess regressions share host applicability; existing native-platform CI runs the diagnostic tests. ## Contract and boundary impact - Semantic owner: Ghidra resolves source coordinates; the shared segment parser owns overview observations. - Scope decision: not applicable; ordinary static analysis within existing [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope). - CLI/MCP: additive offset selection through the existing decompile operation; no new MCP tool. - Compatibility: Ghidra coordinate conversion; other procedure operations retain address semantics. - Evidence: original selector, mapped address, procedure entry and file-offset locations remain available. - Authority/cleanup: caller-selected immutable target, owned analyzer and temporary project; no target execution or instrumentation injection. - Metadata/package: authored docs/skill updated; generated projections checked. No dependency, release-version or installer change. Expected effects: - [x] Read existing files or captured data. - [x] Launch or attach to a process (owned Ghidra analyzer). - [x] Inject instrumentation or run scripts/code (REA bridge script only). - [x] Change process memory, registers, files, or configuration (Ghidra analysis, cache and runtime files; target bytes unchanged). - [x] Contact network services (authenticated loopback RPC; CI fixture restore may contact configured NuGet feeds). - [ ] Recover protected code or work around protections. - [ ] Documentation or usability only; no target interaction. - [ ] Not sure yet; investigation or maintainer guidance needed. ## Evidence and regression coverage - Reproduced the original offset failure. A real Ghidra multi-source probe also reproduced selecting `foreign_only` at `0x910` in a 1536-byte target and reporting its auxiliary offset in reverse. - Regressions cover source identity, partial/auxiliary rejection, ambiguity, SDK schemas, Evidence and replay. Real CLI/MCP reproduces mapped bytes without a procedure; injected backend failures verify retained coordinates/codes without claiming natural engine failures. - Representative `raw_result.selection` from the controlled PE fixture: ```json {"file_offset":528,"mapped_address":"0x140001010","procedure_address":"0x140001010"} ``` - Remaining proof gaps: real Hopper/IDA execution is unverified. Backend failure/cancellation is injected, not a natural engine-failure reproduction. - [x] Observed, derived, and inferred claims remain distinguishable. - [x] Artifact identity, source provenance, and failed attempts remain preserved. - [x] Unsupported, incomplete, unavailable, or uncertain outcomes remain visible. ## Validation performed - `npm run check` and `npm run docs:check` — passed. - Focused mapping/Evidence/schema/replay/CLI and CI diagnostics — 118 tests passed. - `node scripts/verify-real-ghidra-windows.mjs` — real Windows x64 baseline passed; its public file-offset helper also passed after the final bridge fix. - `ReaFileOffsetSourceProbe.java` — real Ghidra source-boundary regression and injected failure/cancellation checks passed; steps are in `docs/testing.md`. - `node scripts/verify-real-ghidra-address-context.mjs` — real Linux ELF CLI/MCP workflow, explicit reverse-mapping round trip, unchanged bytes and empty process lineage passed. - NativeAOT diagnostics/build and CI-scope regressions — 92 passed, 1 unsupported-host case skipped on Windows x64. ## Compatibility, safety, and release - Existing symbol/address selection and main's overview contract are preserved; unverified original-file mappings now fail explicitly. - Real-provider coverage: Windows x64, Node 24.18.0, JDK 25, Ghidra 12.1.4. Linux x64 uses GCC 13.3, Node 24.18.0, JDK 21 and Ghidra 12.1.4. - Source bytes remain unchanged; owned runtime cleanup was verified. Verifier-wide Windows lineage remains unavailable; provider-owned native controls passed. - No release or dependency changes. CPU/memory costs were not measured. - Final commit `321a283` [fork CI](https://github.com/Zhong-fan/rea/actions/runs/38358039284) passed all 36 executed jobs, including six full-suite shards and diagnostic regressions on macOS/Linux x64/arm64. Upstream workflows require maintainer approval; maintainer review remains pending. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved; a separate maintainer scope decision is not applicable. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] Final diff checked for secrets, unrelated cleanup, and unsupported claims. - [ ] Current-head upstream CI has passed. - [ ] Maintainer review is complete. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-31 00:58:58 +00:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#5027

No description provided.

Rows
Columns