[PR #2904] feat(ghidra)!: preserve paged warnings and require observed comparison coverage #5484

New issue

Open

opened 2026-10-31 01:07:00 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:07:00 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/2904
Author: @Bobtoshi
Created: 10/13/2026
Status: 🔄 Open

Base: main ← Head: codex/decompiler-completeness

📝 Commits (10+)

487aa7e feat(ghidra): expose decompiler warning completeness observations
d46c835 fix(ghidra): verify and package decompiler warning extraction
df73145 fix(ghidra): preserve warning provenance in native workflows
d2ba06b fix(ghidra): retain batch limitations and neutral warning semantics
c5d6dfd Merge commit '2895a18ca9' into codex/ci-review-pr2904-20261014
eac4d7f fix(ghidra): page warnings and retain annotation limitations
7f24331 fix(comparison)!: retain warning coordinates and require observed coverage
6cfe3e8 fix(domain): align function comparison consumers with revision 2
1d921ba chore: merge current upstream and retain comparison consumer fixes
a73ec12 Merge remote-tracking branch 'origin/main' into sweep/2904

📊 Changes

48 files changed (+1768 additions, -67 deletions)

View changed files

📝 .github/workflows/ci.yml (+3 -0)
📝 bridge/ghidra/ReaGhidraBridge.java (+68 -10)
➕ bridge/ghidra/ReaGhidraDecompilerWarnings.java (+37 -0)
📝 docs/ghidra-provider.md (+45 -0)
📝 docs/testing.md (+4 -0)
📝 knip.json (+1 -0)
📝 package.json (+2 -0)
➕ scripts/test-ghidra-decompiler-warnings.mjs (+46 -0)
📝 scripts/verify-packaged-ghidra-windows.mjs (+1 -0)
📝 scripts/verify-packed-bridge.mjs (+3 -0)
➕ scripts/verify-real-ghidra-completeness.mjs (+273 -0)
📝 scripts/verify-windows-package.mjs (+1 -0)
📝 src/application/EnhancedTools.ts (+8 -3)
📝 src/application/InvestigationProviders.ts (+13 -14)
📝 src/application/WorkflowEvidence.ts (+56 -0)
📝 src/application/analysisView/AnalysisViewService.test.ts (+86 -0)
📝 src/application/native/NativeApiInspection.test.ts (+42 -0)
📝 src/application/native/NativeApiInspection.ts (+16 -1)
📝 src/application/native/NativeValueTrace.test.ts (+61 -0)
📝 src/application/native/NativeValueTrace.ts (+9 -0)

...and 28 more files

📄 Description

A Ghidra function can report removed blocks while an agent-facing result loses the warning. This adds structured decompiler observations to dossiers, keeps warned or unobserved-warning pseudocode comparisons unresolved, and retains warnings in pseudocode, API/value/batch workflows and refreshed annotation Evidence.

Related to #2735. Authored Listing comments remain separate from automatic warnings. Shared completeness facts retain producer provenance; Ghidra validates its own wire identity. Dropped-block ends remain unknown, and silence does not establish complete recovery. The Ghidra interpretation is v6; composed workflow revisions prevent stale replay.

Compact views expose a scalar completeness summary, separate warning/dropped-block record pages, diagnostic text pages, and provider-limitation pages. Warned views carry a short notice pointing to their complete parent Evidence. Non-warning native views retain their existing inline limitations. Analysis-view producer revision is 2. Metadata projection avoids reparsing a validated API/value graph merely to extract its limitations.

Validation:

360 focused domain, adapter, workflow, reconstruction and MCP tests passed; current-head static, module-boundary, architecture and docs checks passed. Discovery includes every selected test once.
The warning-heavy regression fails on the prior producer with a 6,017,540-character view; fixed selected replies stay below 12 KiB while the full parent and warning indices remain available.
Real Ghidra 12.1.4/macOS arm64 through production CLI/MCP verified actual removed-block warnings, authored-comment exclusion, API/value/batch/annotation Evidence retention and all paged facets. Original input hash stayed unchanged; the target was never executed.
Six pure Java producer cases, including raw offsets after normalization and failed resolution, and fresh packaged-install/lifecycle checks passed. Linux/Windows native coverage remains unrun.
A 3,003,659-byte graph (2,000 nodes/12,000 edges) produced identical Evidence IDs before/after metadata projection. Four-call wall samples were 43.9–59.6 ms before and 40.8–65.9 ms after; whole-process CPU totals were 434/437 ms and peak process RSS 164/220 MiB. Runs overlapped other verification; process-family sampling and agent tokens were not measured. These observations do not establish a speed or memory improvement.

Dropped blocks preserve the raw reported_offset before canonical resolution, including unresolved address spaces. Legacy records default missing offsets to explicit null; the Ghidra wire producer requires its observation. Comparison provider revision is 2, and reconstruction verification accepts that revision while refusing old comparison Evidence. Legacy source Evidence without warning metadata yields unknown pseudocode across producers without parsing authored text or inventing warning coverage. The regression fails on the previous consumer.

Both CLI/MCP verifier children use declared minimal environments and process-run tokens. Success requires verified empty descendants; failures retain diagnostics. The native check also passed with a poisoned ambient seed/analyzer configuration.

BREAKING CHANGE: pseudocode comparisons now require explicit warning observations. Re-analyze with a producer that records them, or compare independently observed assembly; recreate function-comparison Evidence at revision 2 before reconstruction verification. Missing metadata from existing IDA/Hopper or retained Ghidra Evidence is not evidence of no warnings.

The maintainer's latest rebase is preserved. No real IDA/Hopper or broad firmware correctness claim, and no assertion that pending upstream CI or maintainer review has passed.

Current-head follow-up:

Incorporated jaelliot's commit 2f4dd45c17c78582b1b8d906052ab02599244000 with authorship preserved. The function-comparison identity now has one domain owner, used by the producer, changed-behavior/static-runtime/reconstruction consumers and examples. Legacy/foreign identities still fail; artifact comparisons retain revision 1.
Independently reproduced both v2-consumer failures using the old consumer implementations; the same selected acceptance cases pass with the patch. The current 360-test selection includes consumers and production MCP integration. A separate 36-test consumer run also passed.
Merged tested upstream efbc486b9b, preserving its cancellation-aware batches, call-relationship coverage and source-observation limitation collection. Warnings remain per batch item and in final Evidence. Composed revision keys account for both upstream and this PR's interpretations.
Current static/module/architecture, docs, all 1,290-file test discovery, six Java cases, package-install/lifecycle checks and real Ghidra 12.1.4/macOS arm64 CLI/MCP passed with unchanged inputs, no target execution and verified empty descendants. Linux/Windows native runs remain unrun.

Latest verified head: 1d921baa98. The numerical graph measurements above describe the earlier metadata-projection experiment, not a speedup claim for this merged head. GitHub PR Actions currently require maintainer approval for fork-origin code; no upstream CI pass is claimed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/2904 **Author:** [@Bobtoshi](https://github.com/Bobtoshi) **Created:** 10/13/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `codex/decompiler-completeness` --- ### 📝 Commits (10+) - [`487aa7e`](https://github.com/morluto/rea/commit/487aa7e0b57092e01aeb12a9829e0d0f2536c0a2) feat(ghidra): expose decompiler warning completeness observations - [`d46c835`](https://github.com/morluto/rea/commit/d46c8350ae14bb2b7a99b616f35242244155fb82) fix(ghidra): verify and package decompiler warning extraction - [`df73145`](https://github.com/morluto/rea/commit/df73145a960471365eb2e94a391b421a6ef5d299) fix(ghidra): preserve warning provenance in native workflows - [`d2ba06b`](https://github.com/morluto/rea/commit/d2ba06be12f474976241b275ef30c52d8b14d06d) fix(ghidra): retain batch limitations and neutral warning semantics - [`c5d6dfd`](https://github.com/morluto/rea/commit/c5d6dfd61617b45f2934a6cac7d0715a4fc2965b) Merge commit '2895a18ca92f415ae62f87167ba9c7a672e3b7dc' into codex/ci-review-pr2904-20261014 - [`eac4d7f`](https://github.com/morluto/rea/commit/eac4d7f1d18855365ba0e0c97828edfc27e16574) fix(ghidra): page warnings and retain annotation limitations - [`7f24331`](https://github.com/morluto/rea/commit/7f24331720b75b296cf476e5e593840baac684e7) fix(comparison)!: retain warning coordinates and require observed coverage - [`6cfe3e8`](https://github.com/morluto/rea/commit/6cfe3e8eafd299b4470f227252941838aff26b1e) fix(domain): align function comparison consumers with revision 2 - [`1d921ba`](https://github.com/morluto/rea/commit/1d921baa98a77bccea2f6d56d61900e701a52136) chore: merge current upstream and retain comparison consumer fixes - [`a73ec12`](https://github.com/morluto/rea/commit/a73ec1251655ec8de332079557a4b082493763f0) Merge remote-tracking branch 'origin/main' into sweep/2904 ### 📊 Changes **48 files changed** (+1768 additions, -67 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/ci.yml` (+3 -0) 📝 `bridge/ghidra/ReaGhidraBridge.java` (+68 -10) ➕ `bridge/ghidra/ReaGhidraDecompilerWarnings.java` (+37 -0) 📝 `docs/ghidra-provider.md` (+45 -0) 📝 `docs/testing.md` (+4 -0) 📝 `knip.json` (+1 -0) 📝 `package.json` (+2 -0) ➕ `scripts/test-ghidra-decompiler-warnings.mjs` (+46 -0) 📝 `scripts/verify-packaged-ghidra-windows.mjs` (+1 -0) 📝 `scripts/verify-packed-bridge.mjs` (+3 -0) ➕ `scripts/verify-real-ghidra-completeness.mjs` (+273 -0) 📝 `scripts/verify-windows-package.mjs` (+1 -0) 📝 `src/application/EnhancedTools.ts` (+8 -3) 📝 `src/application/InvestigationProviders.ts` (+13 -14) 📝 `src/application/WorkflowEvidence.ts` (+56 -0) 📝 `src/application/analysisView/AnalysisViewService.test.ts` (+86 -0) 📝 `src/application/native/NativeApiInspection.test.ts` (+42 -0) 📝 `src/application/native/NativeApiInspection.ts` (+16 -1) 📝 `src/application/native/NativeValueTrace.test.ts` (+61 -0) 📝 `src/application/native/NativeValueTrace.ts` (+9 -0) _...and 28 more files_ </details> ### 📄 Description A Ghidra function can report removed blocks while an agent-facing result loses the warning. This adds structured decompiler observations to dossiers, keeps warned or unobserved-warning pseudocode comparisons unresolved, and retains warnings in pseudocode, API/value/batch workflows and refreshed annotation Evidence. Related to #2735. Authored Listing comments remain separate from automatic warnings. Shared completeness facts retain producer provenance; Ghidra validates its own wire identity. Dropped-block ends remain unknown, and silence does not establish complete recovery. The Ghidra interpretation is v6; composed workflow revisions prevent stale replay. Compact views expose a scalar completeness summary, separate warning/dropped-block record pages, diagnostic text pages, and provider-limitation pages. Warned views carry a short notice pointing to their complete parent Evidence. Non-warning native views retain their existing inline limitations. Analysis-view producer revision is 2. Metadata projection avoids reparsing a validated API/value graph merely to extract its limitations. Validation: - 360 focused domain, adapter, workflow, reconstruction and MCP tests passed; current-head static, module-boundary, architecture and docs checks passed. Discovery includes every selected test once. - The warning-heavy regression fails on the prior producer with a 6,017,540-character view; fixed selected replies stay below 12 KiB while the full parent and warning indices remain available. - Real Ghidra 12.1.4/macOS arm64 through production CLI/MCP verified actual removed-block warnings, authored-comment exclusion, API/value/batch/annotation Evidence retention and all paged facets. Original input hash stayed unchanged; the target was never executed. - Six pure Java producer cases, including raw offsets after normalization and failed resolution, and fresh packaged-install/lifecycle checks passed. Linux/Windows native coverage remains unrun. - A 3,003,659-byte graph (2,000 nodes/12,000 edges) produced identical Evidence IDs before/after metadata projection. Four-call wall samples were 43.9–59.6 ms before and 40.8–65.9 ms after; whole-process CPU totals were 434/437 ms and peak process RSS 164/220 MiB. Runs overlapped other verification; process-family sampling and agent tokens were not measured. These observations do not establish a speed or memory improvement. Dropped blocks preserve the raw `reported_offset` before canonical resolution, including unresolved address spaces. Legacy records default missing offsets to explicit null; the Ghidra wire producer requires its observation. Comparison provider revision is 2, and reconstruction verification accepts that revision while refusing old comparison Evidence. Legacy source Evidence without warning metadata yields unknown pseudocode across producers without parsing authored text or inventing warning coverage. The regression fails on the previous consumer. Both CLI/MCP verifier children use declared minimal environments and process-run tokens. Success requires verified empty descendants; failures retain diagnostics. The native check also passed with a poisoned ambient seed/analyzer configuration. BREAKING CHANGE: pseudocode comparisons now require explicit warning observations. Re-analyze with a producer that records them, or compare independently observed assembly; recreate function-comparison Evidence at revision 2 before reconstruction verification. Missing metadata from existing IDA/Hopper or retained Ghidra Evidence is not evidence of no warnings. The maintainer's latest rebase is preserved. No real IDA/Hopper or broad firmware correctness claim, and no assertion that pending upstream CI or maintainer review has passed. Current-head follow-up: - Incorporated jaelliot's commit 2f4dd45c17c78582b1b8d906052ab02599244000 with authorship preserved. The function-comparison identity now has one domain owner, used by the producer, changed-behavior/static-runtime/reconstruction consumers and examples. Legacy/foreign identities still fail; artifact comparisons retain revision 1. - Independently reproduced both v2-consumer failures using the old consumer implementations; the same selected acceptance cases pass with the patch. The current 360-test selection includes consumers and production MCP integration. A separate 36-test consumer run also passed. - Merged tested upstream efbc486b9bf8ab46da16fdd6ed88128849e30e81, preserving its cancellation-aware batches, call-relationship coverage and source-observation limitation collection. Warnings remain per batch item and in final Evidence. Composed revision keys account for both upstream and this PR's interpretations. - Current static/module/architecture, docs, all 1,290-file test discovery, six Java cases, package-install/lifecycle checks and real Ghidra 12.1.4/macOS arm64 CLI/MCP passed with unchanged inputs, no target execution and verified empty descendants. Linux/Windows native runs remain unrun. Latest verified head: 1d921baa98a77bccea2f6d56d61900e701a52136. The numerical graph measurements above describe the earlier metadata-projection experiment, not a speedup claim for this merged head. GitHub PR Actions currently require maintainer approval for fork-origin code; no upstream CI pass is claimed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:07:00 +00:00

bsakaguchi referenced this issue

2026-11-03 13:39:01 +00:00

[PR #7042] [MERGED] feat(cli): report opt-in command timings #7598