[PR #4439] feat(ghidra)!: stream progress and cache completed analysis #6242

New issue

Open

opened 2026-10-31 01:12:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:12:58 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4439
Author: @nanashili
Created: 10/16/2026
Status: 🔄 Open

Base: main ← Head: feat/ghidra-progress-and-scalable-analysis

📝 Commits (3)

5c68b25 feat: stream and cache Ghidra analysis
f2c5428 fix(ghidra): reconcile streaming analysis and retained inventory contracts
5c697c0 Merge main and preserve inventory view integration

📊 Changes

45 files changed (+3131 additions, -671 deletions)

View changed files

➕ bridge/ghidra/ReaGhidraAnalysisCache.java (+219 -0)
📝 bridge/ghidra/ReaGhidraBridge.java (+553 -39)
📝 docs/ghidra-provider.md (+24 -3)
📝 docs/mcp-contracts.md (+15 -5)
📝 package.json (+1 -0)
📝 scripts/lib/mcp-verifier-results.mjs (+77 -13)
📝 scripts/lib/real-hopper-inventory.mjs (+8 -2)
📝 scripts/lib/real-hopper-owner-exit.mjs (+3 -2)
📝 scripts/verify-real-hopper.mjs (+3 -1)
📝 src/application/binary/BinarySession.ts (+17 -2)
📝 src/application/binary/BinarySessionLiveness.test.ts (+40 -9)
📝 src/application/binary/BinarySessionOpen.ts (+2 -0)
📝 src/application/binary/BinarySessionPort.ts (+1 -0)
📝 src/contracts/analysisViewToolContracts.ts (+2 -2)
📝 src/contracts/officialToolContracts.ts (+2 -2)
📝 src/contracts/toolOutputSchemaGroups.ts (+14 -3)
📝 src/contracts/toolOutputSchemaPrimitives.ts (+33 -1)
➕ src/domain/analysisView/addressedInventoryView.ts (+99 -0)
📝 src/domain/analysisView/analysisView.ts (+32 -7)
📝 src/domain/analysisView/binaryLayoutView.ts (+6 -2)

...and 25 more files

📄 Description

Summary

Ghidra now reports bounded procedure previews while analysis runs, can reuse an opt-in completed-analysis cache, and keeps oversized procedure/string inventories accessible through retained Evidence pages. The branch incorporates current main lifecycle and cleanup fixes.

Problem and expected behavior

Long analysis previously supplied only waiting notifications, and complete inventories could exceed the MCP response budget. Callers now receive progress previews and a successful retained-inventory continuation when the complete Evidence cannot fit inline. Previews never establish complete coverage; paged results preserve every retained row and its parent Evidence identity.

Change and scope

Own analysis, finalization, and shutdown in one Ghidra worker lifecycle. Split launcher preparation, command inspection, ownership recording, and rollback into named operations while preserving primary and cleanup failures.
Cache completed baselines before extensions and interactive edits. Validate source/profile identity, archive size, and digest; remove partial preparation independently and retain failed cleanup ownership. Windows caching remains disabled.
Define inline Evidence and receipts in the canonical inventory contracts, including an MCP-compatible object root. Remove registration-only schema widening and duplicate size estimation. Inventory readers follow validated continuations without rerunning provider analysis.
Fix the cache test's false-positive indexOf comparison by asserting the complete analysis plan, including required: false. Correct page guidance: strings continuations supply limit: 16; page requests still require an explicit limit.

Contribution scope: existing Ghidra analysis and session boundaries; no new target execution, injection, or protection-recovery authority. No separate issue is linked; this addresses the PR's three review findings and base conflicts.

Contract and boundary impact

Semantic owners: canonical output contracts, retained Evidence views, and the Ghidra launcher/bridge lifecycle.
MCP: oversized list_procedures/list_strings successes may be receipts. Follow continuation.arguments, then coverage.next_offset. CLI inventory output remains complete.
Evidence: original target/provider provenance and retained parent identity remain available; paging does not rerun analysis.
Files/processes: existing owned Ghidra process launch and ephemeral session database, plus opt-in persistent cache archives selected by REA_GHIDRA_CACHE_DIR. Interactive changes never write back to those archives or executable bytes.
Packaging: include the cache Java source through the bridge source manifest. No dependencies or generated build outputs are committed.

Evidence and regression coverage

The canonical receipt regressions failed before the contract fix. Public SDK MCP tests now validate advertised schemas and traverse all 600 procedure/string rows with exact identity/coverage and one provider execution. Additional cases cover absent/failed retention, cache cancellation, partial cleanup failure, launch rollback, and analysis completion before startup succeeds. All three automated review findings are addressed; no directly related cleanup is deferred.

Validation performed

npm run check — passed typecheck, lint, formatting, Knip, metadata, and architecture guards.
npm run docs:check — passed, including five documentation consistency tests.
npm run test:focused -- … — 229 targeted tests passed; three Windows-only tests skipped on macOS.
npm run test:local -- … — 105 targeted analysis-view, progress, session, and liveness tests passed.
After the final merge of main, 118 analysis-view, MCP schema, and browser-scenario regressions passed; static and documentation checks were rerun. Provider, cache, and package implementation files are unchanged from the real-provider and package validations below.
npm run verify:package — passed package installation/update, CLI, MCP, discovery, Evidence, setup/uninstall, and cleanup verification.
npm run verify:ghidra:bridge-sources — all ten Java sources compiled against Ghidra 12.1.4/JDK 21.0.12.1.
npm run verify:ghidra — passed on macOS ARM64 with Ghidra 12.1.4/JDK 21.0.12.1 and locally built upstream native helpers. Debug, stripped, type-layout, and jump-table fixtures passed; 373 successful/52 rejected boundary calls verified CLI/MCP parity, annotations/rollback, source admission, oversized-result export, cancellation, and complete cleanup.
Real Ghidra cache round trip on macOS ARM64 — passed. Cold/warm inventories matched (12 procedures, 49 strings); a later session rename was absent after reopening the cached baseline. Startup took 13.100s cold and 8.793s warm in one sample. The complete two-session run took 25.21s wall time and 35.12s user/2.84s system CPU; time -l reported 647,135,232 bytes maximum RSS. Peak aggregate process-family memory and agent tokens were not measured. This is a functional sample, not a general speedup claim or a before/after baseline comparison.

Compatibility, safety, and release

Breaking: native decompilation now has a 30-second per-function deadline instead of waiting indefinitely. Longer calls return a typed analysis timeout. Callers must handle that result or use instruction-only queries when suitable; no per-call deadline override is exposed. The deadline bounds how long one decompiler request can occupy the session.

Cache entries persist until the selected directory is cleared; there is no automatic eviction. Real Windows, Linux, and Hopper execution and cross-format Ghidra lanes were not run on this macOS host. No claim of remote CI qualification or human readiness is made.

Author readiness

Human owner: @nanashili

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4439 **Author:** [@nanashili](https://github.com/nanashili) **Created:** 10/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/ghidra-progress-and-scalable-analysis` --- ### 📝 Commits (3) - [`5c68b25`](https://github.com/morluto/rea/commit/5c68b25ec1f999d80487966c2f76f80c8c2e70eb) feat: stream and cache Ghidra analysis - [`f2c5428`](https://github.com/morluto/rea/commit/f2c5428f452a47273b7b57b926ce4d178c748778) fix(ghidra): reconcile streaming analysis and retained inventory contracts - [`5c697c0`](https://github.com/morluto/rea/commit/5c697c054586af80a162067959e4140c5bf8c370) Merge main and preserve inventory view integration ### 📊 Changes **45 files changed** (+3131 additions, -671 deletions) <details> <summary>View changed files</summary> ➕ `bridge/ghidra/ReaGhidraAnalysisCache.java` (+219 -0) 📝 `bridge/ghidra/ReaGhidraBridge.java` (+553 -39) 📝 `docs/ghidra-provider.md` (+24 -3) 📝 `docs/mcp-contracts.md` (+15 -5) 📝 `package.json` (+1 -0) 📝 `scripts/lib/mcp-verifier-results.mjs` (+77 -13) 📝 `scripts/lib/real-hopper-inventory.mjs` (+8 -2) 📝 `scripts/lib/real-hopper-owner-exit.mjs` (+3 -2) 📝 `scripts/verify-real-hopper.mjs` (+3 -1) 📝 `src/application/binary/BinarySession.ts` (+17 -2) 📝 `src/application/binary/BinarySessionLiveness.test.ts` (+40 -9) 📝 `src/application/binary/BinarySessionOpen.ts` (+2 -0) 📝 `src/application/binary/BinarySessionPort.ts` (+1 -0) 📝 `src/contracts/analysisViewToolContracts.ts` (+2 -2) 📝 `src/contracts/officialToolContracts.ts` (+2 -2) 📝 `src/contracts/toolOutputSchemaGroups.ts` (+14 -3) 📝 `src/contracts/toolOutputSchemaPrimitives.ts` (+33 -1) ➕ `src/domain/analysisView/addressedInventoryView.ts` (+99 -0) 📝 `src/domain/analysisView/analysisView.ts` (+32 -7) 📝 `src/domain/analysisView/binaryLayoutView.ts` (+6 -2) _...and 25 more files_ </details> ### 📄 Description ## Summary Ghidra now reports bounded procedure previews while analysis runs, can reuse an opt-in completed-analysis cache, and keeps oversized procedure/string inventories accessible through retained Evidence pages. The branch incorporates current `main` lifecycle and cleanup fixes. ## Problem and expected behavior Long analysis previously supplied only waiting notifications, and complete inventories could exceed the MCP response budget. Callers now receive progress previews and a successful retained-inventory continuation when the complete Evidence cannot fit inline. Previews never establish complete coverage; paged results preserve every retained row and its parent Evidence identity. ## Change and scope - Own analysis, finalization, and shutdown in one Ghidra worker lifecycle. Split launcher preparation, command inspection, ownership recording, and rollback into named operations while preserving primary and cleanup failures. - Cache completed baselines before extensions and interactive edits. Validate source/profile identity, archive size, and digest; remove partial preparation independently and retain failed cleanup ownership. Windows caching remains disabled. - Define inline Evidence and receipts in the canonical inventory contracts, including an MCP-compatible object root. Remove registration-only schema widening and duplicate size estimation. Inventory readers follow validated continuations without rerunning provider analysis. - Fix the cache test's false-positive `indexOf` comparison by asserting the complete analysis plan, including `required: false`. Correct page guidance: strings continuations supply `limit: 16`; page requests still require an explicit limit. Contribution scope: existing Ghidra analysis and session boundaries; no new target execution, injection, or protection-recovery authority. No separate issue is linked; this addresses the PR's three review findings and base conflicts. ## Contract and boundary impact - Semantic owners: canonical output contracts, retained Evidence views, and the Ghidra launcher/bridge lifecycle. - MCP: oversized `list_procedures`/`list_strings` successes may be receipts. Follow `continuation.arguments`, then `coverage.next_offset`. CLI inventory output remains complete. - Evidence: original target/provider provenance and retained parent identity remain available; paging does not rerun analysis. - Files/processes: existing owned Ghidra process launch and ephemeral session database, plus opt-in persistent cache archives selected by `REA_GHIDRA_CACHE_DIR`. Interactive changes never write back to those archives or executable bytes. - Packaging: include the cache Java source through the bridge source manifest. No dependencies or generated build outputs are committed. ## Evidence and regression coverage The canonical receipt regressions failed before the contract fix. Public SDK MCP tests now validate advertised schemas and traverse all 600 procedure/string rows with exact identity/coverage and one provider execution. Additional cases cover absent/failed retention, cache cancellation, partial cleanup failure, launch rollback, and analysis completion before startup succeeds. All three automated review findings are addressed; no directly related cleanup is deferred. ## Validation performed - `npm run check` — passed typecheck, lint, formatting, Knip, metadata, and architecture guards. - `npm run docs:check` — passed, including five documentation consistency tests. - `npm run test:focused -- …` — 229 targeted tests passed; three Windows-only tests skipped on macOS. - `npm run test:local -- …` — 105 targeted analysis-view, progress, session, and liveness tests passed. - After the final merge of `main`, 118 analysis-view, MCP schema, and browser-scenario regressions passed; static and documentation checks were rerun. Provider, cache, and package implementation files are unchanged from the real-provider and package validations below. - `npm run verify:package` — passed package installation/update, CLI, MCP, discovery, Evidence, setup/uninstall, and cleanup verification. - `npm run verify:ghidra:bridge-sources` — all ten Java sources compiled against Ghidra 12.1.4/JDK 21.0.12.1. - `npm run verify:ghidra` — passed on macOS ARM64 with Ghidra 12.1.4/JDK 21.0.12.1 and locally built upstream native helpers. Debug, stripped, type-layout, and jump-table fixtures passed; 373 successful/52 rejected boundary calls verified CLI/MCP parity, annotations/rollback, source admission, oversized-result export, cancellation, and complete cleanup. - Real Ghidra cache round trip on macOS ARM64 — passed. Cold/warm inventories matched (12 procedures, 49 strings); a later session rename was absent after reopening the cached baseline. Startup took 13.100s cold and 8.793s warm in one sample. The complete two-session run took 25.21s wall time and 35.12s user/2.84s system CPU; `time -l` reported 647,135,232 bytes maximum RSS. Peak aggregate process-family memory and agent tokens were not measured. This is a functional sample, not a general speedup claim or a before/after baseline comparison. ## Compatibility, safety, and release **Breaking:** native decompilation now has a 30-second per-function deadline instead of waiting indefinitely. Longer calls return a typed analysis timeout. Callers must handle that result or use instruction-only queries when suitable; no per-call deadline override is exposed. The deadline bounds how long one decompiler request can occupy the session. Cache entries persist until the selected directory is cleared; there is no automatic eviction. Real Windows, Linux, and Hopper execution and cross-format Ghidra lanes were not run on this macOS host. No claim of remote CI qualification or human readiness is made. ## Author readiness Human owner: @nanashili - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:12:58 +00:00

bsakaguchi referenced this issue

2026-11-01 21:46:24 +00:00

[PR #6263] [CLOSED] docs(browser): explain destination-dependent case isolation in script export #7237