[GH-ISSUE #4746] [Feature] Avoid full Ghidra inventory transport for binary overview counts #2474

New issue

Closed

opened 2026-10-17 21:05:19 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-17 21:05:19 +00:00

Owner

Originally created by @N0zoM1z0 on GitHub (Oct 17, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4746

Before submitting

I searched open and closed issues and pull requests for related work.
I reviewed the contribution scope; this changes existing read-only analysis only.
No credentials, private samples, or sensitive target data are included.

Area

Native analysis or decompilation

Current limitation

binary_overview needs procedure/string counts but obtains complete list_procedures and list_strings inventories. Ghidra builds, serializes, transports, validates and retains those rows before the application counts them. Repeated overview calls also repeat these unscoped inventory reads. The provider already owns sorted cached inventories, so the summary could avoid transporting every row without changing discovery semantics.

Source verified at e09a503f44; affected application/Ghidra paths remain unchanged at latest main f85a932d71. See src/application/EnhancedTools.ts (#binaryOverview) and bridge/ghidra/ReaGhidraBridge.java (listProcedures, listStrings, procedures, strings).

Use cases

An agent asks for the overview of a real Linux library before choosing functions or strings. A production stdio MCP run against /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (SHA256 2e6eef9a727f081f0d453b4e5e6cbd8b9ef8b6f86cbf7681cbad444d3b0b55c8) returned 4,109 procedures and 2,856 defined strings. First/repeated overview took 575/256 ms; bridge logs confirmed two complete procedure reads and two complete string reads. These are individual observations, not a statistical speedup claim.

Desired outcome and success criteria

Keep the existing CLI/MCP overview contract, identity, exact count semantics, segment facts, provenance and coverage/limitations.
Produce needed aggregates at the owning provider boundary, avoiding complete inventory transport solely for counts; preserve complete inventory access through existing tools.
Retain compatibility for providers without the aggregate capability. Do not cache ambiguous document selection or invent absence from incomplete coverage.
Compare equivalent real MCP tasks before/after on a source-built fixture and SQLite, including fresh-session and warm timings, whole-task CPU and process-family peak memory, cleanup, and independent correctness checks.
Report token/context costs honestly: output is already compact; no model-token saving is established by reducing internal JSON.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Existing ephemeral Ghidra analysis only: no target execution, executable modification, GUI control, persistent provider database or new mutation authority. Shared application workflow for CLI/MCP; no codemode/atomic experiment changes.

Verification available so far

I have a prototype or observed results

Verification details (optional)

Linux x86-64, Node 24.18.0, Ghidra 12.1.4, Temurin 21.0.12.1+1; one CPU and 1 GiB JVM heap, bounded process family. Production stdio MCP verified a source-built inventory fixture (31 procedures, 95 defined strings), source-known call relationships and independently mapped ELF string bytes; real SQLite verified its marker against mapped file bytes. Target digests remained unchanged. Completed fixture/SQLite tasks took 38.40/140.27 s wall, 37.01/139.11 s CPU, 1,330.9/1,400.1 MiB process-family peak memory including file cache. Import/analysis dominates the SQLite cold task (~125 s). All owned process cleanup verified; no OOM. OS caches were not controlled. These are baseline observations, not optimized measurements.

Alternatives considered

Explicitly document-scoping existing full-list reads could reuse application snapshots but still transports and retains the entire inventory on the first read. Public inventory summary/paging (#4249) addresses a related agent-delivery problem; this issue concerns hidden provider work behind the already compact binary_overview and can preserve its public schema.

Possible direction

Use the existing operation/capability architecture to obtain a validated provider-owned overview/aggregate, with the existing shared composition as a fallback. Reuse the same inventory semantics (including external procedures and Ghidra's defined-string coverage) rather than approximate native counters. Select the narrowest implementation after inspecting the current ownership boundaries.

References

Related: #4249 (public inventory summary/paging), #1065 (bounded-resource usability/performance). Other audit candidates, not established fixes in this issue: default-document snapshot reuse, sorted exact-address lookup, bounded batch dispatch, and selected function-fact production. These need separate scopes and evidence.

Originally created by @N0zoM1z0 on GitHub (Oct 17, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4746 ### Before submitting - [x] I searched open and closed issues and pull requests for related work. - [x] I reviewed the contribution scope; this changes existing read-only analysis only. - [x] No credentials, private samples, or sensitive target data are included. ### Area Native analysis or decompilation ### Current limitation `binary_overview` needs procedure/string counts but obtains complete `list_procedures` and `list_strings` inventories. Ghidra builds, serializes, transports, validates and retains those rows before the application counts them. Repeated overview calls also repeat these unscoped inventory reads. The provider already owns sorted cached inventories, so the summary could avoid transporting every row without changing discovery semantics. Source verified at e09a503f44fba1639b71907a018d71161360f5ae; affected application/Ghidra paths remain unchanged at latest main f85a932d71279415dd95cc30083137d1161f83b8. See `src/application/EnhancedTools.ts` (`#binaryOverview`) and `bridge/ghidra/ReaGhidraBridge.java` (`listProcedures`, `listStrings`, `procedures`, `strings`). ### Use cases An agent asks for the overview of a real Linux library before choosing functions or strings. A production stdio MCP run against `/usr/lib/x86_64-linux-gnu/libsqlite3.so.0` (SHA256 `2e6eef9a727f081f0d453b4e5e6cbd8b9ef8b6f86cbf7681cbad444d3b0b55c8`) returned 4,109 procedures and 2,856 defined strings. First/repeated overview took 575/256 ms; bridge logs confirmed two complete procedure reads and two complete string reads. These are individual observations, not a statistical speedup claim. ### Desired outcome and success criteria - Keep the existing CLI/MCP overview contract, identity, exact count semantics, segment facts, provenance and coverage/limitations. - Produce needed aggregates at the owning provider boundary, avoiding complete inventory transport solely for counts; preserve complete inventory access through existing tools. - Retain compatibility for providers without the aggregate capability. Do not cache ambiguous document selection or invent absence from incomplete coverage. - Compare equivalent real MCP tasks before/after on a source-built fixture and SQLite, including fresh-session and warm timings, whole-task CPU and process-family peak memory, cleanup, and independent correctness checks. - Report token/context costs honestly: output is already compact; no model-token saving is established by reducing internal JSON. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) Existing ephemeral Ghidra analysis only: no target execution, executable modification, GUI control, persistent provider database or new mutation authority. Shared application workflow for CLI/MCP; no codemode/atomic experiment changes. ### Verification available so far I have a prototype or observed results ### Verification details (optional) Linux x86-64, Node 24.18.0, Ghidra 12.1.4, Temurin 21.0.12.1+1; one CPU and 1 GiB JVM heap, bounded process family. Production stdio MCP verified a source-built inventory fixture (31 procedures, 95 defined strings), source-known call relationships and independently mapped ELF string bytes; real SQLite verified its marker against mapped file bytes. Target digests remained unchanged. Completed fixture/SQLite tasks took 38.40/140.27 s wall, 37.01/139.11 s CPU, 1,330.9/1,400.1 MiB process-family peak memory including file cache. Import/analysis dominates the SQLite cold task (~125 s). All owned process cleanup verified; no OOM. OS caches were not controlled. These are baseline observations, not optimized measurements. ### Alternatives considered Explicitly document-scoping existing full-list reads could reuse application snapshots but still transports and retains the entire inventory on the first read. Public inventory summary/paging (#4249) addresses a related agent-delivery problem; this issue concerns hidden provider work behind the already compact `binary_overview` and can preserve its public schema. ### Possible direction Use the existing operation/capability architecture to obtain a validated provider-owned overview/aggregate, with the existing shared composition as a fallback. Reuse the same inventory semantics (including external procedures and Ghidra's defined-string coverage) rather than approximate native counters. Select the narrowest implementation after inspecting the current ownership boundaries. ### References Related: #4249 (public inventory summary/paging), #1065 (bounded-resource usability/performance). Other audit candidates, not established fixes in this issue: default-document snapshot reuse, sorted exact-address lookup, bounded batch dispatch, and selected function-fact production. These need separate scopes and evidence.

bsakaguchi

2026-10-17 21:05:19 +00:00

closed this issue
added the
triage

enhancement
labels

bsakaguchi commented

2026-10-17 21:05:21 +00:00

Author

Owner

@N0zoM1z0 commented on GitHub (Oct 17, 2026):

Implemented and merged in #4754 (ae029fe2). Ghidra computes exact counts at the inventory owner, shares overview projection across CLI/MCP, and reuses a small target/profile-bound observation. Complete inventories and their coverage remain accessible.

Real Linux fixture and SQLite results matched full inventories, independently mapped ELF marker bytes and target digests. Catalog hash unchanged.
Five warm SQLite overview calls: 1,555 → 79 ms and 2,464 → 125 ms in two matched pairs (~20× at this query phase). Cold/whole-task variance and memory overlap do not establish stable improvements; offline transport token estimates were effectively unchanged. Full measurements and limits are in the PR.
Required PR CI passed, including full typecheck, five test shards and packaged Ghidra Java compilation. Greptile: 5/5, no actionable defects. Codex review was unavailable due repository quota.
After rebuilding the actual merged source, 17 focused files passed (377 tests, 2 platform skips) and fresh real Ghidra MCP smoke passed; facts and byte oracle still match, owned cleanup verified, OOM=0. The stale-runtime attempt is retained and explicitly excluded. Both requested co-authors are present in the squash commit.

This closes the overview bottleneck. Other audit candidates listed above require separate workload and correctness decisions; this does not claim that all Ghidra optimization work is complete.

@N0zoM1z0 commented on GitHub (Oct 17, 2026): Implemented and merged in #4754 (`ae029fe2`). Ghidra computes exact counts at the inventory owner, shares overview projection across CLI/MCP, and reuses a small target/profile-bound observation. Complete inventories and their coverage remain accessible. - Real Linux fixture and SQLite results matched full inventories, independently mapped ELF marker bytes and target digests. Catalog hash unchanged. - Five warm SQLite overview calls: **1,555 → 79 ms** and **2,464 → 125 ms** in two matched pairs (~20× at this query phase). Cold/whole-task variance and memory overlap do not establish stable improvements; offline transport token estimates were effectively unchanged. Full measurements and limits are in the PR. - Required PR CI passed, including full typecheck, five test shards and packaged Ghidra Java compilation. Greptile: 5/5, no actionable defects. Codex review was unavailable due repository quota. - After rebuilding the actual merged source, 17 focused files passed (**377 tests, 2 platform skips**) and fresh real Ghidra MCP smoke passed; facts and byte oracle still match, owned cleanup verified, OOM=0. The stale-runtime attempt is retained and explicitly excluded. Both requested co-authors are present in the squash commit. This closes the overview bottleneck. Other audit candidates listed above require separate workload and correctness decisions; this does not claim that all Ghidra optimization work is complete.

bsakaguchi referenced this issue

2026-10-31 00:50:51 +00:00

[GH-ISSUE #7644] [Bug] Hopper guide mentions loader overrides without documenting their setting #4568

bsakaguchi referenced this issue

2026-10-31 01:05:13 +00:00

[PR #2474] [MERGED] fix(config): reject NUL bytes in Hopper loader arguments #5257