morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8374] [Bug] Ghidra searches drop provider facts from string and procedure inventory rows #6575

New issue

Open

opened 2026-11-01 21:12:22 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-11-01 21:12:22 +00:00

Owner

Originally created by @morluto on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8374

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

Ghidra inventory search drops provider facts that are present on the corresponding list rows. search_strings loses encoding, termination and byte length; search_procedures loses the procedure external/thunk classification. trace_feature uses these searches and therefore also omits the facts. Addresses and text are preserved; this report concerns the missing qualifications.

Reproduction

On macOS arm64, compile this self-owned fixture without executing it:

const char message[] = "rea_encoding_marker";
int main(void) { return message[0]; }

Run clang -O0 -g -arch arm64 simple.c -o simple, open it through REA MCP with Ghidra 12.1.4, then call list_strings({detail:"complete"}), search_strings({pattern:"rea_encoding_marker", mode:"literal", case_sensitive:false, detail:"complete"}), and trace_feature({query:"rea_encoding_marker"}). On this build, the same address/value appeared as follows:

list_strings:   {"address":"0x100000390","value":"rea_encoding_marker","string":{"encoding":"US-ASCII","termination":"present_or_not_required","byte_length":20}}
search_strings: {"address":"0x100000390","value":"rea_encoding_marker"}
trace_feature:  {"address":"0x100000390","value":"rea_encoding_marker","type":"string"}

A separate real Ghidra control on current main used a synthetic Mach-O function _Kelvin_marker at 0x1000002ec: list_procedures included procedure: {external:false, thunk:false, thunk_target:null}; search_procedures({pattern:"Kelvin",mode:"literal",case_sensitive:false,detail:"complete"}) and the procedure match in trace_feature({query:"Kelvin",case_sensitive:false}) retained its symbol metadata but omitted procedure. This proves the loss for one ordinary function; external and thunk entries were not separately exercised.

A separate selected Ghidra project defined the same Unicode text at two addresses, once as UTF-8 and once as UTF-16. list_strings exposed the distinct types and byte lengths; search_strings and trace_feature returned both rows without that information. On that project, CLI rea search ... Σ --provider ghidra --json and rea trace ... Σ --provider ghidra --json also omitted it. The original target and all source-project files were unchanged by REA. Both MCP sessions closed normally; successful MCP calls used the full TypeScript SDK-validated schemas.

Expected behavior

Preserve the provider's string qualifications and procedure classification when selecting the same inventory rows, including in the composed trace and CLI/MCP results. An encoding is the selected Ghidra data interpretation, not independent proof of the original source encoding. Preserve unknown or absent facts when the provider did not supply them. This follows REA's requirement to preserve source evidence and uncertainty across boundaries.

Impact and scope

An agent cannot distinguish UTF-8 from UTF-16 matches using search or trace output, despite that information being present in the same provider inventory. Procedure search likewise omits the inventory classification in the tested case. list_strings supplies a same-session workaround, with address and value used to correlate rows. Reproduced on the ASCII fixture and the selected Unicode project on one Ghidra/macOS host. The corresponding Kuna 1.986 control retained its UTF-8 encoding metadata through inventory, search and trace. Other providers, targets and hosts are unverified.

REA version

Current main d54a4661da20382ad8456564a08569fdd24131c5.

Runtime and platform

Node 24.22.1; macOS 26.3 arm64; Ghidra 12.1.4 / JDK 21; Kuna 1.986 control; TypeScript MCP SDK 2.3.1.

Target and artifact context

Synthetic unexecuted arm64 Mach-O; simple fixture SHA-256 b68170876fd69e97cf00d2a06250ac3a213c4110a2ddc55fd730e288afdb7314. The selected-project control is a separate synthetic fixture with explicit analyst-defined data types.

Interface

MCP and CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The smallest same-session MCP output is above. The CLI control used a separate source project with UTF-8 and UTF-16 definitions and returned the two matching rows without string metadata. A successful trace need not imply complete reference discovery; that question is separate.

Analysis or suspected cause

Source-derived: the Ghidra bridge's search loop passes null as the facts name, so resultItem omits its facts. The same bridge search loop handles both strings and procedures. The Ghidra search output parser also selects the unqualified addressed-row schema for search_strings. The shared trace already carries fields that reach it; fixing only its projection would not recover facts discarded upstream.

Additional context

Related closed #5503 concerned Hopper qualifications lost between search and trace; here the earliest observed loss is Ghidra inventory to search. Related #8218 covers finding text not represented correctly in the inventory.

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Originally created by @morluto on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8374 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior Ghidra inventory search drops provider facts that are present on the corresponding list rows. `search_strings` loses encoding, termination and byte length; `search_procedures` loses the procedure external/thunk classification. `trace_feature` uses these searches and therefore also omits the facts. Addresses and text are preserved; this report concerns the missing qualifications. ### Reproduction On macOS arm64, compile this self-owned fixture without executing it: ```c const char message[] = "rea_encoding_marker"; int main(void) { return message[0]; } ``` Run `clang -O0 -g -arch arm64 simple.c -o simple`, open it through REA MCP with Ghidra 12.1.4, then call `list_strings({detail:"complete"})`, `search_strings({pattern:"rea_encoding_marker", mode:"literal", case_sensitive:false, detail:"complete"})`, and `trace_feature({query:"rea_encoding_marker"})`. On this build, the same address/value appeared as follows: ```json list_strings: {"address":"0x100000390","value":"rea_encoding_marker","string":{"encoding":"US-ASCII","termination":"present_or_not_required","byte_length":20}} search_strings: {"address":"0x100000390","value":"rea_encoding_marker"} trace_feature: {"address":"0x100000390","value":"rea_encoding_marker","type":"string"} ``` A separate real Ghidra control on current main used a synthetic Mach-O function `_Kelvin_marker` at `0x1000002ec`: `list_procedures` included `procedure: {external:false, thunk:false, thunk_target:null}`; `search_procedures({pattern:"Kelvin",mode:"literal",case_sensitive:false,detail:"complete"})` and the procedure match in `trace_feature({query:"Kelvin",case_sensitive:false})` retained its symbol metadata but omitted `procedure`. This proves the loss for one ordinary function; external and thunk entries were not separately exercised. A separate selected Ghidra project defined the same Unicode text at two addresses, once as UTF-8 and once as UTF-16. `list_strings` exposed the distinct types and byte lengths; `search_strings` and `trace_feature` returned both rows without that information. On that project, CLI `rea search ... Σ --provider ghidra --json` and `rea trace ... Σ --provider ghidra --json` also omitted it. The original target and all source-project files were unchanged by REA. Both MCP sessions closed normally; successful MCP calls used the full TypeScript SDK-validated schemas. ### Expected behavior Preserve the provider's string qualifications and procedure classification when selecting the same inventory rows, including in the composed trace and CLI/MCP results. An encoding is the selected Ghidra data interpretation, not independent proof of the original source encoding. Preserve unknown or absent facts when the provider did not supply them. This follows REA's requirement to preserve source evidence and uncertainty across boundaries. ### Impact and scope An agent cannot distinguish UTF-8 from UTF-16 matches using search or trace output, despite that information being present in the same provider inventory. Procedure search likewise omits the inventory classification in the tested case. `list_strings` supplies a same-session workaround, with address and value used to correlate rows. Reproduced on the ASCII fixture and the selected Unicode project on one Ghidra/macOS host. The corresponding Kuna 1.986 control retained its UTF-8 encoding metadata through inventory, search and trace. Other providers, targets and hosts are unverified. ### REA version Current main `d54a4661da20382ad8456564a08569fdd24131c5`. ### Runtime and platform Node 24.22.1; macOS 26.3 arm64; Ghidra 12.1.4 / JDK 21; Kuna 1.986 control; TypeScript MCP SDK 2.3.1. ### Target and artifact context Synthetic unexecuted arm64 Mach-O; simple fixture SHA-256 `b68170876fd69e97cf00d2a06250ac3a213c4110a2ddc55fd730e288afdb7314`. The selected-project control is a separate synthetic fixture with explicit analyst-defined data types. ### Interface MCP and CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The smallest same-session MCP output is above. The CLI control used a separate source project with UTF-8 and UTF-16 definitions and returned the two matching rows without `string` metadata. A successful trace need not imply complete reference discovery; that question is separate. ### Analysis or suspected cause Source-derived: the Ghidra bridge's [search loop passes `null` as the facts name](https://github.com/morluto/rea/blob/d54a4661da20382ad8456564a08569fdd24131c5/bridge/ghidra/ReaGhidraBridge.java#L2292-L2299), so [`resultItem` omits its facts](https://github.com/morluto/rea/blob/d54a4661da20382ad8456564a08569fdd24131c5/bridge/ghidra/ReaGhidraBridge.java#L4576-L4585). The same bridge search loop handles both strings and procedures. The [Ghidra search output parser](https://github.com/morluto/rea/blob/d54a4661da20382ad8456564a08569fdd24131c5/src/ghidra/GhidraInventoryValues.ts#L479-L486) also selects the unqualified addressed-row schema for `search_strings`. The shared trace already carries fields that reach it; fixing only its projection would not recover facts discarded upstream. ### Additional context Related closed #5503 concerned Hopper qualifications lost between search and trace; here the earliest observed loss is Ghidra inventory to search. Related #8218 covers finding text not represented correctly in the inventory. cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

bsakaguchi added the

bug

triage

labels

2026-11-01 21:12:22 +00:00

bsakaguchi commented

2026-11-01 21:12:23 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

A disposable, uncommitted prototype tests the producing boundary on main d54a4661: the Ghidra search bridge includes the same string or procedure facts as its inventory row, and the Ghidra search output parser accepts the matching inventory row contract. It adds no new provider operation or fallback. This is exploratory evidence, not a proposed ready-to-merge patch.

Real Ghidra 12.1.4 MCP on the same selected Unicode project: baseline search and trace omitted the two string qualifications; the prototype preserved the exact UTF-8/16 encoding, termination and byte lengths from list_strings in both. On the separate procedure fixture, it likewise retained the list classification in search and trace. The selected source project and target were unchanged.
A separate unexecuted Mach-O importing puts exercised real external and thunk rows. In the prototype, /usr/lib/libSystem.B.dylib::_puts retained {external:true,thunk:false,thunk_target:null} and _puts retained {external:false,thunk:true,thunk_target:"EXTERNAL:0x1"} through both search and trace. Its SHA-256 remained dc6226fe389901d641ba5a2fcd1ff6b9298cf2708095b02b4c27ec4e56d3fa40.
With full and compact advertised MCP schema profiles, real Ghidra string and procedure searches preserved the selected facts through summary, native-inventory pages, Evidence export, close, and import into the target-free session. A real Kuna string-search control also passed those paths. These are fixture/host controls, not coverage of all providers, formats or clients.
The first focused run had 3 failures and 68 passes: existing Ghidra fake-bridge/captured-row tests supplied the previous unqualified search shape. After updating those fixture expectations in the disposable checkout, the four selected files passed 71 tests. This illustrates why those tests did not detect the loss: they modeled the old producer output, rather than comparing list, search and trace facts through a real provider. No real-provider regression has yet been added to the repository.

A production change should add that observable cross-workflow regression, cover malformed/missing producer facts and cancellation, validate both output profiles and offline paging, and assess any client impact from the additional row fields. Other platforms and Ghidra releases remain unverified; no full local or CI gate was run for the prototype.

Real-verifier regression experiment: I added a temporary check to the existing public-MCP Ghidra boundary lane that compares the exact inventory row against its corresponding search row and trace_feature match for a defined string, selected function, external function and, when present, a thunk. A direct run of that same check against main fails at the string comparison; against the prototype it passes all four, including the real thunk. Both sessions closed, and the source-built target hash was unchanged.

The normal host-native verifier initially hit existing #8172 before the boundary lane. After changing both comparisons in the disposable verifier to use the already-retained Evidence references (the separate fix is tracked in PR #8258), it reached this check and exposed another existing assertion that expected a long string search to omit metadata. Updating that assertion to compare the original inventory row allowed the complete node scripts/verify-real-ghidra.mjs run to finish with ok:true, cleanup:complete, debug/stripped/type-layout/jump-table fixtures, 441 successful and 52 rejected boundary calls, and a verified empty descendant list. This was a built runtime in the combined temporary worktree, with Ghidra 12.1.4, explicit JDK 21 and Node 24.22.1 on macOS arm64; it is not a clean-main or hosted CI pass. The uncommitted regression and verifier changes are not an integrated fix.

The long-string assertion is an additional reason existing real-provider verification missed the problem: it explicitly required the incomplete {address,value} search projection. Comparing the search and trace with the list row catches the loss at the public workflow boundary without turning a provider inventory into a claim of exhaustive discovery.

Regex and interrupted paging control: with the same uncommitted prototype and real Ghidra, mode:"regex" search preserved inventory facts for three selected-project string rows and two external/thunk procedure rows. With each MCP schema profile, I requested a one-row native-inventory page, exported Evidence, closed the binary, imported the bundle into the target-free session, then continued from next_offset; all rows and explicitly selected facts matched the original inventory. Invalid [ regexes returned invalid_request, after which the same session successfully searched and paged. The selected source project and both target digests remained unchanged. The complete trace checks were literal searches, because trace_feature does not offer regex mode. This is a bounded fixture control, not a cancellation, multi-host or full-provider conformance result.

Malformed producer controls: in the disposable MCP boundary harness, valid address/value rows missing the now-required Ghidra string or procedure facts, and a string fact with a negative byte length, returned unreadable_output without an Evidence ID. The two affected focused MCP test files passed 11 tests after adding those controlled cases. This uses an injected bridge response; it is not a real Ghidra corruption or cancellation test.

@morluto commented on GitHub (Oct 31, 2026): A **disposable, uncommitted prototype** tests the producing boundary on main `d54a4661`: the Ghidra search bridge includes the same `string` or `procedure` facts as its inventory row, and the Ghidra search output parser accepts the matching inventory row contract. It adds no new provider operation or fallback. This is exploratory evidence, not a proposed ready-to-merge patch. - Real Ghidra 12.1.4 MCP on the same selected Unicode project: baseline search and trace omitted the two string qualifications; the prototype preserved the exact UTF-8/16 encoding, termination and byte lengths from `list_strings` in both. On the separate procedure fixture, it likewise retained the list classification in search and trace. The selected source project and target were unchanged. - A separate unexecuted Mach-O importing `puts` exercised real external and thunk rows. In the prototype, `/usr/lib/libSystem.B.dylib::_puts` retained `{external:true,thunk:false,thunk_target:null}` and `_puts` retained `{external:false,thunk:true,thunk_target:"EXTERNAL:0x1"}` through both search and trace. Its SHA-256 remained `dc6226fe389901d641ba5a2fcd1ff6b9298cf2708095b02b4c27ec4e56d3fa40`. - With full and compact advertised MCP schema profiles, real Ghidra string and procedure searches preserved the selected facts through summary, `native-inventory` pages, Evidence export, close, and import into the target-free session. A real Kuna string-search control also passed those paths. These are fixture/host controls, not coverage of all providers, formats or clients. - The first focused run had **3 failures and 68 passes**: existing Ghidra fake-bridge/captured-row tests supplied the previous unqualified search shape. After updating those fixture expectations in the disposable checkout, the four selected files passed **71 tests**. This illustrates why those tests did not detect the loss: they modeled the old producer output, rather than comparing list, search and trace facts through a real provider. No real-provider regression has yet been added to the repository. A production change should add that observable cross-workflow regression, cover malformed/missing producer facts and cancellation, validate both output profiles and offline paging, and assess any client impact from the additional row fields. Other platforms and Ghidra releases remain unverified; no full local or CI gate was run for the prototype. **Real-verifier regression experiment:** I added a temporary check to the existing public-MCP Ghidra boundary lane that compares the exact inventory row against its corresponding search row and `trace_feature` match for a defined string, selected function, external function and, when present, a thunk. A direct run of that same check against main fails at the string comparison; against the prototype it passes all four, including the real thunk. Both sessions closed, and the source-built target hash was unchanged. The normal host-native verifier initially hit existing #8172 before the boundary lane. After changing both comparisons in the disposable verifier to use the already-retained Evidence references (the separate fix is tracked in PR #8258), it reached this check and exposed another existing assertion that expected a long string search to omit metadata. Updating that assertion to compare the original inventory row allowed the complete `node scripts/verify-real-ghidra.mjs` run to finish with `ok:true`, `cleanup:complete`, debug/stripped/type-layout/jump-table fixtures, 441 successful and 52 rejected boundary calls, and a verified empty descendant list. This was a built runtime in the combined temporary worktree, with Ghidra 12.1.4, explicit JDK 21 and Node 24.22.1 on macOS arm64; it is not a clean-main or hosted CI pass. The uncommitted regression and verifier changes are not an integrated fix. The long-string assertion is an additional reason existing real-provider verification missed the problem: it explicitly required the incomplete `{address,value}` search projection. Comparing the search and trace with the list row catches the loss at the public workflow boundary without turning a provider inventory into a claim of exhaustive discovery. **Regex and interrupted paging control:** with the same uncommitted prototype and real Ghidra, `mode:"regex"` search preserved inventory facts for three selected-project string rows and two external/thunk procedure rows. With each MCP schema profile, I requested a one-row `native-inventory` page, exported Evidence, closed the binary, imported the bundle into the target-free session, then continued from `next_offset`; all rows and explicitly selected facts matched the original inventory. Invalid `[` regexes returned `invalid_request`, after which the same session successfully searched and paged. The selected source project and both target digests remained unchanged. The complete trace checks were literal searches, because `trace_feature` does not offer regex mode. This is a bounded fixture control, not a cancellation, multi-host or full-provider conformance result. **Malformed producer controls:** in the disposable MCP boundary harness, valid address/value rows missing the now-required Ghidra string or procedure facts, and a string fact with a negative byte length, returned `unreadable_output` without an Evidence ID. The two affected focused MCP test files passed 11 tests after adding those controlled cases. This uses an injected bridge response; it is not a real Ghidra corruption or cancellation test.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6575

No description provided.

Rows
Columns