morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #1659] [Bug] search_strings misses some Rust string literals and does not say so #561

New issue

Closed

opened 2026-10-15 18:26:10 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:26:10 +00:00

Owner

Originally created by @CassieFox23 on GitHub (Oct 11, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/1659

TL;DR: REA's string search only sees text that Ghidra has marked as a string. In stripped Rust files, many messages are missed. The list tool says so in its limits, but the search tool does not.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

I searched stripped Rust libraries for text I knew was inside them. The search found none of it, or a small part.

The message "projection cache ignored" is in amethyst_filegraph.abi3.so once. REA's search_strings returns an empty list for it. The file is the Rust module of Amethyst Mod Manager 2.5.3 (GPL-3.0-or-later). It is built with PyO3 0.27.2 and is stripped.

Here are more searches. Each count is case-insensitive, which is the default of search_strings. The file column comes from grep -a -o -i -F.

file       text                        in the file  REA search_strings
filegraph  projection cache ignored    1            0
filegraph  projection                  9            1
filegraph  filegraph                   32           4
loot       masterlist                  23           15
loot       libloot                     170          163

Plain Ghidra gives the same picture, so the partial list comes from Ghidra. I ran analyzeHeadless with default analysis and no REA script on the filegraph file. It defines 2,551 strings, and none of them contains "projection cache ignored". REA's list_strings returns 2,551 for the same file.

In these files, the text sits back to back with no end byte, and the length is in the code. For example, the panic message has its length of 0x1e loaded in the code. The bytes are all there.

A way in does exist if you already hold the address. A read of 48 bytes at 0x13f787 returns the message. The xrefs call at 0x13f787 returns two code sites, 0x31ab97 and 0x31b8f7. The disassembly has an lea to that spot at file offsets 0x21ab97 and 0x21b8f7. But nothing in REA finds 0x13f787 from the text.

Reproduction

The smallest case is a short Rust library of about 45 lines. No crates are needed.

Save the source from the fold below as repro.rs.
Build it: rustc --edition 2021 -C opt-level=3 --crate-type cdylib -C strip=symbols -o libreprolib.so repro.rs. I used rustc 1.97.0 (2d8144b78 2026-07-07). The bytes can differ on other versions.
Check the text is there: strings -a libreprolib.so | grep -c "projection cache ignored" prints 2.
Run rea search /abs/path/libreprolib.so "projection cache ignored" --kind strings --provider ghidra --format json. Over MCP, call open_binary with provider_id set to ghidra, then search_strings.
The result is an empty list.

The same search for "reprolib" finds 2 strings. Both are export names, reprolib_describe and reprolib_enter. The file holds 9 copies of that word, and the rest sit in messages.

On the real file, get Amethyst Mod Manager 2.5.3 from its v2.5.3 release. The Flatpak is attached as AmethystModManager.flatpak. The file is /app/lib/python3.13/site-packages/amethyst_filegraph.abi3.so inside it. Its SHA-256 should be 288f8035cfd75c0262be2eb4735809b5a65797eb99642fbc56bf39222c99d919. I took my copy from an installed Flatpak and did not compare the release asset byte for byte (unverified).

The first import took about 90 seconds for the small library, and 4 to 5 minutes for the real file. After open_binary, later calls on the same MCP session answered in under a second.

Expected behavior

A caller who gets an empty list should know the list may be short. The list tool already carries the line Only Ghidra-defined string Data is observed. The search tool should carry it too.

The docs and the tool text should say what "complete" means here. It means complete over the strings Ghidra defined. It does not mean complete over the bytes in the file.

Impact and scope

Affected: search_strings and the CLI search --kind strings. A caller reads an empty list as "this text is not in the file".
Not affected: strings Ghidra does define. Most of the 170 "libloot" paths in the loot file were found. read_bytes and xrefs work when you hold an address.
Rust files are the case I saw. Other languages that pack text, such as Go, may behave the same. I have not tried them.

REA version

Main branch, at commit 4999e4be2 (#1651). Its package.json says 6.3.0. The released 6.3.0 package gave the same empty result on the filegraph file in an earlier run. I did not re-run it today.

Main has moved to 639b73097 since then. The string code, docs and limits have not changed there. I checked with git diff.

Runtime and platform

Node 24.18.0; Nobara 44 (Fedora 44 base), Linux 7.2.9, x86-64; Ghidra 12.1.4 (Apache-2.0); JDK 21.0.12.1

Target and artifact context

ELF64 x86-64 shared objects, stripped, Rust with PyO3. One file is 4,642,008 bytes and one is 5,836,952 bytes. The third is the short library above.

Interface

MCP

Earliest failure stage

Transport or result projection

Relevant output or evidence

search_strings pattern="projection cache ignored" -> []
list_strings -> 2551 items, none contains the phrase
read_bytes 0x13f787, 48 bytes -> "&[filegraph] projection cache ignored: ..."
xrefs 0x13f787 -> ["0x31ab97", "0x31b8f7"]

Where the docs and code say this today:

The Strings row of the inventory table in docs/ghidra-provider.md says Only Ghidra-defined string Data is observed.
In src/ghidra/GhidraProviderCapabilities.ts, the list case carries that line. The search case carries only the regex line.
In src/contracts/officialToolContracts.ts, the search tool says its results are "complete over the native inventory".

Typed outcome or recovery expected

The search_strings result carries the same limit as list_strings. The limit could also name the way around it. That way is read_bytes plus xrefs, once the address is known.

Analysis or suspected cause

This part is read from source and from Ghidra's own counts. The REA bridge lists strings with getDefinedData(true) and keeps the ones with a string value. So the search can only see what Ghidra defined. I did not look at why Ghidra defines some Rust literals and not others. One idea is the packed text with no end byte. I have not checked it.

A tool that finds a byte pattern would help. That is a feature, and I can open it as its own issue if you prefer.

Fixture source

use std::cell::Cell;

thread_local! {
    static DEPTH: Cell<isize> = const { Cell::new(0) };
}

#[inline(never)]
fn work(name: &str) -> Result<usize, String> {
    if name.len() > 100 {
        Err(format!("[reprolib] projection cache ignored: {name}"))
    } else {
        Ok(name.len())
    }
}

#[no_mangle]
pub extern "C" fn reprolib_enter(name: *const u8, len: usize) -> isize {
    let s = unsafe { std::str::from_utf8_unchecked(std::slice::from_raw_parts(name, len)) };
    DEPTH.with(|d| {
        let n = d.get();
        if n < 0 {
            return -1;
        }
        d.set(n + 1);
        let r = match work(s) {
            Ok(v) => v as isize,
            Err(e) => {
                eprintln!("[reprolib] uncaught failure at ffi boundary: {e}");
                -2
            }
        };
        d.set(d.get() - 1);
        r
    })
}

#[no_mangle]
pub extern "C" fn reprolib_describe(kind: u32) -> *const u8 {
    match kind {
        0 => "[reprolib] cache directory is missing".as_ptr(),
        1 => "[reprolib] catalog is corrupt:".as_ptr(),
        2 => "[reprolib] database error:".as_ptr(),
        3 => "[reprolib] projection cache ignored: unreadable".as_ptr(),
        _ => "[reprolib] unknown".as_ptr(),
    }
}

Commands

export GHIDRA_INSTALL_DIR=/path/to/ghidra_12.1.4_PUBLIC JAVA_HOME=/path/to/jdk-21
export REA_GHIDRA_STARTUP_TIMEOUT_MS=600000

node scripts/rea.mjs search /abs/amethyst_filegraph.abi3.so "projection cache ignored" --kind strings --provider ghidra --format json

grep -a -o -i -F "projection cache ignored" /abs/amethyst_filegraph.abi3.so | wc -l
strings -a -t x /abs/amethyst_filegraph.abi3.so | grep -F "projection cache ignored"

The script P4Dump2.java counts Ghidra's defined strings and the raw byte matches for a list of terms. Run it with -postScript P4Dump2.java label PyInit_ "projection cache ignored,filegraph". I can share it if that helps.

Test setup

Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
RAM: 32 GB DDR4-3600
Node: 24.18.0
Main: 4999e4be2 (#1651)
Ghidra: 12.1.4 (Apache-2.0), with JDK 21.0.12.1
Target: Amethyst Mod Manager 2.5.3 (GPL-3.0-or-later)
Target: libloot 0.29.6 (GPL-3.0-or-later)

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

Originally created by @CassieFox23 on GitHub (Oct 11, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/1659 **TL;DR:** REA's string search only sees text that Ghidra has marked as a string. In stripped Rust files, many messages are missed. The list tool says so in its limits, but the search tool does not. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior I searched stripped Rust libraries for text I knew was inside them. The search found none of it, or a small part. The message "projection cache ignored" is in `amethyst_filegraph.abi3.so` once. REA's `search_strings` returns an empty list for it. The file is the Rust module of Amethyst Mod Manager 2.5.3 (GPL-3.0-or-later). It is built with PyO3 0.27.2 and is stripped. Here are more searches. Each count is case-insensitive, which is the default of `search_strings`. The file column comes from `grep -a -o -i -F`. ```text file text in the file REA search_strings filegraph projection cache ignored 1 0 filegraph projection 9 1 filegraph filegraph 32 4 loot masterlist 23 15 loot libloot 170 163 ``` Plain Ghidra gives the same picture, so the partial list comes from Ghidra. I ran `analyzeHeadless` with default analysis and no REA script on the filegraph file. It defines 2,551 strings, and none of them contains "projection cache ignored". REA's `list_strings` returns 2,551 for the same file. In these files, the text sits back to back with no end byte, and the length is in the code. For example, the panic message has its length of 0x1e loaded in the code. The bytes are all there. A way in does exist if you already hold the address. A read of 48 bytes at 0x13f787 returns the message. The xrefs call at 0x13f787 returns two code sites, 0x31ab97 and 0x31b8f7. The disassembly has an `lea` to that spot at file offsets 0x21ab97 and 0x21b8f7. But nothing in REA finds 0x13f787 from the text. ### Reproduction The smallest case is a short Rust library of about 45 lines. No crates are needed. 1. Save the source from the fold below as `repro.rs`. 2. Build it: `rustc --edition 2021 -C opt-level=3 --crate-type cdylib -C strip=symbols -o libreprolib.so repro.rs`. I used rustc 1.97.0 (2d8144b78 2026-07-07). The bytes can differ on other versions. 3. Check the text is there: `strings -a libreprolib.so | grep -c "projection cache ignored"` prints 2. 4. Run `rea search /abs/path/libreprolib.so "projection cache ignored" --kind strings --provider ghidra --format json`. Over MCP, call `open_binary` with `provider_id` set to `ghidra`, then `search_strings`. 5. The result is an empty list. The same search for "reprolib" finds 2 strings. Both are export names, `reprolib_describe` and `reprolib_enter`. The file holds 9 copies of that word, and the rest sit in messages. On the real file, get Amethyst Mod Manager 2.5.3 from its [v2.5.3 release](https://github.com/ChrisDKN/Amethyst-Mod-Manager/releases/tag/v2.5.3). The Flatpak is attached as `AmethystModManager.flatpak`. The file is `/app/lib/python3.13/site-packages/amethyst_filegraph.abi3.so` inside it. Its SHA-256 should be `288f8035cfd75c0262be2eb4735809b5a65797eb99642fbc56bf39222c99d919`. I took my copy from an installed Flatpak and did not compare the release asset byte for byte (unverified). The first import took about 90 seconds for the small library, and 4 to 5 minutes for the real file. After `open_binary`, later calls on the same MCP session answered in under a second. ### Expected behavior A caller who gets an empty list should know the list may be short. The list tool already carries the line `Only Ghidra-defined string Data is observed`. The search tool should carry it too. The docs and the tool text should say what "complete" means here. It means complete over the strings Ghidra defined. It does not mean complete over the bytes in the file. ### Impact and scope - Affected: `search_strings` and the CLI `search --kind strings`. A caller reads an empty list as "this text is not in the file". - Not affected: strings Ghidra does define. Most of the 170 "libloot" paths in the loot file were found. `read_bytes` and `xrefs` work when you hold an address. - Rust files are the case I saw. Other languages that pack text, such as Go, may behave the same. I have not tried them. ### REA version Main branch, at commit 4999e4be2 (#1651). Its package.json says 6.3.0. The released 6.3.0 package gave the same empty result on the filegraph file in an earlier run. I did not re-run it today. Main has moved to 639b73097 since then. The string code, docs and limits have not changed there. I checked with git diff. ### Runtime and platform Node 24.18.0; Nobara 44 (Fedora 44 base), Linux 7.2.9, x86-64; Ghidra 12.1.4 (Apache-2.0); JDK 21.0.12.1 ### Target and artifact context ELF64 x86-64 shared objects, stripped, Rust with PyO3. One file is 4,642,008 bytes and one is 5,836,952 bytes. The third is the short library above. ### Interface MCP ### Earliest failure stage Transport or result projection ### Relevant output or evidence ```text search_strings pattern="projection cache ignored" -> [] list_strings -> 2551 items, none contains the phrase read_bytes 0x13f787, 48 bytes -> "&[filegraph] projection cache ignored: ..." xrefs 0x13f787 -> ["0x31ab97", "0x31b8f7"] ``` Where the docs and code say this today: - The Strings row of the inventory table in `docs/ghidra-provider.md` says `Only Ghidra-defined string Data is observed`. - In `src/ghidra/GhidraProviderCapabilities.ts`, the list case carries that line. The search case carries only the regex line. - In `src/contracts/officialToolContracts.ts`, the search tool says its results are "complete over the native inventory". ### Typed outcome or recovery expected The `search_strings` result carries the same limit as `list_strings`. The limit could also name the way around it. That way is `read_bytes` plus `xrefs`, once the address is known. ### Analysis or suspected cause This part is read from source and from Ghidra's own counts. The REA bridge lists strings with `getDefinedData(true)` and keeps the ones with a string value. So the search can only see what Ghidra defined. I did not look at why Ghidra defines some Rust literals and not others. One idea is the packed text with no end byte. I have not checked it. A tool that finds a byte pattern would help. That is a feature, and I can open it as its own issue if you prefer. <details> <summary>Fixture source</summary> ```rust use std::cell::Cell; thread_local! { static DEPTH: Cell<isize> = const { Cell::new(0) }; } #[inline(never)] fn work(name: &str) -> Result<usize, String> { if name.len() > 100 { Err(format!("[reprolib] projection cache ignored: {name}")) } else { Ok(name.len()) } } #[no_mangle] pub extern "C" fn reprolib_enter(name: *const u8, len: usize) -> isize { let s = unsafe { std::str::from_utf8_unchecked(std::slice::from_raw_parts(name, len)) }; DEPTH.with(|d| { let n = d.get(); if n < 0 { return -1; } d.set(n + 1); let r = match work(s) { Ok(v) => v as isize, Err(e) => { eprintln!("[reprolib] uncaught failure at ffi boundary: {e}"); -2 } }; d.set(d.get() - 1); r }) } #[no_mangle] pub extern "C" fn reprolib_describe(kind: u32) -> *const u8 { match kind { 0 => "[reprolib] cache directory is missing".as_ptr(), 1 => "[reprolib] catalog is corrupt:".as_ptr(), 2 => "[reprolib] database error:".as_ptr(), 3 => "[reprolib] projection cache ignored: unreadable".as_ptr(), _ => "[reprolib] unknown".as_ptr(), } } ``` </details> <details> <summary>Commands</summary> ```sh export GHIDRA_INSTALL_DIR=/path/to/ghidra_12.1.4_PUBLIC JAVA_HOME=/path/to/jdk-21 export REA_GHIDRA_STARTUP_TIMEOUT_MS=600000 node scripts/rea.mjs search /abs/amethyst_filegraph.abi3.so "projection cache ignored" --kind strings --provider ghidra --format json grep -a -o -i -F "projection cache ignored" /abs/amethyst_filegraph.abi3.so | wc -l strings -a -t x /abs/amethyst_filegraph.abi3.so | grep -F "projection cache ignored" ``` The script `P4Dump2.java` counts Ghidra's defined strings and the raw byte matches for a list of terms. Run it with `-postScript P4Dump2.java label PyInit_ "projection cache ignored,filegraph"`. I can share it if that helps. </details> <details> <summary>Test setup</summary> - Box: Nobara 44 (Fedora 44 base), Linux 7.2.9 - CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads - RAM: 32 GB DDR4-3600 - Node: 24.18.0 - Main: 4999e4be2 (#1651) - Ghidra: 12.1.4 (Apache-2.0), with JDK 21.0.12.1 - Target: [Amethyst Mod Manager](https://github.com/ChrisDKN/Amethyst-Mod-Manager) 2.5.3 ([GPL-3.0-or-later](https://github.com/ChrisDKN/Amethyst-Mod-Manager/blob/main/LICENSE)) - Target: [libloot](https://github.com/loot/libloot) 0.29.6 ([GPL-3.0-or-later](https://github.com/loot/libloot/blob/master/LICENSE)) </details> 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

bsakaguchi

2026-10-15 18:26:10 +00:00

closed this issue
added the
area: native

bug
labels

bsakaguchi commented

2026-10-15 18:26:11 +00:00

Author

Owner

@CassieFox23 commented on GitHub (Oct 11, 2026):

TL;DR: The limit now shows on search_strings, as this issue asked, in 4733fc810. On Linux, the missing text still reproduces: Ghidra does not define it as a string, and the new limit says so.

What I saw

I built main at 4733fc810 and ran both the CLI and an MCP session (open_binary with provider_id set to ghidra, then search_strings and list_strings).

The search result for "projection cache ignored" now carries this line, and so does the list_strings result:

Only Ghidra-defined string Data is observed; packed or undefined string bytes may be absent. Completeness applies to that inventory, and an empty search does not establish that text is absent from the artifact. If an address is known, use read_bytes and xrefs to inspect bytes and references.

Your test used a Mach-O file on macOS arm64. Mine are ELF files on Linux x86-64, so the two results can both be true. To repeat mine:

rustc --version
rustc 1.97.0 (2d8144b78 2026-07-07)
rustc --edition 2021 -C opt-level=3 --crate-type cdylib -C strip=symbols -o libreprolib.so repro.rs

The source is the "Fixture source" fold in the issue body. The bytes can differ on other rustc versions.

File	Phrase in the file	Ghidra alone: defined strings that hold it	REA `search_strings`	Limit line on the result
libreprolib.so (built as above)	2	0	empty list, CLI and MCP	yes
amethyst_filegraph.abi3.so	1	0	empty list, CLI and MCP	yes

"Phrase in the file" counts raw byte matches in the loaded image, found by a Ghidra memory search. For the fixture, strings -a agrees: 2 lines. "Ghidra alone" is analyzeHeadless with default analysis and a short script, with no REA code.

The fixture has 214 defined strings. The word "reprolib" has 9 raw copies and 2 defined ones, both export names. REA finds those 2.

Your new test file, tests/boundary/mcp/ghidraInventoryMcp.test.ts, passes on main: 2 tests, exit code 0. Main has since moved to 69a8b3a95. That commit does not touch the string inventory code.

Not run

Hopper and IDA providers.
A Mach-O build of the fixture on macOS.
Other rustc versions, or other Linux builds of the same code.
read_bytes and xrefs again on this commit. The issue body has those results.
The loot file, and the full test suite.

Thanks, morluto, for the fix.

Test setup

Box: Nobara 44 (Fedora 44 base), Linux 7.2.9
CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads
RAM: 32 GB DDR4-3600
Node: 24.18.0
Ghidra: 12.1.4 (Apache-2.0) with JDK 21.0.12.1
Compiler: rustc 1.97.0
Head: 4733fc810

🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

@CassieFox23 commented on GitHub (Oct 11, 2026): **TL;DR:** The limit now shows on `search_strings`, as this issue asked, in 4733fc810. On Linux, the missing text still reproduces: Ghidra does not define it as a string, and the new limit says so. ### What I saw I built main at 4733fc810 and ran both the CLI and an MCP session (`open_binary` with `provider_id` set to `ghidra`, then `search_strings` and `list_strings`). The search result for "projection cache ignored" now carries this line, and so does the `list_strings` result: ```text Only Ghidra-defined string Data is observed; packed or undefined string bytes may be absent. Completeness applies to that inventory, and an empty search does not establish that text is absent from the artifact. If an address is known, use read_bytes and xrefs to inspect bytes and references. ``` Your test used a Mach-O file on macOS arm64. Mine are ELF files on Linux x86-64, so the two results can both be true. To repeat mine: ```text rustc --version rustc 1.97.0 (2d8144b78 2026-07-07) rustc --edition 2021 -C opt-level=3 --crate-type cdylib -C strip=symbols -o libreprolib.so repro.rs ``` The source is the "Fixture source" fold in the issue body. The bytes can differ on other rustc versions. | File | Phrase in the file | Ghidra alone: defined strings that hold it | REA `search_strings` | Limit line on the result | |---|---|---|---|---| | libreprolib.so (built as above) | 2 | 0 | empty list, CLI and MCP | yes | | amethyst_filegraph.abi3.so | 1 | 0 | empty list, CLI and MCP | yes | "Phrase in the file" counts raw byte matches in the loaded image, found by a Ghidra memory search. For the fixture, `strings -a` agrees: 2 lines. "Ghidra alone" is `analyzeHeadless` with default analysis and a short script, with no REA code. The fixture has 214 defined strings. The word "reprolib" has 9 raw copies and 2 defined ones, both export names. REA finds those 2. Your new test file, `tests/boundary/mcp/ghidraInventoryMcp.test.ts`, passes on main: 2 tests, exit code 0. Main has since moved to 69a8b3a95. That commit does not touch the string inventory code. ### Not run - Hopper and IDA providers. - A Mach-O build of the fixture on macOS. - Other rustc versions, or other Linux builds of the same code. - `read_bytes` and `xrefs` again on this commit. The issue body has those results. - The `loot` file, and the full test suite. Thanks, morluto, for the fix. <details> <summary>Test setup</summary> - Box: Nobara 44 (Fedora 44 base), Linux 7.2.9 - CPU: AMD Ryzen 9 5900X, 12 cores / 24 threads - RAM: 32 GB DDR4-3600 - Node: 24.18.0 - Ghidra: 12.1.4 ([Apache-2.0](https://github.com/NationalSecurityAgency/ghidra/blob/master/LICENSE)) with JDK 21.0.12.1 - Compiler: rustc 1.97.0 - Head: 4733fc810 </details> 🤖 Generated with Claude Code (Sonnet 5.5, verified by Opus 5.5) 🦊

bsakaguchi referenced this issue

2026-10-16 21:14:14 +00:00

[PR #597] [MERGED] fix!: repair boundary contracts across CLI and MCP #2097

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#561

No description provided.

Rows
Columns