morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8218] [Design] Find short or encoded markers outside provider string inventories #6495

New issue

Open

opened 2026-11-01 21:11:38 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-11-01 21:11:38 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8218

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

How should an agent find a known short marker or explicitly encoded literal when it is absent from the selected provider's string inventory, without confusing byte matches with defined strings or forcing an unbounded whole-program scan?

Context and evidence

On macOS arm64 with REA 53a4ab78451a0654e9c3beb3d43a5e371a288fcc, Ghidra 12.1.4 and Kuna 1.986, a self-built, unexecuted Mach-O contained an isolated ASCII QZ7\0, a longer QZ7_LONG_MARK!\0, and UTF-16LE bytes for 日本\0.

Real MCP search_strings({pattern:"QZ7"}) returned only the longer string with both providers; search_strings({pattern:"日本"}) returned no rows. Both disclosed their different inventory limitations. This is expected given the observed inventories, not a claim that these tools promised exhaustive raw-byte search.
Direct Kuna strings --encoding all --min-length 3 --no-xrefs --json reported the three-character ASCII row; with its five-character minimum it returned only the longer row. Direct Kuna --encoding utf16 --min-length 2 did not find the non-Latin wide literal on this fixture.
REA/Ghidra search_bytes with exact ASCII and UTF-16LE byte sequences located both in initialized memory, including file offsets and coverage. REA/Kuna reported search_bytes unavailable. These are byte matches; their text interpretation comes from the known fixture, not from a Ghidra defined-string observation.
Both providers' exact xrefs for the short marker returned no sites, while pseudocode for the fixture's indexing function showed address-plus-index expressions. Empty exact references therefore did not establish that the literal is unused. No runtime behavior was observed.

The target digest was unchanged and both sessions closed successfully. This is one fixture, not a prevalence, correctness or performance benchmark.

Expected interaction with targets

Read caller-selected static image bytes and, where applicable, query an existing static-analysis provider. No target execution or modification is proposed.

Boundary details (optional)

Separate raw file bytes, initialized provider memory, defined strings, decoded candidates and inferred use. Preserve selected encodings/termination, exact artifact/slice and address mapping, bounded ranges/pagination, completeness and unknown reference coverage. A literal query may be a substring of a longer string; do not replace existing provider inventory semantics or claim a byte match proves a string or a code use. Low minimum lengths can cause very large inventories and different cache/identity costs.

Options and tradeoffs

Improve discovery of the existing Ghidra byte-search fallback; consider a bounded task-oriented literal/encoded-byte search usable with a verified local image mapping; or consider a caller-selected narrower Kuna candidate scan when semantics and resource bounds are established. Compare these against #8170 and existing file-layout evidence. Do not add a provider command wrapper or lower every default inventory threshold solely for parity.

Verification available so far

I have a prototype or observed results

Success criteria and open questions

Exercise three-character, long-substring, encoded, unterminated and false-positive bytes; mapped/file-only/uninitialized boundaries; exact versus interior references and unknown use; both providers and partial/error cases. Measure representative whole-task cost before changing an inventory policy. A controlled agent trace is needed to establish discoverability impact.

References

REA Kuna string policy, search_bytes contract, #8170 (Kuna byte-reading boundary), #8184 (feature trace coverage), and #8190 (outgoing-reference semantics).

Explicit provider recovery for an encoded literal

On macOS arm64, I tested an isolated, manually conflict-resolved combination of #8817's catalog wording and #8567 head b282fece7. With a source-owned, unexecuted Mach-O and real Kuna 1.990 / Ghidra 12.1.4, both compact and full MCP profiles followed a caller-driven UTF-16LE task: Kuna search_strings returned no match; its unavailable search_bytes response directed candidate discovery; complete binary_session showed Ghidra available, target-supported, and declaring byte search; explicitly selecting Ghidra found the known byte sequence at 0x1000003b0 with file offset 944. An exported bundle retained the original empty Kuna search Evidence exactly. Target SHA-256 ab0b231c1480722ebbd6d61a020d8f8b39928c94f471ea0ec23b33559d408ee5 was unchanged. A byte match is not a defined string or proof of use.

This validates one local combination and explicit handoff, not automatic provider switching, a naturally selected agent workflow, unavailable-Ghidra recovery, or an automatic merge. It does not resolve broader mapped-image/encoding questions in this issue.

Termination and non-text candidate control

On #8817 head 50ba6cb22a242a60f7e4bd12fc74c05ec91191ab, real Ghidra 12.1.4 on macOS arm64 and a separate source-owned arm64 Mach-O, search_strings for QZ7 returned no defined-string rows in either compact or full MCP. search_bytes for 51 5a 37 returned two initialized-memory matches: one embedded in the deliberately non-text bytes 13 51 5a 37 85 c7 fe 42, and a separate 51 5a 37 00 sequence. Searching for the four bytes including the terminator returned only the latter. A separate read_bytes returned 13515a3785c7fe42515a3700 for the 12-byte surrounding region in both profiles. The hit rows preserved addresses, file offsets and searched/skipped coverage; both had no containing procedure. The test's source establishes intended bytes, not runtime use or string classification by Ghidra. Target SHA-256 b151ae528adfe2fd8b7a8c9507070275258de02d87eddc10e323dd9456ce6755 was unchanged and it was not executed. This supports the wording that byte hits are not defined strings, and shows that caller-selected termination can narrow candidates; one constructed fixture does not establish false-positive rates or general completeness.

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8218 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed How should an agent find a known short marker or explicitly encoded literal when it is absent from the selected provider's string inventory, without confusing byte matches with defined strings or forcing an unbounded whole-program scan? ### Context and evidence On macOS arm64 with REA `53a4ab78451a0654e9c3beb3d43a5e371a288fcc`, Ghidra 12.1.4 and Kuna 1.986, a self-built, unexecuted Mach-O contained an isolated ASCII `QZ7\0`, a longer `QZ7_LONG_MARK!\0`, and UTF-16LE bytes for `日本\0`. - Real MCP `search_strings({pattern:"QZ7"})` returned only the longer string with **both** providers; `search_strings({pattern:"日本"})` returned no rows. Both disclosed their different inventory limitations. This is expected given the observed inventories, not a claim that these tools promised exhaustive raw-byte search. - Direct Kuna `strings --encoding all --min-length 3 --no-xrefs --json` reported the three-character ASCII row; with its five-character minimum it returned only the longer row. Direct Kuna `--encoding utf16 --min-length 2` did **not** find the non-Latin wide literal on this fixture. - REA/Ghidra `search_bytes` with exact ASCII and UTF-16LE byte sequences located both in initialized memory, including file offsets and coverage. REA/Kuna reported `search_bytes` unavailable. These are byte matches; their text interpretation comes from the known fixture, not from a Ghidra defined-string observation. - Both providers' exact `xrefs` for the short marker returned no sites, while pseudocode for the fixture's indexing function showed address-plus-index expressions. Empty exact references therefore did not establish that the literal is unused. No runtime behavior was observed. The target digest was unchanged and both sessions closed successfully. This is one fixture, not a prevalence, correctness or performance benchmark. ### Expected interaction with targets Read caller-selected static image bytes and, where applicable, query an existing static-analysis provider. No target execution or modification is proposed. ### Boundary details (optional) Separate raw file bytes, initialized provider memory, defined strings, decoded candidates and inferred use. Preserve selected encodings/termination, exact artifact/slice and address mapping, bounded ranges/pagination, completeness and unknown reference coverage. A literal query may be a substring of a longer string; do not replace existing provider inventory semantics or claim a byte match proves a string or a code use. Low minimum lengths can cause very large inventories and different cache/identity costs. ### Options and tradeoffs Improve discovery of the existing Ghidra byte-search fallback; consider a bounded task-oriented literal/encoded-byte search usable with a verified local image mapping; or consider a caller-selected narrower Kuna candidate scan when semantics and resource bounds are established. Compare these against #8170 and existing file-layout evidence. Do not add a provider command wrapper or lower every default inventory threshold solely for parity. ### Verification available so far I have a prototype or observed results ### Success criteria and open questions Exercise three-character, long-substring, encoded, unterminated and false-positive bytes; mapped/file-only/uninitialized boundaries; exact versus interior references and unknown use; both providers and partial/error cases. Measure representative whole-task cost before changing an inventory policy. A controlled agent trace is needed to establish discoverability impact. ### References [REA Kuna string policy](https://github.com/morluto/rea/blob/53a4ab78451a0654e9c3beb3d43a5e371a288fcc/docs/kuna-provider.md), `search_bytes` contract, #8170 (Kuna byte-reading boundary), #8184 (feature trace coverage), and #8190 (outgoing-reference semantics). ### Explicit provider recovery for an encoded literal On macOS arm64, I tested an isolated, manually conflict-resolved combination of #8817's catalog wording and #8567 head `b282fece7`. With a source-owned, unexecuted Mach-O and real Kuna 1.990 / Ghidra 12.1.4, both compact and full MCP profiles followed a caller-driven UTF-16LE task: Kuna `search_strings` returned no match; its unavailable `search_bytes` response directed candidate discovery; complete `binary_session` showed Ghidra available, target-supported, and declaring byte search; explicitly selecting Ghidra found the known byte sequence at `0x1000003b0` with file offset 944. An exported bundle retained the original empty Kuna search Evidence exactly. Target SHA-256 `ab0b231c1480722ebbd6d61a020d8f8b39928c94f471ea0ec23b33559d408ee5` was unchanged. A byte match is not a defined string or proof of use. This validates one local combination and explicit handoff, not automatic provider switching, a naturally selected agent workflow, unavailable-Ghidra recovery, or an automatic merge. It does not resolve broader mapped-image/encoding questions in this issue. ### Termination and non-text candidate control On #8817 head `50ba6cb22a242a60f7e4bd12fc74c05ec91191ab`, real Ghidra 12.1.4 on macOS arm64 and a separate source-owned arm64 Mach-O, `search_strings` for `QZ7` returned no defined-string rows in either compact or full MCP. `search_bytes` for `51 5a 37` returned two initialized-memory matches: one embedded in the deliberately non-text bytes `13 51 5a 37 85 c7 fe 42`, and a separate `51 5a 37 00` sequence. Searching for the four bytes including the terminator returned only the latter. A separate `read_bytes` returned `13515a3785c7fe42515a3700` for the 12-byte surrounding region in both profiles. The hit rows preserved addresses, file offsets and searched/skipped coverage; both had no containing procedure. The test's source establishes intended bytes, not runtime use or string classification by Ghidra. Target SHA-256 `b151ae528adfe2fd8b7a8c9507070275258de02d87eddc10e323dd9456ce6755` was unchanged and it was not executed. This supports the wording that byte hits are not defined strings, and shows that caller-selected termination can narrow candidates; one constructed fixture does not establish false-positive rates or general completeness.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:11:38 +00:00

bsakaguchi commented

2026-11-01 21:11:39 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

A controlled x86-64 ELF exercises termination, encoding, file-only and alignment boundaries on Kuna 1.986 and REA/Ghidra 12.1.4. Targets were not executed and their digests stayed unchanged.

The same RAW4 bytes occurred once followed by ff and once by NUL. Direct Kuna strings --encoding ascii --min-length 3 --termination any --no-xrefs returned both, marking only the latter NUL-terminated; --termination nul returned only the latter. Bounded REA/Ghidra search_bytes for 52415734 returned both addresses; including 00 returned only the NUL-ended occurrence. These are byte candidates, not proof of text use. The --no-xrefs outputs intentionally did not examine references.
An explicitly UTF-8 éclair was returned by direct Kuna's UTF-8 mode at its true start; ASCII mode returned clair from two bytes later. REA/Kuna's default string search returned the UTF-8 value; REA/Ghidra's defined-string search did not, but Ghidra byte search found its known bytes. An unrelated ASCII candidate e,g came from byte pairs within the UTF-16LE Japanese literal, illustrating that a printable run is not necessarily the intended text interpretation.
A file-only appended marker was absent from bounded initialized-memory results; Ghidra listed the corresponding non-loaded unallocated_0 among skipped blocks. A zero search restricted to .bss examined zero bytes and listed the block as uninitialized despite complete:true for the initialized-memory search. Independent LLVM headers establish these fixture boundaries; this is not an exhaustive file-search or runtime claim.
Two otherwise comparable ELF fixtures place the same NUL-ended UTF-16LE WIDER at odd address 0x20017d or even address 0x20017e. With minimum length five, direct Kuna --encoding utf16 and --encoding all and real REA/Kuna search_strings found the aligned value and omitted the odd-address value. REA/Ghidra defined-string and exact-byte searches found both. This controls the minimum length, while the different files and surrounding layouts limit causal certainty. Pinned Kuna source and test explicitly keep UTF-16 units on even addresses; the omission is a producer coverage boundary, not evidence of an upstream defect. REA currently notes Latin-range UTF-16LE and that other text can remain undiscovered, but does not name this alignment limitation.

These controls reinforce keeping encoding, termination, byte extent, scan domain and unknown coverage visible, with user-selected bounds. They do not establish prevalence or performance. Related #8245 asks separately about loaded versus metadata string provenance.

@morluto commented on GitHub (Oct 30, 2026): A controlled x86-64 ELF exercises termination, encoding, file-only and alignment boundaries on Kuna 1.986 and REA/Ghidra 12.1.4. Targets were not executed and their digests stayed unchanged. - The same `RAW4` bytes occurred once followed by `ff` and once by NUL. Direct Kuna `strings --encoding ascii --min-length 3 --termination any --no-xrefs` returned both, marking only the latter NUL-terminated; `--termination nul` returned only the latter. Bounded REA/Ghidra `search_bytes` for `52415734` returned both addresses; including `00` returned only the NUL-ended occurrence. These are byte candidates, not proof of text use. The `--no-xrefs` outputs intentionally did not examine references. - An explicitly UTF-8 `éclair` was returned by direct Kuna's UTF-8 mode at its true start; ASCII mode returned `clair` from two bytes later. REA/Kuna's default string search returned the UTF-8 value; REA/Ghidra's defined-string search did not, but Ghidra byte search found its known bytes. An unrelated ASCII candidate `e,g` came from byte pairs within the UTF-16LE Japanese literal, illustrating that a printable run is not necessarily the intended text interpretation. - A file-only appended marker was absent from bounded initialized-memory results; Ghidra listed the corresponding non-loaded `unallocated_0` among skipped blocks. A zero search restricted to `.bss` examined zero bytes and listed the block as uninitialized despite `complete:true` for the initialized-memory search. Independent LLVM headers establish these fixture boundaries; this is not an exhaustive file-search or runtime claim. - Two otherwise comparable ELF fixtures place the same NUL-ended UTF-16LE `WIDER` at odd address `0x20017d` or even address `0x20017e`. With minimum length five, direct Kuna `--encoding utf16` and `--encoding all` and real REA/Kuna `search_strings` found the aligned value and omitted the odd-address value. REA/Ghidra defined-string and exact-byte searches found both. This controls the minimum length, while the different files and surrounding layouts limit causal certainty. Pinned Kuna [source and test](https://github.com/Noelo-Lab/kuna/blob/v1.986/decompiler/crates/kuna-analysis/src/analyzers/strings/kuna_stringinv.rs#L426-L436) explicitly keep UTF-16 units on even addresses; the omission is a producer coverage boundary, not evidence of an upstream defect. REA currently notes Latin-range UTF-16LE and that other text can remain undiscovered, but does not name this alignment limitation. These controls reinforce keeping encoding, termination, byte extent, scan domain and unknown coverage visible, with user-selected bounds. They do not establish prevalence or performance. Related [#8245](https://github.com/morluto/rea/issues/8245) asks separately about loaded versus metadata string provenance.

bsakaguchi commented

2026-11-01 21:11:39 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:39 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Provider-aware recovery control with real REA b43c329 MCP, Ghidra 12.1.4 and Kuna 1.986 on the original source-owned short/encoded-string Mach-O:

The advertised search_strings description says that at a known address read_bytes and xrefs inspect bytes and references. binary_session(detail:"complete") reports all three relevant tools (read_bytes, search_bytes, xrefs) available for Ghidra, but reports the first two provider_missing for Kuna while xrefs remains available.
search_strings({pattern:"日本"}) returned no rows under either provider with their respective inventory limitations. At known address 0x3e4, Ghidra read_bytes returned the six initialized bytes e5652c670000, and search_bytes found that address with file offset 996. The source fixture establishes the intended UTF-16LE interpretation. Kuna returned capability_unavailable for both byte operations; its exact xrefs returned no rows with its explicit incomplete-reference limitation.

The generic catalog guidance is valid only when the selected provider offers the operations; discovery already exposes that distinction, so this is an agent recovery/discoverability question rather than a claim that Kuna promised byte reading. A switch to Ghidra can be useful when explicitly available and selected, but it is not a transparent fallback: it changes the bound provider and can retire live session edits (#8289). Keep byte matches distinct from strings and runtime use. Both sessions closed and the unexecuted target's hash was unchanged.

@morluto commented on GitHub (Oct 31, 2026): Provider-aware recovery control with real REA `b43c329` MCP, Ghidra 12.1.4 and Kuna 1.986 on the original source-owned short/encoded-string Mach-O: - The advertised `search_strings` description says that at a known address `read_bytes` and `xrefs` inspect bytes and references. `binary_session(detail:"complete")` reports all three relevant tools (`read_bytes`, `search_bytes`, `xrefs`) available for Ghidra, but reports the first two `provider_missing` for Kuna while `xrefs` remains available. - `search_strings({pattern:"日本"})` returned no rows under either provider with their respective inventory limitations. At known address `0x3e4`, Ghidra `read_bytes` returned the six initialized bytes `e5652c670000`, and `search_bytes` found that address with file offset 996. The source fixture establishes the intended UTF-16LE interpretation. Kuna returned `capability_unavailable` for both byte operations; its exact `xrefs` returned no rows with its explicit incomplete-reference limitation. The generic catalog guidance is valid only when the selected provider offers the operations; discovery already exposes that distinction, so this is an agent recovery/discoverability question rather than a claim that Kuna promised byte reading. A switch to Ghidra can be useful when explicitly available and selected, but it is not a transparent fallback: it changes the bound provider and can retire live session edits (#8289). Keep byte matches distinct from strings and runtime use. Both sessions closed and the unexecuted target's hash was unchanged.

bsakaguchi commented

2026-11-01 21:11:39 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Additional encoding and project-selection control on macOS arm64, with the public MCP tools, Ghidra 12.1.4 and verified Kuna 1.986. The unexecuted Mach-O contains alpha-ΟΣ-beta as UTF-8 and UTF-16LE; target SHA-256 35ece1208b6299e6d4c426da84439c15a88a08caecc088d1f0fda39f33310bd3 was unchanged.

Default Ghidra import returned two UTF-8 byte occurrences as defined US-ASCII strings with replacement characters, and did not list the non-Latin UTF-16LE text. search_strings and trace_feature for Σ returned no matches, while preserving the existing limitation that an empty defined-string search does not establish absence from the artifact. Ghidra search_bytes with the independently known full UTF-8 and UTF-16LE sequences found the respective byte locations in initialized memory. Byte observations alone do not prove text interpretation or use.
Kuna's inventory returned the UTF-8 value at two addresses and a shorter alpha- UTF-16 fragment; search_strings and trace_feature matched both UTF-8 rows with case-insensitive Σ, σ, or ς. Case-sensitive σ did not match the uppercase text. This is consistent with the reported Latin-range UTF-16 scanning limit.
For a project control, a headless Ghidra script defined one UTF-8 and the UTF-16LE occurrence with their actual string data types in a caller-owned project. Opening that selected project in REA yielded two exact alpha-ΟΣ-beta inventory rows; both search and trace matched the three insensitive sigma forms, and the sensitive controls behaved as expected. A second UTF-8 occurrence remained US-ASCII with replacement characters. REA reported the selected project provenance; all 10 source-project files were byte-identical before and after the REA session.

This suggests that discovery guidance should cover misdecoded defined data as well as absent or undefined strings, while preserving the provider's recorded encoding and the difference between an explicit byte interpretation and an existing project definition. It does not establish the intended source-level encoding for arbitrary binaries. The tested REA build was #8368 head 0ebfcabf; successful MCP outputs were checked against full advertised schemas. One fixture and host are not a prevalence or general encoding-coverage measurement.

Follow-up: Ghidra search_strings drops the encoding and length even for correctly defined strings; this distinct producer-boundary defect and its trace_feature impact are tracked in #8374.

@morluto commented on GitHub (Oct 31, 2026): Additional encoding and project-selection control on macOS arm64, with the public MCP tools, Ghidra 12.1.4 and verified Kuna 1.986. The unexecuted Mach-O contains `alpha-ΟΣ-beta` as UTF-8 and UTF-16LE; target SHA-256 `35ece1208b6299e6d4c426da84439c15a88a08caecc088d1f0fda39f33310bd3` was unchanged. - Default Ghidra import returned two UTF-8 byte occurrences as defined `US-ASCII` strings with replacement characters, and did not list the non-Latin UTF-16LE text. `search_strings` and `trace_feature` for `Σ` returned no matches, while preserving the existing limitation that an empty defined-string search does not establish absence from the artifact. Ghidra `search_bytes` with the independently known full UTF-8 and UTF-16LE sequences found the respective byte locations in initialized memory. Byte observations alone do not prove text interpretation or use. - Kuna's inventory returned the UTF-8 value at two addresses and a shorter `alpha-` UTF-16 fragment; `search_strings` and `trace_feature` matched both UTF-8 rows with case-insensitive `Σ`, `σ`, or `ς`. Case-sensitive `σ` did not match the uppercase text. This is consistent with the reported Latin-range UTF-16 scanning limit. - For a project control, a headless Ghidra script defined one UTF-8 and the UTF-16LE occurrence with their actual string data types in a caller-owned project. Opening that selected project in REA yielded two exact `alpha-ΟΣ-beta` inventory rows; both search and trace matched the three insensitive sigma forms, and the sensitive controls behaved as expected. A second UTF-8 occurrence remained US-ASCII with replacement characters. REA reported the selected project provenance; all 10 source-project files were byte-identical before and after the REA session. This suggests that discovery guidance should cover *misdecoded defined data* as well as absent or undefined strings, while preserving the provider's recorded encoding and the difference between an explicit byte interpretation and an existing project definition. It does not establish the intended source-level encoding for arbitrary binaries. The tested REA build was #8368 head `0ebfcabf`; successful MCP outputs were checked against full advertised schemas. One fixture and host are not a prevalence or general encoding-coverage measurement. Follow-up: Ghidra `search_strings` drops the encoding and length even for correctly defined strings; this distinct producer-boundary defect and its `trace_feature` impact are tracked in #8374.

bsakaguchi referenced this issue

2026-11-01 21:47:21 +00:00

[PR #6495] [MERGED] fix(entropy): name offset when the byte window starts past EOF #7368

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 16:33:09 +00:00