morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8563] [Feature] Investigate byte signatures across adjacent Ghidra blocks #6654

New issue

Closed

opened 2026-11-01 21:12:58 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:58 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8563

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

search_bytes deliberately searches Ghidra memory blocks independently; its tool description and returned limitations disclose that patterns never cross block boundaries. This can miss a byte sequence that is contiguous and readable in the analyzed address space even when both adjacent blocks were searched and the page says complete: true.

Use cases

An analyst searches a signature whose location is unknown. Executable section or memory-block boundaries are analysis metadata and can fall inside the sought sequence; asking the caller to predict the boundary defeats a whole-range search.

Desired outcome and success criteria

Investigate supporting signatures across adjacent initialized blocks in the same address space, or expose a bounded way to identify and check eligible boundary windows. Preserve pagination, overlapping matches, exact source coverage and explicit skipped/unknown regions. Do not search across unmapped gaps, uninitialized blocks, different spaces or noncontiguous addresses. Clarify what complete covers; do not change it to imply binary-wide absence.

Expected interaction with targets

Read existing files and launch the caller-supplied Ghidra provider for static analysis. No target execution or mutation is proposed.

Verification available so far

I have observed results.

Verification details

On macOS arm64, Node 24.22.1, Ghidra 12.1.4, and REA 1662c037b900d0cc12346cffdb32bbbc69e9dc4b, an owned unexecuted Mach-O contained contiguous 32-byte __first and __second sections. The four-byte sequence deadbeef straddled their boundary. In a compact stdio MCP session:

read_bytes at 0x10000401e, length 4, returned deadbeef, complete: true.
search_bytes for de ad be ef bounded to 0x100004000–0x10000403f returned no matches, complete: true, and two searched 32-byte blocks. The returned limitation correctly said patterns never cross block boundaries.

The target SHA-256 87fa65dee009e59a8709a70a2aa731a0dc9cae47cf65d4c69292f7cad4897d93 was unchanged. This is one real-provider fixture and a known documented restriction, not a claim that the current result violates its declared contract. Independent Ghidra/headless, other formats, cursor behavior around boundaries, and agent behavior were not tested.

Alternatives considered

list_segments plus targeted read_bytes can inspect a known boundary. It requires separate calls and caller-side pattern matching at every eligible boundary, including mask and wildcard semantics. Search already provides cursor paging and coverage, making this a candidate for a task-level improvement rather than a new raw provider wrapper.

Possible direction

Consider bounded carryover of at most pattern-length-minus-one bytes across adjacent eligible blocks, without duplicate matches or altering per-block permissions/coverage semantics. This is a design option, not a prescribed implementation. Test overlaps and cursors at boundaries, explicit range endpoints, unreadable/unknown gaps, non-loaded and overlay rules, and resource limits before integration.

References

Ghidra masked byte-search contract
Related large-read design: #8560

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Split-signature recovery control

On REA #8817 head 50ba6cb22a242a60f7e4bd12fc74c05ec91191ab, Ghidra 12.1.4/JDK 21 and macOS arm64, a separately built Mach-O placed de ad at the end of a 32-byte __first section and be ef at the beginning of adjacent __second. In both compact and full MCP, the bounded full-pattern search returned no matches and complete: true with two searched blocks and the existing no-cross-block limitation. Searches for each two-byte half separately returned 0x10000401e and 0x100004020; a four-byte read_bytes at the first address returned deadbeef complete. Independent llvm-objdump also showed the two 32-byte sections at 0x100004000 and 0x100004020. The artifact SHA-256 1119d01b0d7d90029c173c907ab309f1e3936d429b5d2776ebf55567a239108a remained unchanged; it was not executed. This is a manual known-boundary recovery, not an automated or complete workaround for arbitrary masks, multiple matches, gaps, or paged searches.

Independent Ghidra search control

For the split-signature fixture above, I imported the same Mach-O into a separate disposable Ghidra 12.1.4 headless project with -noanalysis and ran a Java GhidraScript calling currentProgram.getMemory().findBytes for de ad be ef, with candidate-start bounds 0x100004000 through 0x10000403c. Ghidra returned 0x10000401e, crossing the two adjacent initialized sections; the input SHA-256 remained unchanged. This establishes that Ghidra's primitive can find this specific cross-block pattern when given a range spanning both blocks. Source inspection shows REA intentionally bounds each findBytes call to one memory block. It does not establish safe behavior across uninitialized or non-loaded blocks, overlays, address spaces, wildcard masks, pagination or every Ghidra version; those still require explicit eligibility and coverage rules.

Ghidra masked search and an uninitialized gap

In a disposable Ghidra 12.1.4 headless project, a Java script created two adjacent initialized blocks containing de ad | 11 22 be ef and searched with bytes de ad 00 00 be ef, mask ff ff 00 00 ff ff. Memory.findBytes returned the first block's start. In a separate synthetic region the script created initialized de ad, an adjacent two-byte uninitialized block, and initialized be ef; both the exact zero-filled pattern and the same wildcard-mask search returned null. The test used a rolled-back project transaction; it did not modify or execute the source Mach-O. This is one positive and one negative control, not a guarantee about all gaps, overlays, address spaces or Ghidra versions. REA still needs explicit eligibility, skipped-coverage and pagination rules instead of relying solely on this observed behavior.

Adjacent virtual blocks with noncontiguous file offsets

A source-generated ELF64 x86-64 maps four bytes at 0x401000 from file offset 0x1000 and the adjacent four bytes at 0x401004 from offset 0x3000. On PR #8829 head 33f2d251e, real Ghidra 12.1.4 read_bytes returned 1122334455667788 across the virtual boundary, and address-to-file-offset queries confirmed the two distinct offsets. In both compact/full MCP, a bounded search_bytes for that same eight-byte sequence returned no matches, reported each four-byte block searched, and complete: true under the existing documented block-local limitation. The unexecuted target SHA-256 e66ea8131ec8479d0fbfbdc2d57ff30137700d1956879602ff6b7bd289d54f3d remained unchanged.

If cross-block matching is introduced, the match's existing starting file offset cannot describe the rest of this sequence as a contiguous file span. Preserve the actual memory blocks and their mapping separately, keep file_offset scoped to its start, and test cross-boundary pages and masks. This control does not test implementation of the proposed change or establish runtime ELF loading.

Uncommitted contiguous-run prototype

An isolated, uncommitted prototype groups adjacent eligible initialized Ghidra blocks into search runs while retaining per-block coverage and the match's starting block and file offset. On real Ghidra 12.1.4 compact/full MCP, it found the eight-byte sequence spanning adjacent virtual blocks backed by file offsets 0x1000 and 0x3000. A second owned fixture containing eight aa bytes split over those blocks produced all six overlapping three-byte matches in three two-match pages, including the two boundary-crossing starts; both an exact pattern and aa ?? aa passed. Restricting the inclusive end to 0x401005 produced the first four matches. A separate fixture with a one-byte unmapped gap returned no cross-gap wildcard match and a short read_bytes result. Targets remained unchanged and were not executed.

This is a design experiment, not an integrated patch or complete verification. Its existing catalog, limitation text and profile still describe block-local search and must be updated before use. It still needs bounded/unbounded overlay and unloaded-block controls, initialized/uninitialized boundaries, multi-space ordering, cursor mismatch and snapshot replay, cancellation/cleanup, production tests and schema validation. In particular, do not interpret the starting file offset as a contiguous file span. No upstream Ghidra change is implied.

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8563 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation `search_bytes` deliberately searches Ghidra memory blocks independently; its tool description and returned limitations disclose that patterns never cross block boundaries. This can miss a byte sequence that is contiguous and readable in the analyzed address space even when both adjacent blocks were searched and the page says `complete: true`. ### Use cases An analyst searches a signature whose location is unknown. Executable section or memory-block boundaries are analysis metadata and can fall inside the sought sequence; asking the caller to predict the boundary defeats a whole-range search. ### Desired outcome and success criteria Investigate supporting signatures across adjacent initialized blocks in the same address space, or expose a bounded way to identify and check eligible boundary windows. Preserve pagination, overlapping matches, exact source coverage and explicit skipped/unknown regions. Do not search across unmapped gaps, uninitialized blocks, different spaces or noncontiguous addresses. Clarify what `complete` covers; do not change it to imply binary-wide absence. ### Expected interaction with targets Read existing files and launch the caller-supplied Ghidra provider for static analysis. No target execution or mutation is proposed. ### Verification available so far I have observed results. ### Verification details On macOS arm64, Node 24.22.1, Ghidra 12.1.4, and REA `1662c037b900d0cc12346cffdb32bbbc69e9dc4b`, an owned unexecuted Mach-O contained contiguous 32-byte `__first` and `__second` sections. The four-byte sequence `deadbeef` straddled their boundary. In a compact stdio MCP session: - `read_bytes` at `0x10000401e`, length 4, returned `deadbeef`, `complete: true`. - `search_bytes` for `de ad be ef` bounded to `0x100004000`–`0x10000403f` returned no matches, `complete: true`, and two searched 32-byte blocks. The returned limitation correctly said patterns never cross block boundaries. The target SHA-256 `87fa65dee009e59a8709a70a2aa731a0dc9cae47cf65d4c69292f7cad4897d93` was unchanged. This is one real-provider fixture and a known documented restriction, not a claim that the current result violates its declared contract. Independent Ghidra/headless, other formats, cursor behavior around boundaries, and agent behavior were not tested. ### Alternatives considered `list_segments` plus targeted `read_bytes` can inspect a known boundary. It requires separate calls and caller-side pattern matching at every eligible boundary, including mask and wildcard semantics. Search already provides cursor paging and coverage, making this a candidate for a task-level improvement rather than a new raw provider wrapper. ### Possible direction Consider bounded carryover of at most pattern-length-minus-one bytes across adjacent eligible blocks, without duplicate matches or altering per-block permissions/coverage semantics. This is a design option, not a prescribed implementation. Test overlaps and cursors at boundaries, explicit range endpoints, unreadable/unknown gaps, non-loaded and overlay rules, and resource limits before integration. ### References - [Ghidra masked byte-search contract](https://github.com/morluto/rea/blob/1662c037b900d0cc12346cffdb32bbbc69e9dc4b/docs/ghidra-provider.md#masked-byte-search) - Related large-read design: #8560 cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Split-signature recovery control On REA #8817 head `50ba6cb22a242a60f7e4bd12fc74c05ec91191ab`, Ghidra 12.1.4/JDK 21 and macOS arm64, a separately built Mach-O placed `de ad` at the end of a 32-byte `__first` section and `be ef` at the beginning of adjacent `__second`. In both compact and full MCP, the bounded full-pattern search returned no matches and `complete: true` with two searched blocks and the existing no-cross-block limitation. Searches for each two-byte half separately returned `0x10000401e` and `0x100004020`; a four-byte `read_bytes` at the first address returned `deadbeef` complete. Independent `llvm-objdump` also showed the two 32-byte sections at `0x100004000` and `0x100004020`. The artifact SHA-256 `1119d01b0d7d90029c173c907ab309f1e3936d429b5d2776ebf55567a239108a` remained unchanged; it was not executed. This is a manual known-boundary recovery, not an automated or complete workaround for arbitrary masks, multiple matches, gaps, or paged searches. ### Independent Ghidra search control For the split-signature fixture above, I imported the same Mach-O into a separate disposable Ghidra 12.1.4 headless project with `-noanalysis` and ran a Java `GhidraScript` calling `currentProgram.getMemory().findBytes` for `de ad be ef`, with candidate-start bounds `0x100004000` through `0x10000403c`. Ghidra returned `0x10000401e`, crossing the two adjacent initialized sections; the input SHA-256 remained unchanged. This establishes that Ghidra's primitive can find this specific cross-block pattern when given a range spanning both blocks. Source inspection shows REA intentionally bounds each `findBytes` call to one memory block. It does not establish safe behavior across uninitialized or non-loaded blocks, overlays, address spaces, wildcard masks, pagination or every Ghidra version; those still require explicit eligibility and coverage rules. ### Ghidra masked search and an uninitialized gap In a disposable Ghidra 12.1.4 headless project, a Java script created two adjacent initialized blocks containing `de ad | 11 22 be ef` and searched with bytes `de ad 00 00 be ef`, mask `ff ff 00 00 ff ff`. `Memory.findBytes` returned the first block's start. In a separate synthetic region the script created initialized `de ad`, an adjacent two-byte **uninitialized** block, and initialized `be ef`; both the exact zero-filled pattern and the same wildcard-mask search returned null. The test used a rolled-back project transaction; it did not modify or execute the source Mach-O. This is one positive and one negative control, not a guarantee about all gaps, overlays, address spaces or Ghidra versions. REA still needs explicit eligibility, skipped-coverage and pagination rules instead of relying solely on this observed behavior. ### Adjacent virtual blocks with noncontiguous file offsets A source-generated ELF64 x86-64 maps four bytes at `0x401000` from file offset `0x1000` and the adjacent four bytes at `0x401004` from offset `0x3000`. On PR #8829 head `33f2d251e`, real Ghidra 12.1.4 `read_bytes` returned `1122334455667788` across the virtual boundary, and address-to-file-offset queries confirmed the two distinct offsets. In both compact/full MCP, a bounded `search_bytes` for that same eight-byte sequence returned no matches, reported each four-byte block searched, and `complete: true` under the existing documented block-local limitation. The unexecuted target SHA-256 `e66ea8131ec8479d0fbfbdc2d57ff30137700d1956879602ff6b7bd289d54f3d` remained unchanged. If cross-block matching is introduced, the match's existing starting file offset cannot describe the rest of this sequence as a contiguous file span. Preserve the actual memory blocks and their mapping separately, keep `file_offset` scoped to its start, and test cross-boundary pages and masks. This control does not test implementation of the proposed change or establish runtime ELF loading. ### Uncommitted contiguous-run prototype An isolated, uncommitted prototype groups adjacent eligible initialized Ghidra blocks into search runs while retaining per-block coverage and the match's starting block and file offset. On real Ghidra 12.1.4 compact/full MCP, it found the eight-byte sequence spanning adjacent virtual blocks backed by file offsets `0x1000` and `0x3000`. A second owned fixture containing eight `aa` bytes split over those blocks produced all six overlapping three-byte matches in three two-match pages, including the two boundary-crossing starts; both an exact pattern and `aa ?? aa` passed. Restricting the inclusive end to `0x401005` produced the first four matches. A separate fixture with a one-byte unmapped gap returned no cross-gap wildcard match and a short `read_bytes` result. Targets remained unchanged and were not executed. This is a design experiment, not an integrated patch or complete verification. Its existing catalog, limitation text and profile still describe block-local search and must be updated before use. It still needs bounded/unbounded overlay and unloaded-block controls, initialized/uninitialized boundaries, multi-space ordering, cursor mismatch and snapshot replay, cancellation/cleanup, production tests and schema validation. In particular, do not interpret the starting file offset as a contiguous file span. No upstream Ghidra change is implied.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:58 +00:00

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 17:35:00 +00:00

bsakaguchi closed this issue