morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8360] [Design] Investigate explicit non-mutating decode of an alternate instruction stream #6564

New issue

Open

opened 2026-11-01 21:12:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:16 +00:00

Owner

Originally created by @morluto on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8360

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Should REA offer an explicitly requested, bounded, non-mutating decode at a selected address when an analyst needs to inspect an alternative instruction stream that the provider's listing did not retain? If so, should that be a mode of the existing instruction-inspection workflow?

Context and evidence

On REA main d54a4661d, I built an x86-64 Mach-O dylib containing the following unexecuted bytes at 0x2d0:

85 ff 74 01 b8 b8 2a 00 00 00 c3 c3

The conditional branch at 0x2d2 names 0x2d5, inside the byte span of the fall-through instruction starting at 0x2d4. Independently requested decodes establish two byte-valid static alternatives: 0x2d4: b8 b8 2a 00 00 (MOV EAX,0x2ab8), and 0x2d5: b8 2a 00 00 00 (MOV EAX,0x2a), followed by RET at 0x2da. These are conditional static possibilities, not observed runtime execution.

Real REA/Kuna 1.986 procedure_assembly(_audit_overlap) returned the first stream and included the branch to 0x2d5, but no row starting there or a specific overlap caveat. Its separate pseudocode contained an overlapbranch warning and truncated the fall-through. inspect_native_instruction and read_bytes were explicitly unavailable with Kuna selected.
Real REA/Ghidra 12.1.4 inspect_native_instruction(0x2d5) returned not-in-listing and explicitly said no decode was attempted; read_bytes returned the authored 12 bytes. Ghidra did not create the _audit_overlap procedure in this fresh import, so a procedure lookup did not resolve it. The fixed-function control did resolve.
In a separate headless Ghidra import of the same target, new PseudoDisassembler(currentProgram).disassemble(address) produced both byte sequences above. Ghidra's shipped API documentation states this operation does not affect the program. This is one direct Ghidra observation, not an existing REA feature or proof of general decoder correctness.
Direct Kuna's disassemble --follow did not produce the overlapping target on this fixture; it emitted contradictory recovery notes. I reported the reproduced upstream behavior separately: https://github.com/Noelo-Lab/kuna/issues/1363. An REA design should not simply forward --follow as a completeness guarantee.

The target digest was unchanged across REA calls; successful MCP responses were validated with the advertised compact schemas. The fixture and standalone Ghidra project were disposable.

Expected interaction with targets

Read explicitly selected bytes and provider decoder context. No target execution or mutation of the target or selected Ghidra project is proposed. Scope and provider admission still need maintainer review.

Boundary details

Keep the exact requested address, address space, source bytes, architecture and mode/context with each candidate. A decode does not establish function identity, listing membership, reachability or runtime execution. Preserve overlapping alternatives; do not replace a listed instruction or silently decode uninitialized, unmapped, ambiguous-context or non-executable data. Existing not-in-listing, not-instruction-boundary, data and unreadable results must retain their meaning. Bound the requested range and output.

Options and tradeoffs

Compare the existing listing-only result plus explicit caller-side recovery; an opt-in candidate decode under inspect_native_instruction; or another task-level address-context workflow. Evaluate the result and provenance before proposing integration. Do not add a provider-specific CLI wrapper merely for API parity.

Verification available so far

I have observed results. One x86-64 Mach-O and the exact inputs above were exercised; broad architectures, ARM/Thumb mode, overlays, selected edited projects, instruction context changes, cancellation and CLI parity are untested. The source file's SHA-256 is bae32cfbc3dab050235ece58c19cf38bf99d0ea198d1511c9e3512a48199216c.

References

Related #8241 concerns finding the containing listed instruction without silently moving a search hit; #7490 concerns correct not-in-listing status; #8192 concerns analyst-selected function entries. Ghidra's PseudoDisassembler is a possible implementation primitive, not a prescribed public contract.

@gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Additional AArch64 boundary control

On REA 9ebbe67e05af08b567e215340da993c08dae378f, macOS arm64, a self-built AArch64 Mach-O (SHA-256 ae0f2a349b22332113ab60ab4c78024eb18153c76bdeefb01c4b1d89a26c0003) was inspected through real compact-schema MCP with Ghidra 12.1.4 and Kuna 1.989. Ghidra returned decoded for 0x100000328 (ff4300d1, sub) and 0x10000032c (e00f00b9, str), not-instruction-boundary for the interior byte 0x100000329, and outside-memory for 0x200000000. REA's inspect_native_instruction explicitly reported unavailable with Kuna selected.

Direct kuna disassemble at 0x100000328 --addr --count 2 --json returned the same two instruction bytes; the same command at 0x100000329 --addr --count 1 --json emitted a one-byte .byte 0x43 row. This is a raw decoding result, not evidence of a listed instruction boundary or runtime execution. The target digest remained unchanged. Only this fixture and MCP output profile were checked; this does not establish general ARM mode or decoder behavior.

Kuna 1.990 overlap control and upstream status

The earlier --follow failure described above was fixed upstream in Kuna #1370, closing #1363. On macOS arm64 I rebuilt the x86-64 Mach-O fixture (entry 0x2b0, SHA-256 75423e5e9b6514152915e72b04f6b7c01742af2223ed03eea435524de12e6d88). Direct Kuna 1.990 disassemble --follow --json reported both 0x2b4: b8b82a0000 and 0x2b5: b82a000000, with an explicit note that rows overlap. An independent direct non-follow decode at 0x2b5 returned the second instruction and RET at 0x2ba.

In compact and full public MCP sessions on REA's 1.990 validation branch, procedure_assembly for 0x2b0 returned the straight-line listing and branch to 0x2b5 but no row at 0x2b5; read_function_instructions and inspect_native_instruction returned capability_unavailable for Kuna. The target was not executed and its digest stayed unchanged. This is one fixture and does not imply that an alternate decode proves reachability or runtime execution, or that exposing Kuna's full CLI is the right REA design.

Pseudocode and dossier warning in 1.990

On the same 1.990 fixture, separate fresh compact and full MCP sessions requested cold procedure_pseudo_code, complete analyze_function, and pseudocode again after the project export. Each result retained Kuna's inline overlapbranch warning naming 0x2b5 and saying the fall-through was truncated; the dossier also included the straight-line assembly without the alternate row. The cold result disclosed its single-function source, and the completed-project result did not carry that source limitation. The warning is provider text rather than a separately typed overlap record; this test did not establish how an autonomous agent interprets it or whether every overlap generates a warning. The fixture hash remained unchanged.

Explicit cross-provider handoff on 1.990 fixture

On the 1.990 overlap fixture above, compact and full MCP sessions first obtained Kuna pseudocode containing its overlap warning, then explicitly reopened the same target with Ghidra 12.1.4. inspect_native_instruction(0x2b5) returned not-in-listing and explicitly said no decode was attempted; read_bytes(0x2b0, 12) returned 85ff7401b8b82a000000c3c3. The original Kuna pseudocode Evidence was still retained exactly in the same investigation bundle after the switch. Direct Kuna disassemble at 0x2b5 separately produced the alternate decode described above. The source digest remained unchanged. This is a manual cross-provider handoff, not a typed alternate decode in REA or proof of instruction reachability.

Direct Ghidra candidate decode

On the same x86-64 overlap fixture, a separate Ghidra 12.1.4 headless import ran an owned post-script calling new PseudoDisassembler(currentProgram).disassemble(toAddr("2b5")). Ghidra returned MOV EAX,0x2a, bytes b82a000000, length 5; getInstructionAt(0x2b5) was null both before and after the call. This agrees on the candidate bytes with direct Kuna's independent decode. The disposable project was deleted, the unexecuted target retained SHA-256 75423e5e9b6514152915e72b04f6b7c01742af2223ed03eea435524de12e6d88, and no REA capability was added.

This establishes a possible non-listing-mutating Ghidra primitive on one x86 fixture, not general non-mutation of every project property, correctness across architectures or modes, reachability, or runtime execution. A task-level contract must distinguish such a candidate from the observed listing, expose requested address and decoder context, preserve failures and resource bounds, and avoid silently rewriting the database.

Candidate decoder address boundaries

A direct Ghidra 12.1.4 headless follow-up on the same fixture asked PseudoDisassembler.disassemble at three exact addresses. The unlisted 0x2b5 decoded MOV EAX,0x2a; 0x2b6, an interior byte of that candidate, also decoded a different plausible SUB instruction. Neither became a listed instruction. At unmapped 0x200000000, the decoder returned null. The first probe incorrectly dereferenced that null and raised NullPointerException in the probe itself; a corrected second run distinguished the null result from an exception. Both temporary projects were deleted.

This reinforces that a successful candidate decode alone cannot choose an instruction boundary or establish execution. A REA workflow would need to validate mapped/readable input before invoking the primitive and return an honest typed failure; the null observation is specific to this fixture and Ghidra version, not a general specification of Ghidra's API.

Candidate decode from non-executable data

A separate direct Ghidra 12.1.4 headless control used the owned arm64 pointer fixture from #8510. Its slot at 0x100004000 was in a non-executable block and identified as data in the REA control. PseudoDisassembler.disassemble nevertheless returned udf 0x3c0 from loaded bytes c0030000 with setRespectExecuteFlag(false) and with setRespectExecuteFlag(true). This tests those two direct calls, not the behavior of Ghidra's other subroutine-validation methods. The disposable project was deleted and the target was not executed.

A future caller-selected candidate decode should report memory permissions and loaded-byte provenance and define its policy for non-executable regions explicitly; the decoder producing a plausible instruction does not make data executable or prove code. Do not assume the setRespectExecuteFlag setting alone enforces that policy for the tested direct disassemble call.

Ghidra source interpretation

The Ghidra 12.1.4 PseudoDisassembler source documents single-instruction decoding as not changing the program. Source inspection shows the disassemble(Address) path delegates to a one-instruction pseudo decode; the respectExecuteFlag field is consulted in validation paths rather than that direct method. Its setter documentation is broad, but the observed direct decode matches the implementation. This is not an upstream defect report: REA should own its admission policy and cannot infer execute enforcement from the setter alone. The local controls above remain observations on the listed fixtures and host.

Originally created by @morluto on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8360 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Should REA offer an explicitly requested, bounded, non-mutating decode at a selected address when an analyst needs to inspect an alternative instruction stream that the provider's listing did not retain? If so, should that be a mode of the existing instruction-inspection workflow? ### Context and evidence On REA main `d54a4661d`, I built an x86-64 Mach-O dylib containing the following unexecuted bytes at `0x2d0`: ```text 85 ff 74 01 b8 b8 2a 00 00 00 c3 c3 ``` The conditional branch at `0x2d2` names `0x2d5`, inside the byte span of the fall-through instruction starting at `0x2d4`. Independently requested decodes establish two byte-valid static alternatives: `0x2d4: b8 b8 2a 00 00` (`MOV EAX,0x2ab8`), and `0x2d5: b8 2a 00 00 00` (`MOV EAX,0x2a`), followed by `RET` at `0x2da`. These are conditional static possibilities, not observed runtime execution. - Real REA/Kuna 1.986 `procedure_assembly(_audit_overlap)` returned the first stream and included the branch to `0x2d5`, but no row starting there or a specific overlap caveat. Its separate pseudocode contained an `overlapbranch` warning and truncated the fall-through. `inspect_native_instruction` and `read_bytes` were explicitly unavailable with Kuna selected. - Real REA/Ghidra 12.1.4 `inspect_native_instruction(0x2d5)` returned `not-in-listing` and explicitly said no decode was attempted; `read_bytes` returned the authored 12 bytes. Ghidra did not create the `_audit_overlap` procedure in this fresh import, so a procedure lookup did not resolve it. The fixed-function control did resolve. - In a separate headless Ghidra import of the same target, `new PseudoDisassembler(currentProgram).disassemble(address)` produced both byte sequences above. Ghidra's shipped API documentation states this operation does not affect the program. This is one direct Ghidra observation, not an existing REA feature or proof of general decoder correctness. - Direct Kuna's `disassemble --follow` did not produce the overlapping target on this fixture; it emitted contradictory recovery notes. I reported the reproduced upstream behavior separately: https://github.com/Noelo-Lab/kuna/issues/1363. An REA design should not simply forward `--follow` as a completeness guarantee. The target digest was unchanged across REA calls; successful MCP responses were validated with the advertised compact schemas. The fixture and standalone Ghidra project were disposable. ### Expected interaction with targets Read explicitly selected bytes and provider decoder context. No target execution or mutation of the target or selected Ghidra project is proposed. Scope and provider admission still need maintainer review. ### Boundary details Keep the exact requested address, address space, source bytes, architecture and mode/context with each candidate. A decode does not establish function identity, listing membership, reachability or runtime execution. Preserve overlapping alternatives; do not replace a listed instruction or silently decode uninitialized, unmapped, ambiguous-context or non-executable data. Existing `not-in-listing`, `not-instruction-boundary`, data and unreadable results must retain their meaning. Bound the requested range and output. ### Options and tradeoffs Compare the existing listing-only result plus explicit caller-side recovery; an opt-in candidate decode under `inspect_native_instruction`; or another task-level address-context workflow. Evaluate the result and provenance before proposing integration. Do not add a provider-specific CLI wrapper merely for API parity. ### Verification available so far I have observed results. One x86-64 Mach-O and the exact inputs above were exercised; broad architectures, ARM/Thumb mode, overlays, selected edited projects, instruction context changes, cancellation and CLI parity are untested. The source file's SHA-256 is `bae32cfbc3dab050235ece58c19cf38bf99d0ea198d1511c9e3512a48199216c`. ### References Related #8241 concerns finding the containing *listed* instruction without silently moving a search hit; #7490 concerns correct `not-in-listing` status; #8192 concerns analyst-selected function entries. Ghidra's `PseudoDisassembler` is a possible implementation primitive, not a prescribed public contract. @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Additional AArch64 boundary control On REA `9ebbe67e05af08b567e215340da993c08dae378f`, macOS arm64, a self-built AArch64 Mach-O (SHA-256 `ae0f2a349b22332113ab60ab4c78024eb18153c76bdeefb01c4b1d89a26c0003`) was inspected through real compact-schema MCP with Ghidra 12.1.4 and Kuna 1.989. Ghidra returned `decoded` for `0x100000328` (`ff4300d1`, `sub`) and `0x10000032c` (`e00f00b9`, `str`), `not-instruction-boundary` for the interior byte `0x100000329`, and `outside-memory` for `0x200000000`. REA's `inspect_native_instruction` explicitly reported unavailable with Kuna selected. Direct `kuna disassemble` at `0x100000328 --addr --count 2 --json` returned the same two instruction bytes; the same command at `0x100000329 --addr --count 1 --json` emitted a one-byte `.byte 0x43` row. This is a raw decoding result, not evidence of a listed instruction boundary or runtime execution. The target digest remained unchanged. Only this fixture and MCP output profile were checked; this does not establish general ARM mode or decoder behavior. ### Kuna 1.990 overlap control and upstream status The earlier `--follow` failure described above was fixed upstream in [Kuna #1370](https://github.com/Noelo-Lab/kuna/pull/1370), closing [#1363](https://github.com/Noelo-Lab/kuna/issues/1363). On macOS arm64 I rebuilt the x86-64 Mach-O fixture (entry `0x2b0`, SHA-256 `75423e5e9b6514152915e72b04f6b7c01742af2223ed03eea435524de12e6d88`). Direct Kuna 1.990 `disassemble --follow --json` reported both `0x2b4: b8b82a0000` and `0x2b5: b82a000000`, with an explicit note that rows overlap. An independent direct non-follow decode at `0x2b5` returned the second instruction and `RET` at `0x2ba`. In compact and full public MCP sessions on REA's 1.990 validation branch, `procedure_assembly` for `0x2b0` returned the straight-line listing and branch to `0x2b5` but no row at `0x2b5`; `read_function_instructions` and `inspect_native_instruction` returned `capability_unavailable` for Kuna. The target was not executed and its digest stayed unchanged. This is one fixture and does not imply that an alternate decode proves reachability or runtime execution, or that exposing Kuna's full CLI is the right REA design. ### Pseudocode and dossier warning in 1.990 On the same 1.990 fixture, separate fresh compact and full MCP sessions requested cold `procedure_pseudo_code`, complete `analyze_function`, and pseudocode again after the project export. Each result retained Kuna's inline `overlapbranch` warning naming `0x2b5` and saying the fall-through was truncated; the dossier also included the straight-line assembly without the alternate row. The cold result disclosed its single-function source, and the completed-project result did not carry that source limitation. The warning is provider text rather than a separately typed overlap record; this test did not establish how an autonomous agent interprets it or whether every overlap generates a warning. The fixture hash remained unchanged. ### Explicit cross-provider handoff on 1.990 fixture On the 1.990 overlap fixture above, compact and full MCP sessions first obtained Kuna pseudocode containing its overlap warning, then explicitly reopened the same target with Ghidra 12.1.4. `inspect_native_instruction(0x2b5)` returned `not-in-listing` and explicitly said no decode was attempted; `read_bytes(0x2b0, 12)` returned `85ff7401b8b82a000000c3c3`. The original Kuna pseudocode Evidence was still retained exactly in the same investigation bundle after the switch. Direct Kuna `disassemble` at `0x2b5` separately produced the alternate decode described above. The source digest remained unchanged. This is a manual cross-provider handoff, not a typed alternate decode in REA or proof of instruction reachability. ### Direct Ghidra candidate decode On the same x86-64 overlap fixture, a separate Ghidra 12.1.4 headless import ran an owned post-script calling `new PseudoDisassembler(currentProgram).disassemble(toAddr("2b5"))`. Ghidra returned `MOV EAX,0x2a`, bytes `b82a000000`, length 5; `getInstructionAt(0x2b5)` was null both before and after the call. This agrees on the candidate bytes with direct Kuna's independent decode. The disposable project was deleted, the unexecuted target retained SHA-256 `75423e5e9b6514152915e72b04f6b7c01742af2223ed03eea435524de12e6d88`, and no REA capability was added. This establishes a possible non-listing-mutating Ghidra primitive on one x86 fixture, not general non-mutation of every project property, correctness across architectures or modes, reachability, or runtime execution. A task-level contract must distinguish such a candidate from the observed listing, expose requested address and decoder context, preserve failures and resource bounds, and avoid silently rewriting the database. ### Candidate decoder address boundaries A direct Ghidra 12.1.4 headless follow-up on the same fixture asked `PseudoDisassembler.disassemble` at three exact addresses. The unlisted `0x2b5` decoded `MOV EAX,0x2a`; `0x2b6`, an interior byte of that candidate, also decoded a different plausible `SUB` instruction. Neither became a listed instruction. At unmapped `0x200000000`, the decoder returned null. The first probe incorrectly dereferenced that null and raised `NullPointerException` in the probe itself; a corrected second run distinguished the null result from an exception. Both temporary projects were deleted. This reinforces that a successful candidate decode alone cannot choose an instruction boundary or establish execution. A REA workflow would need to validate mapped/readable input before invoking the primitive and return an honest typed failure; the null observation is specific to this fixture and Ghidra version, not a general specification of Ghidra's API. ### Candidate decode from non-executable data A separate direct Ghidra 12.1.4 headless control used the owned arm64 pointer fixture from #8510. Its slot at `0x100004000` was in a non-executable block and identified as data in the REA control. `PseudoDisassembler.disassemble` nevertheless returned `udf 0x3c0` from loaded bytes `c0030000` with `setRespectExecuteFlag(false)` **and** with `setRespectExecuteFlag(true)`. This tests those two direct calls, not the behavior of Ghidra's other subroutine-validation methods. The disposable project was deleted and the target was not executed. A future caller-selected candidate decode should report memory permissions and loaded-byte provenance and define its policy for non-executable regions explicitly; the decoder producing a plausible instruction does not make data executable or prove code. Do not assume the `setRespectExecuteFlag` setting alone enforces that policy for the tested direct `disassemble` call. ### Ghidra source interpretation The [Ghidra 12.1.4 `PseudoDisassembler` source](https://github.com/NationalSecurityAgency/ghidra/blob/Ghidra_12.1.4_build/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/app/util/PseudoDisassembler.java) documents single-instruction decoding as not changing the program. Source inspection shows the `disassemble(Address)` path delegates to a one-instruction pseudo decode; the `respectExecuteFlag` field is consulted in validation paths rather than that direct method. Its setter documentation is broad, but the observed direct decode matches the implementation. This is not an upstream defect report: REA should own its admission policy and cannot infer execute enforcement from the setter alone. The local controls above remain observations on the listed fixtures and host.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:16 +00:00

bsakaguchi referenced this issue

2026-11-03 07:16:59 +00:00

[GH-ISSUE #7621] [Bug] environment-compare does not disclose when a step's difference list is truncated #4549

bsakaguchi added

enhancement

and removed

enhancement