morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8192] [Design] Analyze analyst-selected function entries absent from provider inventories #6476

New issue

Open

opened 2026-11-01 21:11:27 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-11-01 21:11:27 +00:00

Owner

Originally created by @morluto on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8192

Before submitting

I searched open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

How should an analyst ask REA to analyze a suspected function entry that a provider did not discover, and how should that caller assertion affect later inventory, function boundaries, project export, Evidence, and cache identity?

Context and evidence

A self-owned, unexecuted arm64 Mach-O contains _main immediately followed by eight unlabeled instruction bytes (mov w0, #42; ret) at 0x1000002e0 (SHA-256 a6bd938bf4ae8031a33d270ca7698ee1ced5b78347a9c4fa3750b0e57f6b126c).

With neither seed nor explicit definition, REA/Ghidra 12.1.4 and REA/Kuna 1.986 both omitted that entry from list_procedures. REA/Kuna rejected procedure_pseudo_code for the address because it was not an inventory entry.
With a caller-provided Ghidra seed file containing 0x1000002e0<TAB>function<TAB>audit_hidden, a new REA/Ghidra session listed the entry and returned return 0x2a;. Evidence included the seed-file digest, the created-function outcome, and the limitation that seeds are caller assertions. This uses the existing server-start environment configuration and ephemeral Ghidra database.
Direct Kuna decompile at the address returned return 0x2a; and warned that its original inventory considered the address interior to _main. Direct Kuna functions --define-function 0x1000002e0-0x1000002e8=audit_hidden changed the inventory to separate _main and audit_hidden, with eight-byte extents each; direct decompilation with the definition returned the named result. No REA/Kuna integration of this setting was tested.

This is one authored fixture: correct entry and exclusive end are known from the test input, but a real analyst's choice is a hypothesis, not evidence that the bytes are a function or that the boundary is correct.

Follow-up: Kuna 1.988 and recovery (macOS arm64)

On the same target and current REA provider workflow, Kuna 1.988 still lists only _main; REA procedure_pseudo_code at 0x1000002e0 returns invalid_request. The message identifies the missing inventory entry, but structured recovery says inspect_input_contract and asks the caller to correct the schema, even though the address argument is well formed. resolve_containing_procedure is unavailable with Kuna. Direct Kuna still decompiles that exact address with an interior-entry warning; an interior address at _main+4 also decompiles, showing that this behavior alone does not prove function identity.

Direct functions --define-function with the known valid range splits the inventory as before. A definition beginning at unmapped 0x999999999 is also added as an eight-byte inventory entry with exit 0 and no warning, while direct decompile at that address fails as unmapped. An overlapping interior definition shortens the existing extent; an end below the start is rejected. These are observations for one fixture, not a proposal to accept or automatically apply arbitrary definitions. The unmapped-inventory question is reported upstream as Kuna #1438. Validation, conflicts, caller assertions, and real recovery must be explicit before integration.

On the same Mach-O, a fresh, separately selected REA/Ghidra 12.1.4 session returned eight initialized bytes 40058052c0035fd6 from read_bytes(0x1000002e0, 8) and inspect_native_instruction returned not-in-listing, explicitly saying it did not attempt a decode. Neither observation establishes a function. REA/Kuna 1.988 returned an empty exact-address xrefs result with its existing coverage caveat, but read_bytes and inspect_native_instruction were unavailable. These are observations from one compact stdio MCP session per provider; they do not establish address equivalence for other targets, loader states, or projects. The target digest stayed unchanged. Related: #8170 (Kuna bytes), #8360 (explicit candidate decode), and #8565 (unsupported-operation recovery).

In the Ghidra session, procedure_assembly for the address placed a missing-procedure explanation in details.issues, but its headline and recovery instead blamed invalid inputs. procedure_pseudo_code used its documented bare-hex file-offset fallback after no procedure was found and reported an unmapped file offset in the details; see #8178. The observed initialized bytes do not make that fallback an analysis-address selector. On Ghidra recovery PR #8470 head 3dc41af77, the same real compact MCP control surfaces the missing-procedure or missing-file-mapping explanation and relevant remediation directly, without schema-repair recovery; the bare numeric decompile fallback still applies as designed. The session stayed usable to close and the target digest remained unchanged.

Expected interaction with targets

Read existing target bytes, launch a caller-selected static provider, and apply caller-provided analysis assumptions. A definition can alter analysis control-flow and boundaries. No executable modification or execution is proposed.

Boundary details (optional)

Any shared workflow must preserve the exact selected target and architecture, address space, explicit entry/range, applied/rejected result, and caller assertion provenance. Distinguish a one-off analysis at an arbitrary address from creating a session-wide function definition; those have different consequences. Bind assumptions to profile/cache identity. Existing Ghidra seeds are startup-scoped and may fail if a seed conflicts with decoded instructions; Kuna's definition is per-run and affects its inventory/export. Windows Ghidra P0 remains read-only. A maintainer scope decision is needed before integrating a capability that changes analysis/control-flow interpretation.

Options and tradeoffs

Evaluate a typed, bounded analyst-hypothesis workflow over existing provider-specific mechanisms, or offer an explicitly scoped single-address analysis with clear non-inventory status. Compare against the existing Ghidra seed file; do not simply expose --define-function as an unvalidated pass-through, silently assert function identity, or claim that providers persist edits in the same way.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Test no seed, valid seed, an interior address, overlapping/conflicting definitions, invalid/unmapped bytes, multiple sessions/targets, failed application, cache isolation, and CLI/MCP provenance. State how ranges and any inferred bodies differ per provider and verify source bytes remain unchanged.

References

Related: #8166 (type/name assertions), #8165 (durable Ghidra state), and #8178 (ambiguous address versus file-offset lookup). REA baseline d48e472fa plus unrelated Kuna 1.986 baseline changes in #8187; macOS arm64.

@gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Originally created by @morluto on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8192 ### Before submitting - [x] I searched open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed How should an analyst ask REA to analyze a suspected function entry that a provider did not discover, and how should that caller assertion affect later inventory, function boundaries, project export, Evidence, and cache identity? ### Context and evidence A self-owned, unexecuted arm64 Mach-O contains `_main` immediately followed by eight unlabeled instruction bytes (`mov w0, #42; ret`) at `0x1000002e0` (SHA-256 `a6bd938bf4ae8031a33d270ca7698ee1ced5b78347a9c4fa3750b0e57f6b126c`). - With neither seed nor explicit definition, REA/Ghidra 12.1.4 and REA/Kuna 1.986 both omitted that entry from `list_procedures`. REA/Kuna rejected `procedure_pseudo_code` for the address because it was not an inventory entry. - With a caller-provided Ghidra seed file containing `0x1000002e0<TAB>function<TAB>audit_hidden`, a new REA/Ghidra session listed the entry and returned `return 0x2a;`. Evidence included the seed-file digest, the created-function outcome, and the limitation that seeds are caller assertions. This uses the existing server-start environment configuration and ephemeral Ghidra database. - Direct Kuna `decompile` at the address returned `return 0x2a;` and warned that its original inventory considered the address interior to `_main`. Direct Kuna `functions --define-function 0x1000002e0-0x1000002e8=audit_hidden` changed the inventory to separate `_main` and `audit_hidden`, with eight-byte extents each; direct decompilation with the definition returned the named result. No REA/Kuna integration of this setting was tested. This is one authored fixture: correct entry and exclusive end are known from the test input, but a real analyst's choice is a hypothesis, not evidence that the bytes are a function or that the boundary is correct. ### Follow-up: Kuna 1.988 and recovery (macOS arm64) On the same target and current REA provider workflow, Kuna 1.988 still lists only `_main`; REA `procedure_pseudo_code` at `0x1000002e0` returns `invalid_request`. The message identifies the missing inventory entry, but structured recovery says `inspect_input_contract` and asks the caller to correct the schema, even though the address argument is well formed. `resolve_containing_procedure` is unavailable with Kuna. Direct Kuna still decompiles that exact address with an interior-entry warning; an interior address at `_main+4` also decompiles, showing that this behavior alone does not prove function identity. Direct `functions --define-function` with the known valid range splits the inventory as before. A definition beginning at unmapped `0x999999999` is also added as an eight-byte inventory entry with exit 0 and no warning, while direct decompile at that address fails as unmapped. An overlapping interior definition shortens the existing extent; an end below the start is rejected. These are observations for one fixture, not a proposal to accept or automatically apply arbitrary definitions. The unmapped-inventory question is reported upstream as [Kuna #1438](https://github.com/Noelo-Lab/kuna/issues/1438). Validation, conflicts, caller assertions, and real recovery must be explicit before integration. On the same Mach-O, a fresh, separately selected REA/Ghidra 12.1.4 session returned eight initialized bytes `40058052c0035fd6` from `read_bytes(0x1000002e0, 8)` and `inspect_native_instruction` returned `not-in-listing`, explicitly saying it did not attempt a decode. Neither observation establishes a function. REA/Kuna 1.988 returned an empty exact-address `xrefs` result with its existing coverage caveat, but `read_bytes` and `inspect_native_instruction` were unavailable. These are observations from one compact stdio MCP session per provider; they do not establish address equivalence for other targets, loader states, or projects. The target digest stayed unchanged. Related: #8170 (Kuna bytes), #8360 (explicit candidate decode), and #8565 (unsupported-operation recovery). In the Ghidra session, `procedure_assembly` for the address placed a missing-procedure explanation in `details.issues`, but its headline and recovery instead blamed invalid inputs. `procedure_pseudo_code` used its documented bare-hex file-offset fallback after no procedure was found and reported an unmapped *file offset* in the details; see #8178. The observed initialized bytes do not make that fallback an analysis-address selector. On Ghidra recovery PR #8470 head `3dc41af77`, the same real compact MCP control surfaces the missing-procedure or missing-file-mapping explanation and relevant remediation directly, without schema-repair recovery; the bare numeric decompile fallback still applies as designed. The session stayed usable to close and the target digest remained unchanged. ### Expected interaction with targets Read existing target bytes, launch a caller-selected static provider, and apply caller-provided analysis assumptions. A definition can alter analysis control-flow and boundaries. No executable modification or execution is proposed. ### Boundary details (optional) Any shared workflow must preserve the exact selected target and architecture, address space, explicit entry/range, applied/rejected result, and caller assertion provenance. Distinguish a one-off analysis at an arbitrary address from creating a session-wide function definition; those have different consequences. Bind assumptions to profile/cache identity. Existing Ghidra seeds are startup-scoped and may fail if a seed conflicts with decoded instructions; Kuna's definition is per-run and affects its inventory/export. Windows Ghidra P0 remains read-only. A maintainer scope decision is needed before integrating a capability that changes analysis/control-flow interpretation. ### Options and tradeoffs Evaluate a typed, bounded analyst-hypothesis workflow over existing provider-specific mechanisms, or offer an explicitly scoped single-address analysis with clear non-inventory status. Compare against the existing Ghidra seed file; do not simply expose `--define-function` as an unvalidated pass-through, silently assert function identity, or claim that providers persist edits in the same way. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Test no seed, valid seed, an interior address, overlapping/conflicting definitions, invalid/unmapped bytes, multiple sessions/targets, failed application, cache isolation, and CLI/MCP provenance. State how ranges and any inferred bodies differ per provider and verify source bytes remain unchanged. ### References Related: #8166 (type/name assertions), #8165 (durable Ghidra state), and #8178 (ambiguous address versus file-offset lookup). REA baseline `d48e472fa` plus unrelated Kuna 1.986 baseline changes in #8187; macOS arm64. @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:11:27 +00:00

bsakaguchi commented

2026-11-01 21:11:27 +00:00

Author

Owner

@morluto commented on GitHub (Oct 30, 2026):

Additional real Ghidra check on macOS arm64: npm run verify:ghidra:seeds passed with Ghidra 12.1.4 and JDK 21. Its verifier reports descending function ordering, out-of-range seed handling, a BSS label, BSS code rejection, analyzer option readback, not-applicable and unknown analyzer behavior, and owned-process cleanup. This is a real Ghidra pre-script regression lane, not evidence of a new REA/Kuna seed workflow or full CLI/MCP seed coverage. The separate same-target MCP observation in the issue remains the end-to-end Ghidra example.

@morluto commented on GitHub (Oct 30, 2026): Additional real Ghidra check on macOS arm64: `npm run verify:ghidra:seeds` passed with Ghidra 12.1.4 and JDK 21. Its verifier reports descending function ordering, out-of-range seed handling, a BSS label, BSS code rejection, analyzer option readback, not-applicable and unknown analyzer behavior, and owned-process cleanup. This is a real Ghidra pre-script regression lane, not evidence of a new REA/Kuna seed workflow or full CLI/MCP seed coverage. The separate same-target MCP observation in the issue remains the end-to-end Ghidra example.

bsakaguchi commented

2026-11-01 21:11:27 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the Ghidra, Kuna, and REA tool-design investigation.

bsakaguchi referenced this issue

2026-11-01 21:47:21 +00:00

[PR #6492] [MERGED] fix(python): keep pyc string previews on Unicode boundaries #7366

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 00:29:41 +00:00