morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8336] [Design] Clarify or represent direct memory varnodes in native value-flow effects #6552

New issue

Open

opened 2026-11-01 21:12:10 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-11-01 21:12:10 +00:00

Owner

Originally created by @morluto on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8336

Before submitting

I searched the open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

How should the native value-flow workflow expose or qualify direct memory varnode reads and writes, so that an empty value_flow_effects view is not mistaken for an absence of memory access?

Context and evidence

A source-built x86-64 ELF contains four small functions: a direct volatile global setter/getter, and corresponding setter/getter through a pointer parameter. Independent llvm-objdump shows one memory write and one memory read in each pair. On REA a989c0e with real Ghidra 12.1.4 through MCP:

Function	Ghidra high p-code observation	`native_value_flow.effects`	Independent REA reference
Direct global setter	`COPY` from parameter to address varnode `0x103438`	`[]`	`WRITE` to `0x103438`
Direct global getter	`COPY` from address varnode `0x103438`	`[]`	`READ` from `0x103438`
Pointer setter	`STORE`	one `memory_write`	pointer target not statically identified
Pointer getter	`LOAD`	one `memory_read`	pointer target not statically identified

All four value flows were available and untruncated for the selected limits. The retained value_flow_effects view for each direct function returned an empty, exhausted view with no unknowns; its limitations say that memory effects can include stack and temporary accesses, but do not explicitly say direct address-varnode reads/writes are excluded. trace_native_values preserved the COPY operations and also returned no residual unknowns for these examples. Real CLI rea function ... --detail complete reproduced the direct-setter empty effect list and pointer-setter write effect.

The current implementation is consistent with its internal definition: addPcodeEffect creates memory effects for LOAD and STORE opcodes; the domain type describes effects as p-code-named operations. This report does not claim the collected p-code operations or empty page are wrong. The question is whether that scope is sufficiently explicit/useful to an agent asking what memory a function accesses. Ghidra's p-code documentation defines COPY between varnodes and LOAD/STORE for dynamic pointer dereferences.

Expected interaction with targets

Read caller-selected binaries and provider analysis; no target execution, injection, mutation or provider switch is proposed.

Boundary details (optional)

Keep high-p-code observations, instruction-level reference classifications, runtime accesses and persistent-state inferences distinct. Memory-space inputs/outputs can occur on operations other than COPY; stack, register, unique, overlay and other address spaces must not be conflated. High p-code can include detached or synthetic operations: this fixture also contains a return-time self-copy of the global, which must not be blindly counted as a second machine write. Some memory addresses or aliases can remain unknown. The current effect schema indexes operation inputs, while a direct write can have its memory location in the output, so a change may need a different typed representation.

Options and tradeoffs

At minimum, make the existing effect projection's LOAD/STORE scope and the meaning of empty coverage explicit, and point to independently observed procedure_references where supported. Alternatively, design an evidence-backed representation of direct memory-space varnode accesses, preserving source operation/operand location, provenance, synthetic membership and uncertainty. Do not silently combine high-p-code effects with instruction references or promote static observations to runtime facts.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Use an unmocked CLI/MCP fixture with direct global and pointer reads/writes, plus synthetic-return/self-copy controls; test truncation, missing operations, stack and non-default address spaces. Explicitly verify both positive effects and honest no-match coverage. The four controlled functions above do not establish completeness on other architectures or arbitrary Ghidra projects.

References

Related: #8190 (outgoing reference evidence), #8283 (bounded graph coverage). Fixture SHA-256 e8f487fb48af4db5dc1d20067e6f1f69c4ecc2bb024cf480a70829fdd7528eef, unchanged; target not executed. Node 24.22.1, macOS arm64, Ghidra 12.1.4, real x86-64 ELF analyzed statically.

@gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Independent x86-64 outgoing-reference control

With real Ghidra 12.1.4 through REA compact stdio MCP at ae8223cc, another source-owned x86-64 ELF contains separate direct RIP-relative global read and write, address-only LEA, pointer-parameter read, and immediate-only functions. Independent disassembly confirms the five instructions. procedure_references reported READ/WRITE/DATA for the first three, but returned empty recorded reference lists for both the indirect read and immediate-only functions. analyze_function reported one high-p-code memory_read effect for the indirect read and no effects for the direct global read/write, address-only or immediate-only functions. This corroborates the complementary scopes rather than showing either representation is complete. The unexecuted target SHA-256 d3baa5cf385b68e3a8de02ed0884d9c9e781bd2c6b63201250635e4dc37655e8 was unchanged. Full and compact profiles were checked for references and assembly; the dossier was checked in compact only. Related #8190 records direct Kuna controls on the same fixture.

Retained Evidence handoff control

Draft #8651 adds the LOAD/STORE scope limitation at the Ghidra producer and carries it into retained native views. With real Ghidra 12.1.4 on the x86-64 direct-read fixture above, I exported one dossier from baseline ae8223cc and one from #8651 head da308f399. In fresh target-free MCP sessions using compact and full profiles, the new retained effects view preserved the limitation; the imported older record remained unchanged and its view did not gain a limitation that the historical source never recorded. Both bundles and the target kept their SHA-256 values. Successful outputs validated against the advertised schemas. This tests one record per version, not all historical bundles. #8651 does not introduce direct-memory effects or claim complete access coverage.

Composed trace and offline handoff

On #8651 head c103de06f, real Ghidra stdio MCP for the x86-64 direct-read fixture returned the analyze_function scope limitation inline in trace_native_values in both profiles; the prior PR head retained the source dossier but omitted its limitations from the trace. The revised trace links that dossier. Exporting the trace and source, then importing them into a new target-free MCP session in each profile, preserved the trace record exactly and the source effects-view limitation. Bundle and target digests were unchanged and successful replies validated against advertised schemas. This is one trace on one host, not proof of general migration or memory-access completeness. A controlled snapshot regression checks that revision-1 value traces are not replayed as revision 2.

Real snapshot replay boundary

On #8651 before its producer-profile change, a Ghidra snapshot created at baseline ae8223cc reopened under the new workflow profile, but the new trace consumed an old cached function dossier and omitted the new LOAD/STORE scope. Following its synthesized source ID also encountered the separate #8282 source-ownership problem. Thus changing only the workflow profile did not establish fresh source semantics.

At #8651 head ecd7e86e, the Ghidra profile commits value_flow_effect_scope: pcode-load-store-explicit-v1. Real compact and full MCP controls rejected the old snapshot with evidence_integrity_mismatch, imported its separately extracted historical Evidence bundle unchanged, and produced the qualified trace after opening the target without that snapshot. A snapshot saved under the new profile reopened and retained the scope. The old snapshot, extracted bundle and target digests remained unchanged. This is one snapshot and host; it does not establish general migration. The extraction was a local manual step, not a dedicated REA snapshot-export command.

Originally created by @morluto on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8336 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed How should the native value-flow workflow expose or qualify direct memory varnode reads and writes, so that an empty `value_flow_effects` view is not mistaken for an absence of memory access? ### Context and evidence A source-built x86-64 ELF contains four small functions: a direct volatile global setter/getter, and corresponding setter/getter through a pointer parameter. Independent `llvm-objdump` shows one memory write and one memory read in each pair. On REA `a989c0e` with real Ghidra 12.1.4 through MCP: | Function | Ghidra high p-code observation | `native_value_flow.effects` | Independent REA reference | | --- | --- | --- | --- | | Direct global setter | `COPY` from parameter to address varnode `0x103438` | `[]` | `WRITE` to `0x103438` | | Direct global getter | `COPY` from address varnode `0x103438` | `[]` | `READ` from `0x103438` | | Pointer setter | `STORE` | one `memory_write` | pointer target not statically identified | | Pointer getter | `LOAD` | one `memory_read` | pointer target not statically identified | All four value flows were available and untruncated for the selected limits. The retained `value_flow_effects` view for each direct function returned an empty, exhausted **view** with no unknowns; its limitations say that memory effects can include stack and temporary accesses, but do not explicitly say direct address-varnode reads/writes are excluded. `trace_native_values` preserved the `COPY` operations and also returned no residual unknowns for these examples. Real CLI `rea function ... --detail complete` reproduced the direct-setter empty effect list and pointer-setter write effect. The current implementation is consistent with its internal definition: [`addPcodeEffect`](https://github.com/morluto/rea/blob/a989c0e458c91af177e097abd7be0fd021b0401d/bridge/ghidra/ReaGhidraBridge.java#L1880) creates memory effects for `LOAD` and `STORE` opcodes; the domain type describes effects as p-code-named operations. This report does **not** claim the collected p-code operations or empty page are wrong. The question is whether that scope is sufficiently explicit/useful to an agent asking what memory a function accesses. [Ghidra's p-code documentation](https://ghidra.re/ghidra_docs/languages/html/pcodedescription.html) defines `COPY` between varnodes and `LOAD`/`STORE` for dynamic pointer dereferences. ### Expected interaction with targets Read caller-selected binaries and provider analysis; no target execution, injection, mutation or provider switch is proposed. ### Boundary details (optional) Keep high-p-code observations, instruction-level reference classifications, runtime accesses and persistent-state inferences distinct. Memory-space inputs/outputs can occur on operations other than `COPY`; stack, register, unique, overlay and other address spaces must not be conflated. High p-code can include detached or synthetic operations: this fixture also contains a return-time self-copy of the global, which must not be blindly counted as a second machine write. Some memory addresses or aliases can remain unknown. The current effect schema indexes operation inputs, while a direct write can have its memory location in the output, so a change may need a different typed representation. ### Options and tradeoffs At minimum, make the existing effect projection's `LOAD`/`STORE` scope and the meaning of empty coverage explicit, and point to independently observed `procedure_references` where supported. Alternatively, design an evidence-backed representation of direct memory-space varnode accesses, preserving source operation/operand location, provenance, synthetic membership and uncertainty. Do not silently combine high-p-code effects with instruction references or promote static observations to runtime facts. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Use an unmocked CLI/MCP fixture with direct global and pointer reads/writes, plus synthetic-return/self-copy controls; test truncation, missing operations, stack and non-default address spaces. Explicitly verify both positive effects and honest no-match coverage. The four controlled functions above do not establish completeness on other architectures or arbitrary Ghidra projects. ### References Related: #8190 (outgoing reference evidence), #8283 (bounded graph coverage). Fixture SHA-256 `e8f487fb48af4db5dc1d20067e6f1f69c4ecc2bb024cf480a70829fdd7528eef`, unchanged; target not executed. Node 24.22.1, macOS arm64, Ghidra 12.1.4, real x86-64 ELF analyzed statically. @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Independent x86-64 outgoing-reference control With real Ghidra 12.1.4 through REA compact stdio MCP at `ae8223cc`, another source-owned x86-64 ELF contains separate direct RIP-relative global read and write, address-only LEA, pointer-parameter read, and immediate-only functions. Independent disassembly confirms the five instructions. `procedure_references` reported READ/WRITE/DATA for the first three, but returned empty recorded reference lists for both the indirect read and immediate-only functions. `analyze_function` reported one high-p-code `memory_read` effect for the indirect read and no effects for the direct global read/write, address-only or immediate-only functions. This corroborates the complementary scopes rather than showing either representation is complete. The unexecuted target SHA-256 `d3baa5cf385b68e3a8de02ed0884d9c9e781bd2c6b63201250635e4dc37655e8` was unchanged. Full and compact profiles were checked for references and assembly; the dossier was checked in compact only. Related #8190 records direct Kuna controls on the same fixture. ### Retained Evidence handoff control Draft #8651 adds the `LOAD`/`STORE` scope limitation at the Ghidra producer and carries it into retained native views. With real Ghidra 12.1.4 on the x86-64 direct-read fixture above, I exported one dossier from baseline `ae8223cc` and one from #8651 head `da308f399`. In fresh target-free MCP sessions using compact and full profiles, the new retained effects view preserved the limitation; the imported older record remained unchanged and its view did not gain a limitation that the historical source never recorded. Both bundles and the target kept their SHA-256 values. Successful outputs validated against the advertised schemas. This tests one record per version, not all historical bundles. #8651 does not introduce direct-memory effects or claim complete access coverage. ### Composed trace and offline handoff On #8651 head `c103de06f`, real Ghidra stdio MCP for the x86-64 direct-read fixture returned the `analyze_function` scope limitation inline in `trace_native_values` in both profiles; the prior PR head retained the source dossier but omitted its limitations from the trace. The revised trace links that dossier. Exporting the trace and source, then importing them into a new target-free MCP session in each profile, preserved the trace record exactly and the source effects-view limitation. Bundle and target digests were unchanged and successful replies validated against advertised schemas. This is one trace on one host, not proof of general migration or memory-access completeness. A controlled snapshot regression checks that revision-1 value traces are not replayed as revision 2. ### Real snapshot replay boundary On #8651 before its producer-profile change, a Ghidra snapshot created at baseline `ae8223cc` reopened under the new workflow profile, but the new trace consumed an old cached function dossier and omitted the new `LOAD`/`STORE` scope. Following its synthesized source ID also encountered the separate #8282 source-ownership problem. Thus changing only the workflow profile did not establish fresh source semantics. At #8651 head `ecd7e86e`, the Ghidra profile commits `value_flow_effect_scope: pcode-load-store-explicit-v1`. Real compact and full MCP controls rejected the old snapshot with `evidence_integrity_mismatch`, imported its separately extracted historical Evidence bundle unchanged, and produced the qualified trace after opening the target without that snapshot. A snapshot saved under the new profile reopened and retained the scope. The old snapshot, extracted bundle and target digests remained unchanged. This is one snapshot and host; it does not establish general migration. The extraction was a local manual step, not a dedicated REA snapshot-export command.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:10 +00:00

bsakaguchi commented

2026-11-01 21:12:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Testing note: the existing real-Ghidra value-flow E2E in scripts/lib/native-value-e2e.mjs validates argument, parameter-use and return bindings. Source inspection found no memory_read/memory_write or direct-memory-varnode assertion in that script. This does not imply the repository has no other memory tests; it explains why this particular higher-level verifier would not exercise the direct/indirect distinction. No test suite was run as part of that source review.

@morluto commented on GitHub (Oct 31, 2026): Testing note: the existing real-Ghidra value-flow E2E in [`scripts/lib/native-value-e2e.mjs`](https://github.com/morluto/rea/blob/a989c0e458c91af177e097abd7be0fd021b0401d/scripts/lib/native-value-e2e.mjs#L52) validates argument, parameter-use and return bindings. Source inspection found no `memory_read`/`memory_write` or direct-memory-varnode assertion in that script. This does not imply the repository has no other memory tests; it explains why this particular higher-level verifier would not exercise the direct/indirect distinction. No test suite was run as part of that source review.

bsakaguchi commented

2026-11-01 21:12:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

arm64 Mach-O and offline-view control

The same four source-authored access patterns reproduce on an independently built arm64 Mach-O dylib with REA a989c0e and real Ghidra 12.1.4. llvm-objdump shows ADRP + STR for the direct global setter and ADRP + LDR for the getter. In the high-p-code dossier both direct accesses appear as COPY involving the global at 0x4000, and effects is empty; the pointer-parameter STR/LDR appear as STORE/LOAD and have memory_write/memory_read effects. The flows were not truncated, and the tests do not establish behavior for other access forms.

A useful manual alternative is available on this fixture: Ghidra xrefs(0x4000) found 0x3ac and 0x3b8, and inspect_native_instruction classified those exact sites as WRITE and READ. Kuna 1.986 through REA instead returned the preceding address-forming sites 0x3a8 and 0x3b4; its instruction-inspection operation was unavailable, although its procedure assembly showed the following STR/LDR. Direct Kuna xrefs --from labelled the address-forming observations as data, not read or write. These are different observations and must not be automatically equated; see #8190.

The TypeScript MCP SDK accepted advertised compact-profile output schemas for the successful summary/effect-view workflow. After Evidence export and import into a target-free server, both the empty direct-access effects view and the pointer-write effect view had exactly the same normalized results and Evidence IDs as live. This confirms retained projection, not that the effect list covers all machine memory accesses. Fixture SHA-256 4a86fb15551a497403c61dc6d4283851a32d0133887c2bac388598f5d6b91597 and bundle hash were unchanged; the target was not executed.

@morluto commented on GitHub (Oct 31, 2026): ### arm64 Mach-O and offline-view control The same four source-authored access patterns reproduce on an independently built arm64 Mach-O dylib with REA `a989c0e` and real Ghidra 12.1.4. `llvm-objdump` shows `ADRP` + `STR` for the direct global setter and `ADRP` + `LDR` for the getter. In the high-p-code dossier both direct accesses appear as `COPY` involving the global at `0x4000`, and `effects` is empty; the pointer-parameter `STR`/`LDR` appear as `STORE`/`LOAD` and have `memory_write`/`memory_read` effects. The flows were not truncated, and the tests do not establish behavior for other access forms. A useful manual alternative is available on this fixture: Ghidra `xrefs(0x4000)` found `0x3ac` and `0x3b8`, and `inspect_native_instruction` classified those exact sites as WRITE and READ. Kuna 1.986 through REA instead returned the preceding address-forming sites `0x3a8` and `0x3b4`; its instruction-inspection operation was unavailable, although its procedure assembly showed the following `STR`/`LDR`. Direct Kuna `xrefs --from` labelled the address-forming observations as `data`, not `read` or `write`. These are different observations and must not be automatically equated; see #8190. The TypeScript MCP SDK accepted advertised compact-profile output schemas for the successful summary/effect-view workflow. After Evidence export and import into a target-free server, both the empty direct-access effects view and the pointer-write effect view had exactly the same normalized results and Evidence IDs as live. This confirms retained projection, not that the effect list covers all machine memory accesses. Fixture SHA-256 `4a86fb15551a497403c61dc6d4283851a32d0133887c2bac388598f5d6b91597` and bundle hash were unchanged; the target was not executed.

bsakaguchi commented

2026-11-01 21:12:10 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Additional ARM64 Mach-O control on current REA d54a4661da20382ad8456564a08569fdd24131c5 / Ghidra 12.1.4: an owned function performs ldr w9,[x8], adds one, str w9,[x8], and returns the global's address. REA's instruction and function-reference tools report the corresponding READ and WRITE to 0x100004000. The complete function dossier's high-p-code value flow is available and untruncated, but effects is empty: the global appears as a direct address input to INT_ADD and as the output of COPY, not as LOAD/STORE. trace_native_values retains those operations.

The dossier also contains a later synthetic-looking global self-copy at the return, so counting every address varnode as a machine access would overstate the assembly. The operations are flagged is_dead while also marked syntax-tree members; the existing limitation explicitly says to interpret those fields separately. This broadens the earlier x86-64 example to one ARM64 case and supports qualifying or carefully modeling direct-memory effects, not blindly turning p-code rows into runtime accesses. The target SHA-256 9b1f167a30cd17cb5d4393d720336fc14d7ac3bce20ade06182b941ecf8918fd was unchanged and the target was not executed.

@morluto commented on GitHub (Oct 31, 2026): Additional ARM64 Mach-O control on current REA `d54a4661da20382ad8456564a08569fdd24131c5` / Ghidra 12.1.4: an owned function performs `ldr w9,[x8]`, adds one, `str w9,[x8]`, and returns the global's address. REA's instruction and function-reference tools report the corresponding `READ` and `WRITE` to `0x100004000`. The complete function dossier's high-p-code value flow is available and untruncated, but `effects` is empty: the global appears as a direct address input to `INT_ADD` and as the output of `COPY`, not as `LOAD`/`STORE`. `trace_native_values` retains those operations. The dossier also contains a later synthetic-looking global self-copy at the return, so counting every address varnode as a machine access would overstate the assembly. The operations are flagged `is_dead` while also marked syntax-tree members; the existing limitation explicitly says to interpret those fields separately. This broadens the earlier x86-64 example to one ARM64 case and supports qualifying or carefully modeling direct-memory effects, not blindly turning p-code rows into runtime accesses. The target SHA-256 `9b1f167a30cd17cb5d4393d720336fc14d7ac3bce20ade06182b941ecf8918fd` was unchanged and the target was not executed.

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 00:29:56 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6552

No description provided.

Rows
Columns