morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #8593] [Design] Page retained native instruction references #6661

New issue

Open

opened 2026-11-01 21:13:01 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:13:01 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8593

Before submitting

I searched open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

With real Ghidra, inspect_native_instruction can complete with a typed incoming-reference collection larger than the default MCP response budget. REA returns a retained Evidence ID, but inspect_analysis_view rejects that parent, so the typed rows cannot be paged through MCP. xrefs returns the addresses on this fixture, without their reference kinds.

Reproduction

On macOS arm64, use an unexecuted, source-built Mach-O containing one audit_target function and 24,000 initialized function-pointer slots to it (a stress fixture, not a prevalence claim). Open it with Ghidra 12.1.4 and call inspect_native_instruction at 0x1000003c0 through compact stdio MCP on #8591 head 6c7592d1f.
The call returns resource_constraint for the 10 MiB response budget and retained ID ev_c364358aa1b3734ab638d329ac975c39a5486c939947669ee2e93343f3a62761. inspect_analysis_view with that ID and summary returns unsupported_target: this parent operation does not support analysis views.
export_evidence_bundle succeeds. The saved instruction Evidence contains 24,001 incoming rows, all Ghidra DATA references. The same target's xrefs returns 24,001 untyped addresses in a 727,419-byte SDK-serialized response.

A 3,000-slot control returned 3,001 typed instruction references successfully, with a 1,737,893-byte SDK-serialized response. Both binaries remained unchanged and were not executed.

Expected behavior

Provide a bounded, discoverable way to select retained instruction reference observations and their kind/provenance without repeating analysis. Preserve the instruction's own status, exact source and target identities, reference ordering, source limitations, and the difference between recorded-row coverage and complete discovery. Consider both code and data addresses; a function-only workaround cannot be assumed to apply to every selected address.

Impact and scope

An agent using the documented typed alternative to xrefs can receive an overflow error without an MCP page continuation for the completed typed observation. Exporting the complete Evidence bundle is a working file-based recovery. The 24,000-row case is an intentionally large fixture; no frequency, token-cost, autonomous-agent behavior, other schema profile or provider completeness claim is made.

REA version

#8591 6c7592d1fbfc796159ba9be1441e9b6a00b29b08, based on main d3ab0faf52a02ff5a618f3ed7beb608b78752abd.

Runtime and platform

Node 24.22.1, macOS arm64, Ghidra 12.1.4, JDK 21, compact stdio MCP.

Target and artifact context

Source-built Mach-O arm64, SHA-256 3e15b3d7c8ff848d4399d2f9ab27f09269efca437a72706be8070968cb410016; unchanged and not executed.

Interface

MCP

Earliest failure stage

MCP response delivery and retained Evidence recovery.

Relevant output or evidence

The output reported resource_constraint, retained Evidence ID and boundary: mcp-response; the attempted summary view reported unsupported_target. The bundle contained all 24,001 typed incoming rows.

A separate real Ghidra control on draft #8588 0fabb2f7b used procedure_references with detail: summary and its paged view. It returned 24,001 incoming rows in 25 page calls; all 24,001 (source, target, type) triples matched the exported instruction observation on this fixture. That is a useful function-specific alternative, not a guarantee of identical semantics for other targets or a shipped main-branch view. On #8591, complete procedure_references itself also exceeded the response budget.

Typed outcome or recovery expected

Design a bounded instruction-observation view or another shared selection contract; compare it with #8587/#8588 before adding a new tool. Exercise full and compact output schemas, non-function data addresses, empty and unavailable collections, duplicate rows, byte limits, retained/offline handoff, and unchanged parent Evidence.

Analysis or suspected cause

Source inspection shows the Ghidra bridge constructs an instruction's incoming_references array, while the current analysis-view owner registry does not recognize inspect_native_instruction parents. This is distinct from #8587's procedure-wide reference pages and #8510's missing Kuna pointer rows.

Additional context

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Draft bounded view

Draft #8594 adds an instruction-reference view and an overflow continuation. Real full and compact Ghidra stdio MCP controls reconstructed all 24,001 typed incoming rows and original indices from the retained parent; an exact data-address control also matched its complete incoming references. An offline CLI request read the final row from the exported bundle. The fixture remained unchanged and unexecuted. Separate synthetic controls distinguish absent, null and empty incoming collections and check oversized-row recovery. This draft does not establish real Hopper, other-host or autonomous-agent behavior; full source generation precedes page delivery.

One-shot CLI discovery and provenance

On #8594 c5978bb26, the stock rea inspect-native-instruction <target> <address> --detail summary --evidence_output <new-file> --json rejected --detail as an unknown flag and wrote no file. A separate default CLI control on a selected Ghidra project with a historical analysis report returned an instruction Evidence record linked to that history Evidence, but no linked source record in its one-shot output. A large complete JSON result can be redirected to a file and paged offline with #8594, but that does not itself save the linked source closure.

Consider reusing the shared CLI Evidence delivery contract for a compact instruction summary and explicit complete bundle, retaining the exact linked source closure before the session closes. The summary should still answer the instruction question (status and decoded identity/flow), rather than only displaying its reference list. Related: #8590 (default one-shot source availability) and #8589 (final-close cancellation after capture). The observed unknown flag is not a violation of the current CLI contract; this is a design opportunity.

Unmerged CLI summary experiment

In an isolated temporary integration of #8594 c5978bb26 and #8585 2a0415cd5 (one view-registry conflict resolved for the experiment), an unmerged CLI prototype reused the existing Evidence-output and source-closure owners and added an instruction summary. On the unexecuted 48,000-slot Ghidra fixture, --detail summary --evidence_output <new-file> returned 4,610 bytes of JSON with the decoded mnemonic, bytes, flow and 48,001 incoming-reference count; the local bundle held the complete instruction record. The offline instruction-reference view read its final row. This is one output-size observation, not a token, time, CPU or memory benchmark.

On a separate caller-owned existing Ghidra project with analysis-history Evidence, the prototype bundle contained both the instruction and its exact linked history source. A missing output path was rejected before opening a nonexistent target, and an existing output bundle remained byte-identical after refusal. Target hashes were unchanged; targets were not executed. Real full/compact MCP controls also validated the summary for decoded code and data against the fetched output schemas.

This experiment is not included in either PR or a proposed final contract. It has not established late-cancellation/cleanup source handoff (#8589), oversized-summary recovery, full provider/platform coverage, or a resolution of the overlapping source-closure changes in #8297/#8585. A follow-up prototype preserves structured operands, optional decoder context, and flow (including direct destinations and classification evidence) in the summary; reference collections remain in the complete parent and paged views. On the same 48,000-slot fixture the summary JSON was 5,104 bytes and preserved the sub sp, sp, #0x10 register and immediate observations; the complete bundle retained all 48,001 incoming references. A separate direct-call control preserved Ghidra’s 0x100000410 address operand and direct destination in both full and compact MCP schemas, and a data-address and target-free bundle handoff also passed. The target bytes were unchanged. Five focused MCP tests and check:fast passed in the temporary integration. The earlier 4,610-byte observation describes the superseded counts-only summary; these byte counts are not a token or performance benchmark.

Source handoff after publication refusal

A real Ghidra control of the unmerged CLI prototype selected the existing project and requested a summary bundle at a pre-existing test-owned sentinel file. The command exited 1 with invalid_request and correctly left the sentinel unchanged. Its completed partial instruction (ev_a1a56314a70a2e5dac960218f8fc94616aba3017c17c7c0c55ed57cec9b309d7) linked the historical health record (ev_bd8f42f000d2c1f6f9e24e412d5303b694a5dc2b286d3193b73e000e8ebb60a8), but the error contained no linked source record or publication receipt. The suggested retry at another path requires a new one-shot analysis; it cannot guarantee the same project or source. No cleanup fault or cancellation was injected in this control.

The shared publisher receives the captured sources but projects publication failure with only the primary partial observation. This is a source-derived explanation for the prototype; it does not establish all CLI commands or #8297 have the same behavior. Consider failure delivery together with #8589 before treating successful source-closure export as complete recovery.

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8593 ### Before submitting - [x] I searched open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior With real Ghidra, `inspect_native_instruction` can complete with a typed incoming-reference collection larger than the default MCP response budget. REA returns a retained Evidence ID, but `inspect_analysis_view` rejects that parent, so the typed rows cannot be paged through MCP. `xrefs` returns the addresses on this fixture, without their reference kinds. ### Reproduction 1. On macOS arm64, use an unexecuted, source-built Mach-O containing one `audit_target` function and 24,000 initialized function-pointer slots to it (a stress fixture, not a prevalence claim). Open it with Ghidra 12.1.4 and call `inspect_native_instruction` at `0x1000003c0` through compact stdio MCP on #8591 head `6c7592d1f`. 2. The call returns `resource_constraint` for the 10 MiB response budget and retained ID `ev_c364358aa1b3734ab638d329ac975c39a5486c939947669ee2e93343f3a62761`. `inspect_analysis_view` with that ID and `summary` returns `unsupported_target`: this parent operation does not support analysis views. 3. `export_evidence_bundle` succeeds. The saved instruction Evidence contains 24,001 incoming rows, all Ghidra `DATA` references. The same target's `xrefs` returns 24,001 untyped addresses in a 727,419-byte SDK-serialized response. A 3,000-slot control returned 3,001 typed instruction references successfully, with a 1,737,893-byte SDK-serialized response. Both binaries remained unchanged and were not executed. ### Expected behavior Provide a bounded, discoverable way to select retained instruction reference observations and their kind/provenance without repeating analysis. Preserve the instruction's own status, exact source and target identities, reference ordering, source limitations, and the difference between recorded-row coverage and complete discovery. Consider both code and data addresses; a function-only workaround cannot be assumed to apply to every selected address. ### Impact and scope An agent using the documented typed alternative to `xrefs` can receive an overflow error without an MCP page continuation for the completed typed observation. Exporting the complete Evidence bundle is a working file-based recovery. The 24,000-row case is an intentionally large fixture; no frequency, token-cost, autonomous-agent behavior, other schema profile or provider completeness claim is made. ### REA version #8591 `6c7592d1fbfc796159ba9be1441e9b6a00b29b08`, based on main `d3ab0faf52a02ff5a618f3ed7beb608b78752abd`. ### Runtime and platform Node 24.22.1, macOS arm64, Ghidra 12.1.4, JDK 21, compact stdio MCP. ### Target and artifact context Source-built Mach-O arm64, SHA-256 `3e15b3d7c8ff848d4399d2f9ab27f09269efca437a72706be8070968cb410016`; unchanged and not executed. ### Interface MCP ### Earliest failure stage MCP response delivery and retained Evidence recovery. ### Relevant output or evidence The output reported `resource_constraint`, retained Evidence ID and `boundary: mcp-response`; the attempted summary view reported `unsupported_target`. The bundle contained all 24,001 typed incoming rows. A separate real Ghidra control on draft #8588 `0fabb2f7b` used `procedure_references` with `detail: summary` and its paged view. It returned 24,001 incoming rows in 25 page calls; all 24,001 `(source, target, type)` triples matched the exported instruction observation on this fixture. That is a useful function-specific alternative, not a guarantee of identical semantics for other targets or a shipped main-branch view. On #8591, complete `procedure_references` itself also exceeded the response budget. ### Typed outcome or recovery expected Design a bounded instruction-observation view or another shared selection contract; compare it with #8587/#8588 before adding a new tool. Exercise full and compact output schemas, non-function data addresses, empty and unavailable collections, duplicate rows, byte limits, retained/offline handoff, and unchanged parent Evidence. ### Analysis or suspected cause Source inspection shows the Ghidra bridge constructs an instruction's `incoming_references` array, while the current analysis-view owner registry does not recognize `inspect_native_instruction` parents. This is distinct from #8587's procedure-wide reference pages and #8510's missing Kuna pointer rows. ### Additional context cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Draft bounded view Draft #8594 adds an instruction-reference view and an overflow continuation. Real full and compact Ghidra stdio MCP controls reconstructed all 24,001 typed incoming rows and original indices from the retained parent; an exact data-address control also matched its complete incoming references. An offline CLI request read the final row from the exported bundle. The fixture remained unchanged and unexecuted. Separate synthetic controls distinguish absent, null and empty incoming collections and check oversized-row recovery. This draft does not establish real Hopper, other-host or autonomous-agent behavior; full source generation precedes page delivery. ### One-shot CLI discovery and provenance On #8594 `c5978bb26`, the stock `rea inspect-native-instruction <target> <address> --detail summary --evidence_output <new-file> --json` rejected `--detail` as an unknown flag and wrote no file. A separate default CLI control on a selected Ghidra project with a historical analysis report returned an instruction Evidence record linked to that history Evidence, but no linked source record in its one-shot output. A large complete JSON result can be redirected to a file and paged offline with #8594, but that does not itself save the linked source closure. Consider reusing the shared CLI Evidence delivery contract for a compact instruction summary and explicit complete bundle, retaining the exact linked source closure before the session closes. The summary should still answer the instruction question (status and decoded identity/flow), rather than only displaying its reference list. Related: #8590 (default one-shot source availability) and #8589 (final-close cancellation after capture). The observed unknown flag is not a violation of the current CLI contract; this is a design opportunity. ### Unmerged CLI summary experiment In an isolated temporary integration of #8594 `c5978bb26` and #8585 `2a0415cd5` (one view-registry conflict resolved for the experiment), an unmerged CLI prototype reused the existing Evidence-output and source-closure owners and added an instruction summary. On the unexecuted 48,000-slot Ghidra fixture, `--detail summary --evidence_output <new-file>` returned 4,610 bytes of JSON with the decoded mnemonic, bytes, flow and 48,001 incoming-reference count; the local bundle held the complete instruction record. The offline instruction-reference view read its final row. This is one output-size observation, not a token, time, CPU or memory benchmark. On a separate caller-owned existing Ghidra project with analysis-history Evidence, the prototype bundle contained both the instruction and its exact linked history source. A missing output path was rejected before opening a nonexistent target, and an existing output bundle remained byte-identical after refusal. Target hashes were unchanged; targets were not executed. Real full/compact MCP controls also validated the summary for decoded code and data against the fetched output schemas. This experiment is not included in either PR or a proposed final contract. It has not established late-cancellation/cleanup source handoff (#8589), oversized-summary recovery, full provider/platform coverage, or a resolution of the overlapping source-closure changes in #8297/#8585. A follow-up prototype preserves structured operands, optional decoder context, and flow (including direct destinations and classification evidence) in the summary; reference collections remain in the complete parent and paged views. On the same 48,000-slot fixture the summary JSON was 5,104 bytes and preserved the `sub sp, sp, #0x10` register and immediate observations; the complete bundle retained all 48,001 incoming references. A separate direct-call control preserved Ghidra’s `0x100000410` address operand and direct destination in both full and compact MCP schemas, and a data-address and target-free bundle handoff also passed. The target bytes were unchanged. Five focused MCP tests and `check:fast` passed in the temporary integration. The earlier 4,610-byte observation describes the superseded counts-only summary; these byte counts are not a token or performance benchmark. ### Source handoff after publication refusal A real Ghidra control of the unmerged CLI prototype selected the existing project and requested a summary bundle at a pre-existing test-owned sentinel file. The command exited 1 with `invalid_request` and correctly left the sentinel unchanged. Its completed partial instruction (`ev_a1a56314a70a2e5dac960218f8fc94616aba3017c17c7c0c55ed57cec9b309d7`) linked the historical `health` record (`ev_bd8f42f000d2c1f6f9e24e412d5303b694a5dc2b286d3193b73e000e8ebb60a8`), but the error contained no linked source record or publication receipt. The suggested retry at another path requires a new one-shot analysis; it cannot guarantee the same project or source. No cleanup fault or cancellation was injected in this control. The shared publisher receives the captured sources but projects publication failure with only the primary partial observation. This is a source-derived explanation for the prototype; it does not establish all CLI commands or #8297 have the same behavior. Consider failure delivery together with #8589 before treating successful source-closure export as complete recovery.

bsakaguchi added the

triage

label

2026-11-01 21:13:01 +00:00

bsakaguchi referenced this issue

2026-11-04 12:09:24 +00:00

[GH-ISSUE #8023] [Bug] A GDB console command whose output is one MI line over 16 MiB ends the session with no way to detach, kill or close it #4659