[GH-ISSUE #8560] [Design] Bound Ghidra byte-read allocation before response construction #6650

New issue

Open

opened 2026-11-01 21:12:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:56 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8560

Decision needed

How should read_bytes bound or page large Ghidra reads before allocating the provider response, while keeping exact requested and returned coverage and allowing legitimate larger investigations?

Context and evidence

On macOS arm64, Node 24.22.1, Ghidra 12.1.4 and REA 1662c037b900d0cc12346cffdb32bbbc69e9dc4b, an owned arm64 Mach-O with a 1 MiB initialized data section was opened in real compact stdio MCP. read_bytes returned exact bytes and complete: true for 65,536, 262,144 and 1,048,576-byte reads. Serialized Evidence response sizes were 134,506, 527,725 and 2,100,592 bytes respectively. These are output sizes, not timing, token, CPU or memory measurements. The fixture was not executed and its SHA-256 stayed 501ec1bc5d8497cb40e017d52cc5682186a480a5dbb64d083a06909ffe1bab9d.

Source-derived concern: the public input accepts positive safe integer lengths. The Ghidra bridge accumulates initialized bytes in a ByteArrayOutputStream, then copies them into a hexadecimal JSON value before returning. A downstream MCP response limit does not by itself prevent those earlier allocations. No resource exhaustion, large-request failure, or production frequency was measured.

MCP response-budget control

On the same host and build, a separate owned, unexecuted Mach-O with 6 MiB of initialized 0x5a bytes was read through real Ghidra compact stdio MCP with REA_MCP_MAX_RESPONSE_BYTES=10485760. The read completed but returned a typed, nonretryable resource_constraint for the MCP response budget, including a retained Evidence ID and export guidance. An exact-ID summary found the complete retained record; export_evidence_bundle then wrote it, and a local byte-for-byte check confirmed all 6,291,456 expected bytes and complete: true. A subsequent eight-byte read and session close succeeded. The target SHA-256 1648bb37597bde7c0f1e4d7d89e349cd75297beb753ac7cf958f63c729fbb2d2 was unchanged.

This confirms the existing response budget provides useful delivery recovery after this completed read; it does not establish a producer-side allocation bound or quantify peak memory. It does not prove behavior at larger sizes, cancellation, other transports or providers.

Existing search alternative

On the same 1 MiB Ghidra fixture and compact MCP profile, bounded search_bytes for an absent 5a 5b pattern over the initialized 1 MiB data section returned zero matches, complete: true, searched.bytes: 1048576, and a 3,822-byte serialized Evidence response. A 5a 5a search over 16 bytes with limit: 3 returned all 15 overlapping matches in five cursor pages without gaps or duplicates; coverage overlaps across pages as documented. The fixture digest remained unchanged. These are one workflow's output sizes, not performance or token measurements.

Searching is an existing smaller-delivery alternative when the analyst knows a pattern; it does not return arbitrary bytes and searches initialized blocks separately, so its completeness is limited to the reported eligible scope. It does not replace a bounded byte-read/export workflow.

Expected interaction with targets

Read existing files and launch the caller-supplied Ghidra provider for static analysis; no target execution or target mutation is proposed.

Boundary details

Any resource control should account for provider-side accumulation, hex expansion, JSON and Evidence copies before allocation; preserve exact addresses, requested/returned length and incompleteness. Avoid interpreting a resource refusal as an unmapped or uninitialized range. Scope concerns the read workflow, not a claim that every query needs a fixed arbitrary cap.

Options and tradeoffs

Consider bounded paging or a documented, justified caller-adjustable read budget with a typed recovery path and a safe way to retain/inspect large evidence. Investigate existing bridge and MCP budgets first; no particular threshold is proposed. A consumer-side output cap alone would leave the provider work intact.

Verification available so far

I have observed results. Real Ghidra 12.1.4, one macOS arm64 fixture, compact MCP; bounded reads up to 1 MiB. Other providers, formats, error and cancellation behavior, and realistic peak process-family memory remain unverified.

Success criteria and open questions

Measure representative complete tasks and inspect producer allocations before choosing limits. Exercise just-below/above the chosen real bound, partial/uninitialized ranges, cancellation/cleanup, and caller recovery. Include byte-for-byte checks and unchanged target identity. Determine whether existing output and transport budgets already impose a meaningful earlier bound.

References

Related: #8170 (Kuna read provenance and producer expansion); #8556 (byte qualifications in function matching).

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8560 ### Decision needed How should `read_bytes` bound or page large Ghidra reads before allocating the provider response, while keeping exact requested and returned coverage and allowing legitimate larger investigations? ### Context and evidence On macOS arm64, Node 24.22.1, Ghidra 12.1.4 and REA `1662c037b900d0cc12346cffdb32bbbc69e9dc4b`, an owned arm64 Mach-O with a 1 MiB initialized data section was opened in real compact stdio MCP. `read_bytes` returned exact bytes and `complete: true` for 65,536, 262,144 and 1,048,576-byte reads. Serialized Evidence response sizes were 134,506, 527,725 and 2,100,592 bytes respectively. These are output sizes, not timing, token, CPU or memory measurements. The fixture was not executed and its SHA-256 stayed `501ec1bc5d8497cb40e017d52cc5682186a480a5dbb64d083a06909ffe1bab9d`. Source-derived concern: the [public input](https://github.com/morluto/rea/blob/1662c037b900d0cc12346cffdb32bbbc69e9dc4b/src/contracts/hopperMemoryContracts.ts#L27) accepts positive safe integer lengths. The [Ghidra bridge](https://github.com/morluto/rea/blob/1662c037b900d0cc12346cffdb32bbbc69e9dc4b/bridge/ghidra/ReaGhidraBridge.java#L736) accumulates initialized bytes in a `ByteArrayOutputStream`, then copies them into a hexadecimal JSON value before returning. A downstream MCP response limit does not by itself prevent those earlier allocations. No resource exhaustion, large-request failure, or production frequency was measured. ### MCP response-budget control On the same host and build, a separate owned, unexecuted Mach-O with 6 MiB of initialized `0x5a` bytes was read through real Ghidra compact stdio MCP with `REA_MCP_MAX_RESPONSE_BYTES=10485760`. The read completed but returned a typed, nonretryable `resource_constraint` for the MCP response budget, including a retained Evidence ID and export guidance. An exact-ID summary found the complete retained record; `export_evidence_bundle` then wrote it, and a local byte-for-byte check confirmed all 6,291,456 expected bytes and `complete: true`. A subsequent eight-byte read and session close succeeded. The target SHA-256 `1648bb37597bde7c0f1e4d7d89e349cd75297beb753ac7cf958f63c729fbb2d2` was unchanged. This confirms the existing response budget provides useful delivery recovery after this completed read; it does not establish a producer-side allocation bound or quantify peak memory. It does not prove behavior at larger sizes, cancellation, other transports or providers. ### Existing search alternative On the same 1 MiB Ghidra fixture and compact MCP profile, bounded `search_bytes` for an absent `5a 5b` pattern over the initialized 1 MiB data section returned zero matches, `complete: true`, `searched.bytes: 1048576`, and a 3,822-byte serialized Evidence response. A `5a 5a` search over 16 bytes with `limit: 3` returned all 15 overlapping matches in five cursor pages without gaps or duplicates; coverage overlaps across pages as documented. The fixture digest remained unchanged. These are one workflow's output sizes, not performance or token measurements. Searching is an existing smaller-delivery alternative when the analyst knows a pattern; it does not return arbitrary bytes and searches initialized blocks separately, so its completeness is limited to the reported eligible scope. It does not replace a bounded byte-read/export workflow. ### Expected interaction with targets Read existing files and launch the caller-supplied Ghidra provider for static analysis; no target execution or target mutation is proposed. ### Boundary details Any resource control should account for provider-side accumulation, hex expansion, JSON and Evidence copies before allocation; preserve exact addresses, requested/returned length and incompleteness. Avoid interpreting a resource refusal as an unmapped or uninitialized range. Scope concerns the read workflow, not a claim that every query needs a fixed arbitrary cap. ### Options and tradeoffs Consider bounded paging or a documented, justified caller-adjustable read budget with a typed recovery path and a safe way to retain/inspect large evidence. Investigate existing bridge and MCP budgets first; no particular threshold is proposed. A consumer-side output cap alone would leave the provider work intact. ### Verification available so far I have observed results. Real Ghidra 12.1.4, one macOS arm64 fixture, compact MCP; bounded reads up to 1 MiB. Other providers, formats, error and cancellation behavior, and realistic peak process-family memory remain unverified. ### Success criteria and open questions Measure representative complete tasks and inspect producer allocations before choosing limits. Exercise just-below/above the chosen real bound, partial/uninitialized ranges, cancellation/cleanup, and caller recovery. Include byte-for-byte checks and unchanged target identity. Determine whether existing output and transport budgets already impose a meaningful earlier bound. ### References Related: #8170 (Kuna read provenance and producer expansion); #8556 (byte qualifications in function matching). cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:56 +00:00

bsakaguchi referenced this issue

2026-11-01 21:46:41 +00:00

[PR #6316] [MERGED] fix(runtime): align remaining Node security pins and setup guidance #7277