[GH-ISSUE #8556] [Design] Preserve byte-observation limitations in function matching #6647

New issue

Open

opened 2026-11-01 21:12:55 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:12:55 +00:00

Owner

Originally created by @morluto on GitHub (Nov 1, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8556

Decision needed

How should function-matching results convey limitations of the specific byte observations used for a fingerprint, without duplicating unrelated provider limitations or growing output with every input?

Context and evidence

Observed on macOS arm64, Ghidra 12.1.4 and REA commit 7f344af1aa0a78fd4e331dbbc9c30d38e4cfb796: imported authentic Ghidra Evidence from two owned stripped Mach-O targets into REA's stdio MCP server, then supplied the inventories, procedure_info and read_bytes observations to match_native_functions in both compact and full schema profiles. Each run produced an identical-bytes match. Its reason correctly states that relocations are not normalized. The Ghidra byte-read Evidence also warns that the bytes reflect initialized memory, including loader fixups; this warning did not appear in the complete matching report, its member limitations, or a paged function-matches view. Source Evidence IDs were retained in the result and member links, and the source records remained in the imported bundles. The fixture did not establish that these particular function bytes contain a fixup.

Source-derived explanation: NativeFunctionMatchingService builds report limitations from the two inventories and general matching caveats; functionInventory consumes byte observations and retains their Evidence links. Consequently an agent reading the matching output can miss a qualification attached to the bytes it matched, even though the original record is retained. This is related to, but distinct from, byte-read provenance.

Current main source-record recovery

Merged #8528 now allows get_evidence_bundle with detail: "records" and an exact Evidence ID. On current main 85013da0c, real Ghidra read_bytes, xrefs, and procedure_references records survived export/import and were retrieved individually with unchanged records and explicit historical/non-importable-selection limitations in both MCP output-schema profiles, without an open target. Thus the older full-bundle-only observation below is historical. This improves recovery but does not itself put the read-specific qualification in a matching result or its default view; the source-association question remains.

Source recovery control

In a fresh target-free stdio MCP session in each schema profile, importing one of the same Ghidra bundles and filtering get_evidence_bundle by the exact read_bytes Evidence ID returned one retained-record summary with limitation_count: 6, but no limitation text. inspect_analysis_view with that ID and kind: summary returned unsupported_target for read_bytes; complete bundle delivery with a filter returned invalid_request because filters require summary. Unfiltered complete delivery contained the warning. For this small seven-record fixture, the serialized complete response was 26,880 bytes and the selected read record was 3,593 bytes; these are sizes, not token counts or performance measurements. Exporting a complete bundle is another documented path, but this control specifically exercised complete inline delivery. This extends the related discovery/navigation concern in #8549 to a source needed to interpret a byte match.

Member limitation projection

A second target-free MCP control used the same Ghidra inventories and procedure_info Evidence but supplied no byte-read observations. The default function-matches function page (matched state, left side, fields omitted) returned the matched entry member without a limitations field. A separate page requesting fields: ["name", "limitations"] returned that same member with No complete unambiguous bytes for the observed function body. This occurred in both schema profiles. Thus simply adding read qualifications to member limitations would not make them visible in default function pages; callers can explicitly select that field today. Whether the default should include a concise qualification, an indicator, or a different representation needs to account for response size.

Overlap and source association control

A public MCP control combined the same real Ghidra Evidence with a synthetic, REA-created second read_bytes record having the same target, profile, address and bytes, plus a synthetic qualification. Reversing the two equal-range inputs in both schema profiles left the function fingerprint unchanged, but switched its member-level link between the two read IDs. Both IDs remained linked at the overall matching Evidence level. This demonstrates that the member currently cites the selected covering read, not every corroborating observation. It does not claim that Ghidra produced the synthetic qualification.

Source inspection: coverage sorts ranges and retains the first already-admitted overlapping bytes; identical equal-range input order decides which source supplies the member link. A solution must define whether it reports qualifications for the selected covering source or all applicable corroborating sources, retain their association honestly, and bound any expansion before constructing output. Blindly propagating all supplied read limitations to every member or assuming member links cover all observations would be misleading.

Expected interaction with targets

Read existing files or captured data; launch a provider process for acquisition. Matching itself consumes supplied Evidence and does not require opening a target. This proposal adds no target interaction.

Boundary details

Keep each qualification associated with the observations and sides to which it applies; preserve Evidence links and uncertainty. Avoid treating analyzed-memory bytes as necessarily identical to file bytes. Do not assume every source has the same caveat.

Options and tradeoffs

Consider a bounded, deduplicated representation of material source limitations in matching results and paged views, with links to the relevant original Evidence. Alternatively, make the byte basis and a direct recovery path explicit in the contract. Avoid copying every source warning onto every member: large reads can support many functions, and irrelevant operational warnings can drown out the relevant facts.

Verification available so far

I have observed results. The control above used previously captured real-provider Evidence; it did not measure autonomous-agent behavior or validate providers beyond Ghidra on macOS. Targets and input bundles were unchanged.

Success criteria and open questions

Exercise a real byte-based match and its paged view, including observations with different qualifications on each side. Check that an agent can distinguish the bytes actually compared and recover applicable source limitations without unbounded duplication. Determine how partial or overlapping reads and omitted/unknown source semantics should be represented.

References

Native function matching guide
Related Evidence navigation discussion: #8549

cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0

Real relocation inside the matched function

On current main d3ab0faf5, two source-built, unexecuted x86-64 ELF shared objects contained differently named 11-byte functions with the same movabs operand targeting a data symbol. LLVM readelf reported R_X86_64_64 at 0x12a2 in each function. The original file bytes were 48b80000000000000000c3; real Ghidra 12.1.4 read_bytes at its rebased function address returned 48b86033100000000000c3 after loader relocation.

In both full and compact stdio MCP, match_native_functions used the two actual function-body and byte-read records and produced one identical-bytes match, with the existing warning that relocations are not normalized. The matching Evidence linked both byte reads but did not carry their warning that bytes reflect Ghidra initialized memory including loader fixups. Both source files were unchanged (SHA-256 aceed94a32d0e97269b70df6283fad9e445cfb124e0628947dfc42d7b3ea7018 and f8263c87fcc8aadcadd7db3e163f1aeb272352005d7f6e6014f6ecd387a43e32). This demonstrates that the qualification applies to bytes actually used by a match; it does not show a false match or runtime behavior. The original file bytes were also identical in this controlled pair.

A follow-up changed one byte in a disposable copy's file-backed relocation slot while keeping its R_X86_64_64 relocation. The two function file spans were then 48b80000000000000000c3 and 48b8ab00000000000000c3; Ghidra's loaded observations for both were 48b86033100000000000c3. Real full and compact MCP still returned an identical-bytes match with the same “relocations are not normalized” reason and the byte-source Evidence links, but without the source warning about initialized memory and loader fixups. The two test files remained unchanged during analysis (SHA-256 aceed94a32d0e97269b70df6283fad9e445cfb124e0628947dfc42d7b3ea7018 and dbdc08112473081a579450b962efa10345a1a273ea0804f951f7e9e7cdff4e42). This controlled byte edit was fixture construction, not target execution or a claim that the match is wrong for the supplied loaded-memory observations. It shows that identical loaded bytes do not imply identical file bytes and makes the missing source basis consequential.

Originally created by @morluto on GitHub (Nov 1, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8556 ### Decision needed How should function-matching results convey limitations of the specific byte observations used for a fingerprint, without duplicating unrelated provider limitations or growing output with every input? ### Context and evidence Observed on macOS arm64, Ghidra 12.1.4 and REA commit `7f344af1aa0a78fd4e331dbbc9c30d38e4cfb796`: imported authentic Ghidra Evidence from two owned stripped Mach-O targets into REA's stdio MCP server, then supplied the inventories, `procedure_info` and `read_bytes` observations to `match_native_functions` in both compact and full schema profiles. Each run produced an `identical-bytes` match. Its reason correctly states that relocations are not normalized. The Ghidra byte-read Evidence also warns that the bytes reflect initialized memory, including loader fixups; this warning did not appear in the complete matching report, its member limitations, or a paged `function-matches` view. Source Evidence IDs were retained in the result and member links, and the source records remained in the imported bundles. The fixture did not establish that these particular function bytes contain a fixup. Source-derived explanation: [`NativeFunctionMatchingService`](https://github.com/morluto/rea/blob/7f344af1aa0a78fd4e331dbbc9c30d38e4cfb796/src/application/native/NativeFunctionMatchingService.ts#L95) builds report limitations from the two inventories and general matching caveats; [`functionInventory`](https://github.com/morluto/rea/blob/7f344af1aa0a78fd4e331dbbc9c30d38e4cfb796/src/domain/native/functionInventory.ts#L169) consumes byte observations and retains their Evidence links. Consequently an agent reading the matching output can miss a qualification attached to the bytes it matched, even though the original record is retained. This is related to, but distinct from, [byte-read provenance](https://github.com/morluto/rea/issues/8170). ### Current main source-record recovery Merged #8528 now allows `get_evidence_bundle` with `detail: "records"` and an exact Evidence ID. On current main `85013da0c`, real Ghidra `read_bytes`, `xrefs`, and `procedure_references` records survived export/import and were retrieved individually with unchanged records and explicit historical/non-importable-selection limitations in both MCP output-schema profiles, without an open target. Thus the older full-bundle-only observation below is historical. This improves recovery but does not itself put the read-specific qualification in a matching result or its default view; the source-association question remains. ### Source recovery control In a fresh target-free stdio MCP session in each schema profile, importing one of the same Ghidra bundles and filtering `get_evidence_bundle` by the exact `read_bytes` Evidence ID returned one retained-record summary with `limitation_count: 6`, but no limitation text. `inspect_analysis_view` with that ID and `kind: summary` returned `unsupported_target` for `read_bytes`; complete bundle delivery with a filter returned `invalid_request` because filters require summary. Unfiltered complete delivery contained the warning. For this small seven-record fixture, the serialized complete response was 26,880 bytes and the selected read record was 3,593 bytes; these are sizes, not token counts or performance measurements. Exporting a complete bundle is another documented path, but this control specifically exercised complete inline delivery. This extends the related discovery/navigation concern in #8549 to a source needed to interpret a byte match. ### Member limitation projection A second target-free MCP control used the same Ghidra inventories and `procedure_info` Evidence but supplied no byte-read observations. The default `function-matches` function page (matched state, left side, fields omitted) returned the matched `entry` member without a `limitations` field. A separate page requesting `fields: ["name", "limitations"]` returned that same member with `No complete unambiguous bytes for the observed function body.` This occurred in both schema profiles. Thus simply adding read qualifications to member limitations would not make them visible in default function pages; callers can explicitly select that field today. Whether the default should include a concise qualification, an indicator, or a different representation needs to account for response size. ### Overlap and source association control A public MCP control combined the same real Ghidra Evidence with a **synthetic**, REA-created second `read_bytes` record having the same target, profile, address and bytes, plus a synthetic qualification. Reversing the two equal-range inputs in both schema profiles left the function fingerprint unchanged, but switched its member-level link between the two read IDs. Both IDs remained linked at the overall matching Evidence level. This demonstrates that the member currently cites the selected covering read, not every corroborating observation. It does not claim that Ghidra produced the synthetic qualification. Source inspection: coverage sorts ranges and retains the first already-admitted overlapping bytes; identical equal-range input order decides which source supplies the member link. A solution must define whether it reports qualifications for the selected covering source or all applicable corroborating sources, retain their association honestly, and bound any expansion before constructing output. Blindly propagating all supplied read limitations to every member or assuming member links cover all observations would be misleading. ### Expected interaction with targets Read existing files or captured data; launch a provider process for acquisition. Matching itself consumes supplied Evidence and does not require opening a target. This proposal adds no target interaction. ### Boundary details Keep each qualification associated with the observations and sides to which it applies; preserve Evidence links and uncertainty. Avoid treating analyzed-memory bytes as necessarily identical to file bytes. Do not assume every source has the same caveat. ### Options and tradeoffs Consider a bounded, deduplicated representation of material source limitations in matching results and paged views, with links to the relevant original Evidence. Alternatively, make the byte basis and a direct recovery path explicit in the contract. Avoid copying every source warning onto every member: large reads can support many functions, and irrelevant operational warnings can drown out the relevant facts. ### Verification available so far I have observed results. The control above used previously captured real-provider Evidence; it did not measure autonomous-agent behavior or validate providers beyond Ghidra on macOS. Targets and input bundles were unchanged. ### Success criteria and open questions Exercise a real byte-based match and its paged view, including observations with different qualifications on each side. Check that an agent can distinguish the bytes actually compared and recover applicable source limitations without unbounded duplication. Determine how partial or overlapping reads and omitted/unknown source semantics should be represented. ### References - [Native function matching guide](https://github.com/morluto/rea/blob/7f344af1aa0a78fd4e331dbbc9c30d38e4cfb796/docs/native-function-matching.md) - Related Evidence navigation discussion: #8549 cc @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 ### Real relocation inside the matched function On current main `d3ab0faf5`, two source-built, unexecuted x86-64 ELF shared objects contained differently named 11-byte functions with the same `movabs` operand targeting a data symbol. LLVM readelf reported `R_X86_64_64` at `0x12a2` in each function. The original file bytes were `48b80000000000000000c3`; real Ghidra 12.1.4 `read_bytes` at its rebased function address returned `48b86033100000000000c3` after loader relocation. In both full and compact stdio MCP, `match_native_functions` used the two actual function-body and byte-read records and produced one `identical-bytes` match, with the existing warning that relocations are not normalized. The matching Evidence linked both byte reads but did not carry their warning that bytes reflect Ghidra initialized memory including loader fixups. Both source files were unchanged (SHA-256 `aceed94a32d0e97269b70df6283fad9e445cfb124e0628947dfc42d7b3ea7018` and `f8263c87fcc8aadcadd7db3e163f1aeb272352005d7f6e6014f6ecd387a43e32`). This demonstrates that the qualification applies to bytes actually used by a match; it does not show a false match or runtime behavior. The original file bytes were also identical in this controlled pair. A follow-up changed one byte in a disposable copy's file-backed relocation slot while keeping its `R_X86_64_64` relocation. The two function file spans were then `48b80000000000000000c3` and `48b8ab00000000000000c3`; Ghidra's loaded observations for both were `48b86033100000000000c3`. Real full and compact MCP still returned an `identical-bytes` match with the same “relocations are not normalized” reason and the byte-source Evidence links, but without the source warning about initialized memory and loader fixups. The two test files remained unchanged during analysis (SHA-256 `aceed94a32d0e97269b70df6283fad9e445cfb124e0628947dfc42d7b3ea7018` and `dbdc08112473081a579450b962efa10345a1a273ea0804f951f7e9e7cdff4e42`). This controlled byte edit was fixture construction, not target execution or a claim that the match is wrong for the supplied loaded-memory observations. It shows that identical loaded bytes do not imply identical file bytes and makes the missing source basis consequential.

bsakaguchi added the

enhancement

triage

labels

2026-11-01 21:12:55 +00:00

bsakaguchi added

enhancement

and removed

enhancement

labels

2026-11-03 00:30:06 +00:00