morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8287] [Design] Clarify provider inventory scope for binary overview counts #6524

New issue

Closed

opened 2026-11-01 21:11:55 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-11-01 21:11:55 +00:00

Owner

Originally created by @morluto on GitHub (Oct 31, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8287

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched open and closed issues and pull requests for a related discussion.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This discussion does not include credentials, private samples, or sensitive target data.

Decision needed

Should binary_overview explicitly label its counts as counts of provider-reported inventories, and expose their discovery scope in a machine-readable way when known?

Context and evidence

The live MCP tool description says “exhaustive procedure/string counts.” On a source-owned arm64 Mach-O at REA b43c329df779708e43ae297f498a844e45d9c923, real stdio MCP returned:

Provider	Overview procedure count	Corresponding inventory rows	Overview string count	Corresponding inventory rows
Ghidra 12.1.4	3	3	24	24
Kuna 1.986	6	6	1	1

The counts exactly match each provider's returned inventory in this sample. They do not describe interchangeable or necessarily complete artifact-wide inventories: Kuna's procedure rows included the Mach-O header and two data labels, including global_count at 0x100004000; inspect_macho independently reported that address in a writable, non-executable __DATA,__data section. #8168 tracks the separate false-procedure issue. Ghidra's limitations say it counts defined strings and known functions without establishing discovery of undefined strings or unidentified functions; Kuna's limitations describe inferred/split functions and a restricted printable-string scan.

Those caveats are present in the returned Evidence. The question is whether the unqualified catalog description and scalar-only result make it too easy for an agent to read “exhaustive” as discovery or as comparable counts across providers. No autonomous-agent behavior or broad corpus accuracy was measured. The target was not executed, and its SHA-256 stayed unchanged.

Expected interaction with targets

Read existing files or captured data.

Boundary details (optional)

Preserve exact totals over all rows the provider actually returns; do not reinterpret incomplete discovery as zero or suppress existing limitations. Distinguish provider inventory scope, pagination completeness, candidate classification and discovery completeness. Segments may be unavailable independently of procedures and strings.

Options and tradeoffs

Compare clarifying the existing description and result caveats with adding a small provider-neutral scope/coverage record beside each count. Reuse existing inventory Evidence or coverage concepts if adequate; avoid a provider-specific wrapper or a large duplicated inventory. Evaluate output/schema cost and compatibility before selecting a contract. #28's closed work on counting all returned pages remains useful and is not a request to reintroduce a page cap.

Verification available so far

I have a prototype or observed results.

Success criteria and open questions

Exercise Ghidra and Kuna on symbolized/stripped binaries, explicitly selected incomplete projects, missing/empty inventories and multiple pages. Check that the reported count remains the actual inventory count, discovery uncertainty remains distinct, and CLI/MCP plus converted schemas preserve the same semantics. No proposed new check has been run yet.

References

Related: #8168 (Kuna false procedure candidates), #8184 (feature-reference coverage), #4249 (retained inventory pages), #28 (all returned pages), and the catalog contract.

Originally created by @morluto on GitHub (Oct 31, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8287 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched open and closed issues and pull requests for a related discussion. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This discussion does not include credentials, private samples, or sensitive target data. ### Decision needed Should `binary_overview` explicitly label its counts as counts of provider-reported inventories, and expose their discovery scope in a machine-readable way when known? ### Context and evidence The live MCP tool description says “exhaustive procedure/string counts.” On a source-owned arm64 Mach-O at REA `b43c329df779708e43ae297f498a844e45d9c923`, real stdio MCP returned: | Provider | Overview procedure count | Corresponding inventory rows | Overview string count | Corresponding inventory rows | | --- | ---: | ---: | ---: | ---: | | Ghidra 12.1.4 | 3 | 3 | 24 | 24 | | Kuna 1.986 | 6 | 6 | 1 | 1 | The counts exactly match each provider's returned inventory in this sample. They do not describe interchangeable or necessarily complete artifact-wide inventories: Kuna's procedure rows included the Mach-O header and two data labels, including `global_count` at `0x100004000`; `inspect_macho` independently reported that address in a writable, non-executable `__DATA,__data` section. #8168 tracks the separate false-procedure issue. Ghidra's limitations say it counts defined strings and known functions without establishing discovery of undefined strings or unidentified functions; Kuna's limitations describe inferred/split functions and a restricted printable-string scan. Those caveats are present in the returned Evidence. The question is whether the unqualified catalog description and scalar-only result make it too easy for an agent to read “exhaustive” as discovery or as comparable counts across providers. No autonomous-agent behavior or broad corpus accuracy was measured. The target was not executed, and its SHA-256 stayed unchanged. ### Expected interaction with targets Read existing files or captured data. ### Boundary details (optional) Preserve exact totals over all rows the provider actually returns; do not reinterpret incomplete discovery as zero or suppress existing limitations. Distinguish provider inventory scope, pagination completeness, candidate classification and discovery completeness. Segments may be unavailable independently of procedures and strings. ### Options and tradeoffs Compare clarifying the existing description and result caveats with adding a small provider-neutral scope/coverage record beside each count. Reuse existing inventory Evidence or coverage concepts if adequate; avoid a provider-specific wrapper or a large duplicated inventory. Evaluate output/schema cost and compatibility before selecting a contract. #28's closed work on counting all returned pages remains useful and is not a request to reintroduce a page cap. ### Verification available so far I have a prototype or observed results. ### Success criteria and open questions Exercise Ghidra and Kuna on symbolized/stripped binaries, explicitly selected incomplete projects, missing/empty inventories and multiple pages. Check that the reported count remains the actual inventory count, discovery uncertainty remains distinct, and CLI/MCP plus converted schemas preserve the same semantics. No proposed new check has been run yet. ### References Related: #8168 (Kuna false procedure candidates), #8184 (feature-reference coverage), #4249 (retained inventory pages), #28 (all returned pages), and [the catalog contract](https://github.com/morluto/rea/blob/b43c329df779708e43ae297f498a844e45d9c923/src/contracts/enhancedToolContracts.ts).

bsakaguchi

2026-11-01 21:11:55 +00:00

closed this issue
added the
enhancement

triage
labels

bsakaguchi commented

2026-11-01 21:11:56 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the REA tool-design investigation.

@morluto commented on GitHub (Oct 31, 2026): Looping in @gensokyo-dev @kaoru0822-kitauji @N0zoM1z0 for visibility on the REA tool-design investigation.

bsakaguchi commented

2026-11-01 21:11:56 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Selected-project discovery coverage control

I repeated the overview on the same source-owned arm64 Mach-O using real Ghidra 12.1.4 MCP at REA b43c329df779708e43ae297f498a844e45d9c923. A fresh import returned three procedure rows and procedure_count: 3; a separately selected project created with Ghidra -noanalysis returned one row and procedure_count: 1. Both returned 24 defined-string rows. The selected-project overview Evidence warned that original analysis completeness/settings were unknown and that REA did not rerun analysis.

The existing list_procedures({detail:"summary"}) path already exposes the useful distinction: its retained inventory summary reported row_count: 1, discovery_coverage: "unknown", and a page with exhausted: true for the one retained row. The fresh import also reported discovery coverage unknown. A literal search found _evaluate in the fresh import and returned no rows in the selected project; that absence describes its retained inventory, not an assertion about the source file.

This narrows the design choice: the overview could potentially reuse the existing inventory-scope vocabulary rather than invent another meaning for completeness. Do not silently discard the user's selected project or rerun analysis to make counts match. Target and ten selected-project file hashes remained unchanged; the target was not executed. One fixture and host do not establish general provider discovery behavior. Related recovery discussion: #8208.

@morluto commented on GitHub (Oct 31, 2026): ### Selected-project discovery coverage control I repeated the overview on the same source-owned arm64 Mach-O using real Ghidra 12.1.4 MCP at REA `b43c329df779708e43ae297f498a844e45d9c923`. A fresh import returned three procedure rows and `procedure_count: 3`; a separately selected project created with Ghidra `-noanalysis` returned one row and `procedure_count: 1`. Both returned 24 defined-string rows. The selected-project overview Evidence warned that original analysis completeness/settings were unknown and that REA did not rerun analysis. The existing `list_procedures({detail:"summary"})` path already exposes the useful distinction: its retained inventory summary reported `row_count: 1`, `discovery_coverage: "unknown"`, and a page with `exhausted: true` for the one retained row. The fresh import also reported discovery coverage unknown. A literal search found `_evaluate` in the fresh import and returned no rows in the selected project; that absence describes its retained inventory, not an assertion about the source file. This narrows the design choice: the overview could potentially reuse the existing inventory-scope vocabulary rather than invent another meaning for completeness. Do not silently discard the user's selected project or rerun analysis to make counts match. Target and ten selected-project file hashes remained unchanged; the target was not executed. One fixture and host do not establish general provider discovery behavior. Related recovery discussion: #8208.

bsakaguchi commented

2026-11-01 21:11:56 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

A controlled producer comparison on current REA a989c0e illustrates what the overview count means. Using the same owned Mach-O, same host and SLEIGH tree, two source-built Kuna CLIs differed only by a diagnostic suppression of Mach-O export promotion (details in #8168). Real MCP returned:

Kuna build	Overview procedures	`list_procedures` rows	Overview strings
Unmodified upstream `d85e7cc`	4	4	0
Diagnostic variant	3	3	0

The one-row difference was the exported data global. Both inventories still included __mh_execute_header, so the diagnostic variant is not an independently correct function inventory and is not a candidate fix. The same known code functions appeared in both. This strengthens the case that the scalar is an exact count of the selected producer's returned inventory, not proof of discovered callable functions; it does not measure autonomous-agent decisions. Both source-built executables reported version 0.1.0/unverified, the target hash remained unchanged, and the target was not executed.

@morluto commented on GitHub (Oct 31, 2026): A controlled producer comparison on current REA `a989c0e` illustrates what the overview count means. Using the same owned Mach-O, same host and SLEIGH tree, two source-built Kuna CLIs differed only by a diagnostic suppression of Mach-O export promotion (details in [#8168](https://github.com/morluto/rea/issues/8168#issuecomment-6545057115)). Real MCP returned: | Kuna build | Overview procedures | `list_procedures` rows | Overview strings | | --- | ---: | ---: | ---: | | Unmodified upstream `d85e7cc` | 4 | 4 | 0 | | Diagnostic variant | 3 | 3 | 0 | The one-row difference was the exported data global. Both inventories still included `__mh_execute_header`, so the diagnostic variant is **not** an independently correct function inventory and is not a candidate fix. The same known code functions appeared in both. This strengthens the case that the scalar is an exact count of the selected producer's returned inventory, not proof of discovered callable functions; it does not measure autonomous-agent decisions. Both source-built executables reported version `0.1.0`/unverified, the target hash remained unchanged, and the target was not executed.

bsakaguchi commented

2026-11-01 21:11:56 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 31, 2026):

Proposal (I'm picking this up). It reuses the existing inventory vocabulary rather than adding a new meaning:

Description: replace "exhaustive procedure/string counts" with counts of every row in the bound provider's procedure and string inventories, across all pages. Say that these are not a discovery of every function or string in the file, and are not comparable across providers.
Result: add one small provider-neutral record beside the counts, produced once in the shared projectBinaryOverview, so Ghidra's direct summary and the generic fallback both carry it:
```
"inventory_coverage": { "scope": "provider_inventory", "discovery_coverage": "unknown" }
```
discovery_coverage is the same "unknown" literal that list_procedures({detail:"summary"}) already reports in nativeInventorySummarySchema. scope names what the counts measure. No provider reports discovery completeness today, so nothing claims complete; a provider that can establish it can extend the enum later.
Unchanged: procedure_count / string_count stay exact totals over the rows the provider returned (#28). Segment fields keep their own availability semantics. Existing limitations are not suppressed.

Cost: about 80 bytes per overview and one short schema property. Verification: overview contract tests on both producer paths (direct summary and listing fallback), CLI and MCP output schema parity, and an agent-facing description check. Real Ghidra/Kuna runs aren't available on my host, so I won't claim real-provider verification.

@kaoru0822-kitauji commented on GitHub (Oct 31, 2026): Proposal (I'm picking this up). It reuses the existing inventory vocabulary rather than adding a new meaning: - **Description:** replace "exhaustive procedure/string counts" with counts of every row in the bound provider's procedure and string inventories, across all pages. Say that these are not a discovery of every function or string in the file, and are not comparable across providers. - **Result:** add one small provider-neutral record beside the counts, produced once in the shared `projectBinaryOverview`, so Ghidra's direct summary and the generic fallback both carry it: ```json "inventory_coverage": { "scope": "provider_inventory", "discovery_coverage": "unknown" } ``` `discovery_coverage` is the same `"unknown"` literal that `list_procedures({detail:"summary"})` already reports in `nativeInventorySummarySchema`. `scope` names what the counts measure. No provider reports discovery completeness today, so nothing claims `complete`; a provider that can establish it can extend the enum later. - **Unchanged:** `procedure_count` / `string_count` stay exact totals over the rows the provider returned (#28). Segment fields keep their own availability semantics. Existing limitations are not suppressed. Cost: about 80 bytes per overview and one short schema property. Verification: overview contract tests on both producer paths (direct summary and listing fallback), CLI and MCP output schema parity, and an agent-facing description check. Real Ghidra/Kuna runs aren't available on my host, so I won't claim real-provider verification.

bsakaguchi commented

2026-11-01 21:11:56 +00:00

Author

Owner

@morluto commented on GitHub (Oct 31, 2026):

Post-merge real Ghidra/Kuna and CLI/MCP verification, including pre-change Ghidra snapshots and offline Evidence closure, is recorded at https://github.com/morluto/rea/pull/8338#issuecomment-6547324476. On the tested fixture the new scope and unknown discovery coverage survived the fresh, selected-project, replay and export/import paths. The separate Kuna classification issue #8352 remains; inventory scope does not assert that every candidate was correctly classified.

@morluto commented on GitHub (Oct 31, 2026): Post-merge real Ghidra/Kuna and CLI/MCP verification, including pre-change Ghidra snapshots and offline Evidence closure, is recorded at https://github.com/morluto/rea/pull/8338#issuecomment-6547324476. On the tested fixture the new scope and unknown discovery coverage survived the fresh, selected-project, replay and export/import paths. The separate Kuna classification issue #8352 remains; inventory scope does not assert that every candidate was correctly classified.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6524

No description provided.

Rows
Columns