[GH-ISSUE #6406] [Feature] match_native_functions: admit inventories of 20k–50k functions per side (Ghidra-shaped rows are refused above ~10k) #3838

New issue

Closed

opened 2026-10-31 00:44:31 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:44:31 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 22, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6406

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Native analysis or decompilation

Current limitation

match_native_functions (merged in #5390; on main only, not in the 6.7.0 release) refuses ordinary large builds outright. Its admission charges a fixed 96 MiB "representation allowance" per call, shared by both sides (src/application/native/NativeMatchingAdmission.ts), and the refusal says "This release has no runtime override." docs/native-function-matching.md states that this budget "is not a bound on total process RSS".

Observed at REA main e83e3ceadd by calling the application service that MCP and CLI share (matchNativeFunctions in src/application/native/NativeFunctionMatchingService.ts) with synthetic authenticated list_procedures Evidence, distinct names, same names on both sides:

Rows per side	Row shape	Result	Elapsed
19,000	address + name	matched 19,000	484 ms
19,500	address + name	`AnalysisResourceConstraintError`, 96 MiB allowance	153 ms
10,000	address + name + Ghidra `symbol` and `procedure` facts	matched 10,000	491 ms
11,000	same Ghidra-shaped rows	`AnalysisResourceConstraintError`, 96 MiB allowance	355 ms

Peak resident memory of the whole Node process, including building both synthetic inventories and both runs of each pair, was between about 290 MB and 350 MB across two repetitions (the row thresholds and timings were stable; memory varied between runs). So the refusal happens at roughly a third of a gigabyte of real memory and well under a second of work.

Source reading of the accounting (src/domain/native/functionInventory.ts, NativeMatchingAdmission.ts): each row is charged 1024 + 4 × (address + name length), each string 64 + 4 × length, and each function is charged 256 + 8 × id length for every Evidence link, including the same inventory Evidence ID repeated for every function. Byte observations are charged per body byte as well, so exact-byte matching should top out at a few MiB of observed code per side (inferred from source, not run).

Use cases

"Which functions were added, removed, or byte-identical between v1 and v2 of this library?" for common large targets: browser engines, game executables, Electron/Chromium native modules, and firmware images routinely have 20k–100k functions.
Feeding matched pairs to compare_functions for a patch diff of a large product, the workflow this tool was built for (#5343).

Desired outcome and success criteria

Inventories of at least 50,000 Ghidra-shaped functions per side match under the default configuration, or under a documented setting that follows docs/design/limits-and-recovery.md (default equal to today's constant, floor and maximum, copied by rea setup, named in the refusal text).
The admission charge reflects retained representation: an inventory-level link that is identical for every function is not charged once per function.
Elapsed time and peak process memory are reported for 20k and 50k per side, cold, in the change's verification.
One real pair of large stripped Ghidra inventories is measured end to end through MCP or CLI.
Refusals that remain keep naming the limit and its override, and stay non-retryable (fixedNativeAllowance in src/domain/native/nativeDeadlines.ts already returns retryable: false at the examined revision, so #6035 may already be resolved).

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Input is existing authenticated Evidence (inventories, optional dossiers and byte observations); nothing is executed. Non-goals: fuzzy or similarity matching (#6407), and changing what a match asserts.

Verification available so far

I have a prototype or observed results

Verification details (optional)

Reproduction: in a scratch copy of the repository at the examined revision, run npm ci --ignore-scripts, add a script that builds two list_procedures Evidence records with createEvidence (src/domain/evidence.ts) containing N rows each (address as 0x-hex, value as subsystem_function_<i>; for the Ghidra-shaped run also symbol: {primary: true, dynamic: false, external: false, type: "Function", source: "imported"} and procedure: {external: false, thunk: false, thunk_target: null}), then call matchNativeFunctions({left, right, detail: "summary"}, {signal}) and run it with tsx. Run on Node 26.12.1, macOS arm64; the repository pins Node 24.18.0, which I did not use. Peak memory came from /usr/bin/time -l. Not run: a real Ghidra inventory pair, MCP/CLI transport, and byte-observation matching.

Alternatives considered

Splitting each inventory into slices and matching slice by slice: breaks the global uniqueness that exact-name and exact-body matching rely on, so results across slices are wrong.
Filtering inventories before matching: requires knowing in advance which functions changed, which is the question being asked.

Possible direction

Add the ceiling to the configurable-ceiling work in #6176 (native matching is not listed there), and/or charge shared links once per collection. The refusal text already follows the "keep a fixed ceiling honest" rule; this proposal is about headroom, not honesty.

References

Related: #5390, #5343, #6035, #6176.

Originally created by @coygeek on GitHub (Oct 22, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6406 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [X] I searched the open and closed issues and pull requests for a related proposal. - [X] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [X] This request does not include credentials, private samples, or sensitive target data. ### Area Native analysis or decompilation ### Current limitation `match_native_functions` (merged in #5390; on `main` only, not in the 6.7.0 release) refuses ordinary large builds outright. Its admission charges a fixed 96 MiB "representation allowance" per call, shared by both sides (`src/application/native/NativeMatchingAdmission.ts`), and the refusal says "This release has no runtime override." `docs/native-function-matching.md` states that this budget "is not a bound on total process RSS". Observed at REA `main` e83e3ceadd97770a50b8afa97f4077916e4d2b81 by calling the application service that MCP and CLI share (`matchNativeFunctions` in `src/application/native/NativeFunctionMatchingService.ts`) with synthetic authenticated `list_procedures` Evidence, distinct names, same names on both sides: | Rows per side | Row shape | Result | Elapsed | |---|---|---|---| | 19,000 | address + name | matched 19,000 | 484 ms | | 19,500 | address + name | `AnalysisResourceConstraintError`, 96 MiB allowance | 153 ms | | 10,000 | address + name + Ghidra `symbol` and `procedure` facts | matched 10,000 | 491 ms | | 11,000 | same Ghidra-shaped rows | `AnalysisResourceConstraintError`, 96 MiB allowance | 355 ms | Peak resident memory of the whole Node process, including building both synthetic inventories and both runs of each pair, was between about 290 MB and 350 MB across two repetitions (the row thresholds and timings were stable; memory varied between runs). So the refusal happens at roughly a third of a gigabyte of real memory and well under a second of work. Source reading of the accounting (`src/domain/native/functionInventory.ts`, `NativeMatchingAdmission.ts`): each row is charged `1024 + 4 × (address + name length)`, each string `64 + 4 × length`, and each function is charged `256 + 8 × id length` for every Evidence link, including the same inventory Evidence ID repeated for every function. Byte observations are charged per body byte as well, so exact-byte matching should top out at a few MiB of observed code per side (inferred from source, not run). ### Use cases - "Which functions were added, removed, or byte-identical between v1 and v2 of this library?" for common large targets: browser engines, game executables, Electron/Chromium native modules, and firmware images routinely have 20k–100k functions. - Feeding matched pairs to `compare_functions` for a patch diff of a large product, the workflow this tool was built for (#5343). ### Desired outcome and success criteria 1. Inventories of at least 50,000 Ghidra-shaped functions per side match under the default configuration, or under a documented setting that follows `docs/design/limits-and-recovery.md` (default equal to today's constant, floor and maximum, copied by `rea setup`, named in the refusal text). 2. The admission charge reflects retained representation: an inventory-level link that is identical for every function is not charged once per function. 3. Elapsed time and peak process memory are reported for 20k and 50k per side, cold, in the change's verification. 4. One real pair of large stripped Ghidra inventories is measured end to end through MCP or CLI. 5. Refusals that remain keep naming the limit and its override, and stay non-retryable (`fixedNativeAllowance` in `src/domain/native/nativeDeadlines.ts` already returns `retryable: false` at the examined revision, so #6035 may already be resolved). ### Expected interaction with targets - Read existing files or captured data ### Target and scope details (optional) Input is existing authenticated Evidence (inventories, optional dossiers and byte observations); nothing is executed. Non-goals: fuzzy or similarity matching (#6407), and changing what a match asserts. ### Verification available so far I have a prototype or observed results ### Verification details (optional) Reproduction: in a scratch copy of the repository at the examined revision, run `npm ci --ignore-scripts`, add a script that builds two `list_procedures` Evidence records with `createEvidence` (`src/domain/evidence.ts`) containing N rows each (`address` as 0x-hex, `value` as `subsystem_function_<i>`; for the Ghidra-shaped run also `symbol: {primary: true, dynamic: false, external: false, type: "Function", source: "imported"}` and `procedure: {external: false, thunk: false, thunk_target: null}`), then call `matchNativeFunctions({left, right, detail: "summary"}, {signal})` and run it with `tsx`. Run on Node 26.12.1, macOS arm64; the repository pins Node 24.18.0, which I did not use. Peak memory came from `/usr/bin/time -l`. Not run: a real Ghidra inventory pair, MCP/CLI transport, and byte-observation matching. ### Alternatives considered - Splitting each inventory into slices and matching slice by slice: breaks the global uniqueness that exact-name and exact-body matching rely on, so results across slices are wrong. - Filtering inventories before matching: requires knowing in advance which functions changed, which is the question being asked. ### Possible direction Add the ceiling to the configurable-ceiling work in #6176 (native matching is not listed there), and/or charge shared links once per collection. The refusal text already follows the "keep a fixed ceiling honest" rule; this proposal is about headroom, not honesty. ### References - Related: #5390, #5343, #6035, #6176.

bsakaguchi closed this issue

2026-10-31 00:44:31 +00:00

bsakaguchi commented

2026-10-31 00:44:32 +00:00

Author

Owner

@kaoru0822-kitauji commented on GitHub (Oct 25, 2026):

Working on this in two independent PRs:

#7108 changes the accounting (criterion 2). Borrowed retained inventories are no longer walked, the shared inventory ID is charged once, and result copies are charged up front so an oversized side is refused before parsing. Retained references go from ~10k to ~22k Ghidra-shaped rows per side under the unchanged 96 MiB default.
#7112 adds native.matching_representation_bytes / REA_NATIVE_MATCHING_REPRESENTATION_BYTES (default 96 MiB, max 1 GiB, copied by setup, named in details.limit) (criteria 1, 3 and 5). At 512 MiB, 50k rows per side matched: CLI 8.7 s, 1.34 GB peak; MCP 2.7 s.

Criterion 4 (a real large stripped Ghidra pair) is not covered: Ghidra isn't available on my host, so all numbers come from synthetic Ghidra-shaped Evidence.

@kaoru0822-kitauji commented on GitHub (Oct 25, 2026): Working on this in two independent PRs: - #7108 changes the accounting (criterion 2). Borrowed retained inventories are no longer walked, the shared inventory ID is charged once, and result copies are charged up front so an oversized side is refused before parsing. Retained references go from ~10k to ~22k Ghidra-shaped rows per side under the unchanged 96 MiB default. - #7112 adds `native.matching_representation_bytes` / `REA_NATIVE_MATCHING_REPRESENTATION_BYTES` (default 96 MiB, max 1 GiB, copied by setup, named in `details.limit`) (criteria 1, 3 and 5). At 512 MiB, 50k rows per side matched: CLI 8.7 s, 1.34 GB peak; MCP 2.7 s. Criterion 4 (a real large stripped Ghidra pair) is not covered: Ghidra isn't available on my host, so all numbers come from synthetic Ghidra-shaped Evidence.

bsakaguchi referenced this issue

2026-10-31 01:10:37 +00:00

[PR #3842] [CLOSED] fix(mcp): give the compact any-value schema an item schema #5942

bsakaguchi referenced this issue

2026-10-31 01:10:48 +00:00

[PR #3921] [CLOSED] fix(mcp): give the compact any-value schema an items schema #5966