morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #5390] [MERGED] feat(native): match retained function inventories across builds #6916

New issue

Closed

opened 2026-11-01 21:44:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-11-01 21:44:02 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/5390
Author: @gensokyo-dev
Created: 10/20/2026
Status: ✅ Merged
Merged: 10/20/2026
Merged by: @gensokyo-dev

Base: main ← Head: feat/5343-function-matching

📝 Commits (7)

4fb7fd2 feat(native): match retained function inventories across builds
6388761 Merge remote-tracking branch 'origin/main' into feat/5343-function-matching
0750e72 fix(native): reuse canonical inventory identity and preserve unknown facets
30b5f37 fix(native): retain unknown origin for unmatched symbol aliases
3d835bc fix(native): preserve matching provenance and bounded facts
b878685 Merge commit 'd532b18123' into feat/5343-function-matching
27fa8c9 fix(native): preserve conflicting symbol origins

📊 Changes

38 files changed (+3422 additions, -72 deletions)

View changed files

📝 .agents/skills/reverse-engineer-anything/references/native-and-artifacts.md (+11 -0)
📝 docs/mcp-contracts.md (+4 -0)
➕ docs/native-function-matching.md (+120 -0)
➕ src/application/native/NativeFunctionMatchingService.test.ts (+707 -0)
➕ src/application/native/NativeFunctionMatchingService.ts (+250 -0)
➕ src/application/native/NativeFunctionMatchingViews.test.ts (+101 -0)
➕ src/application/native/NativeMatchingAdmission.ts (+60 -0)
➕ src/application/native/NativeMatchingBundleSources.ts (+165 -0)
📝 src/cli.ts (+2 -0)
📝 src/cli/analysisEvidenceDelivery.ts (+10 -1)
➕ src/cli/nativeFunctionMatchingCommands.ts (+103 -0)
📝 src/cliCommandNames.ts (+1 -0)
📝 src/contracts/evidenceInputContracts.ts (+5 -18)
➕ src/contracts/native/functionMatchingContracts.ts (+30 -0)
📝 src/contracts/sessionToolContracts.ts (+7 -1)
📝 src/contracts/toolContractExamples.ts (+4 -0)
📝 src/contracts/toolEffects.ts (+1 -0)
📝 src/contracts/toolOutputSchemaGroups.ts (+10 -9)
📝 src/contracts/toolOutputSchemaPrimitives.ts (+2 -15)
📝 src/domain/analysisView/analysisView.ts (+27 -0)

...and 18 more files

📄 Description

Summary

Fixes #5343. match_native_functions pairs two observed native inventories before callers select compare_functions: unique exact symbols first, then complete identical bytes. Supplied compatible CFG/call/string signatures propose unresolved candidates only. Unpaired observations never establish source addition/removal or behavioral equivalence.

One shared application workflow serves CLI/MCP, using retained or portable Evidence without target execution, provider startup or all-function decompilation. Existing dossier pairs include directly usable comparison inputs.
Reuses main's native inventory owner and source_index identity. Batch IDs/names, fields and paging retain complete Evidence. Duplicate candidates use connected groups instead of Cartesian expansion; immutable-parent validation/indexes and one recent filter are reused.
Preserves address spaces, complete inclusive body coverage, partial/conflicting reads, unknown facets, contradictory symbol origins and ambiguous dossier selection. Repeated provenance expansion is admitted before allocation within a shared 96 MiB representation budget; cancellation yields cooperatively.
CLI indexes bounded graph metadata and retains the selected transitive Evidence closure, rejecting missing/changed dependencies and excluding unrelated payloads. Exports are exclusive. Two-input views have no fabricated artifact path. Existing file/process cleanup owners remain responsible.

Workflow, examples and limitations. close_binary clears the ledger: export before closing, then import both bundles. No new execution/injection/mutation/protection-recovery scope or dependencies.

Executed validation

Head 27fa8c918, main d532b1812 incorporated:

212 source tests, 5 public CLI/MCP tests, advertised SDK/AJV2020 schemas, docs checks and 5 README tests passed. Typecheck/lint and pre-push checks passed. Full npm run check passed at 3d835bc13 before the final symbol-conflict follow-up; final static checks also run in CI.
Actual Ghidra 12.1.4 / JDK21.0.12.1+1, Linux x86-64, self-owned v1/v2/stripped C fixtures and official zlib minigzip 1.2.13/1.3.1 (GCC12.2.0, static, -O2 -g0). Exact-name/byte-renamed, changed/inlined, aliases, changed-renamed structural candidates and selected compare_functions inputs passed. Original artifact hashes stayed unchanged.
Independent stdlib ELF64 decoding, GNU nm fixture extents and raw file/memory text equality verified 323 per-result body fingerprints. This does not prove exhaustive discovery. Full acquisition/verification at 0750e725a: 116.83 s wall /194.10 s family CPU /2.344 GiB sampled RSS. Final-head authenticated historical replay: 8.60 s /10.76 s CPU /0.949 GiB; fresh final-head Ghidra acquisition is measured below.
Final public CLI read two real zlib bundles and exclusively published 402 Evidence records; every link resolved, sources stayed unchanged and the matching identity agreed with MCP: 6.07 s /6.04 s CPU /0.393 GiB (read/match/publish only). Synthetic regressions separately cover transitive ancestors across files, missing closure, symbol conflicts, consistent duplicate dossiers and capacity rejection before large link expansion.
Latest-main integration preview f02e79524 (base e84468c0b, tree ec7030643d81b820cf9e1c64f929a4575f91eeda): 212 source +5 public CLI/MCP tests passed. Fresh Ghidra open/inventory/export/close/import/selected-matching task also passed: 108.18 s /93.43 s CPU /1.619 GiB, no resource stops. This separate integration run is not a direct latency comparison with the head benchmarks below.

Completed-task measurements

Final-head tokens, latency, CPU, memory and warm reads

Task: summarize exact correspondence within two observed inventories and read one named pair with identity/provenance. Full versus summary plus selected function page use the same revision; no earlier matcher existed. Reuses main's native-inventory summaries (credit #5378).

Tokenx estimates include used input declarations, arguments, complete replies and extra RPCs. Actual model usage/rendering/tool selection is unmeasured. Whole-wire artifacts also count catalog output schemas.

Inputs	Full tokens	Selected tokens	All input declarations: full / selected
5 generated/side	11,420	24,078	245,359 / 243,819
Real zlib, 230/231 functions	207,267	27,280	441,206 / 247,021
2,001 generated/side	1,777,871	24,083	2,011,810 / 243,824

One fresh process per variant/workload, including client/catalog validation, imports, token estimation and teardown. Historical inputs exclude acquisition. Fresh Ghidra includes sequential open, inventory, export, close, import and matching: 254,177 → 34,236 used-declaration tokens (486,734 → 252,595 with all input declarations).

Task / delivery	Wall s	Family CPU s	Sampled family RSS GiB
5 generated/side / full	5.40	6.50	0.638
5 generated/side / selected	8.28	10.03	0.866
Real zlib, 230/231 functions / full	4.81	5.60	0.641
Real zlib, 230/231 functions / selected	8.13	9.69	0.897
2,001 generated/side / full	6.80	8.18	0.904
2,001 generated/side / selected	9.56	11.02	0.848
Fresh Ghidra zlib / full	46.89	79.46	1.565
Fresh Ghidra zlib / selected	55.06	91.32	1.523

Separate retained reads after removing the private source bundle: full pages [4234.88, 1998.08, 1707.43] ms, selected pair [17.36, 17.49, 21.4] ms. The first full probe includes first exposure to the generic view schema; use the last two for warm comparison. Probes are excluded from primary counters.

Small inputs regress in used-schema tokens, latency and memory; complete detail can be cheaper. First selected reads incur extra schema/RPC cost. No universal cold-speed or memory improvement is claimed. Earlier alternating runs at 0750e725a support scoped context reduction/reuse.

Node24.18/Linux, two CPUs, 3 GiB V8 heaps, 2 GiB Ghidra heaps, 8 GiB family stop guard; no retries/resource stops. RSS sampled every20ms sums process-family RSS, potentially double-counting shared pages/missing short peaks. Runtime caches fresh; OS caches/host load uncontrolled. Engine-only costs unmeasured. Generated histories qualify representation stress, not real providers. The 96 MiB admission budget is not a total-RSS bound.

Expected effects:

Read existing files or captured data.
Change files: only explicitly requested exclusive CLI Evidence output; source inputs are unchanged.

Real provider/platform qualification is Linux/Ghidra only. Other providers/platforms remain unverified. Bytes are not normalized for relocations; missing/incompatible profiles, unsupported CFG spaces, incomplete calls or uninformative structural facts cannot establish structural candidates. Full Evidence and limitations remain available.

Author readiness

Human owner: @gensokyo-dev

The human owner explicitly authorized this work, delegated the technical review and validation to Codex, directed these declarations to be recorded on their behalf, and requested ready-for-review submission and normal merge. This records that direction; it does not assert a separate, independent human diff review. Codex performed the documented technical review and checks.

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

This PR addresses a concrete problem or an agreed enhancement.
Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff.
The PR has one focused outcome and the title follows type(scope): outcome.
Related issue is linked, or the reason for not linking one is stated above.
Tests cover changed observable behavior and meaningful failure paths.
Owning docs, contracts, and generated metadata are updated where needed.
User-visible CLI/MCP changes include representative output.
I checked the final diff for secrets, unrelated cleanup, and unsupported claims.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/5390 **Author:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Created:** 10/20/2026 **Status:** ✅ Merged **Merged:** 10/20/2026 **Merged by:** [@gensokyo-dev](https://github.com/gensokyo-dev) **Base:** `main` ← **Head:** `feat/5343-function-matching` --- ### 📝 Commits (7) - [`4fb7fd2`](https://github.com/morluto/rea/commit/4fb7fd2107839ffea14cd5eb803bfaa7d3e9002f) feat(native): match retained function inventories across builds - [`6388761`](https://github.com/morluto/rea/commit/6388761255ae0f925ec2cad0ab61ec65bd60802a) Merge remote-tracking branch 'origin/main' into feat/5343-function-matching - [`0750e72`](https://github.com/morluto/rea/commit/0750e725a5ec01d21fa4a1d9feb9ef150ce53011) fix(native): reuse canonical inventory identity and preserve unknown facets - [`30b5f37`](https://github.com/morluto/rea/commit/30b5f379b328b78a46f98674787d3aef850eb92e) fix(native): retain unknown origin for unmatched symbol aliases - [`3d835bc`](https://github.com/morluto/rea/commit/3d835bc13f6b57bae72563164f5de16201c31620) fix(native): preserve matching provenance and bounded facts - [`b878685`](https://github.com/morluto/rea/commit/b8786850ffd5c857e0850c9c62b6b9d32f9ef15e) Merge commit 'd532b18123916a9e88f98c7880ba4f984e23d715' into feat/5343-function-matching - [`27fa8c9`](https://github.com/morluto/rea/commit/27fa8c9187667f8d5d7728f40f1f8d4e853bf451) fix(native): preserve conflicting symbol origins ### 📊 Changes **38 files changed** (+3422 additions, -72 deletions) <details> <summary>View changed files</summary> 📝 `.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md` (+11 -0) 📝 `docs/mcp-contracts.md` (+4 -0) ➕ `docs/native-function-matching.md` (+120 -0) ➕ `src/application/native/NativeFunctionMatchingService.test.ts` (+707 -0) ➕ `src/application/native/NativeFunctionMatchingService.ts` (+250 -0) ➕ `src/application/native/NativeFunctionMatchingViews.test.ts` (+101 -0) ➕ `src/application/native/NativeMatchingAdmission.ts` (+60 -0) ➕ `src/application/native/NativeMatchingBundleSources.ts` (+165 -0) 📝 `src/cli.ts` (+2 -0) 📝 `src/cli/analysisEvidenceDelivery.ts` (+10 -1) ➕ `src/cli/nativeFunctionMatchingCommands.ts` (+103 -0) 📝 `src/cliCommandNames.ts` (+1 -0) 📝 `src/contracts/evidenceInputContracts.ts` (+5 -18) ➕ `src/contracts/native/functionMatchingContracts.ts` (+30 -0) 📝 `src/contracts/sessionToolContracts.ts` (+7 -1) 📝 `src/contracts/toolContractExamples.ts` (+4 -0) 📝 `src/contracts/toolEffects.ts` (+1 -0) 📝 `src/contracts/toolOutputSchemaGroups.ts` (+10 -9) 📝 `src/contracts/toolOutputSchemaPrimitives.ts` (+2 -15) 📝 `src/domain/analysisView/analysisView.ts` (+27 -0) _...and 18 more files_ </details> ### 📄 Description ## Summary Fixes #5343. `match_native_functions` pairs two observed native inventories before callers select `compare_functions`: unique exact symbols first, then complete identical bytes. Supplied compatible CFG/call/string signatures propose unresolved candidates only. Unpaired observations never establish source addition/removal or behavioral equivalence. - One shared application workflow serves CLI/MCP, using retained or portable Evidence without target execution, provider startup or all-function decompilation. Existing dossier pairs include directly usable comparison inputs. - Reuses main's native inventory owner and `source_index` identity. Batch IDs/names, fields and paging retain complete Evidence. Duplicate candidates use connected groups instead of Cartesian expansion; immutable-parent validation/indexes and one recent filter are reused. - Preserves address spaces, complete inclusive body coverage, partial/conflicting reads, unknown facets, contradictory symbol origins and ambiguous dossier selection. Repeated provenance expansion is admitted before allocation within a shared 96 MiB representation budget; cancellation yields cooperatively. - CLI indexes bounded graph metadata and retains the selected transitive Evidence closure, rejecting missing/changed dependencies and excluding unrelated payloads. Exports are exclusive. Two-input views have no fabricated artifact path. Existing file/process cleanup owners remain responsible. [Workflow, examples and limitations](https://github.com/morluto/rea/blob/main/docs/native-function-matching.md). `close_binary` clears the ledger: export before closing, then import both bundles. No new execution/injection/mutation/protection-recovery scope or dependencies. ## Executed validation Head `27fa8c918`, main `d532b1812` incorporated: - **212 source tests**, **5 public CLI/MCP tests**, advertised SDK/AJV2020 schemas, docs checks and **5 README tests** passed. Typecheck/lint and pre-push checks passed. Full `npm run check` passed at `3d835bc13` before the final symbol-conflict follow-up; final static checks also run in CI. - Actual **Ghidra 12.1.4 / JDK21.0.12.1+1, Linux x86-64**, self-owned v1/v2/stripped C fixtures and official zlib minigzip 1.2.13/1.3.1 (GCC12.2.0, static, `-O2 -g0`). Exact-name/byte-renamed, changed/inlined, aliases, changed-renamed structural candidates and selected `compare_functions` inputs passed. Original artifact hashes stayed unchanged. - Independent stdlib ELF64 decoding, GNU nm fixture extents and raw file/memory text equality verified **323 per-result body fingerprints**. This does not prove exhaustive discovery. Full acquisition/verification at `0750e725a`: 116.83 s wall /194.10 s family CPU /2.344 GiB sampled RSS. Final-head authenticated historical replay: **8.60 s /10.76 s CPU /0.949 GiB**; fresh final-head Ghidra acquisition is measured below. - Final public CLI read two real zlib bundles and exclusively published **402 Evidence records**; every link resolved, sources stayed unchanged and the matching identity agreed with MCP: **6.07 s /6.04 s CPU /0.393 GiB** (read/match/publish only). Synthetic regressions separately cover transitive ancestors across files, missing closure, symbol conflicts, consistent duplicate dossiers and capacity rejection before large link expansion. - Latest-main integration preview `f02e79524` (base `e84468c0b`, tree `ec7030643d81b820cf9e1c64f929a4575f91eeda`): **212 source +5 public CLI/MCP tests passed**. Fresh Ghidra open/inventory/export/close/import/selected-matching task also passed: **108.18 s /93.43 s CPU /1.619 GiB**, no resource stops. This separate integration run is not a direct latency comparison with the head benchmarks below. ## Completed-task measurements <details> <summary>Final-head tokens, latency, CPU, memory and warm reads</summary> Task: summarize exact correspondence within two observed inventories and read one named pair with identity/provenance. Full versus summary plus selected function page use the same revision; no earlier matcher existed. Reuses main's native-inventory summaries (credit #5378). Tokenx estimates include used input declarations, arguments, complete replies and extra RPCs. Actual model usage/rendering/tool selection is unmeasured. Whole-wire artifacts also count catalog output schemas. | Inputs | Full tokens | Selected tokens | All input declarations: full / selected | | --- | ---: | ---: | ---: | | 5 generated/side | 11,420 | 24,078 | 245,359 / 243,819 | | Real zlib, 230/231 functions | 207,267 | 27,280 | 441,206 / 247,021 | | 2,001 generated/side | 1,777,871 | 24,083 | 2,011,810 / 243,824 | One fresh process per variant/workload, including client/catalog validation, imports, token estimation and teardown. Historical inputs exclude acquisition. Fresh Ghidra includes sequential open, inventory, export, close, import and matching: **254,177 → 34,236** used-declaration tokens (486,734 → 252,595 with all input declarations). | Task / delivery | Wall s | Family CPU s | Sampled family RSS GiB | | --- | ---: | ---: | ---: | | 5 generated/side / full | 5.40 | 6.50 | 0.638 | | 5 generated/side / selected | 8.28 | 10.03 | 0.866 | | Real zlib, 230/231 functions / full | 4.81 | 5.60 | 0.641 | | Real zlib, 230/231 functions / selected | 8.13 | 9.69 | 0.897 | | 2,001 generated/side / full | 6.80 | 8.18 | 0.904 | | 2,001 generated/side / selected | 9.56 | 11.02 | 0.848 | | Fresh Ghidra zlib / full | 46.89 | 79.46 | 1.565 | | Fresh Ghidra zlib / selected | 55.06 | 91.32 | 1.523 | Separate retained reads after removing the private source bundle: full pages **[4234.88, 1998.08, 1707.43] ms**, selected pair **[17.36, 17.49, 21.4] ms**. The first full probe includes first exposure to the generic view schema; use the last two for warm comparison. Probes are excluded from primary counters. Small inputs regress in used-schema tokens, latency and memory; complete detail can be cheaper. First selected reads incur extra schema/RPC cost. No universal cold-speed or memory improvement is claimed. Earlier alternating runs at `0750e725a` support scoped context reduction/reuse. Node24.18/Linux, two CPUs, 3 GiB V8 heaps, 2 GiB Ghidra heaps, 8 GiB family stop guard; no retries/resource stops. RSS sampled every20ms sums process-family RSS, potentially double-counting shared pages/missing short peaks. Runtime caches fresh; OS caches/host load uncontrolled. Engine-only costs unmeasured. Generated histories qualify representation stress, not real providers. The 96 MiB admission budget is not a total-RSS bound. </details> Expected effects: - [x] Read existing files or captured data. - [x] Change files: only explicitly requested exclusive CLI Evidence output; source inputs are unchanged. Real provider/platform qualification is Linux/Ghidra only. Other providers/platforms remain unverified. Bytes are not normalized for relocations; missing/incompatible profiles, unsupported CFG spaces, incomplete calls or uninformative structural facts cannot establish structural candidates. Full Evidence and limitations remain available. ## Author readiness Human owner: @gensokyo-dev The human owner explicitly authorized this work, delegated the technical review and validation to Codex, directed these declarations to be recorded on their behalf, and requested ready-for-review submission and normal merge. This records that direction; it does not assert a separate, independent human diff review. Codex performed the documented technical review and checks. - [x] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [x] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [x] I declare this PR ready for initial maintainer review. ## Before merge - [x] This PR addresses a concrete problem or an agreed enhancement. - [x] Material scope and authority questions are resolved, and any required maintainer scope decision covers the final diff. - [x] The PR has one focused outcome and the title follows `type(scope): outcome`. - [x] Related issue is linked, or the reason for not linking one is stated above. - [x] Tests cover changed observable behavior and meaningful failure paths. - [x] Owning docs, contracts, and generated metadata are updated where needed. - [x] User-visible CLI/MCP changes include representative output. - [x] I checked the final diff for secrets, unrelated cleanup, and unsupported claims. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-11-01 21:44:02 +00:00

closed this issue
added the
pull-request
label

bsakaguchi referenced this issue

2026-11-03 13:38:42 +00:00

[PR #6996] [MERGED] feat(guidance): search installed workflow sections #7559

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#6916

No description provided.

Rows
Columns