morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[PR #4559] fix(native): scale NativeAOT work budget and resolve base types by index #6307

New issue

Open

opened 2026-10-31 01:13:27 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 01:13:27 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/morluto/rea/pull/4559
Author: @Nitin13-git
Created: 10/16/2026
Status: 🔄 Open

Base: main ← Head: fix/nativeaot-work-budget

📝 Commits (3)

0b58e58 fix(native): scale NativeAOT work budget and resolve base types by index
acb529a fix(native): reserve closure queue memory before allocating it
ddf9bae Merge origin/main into fix/nativeaot-work-budget

📊 Changes

5 files changed (+221 additions, -60 deletions)

View changed files

📝 src/domain/native/nativeAotPe.budget.test.ts (+80 -0)
📝 src/domain/native/nativeAotPe.ts (+113 -51)
📝 src/ghidra/GhidraAnalysisProfile.ts (+5 -2)
📝 src/ghidra/GhidraProvider.behavior.test.ts (+3 -2)
📝 tests/fixtures/nativeaotPe.ts (+20 -5)

📄 Description

Summary

Fixes #4445: NativeAOT metadata recovery returned no MethodTables, types or frozen strings for a real 26 MB .NET 8 application because of a fixed work budget and a quadratic base-type closure.

Problem and expected behavior

On bflat.exe 8.0.2 (sha256 09c8a41e…93d3a2, the image from the issue) the parser returned partial with work-unit-budget truncation. Two causes, measured by running parseNativeAotPe directly on that file:

The pointer scan alone needs about 71M work units; the budget was 64 MiB (about 67M).
After the scan, the inclusion closure rescanned every candidate for each included base, so cost grew with the square of the candidate count (about 13k candidates, over 1G units). Raising the budget 16x still did not finish (12.7 s, truncated).

Expected: an ordinary large NativeAOT app recovers within bounded work, and truncation still reports honestly when a real bound is hit.

Change and scope

nativeAotPe.ts: work budget is now 64 MiB + 8 units per admitted source byte (nativeAotParserLimitsFor), bounded by the existing 128 MiB source ceiling. Descendants are resolved through a sorted related-address index and a queue, so each candidate joins once. Index and queue memory is charged to the existing working-memory budget, and index construction is charged to the work budget.
GhidraAnalysisProfile.ts: the profile records base_work_units and work_units_per_source_byte and the contract revision moves to rtr-9.1-x64-pe-read-only-v6 (main was at v5 when this branch last merged it, so the new budget policy is a further revision), so analysis keyed on the old budget is not reused.
nativeAotSnapshotOverCapacity now defaults to the ceiling limits, so an over-size report shows a work budget of about 1.14 billion units (64 MiB base + 8 units x 128 MiB) instead of the old 67 million. The report is otherwise unchanged.
Not included: a user-facing setting for the budget, and the separate question of why the PDB lists about 62k MethodTable symbols while the parser recovers about 13k (the PDB count may include non-unique or non-type symbols; not investigated).
Scope decision: not applicable (existing read-only parser, no new capability).

Contract and boundary impact

Semantic owner and earliest changed stage: NativeAOT PE parser (domain), profile identity (Ghidra adapter)
CLI and MCP/tool-catalog contract: none; reports gain more complete results
Provider/platform compatibility: Ghidra PE profiles; profile digest changes for PE targets
Evidence/provenance contract: coverage fields unchanged; work_budget_units now reflects the scaled budget
Read existing files or captured data.

Evidence and regression coverage

Executed on macOS arm64, Node 24.18.0, directly against the real image (parser only; no Ghidra run):

	status	MethodTables	frozen strings	work units	time
before (64 MiB)	partial	0	0	67,108,860 (exhausted)	1.2 s
before (16x budget)	partial	11,977	0	exhausted	12.7 s
after	complete	12,973	6,685	87,585,929 of 276,549,632	1.4 s, RSS 142 MB

The 6,685 frozen strings equal the PDB's __Str_* count in the issue.

New tests in nativeAotPe.budget.test.ts: budget scaling and ceiling, and 1,500 descendants of one base completing within 2M units (the old closure needs about 18M); both fail on the base revision.
GhidraProvider.behavior.test.ts updated for the new profile fields; nativeAotPe*.test.ts, ghidraInventoryMcp.test.ts and npm run check:fast pass.
Not run: real Ghidra import, the optional adapter JAR, Windows, full check:pr. Worst-case cost at the 128 MiB ceiling is not measured; the budget there is about 1.1G units, roughly 12 s at the measured throughput (estimate).

Validation performed

Run on macOS arm64, Node 24.18.0, on the branch merged with origin/main at 969aafcf2 (the head ddf9bae):

Real image from #4445 (bflat.exe 8.0.2, SHA-256 09c8a41e...93d3a2), parser only: complete, 12,973 MethodTables, 6,685 frozen strings, 0.7 s. The same image on main is partial with 0 types under the fixed 64 MiB budget.
nativeAotPe.budget.test.ts, nativeAotPe.test.ts and GhidraProvider.behavior.test.ts: 141 passed. NativeAOT header-range, section-range, recovery, snapshot, load-image-attest, extension and config tests: 127 passed. ghidraInventoryMcp, ghidraFailureDiagnostics and ghidraProfileArtifactOwnership: 30 passed.
npm run check:fast: exit 0.
CI on ddf9bae8: all checks passed, including Portable tests and CI required.
Not run: npm run check:pr, a real Ghidra import of the NativeAOT image, Windows.

Compatibility, safety, and release

Breaking changes or migration steps: none for callers. PE analysis profile identity changes (contract revision v6), so cached analysis from earlier revisions is not reused.
Worst-case parser work grows with the admitted image size, bounded by the existing 128 MiB source ceiling (about 1.14 billion work units, roughly 12 s at the measured throughput; an estimate, not a measurement at the ceiling).
Real Hopper/Ghidra, browser, or OS coverage: none of this PR's parse path was run through a real Ghidra import.
Package or release metadata impact: none.
Security, privacy, process, or containment review: not applicable; read-only parsing within existing resource bounds.

Author readiness

Human owner:

I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
I declare this PR ready for initial maintainer review.

Before merge

Scope/authority and applicable lifecycle verification resolved.
Latest-head CI and actionable review findings assessed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/morluto/rea/pull/4559 **Author:** [@Nitin13-git](https://github.com/Nitin13-git) **Created:** 10/16/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/nativeaot-work-budget` --- ### 📝 Commits (3) - [`0b58e58`](https://github.com/morluto/rea/commit/0b58e58cde1d9cb32f9b9072122a0e4ad89ea8bf) fix(native): scale NativeAOT work budget and resolve base types by index - [`acb529a`](https://github.com/morluto/rea/commit/acb529a8ed683cd05e82c138e26180a8a58343e4) fix(native): reserve closure queue memory before allocating it - [`ddf9bae`](https://github.com/morluto/rea/commit/ddf9bae8220f91f24664669762fdbba0787f729d) Merge origin/main into fix/nativeaot-work-budget ### 📊 Changes **5 files changed** (+221 additions, -60 deletions) <details> <summary>View changed files</summary> 📝 `src/domain/native/nativeAotPe.budget.test.ts` (+80 -0) 📝 `src/domain/native/nativeAotPe.ts` (+113 -51) 📝 `src/ghidra/GhidraAnalysisProfile.ts` (+5 -2) 📝 `src/ghidra/GhidraProvider.behavior.test.ts` (+3 -2) 📝 `tests/fixtures/nativeaotPe.ts` (+20 -5) </details> ### 📄 Description ## Summary Fixes #4445: NativeAOT metadata recovery returned no MethodTables, types or frozen strings for a real 26 MB .NET 8 application because of a fixed work budget and a quadratic base-type closure. ## Problem and expected behavior On `bflat.exe` 8.0.2 (sha256 `09c8a41e…93d3a2`, the image from the issue) the parser returned `partial` with `work-unit-budget` truncation. Two causes, measured by running `parseNativeAotPe` directly on that file: - The pointer scan alone needs about 71M work units; the budget was 64 MiB (about 67M). - After the scan, the inclusion closure rescanned every candidate for each included base, so cost grew with the square of the candidate count (about 13k candidates, over 1G units). Raising the budget 16x still did not finish (12.7 s, truncated). Expected: an ordinary large NativeAOT app recovers within bounded work, and truncation still reports honestly when a real bound is hit. ## Change and scope - `nativeAotPe.ts`: work budget is now `64 MiB + 8 units per admitted source byte` (`nativeAotParserLimitsFor`), bounded by the existing 128 MiB source ceiling. Descendants are resolved through a sorted related-address index and a queue, so each candidate joins once. Index and queue memory is charged to the existing working-memory budget, and index construction is charged to the work budget. - `GhidraAnalysisProfile.ts`: the profile records `base_work_units` and `work_units_per_source_byte` and the contract revision moves to `rtr-9.1-x64-pe-read-only-v6` (`main` was at v5 when this branch last merged it, so the new budget policy is a further revision), so analysis keyed on the old budget is not reused. - `nativeAotSnapshotOverCapacity` now defaults to the ceiling limits, so an over-size report shows a work budget of about 1.14 billion units (64 MiB base + 8 units x 128 MiB) instead of the old 67 million. The report is otherwise unchanged. - Not included: a user-facing setting for the budget, and the separate question of why the PDB lists about 62k MethodTable symbols while the parser recovers about 13k (the PDB count may include non-unique or non-type symbols; not investigated). - Scope decision: not applicable (existing read-only parser, no new capability). ## Contract and boundary impact - Semantic owner and earliest changed stage: NativeAOT PE parser (domain), profile identity (Ghidra adapter) - CLI and MCP/tool-catalog contract: none; reports gain more complete results - Provider/platform compatibility: Ghidra PE profiles; profile digest changes for PE targets - Evidence/provenance contract: coverage fields unchanged; `work_budget_units` now reflects the scaled budget - [x] Read existing files or captured data. ## Evidence and regression coverage Executed on macOS arm64, Node 24.18.0, directly against the real image (parser only; no Ghidra run): | | status | MethodTables | frozen strings | work units | time | | --- | --- | ---: | ---: | ---: | ---: | | before (64 MiB) | partial | 0 | 0 | 67,108,860 (exhausted) | 1.2 s | | before (16x budget) | partial | 11,977 | 0 | exhausted | 12.7 s | | after | complete | 12,973 | 6,685 | 87,585,929 of 276,549,632 | 1.4 s, RSS 142 MB | The 6,685 frozen strings equal the PDB's `__Str_*` count in the issue. - New tests in `nativeAotPe.budget.test.ts`: budget scaling and ceiling, and 1,500 descendants of one base completing within 2M units (the old closure needs about 18M); both fail on the base revision. - `GhidraProvider.behavior.test.ts` updated for the new profile fields; `nativeAotPe*.test.ts`, `ghidraInventoryMcp.test.ts` and `npm run check:fast` pass. - Not run: real Ghidra import, the optional adapter JAR, Windows, full `check:pr`. Worst-case cost at the 128 MiB ceiling is not measured; the budget there is about 1.1G units, roughly 12 s at the measured throughput (estimate). ## Validation performed Run on macOS arm64, Node 24.18.0, on the branch merged with `origin/main` at `969aafcf2` (the head `ddf9bae`): - Real image from #4445 (`bflat.exe` 8.0.2, SHA-256 `09c8a41e...93d3a2`), parser only: `complete`, 12,973 MethodTables, 6,685 frozen strings, 0.7 s. The same image on `main` is `partial` with 0 types under the fixed 64 MiB budget. - `nativeAotPe.budget.test.ts`, `nativeAotPe.test.ts` and `GhidraProvider.behavior.test.ts`: 141 passed. NativeAOT header-range, section-range, recovery, snapshot, load-image-attest, extension and config tests: 127 passed. `ghidraInventoryMcp`, `ghidraFailureDiagnostics` and `ghidraProfileArtifactOwnership`: 30 passed. - `npm run check:fast`: exit 0. - CI on `ddf9bae8`: all checks passed, including `Portable tests` and `CI required`. - Not run: `npm run check:pr`, a real Ghidra import of the NativeAOT image, Windows. ## Compatibility, safety, and release - Breaking changes or migration steps: none for callers. PE analysis profile identity changes (contract revision v6), so cached analysis from earlier revisions is not reused. - Worst-case parser work grows with the admitted image size, bounded by the existing 128 MiB source ceiling (about 1.14 billion work units, roughly 12 s at the measured throughput; an estimate, not a measurement at the ceiling). - Real Hopper/Ghidra, browser, or OS coverage: none of this PR's parse path was run through a real Ghidra import. - Package or release metadata impact: none. - Security, privacy, process, or containment review: not applicable; read-only parsing within existing resource bounds. ## Author readiness Human owner: - [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work. - [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps. - [ ] I declare this PR ready for initial maintainer review. ## Before merge - [ ] Scope/authority and applicable lifecycle verification resolved. - [ ] Latest-head CI and actionable review findings assessed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-31 01:13:27 +00:00

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425

bsakaguchi referenced this issue

2026-11-01 21:46:39 +00:00

[PR #6307] test(correlation): cover contradictory findings and retained evidence #7271

bsakaguchi referenced this issue

2026-11-03 13:39:11 +00:00