morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5890] [Bug] NativeAOT RTR scan omits unvisited sections from omitted-byte coverage #3568

New issue

Closed

opened 2026-10-31 00:41:58 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:41:58 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5890

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

The built-in NativeAOT PE parser understates omitted RTR signature-scan bytes when its work budget stops the scan before the last eligible section. An authored PE with two readable, non-executable data sections returns 1,008 omitted bytes instead of 2,024 with an 8-unit work limit. A larger case using the parser's unchanged default limits returns 1,080 instead of 2,096. Both omit exactly the later section's 1,016 candidate bytes.

This was executed directly against parseNativeAotPe. The supported entry points are CLI rea inspect-native-load-image <target> --provider ghidra --json and MCP inspect_native_load_image with provider ghidra. Their load-image route reaches this parser according to source; a real Ghidra, CLI, or MCP session was not run. The report correctly remains partial and truncated: true.

Reproduction

Save this test as src/domain/native/nativeAotPe.rtrCoverage.test.ts, then run npx vitest run --project domain src/domain/native/nativeAotPe.rtrCoverage.test.ts --maxWorkers 1 with the locked dependencies. It extends the existing authored PE fixture while retaining its RTR9.1 header. The first case is a single-section control; the second adds one later eligible section; the third reaches the production default work limit without changing parser limits.

import { expect, it } from "vitest";
import {
  DEFAULT_NATIVE_AOT_PARSER_LIMITS,
  NATIVE_AOT_MAX_REPORT_BYTES,
  parseNativeAotPe,
} from "./nativeAotPe.js";
import {
  nativeAotPeDigest,
  nativeAotPeFixture,
} from "../../../tests/fixtures/nativeaotPe.js";

const fixture = (firstSize: number, laterSize: number) => {
  const bytes = Buffer.alloc(0x600 + firstSize + laterSize);
  nativeAotPeFixture().copy(bytes);
  const data = 0x98 + 0xf0 + 40;
  bytes.writeUInt32LE(firstSize, data + 8);
  bytes.writeUInt32LE(firstSize, data + 16);
  bytes.writeUInt32LE(0x2000 + firstSize + laterSize, 0x98 + 56);
  if (laterSize > 0) {
    bytes.writeUInt16LE(3, 0x86);
    const later = data + 40;
    bytes.write(".data", later, "ascii");
    bytes.writeUInt32LE(laterSize, later + 8);
    bytes.writeUInt32LE(0x2000 + firstSize, later + 12);
    bytes.writeUInt32LE(laterSize, later + 16);
    bytes.writeUInt32LE(0x600 + firstSize, later + 20);
    bytes.writeUInt32LE(0xc0000040, later + 36);
  }
  return bytes;
};

it.each([
  { firstSize: 0x400, laterSize: 0, workUnits: 8 },
  { firstSize: 0x400, laterSize: 0x400, workUnits: 8 },
  {
    firstSize: 64 * 1024 * 1024 + 0x400,
    laterSize: 0x400,
    workUnits: DEFAULT_NATIVE_AOT_PARSER_LIMITS.workUnits,
  },
])("accounts for all eligible sections: %j", ({ firstSize, laterSize, workUnits }) => {
  const bytes = fixture(firstSize, laterSize);
  const summary = parseNativeAotPe(
    bytes,
    nativeAotPeDigest(bytes),
    NATIVE_AOT_MAX_REPORT_BYTES,
    undefined,
    workUnits === DEFAULT_NATIVE_AOT_PARSER_LIMITS.workUnits
      ? undefined
      : { ...DEFAULT_NATIVE_AOT_PARSER_LIMITS, workUnits },
  ).summary;
  const candidateBytes = firstSize - 8 + (laterSize ? laterSize - 8 : 0);
  const examined = summary.coverage.rtr_signature_scan_bytes_examined!;
  const omitted = summary.coverage.rtr_signature_scan_bytes_omitted;
  console.log(JSON.stringify({ firstSize, laterSize, workUnits, status: summary.status, truncated: summary.truncated, candidateBytes, examined, omitted, expectedOmitted: candidateBytes - examined, coverage: summary.coverage }));
  expect(summary.status).toBe("partial");
  expect(summary.truncated).toBe(true);
  expect(summary.coverage.truncation_reason).toBe("work-unit-budget");
  expect(summary.coverage.scan_bytes_omitted).toBe(omitted);
  expect(omitted).toBe(candidateBytes - examined);
});

The final focused run produced one passing control and two failing omitted-byte assertions. No target executable was launched.

Expected behavior

rtr_signature_scan_bytes_examined + rtr_signature_scan_bytes_omitted should cover the candidate positions across every eligible section, including those skipped after the budget exit. The same applies to scan_bytes_omitted on this early partial outcome. An aligned 1,024-byte section contributes 1,016 candidate bytes under the existing eight-byte stride and 16-byte-header requirement.

The parser limitation text promises examined and unexamined coverage for incomplete scans. The documented evidence boundaries describe the signature scan and independent resource policies. This does not require removing or raising those limits.

Impact and scope

P2: quantitative coverage understates how much of the admitted image was left unexamined. The two authored multi-section cases each lose the entire later section from the omitted count. Both the general and RTR-specific omitted fields are wrong on the early partial result, but partial status and truncation reason remain accurate.

Natural-file frequency and real-provider occurrence are unknown. No recovery workaround was tested. Callers can respect the partial status, but cannot reconstruct the missing section coverage from this report alone. No hydration corruption or false complete result is claimed.

REA version

Source revision 52415975942a4b9c53a817915b313ccc404f280e; package manifest version 6.7.0.

Runtime and platform

macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3. This run used Node 26 rather than the repository's pinned Node 24.18.0.

Target and artifact context

Authored in-memory x64 PE32+/RTR9.1 buffers based on tests/fixtures/nativeaotPe.ts; one or two initialized readable, non-executable data sections. The default-limit case is 67,112,448 bytes, below the 128 MiB source cap. No private binary or external analysis engine was used.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Direct production-parser results from the focused test:

Case                       Candidate bytes   Examined     Omitted   Expected omitted
Single section, work=8              1016          8         1008               1008
Two sections, work=8                2032          8         1008               2024
Two sections, default work     67110896   67108800         1080               2096

All cases returned partial, truncated: true, and truncation_reason: "work-unit-budget". General and RTR-specific omitted fields agreed with each other. The default case used the original 67,108,864-unit limit. Its diagnostic stated that no metadata hydration was attempted because signature coverage was incomplete.

Source reachability: Ghidra load-image derivation calls the snapshot parser with default limits. The parser projects the truncated signature scan into both omitted-byte fields.

Typed outcome or recovery expected

Retain the partial outcome and work-budget reason, with omitted candidate-byte counts covering all eligible sections. The work budget should still bound scanning; accounting for unvisited section extents need not read their contents.

Analysis or suspected cause

Source-derived: findDirectory increments totalScanBytes only when entering a section. Each work-budget failure breaks the entire labeled scan, leaving later sections out of that total. The returned omitted count subtracts examined bytes from the incomplete total.

Precomputing the eligible candidate total, or accounting for remaining sections before returning, would address the arithmetic. No implementation change was made or tested.

Additional context

The fetched upstream main revision c324d31b45fcab77e4ed7320ed60f0bf5a4ba188 has no changes to the relevant parser or provider path compared with the reported revision.

Issue #5882 concerns lost COPY payload bytes when the hydration output buffer grows. This case fails before hydration and requires a different fix. Issue #4445 and PR #4559 concern work-budget scaling and MethodTable base-type indexing; the PR diff does not change the signature scan's omitted-byte accounting. Issues #4750 and #4763 concern PE mapping admission. PR #1679 concerns raw/virtual extent bounds, and PR #759 concerns optional extension result validation.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5890 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior The built-in NativeAOT PE parser understates omitted RTR signature-scan bytes when its work budget stops the scan before the last eligible section. An authored PE with two readable, non-executable data sections returns 1,008 omitted bytes instead of 2,024 with an 8-unit work limit. A larger case using the parser's unchanged default limits returns 1,080 instead of 2,096. Both omit exactly the later section's 1,016 candidate bytes. This was executed directly against `parseNativeAotPe`. The supported entry points are CLI `rea inspect-native-load-image <target> --provider ghidra --json` and MCP `inspect_native_load_image` with provider `ghidra`. Their load-image route reaches this parser according to source; a real Ghidra, CLI, or MCP session was not run. The report correctly remains `partial` and `truncated: true`. ### Reproduction Save this test as `src/domain/native/nativeAotPe.rtrCoverage.test.ts`, then run `npx vitest run --project domain src/domain/native/nativeAotPe.rtrCoverage.test.ts --maxWorkers 1` with the locked dependencies. It extends the existing authored PE fixture while retaining its RTR9.1 header. The first case is a single-section control; the second adds one later eligible section; the third reaches the production default work limit without changing parser limits. ```ts import { expect, it } from "vitest"; import { DEFAULT_NATIVE_AOT_PARSER_LIMITS, NATIVE_AOT_MAX_REPORT_BYTES, parseNativeAotPe, } from "./nativeAotPe.js"; import { nativeAotPeDigest, nativeAotPeFixture, } from "../../../tests/fixtures/nativeaotPe.js"; const fixture = (firstSize: number, laterSize: number) => { const bytes = Buffer.alloc(0x600 + firstSize + laterSize); nativeAotPeFixture().copy(bytes); const data = 0x98 + 0xf0 + 40; bytes.writeUInt32LE(firstSize, data + 8); bytes.writeUInt32LE(firstSize, data + 16); bytes.writeUInt32LE(0x2000 + firstSize + laterSize, 0x98 + 56); if (laterSize > 0) { bytes.writeUInt16LE(3, 0x86); const later = data + 40; bytes.write(".data", later, "ascii"); bytes.writeUInt32LE(laterSize, later + 8); bytes.writeUInt32LE(0x2000 + firstSize, later + 12); bytes.writeUInt32LE(laterSize, later + 16); bytes.writeUInt32LE(0x600 + firstSize, later + 20); bytes.writeUInt32LE(0xc0000040, later + 36); } return bytes; }; it.each([ { firstSize: 0x400, laterSize: 0, workUnits: 8 }, { firstSize: 0x400, laterSize: 0x400, workUnits: 8 }, { firstSize: 64 * 1024 * 1024 + 0x400, laterSize: 0x400, workUnits: DEFAULT_NATIVE_AOT_PARSER_LIMITS.workUnits, }, ])("accounts for all eligible sections: %j", ({ firstSize, laterSize, workUnits }) => { const bytes = fixture(firstSize, laterSize); const summary = parseNativeAotPe( bytes, nativeAotPeDigest(bytes), NATIVE_AOT_MAX_REPORT_BYTES, undefined, workUnits === DEFAULT_NATIVE_AOT_PARSER_LIMITS.workUnits ? undefined : { ...DEFAULT_NATIVE_AOT_PARSER_LIMITS, workUnits }, ).summary; const candidateBytes = firstSize - 8 + (laterSize ? laterSize - 8 : 0); const examined = summary.coverage.rtr_signature_scan_bytes_examined!; const omitted = summary.coverage.rtr_signature_scan_bytes_omitted; console.log(JSON.stringify({ firstSize, laterSize, workUnits, status: summary.status, truncated: summary.truncated, candidateBytes, examined, omitted, expectedOmitted: candidateBytes - examined, coverage: summary.coverage })); expect(summary.status).toBe("partial"); expect(summary.truncated).toBe(true); expect(summary.coverage.truncation_reason).toBe("work-unit-budget"); expect(summary.coverage.scan_bytes_omitted).toBe(omitted); expect(omitted).toBe(candidateBytes - examined); }); ``` The final focused run produced one passing control and two failing omitted-byte assertions. No target executable was launched. ### Expected behavior `rtr_signature_scan_bytes_examined + rtr_signature_scan_bytes_omitted` should cover the candidate positions across every eligible section, including those skipped after the budget exit. The same applies to `scan_bytes_omitted` on this early partial outcome. An aligned 1,024-byte section contributes 1,016 candidate bytes under the existing eight-byte stride and 16-byte-header requirement. The [parser limitation text](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L1411-L1413) promises examined and unexamined coverage for incomplete scans. The [documented evidence boundaries](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/ghidra-nativeaot.md#L104-L123) describe the signature scan and independent resource policies. This does not require removing or raising those limits. ### Impact and scope P2: quantitative coverage understates how much of the admitted image was left unexamined. The two authored multi-section cases each lose the entire later section from the omitted count. Both the general and RTR-specific omitted fields are wrong on the early partial result, but partial status and truncation reason remain accurate. Natural-file frequency and real-provider occurrence are unknown. No recovery workaround was tested. Callers can respect the partial status, but cannot reconstruct the missing section coverage from this report alone. No hydration corruption or false complete result is claimed. ### REA version Source revision `52415975942a4b9c53a817915b313ccc404f280e`; package manifest version 6.7.0. ### Runtime and platform macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3. This run used Node 26 rather than the repository's pinned Node 24.18.0. ### Target and artifact context Authored in-memory x64 PE32+/RTR9.1 buffers based on `tests/fixtures/nativeaotPe.ts`; one or two initialized readable, non-executable data sections. The default-limit case is 67,112,448 bytes, below the 128 MiB source cap. No private binary or external analysis engine was used. ### Interface Other ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Direct production-parser results from the focused test: ```text Case Candidate bytes Examined Omitted Expected omitted Single section, work=8 1016 8 1008 1008 Two sections, work=8 2032 8 1008 2024 Two sections, default work 67110896 67108800 1080 2096 ``` All cases returned `partial`, `truncated: true`, and `truncation_reason: "work-unit-budget"`. General and RTR-specific omitted fields agreed with each other. The default case used the original 67,108,864-unit limit. Its diagnostic stated that no metadata hydration was attempted because signature coverage was incomplete. Source reachability: [Ghidra load-image derivation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/ghidra/GhidraProviderClient.ts#L542-L566) calls the [snapshot parser with default limits](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/ghidra/GhidraNativeAotSnapshotOwner.ts#L270-L287). The parser [projects the truncated signature scan](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L1262-L1298) into both omitted-byte fields. ### Typed outcome or recovery expected Retain the partial outcome and work-budget reason, with omitted candidate-byte counts covering all eligible sections. The work budget should still bound scanning; accounting for unvisited section extents need not read their contents. ### Analysis or suspected cause Source-derived: [`findDirectory`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L633-L669) increments `totalScanBytes` only when entering a section. Each work-budget failure breaks the entire labeled scan, leaving later sections out of that total. The [returned omitted count](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L719-L725) subtracts examined bytes from the incomplete total. Precomputing the eligible candidate total, or accounting for remaining sections before returning, would address the arithmetic. No implementation change was made or tested. ### Additional context The fetched upstream main revision `c324d31b45fcab77e4ed7320ed60f0bf5a4ba188` has no changes to the relevant parser or provider path compared with the reported revision. Issue #5882 concerns lost COPY payload bytes when the hydration output buffer grows. This case fails before hydration and requires a different fix. Issue #4445 and PR #4559 concern work-budget scaling and MethodTable base-type indexing; the PR diff does not change the signature scan's omitted-byte accounting. Issues #4750 and #4763 concern PE mapping admission. PR #1679 concerns raw/virtual extent bounds, and PR #759 concerns optional extension result validation.

bsakaguchi closed this issue

2026-10-31 00:41:58 +00:00

bsakaguchi commented

2026-10-31 00:41:58 +00:00

Author

Owner

@coygeek commented on GitHub (Oct 24, 2026):

Checked the reported fix on main at d87e73f2db.

findDirectory now computes eligible ranges and their candidate-byte contributions for all readable non-executable sections before scanning, using the same aligned-eight-byte stride and 16-byte-header eligibility as the scan. It sums totalScanBytes before a work-budget exit can break the scan, and returns total minus examined as omitted. The early partial projection still publishes that result into both scan_bytes_omitted and rtr_signature_scan_bytes_omitted, retaining partial/truncated/work-budget semantics and no hydration.

Implementation: nativeAotPe.ts, nativeAotPe.ts.

The relevant regression checks in nativeAotPe.budget.test.ts passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

@coygeek commented on GitHub (Oct 24, 2026): Checked the reported fix on `main` at d87e73f2db50be72d33a4b3c7199af35518b441c. findDirectory now computes eligible ranges and their candidate-byte contributions for all readable non-executable sections before scanning, using the same aligned-eight-byte stride and 16-byte-header eligibility as the scan. It sums totalScanBytes before a work-budget exit can break the scan, and returns total minus examined as omitted. The early partial projection still publishes that result into both scan_bytes_omitted and rtr_signature_scan_bytes_omitted, retaining partial/truncated/work-budget semantics and no hydration. Implementation: [nativeAotPe.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/native/nativeAotPe.ts#L733), [nativeAotPe.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/native/nativeAotPe.ts#L751). The relevant regression checks in [nativeAotPe.budget.test.ts](https://github.com/morluto/rea/blob/d87e73f2db50be72d33a4b3c7199af35518b441c/src/domain/native/nativeAotPe.budget.test.ts) passed in the focused test run on this revision. This closure is based on the current source and those checks; provider fixtures do not establish a real-engine run. Closing as completed on main.

bsakaguchi referenced this issue

2026-10-31 01:09:46 +00:00

[PR #3568] [MERGED] fix(setup): retain Hermes YAML scalar source during updates #5832

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3568

No description provided.

Rows
Columns