[GH-ISSUE #5882] [Bug] NativeAOT hydration loses COPY bytes when the output buffer grows #3566

New issue

Closed

opened 2026-10-31 00:41:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:41:56 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5882

Originally assigned to: @kaoru0822-kitauji on GitHub.

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Native analysis or decompilation

Observed behavior

The built-in NativeAOT PE parser loses COPY payload bytes when that command grows the hydration buffer. A synthetic RTR9.1 PE with one 65,537-byte COPY returns derived_overlay.size_bytes: 65537, status: complete and no diagnostics, but the overlay SHA-256 differs from the independently hashed payload. A 65,536-byte control matches.

This was executed directly against parseNativeAotPe; no real Ghidra, CLI or MCP run was performed. The documented inspect-native-load-image --provider ghidra / inspect_native_load_image path uses this parser, as established from source.

Reproduction

Save the following test beside src/domain/native/nativeAotPe.test.ts as nativeAotPe.copyGrowth.test.ts, then run npx vitest run --project domain src/domain/native/nativeAotPe.copyGrowth.test.ts --maxWorkers 1 with the repository's locked dependencies. It expands the existing authored PE fixture, preserves the original tables, and places the COPY stream and hydration output within the expanded mapped section.

import { expect, it } from "vitest";
import { NATIVE_AOT_MAX_REPORT_BYTES, parseNativeAotPe } from "./nativeAotPe.js";
import { nativeAotPeDigest, nativeAotPeFixture } from "../../../tests/fixtures/nativeaotPe.js";

const fixture = (lengths: number[]) => {
  const bytes = Buffer.alloc(0x60600);
  nativeAotPeFixture().copy(bytes);
  const data = 0x98 + 0xf0 + 40;
  bytes.writeUInt32LE(0x60000, data + 8);
  bytes.writeUInt32LE(0x60000, data + 16);
  bytes.writeUInt32LE(0x62000, 0x98 + 56);
  const chunks = lengths.map((length, index) => Buffer.alloc(length, 0xa5 + index));
  let cursor = 0x1604;
  bytes.writeInt32LE(0x40000 - 0x3000, 0x1600);
  bytes.writeBigUInt64LE(0x140003000n, 0x618);
  for (const chunk of chunks) {
    bytes[cursor++] = 31 << 3;
    bytes.writeUIntLE(chunk.length - 28, cursor, 3);
    cursor += 3;
    chunk.copy(bytes, cursor);
    cursor += chunk.length;
  }
  bytes.writeBigUInt64LE(0x140003000n + BigInt(cursor - 0x1600), 0x620);
  return { bytes, expected: Buffer.concat(chunks) };
};

it.each([[65536], [65537], [32768, 32769], [65537, 65537]])(
  "preserves COPY bytes for chunk lengths %j",
  (...lengths: number[]) => {
    const { bytes, expected } = fixture(lengths);
    const original = Buffer.from(bytes);
    const summary = parseNativeAotPe(bytes, nativeAotPeDigest(bytes), NATIVE_AOT_MAX_REPORT_BYTES).summary;
    console.log(JSON.stringify({ lengths, status: summary.status, overlay: summary.derived_overlay, expectedSha256: nativeAotPeDigest(expected), diagnostics: summary.diagnostics }));
    expect(bytes).toEqual(original);
    expect(summary.derived_overlay?.size_bytes).toBe(expected.length);
    expect(summary.derived_overlay?.sha256).toBe(nativeAotPeDigest(expected));
  },
);

Observed once with this final fixture: one passing boundary control and three failing digest assertions. All four source-immutability and output-size assertions passed. The last case exercises two growth events.

Expected behavior

The overlay must contain the concatenated COPY payloads, and its SHA-256 must equal the digest of those intended bytes. Growing internal storage must preserve them.

The documented parser contract identifies the isolated rehydrated overlay by address, size and SHA-256 and keeps source bytes immutable.

Impact and scope

P2: incorrect bytes enter the parser-owned metadata overlay for a COPY command that triggers buffer growth. The exposed digest identifies those incorrect contents while the report can still claim complete recovery. The synthetic cases prove a bounded metadata/evidence integrity failure; they do not measure real-file frequency or demonstrate lost MethodTables or frozen strings. Downstream discovery reads this overlay, so effects on those results require further testing. No workaround was tested.

REA version

Source revision 52415975942a4b9c53a817915b313ccc404f280e; package manifest version 6.7.0.

Runtime and platform

macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3.

Target and artifact context

Authored in-memory x64 PE32+/RTR9.1 fixture derived from tests/fixtures/nativeaotPe.ts; no target executable was launched. COPY output sizes: 65,536, 65,537 and 131,074 bytes.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

The final focused run returned:

COPY payload lengths     status     overlay size   digest matches expected
[65536]                  complete   65536          yes
[65537]                  complete   65537          no
[32768,32769]            complete   65537          no
[65537,65537]            complete   131074         no
All diagnostics arrays were empty; source bytes remained unchanged.

For [65537], hashing the expected payload gave a3253bfa7d790dc6f5d41b75421bcc2c606e8043f9fad4331e8aae0a6c504cfa; the parser reported 6e7867b3f5a2bccb3c2f0f461ce3f691f6382bae2123f448b5cf87c27cd64277. The incorrect digest is not stable across runs because growth uses uninitialized storage.

Source reachability: Ghidra load-image derivation calls the snapshot parser, which hydrates DEHYDRATED_DATA.

Typed outcome or recovery expected

A successful overlay descriptor should identify the actual result of the hydration command stream. If hydration cannot be completed, it should return the existing partial outcome with a meaningful diagnostic rather than complete recovery over incorrect bytes.

Analysis or suspected cause

Source-derived cause: chunk.copy(output, ensureOutput(chunk.length)) evaluates the destination argument before ensureOutput replaces output with a larger Buffer. The triggering COPY therefore writes to the previous Buffer after its existing contents have already been transferred to the retained expanded Buffer. The new payload does not reach the retained overlay. Other opcode writers reserve capacity in a separate statement before accessing output.

The overlay hash and registration use the new Buffer and advanced length. Reserving capacity before evaluating the copy destination should avoid this stale reference; that change was not applied or tested here.

Additional context

Current upstream source was fetched and the parser's relevant lines were unchanged. Related reports #4445 / PR #4559 concern work budgets and base-type indexing; #4750 and #4763 concern PE mapping admission. Their contracts and fixes differ from this COPY growth defect.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5882 Originally assigned to: @kaoru0822-kitauji on GitHub. ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Native analysis or decompilation ### Observed behavior The built-in NativeAOT PE parser loses COPY payload bytes when that command grows the hydration buffer. A synthetic RTR9.1 PE with one 65,537-byte COPY returns `derived_overlay.size_bytes: 65537`, `status: complete` and no diagnostics, but the overlay SHA-256 differs from the independently hashed payload. A 65,536-byte control matches. This was executed directly against `parseNativeAotPe`; no real Ghidra, CLI or MCP run was performed. The documented `inspect-native-load-image --provider ghidra` / `inspect_native_load_image` path uses this parser, as established from source. ### Reproduction Save the following test beside `src/domain/native/nativeAotPe.test.ts` as `nativeAotPe.copyGrowth.test.ts`, then run `npx vitest run --project domain src/domain/native/nativeAotPe.copyGrowth.test.ts --maxWorkers 1` with the repository's locked dependencies. It expands the existing authored PE fixture, preserves the original tables, and places the COPY stream and hydration output within the expanded mapped section. ```ts import { expect, it } from "vitest"; import { NATIVE_AOT_MAX_REPORT_BYTES, parseNativeAotPe } from "./nativeAotPe.js"; import { nativeAotPeDigest, nativeAotPeFixture } from "../../../tests/fixtures/nativeaotPe.js"; const fixture = (lengths: number[]) => { const bytes = Buffer.alloc(0x60600); nativeAotPeFixture().copy(bytes); const data = 0x98 + 0xf0 + 40; bytes.writeUInt32LE(0x60000, data + 8); bytes.writeUInt32LE(0x60000, data + 16); bytes.writeUInt32LE(0x62000, 0x98 + 56); const chunks = lengths.map((length, index) => Buffer.alloc(length, 0xa5 + index)); let cursor = 0x1604; bytes.writeInt32LE(0x40000 - 0x3000, 0x1600); bytes.writeBigUInt64LE(0x140003000n, 0x618); for (const chunk of chunks) { bytes[cursor++] = 31 << 3; bytes.writeUIntLE(chunk.length - 28, cursor, 3); cursor += 3; chunk.copy(bytes, cursor); cursor += chunk.length; } bytes.writeBigUInt64LE(0x140003000n + BigInt(cursor - 0x1600), 0x620); return { bytes, expected: Buffer.concat(chunks) }; }; it.each([[65536], [65537], [32768, 32769], [65537, 65537]])( "preserves COPY bytes for chunk lengths %j", (...lengths: number[]) => { const { bytes, expected } = fixture(lengths); const original = Buffer.from(bytes); const summary = parseNativeAotPe(bytes, nativeAotPeDigest(bytes), NATIVE_AOT_MAX_REPORT_BYTES).summary; console.log(JSON.stringify({ lengths, status: summary.status, overlay: summary.derived_overlay, expectedSha256: nativeAotPeDigest(expected), diagnostics: summary.diagnostics })); expect(bytes).toEqual(original); expect(summary.derived_overlay?.size_bytes).toBe(expected.length); expect(summary.derived_overlay?.sha256).toBe(nativeAotPeDigest(expected)); }, ); ``` Observed once with this final fixture: one passing boundary control and three failing digest assertions. All four source-immutability and output-size assertions passed. The last case exercises two growth events. ### Expected behavior The overlay must contain the concatenated COPY payloads, and its SHA-256 must equal the digest of those intended bytes. Growing internal storage must preserve them. The [documented parser contract](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/ghidra-nativeaot.md#L104-L119) identifies the isolated rehydrated overlay by address, size and SHA-256 and keeps source bytes immutable. ### Impact and scope P2: incorrect bytes enter the parser-owned metadata overlay for a COPY command that triggers buffer growth. The exposed digest identifies those incorrect contents while the report can still claim complete recovery. The synthetic cases prove a bounded metadata/evidence integrity failure; they do not measure real-file frequency or demonstrate lost MethodTables or frozen strings. Downstream discovery reads this overlay, so effects on those results require further testing. No workaround was tested. ### REA version Source revision `52415975942a4b9c53a817915b313ccc404f280e`; package manifest version 6.7.0. ### Runtime and platform macOS 27.0.1 arm64, Node v26.12.0, Vitest 5.0.3. ### Target and artifact context Authored in-memory x64 PE32+/RTR9.1 fixture derived from `tests/fixtures/nativeaotPe.ts`; no target executable was launched. COPY output sizes: 65,536, 65,537 and 131,074 bytes. ### Interface Other ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence The final focused run returned: ```text COPY payload lengths status overlay size digest matches expected [65536] complete 65536 yes [65537] complete 65537 no [32768,32769] complete 65537 no [65537,65537] complete 131074 no All diagnostics arrays were empty; source bytes remained unchanged. ``` For `[65537]`, hashing the expected payload gave `a3253bfa7d790dc6f5d41b75421bcc2c606e8043f9fad4331e8aae0a6c504cfa`; the parser reported `6e7867b3f5a2bccb3c2f0f461ce3f691f6382bae2123f448b5cf87c27cd64277`. The incorrect digest is not stable across runs because growth uses uninitialized storage. Source reachability: [Ghidra load-image derivation](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/ghidra/GhidraProviderClient.ts#L542-L566) calls the [snapshot parser](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/ghidra/GhidraNativeAotSnapshotOwner.ts#L270-L287), which [hydrates DEHYDRATED_DATA](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L1355-L1359). ### Typed outcome or recovery expected A successful overlay descriptor should identify the actual result of the hydration command stream. If hydration cannot be completed, it should return the existing partial outcome with a meaningful diagnostic rather than complete recovery over incorrect bytes. ### Analysis or suspected cause Source-derived cause: [`chunk.copy(output, ensureOutput(chunk.length))`](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L833-L872) evaluates the destination argument before `ensureOutput` replaces `output` with a larger Buffer. The triggering COPY therefore writes to the previous Buffer after its existing contents have already been transferred to the retained expanded Buffer. The new payload does not reach the retained overlay. Other opcode writers reserve capacity in a separate statement before accessing `output`. The [overlay hash and registration](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/domain/native/nativeAotPe.ts#L981-L985) use the new Buffer and advanced length. Reserving capacity before evaluating the copy destination should avoid this stale reference; that change was not applied or tested here. ### Additional context Current upstream source was fetched and the parser's relevant lines were unchanged. Related reports #4445 / PR #4559 concern work budgets and base-type indexing; #4750 and #4763 concern PE mapping admission. Their contracts and fixes differ from this COPY growth defect.

bsakaguchi closed this issue

2026-10-31 00:41:56 +00:00

bsakaguchi referenced this issue

2026-10-31 01:09:44 +00:00

[PR #3566] [CLOSED] fix(javascript): ignore exports writes after the alias is detached #5829